iJSrSSKJr SSKrSSKJrJr SSKJr SSK J r SSK J r "SS \ 5r "S S 5r"S S \5r"SS\5r"SS5rg)zMarkdown text splitters.) annotationsN)Any TypedDict)Document)Language)RecursiveCharacterTextSplitterc0^\rSrSrSrSU4SjjrSrU=r$)MarkdownTextSplitterz=Attempts to split the text along Markdown-formatted headings.c h>UR[R5n[TU]"SSU0UD6 g)z"Initialize a MarkdownTextSplitter. separatorsN)get_separators_for_languagerMARKDOWNsuper__init__)selfkwargsr __class__s q/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/langchain_text_splitters/markdown.pyrMarkdownTextSplitter.__init__s.55h6G6GH  9J9&9r)rrreturnNone)__name__ __module__ __qualname____firstlineno____doc__r__static_attributes__ __classcell__)rs@rr r sG::rr c\\rSrSrSrS S SjjrS SjrS SjrS SjrSr g)MarkdownHeaderTextSplitterz4Splitting markdown files based on specified headers.Nc^X l[USSS9UlX0lU=(d 0Ulg)aCreate a new MarkdownHeaderTextSplitter. Args: headers_to_split_on: Headers we want to track return_each_line: Return each line w/ associated headers strip_headers: Strip split headers from the content of the chunk custom_header_patterns: Optional dict mapping header patterns to their levels. For example: {"**": 1, "***": 2} to treat **Header** as level 1 and ***Header*** as level 2 headers. c[US5$)Nr)len)splits r5MarkdownHeaderTextSplitter.__init__..0s 3uQx=rT)keyreverseN)return_each_linesortedheaders_to_split_on strip_headerscustom_header_patterns)rr/r-r0r1s rr#MarkdownHeaderTextSplitter.__init__s6$!1$* %@$$  +&<&B#rc N^TUR;ag[R"T5nSUSUSUSUS3 n[R"XA5nU(aQUR S5R 5nU(a+[ U4SjURS S 555(dg g) zCheck if line matches a custom header pattern. Args: line: The line to check sep: The separator pattern to match Returns: True if the line matches the custom pattern format F^z(?!z )(.+?)(?# UH oT;v M g7fNr).0cseps r ?MarkdownHeaderTextSplitter._is_custom_header..Rs"N5M85Ms T)r1reescapematchgroupstripallreplace)rliner< escaped_seppatternrCcontents ` r_is_custom_header,MarkdownHeaderTextSplitter._is_custom_header7s d11 1iin  }C }J{m1[MQR S ' kk!n**,Gs"NW__S"5M"NNNrc/nUHnU(a'USSUS:XaUSS==SUS-- ss'M1U(aUSSUS:wav[USS5[US5:aUUSSRS5SSS:Xa4UR(d#USS==SUS-- ss'USUSS'MURU5 M UVs/sHn[ USUSS9PM sn$s snf) zlCombine lines with common metadata into chunks. Args: lines: Line of text / associated header metadata metadatarKz  r# page_contentrP)r'r(r0appendr)rlinesaggregated_chunksrHchunks raggregate_lines_to_chunks4MarkdownHeaderTextSplitter.aggregate_lines_to_chunksVs8 -/D!%b)*5j9II ""%i0FT)_4LL0!%b)*5j9II)"-j9:SjAQ=RR%b))4::4@DQG3N**""%i0FT)_4LL0484D!"%j1"((.9@+ * % "2U:=N O*   sC8cJURS5n/n/n0n/n0nSnSn UGH n U R5n SR[[R U 55n U(dKU R S5(aU RS5S:XaSnSn O5U R S5(aSnSn OU R U 5(aSnSn U(aURU 5 MURGHupU R U 5=(a/ [U 5[U 5:H=(d U [U 5S:HnURX5nU(d U(dMnU bXR;aURU nOU RS 5nU(aNUS S U:aBUR5nUS U;aURUS 5 U(aUS S U:aMBU(a'U [U 5[U 5*R5nOU [U 5S R5nUU US.nURU5 USX}'U(aAURSRU5UR5S.5 UR5 UR (dURU 5 Oc U (aURU 5 OHU(aAURSRU5UR5S.5 UR5 UR5nGM U(a#URSRU5US.5 UR"(dUR%U5$UVs/sHn['USUSS9PM sn$s snf)z4Split markdown file. Args: text: Markdown file rQFr@z```r7Tz~~~r?NrRrOlevelname)r\r]datar^)rKrPrKrPrS)r(rEjoinfilterstr isprintable startswithcountrUr/r'rLr1popcopyclearr0r-rYr)rtextrVlines_with_metadatacurrent_contentcurrent_metadata header_stackinitial_metadata in_code_block opening_fencerH stripped_liner<r]is_standard_headeris_custom_headercurrent_header_level popped_header header_textheaderrXs r split_text%MarkdownHeaderTextSplitter.split_texts 4 .0%'+-)+ +-  D JJLMGGF3??M$JKM  ++E22}7J7J57QUV7V$(M$)M"--e44$(M$)M))-88 % " &&}5"55 %2%=%=c%B& &#c(2TmCH6MQT6T# $(#9#9-#M &)9)9'"="==373N3Ns3S03699S>0) ,R 0 9=Q Q-9,<,<,>M -V48HH 0 4 4]65J K) ,R 0 9=Q Q,+8CCH9*M*S*S*UK+8C *C*I*I*KK&:$($/. %++F317(.'+22+/99_+E,<,A,A,C (--/--'..}=C6F!#**=9$'..'+yy'A(8(=(=(? $))+/446 IL   & &#yy9 0 $$112EF F- , % "2U:=N O,   sN )r1r/r-r0)FTN) r/zlist[tuple[str, str]]r-boolr0ryr1zdict[str, int] | Nonerr)rHrar<rarry)rVzlist[LineType]rlist[Document]rhrarrz) rrrrrrrLrYrwr rrrr#r#s\> "'"8< C2CC C !6 C  C:>) VI rr#c.\rSrSr%SrS\S'S\S'Srg) LineTypei zLine type as typed dict.zdict[str, str]rPrarKrNrrrrr__annotations__r rrrr}r} s" Lrr}c8\rSrSr%SrS\S'S\S'S\S'Srg ) HeaderTypeizHeader type as typed dict.intr\rar]r^rNr~rrrrrs$ J I Irrc\rSrSrSrSSSSSSS .rSSS jjrSS jrSS jrSSjr SSjr SSjr SSjr SSjr Srg )&ExperimentalMarkdownSyntaxTextSplitteria0An experimental text splitter for handling Markdown syntax. This splitter aims to retain the exact whitespace of the original text while extracting structured metadata, such as headers. It is a re-implementation of the MarkdownHeaderTextSplitter with notable changes to the approach and additional features. Key Features: * Retains the original whitespace and formatting of the Markdown text. * Extracts headers, code blocks, and horizontal rules as metadata. * Splits out code blocks and includes the language in the "Code" metadata key. * Splits text on horizontal rules (`---`) as well. * Defaults to sensible splitting behavior, which can be overridden using the `headers_to_split_on` parameter. Example: ```python headers_to_split_on = [ ("#", "Header 1"), ("##", "Header 2"), ] splitter = ExperimentalMarkdownSyntaxTextSplitter( headers_to_split_on=headers_to_split_on ) chunks = splitter.split(text) for chunk in chunks: print(chunk) ``` This class is currently experimental and subject to change based on feedback and further development. zHeader 1zHeader 2zHeader 3zHeader 4zHeader 5zHeader 6)rRz##z###z####z#####z######Nc/Ul[SS9Ul/UlX0lU(a[ U5UlOURUlX lg)aInitialize the text splitter with header splitting and formatting options. This constructor sets up the required configuration for splitting text into chunks based on specified headers and formatting preferences. Args: headers_to_split_on (Union[list[tuple[str, str]], None]): A list of tuples, where each tuple contains a header tag (e.g., "h1") and its corresponding metadata key. If `None`, default headers are used. return_each_line (bool): Whether to return each line as an individual chunk. Defaults to `False`, which aggregates lines into larger chunks. strip_headers (bool): Whether to exclude headers from the resulting chunks. r@rTN) chunksr current_chunkcurrent_header_stackr0dictsplittable_headersDEFAULT_HEADER_KEYSr-)rr/r-r0s rr/ExperimentalMarkdownSyntaxTextSplitter.__init__HsK*') %26;=!* &*+>&?D #&*&>&>D # 0rc URR5 [SS9UlURR5 UR SS9nU(GayUR S5nURU5nURU5nURU5nU(a}UR5 UR(dUR=RU- sl [URS55nURS5nURXx5 OU(aiUR5 UR!X25URl URS5URR"S'UR5 O7U(aUR5 OUR=RU- sl U(aGMyUR5 UR$(arURV V s/sHXn U RR 5H6n U (dM U R'5(aM#[XR"S 9PM8 MZ sn n $UR$s sn n f) a Split the input text into structured chunks. This method processes the input text line by line, identifying and handling specific patterns such as headers, code blocks, and horizontal rules to split it into structured chunks based on headers, code blocks, and horizontal rules. Args: text: The input text to be split into chunks. Returns: A list of `Document` objects representing the structured chunks of the input text. If `return_each_line` is enabled, each line is returned as a separate `Document`. r@rT)keependsrr7CoderS)rrgrrr splitlinesre _match_header _match_code _match_horz_complete_chunk_docr0rTr'rD_resolve_header_stack_resolve_code_chunkrPr-isspace) rrh raw_linesraw_line header_match code_match horz_match header_depthrurXrHs rrw1ExperimentalMarkdownSyntaxTextSplitter.split_texths" %26 !!'')OOTO2  }}Q'H--h7L))(3J))(3J((*))&&33x?3 #<#5#5a#89 *003 **<E((*262J2J3""/7A6F6Fq6I""++F3((*((*""//8;/3i6   "  "[[(E!..99;DE!% Ed^^D;E(  {{ s)I20I2I2c[UR5H#unupEXA:dMURSUUl O URRX45 gr9) enumeraterrU)rrruidepth_s rr'>>Lrc~SVs/sHn[R"X!5PM nn[SU5S5$s snf)N)z^```(.*)z^~~~(.*)c36# UHo(dM Uv M g7fr9rr:rCs rr=EExperimentalMarkdownSyntaxTextSplitter._match_code..9u5UU  rArCnextrrHrulematchess rr2ExperimentalMarkdownSyntaxTextSplitter._match_codes:4NO4ND288D'4NO994@@P :c~SVs/sHn[R"X!5PM nn[SU5S5$s snf)N)z ^\*\*\*+\nz^---+\nz^___+\nc36# UHo(dM Uv M g7fr9rrs rr=EExperimentalMarkdownSyntaxTextSplitter._match_horz..rrrrs rr2ExperimentalMarkdownSyntaxTextSplitter._match_horzsB-T -TTBHHT -T  994@@ r)rrrr-rr0)NFT)r/zlist[tuple[str, str]] | Noner-ryr0ryrrr{)rrrurarr)rrarz list[str]rra)rr)rHrarzre.Match[str] | None)rrrrrrrrwrrrrrrr rrrrrs F =A!&" 1911 1  1@=~F 7AArr)r __future__rrAtypingrrlangchain_core.documentsrlangchain_text_splitters.baser"langchain_text_splitters.characterrr r#r}rrrrrrsZ" !-2M:9:s s lyxAxAr