i2SrSSKJr SSKrSSKrSSKJrJr SSKJ r SSK J r SSK J r JrJrJr SSKJrJr SS KJrJr \ (aSS KJrJrJrJr SS KJr SSKrS rSSK!J"r" S r#\RH"\%5r&\"SSS9r'"SS\\5r("SS\(5r)"SS\*\ 5r+\ "S S9"SS55r,SSjr-g!\ a S rNhf=f!\ a S r#Nmf=f)zText splitter base interface.) annotationsN)ABCabstractmethod) dataclass)Enum) TYPE_CHECKINGAnyLiteralTypeVar)BaseDocumentTransformerDocument)Selfoverride)Callable CollectionIterableSequence)SetTF)PreTrainedTokenizerBaseTS TextSplitter)boundc\rSrSrSrSS\SSS4SSjjr\SSj5rSSS jjr SS jr SS jr SS jr \ SSj5r\ SS \"5S4SSjj5r\SSj5rSrg )r,z)Interface for splitting text into chunks.iFTcUS::aSU3n[U5eUS:aSU3n[U5eX!:aSUSUS3n[U5eXlX lX0lX@lXPlX`lg)aCreate a new TextSplitter. Args: chunk_size: Maximum size of chunks to return chunk_overlap: Overlap in characters between chunks length_function: Function that measures the length of given chunks keep_separator: Whether to keep the separator and where to place it in each corresponding chunk (True='start') add_start_index: If `True`, includes chunk's start index in metadata strip_whitespace: If `True`, strips whitespace from the start and end of every document rzchunk_size must be > 0, got z chunk_overlap must be >= 0, got zGot a larger chunk overlap (z) than chunk size (z), should be smaller.N) ValueError _chunk_size_chunk_overlap_length_function_keep_separator_add_start_index_strip_whitespace)self chunk_size chunk_overlaplength_functionkeep_separatoradd_start_indexstrip_whitespacemsgs m/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/langchain_text_splitters/base.py__init__TextSplitter.__init__/s* ?0 =CS/ ! 1 4]ODCS/ !  %.}o><46 S/ !%+ /- /!1cg)z$Split text into multiple components.N)r$texts r, split_textTextSplitter.split_textWsr/Nc U=(d 0/[U5-n/n[U5HupVSnSnURU5Hn [R"X55n UR (a<Xx-UR - n URU [SU 55nXzS'[U 5n[XS9n URU 5 M M U$)z&Create documents from a list of texts.r start_index) page_contentmetadata) len enumerater3copydeepcopyr"rfindmaxr append) r$texts metadatas metadatas_ documentsir2indexprevious_chunk_lenchunkr8offsetnew_docs r,create_documentsTextSplitter.create_documents[s32$U"3   'GAE!" .==7(("7$:M:MMF IIeSF^.B]J JE=>ook5 ? KK  r/c ^[(d Sn[U5e[T[5(d Sn[U5eSU4SjjnU"SSU0UD6$)z>Text splitter that uses HuggingFace tokenizer to count length.z`Could not import transformers python package. Please install it with `pip install transformers`.zATokenizer received was not an instance of PreTrainedTokenizerBasec8>[TRU55$rQ)r9tokenizer2 tokenizers r,_huggingface_tokenizer_lengthNTextSplitter.from_huggingface_tokenizer.._huggingface_tokenizer_lengthsy))$/0 0r/r'r2strreturnintr1)_HAS_TRANSFORMERSr isinstancer)clsrhkwargsr+ris ` r,from_huggingface_tokenizer'TextSplitter.from_huggingface_tokenizersZ ! E S/ !)%<==UCS/ ! 1K#@KFKKr/gpt2allc ^^^ [(d Sn[U5eUb[R"U5m O[R"U5m SUUU 4Sjjn[ U[ 5(a UUTTS.n0UEUEnU"SSU0UD6$)z9Text splitter that uses tiktoken encoder to count length.zCould not import tiktoken python package. This is needed in order to calculate max_tokens_for_prompt. Please install it with `pip install tiktoken`.c8>[TRUTTS95$N)allowed_specialdisallowed_special)r9encode)r2rzr{encs r,_tiktoken_encoder=TextSplitter.from_tiktoken_encoder.._tiktoken_encoders* $3'9 r/) encoding_name model_namerzr{r'rkr1) _HAS_TIKTOKEN ImportErrortiktokenencoding_for_model get_encoding issubclassTokenTextSplitter) rqrrrzr{rrr+r~ extra_kwargsr}s `` @r,from_tiktoken_encoder"TextSplitter.from_tiktoken_encoders}A  c" "  !--j9C'' 6C   c, - -!.(#2&8 L 0/,/F?#4???r/c 6UR[U55$)z2Transform sequence of documents by splitting them.)rNlist)r$rCrrs r,transform_documents TextSplitter.transform_documentss ##DO44r/)r"rrr!r r#)r%rnr&rnr'zCallable[[str], int]r(zbool | Literal['start', 'end']r)boolr*rrmNoner2rlrm list[str]rQ)r@rrAzlist[dict[Any, Any]] | Nonermlist[Document])rCzIterable[Document]rmr)rTrrUrlrm str | None)r\z Iterable[str]rUrlrmr)rhrrrr rmr) rrlrrrz!Literal['all'] | AbstractSet[str]r{ Literal['all'] | Collection[str]rrr rmr)rCSequence[Document]rrr rmr)__name__ __module__ __qualname____firstlineno____doc__r9r-rr3rJrNrVrb classmethodrssetrrr__static_attributes__r1r/r,rr,sp3 039> %!%&2&2&2. &2 7 &2  &2&2 &2P33JN+F (A *XL/L;>L LL($!%=@U?D (@(@(@; (@ = (@  (@ (@(@T5+57:5 55r/cf^\rSrSrSrSS\"5S4S U4SjjjrS SjrSrU=r $) rz/Splitting text to tokens using model tokenizer.ruNrvc >[TU]"S0UD6 [(d Sn[U5eUb[R "U5nO[R "U5nXplX0lX@l g)zCreate a new TextSplitter.zCould not import tiktoken python package. This is needed in order to for TokenTextSplitter. Please install it with `pip install tiktoken`.Nr1) superr-rrrrr _tokenizer_allowed_special_disallowed_special) r$rrrzr{rrr+r} __class__s r,r-TokenTextSplitter.__init__si "6"}A  c" "  !--j9C'' 6C /#5 r/c^SU4Sjjn[TRTRTRRUS9n[ XS9$)aSplits the input text into smaller chunks based on tokenization. This method uses a custom tokenizer configuration to encode the input text into tokens, processes the tokens in chunks of a specified size with overlap, and decodes them back into text chunks. The splitting is performed using the `split_text_on_tokens` function. Args: text: The input text to be split into smaller chunks. Returns: A list of text chunks, where each chunk is derived from a portion of the input text based on the tokenization and chunking rules. cb>TRRUTRTRS9$ry)rr|rr)_textr$s r,_encode-TokenTextSplitter.split_text.._encodes4??)) $ 5 5#'#;#;* r/)r&tokens_per_chunkdecoder|rg)rrlrmz list[int]) Tokenizerrrrrsplit_text_on_tokens)r$r2rrhs` r,r3TokenTextSplitter.split_textsC  --!--??))  $CCr/)rrr) rrlrrrzrr{rrrr rmrr) rrrrrrr-r3r __classcell__)rs@r,rrsf9$!%=@U?D 666; 6 = 6  6 664DDr/rc\rSrSrSrSrSrSrSrSr Sr S r S r S r S rS rSrSrSrSrSrSrSrSrSrSrSrSrSrSrSrSrSr g) Languagei/z"Enum of the programming languages.cppgojavakotlinjstsphpprotopythonrstrubyrustscalaswiftmarkdownlatexhtmlsolcsharpcobolcluaperlhaskellelixir powershell visualbasic6r1N)!rrrrrCPPGOJAVAKOTLINJSrPHPPROTOPYTHONRSTRUBYRUSTSCALASWIFTMARKDOWNLATEXHTMLSOLCSHARPCOBOLCLUAPERLHASKELLELIXIR POWERSHELL VISUALBASIC6rr1r/r,rr/s, C B D F B B C E F C D D E EH E D C F E A C DG FJ!Lr/r)frozencH\rSrSr%SrS\S'S\S'S\S'S\S 'S rg ) riOzTokenizer data class.rnr&rzCallable[[list[int]], str]rzCallable[[str], list[int]]r|r1N)rrrrr__annotations__rr1r/r,rrOs)*, &&= &&=r/rc/nURU5nSnURUR::a Sn[U5eU[ U5:a[ XAR-[ U55nX4UnU(dU$UR U5nU(aURU5 U[ U5:XaU$XARUR- - nU[ U5:aMU$)z6Split incoming text and return chunks using tokenizer.rz3tokens_per_chunk must be greater than chunk_overlap)r|rr&rr9minrr?) r2rhr\ input_ids start_idxr+cur_idx chunk_idsdecodeds r,rr]sF  &II!!Y%<%<<Co c)n $i"<"<rs#" #!G,HH2ML   8 $ T(B5*CB5J;D ;D|"sD"@ $ > > >A Ms$B?"C ?C  C  CC