9i2dZddlmZddlZddlZddlmZmZddlm Z ddl m Z ddl m Z mZmZmZddlmZmZdd lmZmZe rdd lmZmZmZmZdd lmZ ddlZd Z ddl!m"Z"d Z#ejHe%Z&eddZ'GddeeZ(Gdde(Z)Gdde*e Z+e d GddZ,ddZ-y#e $rd ZYlwxYw#e $rd Z#YpwxYw)zText splitter base interface.) annotationsN)ABCabstractmethod) dataclass)Enum) TYPE_CHECKINGAnyLiteralTypeVar)BaseDocumentTransformerDocument)Selfoverride)Callable CollectionIterableSequence)SetTF)PreTrainedTokenizerBaseTS TextSplitter)boundceZdZdZddedddf ddZeddZ d dd Zdd Z dd Z dd Z e dd Z e ddedf ddZe ddZy)rz)Interface for splitting text into chunks.iFTc|dkrd|}t||dkrd|}t|||kDrd|d|d}t|||_||_||_||_||_||_y)adCreate a new TextSplitter. Args: chunk_size: Maximum size of chunks to return chunk_overlap: Overlap in characters between chunks length_function: Function that measures the length of given chunks keep_separator: Whether to keep the separator and where to place it in each corresponding chunk (True='start') add_start_index: If `True`, includes chunk's start index in metadata strip_whitespace: If `True`, strips whitespace from the start and end of every document rzchunk_size must be > 0, got z chunk_overlap must be >= 0, got zGot a larger chunk overlap (z) than chunk size (z), should be smaller.N) ValueError _chunk_size_chunk_overlap_length_function_keep_separator_add_start_index_strip_whitespace)self chunk_size chunk_overlaplength_functionkeep_separatoradd_start_indexstrip_whitespacemsgs g/var/www/html/backtest/airagagent/rag_env/lib/python3.12/site-packages/langchain_text_splitters/base.py__init__zTextSplitter.__init__/s* ?0 =CS/ ! 1 4]ODCS/ ! : %.}o><46 S/ !%+ /- /!1cy)z$Split text into multiple components.N)r#texts r+ split_textzTextSplitter.split_textWsr-Nc |xsigt|z}g}t|D]\}}d}d}|j|D]} tj||} |j r>||z|j z } |j| td| }|| d<t| }t| | } |j| |S)z&Create documents from a list of texts.r start_index) page_contentmetadata) len enumerater1copydeepcopyr!rfindmaxr append) r#texts metadatas metadatas_ documentsir0indexprevious_chunk_lenchunkr5offsetnew_docs r+create_documentszTextSplitter.create_documents[s32$U"3   ' *GAtE!" . *==A7(("%77$:M:MMF IIeSF^)r<r4r5rG)r#r@r=r>docs r+split_documentszTextSplitter.split_documentsosVry +C LL)) *   S\\ * +$$Ui$@@r-ch|j|}|jr|j}|xsdSN)joinr"strip)r#docs separatorr0s r+ _join_docszTextSplitter._join_docsws.~~d#  ! !::.B]J JE= K>ook95 ? KK  r-c ts d}t|tts d}t|dfd }|dd|i|S)z>Text splitter that uses HuggingFace tokenizer to count length.z`Could not import transformers python package. Please install it with `pip install transformers`.zATokenizer received was not an instance of PreTrainedTokenizerBasec8tj|SrL)r6tokenizer0 tokenizers r+_huggingface_tokenizer_lengthzNTextSplitter.from_huggingface_tokenizer.._huggingface_tokenizer_lengthsy))$/0 0r-r&r0strreturnintr/)_HAS_TRANSFORMERSr isinstancer)clsrakwargsr*rbs ` r+from_huggingface_tokenizerz'TextSplitter.from_huggingface_tokenizersT !E S/ !)%<=UCS/ ! 1K#@KFKKr-gpt2allc  ts d}t||tj| ntj| d fd }t |t r ||d}i||}|dd|i|S)z9Text splitter that uses tiktoken encoder to count length.zCould not import tiktoken python package. This is needed in order to calculate max_tokens_for_prompt. Please install it with `pip install tiktoken`.c>tj|SN)allowed_specialdisallowed_special)r6encode)r0rqrrencs r+_tiktoken_encoderz=TextSplitter.from_tiktoken_encoder.._tiktoken_encoders* $3'9 r-) encoding_name model_namerqrrr&rcr/) _HAS_TIKTOKEN ImportErrortiktokenencoding_for_model get_encoding issubclassTokenTextSplitter) rirvrwrqrrrjr*ru extra_kwargsrts `` @r+from_tiktoken_encoderz"TextSplitter.from_tiktoken_encodersA  c" "  !--j9C'' 6C  c, -!.(#2&8 L 0/,/F?#4???r-c 6|jt|S)z2Transform sequence of documents by splitting them.)rJlist)r#r@rjs r+transform_documentsz TextSplitter.transform_documentss ##DO44r-)r$rfr%rfr&zCallable[[str], int]r'zbool | Literal['start', 'end']r(boolr)rreNoner0rdre list[str]rL)r=rr>zlist[dict[Any, Any]] | Nonerelist[Document])r@zIterable[Document]rer)rOrrPrdre str | None)rVz Iterable[str]rPrdrer)rarrjr rer) rvrdrwrrq!Literal['all'] | AbstractSet[str]rr Literal['all'] | Collection[str]rjr rer)r@Sequence[Document]rjr rer)__name__ __module__ __qualname____doc__r6r,rr1rGrJrQr\ classmethodrksetrrrr/r-r+rr,sp3 039> %!%&2&2&2. &2 7 &2  &2&2 &2P33JN+F (A *XL/L;>L LL($!%=@U?D (@(@(@; (@ = (@  (@ (@(@T5+57:5 55r-cVeZdZdZddedf dfd ZddZxZS) r~z/Splitting text to tokens using model tokenizer.rlNrmc t|di|ts d}t||t j |}nt j |}||_||_||_ y)zCreate a new TextSplitter.zCould not import tiktoken python package. This is needed in order to for TokenTextSplitter. Please install it with `pip install tiktoken`.Nr/) superr,rxryrzr{r| _tokenizer_allowed_special_disallowed_special) r#rvrwrqrrrjr*rt __class__s r+r,zTokenTextSplitter.__init__sm "6"A  c" "  !--j9C'' 6C /#5 r-cdfd }tjjjj|}t ||S)anSplits the input text into smaller chunks based on tokenization. This method uses a custom tokenizer configuration to encode the input text into tokens, processes the tokens in chunks of a specified size with overlap, and decodes them back into text chunks. The splitting is performed using the `split_text_on_tokens` function. Args: text: The input text to be split into smaller chunks. Returns: A list of text chunks, where each chunk is derived from a portion of the input text based on the tokenization and chunking rules. chjj|jjSrp)rrsrr)_textr#s r+_encodez-TokenTextSplitter.split_text.._encodes4??)) $ 5 5#'#;#;* r-)r%tokens_per_chunkdecodersr`)rrdrez list[int]) Tokenizerrrrrsplit_text_on_tokens)r#r0rras` r+r1zTokenTextSplitter.split_textsE  --!--??))  $CCr-) rvrdrwrrqrrrrrjr rerr)rrrrrr,r1 __classcell__)rs@r+r~r~s[9$!%=@U?D 666; 6 = 6  6 64Dr-r~c|eZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZy)Languagez"Enum of the programming languages.cppgojavakotlinjstsphpprotopythonrstrubyrustscalaswiftmarkdownlatexhtmlsolcsharpcobolcluaperlhaskellelixir powershell visualbasic6N)rrrrCPPGOJAVAKOTLINJSrPHPPROTOPYTHONRSTRUBYRUSTSCALASWIFTMARKDOWNLATEXHTMLSOLCSHARPCOBOLCLUAPERLHASKELLELIXIR POWERSHELL VISUALBASIC6r/r-r+rr/s, C B D F B B C E F C D D E EH E D C F E A C DG FJ!Lr-r)frozenc@eZdZUdZded< ded< ded< ded<y ) rzTokenizer data class.rfr%rzCallable[[list[int]], str]rzCallable[[str], list[int]]rsN)rrrr__annotations__r/r-r+rrOs)*, &&= &&=r-rcg}|j|}d}|j|jkr d}t||t |krt ||jzt |}|||}|s |S|j |}|r|j||t |k(r |S||j|jz z }|t |kr|S)z6Split incoming text and return chunks using tokenizer.rz3tokens_per_chunk must be greater than chunk_overlap)rsrr%rr6minrr<) r0rarV input_ids start_idxr*cur_idx chunk_idsdecodeds r+rr]sF  &II!!Y%<%<<Co c)n $i)"<"<rs#" #!G,HH2ML   8 $ T(B5*CB5J;D ;D|"sD"@ $ > > >A Ms$B?C ?C C  CC