itT.SrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSK J r SSK J r SSKJ r JrJrJrJr SSKrSSKrSSKJr SSKJrJr SS KJr SS KJr SS KJrJ r J!r! SS K"J#r# SS K$J%r%J&r& \!"5(aSSKJ'r' SSK(J)r)J*r*J+r+J,r,J-r- SSK.J/r/J0r0J1r1J2r2J3r3J4r4J5r5J6r6J7r7J8r8J9r9J:r:J;r;Jr> SSK?J@r@ \<"5(aSSKAJBrB \>R"\D5rE\"SSS9rF\8"\ "\G5R5rISSSSS.rJ\RS:a \RrLO \RrL"SS\SS9rM"S S!\SS9rN"S"S#\SS9rO"S$S%\SS9rP"S&S'\SS9rQ"S(S)\SS9rR"S*S+\SS9rS"S,S-\SS9rT"S.S/\T\SSS9rU"S0S1\SS9rV\ "S2S355rW"S4S\45rX\7"\XR5\XlY\XRRb5\XRRRS5S6S7S89\XRlgg)9z8 Processing saving/loading class for common processors. N) dataclass)Path)AnyOptional TypedDictTypeVarUnion)EntryNotFoundError) AudioInput load_audio)custom_object_save) BatchFeature)ChannelDimension ImageInputis_vision_available)render_jinja_template) VideoInput VideoMetadata)PILImageResampling)PaddingStrategyPreTokenizedInputPreTrainedTokenizerBase TextInputTruncationStrategy)AUDIO_TOKENIZER_NAMECHAT_TEMPLATE_DIRCHAT_TEMPLATE_FILE#LEGACY_PROCESSOR_CHAT_TEMPLATE_FILEPROCESSOR_NAMEPushToHubMixin TensorType cached_file copy_funcdirect_transformers_import download_urlis_offline_mode is_remote_urlis_torch_availablelist_repo_templateslogging)deprecate_kwarg)PreTrainedAudioTokenizerBaseSpecificProcessorTypeProcessorMixin)boundrFeatureExtractionMixinImageProcessingMixinBaseVideoProcessor) AutoTokenizerAutoFeatureExtractorAutoImageProcessorAutoVideoProcessor) c\rSrSr%Sr\\\\\ \\ \4\ S'\\\\ \\ \4\ S'\\\\\ \\ \4\ S'\\ \ S'\\ \ \ 4\ S'\\ \ \4\ S'\\\ S '\\\ S '\\ \ S '\\\ S '\\ \ S '\\ \ S'\\ \ S'\\ \ S'\\ \ S'\\ \ S'\\ \ S'\\ \ S'\\ \ S'Srg) TextKwargsda Keyword arguments for text processing. For extended documentation, check out tokenization_utils_base methods and docstrings associated. Attributes: add_special_tokens (`bool`, *optional*) Whether or not to add special tokens when encoding the sequences. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*) Activates and controls padding. truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*): Activates and controls truncation. max_length (`int`, *optional*): Controls the maximum length to use by one of the truncation/padding parameters. stride (`int`, *optional*): If set, the overflowing tokens will contain some tokens from the end of the truncated sequence. is_split_into_words (`bool`, *optional*): Whether or not the input is already pre-tokenized. pad_to_multiple_of (`int`, *optional*): If set, will pad the sequence to a multiple of the provided value. return_token_type_ids (`bool`, *optional*): Whether to return token type IDs. return_attention_mask (`bool`, *optional*): Whether to return the attention mask. return_overflowing_tokens (`bool`, *optional*): Whether or not to return overflowing token sequences. return_special_tokens_mask (`bool`, *optional*): Whether or not to return special tokens mask information. return_offsets_mapping (`bool`, *optional*): Whether or not to return `(char_start, char_end)` for each token. return_length (`bool`, *optional*): Whether or not to return the lengths of the encoded inputs. verbose (`bool`, *optional*): Whether or not to print more information and warnings. padding_side (`str`, *optional*): The side on which padding will be applied. return_mm_token_type_ids (`bool`, *optional*): Whether to return multimodal token type ids indicating mm placeholder token positions. text_pair text_targettext_pair_targetadd_special_tokenspadding truncation max_lengthstrideis_split_into_wordspad_to_multiple_ofreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbose padding_sidereturn_mm_token_type_idsN)__name__ __module__ __qualname____firstlineno____doc__rr rrlist__annotations__boolstrrrint__static_attributes__rPm/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/transformers/processing_utils.pyr;r;ds#%Ni):DOTRcMddeffy"3T)_dK\F]]^^uY0A4 ?TXYjTk%klmm & 4o- ..dC!3344  SM!$'  %#D>)#D>)'~- (.$TN*D>! d^3-&tn,r\r;F)totalc\\rSrSr%Sr\\\S'\\\ \ 4\S'\\\ \ 4\S'\\ S\ 4\S'\\\S'\\ \S '\\\S '\\ \ \ \ 4\S '\\ \ \ \ 4\S '\\\S '\\\ \ 4\S'\\\S'\\\S'\\ \ \4\S'\\ \S'Srg) ImagesKwargsa Keyword arguments for image processing. For extended documentation, check the appropriate ImageProcessor class methods and docstrings. Attributes: do_resize (`bool`, *optional*): Whether to resize the image. size (`dict[str, int]`, *optional*): Resize the shorter side of the input to `size["shortest_edge"]`. crop_size (`dict[str, int]`, *optional*): Desired output size when applying center-cropping. resample (`PILImageResampling`, *optional*): Resampling filter to use if resizing the image. do_rescale (`bool`, *optional*): Whether to rescale the image by the specified scale `rescale_factor`. rescale_factor (`int` or `float`, *optional*): Scale factor to use if rescaling the image. do_normalize (`bool`, *optional*): Whether to normalize the image. image_mean (`float` or `list[float]`, *optional*): Mean to use if normalizing the image. image_std (`float` or `list[float]`, *optional*): Standard deviation to use if normalizing the image. do_pad (`bool`, *optional*): Whether to pad the image to the `(max_height, max_width)` of the images in the batch. pad_size (`dict[str, int]`, *optional*): The size `{"height": int, "width" int}` to pad the images to. do_center_crop (`bool`, *optional*): Whether to center crop the image. data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the output image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. device (`str`, *optional*): The device to use for processing (e.g. "cpu", "cuda"), only relevant for fast image processing. do_resizesize crop_sizerresample do_rescalerescale_factor do_normalize image_mean image_stddo_padpad_sizedo_center_crop data_formatinput_data_formatdevicerPN)rQrRrSrTrUrrXrWdictrYrZr floatrVrr[rPr\r]r`r`s#J~ 4S> ""S#X''u136788UO#4. ud5k1233eT%[0122 TNtCH~&&TN"*++c+;&; <== SMr\r`c\rSrSr%Sr\\\S'\\\S'\\\ \ 4\S'\\\S'\S\S'\\\S '\\ \S '\\\S '\\ \ \ \ 4\S '\\ \ \ \ 4\S '\\\S'\\\ \ 4\S'\\\S'\\ \ \4\S'\\ \S'\\\S'\\ \\4\S'\\ \ \ 4\S'\\ \S'\\\S'Srg) VideosKwargsak Keyword arguments for video processing. Attributes: do_convert_rgb (`bool`): Whether to convert the video to RGB format. do_resize (`bool`): Whether to resize the video. size (`dict[str, int]`, *optional*): Resize the shorter side of the input to `size["shortest_edge"]`. default_to_square (`bool`, *optional*, defaults to `self.default_to_square`): Whether to default to a square when resizing, if size is an int. resample (`PILImageResampling`, *optional*): Resampling filter to use if resizing the video. do_rescale (`bool`, *optional*): Whether to rescale the video by the specified scale `rescale_factor`. rescale_factor (`int` or `float`, *optional*): Scale factor to use if rescaling the video. do_normalize (`bool`, *optional*): Whether to normalize the video. image_mean (`float` or `list[float]`, *optional*): Mean to use if normalizing the video. image_std (`float` or `list[float]`, *optional*): Standard deviation to use if normalizing the video. do_center_crop (`bool`, *optional*): Whether to center crop the video. do_sample_frames (`bool`, *optional*): Whether to sample frames from the video before processing or to process the whole video. video_metadata (`Union[VideoMetadata, dict]`, *optional*): Metadata of the video containing information about total duration, fps and total number of frames. num_frames (`int`, *optional*): Maximum number of frames to sample when `do_sample_frames=True`. fps (`int` or `float`, *optional*): Target frames to sample per second when `do_sample_frames=True`. crop_size (`dict[str, int]`, *optional*): Desired output size when applying center-cropping. data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the output video. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input video. return_metadata (`ChannelDimension` or `str`, *optional*): Whether to return video metadata or not. do_convert_rgbrbrcdefault_to_squarerrerfrgrhrirjrmrdrnrorpdo_sample_framesvideo_metadatafps num_framesreturn_metadatarPN)rQrRrSrTrUrrXrWrqrYrZrrr rVrrr[rPr\r]rtrts'*XTN"~ 4S> ""~%+,,UO#4. ud5k1233eT%[0122TN"S#X''*++c+;&; <== SMtn$U=$#6788 %U # $$ d^#r\rtc\rSrSr%Sr\\\S'\\\ R\ \ \ \ R\ \ \ 4\S'\\\ \\4\S'\\\S'\\ \S'\\\S'\\ \S 'S rg ) AudioKwargsia; Keyword arguments for audio processing. Attributes: sampling_rate (`int`, *optional*): The sampling rate at which the `raw_speech` input was sampled. raw_speech (`np.ndarray`, `list[float]`, `list[np.ndarray]`, `list[list[float]]`): The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float values, a list of numpy arrays or a list of list of float values. Must be mono channel audio, not stereo, i.e. single float per timestep. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). truncation (`bool`, *optional*): Activates truncation to cut input sequences longer than *max_length* to *max_length*. pad_to_multiple_of (`int`, *optional*): If set, will pad the sequence to a multiple of the provided value. return_attention_mask (`bool`, *optional*): Whether or not [`~ASTFeatureExtractor.__call__`] should return `attention_mask`. sampling_rate raw_speechrArCrBrFrHrPN)rQrRrSrTrUrrZrWr npndarrayrVrrrXrYrr[rPr\r]r~r~s:C= rzz4;RZZ8H$tTY{J[[\]] eD#67 88   %#D>)r\r~c0\rSrSr%\\\\4\S'Sr g) CommonKwargsiBreturn_tensorsrPN) rQrRrSrTrr rYr"rWr[rPr\r]rrBsU3 ?344r\rc\rSrSr%Sr0r0\REr\\S'0\ REr \ \S'0\ REr \ \S'0\ REr\ \S'0\REr\\S'Srg ) ProcessingKwargsiFa" Base class for kwargs passing to processors. In case a model has specific kwargs that are not present in the base class or default values for existing keys, it should have its own `ModelProcessorKwargs` class that inherits from `ProcessingKwargs` to provide: 1) Additional typed keys and that this model requires to process inputs. 2) Default values for existing keys under a `_defaults` attribute. New keys have to be defined as follows to ensure type hinting is done correctly. ```python # adding a new image kwarg for this model class ModelImagesKwargs(ImagesKwargs, total=False): new_image_kwarg: Optional[bool] class ModelProcessorKwargs(ProcessingKwargs, total=False): images_kwargs: ModelImagesKwargs _defaults = { "images_kwargs: { "new_image_kwarg": False, } "text_kwargs": { "padding": "max_length", }, } ``` For Python 3.8 compatibility, when inheriting from this class and overriding one of the kwargs, you need to manually update the __annotations__ dictionary. This can be done as follows: ```python class CustomProcessorKwargs(ProcessingKwargs, total=False): images_kwargs: CustomImagesKwargs CustomProcessorKwargs.__annotations__["images_kwargs"] = CustomImagesKwargs # python 3.8 compatibility ```python common_kwargs text_kwargs images_kwargs videos_kwargs audio_kwargsrPN)rQrRrSrTrU _defaultsrrWrr;rr`rrtrr~rr[rPr\r]rrFs$LI#  & &#M<  $ $K#  & &#M<#  & &#M<!  % %!L+r\rc\rSrSr%SrSr\\\\ S'Sr \\\\ \ 4\ S'Sr \\ \ S'Sr\\ \ S'Sr\\ \ S 'S rg) TokenizerChatTemplateKwargsia Keyword arguments for tokenizer's `apply_chat_template`, when it is called from within a processor. tools (`list[Dict]`, *optional*): A list of tools (callable functions) that will be accessible to the model. If the template does not support function calling, this argument will have no effect. Each tool should be passed as a JSON Schema, giving the name, description and argument types for the tool. See our [chat templating guide](https://huggingface.co/docs/transformers/main/en/chat_templating#automated-function-conversion-for-tool-use) for more information. documents (`list[dict[str, str]]`, *optional*): A list of dicts representing documents that will be accessible to the model if it is performing RAG (retrieval-augmented generation). If the template does not support RAG, this argument will have no effect. We recommend that each document should be a dict containing "title" and "text" keys. Please see the RAG section of the [chat templating guide](https://huggingface.co/docs/transformers/main/en/chat_templating#arguments-for-RAG) for examples of passing documents with chat templates. add_generation_prompt (bool, *optional*): If this is set, a prompt with the token(s) that indicate the start of an assistant message will be appended to the formatted output. This is useful when you want to generate a response from the model. Note that this argument will be passed to the chat template, and so it must be supported in the template for this argument to have any effect. continue_final_message (bool, *optional*): If this is set, the chat will be formatted so that the final message in the chat is open-ended, without any EOS tokens. The model will continue this message rather than starting a new one. This allows you to "prefill" part of the model's response for it. Cannot be used at the same time as `add_generation_prompt`. return_assistant_tokens_mask (`bool`, defaults to `False`): Whether to return a mask of the assistant generated tokens. For tokens generated by the assistant, the mask will contain 1. For user and system tokens, the mask will contain 0. This functionality is only available for chat templates that support it via the `{% generation %}` keyword. Ntools documentsFadd_generation_promptcontinue_final_messagereturn_assistant_tokens_maskrP)rQrRrSrTrUrrrVrqrWrrYrrXrrr[rPr\r]rrse>#'E8DJ &04IxT#s(^,-4,18D>1-2HTN238 (4.8r\rcB\rSrSr%SrSr\\\S'Sr \\ \S'Sr g) ChatTemplateLoadKwargsia Keyword arguments used to load multimodal data in processor chat templates. num_frames (`int`, *optional*): Number of frames to sample uniformly. If not passed, the whole video is loaded. load_audio_from_video (`bool`, *optional*): Whether to use the audio track of input video. If `True` the audio track will be loaded and passed to the processor. This flag has no effect if the model doesn't support audio modality. i>rFload_audio_from_videorPN) rQrRrSrTrUrrrZrWrrXr[rPr\r]rrs&$*M8C=),18D>1r\rcB\rSrSr%SrSr\\\S'Sr \\\S'Sr g)ProcessorChatTemplateKwargsia" Keyword arguments for processor's `apply_chat_template`. tokenize (`bool`, *optional*, defaults to `False`): Whether to tokenize the output or not. return_dict (`bool`, defaults to `False`): Whether to return a dictionary with named outputs. Has no effect if tokenize is `False`. Ftokenize return_dictrPN) rQrRrSrTrUrrrXrWrr[rPr\r]rrs% %Hhtn$"'K$'r\rc4\rSrSr%\\S'\\S'\\S'Srg)AllKwargsForChatTemplateiprocessor_kwargsmm_load_kwargstemplate_kwargsrPN) rQrRrSrTrrWrrr[rPr\r]rrs&&**00r\rc\rSrSr%SrSr\\\\ S'Sr \\\\ S'Sr \\\\ S'Sr \\\\ S'Sr S rS rg) MultiModalDataiaq Dataclass that holds extra useful data for processing multimodal data. Processors currently cannot return keys, unless it is used in model's forward. Thus we have helper methods that calculate and return useful data from processing input multimodals (images/videos). Note that this dataclass is aimed to be used only in vLLM and we might change its API in the future. Nnum_image_tokensnum_video_tokensnum_audio_tokensnum_image_patchesc@[X5=(a [X5SL$N)hasattrgetattrselfkeys r] __contains__MultiModalData.__contains__st!Dgd&8&DDr\c[X5(a [X5$[URRSU35e)Nz has no attribute )rrAttributeError __class__rQrs r] __getitem__MultiModalData.__getitem__s: 4  4% % 7 788J3%PQQr\rP)rQrRrSrTrUrrrVrZrWrrrrrr[rPr\r]rrse-1htCy)0,0htCy)0,0htCy)0-1xS *1ERr\rc\rSrSr%SrSS/rSS/r/r\\ \ S'Sr Sr Sr \rS rS?S \\S \\\\\\\\4S \\S \\S\\4 SjjrSrS@S\\ \44SjjrS@S\ 4SjjrS@S\\ \R@44Sjjr!Sr"SAS\#S\#4Sjjr$\%S\\ \R@4S\&\\ \4\\ \444Sj5r'\%S\\ \44Sj5r(SBS\S\\S\\ \44S jjr)\%SCS!\*\+S\\ \R@4S"\\\ \R@4S#\#S$\#S%\\\ \#4S&\ S\+4S'jj5r,\%SDS(j5r-\%S)5r.\/S*5r0S+r1S,r2\3S-5r4\/S.5r5\6"S/S0S1S29\6"S3S4S5S69SBS7\\\\ \ 4\\\\ \ 44S\\ S\\7S\ 4S8jj55r8S@S9jr9S \\ S:S;S<\\ 4S=jr:S>r;g)Er/izY This is a mixin used to provide saving/loading functionality for all processor classes. feature_extractor tokenizer chat_templateaudio_tokenizeroptional_call_argsNc URHrnURUS5n[XU5 US:XdM)UcM.URX45n[ 5(a[ U[ 5(aMe[SUS35e UH!nX`R;dM[SUS35e [XR5HupxX;a[SUS35eXrU'M [U5[UR5:waJ[S[UR5SS RUR5S [U5S 35eUR5H"upURX5 [XU5 M$ g) NrzTried to use `zW` for audio tokenization. However, this class is not registered for audio tokenization.zUnexpected keyword argument .z!Got multiple values for argument zThis processor requires z arguments: , z. Got z arguments instead.)optional_attributespopsetattrcheck_argument_for_proper_classr) isinstancer- ValueError attributes TypeErrorziplenjoinitems) rargskwargsoptional_attributeoptional_attribute_value proper_classrargattribute_names r]__init__ProcessorMixin.__init__s~#'":": '-zz2Dd'K $ D.F G"%66;S;_#CCDVq *,,>#;C//)">se1 EFF$'t__#= C'"CNCSST UVV),~& $> v;#doo. .*3t+?*@ TYYW[WfWfMgLhhnt9+02  $*<<> N  0 0 E D# .$2r\imagestextvideosaudiorc Uc+Uc(Uc%Uc"[SURR35eUR"UR4S[ US5(aUR RO00UD6nUS4US4US4US4S .n0nURH;n[XS5n XhupU cMU cMU "U 40X[D6n URU 5 M= [U5$) ao Main method to prepare for model inputs. This method forwards the each modality argument to its own processor along with `kwargs`. Please refer to the docstring of the each processor attributes for more information. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. Both channels-first and channels-last formats are supported. text (`TextInput`, `PreTokenizedInput`, `list[TextInput]`, `list[PreTokenizedInput]`, *optional*): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). videos (`np.ndarray`, `torch.Tensor`, `List[np.ndarray]`, `List[torch.Tensor]`): The video or batch of videos to be prepared. Each video can be a 4D NumPy array or PyTorch tensor, or a nested list of 3D frames. Both channels-first and channels-last formats are supported. audio (`np.ndarray`, `torch.Tensor`, `list[np.ndarray]`, `list[torch.Tensor]`): The audio or batch of audio to be prepared. Each audio can be a NumPy array or PyTorch tensor. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] object with processed inputs in a dict format. Nz/You need to provide at least one input to call tokenizer_init_kwargsrrrrr)rimage_processorvideo_processorr) rrrQ _merge_kwargsvalid_processor_kwargsrr init_kwargsrrupdater) rrrrrrattribute_to_kwargsoutputsr attribute input_data input_kwargsattribute_outputs r]__call__ProcessorMixin.__call__sJ >dlv~%-Nt~~OfOfNghi i##  ' ' @Gk@Z@Z$.."<"<`b   . &8 &8"'!8   "ooNd;I':'J $J%)*?#,Z#P6;O#P /0 .G$$r\c 8^[TUS35n[RX35n[U[5(a[ U4SjU55nOTR U5n[X$5(d([ S[U5RSUSUS35eU$)z Checks the passed argument's class against the expected transformers class. In case of an unexpected mismatch between expected and actual class, an error is raise. Otherwise, the proper retrieved class is returned. _classc3N># UHocMTRU5v M g7frget_possibly_dynamic_module).0nrs r] AProcessorMixin.check_argument_for_proper_class..as" jj!D!A!A!!D!Djs%%z Received a z for argument z, but a z was expected.) rAUTO_TO_BASE_CLASS_MAPPINGgetrtuplerrtyperQ)r argument_nameargument class_namers` r]r.ProcessorMixin.check_argument_for_proper_classWs Tm_F#;< /33JK j% ( ( jj jjL;;JGL(11d8n556n]OS[\f[gguv r\returnc ^ [R"UR5n[R"UR 5n[ UR5nUS/- nU(a-UVs/sH oUURR;dMUPM" nnSU;aUS SU;aUS SU;aUS SU;aUS U 4Sjm UR5VVs0sHuupgXd;dM URRS:wdM(U(a[U[5(a U(aMMU[U[5(aUR5OU_Mw nnnT "U5nU(dASU;a;URRRURR S .nXS'URRUS 'U$s snfs snnf) z Serializes this instance to a Python dictionary. Returns: `dict[str, Any]`: Dictionary of all the attributes that make up this processor instance. auto_maprqformer_tokenizerprotein_tokenizerrc>UR5HYup[U[R5(aUR 5X'M8[U[ 5(dMOT"U5X'M[ U$)z Numpy arrays are not serialiazable but can be in pre-processing dicts. This function casts arrays to list, recusring through the nested configs as well. )rrrrtolistrq) dictionaryrvaluecast_array_to_lists r]r2ProcessorMixin.to_dict..cast_array_to_listsY )..0 eRZZ00&+llnJOt,,&8&?JO 1  r\BeamSearchDecoderCTCraudio_tokenizer_classaudio_tokenizer_name_or_pathprocessor_class)copydeepcopy__dict__inspect signaturerrV parametersrrrrQrr!to_dictr name_or_path) rlegacy_serializationoutputsig attrs_to_savexkvaudio_tokenizer_dictrs @r]rProcessorMixin.to_dictlst}}- .S^^, *% (5\ 1$..B[B[9[Q M\ & {# & (*+ & (*+ f $'    &" CKK((,BB C **Q2O2OXl CAjN;;qyy{ B&   $F+$(9V(C)-)=)=)G)G)P)P040D0D0Q0Q$ )=$ %$(NN$;$; ! a]0  s$(F8 F8 F=F=9!F=,F=cRURUS9n[R"USSS9S-$)z Serializes this instance to a JSON string. Returns: `str`: String containing all the attributes that make up this feature_extractor instance in JSON format. rTindent sort_keys )rjsondumps)rrrs r]to_json_stringProcessorMixin.to_json_strings-\\7K\L zz*Q$?$FFr\json_file_pathc[USSS9nURURUS95 SSS5 g!,(df  g=f)z Save this instance to a JSON file. Args: json_file_path (`str` or `os.PathLike`): Path to the JSON file in which this processor instance's parameters will be saved. wutf-8encodingrN)openwriter")rr$rwriters r] to_json_fileProcessorMixin.to_json_files9.# 8F LL,,BV,W X9 8 8s 4 Ac URVs/sHnSUS[[X553PM nnSRU5nURR SUSUR 53$s snf)Nz- z: rz: z )rreprrrrrQr")rnameattributes_reprs r]__repr__ProcessorMixin.__repr__swPTP_P_`P_RvRWT-@(A'BCP_`))O4..))*#o->d4CVCVCXBYZZas$A2 push_to_hubrc URSS5nUb<[R"S[5 UR S5b [ S5eXTS'[ R"USS9 U(arURSS5nURS UR[ RR5S 5nUR"U40UD6nURU5nURbsURV s/sHn [X 5PM n n U V s/sH&n [!U ["5(a U R$OU PM( n n U R'U5 [)XU S 9 UR S S5n URHn U S :XaR[X 5n[+US5(a%UR-UR.R05 UR3XS9 M[U(dMd[X 5n[+US5(a%UR-UR.R05 UR3U5 M URbBURH2n [X 5n[!U["5(dM%UR$S M4 [ RR5U[65n[ RR5U[85n[ RR5U[:5n[ RR5U[<5nUR>GbUR S S5n [!UR>[@5nU (aPU(aI[CUSSS9nUREUR>5 SSS5 [FRISU35 GO|U (aU(dUR>RK5HunnUS:XaL[CUSSS9nUREUR>S5 SSS5 [FRISU35 MX[ R"USS9 [ RR5UUS35n[CUSSS9nUREU5 SSS5 [FRISU35 M OU(ac[LRN"SUR>0SSS9S-n[CUSSS9nUREU5 SSS5 [FRISU35 OUR>b [ S5eU(Ga'[ RR5U[P5nURS5nURW55S1:wa)URYU5 [FRISU35 [UURW55S1:Xa/nOU/nURZbzURZR.R0nURZR\nUUS.n[LRN"USSS9S-n [CUSSS9nUREU 5 SSS5 O,O+URYUSS 9 [FRISU35 U/nU(a"UR_UWWWUR S5S!9 U$s sn fs sn f!,(df  GN4=f!,(df  GN=f!,(df  GNa=f!,(df  GN=f!,(df  N=f)"a Saves the attributes of this processor (feature extractor, tokenizer...) in the specified directory so that it can be reloaded using the [`~ProcessorMixin.from_pretrained`] method. This class method is simply calling [`~feature_extraction_utils.FeatureExtractionMixin.save_pretrained`] and [`~tokenization_utils_base.PreTrainedTokenizerBase.save_pretrained`]. Please refer to the docstrings of the methods above for more information. Args: save_directory (`str` or `os.PathLike`): Directory where the feature extractor JSON file and the tokenizer files will be saved (directory will be created if it does not exist). push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). legacy_serialization (`bool`, *optional*, defaults to `True`): Whether or not to save processor attributes in separate config files (legacy) or in processor's config file as a nested dict. Saving all attributes in a single dict will become the default in future versions. Set to `legacy_serialization=True` until then. kwargs (`dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. use_auth_tokenNrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.tokenV`token` and `use_auth_token` are both specified. Please set only the argument `token`.T)exist_okcommit_messagerepo_id)configsave_jinja_filesr_set_processor_class)r@rr&r'r(zchat template saved in default.jinjarrrrzMultiple chat templates are not supported in the legacy format. Please save them as separate files using the `save_jinja_files` argument.rzprocessor saved in rFr)r<r9)0rwarningswarn FutureWarningrrosmakedirssplitpathsep _create_repo_get_files_timestamps _auto_classrrrrrappendrrrArrQsave_pretrainedrr rrrrrYr*r+loggerinforr r!rrsetkeysr-rr_upload_modified_files)!rsave_directoryr5rrr7r<r=files_timestampsrattrsaconfigsr@routput_processor_fileoutput_chat_template_file_jinja output_chat_template_file_legacychat_template_diris_single_templatef template_nametemplatetemplate_filepathchat_template_json_stringr,output_audio_tokenizer_fileprocessor_dict return_filesrrraudio_tokenizer_jsons! r]rPProcessorMixin.save_pretraineds"8 $4d;  % MME zz'". l-7O NT2 #ZZ(8$?NjjN,@,@,Mb,QRG'':6:G#99.I     'IMY~WT2EYafgaf\]A7N)O)O UVVafGg NN4 tG D!::&8$?"ooN,#D9 9&<==224>>3J3JK)).)\%%#D9 9&<==224>>3J3JK)).9.    '"&//#D9 i)@AA!--j9#2!# ^^ L*,'',,~GY*Z'+-77<< ?, (GGLL9JK    )%zz*,>!D $693QUVGGD../R 56U5VWX!*<04/A/A/G/G/I+M8$ 1!"A3QXY]^GGD$6$6y$ABZ &=>]=^$_` $5E,.GGLL9J}o]cLd,e)!"3S7KqGGH-L &=>O=P$QR0J$JJ1C1CDQZ^_bff*:C'RV\LL!:;S 56V5WXY##/!L *,'',,~G[*\ '!\\^N>&&().?-@@!!"78 12G1HIJ>&&().?-@@! 56 ##/(,(<(<(F(F(O(O%/3/C/C/P/P,-B4P($(,zz2Fq\`'adh'h$5sWMQWLL!56NM0   3%  P KK-.C-DE F12L   ' ' -jj) ( uZgZRQZY LKSRDNMsH;Z)-Z.*Z3'[+[[)7[;3 [ [  [& ) [8; \ pretrained_model_name_or_pathc [R"U5nURSS5nURSS5nURSS5nURSS5nURSS5nURSS5n URS S5n URS S 5n URS S5n URS S5n SU S.nU bXS'[5(aU (d[R S5 Sn [ U5n[RRU5n[RRU5(a$[RRU[5n0n0n[RRU5(a UnSnSnSnSnGOM[U5(aUn[U5nSnSnSnGO(U(ab[U[ 5nUR#5(a<UR%S5H'nUR&n[ SUR(3UU'M) O$[+UU U UUS9Hn[ SUS3UU'M [n[/UUUUUUU UUU U SS9 n[/U[0UUUUU UUU U SS9 n[/U[2UUUUU UUU U SS9 nUR55VVs0sHunnU[/UUUUUUU UUU U SS9 _M nnn[/U[6UUUUU UUU U SS9 nUbR[=USS9n[>R@"URC55nSUS0nU(a [ES 5eSSS5 OgUR55VVs0sH unnU[=US!SS9RC5_M" nnnUb'[=US!SS9nURC5US'SSS5 [GW[H5(aSU;a[KU5S":XaUSnU(aUUS'Uc0nO:[=USS9nURC5nSSS5 [>R@"W5nU(a[R S%U35 O[R S%WS&U35 SU;aUSb[ROS'5 SU;aURS5US'UcS(U;aiUb2[=US!SS9nURC5n [>R@"U 5n OUS(n URQU S)5n!U S*n"U!RR"U"40UD6US('URTHn#URU#S5 M UU4$![,a GNf=fs snnf![8a e[:a [9SUSUS[S35ef=f!,(df  GN=fs snnf!,(df  GN=f!,(df  GN=f![>RLa [9S#US$35ef=f)+a From a `pretrained_model_name_or_path`, resolve to a dictionary of parameters, to be used for instantiating a processor of type [`~processing_utils.ProcessingMixin`] using `from_args_and_dict`. Parameters: pretrained_model_name_or_path (`str` or `os.PathLike`): The identifier of the pre-trained checkpoint from which we want the dictionary of parameters. subfolder (`str`, *optional*, defaults to `""`): In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. Returns: `tuple[Dict, Dict]`: The dictionary(ies) that will be used to instantiate the processor object. cache_dirNforce_downloadFresume_downloadproxiesr9local_files_onlyrevision subfolder_from_pipeline _from_auto processor) file_typefrom_auto_classusing_pipelinez+Offline mode: forcing local_files_only=TrueTz*.jinja/)rprqrlr9rC) rlrmrornrpr9 user_agentrqrr%_raise_exceptions_for_missing_entrieszCan't load processor for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z2' is the correct path to a directory containing a z filer'r(rBraCannot load chat template due to conflicting files - this checkpoint combines a legacy chat_template.json file with separate template files, which is not supported. To resolve this error, replace the legacy chat_template.json file with a modern chat_template.jinja file.rr z"It looks like the config file at 'z' is not a valid JSON file.zloading configuration file z from cache at zChat templates should be in a 'chat_template.jinja' file but found key='chat_template' in the processor's config. Make sure to move your template to its own file.rrr)+rr rr'rQrRrYrGrJisdirrr isfiler(r&rris_dirglobstemr1r*r r#rrrrOSError Exceptionr*r loadsreadrrrqrJSONDecodeError warning_oncerfrom_pretrainedr)$clsrjraudio_tokenizer_kwargsrlrmrnror9rprqrr from_pipelinerxr{is_localprocessor_fileadditional_chat_template_files'resolved_additional_chat_template_filesresolved_processor_fileresolved_chat_template_fileresolved_raw_chat_template_fileresolved_audio_tokenizer_file template_dir template_filerarbreaderchat_template_jsonchat_templatesrfrrraudio_tokenizer_pathrs$ r]get_processor_dict!ProcessorMixin.get_processor_dictzs&"&v!6JJ{D1 $4e< **%6=**Y- 7D)!::&8%@::j$/JJ{B/  #3T:  **\59#.?S  $+8' (   %5 KKE F# (+,I(J%77==!>? 77==6 7 7WW\\*GXN)+&24/ 77>>7 8 8&C #*. '.2 +,0 )H 8 9 9:N&23P&Q #*. '.2 +,0 )#$ACTU &&(()5):):9)E (5(:(: K\J]]^_l_q_q^rHs6}E*F $75)9!)"+# %GXEXXYZbYcciCj6x@%,N\ *51"'#1#$3%5)%':? +'"/:17'#1#$3%5)%':? /+3>1&'#1#$3%5)%':? 3/<9W8\8\8^;9_4 }";5%"+'5 '(7)9##-!)"+>C $ 9_8;$1<1('#1#$3%5)%':? 1-8 ' 21GD%)ZZ %>""+-?-P!Q:$B;ED5\4a4a4c4c0M=tM3INNPP4c /:93QU[06 N9-R nd + + ^0KPSTbPcghPh+I6N &4F? # # *N 1GD!;;=DE!%D!1  KK56M5NO P KK5n5E_UlTmn o n ,1P1\   ^  f $.4jj.IN? + ) 48I^8[,8;S7S'-{{}$'+zz2F'G$'56G'H$$'$C$CDXYpDq$r !#78V#W 0E0U0U$1(>1N, - I   y$ /(v%%e*j;@  01N0OP99V8WX//=.>eE ED RQ$ED'' 89P8QQlm ss9#T?#AU:"UU>V'VV5 W?V1W? U  U U.V V V.1 W;W$W'rfc UR5nURSS5nSU;aUS SU;aUS URU5 URRR SURRR SSnURX%S9upg[U5VV s0sH.upX;dM U[U5:dMXRU 5_M0 n nn [U5VV s/sHupU RX5PM nnn U"U0UD6n [RSU 35 U(aX4$U $s sn nfs sn nf) aT Instantiates a type of [`~processing_utils.ProcessingMixin`] from a Python dictionary of parameters. Args: processor_dict (`dict[str, Any]`): Dictionary that will be used to instantiate the processor object. Such a dictionary can be retrieved from a pretrained checkpoint by leveraging the [`~processing_utils.ProcessingMixin.to_dict`] method. kwargs (`dict[str, Any]`): Additional parameters from which to initialize the processor object. Returns: [`~processing_utils.ProcessingMixin`]: The processor object instantiated from those parameters. return_unused_kwargsFrrNr )processor_config valid_kwargsz Processor ) rrrr__code__ co_varnames co_argcountvalidate_init_kwargs enumeraterrrQrR) rrrfrraccepted_args_and_kwargs unused_kwargsrirargs_to_updatervs r]from_args_and_dict!ProcessorMixin.from_args_and_dict}sf"(,,.%zz*@%H  .01  'z* f%$'<<#8#8#D#DEhs||G\G\GhGh#ijkjl#m '*&>&>+'?' # $$<= =# %()CI  %A$ $=  :C4Ivq""1*I..  j ,- + +  Js) D;8 D; D;/EModelProcessorKwargsrc \^00000S.n00000S.m1Skn[5nTHnURRU05R5TU'URURHLnUcMX;dM[ UR U5(a[UR U5OX(n U TUU'MN M URT5 [U5[U5- n UR5Hup{URURHnXs;a2X7RUS5n U S:waX;a[SUSUS35eOX;aURUS5n OSn [U [5(aU S:wdMqXU'URU5 M M [U4SjU55(aZUR5HEup}UT;dM U R5H$upX;dM XUU'URU5 M& MG OjUR5HVunnUU;dMUURSR;a UUSU'M5UU;dM=[ R#S US 35 MX UR%5HnURUS5 M U$) a Method to merge dictionaries of kwargs cleanly separated by modality within a Processor instance. The order of operations is as follows: 1) kwargs passed as before have highest priority to preserve BC. ```python high_priority_kwargs = {"crop_size" = {"height": 222, "width": 222}, "padding" = "max_length"} processor(..., **high_priority_kwargs) ``` 2) kwargs passed as modality-specific kwargs have second priority. This is the recommended API. ```python processor(..., text_kwargs={"padding": "max_length"}, images_kwargs={"crop_size": {"height": 222, "width": 222}}}) ``` 3) kwargs passed during instantiation of a modality processor have fourth priority. ```python tokenizer = tokenizer_class(..., {"padding": "max_length"}) image_processor = image_processor_class(...) processor(tokenizer, image_processor) # will pass max_length unless overridden by kwargs at call ``` 4) defaults kwargs specified at processor level have lowest priority. ```python class MyProcessingKwargs(ProcessingKwargs, CommonKwargs, TextKwargs, ImagesKwargs, total=False): _defaults = { "text_kwargs": { "padding": "max_length", "max_length": 64, }, } ``` Args: ModelProcessorKwargs (`ProcessingKwargs`): Typed dictionary of kwargs specifically required by the model passed. tokenizer_init_kwargs (`Dict`, *optional*): Dictionary of kwargs the tokenizer was instantiated with and need to take precedence over defaults. Returns: output_kwargs (`Dict`): Dictionary of per-modality kwargs to be passed to each modality-specific processor. )rrrrr>rrrr __empty__zKeyword argument z+ was passed two times: in a dictionary for z and as a **kwarg.c3,># UH oT;v M g7frrP)rrdefault_kwargss r]r/ProcessorMixin._merge_kwargs..s7n$srzKeyword argument `zA` is not a valid argument for this processor and will be ignored.)rSrrrrWrrrrrrrrrYaddanyrQrvalues)rrrr output_kwargspossible_modality_keywords used_keysmodality modality_keyrnon_modality_kwargs output_kwarg kwarg_valuesubdictsubkeysubvaluerkwargrs @r]rProcessorMixin._merge_kwargss^    &K"E 'H';'E'E'I'I(TV'W'\'\'^N8 $ 4 D DX N ^ ^ (49^#4>><@@  =2@ >CN8,\:!_' ^,"&kC ,>>&3&9&9&; "H 4 D DX N ^ ^ %"("2"6"6|["QK"k1l6Y(/ ~>33;*F(3F;%MM&1-<&4%lln Ui'2BB?Sccc>C o6s;$>>++05vw -#))+E LL7 8,r\rrlrmrpr9rqc &X'S'X7S'XGS'XgS'URSS5nUb+[R"S[5 Ub [ S5eUnUbXWS 'UR "U40UD6n UR "U40UD6upUR"X40UD6$) a Instantiate a processor associated with a pretrained model. This class method is simply calling the feature extractor [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`], image processor [`~image_processing_utils.ImageProcessingMixin`] and the tokenizer [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`] methods. Please refer to the docstrings of the methods above for more information. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained feature_extractor hosted inside a model repo on huggingface.co. - a path to a *directory* containing a feature extractor file saved using the [`~SequenceFeatureExtractor.save_pretrained`] method, e.g., `./my_model_directory/`. - a path or url to a saved feature extractor JSON *file*, e.g., `./my_model_directory/preprocessor_config.json`. **kwargs Additional keyword arguments passed along to both [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`] and [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`]. rlrmrprqr7Nr8r:r9)rrDrErFr_get_arguments_from_pretrainedrr) rrjrlrmrpr9rqrr7rrfs r]rProcessorMixin.from_pretrained7sN({#1 %5!"%z$4d;  % MME   l#E  #7O112OZSYZ!$!7!78U!`Y_!`%%dEfEEr\c[U[5(d URnSSKJs Jn [ X!5(d[US35eXlg)aC Register this class with a given auto class. This should only be used for custom feature extractors as the ones in the library are already mapped with `AutoProcessor`. Args: auto_class (`str` or `type`, *optional*, defaults to `"AutoProcessor"`): The auto class to register this new feature extractor with. rNz is not a valid auto class.) rrYrQtransformers.models.automodelsautorrrN)r auto_class auto_modules r]register_for_auto_class&ProcessorMixin.register_for_auto_classvsE*c**#,,J66{// |+FGH H$r\c ^/nTRHn[TUS35n[U[5(ap[U4SjU55nUS:Xa*UR S5nUc[ R S5 OUR SS5nU(a USbUSnOUSnOTRU5nURUR"U40UD65 M U$) aU Identify and instantiate the subcomponents of Processor classes, like image processors and tokenizers. This method uses the Processor attributes like `tokenizer_class` to figure out what class those subcomponents should be. Note that any subcomponents must either be library classes that are accessible in the `transformers` root, or they must be custom code that has been registered with the relevant autoclass, via methods like `AutoTokenizer.register()`. If neither of these conditions are fulfilled, this method will be unable to find the relevant subcomponent class and will raise an error. rc3N># UHobTRU5OSv M g7frr)rrrs r]r@ProcessorMixin._get_arguments_from_pretrained..s'rgqbcm ? ? BY] ]gqs"%ruse_fastaCUsing a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.Tr r) rrrrrrQrrrOr) rrjrrrrclassesrattribute_classs ` r]r-ProcessorMixin._get_arguments_from_pretraineds!nnN (8&?@J*e,,rgqrr!%66%zz*5H'++T &zz*d;H 6&-ajO&-ajO"%"A"A*"M KK778U`Y_` a--0 r\c[[U5(a[[U5$[R[R[R [R [R/nUH|nURR5H[n[U[5(a(UH nUcMURU:XdMUs s s $ M@UcMEURU:XdMWUs s $ M~ [SUS35e)NzCould not find module z in `transformers`. If this is a custom class, it should be registered using the relevant `AutoClass.register()` function so that other functions can find it!)rtransformers_modulerIMAGE_PROCESSOR_MAPPINGVIDEO_PROCESSOR_MAPPINGTOKENIZER_MAPPINGFEATURE_EXTRACTOR_MAPPING$MODEL_FOR_AUDIO_TOKENIZATION_MAPPING_extra_contentrrrrQr) module_namelookup_locationslookup_location custom_classcustom_subclasss r]r*ProcessorMixin.get_possibly_dynamic_modules & 4 4. < <  7 7  7 7  1 1  9 9  D D   0O / > > E E G lE22+7*6?;S;SWb;b#22,8"-,2G2G;2V'' !H 0$[M2+ ,  r\c[US5(d#[SURRS35eURR "U0UD6$)z This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. rzCannot batch decode text:  has no tokenizer.)rrrrQr batch_decoderrrs r]rProcessorMixin.batch_decodesL t[))9$..:Q:Q9RRdef f~~**D;F;;r\c[US5(d#[SURRS35eURR "U0UD6$)z This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. rzCannot decode text: r)rrrrQrdecoders r]rProcessorMixin.decodesL t[))3DNN4K4K3LL^_` `~~$$d5f55r\c/nURH,n[XS5n[US5nURU5 M. U$)Nmodel_input_names)rrextend)rrrrattr_input_namess r]r ProcessorMixin.model_input_namessG"ooNd;I&y2EF   $ $%5 6.! r\c[UR55n[U5nX#- nX#-nU(aUVs0sHofX_M snO0nU(aUVs0sHofX_M snO0nXq4$s snfs snfr)rSrT)rrkwargs_from_configvalid_kwargs_set unused_keys valid_keysrrs r]r#ProcessorMixin.validate_init_kwargss !1!6!6!89|,(; ': ITEA,//EZ\ GQ C 1+.. CWY **FCs A,A1 video_fpsz4.58rz)versionnew_namevideo_load_backendz4.59zd. This function will use `torchcodec` by default, or `torchvision` if `torchcodec` is not installed.)radditional_message conversationc Uc[UR[5(a SUR;aURSnO[UR[5(a5[SSR URR 5535eURb URnOJ[S5e[UR[5(aX R;aURUnO[ US5=(a/ URRRRS5nURSS 5(aDURS S 5(a [S 5eURS S 5(a [S 5eURS S 5(aU(d [S5eSUS'00S.nUHzn[RURHVn[RUn[XS5n URXy5n U cM8[U [5(aMOXUU'MX M| URSS5 USR!U5 [U["[$45(a7[US["[$45(d[ USS5(aSn Un OS n U/n USRSS 5n USRSS 5nUSnU (Gaz//nn/nU GHmn//nnUGH<nUSVs/sHnUSS;dMUPM nnUSVVs/sH%nSHnUU;dM USS:XdMUUPM M' nnnUVVs/sH%nSHnUU;dM USS:XdMUUPM M' nnnUR'U5 UVVs/sH%nSHnUU;dM USS :XdMUUPM M' nnnUR'U5 US!(d(UHnUR)[+UUS"S#95 M! GMUHnUR)[+UUS"S#95 M! GM? UR)U5 UR)U5 GMp [-S2U US$.USDURR.D6unnU (dUSnU (GaU (aUSOUnURR0b/UR3URR05(aS US%'S&U;a)URS'5cURS(5bSUS&'[5S)W55n [5S*W55n!U"S2UU (aUOSU!(aUOSW(aUOSS+.UD6n"U(Ga?USRS S 5(Ga"/n#U"RS,5n$U"S-n%[7[9U%55Hn&S/[9U%U&5-n'U$U&n(U(V)V*s/sHun)n*U)PM n+n)n*UU&Hun,n-[:R<"U+U,5n.[:R<"U+U-5n/U.S:aU(U.SU,s=::a U(U.S.:dO MW[7U.U/(aU/O [9U%U&55Hn0S.U'U0'M M U#R)U'5 M U#U"S/'U"R?URS05S19 U"$U"S-$U$s snfs snnfs snnfs snnfs sn*n)f)3a; Similar to the `apply_chat_template` method on tokenizers, this method applies a Jinja template to input conversations to turn them into a single tokenizable string. The input is expected to be in the following format, where each message content is a list consisting of text and optionally image or video inputs. One can also provide an image, video, URL or local path which will be used to form `pixel_values` when `return_dict=True`. If not provided, one will get only the formatted text, optionally tokenized text. conversation = [ { "role": "user", "content": [ {"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"}, {"type": "text", "text": "Please describe this image in detail."}, ], }, ] Args: conversation (`Union[list[Dict, [str, str]], list[list[dict[str, str]]]]`): The conversation to format. chat_template (`Optional[str]`, *optional*): The Jinja template to use for formatting the conversation. If not provided, the tokenizer's chat template is used. NrBzThe processor has multiple chat templates but none of them are named "default". You need to specify which one to use by passing the `chat_template` argument. Available templates are: rzTCannot use apply_chat_template because this processor does not have a chat template.rFastrFra continue_final_message and add_generation_prompt are not compatible. Use continue_final_message when you want the model to continue the final message, and add_generation_prompt when you want to add a header that will prompt it to start a new assistant message instead.rzKcontinue_final_message is not compatible with return_assistant_tokens_mask.z`return_assistant_tokens_mask` is not possible with slow tokenizers. Make sure you have `tokenizers` installed. If the error persists, open an issue to support a Fast tokenizer for your model.TrK)rrrrrcontentrrrr)imagevideo)rurlrJr)rrrJbase64r)rrrJrrr)r) conversationsrr@rxrzr{c3:# UHoHo"SLv M M g7frrP)rim_listims r]r5ProcessorMixin.apply_chat_template..s^|GV]PR$V]|c3:# UHoHo"SLv M M g7frrP)rvid_listvids r]rrsbXYaRU4Yar)rrrroffset_mapping input_idsr assistant_masksr) tensor_typerP) rrrqrrrTrrrrQendswithrrrWrrrrVrrrOr rspecial_tokens_map bos_token startswithrrangerbisect bisect_leftconvert_to_tensors)1rrrris_tokenizers_fastprocessed_kwargs kwarg_typerkwarg_type_defaults default_valuer is_batchedrrrr batch_images batch_videos batch_audiosrrmessagervisuals audio_fnames vision_info image_fnames video_fnamesfnamepromptgeneration_indices single_prompt images_exist videos_existoutr r r r current_maskoffsetsstartend offset_startsassistant_start_charassistant_end_char start_posend_postoken_ids1 r]apply_chat_template"ProcessorMixin.apply_chat_templates_J  $,,d33 TEWEW8W $ 2 29 = D..55 kyy!3!3!8!8!:;<> ##/ $ 2 2  j$,,d33 I[I[8[ $ 2 2= A $T;7nDNN&N&Nz&Z# '(;$ G  36$Zt-D-D8=Z05 \+  '. *+226: lT5M 2 2 |Au 6 6',q/S\:]:]J(MJ)NM#$56:::uM&'89==mUS )*:; )+R,LL - !#R+G6=i6Hr6H7GTZO_qLqw6HGr(/y'9$'9G#;C'>%/6fo.H% #;%'9!$,3$+2K#EC+-)3>f2E2P) C(#E)+2!$ MM,/,3$+2K#;C+-)3>f2E2P) C(#;)+2!$ MM,/**AB%1E(// 5P^_nPo0pq&2&2E(// 5P^_nPo0pq&29 ,B##F+##F+I!.L&;& ''& 01& nn// & ""AYF *4F1IM~~''3 8P8PQUQ_Q_QiQi8j8j/4+,"/ 5!-L1I1U-1)*^|^^LbbbL'3|'3|&2l   C#$56::;Y[`aa&(O%(WW-=%>N #K 0I"3y>2()sS1->'> "0"3AH(I:5# (IHZ[\H]D02D(.(:(:=J^(_I&,&8&8HZ&[G!*Q$+I$6q$9=Q$iT[\eTfghTi$i!),1)WUXYbcdYeUf,g9: X 6-hI^(..|r?r token_strr6ids ids_countsample text_counts r]_check_special_mm_tokens'ProcessorMixin._check_special_mm_tokenss #H &&9:Itz%;I+>VW>Vsc2>VIW@DEf,,y1JE& #H:-abkallvxBwCCss#XEs $BBrP)NNNN)T)FTr)NFFNmain) AutoProcessor)$&S &"OK-$/P(,hl'+&* <%$<%uY(94 ?DQbLccde<%$ <%  # <% )* <%|*BDcNBH G3 G Y5bkk1A+B Y[ m4m_cm^@&,1#r{{2B,C@& tCH~tCH~- .@&@&D5d38n55t15@.@ (~@ c4i @D8<$!&,0<F ' (<F',S"++-='><FE#r{{"234<F <F  <F c4i() <F<F <F<F|%%*""H  0<6!! + +[&5AB(,TDc3h0$tDcN7K2LLMT }T12 T T B Tlq$T#Y^aefiajr\rvrLzprocessor files)object object_class object_files)[rUrrr r rGsystypingrD dataclassesrpathlibrrrrrr numpyrtyping_extensionshuggingface_hub.errorsr audio_utilsr r dynamic_module_utilsrfeature_extraction_utilsr image_utilsrrrutils.chat_template_utilsr video_utilsrrrtokenization_utils_baserrrrrutilsrrrrr r!r"r#r$r%r&r'r(r)r*r+utils.deprecationr,modeling_utilsr- get_loggerrQrQr.__file__parentrr version_inforOr;r`rtr~rrrrrrrr/r5formatrPr\r]rms)  !;;5/42JJ<2/$/<   H % 7?OP1h1F1FG/40. w ]]F  % %F:-%:-z49E4n@$9E@$F$*)5$*N59E57y7t$9)5$9N 2Ye 2 ("8:U]b (1y1  RR R2K^K\('~'A'AB%%1)7)C)C)K)K)R)RGX*S*N&2r\