9i0dZddlZddlZddlmZddlmZmZddlm Z ddl m Z ddl m Z mZmZmZej"eZeeej*Zd d d d ZGd de Zeej2e_ej2j8ej2jj5dddej2_yy)z9 Processing saving/loading class for common processors. N)Path)OptionalUnion)custom_object_save)PreTrainedTokenizerBase)PushToHubMixin copy_funcdirect_transformers_importloggingrFeatureExtractionMixinImageProcessingMixin) AutoTokenizerAutoFeatureExtractorAutoImageProcessorceZdZdZddgZdZdZdZdZdZ dde fdZ e dd e eej fd ee eej fd e d e d ee ee fdef dZe ddZe dZedZy)ProcessorMixinza This is a mixin used to provide saving/loading functionality for all processor classes. feature_extractor tokenizerNc |D]}||jvstd|dt||jD]\}}||vrtd|d|||<t|t|jk7rJt dt|jddj |jdt|d|j D]\}}t||d }tj||}t|trtd |D}ntt|}t||s(t d t|jd |d |dt|||y)NzUnexpected keyword argument .z!Got multiple values for argument zThis processor requires z arguments: z, z. Got z arguments instead._classc3BK|]}|tt|ywNgetattrtransformers_module.0ns g/var/www/html/backtest/airagagent/rag_env/lib/python3.12/site-packages/transformers/processing_utils.py z*ProcessorMixin.__init__..Ms$j\]\iW-@!%D$jsz Received a z for argument z, but a z was expected.) attributes TypeErrorziplen ValueErrorjoinitemsrAUTO_TO_BASE_CLASS_MAPPINGget isinstancetuplertype__name__setattr)selfargskwargskeyargattribute_name class_name proper_classs r!__init__zProcessorMixin.__init__6s GC$//)">se1 EFF G$'tT__#= - C'"CNCSST UVV),~&  - v;#doo. .*3t+?*@ TYYW[WfWfMgLhhnt9+02  $*<<> / NC .)9'@AJ377 JOJ*e,$$jj$jj &':JG c<0 !$s)"4"4!5^NCSS[\f[gguv D.# . /c |jDcgc]}d|dtt||}}dj|}|jj d|Scc}w)Nz- z:  z: )r#reprrr( __class__r/)r1nameattributes_reprs r!__repr__zProcessorMixin.__repr__XsfPTP_P_`RvRWT4-@(A'BC``))O4..))*#o->??as"A push_to_hubc :|jdd}| This class method is simply calling [`~feature_extraction_utils.FeatureExtractionMixin.save_pretrained`] and [`~tokenization_utils_base.PreTrainedTokenizerBase.save_pretrained`]. Please refer to the docstrings of the methods above for more information. Args: save_directory (`str` or `os.PathLike`): Directory where the feature extractor JSON file and the tokenizer files will be saved (directory will be created if it does not exist). push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). kwargs (`Dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. use_auth_tokenNrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.tokenV`token` and `use_auth_token` are both specified. Please set only the argument `token`.T)exist_okcommit_messagerepo_id)config_set_processor_classauto_map)rIrF)popwarningswarn FutureWarningr+r'osmakedirssplitpathsep _create_repo_get_files_timestamps _auto_classr#rr,r init_kwargsrhasattrrMr>r/save_pretrained_upload_modified_files) r1save_directoryrBr3rDrIrJfiles_timestampsr6attrsaconfigs attributes r!r]zProcessorMixin.save_pretrained]s0 $4d;  % MME zz'4(4 l-F7O NT2 #ZZ(8$?NjjN,@,@,Mb,QRG'd'':6:G#99.I     'IMY~WT>2YEYafg\]A7N)O UVVgGg t^G D"oo 6Nn5Iy"89..t~~/F/FG  % %n 5  6    '"&// :#D.9 i)@A!--j9 :   ' ' -jj) (  'Zgs 6H%Hpretrained_model_name_or_path cache_dirforce_downloadlocal_files_onlyrFrevisionc ||d<||d<||d<||d<|jdd}|)tjdt| t d|}|||d <|j |fi|} || S) a Instantiate a processor associated with a pretrained model. This class method is simply calling the feature extractor [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`], image processor [`~image_processing_utils.ImageProcessingMixin`] and the tokenizer [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`] methods. Please refer to the docstrings of the methods above for more information. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained feature_extractor hosted inside a model repo on huggingface.co. Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a user or organization name, like `dbmdz/bert-base-german-cased`. - a path to a *directory* containing a feature extractor file saved using the [`~SequenceFeatureExtractor.save_pretrained`] method, e.g., `./my_model_directory/`. - a path or url to a saved feature extractor JSON *file*, e.g., `./my_model_directory/preprocessor_config.json`. **kwargs Additional keyword arguments passed along to both [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`] and [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`]. rfrgrhrirDNrErGrF)rOrPrQrRr'_get_arguments_from_pretrained) clsrerfrgrhrFrir3rDr2s r!from_pretrainedzProcessorMixin.from_pretrainedsP({#1 %5!"%z$4d;  % MME   l#E  #F7O1s112OZSYZDzr:ct|ts |j}ddlmcm}t ||st|d||_y)a Register this class with a given auto class. This should only be used for custom feature extractors as the ones in the library are already mapped with `AutoProcessor`. This API is experimental and may have some slight breaking changes in the next releases. Args: auto_class (`str` or `type`, *optional*, defaults to `"AutoProcessor"`): The auto class to register this new feature extractor with. rNz is not a valid auto class.) r,strr/transformers.models.automodelsautor\r'rZ)rl auto_class auto_modules r!register_for_auto_classz&ProcessorMixin.register_for_auto_classsC *c*#,,J66{J/ |+FGH H$r:c <g}|jD]}t||d}t|tr7td|D}|j dd}|r |d|d}n|d}ntt |}|j |j|fi||S)Nrc3DK|]}|tt|ndywrrrs r!r"z@ProcessorMixin._get_arguments_from_pretrained..s#o_`1=(;Q ?VZ Zos use_fastTrr)r#rr,r-r+rappendrm) rlrer3r2r6r7classesrxattribute_classs r!rkz-ProcessorMixin._get_arguments_from_pretraineds!nn bN (8&?@J*e,odnoo!::j$7 6&-ajO&-ajO")*=z"J KK7778U`Y_` a b r:cNt||jd}t|ddS)Nrmodel_input_names)rr#)r1first_attributes r!r}z ProcessorMixin.model_input_namess'!$(:;(;TBBr:)F)NFFNmain) AutoProcessor)r/ __module__ __qualname____doc__r#feature_extractor_classtokenizer_classrZr9rAboolr] classmethodrrorSPathLikerrmrurkpropertyr}r:r!rr*s &{3J"OK /D@ I4IV8<$!&,0<',S"++-='><E#r{{"234< <  < c4i() <<<|%%2"CCr:r processorrzprocessor files)object object_class object_files)rrSrPpathlibrtypingrrdynamic_module_utilsrtokenization_utils_baserutilsr r r r get_loggerr/logger__file__parentrr*rrBformatrr:r!rs "4<QQ   H %1h1F1FG/40lC^lC^'~'A'AB%%1)7)C)C)K)K)R)RGX*S*N&2r: