9i'ddlZddlZddlZddlZddlmZddlmZddlm Z GddZ Gdd Z Gd d eZ Gd d e Z GddZGddZy)N)deepcopy)AcceleratedOptimizer)AcceleratedSchedulercPeZdZdZdZdZdZd dZddZdZ d Z d Z d Z d Z y)HfDeepSpeedConfigaJ This object contains a DeepSpeed configuration dictionary and can be quickly queried for things like zero stage. A `weakref` of this object is stored in the module's globals to be able to access the config from areas where things like the Trainer object is not available (e.g. `from_pretrained` and `_get_resized_embeddings`). Therefore it's important that this object remains alive while the program is still running. [`Trainer`] uses the `HfTrainerDeepSpeedConfig` subclass instead. That subclass has logic to sync the configuration with values of [`TrainingArguments`] by replacing special placeholder values: `"auto"`. Without this special logic the DeepSpeed configuration is not modified in any way. Args: config_file_or_dict (`Union[str, Dict]`): path to DeepSpeed config file or dict. ct|tr t|}ntjj |r7t j|dd5}tj|}dddn: tj|jd}tj|}|_|j%y#1swY!xYw#ttt f$rt!d|wxYw)Nrzutf-8)encodingzoExpected a string path to an existing deepspeed config, or a dictionary, or a base64 encoded string. Received: ) isinstancedictrospathexistsioopenjsonloadbase64urlsafe_b64decodedecodeloadsUnicodeDecodeErrorAttributeError ValueErrorconfigset_stage_and_offload)selfconfig_file_or_dictrfconfig_decodeds d/var/www/html/backtest/airagagent/rag_env/lib/python3.12/site-packages/accelerate/utils/deepspeed.py__init__zHfDeepSpeedConfig.__init__*s )4 012F WW^^/ 0,cGD &1 & & !'!9!9:M!N!U!UV]!^N3   ""$ & & ' C  FGZF[\ sC49CC#C4c$|jdd|_d|_|js|j rRt ddg}t |jd|jdg}t ||zdkDrd |_yyy) Nzzero_optimization.stageFcpunvmez*zero_optimization.offload_optimizer.devicez&zero_optimization.offload_param.devicerT) get_value_stage_offloadis_zero2is_zero3setlen)roffload_devices_validoffload_devicess r"rz'HfDeepSpeedConfig.set_stage_and_offload?snn%>C  ==?dmmo$'$8 !!NN#OPNN#KLO ?%::;a? $ @.c|j}|jd}|j}|D]}|j|}|d|fcS||fS)N.)rsplitpopget)r ds_key_longrnodesds_keynodes r"find_config_nodez"HfDeepSpeedConfig.find_config_nodeRs`!!#& $DZZ%F~V|# $ v~r1NcV|j|\}}||S|j||S)zG Returns the set value or `default` if no value is set )r;r6)rr7defaultrr9s r"r(zHfDeepSpeedConfig.get_value_s3..{; >Nzz&'**r1c|j}|jd}|D]6}|}|j|}||rtd|d|jy|j yy)z Deletes a sub-section of the config file if it's found. Unless `must_exist` is `True` the section doesn't have to exist. r3Nz Can't find z entry in the config: )rr4r6rr5)rr7 must_existrr8r: parent_configs r"del_config_sub_treez%HfDeepSpeedConfig.del_config_sub_treehs !!#& D"MZZ%F~${;-?UVZVaVaUb%cdd   $   d # %r1cB|j|}|dSt|S)z Returns `True`/``False` only if the value is set, always `False` otherwise. So use this method to ask the very specific question of whether the value is set to `True` (and it's not set to `False`` or isn't set). Fr(boolrr7values r"is_truezHfDeepSpeedConfig.is_trues% {+ u64;6r1cD|j|}|dSt| S)z Returns `True`/``False` only if the value is set, always `False` otherwise. So use this method to ask the very specific question of whether the value is set to `False` (and it's not set to `True`` or isn't set). FrCrEs r"is_falsezHfDeepSpeedConfig.is_falses' {+ u:tE{?:r1c |jdk(S)Nrr)rs r"r+zHfDeepSpeedConfig.is_zero2{{ar1c |jdk(S)NrKrLs r"r,zHfDeepSpeedConfig.is_zero3rMr1c|jSN)r*rLs r" is_offloadzHfDeepSpeedConfig.is_offloads }}r1rQ)F)__name__ __module__ __qualname____doc__r#rr;r(rArGrIr+r,rRr1r"rrs9 %*%& +$.7;  r1rceZdZdZdZdZy)DeepSpeedEngineWrapperz Internal wrapper for deepspeed.runtime.engine.DeepSpeedEngine. This is used to follow conventional training loop. Args: engine (deepspeed.runtime.engine.DeepSpeedEngine): deepspeed engine to wrap c||_yrQ)engine)rr[s r"r#zDeepSpeedEngineWrapper.__init__s  r1c r|jj|fi||jjyrQ)r[backwardstep)rlosskwargss r"r]zDeepSpeedEngineWrapper.backwards- T,V, r1N)rSrTrUrVr#r]rWr1r"rYrYs r1rYc@eZdZdZfdZddZdZedZxZ S)DeepSpeedOptimizerWrapperz Internal wrapper around a deepspeed optimizer. Args: optimizer (`torch.optim.optimizer.Optimizer`): The optimizer to wrap. c`t||ddt|jd|_y)NF)device_placementscaleroverflow)superr#hasattr optimizer__has_overflow__)rri __class__s r"r#z"DeepSpeedOptimizerWrapper.__init__s* U4H ' Cr1cyrQrW)r set_to_nones r" zero_gradz#DeepSpeedOptimizerWrapper.zero_grad r1cyrQrWrLs r"r^zDeepSpeedOptimizerWrapper.stepror1cH|jr|jjSy)zTWhether or not the optimizer step was done, or skipped because of gradient overflow.F)rjrirfrLs r"step_was_skippedz*DeepSpeedOptimizerWrapper.step_was_skippeds   >>** *r1rQ) rSrTrUrVr#rnr^propertyrr __classcell__rks@r"rbrbs,D  r1rbc(eZdZdZfdZdZxZS)DeepSpeedSchedulerWrapperz Internal wrapper around a deepspeed scheduler. Args: scheduler (`torch.optim.lr_scheduler.LambdaLR`): The scheduler to wrap. optimizers (one or a list of `torch.optim.Optimizer`): c&t|||yrQ)rgr#)r scheduler optimizersrks r"r#z"DeepSpeedSchedulerWrapper.__init__s J/r1cyrQrWrLs r"r^zDeepSpeedSchedulerWrapper.stepror1)rSrTrUrVr#r^rtrus@r"rwrws0 r1rwceZdZdZddZy) DummyOptima Dummy optimizer presents model parameters or param groups, this is primarily used to follow conventional training loop when optimizer config is specified in the deepspeed config file. Args: lr (float): Learning rate. params (iterable): iterable of parameters to optimize or dicts defining parameter groups weight_decay (float): Weight decay. **kwargs: Other arguments. c <||_||_||_||_yrQ)paramslr weight_decayr`)rrrrr`s r"r#zDummyOptim.__init__s ( r1N)gMbP?rrSrTrUrVr#rWr1r"r}r}s  r1r}ceZdZdZddZy)DummySchedulera Dummy scheduler presents model parameters or param groups, this is primarily used to follow conventional training loop when scheduler config is specified in the deepspeed config file. Args: optimizer (`torch.optim.optimizer.Optimizer`): The optimizer to wrap. total_num_steps (int, *optional*): Total number of steps. warmup_num_steps (int, *optional*): Number of steps for warmup. lr_scheduler_callable (callable, *optional*): A callable function that creates an LR Scheduler. It accepts only one argument `optimizer`. **kwargs: Other arguments. Nc J||_||_||_||_||_yrQ)ritotal_num_stepswarmup_num_stepslr_scheduler_callabler`)rrirrrr`s r"r#zDummyScheduler.__init__ s(". 0%:" r1)NrNrrWr1r"rrs "r1r)rrrrcopyrrirryrrrYrbrwr}rrWr1r"rsa ,,~~B8 46 4 ".r1