;i9SSKrSSKrSSKrSSKJr SSKJr SSKJr SSK J r SSK J r SS K Jr SS KJr S rS r"S S5r"SS5r"SS\5r"SS\ 5r"SS5r"SS5rg)N)deepcopy)optim)AcceleratedOptimizer)AcceleratedScheduler)DistributedType)is_bnb_available)compare_versionscURR5VVs0sHupUS;dM X_M nnnSSKJn Un[ SSS5(aSUS'[ U[ R5n[5(aFU(d?SS K Jn [ XRUR45(aURS :HnOSnU(aS US'[ SSS 5(a{[ U[ R5n[5(aCU(d<SS K Jn [ XRUR45(aURS :HnU(aSS KJn U n[SS9(aK[ SSS5(a9SSK Jn Jn [ X U 45(aURS :Hn U (aSSKJn U nU"UR,40UD6$s snnf![a URRS :HnGN1f=f![a URRS :HnNf=f![a URRS :Hn Nf=f)zx Args: optimizer: torch.optim.Optimizer Returns the DeepSeedCPUOptimizer (deepspeed.ops) version of the optimizer. )lr weight_decayr)DeepSpeedCPUAdam deepspeedz>=z0.3.1F adamw_modeN Tz0.5.5)DeepSpeedCPUAdagradz0.38.0) min_versionz0.11.0)Lion Lion32bit)DeepSpeedCPULion)defaultsitemsdeepspeed.ops.adamrr isinstancerAdamWr bitsandbytes.optim AdamW32bit optim_bitsAttributeErrorargsAdagrad Adagrad32bitdeepspeed.ops.adagradrrrdeepspeed.ops.lionr param_groups) optimizerkvrroptimizer_classis_adawbnb_optis_adarrr is_bnb_32bitsrs j/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/accelerate/utils/deepspeed.pymap_pytorch_optim_to_deepspeedr0s"+!3!3!9!9!;[!;qDZ?Z!;H[4&O T733!&Y 4   g 0)mmW5G5G%HII>'22b8G  %)H\ " T733Iu}}5   f 0)oow7K7K%LMM=&11R7F  A1OH-2B;PTV^2_2_6 i !2 3 3 @ ) 4 4 : ?"2 911 >X >>y\&&>'nn772=G>(&=&^^66".get_active_deepspeed_plugin..tsY%E6%Es# #) distributed_typer DEEPSPEED ValueErrorrdeepspeed_pluginsdictnextvalues)states r/get_active_deepspeed_pluginrAdsk !:!:: ,  e--t 4 4&&& YU%<%<%C%C%EY YYc\\rSrSrSrSrSrSrSSjrSSjr S r S r S r S r S rSrg)HfDeepSpeedConfigwa" This object contains a DeepSpeed configuration dictionary and can be quickly queried for things like zero stage. A `weakref` of this object is stored in the module's globals to be able to access the config from areas where things like the Trainer object is not available (e.g. `from_pretrained` and `_get_resized_embeddings`). Therefore it's important that this object remains alive while the program is still running. [`Trainer`] uses the `HfTrainerDeepSpeedConfig` subclass instead. That subclass has logic to sync the configuration with values of [`TrainingArguments`] by replacing special placeholder values: `"auto"`. Without this special logic the DeepSpeed configuration is not modified in any way. Args: config_file_or_dict (`Union[str, Dict]`): path to DeepSpeed config file or dict. cZ[U[5(a [U5nOf[RR U5(a*[ USS9n[R"U5nSSS5 O[R"U5nWUlUR%5 g!,(df  N&=f![Ra> [R"U5RS5n[R"U5nNzf=f![[[ 4a [!SU35ef=f)Nzutf-8)encodingzoExpected a string path to an existing deepspeed config, or a dictionary, or a base64 encoded string. Received: )rr=rospathexistsopenjsonloadloadsJSONDecodeErrorbase64urlsafe_b64decodedecodeUnicodeDecodeErrorr r;configset_stage_and_offload)selfconfig_file_or_dictrTfconfig_decodeds r/__init__HfDeepSpeedConfig.__init__s  )4 0 012F WW^^/ 0 0)G<1=< 8!ZZ(;Q%R%Y%YZa%bN!ZZ7F8' C  FGZF[\ s1B 2B1 B.1ADDDD$D*c8URSS5UlSUlUR5(dUR 5(aR[ SS/5n[ URS5URS5/5n[ X!-5S:aS Ulggg) Nzzero_optimization.stageFcpunvmez*zero_optimization.offload_optimizer.devicez&zero_optimization.offload_param.devicerT) get_value_stage_offloadis_zero2is_zero3setlen)rVoffload_devices_validoffload_devicess r/rU'HfDeepSpeedConfig.set_stage_and_offloadsnn%>C  ==??dmmoo$'$8 !!NN#OPNN#KLO ?:;a? $ @.rBcURnURS5nUR5nUHnURU5nUbMSU4s $ X$4$)N.)rTsplitpopget)rV ds_key_longrTnodesds_keynodes r/find_config_node"HfDeepSpeedConfig.find_config_nodesY!!#&DZZ%F~V|# ~rBNcTURU5up4UcU$URXB5$)z7 Returns the set value or `default` if no value is set )rsrn)rVrodefaultrTrqs r/r`HfDeepSpeedConfig.get_values/..{; >Nzz&**rBcURnURS5nUH=nUnURU5nUbMU(a[SUSUR35e g WbUR W5 gg)z{ Deletes a sub-section of the config file if it's found. Unless `must_exist` is `True` the section doesn't have to exist. rkNz Can't find z entry in the config: )rTrlrnr;rm)rVro must_existrTrprr parent_configs r/del_config_sub_tree%HfDeepSpeedConfig.del_config_sub_trees !!#&D"MZZ%F~${;-?UVZVaVaUb%cdd  $   d # %rBcDURU5nUcS$[U5$)z Returns `True`/``False` only if the value is set, always `False` otherwise. So use this method to ask the very specific question of whether the value is set to `True` (and it's not set to `False`` or isn't set). Fr`boolrVrovalues r/is_trueHfDeepSpeedConfig.is_trues% {+ u64;6rBcNURU5nUcS$[U5(+$)z Returns `True`/``False` only if the value is set, always `False` otherwise. So use this method to ask the very specific question of whether the value is set to `False` (and it's not set to `True`` or isn't set). Fr~rs r/is_falseHfDeepSpeedConfig.is_falses' {+ u:tE{?:rBc URS:H$)NrrarVs r/rcHfDeepSpeedConfig.is_zero2{{arBc URS:H$)Nrrs r/rdHfDeepSpeedConfig.is_zero3rrBcUR$r3)rbrs r/ is_offloadHfDeepSpeedConfig.is_offloads }}rB)rbrarTr3)F)__name__ __module__ __qualname____firstlineno____doc__rZrUrsr`r{rrrcrdr__static_attributes__rBr/rDrDws9 %4%& +$.7;  rBrDc.\rSrSrSrSrSSjrSrSrg) DeepSpeedEngineWrapperz Internal wrapper for deepspeed.runtime.engine.DeepSpeedEngine. This is used to follow conventional training loop. Args: engine (deepspeed.runtime.engine.DeepSpeedEngine): deepspeed engine to wrap cXlgr3engine)rVrs r/rZDeepSpeedEngineWrapper.__init__s rBc URRUS9 URR"U40UD6 U(aURR5 gg)N) is_boundary)r"set_gradient_accumulation_boundarybackwardstep)rVlosssync_gradientskwargss r/rDeepSpeedEngineWrapper.backwardsN 66>6R T,V,  KK    rBc|URR5n[US5(aUR5$U$)z3Get the global gradient norm from DeepSpeed engine.item)rget_global_grad_normhasattrr)rV grad_norms r/r+DeepSpeedEngineWrapper.get_global_grad_norms4KK446 9f % %>># #rBrN)T) rrrrrrZrrrrrBr/rrs,rBrcL^\rSrSrSrU4SjrSSjrSr\S5r Sr U=r $) DeepSpeedOptimizerWrapperi'z Internal wrapper around a deepspeed optimizer. Args: optimizer (`torch.optim.optimizer.Optimizer`): The optimizer to wrap. cZ>[TU]USSS9 [URS5Ulg)NF)device_placementscaleroverflow)superrZrr'__has_overflow__)rVr' __class__s r/rZ"DeepSpeedOptimizerWrapper.__init__0s* U4H ' CrBcgr3r)rV set_to_nones r/ zero_grad#DeepSpeedOptimizerWrapper.zero_grad4 rBcgr3rrs r/rDeepSpeedOptimizerWrapper.step7rrBcRUR(aURR$g)zTWhether or not the optimizer step was done, or skipped because of gradient overflow.F)rr'rrs r/step_was_skipped*DeepSpeedOptimizerWrapper.step_was_skipped:s   >>** *rB)rr3) rrrrrrZrrpropertyrr __classcell__rs@r/rr's,D  rBrc2^\rSrSrSrU4SjrSrSrU=r$)DeepSpeedSchedulerWrapperiBz Internal wrapper around a deepspeed scheduler. Args: scheduler (`torch.optim.lr_scheduler.LambdaLR`): The scheduler to wrap. optimizers (one or a list of `torch.optim.Optimizer`): c$>[TU]X5 gr3)rrZ)rV scheduler optimizersrs r/rZ"DeepSpeedSchedulerWrapper.__init__Ls /rBcgr3rrs r/rDeepSpeedSchedulerWrapper.stepOrrBr) rrrrrrZrrrrs@r/rrBs0  rBrc"\rSrSrSrSSjrSrg) DummyOptimiSa Dummy optimizer presents model parameters or param groups, this is primarily used to follow conventional training loop when optimizer config is specified in the deepspeed config file. Args: lr (float): Learning rate. params (iterable): iterable of parameters to optimize or dicts defining parameter groups weight_decay (float): Weight decay. **kwargs (additional keyword arguments, *optional*): Other arguments. c 4XlX lX0lX@lgr3)paramsr rr)rVrr rrs r/rZDummyOptim.__init__cs ( rB)rr rrN)gMbP?rrrrrrrZrrrBr/rrSs  rBrc"\rSrSrSrSSjrSrg)DummySchedulerija Dummy scheduler presents model parameters or param groups, this is primarily used to follow conventional training loop when scheduler config is specified in the deepspeed config file. Args: optimizer (`torch.optim.optimizer.Optimizer`): The optimizer to wrap. total_num_steps (int, *optional*): Total number of steps. warmup_num_steps (int, *optional*): Number of steps for warmup. lr_scheduler_callable (callable, *optional*): A callable function that creates an LR Scheduler. It accepts only one argument `optimizer`. **kwargs (additional keyword arguments, *optional*): Other arguments. Nc @XlX lX0lX@lXPlgr3)r'total_num_stepswarmup_num_stepslr_scheduler_callabler)rVr'rrrrs r/rZDummyScheduler.__init__|s". 0%:" rB)rrr'rr)NrNrrrBr/rrjs "rBr)rPrLrHcopyrtorchrr'rrr dataclassesr importsr versionsr r0rArDrrrrrrrBr/rsx ,,(%&D?NZ&CCL''T 46 4 ".rB