;iӓ SSKrSSKrSSKrSSKrSSKrSSKrSSKJr SSKJ r SSK J r SSK J r SSKJrJr SSKrSSKJr S S KJrJrJrJr S S KJr S S KJrJr S S KJrJ r J!r! S SK"J#r# \"\$5r%Sr&Sr'S0Sjr(S0Sjr)Sr*S1Sjr+S1Sjr,S2Sjr-S1Sjr.S3S\/S\/S\04Sjjr1S4S\/S\/S\0S\04Sjjr2S\RfRhS \Rj4S!jr6S\RfRhS"\74S#jr8S$\RrRtS%\74S&jr;S\RfRh4S'jrS+r?S,\7S(\74S-jr@S.\\ \RfRh\/4S(\A\RfR4S/jrCg)5N) defaultdict)Iterable) nullcontext)Path)CallableUnion) get_logger)FSDP_MODEL_NAMEOPTIMIZER_NAMESAFE_WEIGHTS_NAME WEIGHTS_NAME)get_module_class_from_name)get_non_persistent_buffers is_peft_model)get_module_children_bottom_upis_compiled_modulesave)is_torch_versioncxS[R;aS[RS'S[RS'g)zS Enables RAM efficient loading of Hugging Face models for FSDP in the environment. ACCELERATE_USE_FSDPTrueFSDP_CPU_RAM_EFFICIENT_LOADINGNosenvironk/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/accelerate/utils/fsdp_utils.py!enable_fsdp_ram_efficient_loadingr!'s. BJJ.,2 ()39BJJ/0rc*S[RS'g)zT Disables RAM efficient loading of Hugging Face models for FSDP in the environment. FalserNrrrr "disable_fsdp_ram_efficient_loadingr$1s4;BJJ/0rcU(a&[U5(aSSKJn U"XRS9$Ub SSKJn U"XS9$UR 5$)Nr)get_peft_model_state_dict adapter_name)get_model_state_dictoptions)rpeftr&active_adapter'torch.distributed.checkpoint.state_dictr) state_dict)model adapter_only sd_optionsr&r)s r _get_model_state_dictr38sH e,,2(=Q=QRRP#E>>!!rcU(a'[U5(aSSKJn U"XURS9$Ub SSKJn U"XUS9$UR U5$)Nr)set_peft_model_state_dictr')set_model_state_dictr*)rr,r5r-r.r6load_state_dict)r0r/r1r2r5r6s r _set_model_state_dictr8GsN e,,2(I]I]^^P#EzJJ$$Z00rc SnURS:XaUSSKJn SSKJn U"UR UR :H[URSS5[URSS5S9nU$) Nr r)StateDictOptions StateDictTypeoffload_to_cpuF rank0_only)full_state_dict cpu_offloadbroadcast_from_rank0) fsdp_versionr.r:2torch.distributed.fsdp.fully_sharded_data_parallelr<state_dict_typeFULL_STATE_DICTgetattrstate_dict_config) fsdp_pluginr2r:r<s r _prepare_sd_optionsrIVsjJ1$LT%'77=;X;XX = =?OQVW!()F)F V[!\ rcSSKJs Jn SSKJn SSKJn SSKJn [R"USS9 URU R:Xa/URS:n XRlXRlUR S:Xa1URX RURUR"5O [%5n ['U5n U  [)X%U S9n URU R:XaUS:Xa [*S 3O [*S US 3n[R,R/X>5nUR0S:XaF[2R5S U35 [6R8"X5 [2R5S U35 GOfURU R::XaUS:Xa[*S UR0S 3O[*S US UR0S 3n[R,R/X>5n[2R5S U35 [6R8"X5 [2R5S U35 OURU R<:Xa[R,R/U[*S U35n[R"USS9 [2R5S U35 SU 0n UR9U UR?U5U"5S9 [2R5S U35 SSS5 g!,(df  g=f)NrDefaultSavePlannerFullyShardedDataParallelr;Texist_okr r1r2.bin_zSaving model to zModel saved to _rankr0r/storage_writerplanner) torch.distributed.checkpoint distributed checkpoint,torch.distributed.checkpoint.default_plannerrLrCrNr<rmakedirsrDrE num_processesrGr=r>rBoptim_state_dict_configrrIr3r pathjoin process_indexloggerinfotorchrLOCAL_STATE_DICTSHARDED_STATE_DICTFileSystemWriter)rH acceleratorr0 output_dir model_indexr1dist_cprLFSDPr<is_multi_processctxr2r/ weights_nameoutput_model_fileckpt_dirs r save_fsdp_modelrrgs22OcPKK T*""m&C&CC'44q87G%%43C%%0  # #q (  .. 0M0M{OrOr ] %[1J *5Xbc  & &-*G*G G7Ba7Go.d3P_O``abmannrMsL " Z F ((A- ./@.ABC :9 o.?-@AB  ( (M,J,J J!###5)B)B(C4H'(+eK ? JJz 5 KK/*;)<= >  ( (M,L,L Lww||J?2C1[M0RSH KK4 0 KK*8*5 6!:.J LL%&77A*,   KK/(4 5? s HK88 LcSSKJs Jn SSKJn SSKJn SSKJn UR5 URU R:Xa/URS:n XRl XRlURS:Xa1URX RURUR 5O [#5n [%U5n U  URU R:XGa['U5ULaWUR(S:waGUR*(d6UR,(dURS:Xa [/S5eSSS5 gUS:Xa [0S3O [0SUS3n [2R4R7X=5n[8R;S U35 UR*(+=(d UR<nU(a[>R@"US S 9nO0n[8R;S U35 GOjURU RB:XaUS:Xa[0S UR(S3O[0SUS UR(S3n [2R4R7X=5n[8R;S U35 [>R@"US S 9n[8R;S U35 OURU RD:Xa[0U;a)[2R4R7U[0SU35OUn[8R;S U35 S[GX%U S90nURAUURIU5U"5S9 USn[8R;S U35 [KUWX\S9nSSS5 U$!,(df  W$=f)Nr)DefaultLoadPlannerrMr;r zzSet the `sync_module_states` flag to `True` so that model states are synced across processes when initializing FSDP objectrRrSzLoading model from T weights_onlyzModel loaded from rTr0rQ)r/storage_readerrW)&rXrYrZr[rtrCrNr<wait_for_everyonerDrEr]rGr=r>rBr^rrItyperais_fsdp2sync_module_states ValueErrorr rr_r`rbrcis_main_processrdloadrerfr3FileSystemReaderr8)rHrhr0 input_dirrjr1rkrtrlr<rmrnr2roinput_model_file load_modelr/rq load_results r load_fsdp_modelrsO22OcP!!#""m&C&CC'44q87G%%43C%%0  # #q (  .. 0M0M{OrOr ] %[1J  & &-*G*G GE{$&;+D+D+IR]RfRf"55+:R:RVW:W$3 8Ca7Go.d3P_O``abmannrMsL!ww||ID  KK-.>-?@ A(111P[5P5PJ"ZZ(8tL  KK,-=,>? @  ( (M,J,J J!###5)B)B(C4H'(+eK-?@ A$44HJ KK,-=,>? @  ( (M,L,L L&&y8 Y?*;1[M(JK  KK-hZ8 9!#8fp#qrJ LL%&77A*,   $G,J KK,XJ7 8+E:Lp [ \ ] \ s A8M<H$M<< N c&SSKJs Jn SSKJn SSKJn SSKJn [R"USS9 URS:Xa1URX0RURUR5O [5n [U5n U  URS:XaSS KJn U "X2U S 9n OUR%X25n URU R&:XaUR(S:XaUS:Xa [*S 3O [*S US 3n[R,R/XN5n[0R3S U35 [4R6"X5 [0R3SU35 O[R,R/U[*S U35n[R"USS9 [0R3S U35 UR7SU 0UR9U5U"5S9 [0R3SU35 SSS5 g!,(df  g=f)NrrKrMr;TrOr r )get_optimizer_state_dictr*rRrSzSaving Optimizer state to zOptimizer state saved in optimizerrU)rXrYrZr[rLrCrNr<rr\rBrDrGr^rrIr.roptim_state_dictrErar r_r`rbrcrdrrg)rHrhrr0rioptimizer_indexrkrLrlr<rnr2r optim_stateoptim_state_nameoutput_optimizer_filerqs r save_fsdp_optimizerrs22OcPKK T*  # #q (  .. 0M0M{OrOr ] %[1J  # #q ( X25ZXK//AK  & &-*G*G G((A-/>!/C~&d+NK[[\]l\mmqIr!)+ Z(R% 89N8OPQ ;> 78M7NOPww||J>2B!OCT0UVH KK4 0 KK4XJ? @ LL'5&77A*,   KK3H:> ?5 s E0H HcxSSKJs Jn SSKJn SSKJn UR 5 URS:Xa1URX0RURUR5O [5n [U5n U  URU R:XaSn URS:XdURR(dUS:Xa [ S3O [ SUS3n ["R$R'XM5n[(R+SU35 [,R."USS 9n [(R+S U35 O[ U;a)["R$R'U[ SU35OUn[(R+S U35 S UR150n UR/U UUR3U5S 9 U S n [(R+SU35 URS:Xa"UR5X2U S9nUR7U5 O SSKJn U"X2XS9 SSS5 g!,(df  g=f)NrrMr;r rRrSzLoading Optimizer state from TruzOptimizer state loaded from zLoading Optimizer from r) checkpoint_idrwzOptimizer loaded from )r0optimr)set_optimizer_state_dictr*)rXrYrZrCrNr<rxrBrDrGr^rrIrErar>r rr_r`rbrcrdr~r/roptim_state_dict_to_loadr7r.r)rHrhrr0rrr1rkrlr<rnr2roptimizer_nameinput_optimizer_filerq flattened_osdrs r load_fsdp_optimizerrs22cP!!#  # #q (  .. 0M0M{OrOr ] %[1J  & &-*G*G GK((A-[5X5X5c5c/>!/C~&d+NK[[\]l\mmqIr(*ww||I'N$ ;*:!O;L(MN  KK1(< =& (<(<(>?K LL&&77A   &k2K KK0 ; <  # #q ( 99it9uM  % %m 4 X $U{ WC s >F$H++ H9checkpoint_dir save_pathsafe_serializationchSSKJs Jn SSKJs Js Jn 0n[ U5nUR SS9 URUURU5UR5SS9 U(a U[- OU[- n[UR55S:XaU[U5Sn[XQUS9 U$)z Passthrough to `torch.distributed.checkpoint.format_utils.dcp_to_torch_save` Will save under `save_path` as either `model.safetensors` or `pytorch_model.bin`. rNTrO)rwrWno_distr )r)rXrYrZ)torch.distributed.checkpoint.format_utils format_utilsrmkdir_load_state_dictr_EmptyStateDictLoadPlannerrrlenkeyslistr)rrrrkdist_cp_format_utilsr/s r )_distributed_checkpoint_to_merged_weightsrLs32LLJYI OOTO"))//?$??A * 2D --UaIaI :?? "Z 0 34 3EF r output_pathremove_checkpoint_dirc[U5nSSKJn [SS5(d [ S5eUR 5(dUS- R 5nUS- R 5nSUS 3nU(aU(aUS - nUS US US 3- nUS- nO+U(aUS- nUSUS3- nOU(aUS- nUSUS3- n[ U5eU"5nUR (aq[RSU35 [XU5n [RSU 35 U(a.[RSU35 [R"U5 UR5 g)a Merge the weights from sharded FSDP model checkpoints into a single combined checkpoint. Should be used if `SHARDED_STATE_DICT` was used for the model. Weights will be saved to `{output_path}/model.safetensors` if `safe_serialization` else `pytorch_model.bin`. Note: this is a CPU-bound process. Args: checkpoint_dir (`str`): The directory containing the FSDP checkpoints (can be either the model or optimizer). output_path (`str`): The path to save the merged checkpoint. safe_serialization (`bool`, *optional*, defaults to `True`): Whether to save the merged weights with safetensors (recommended). remove_checkpoint_dir (`bool`, *optional*, defaults to `False`): Whether to remove the checkpoint directory after merging. r) PartialStatez>=z2.3.0z/`merge_fsdp_weights` requires PyTorch >= 2.3.0`pytorch_model_fsdp_0 optimizer_0zTried to load from z) but couldn't find a valid metadata file.zE However, potential model and optimizer checkpoint directories exist.zPlease pass in either z/pytorch_model_fsdp_0 or z /optimizer_0zinstead.z8 However, a potential model checkpoint directory exists.zPlease try passing in z/pytorch_model_fsdp_0 instead.z< However, a potential optimizer checkpoint directory exists.z/optimizer_0 instead.zMerging FSDP weights from z.Successfully merged FSDP weights and saved to z"Removing old checkpoint directory N) raccelerate.staterrr|existsr}rbrcrshutilrmtreerx) rrrrrmodel_path_existsoptimizer_path_existserrstaters r merge_fsdp_weightsrhsx(.)N- D' * *JKK  " "+.DDLLN!/-!? G G I#N#33\] !6 Z ZC +N+;;TUcTddpq qC : C  M MC +N+;;YZ ZC " Q QC +N+;;PQ QCo NE  00@AB=n[mn  DYKPQ KK<^[[5nURSS9H3up#[U5T;dMU[U5R U5 M5 T"U5nUR 5H2upEUH'nTUnUc[ X5TU'M[XU5 M) M4 U$)NF)recurse)rrnamed_parametersidappenditemsrFsetattr) r params_to_tienparamid_key _param_names param_name _tied_params param_init_fns r param_init_fn_tied_param7ensure_weights_retied..param_init_fn_tied_params$D) ///>HA%yL(bi(//2? v&%2$7$7$9 F* $V,=,36+FL(F6+%: r)rFsplitr` get_submodulerrdnnModule) rr0r _tied_namesnamermodrrrs ` @r ensure_weights_retiedrs%!5t._infer_parameter_dtypes =55jAI%,E?$C! !:!g{?P?PTYTgTg?g    . .7M%OOM$B)@)@)BMQQ =0:0A0A#q0I -O++O._cast_and_contiguouss.  YYUY+F &&(F r)srcgroup)rrT)assign)torch.distributedrYtorch.distributed.tensorrr/r}ziprvalues device_meshdetachr device_type broadcastrWORLD placementsrdemptysizerr7)rhr0rdistrmeta_sharded_sd sharded_sdrrr full_param sharded_paramrsharded_tensorrr full_tensors r fsdp2_load_full_state_dictrs%:&&(OJR$""7:7==?OLbLbLd7e 3 $Zm'33K#**,// 0G0GHJ NN:1DJJ4D4DN E.z H`H`aN+A, (M 2.Q^_N%3z "8f*9)>)>)@ %J'33K++m&8&8&:;CZCZbobubuvK NN;ATZZ5E5EN F.{KIaIabN+A, (M 2.Q^_N%3z "*A *T2 LrrmappingcSSKJn 0nSX2'URH(nUSVs/sHoQURPM snUS'M* gs snf![a [ S5ef=f)a Switches the parameters of the optimizer to new ones (sharded parameters in usual case). This function modifies the optimizer in-place. Args: optimizer (`torch.optim.Optimizer`): Optimizer instance which contains the original model parameters mapping (`dict`): Mapping from the original parameter (specified by `data_ptr`) to the sharded parameter Raises: KeyError: If a parameter in the optimizer couldn't be switched to its sharded version. This should never happen and indicates a bug. If we kept the original params instead of raising, the training wouldn't be numerically correct and weights wouldn't get updated. r)DTensor _local_tensorparamszA parameter in the optimizer couldn't be switched to its sharded version. This breaks the training. Please raise an issue on GitHub.N)rr  param_groupsdata_ptrKeyError)rrr accessor_mapping param_groupps r !fsdp2_switch_optimizer_parametersrsy1 / $11KBMhBW$XBWQQZZ%8BW$XK !2$X   S   sA A A A A"cnSSKJn [URRU5n[ USS9SSH}upE[ URS55S:aURSS5upgOSnUnU(aURU5OUnU"U5(dMeU"US S 9nURXu5 M U$) a Applies the activation checkpointing to the model. Args: accelerator (`Accelerator`): The accelerator instance model (`torch.nn.Module`): The model to apply the activation checkpointing to Returns: `torch.nn.Module`: The model with the activation checkpointing applied r)checkpoint_wrapperT) return_fqnsNrrr F)preserve_rng_state) ;torch.distributed.algorithms._checkpoint.checkpoint_wrapperrfsdp2_prepare_auto_wrap_policyrrHrrrrrregister_module) rhr0rauto_wrap_policy_func layer_namelayer parent_name child_name parent_modules r fsdp2_apply_acr!9s;;;L;L;X;XZ_`:5dSTWUWX z$ % )&0&7&7Q&? #KK#JTRcg[U[T55$)NF)transformer_cls_names_to_wrapr4r8)rrItransformer_cls_to_wraps r policy.fsdp2_prepare_auto_wrap_policy..policys%99Afe,C&DE Ercb>[SUR555nUTR:$)Nc3@# UHoR5v M g7f)N)numel).0rs r Afsdp2_prepare_auto_wrap_policy..policy..s#K7J!GGII7Js)sum parametersmin_num_params)rmodule_num_paramsrIs r r^r_s. ##Kv7H7H7J#K K $|'B'BB Br)torch.distributed.fsdp.wraprXrYauto_wrap_policyr4 functoolspartialfuncrFrr\setrr|addrdrrbool) rIr0rXrYfnno_split_modulesr\ layer_classtransformer_clsr^r]s ` @r rrsf  & &B"i''(( WW )"5*=tD  #! (,-=(>%  5 5 A,8,V,V )"%%8K8LO& #N{m[i!jkk # ' ' 8 9  F588?? Ft F F M * C588?? Ct C Mrc SSKJn U"S0UD6$)z Returns a `GradScaler` for FSDP2, as the current implementation of `get_grad_scaler` doesn't accept other args. We need this as current `get_grad_scaler` accepts only `distributed_type` as arg, which doesn't differentiate between FSDP1 and FSDP2 r) GradScalerr)torch.amp.grad_scalerrw)kwargsrws r get_fsdp2_grad_scalerrzs 1   r named_paramscUR5VVs0sHupURSS5U_M nnnUR5VVs0sH0upURS5(aURSS5OUU_M2 nnnUR5VVs0sHupURSS5U_M nnnU$s snnfs snnfs snnf)aRemoves parameter name modifiers in order to map them back to their original names. See huggingface/accelerate#3554 for more context. Args: named_params (`dict`): The named parameters dictionary to canonicalize. Returns: `dict`: The canonicalized named parameters dictionary z._checkpoint_wrapped_modulez _orig_mod.z ._orig_mod)rreplace startswith)r{rNrOs r fsdp2_canonicalize_namesr sQ]PbPbPdePdAII;R@!CPdLeXdXjXjXlXlPTPQq||L'A'A ,#q!KXl@L?Q?Q?ST?StqAIIlB/2?SLT  fUsB: 7CCmodulescUc [5$/n[U[5(am[R"U5n/nUR 5H?upgUR U5(dMURU5 URU5 MA UnUH+nUR[UR555 M- [U5$)zConverts modules to parameters where modules can be a string or list of torch.nn.Module Args: modules (`Union[Iterable[torch.nn.Module], str]`): List of modules Returns: `set[torch.nn.Parameter]`: List of parameters ) ror4strrecompile named_modules fullmatchrrextendrrg)rr0rrgregmapped_modulesrrs r r;r;su J'3jj!!//1LD}}T"" &!%%f-2!$v00234 z?r)FN)rF)r)T)TF)Dr@rlrrrrF collectionsrcollections.abcr contextlibrpathlibrtypingrrrdloggingr constantsr r rr dataclassesrmodelingrrotherrrrversionsrr>rbr!r$r3r8rIrrrrrrrqrrrrrrdictrr Optimizerrr!rVrrzrro Parameterr;rrr rs  #$"" WW3?JJ& H :; " 1"76tEP-@`0Xfccgk:kp44&)4?C4cg4n+$+$+$\J588??JTJZ 1F1F QU >uxxBoEHHOOood22HV[V^V^VeVeUfhlUlLm2j 4D& 8EHHOO,c1 2  r