;iaOSSKrSSKrSSKrSSKrSSKJr SSKJr SSKJrJ r SSK J r SSK J r SSKrSSKrSSKJr SSKJr S S KJr S S KJr S S KJr S SKJr S SKJr S SK J!r!J"r"J#r#J$r$J%r% S SK&J'r' S SK(J)r) S SK*J+r+ \"\,5r-\$"5(a SSK.J/s J0r1 S\RdRfS\44Sjr5S\RdRfS\44Sjr6S\RdRfS\44Sjr7S\RdRfS\44Sjr8S\RdRfS\RdRf4Sjr9S\RdRf4Sjr:S\RdRfS\44Sjr;S6S\4S\4S \44S!jjr4S$jr?S7S%\4S&\44S'jjr@\""S(5(a \RO \R^rB\BRR\R\\R/rG\""S)5(a%\GR\RR5 S8S*jrKS+rLS,rMS8S-\ \NS\44S.jjrOS\N4S/jrPS0rQS1rRS2\S4S3jrTS9S\RdRfS4\4S\U\RdRf4S5jjrVg):N)encode) OrderedDict)partialreduce) MethodType)Optional)Version) save_file)write_basic_config) get_logger) PartialState)FSDP_PYTORCH_VERSION)DistributedType)is_deepspeed_availableis_numpy_availableis_torch_distributed_availableis_torch_xla_availableis_weights_only_available)id_tensor_storage) convert_model)is_torch_versionmodulereturnc[[S5(dg[U[RRR 5$)z< Check whether the module was compiled with torch.compile() _dynamoF)hasattrtorch isinstancer eval_frameOptimizedModulers f/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/accelerate/utils/other.pyis_compiled_moduler%6s1 5) $ $ femm66FF GGc[[S5(dgUR(aLUR5H8n[ U[R R R5(dM8 g g)zT Check whether the module has submodules that were compiled with `torch.compile()`. rFT)rr_modulesmodulesr rr!r"r submodules r$has_compiled_regionsr,@sQ 5) $ $ )I)U]]%=%=%M%MNN* r&c^[T[RR5=(a [ U4SjT55$)z Check whether the module is a repeated block, i.e. `torch.nn.ModuleList` with all children of the same class. This is useful to determine whether we should apply regional compilation to the module. c3V># UHn[UTSR5v M g7f)rN)r __class__).0mrs r$ %is_repeated_blocks..Us):ngmbc:aPQI\I\;];]gms&))r rnn ModuleListallr#s`r$is_repeated_blocksr7Os- fehh11 2 ns:ngm:n7nnr&cxUR(a)UR5Hn[U5(dM g g)z Check whether the module has repeated blocks, i.e. `torch.nn.ModuleList` with all children of the same class, at any level of the module hierarchy. This is useful to determine whether we should apply regional compilation to the module. TF)r(r)r7r*s r$has_repeated_blocksr9Xs/ )I!),,* r&c ^S[RRS[RR4U4SjjmT"U40UD6nSUR;aXRS'U$)a Performs regional compilation where we target repeated blocks of the same class and compile them sequentially to hit the compiler's cache. For example, in `GPT2LMHeadModel`, the repeated block/class is `GPT2Block`, and can be accessed as `model.transformer.h[0]`. The rest of the model (e.g. model.lm_head) is compiled separately. This allows us to speed up the compilation overhead / cold start of models like LLMs and Transformers in general. See https://pytorch.org/tutorials/recipes/regional_compilation.html for more details. Args: module (`torch.nn.Module`): The model to compile. **compile_kwargs: Additional keyword arguments to pass to `torch.compile()`. Returns: `torch.nn.Module`: A new instance of the model with some compiled regions. Example: ```python >>> from accelerate.utils import compile_regions >>> from transformers import AutoModelForCausalLM >>> model = AutoModelForCausalLM.from_pretrained("gpt2") >>> compiled_model = compile_regions(model, mode="reduce-overhead") >>> compiled_model.transformer.h[0] OptimizedModule( (_orig_mod): GPT2Block( (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D(nf=2304, nx=768) (c_proj): Conv1D(nf=768, nx=768) (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D(nf=3072, nx=768) (c_proj): Conv1D(nf=768, nx=3072) (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) ) ``` rrc >[U5(aO[RR5nUH)nUR [R "U40UD65 M+ U$[ U5(aURRUR5nURRUR5 0Ul UR5HupCURUT"U40UD65 M U$[R "U40UD6nU$N)r7rr4r5appendcompiler9r/__new____dict__updater(named_children add_module)rcompile_kwargs new_moduler+name_compile_regionss r$rG)compile_regions.._compile_regionss f % %,,.J# !!%-- "L^"LM$! ( ())11&2B2BCJ    & &v 7"$J #)#8#8#:%%d,r9childrencompile_regions_deepspeed)rrDr+childs r$rOrOsd&!!I    / / V $ $__&E %e >~ >' ((r&modelc^[SS5(aSSKJm OSSKJm [ U4SjUR 555$)z Check if the model has DTensor parameters. Args: model (`torch.nn.Module`): The model to check. Returns: `bool`: Whether the model has DTensor parameters. >=z2.5.0r)DTensorc3<># UHn[UT5v M g7fr<)r )r0prTs r$r2$model_has_dtensor..sB/A!z!W%%/As)rtorch.distributed.tensorrTtorch.distributed._tensorany parameters)rQrTs @r$model_has_dtensorr\s4g&&4 6 Bu/?/?/AB BBr&keep_fp32_wrapperkeep_torch_compile recursivec^ [RRR[RR4n[ U5n[ U5nU(aUnURnOU(aUnURSn[5(a SSK J n XH4- n[S[5(a[5(a SSKJn XI4- n[#X5(aUR$n[#X5(aMU(aU 4Sjm T "U5nU(dUR&n URR)SS5n U bF[+U S 5(a%U R,n X:XaO[+U S 5(aM%[/X5Ul[1US S 5(a [3US S 9 U(a*U(a UWlUnU$U(aUWRS'UnU$) a_ Extract a model from its distributed containers. Args: model (`torch.nn.Module`): The model to extract. keep_fp32_wrapper (`bool`, *optional*): Whether to remove mixed precision hooks from the model. keep_torch_compile (`bool`, *optional*): Whether to unwrap compiled model. recursive (`bool`, *optional*, defaults to `False`): Whether to recursively extract all cases of `module.module` from `model` as well as unwrap child sublayers recursively, not just the top-level distributed containers. Returns: `torch.nn.Module`: The extracted model. rIr)DeepSpeedEnginerS)FullyShardedDataParallelc>[US5(aT"UR5nOUnUR5Hup#[XT"U55 M U$)Nr)rrrBsetattr)runwrapped_modulerFrP_recursive_unwraps r$rf6extract_model_from_parallel.._recursive_unwrap sTvx((#4V]]#C #) />>@ (0A%0HI A# #r&_original_forwardN __wrapped__ _converted_to_transformer_engineF)to_transformer_engine)rr4parallelDistributedDataParallel DataParallelr%r,rIr@r deepspeedrarrr2torch.distributed.fsdp.fully_sharded_data_parallelrbr rforwardpoprrirgetattrr) rQr]r^r_options is_compiled has_compiledcompiled_modelraFSDPrqoriginal_forwardrfs @r$extract_model_from_parallelrzs(xx  88%((:O:OPG$U+K'.L {+-%%2338V8X8Xg7 U $ $  U $ $ $"%( -- >>--.A4H  ''=11!--.'=11'w6EM 5 Make sure all processes will reach this instruction otherwise one of your processes will hang forever. N)rwait_for_everyonerMr&r$r|r|/sN$$&r& state_dictc [R"[5nUR5H9up#[ U[ 5(aMU[ U5RU5 M; UR5VVs0sHupE[U5S:dMXE_M nnn[5nUR5H>nUVs/sH o"U;dM UPM nnURUSS5 USSHnX M M@ [U5S:a[RSUS35 UR5V V s0sH7upU [ U [R5(aU R!5OU _M9 nn n U$s snnfs snfs sn n f)z Cleans the state dictionary from a model and removes tensor aliasing if present. Args: state_dict (`dict`): The state dictionary from a model rNrzRemoved shared tensor zk while saving. This should be OK, but check by verifying that you don't receive any warning while reloading) collections defaultdictlistitemsr strrr=lensetvaluesrAloggerwarningrTensor contiguous) r}ptrsrFtensorptrnames shared_ptrs warn_names found_nameskvs r$ clean_state_dict_for_safetensorsr<sa  " "4 (D"((* &#&& "6* + 2 24 8+ 15 O *#E Q:3: KOJ##% ).D1Ct D+ab/*OD $& :$ZL1\ ] WaVfVfVhiVhda!z!U\\'B'BQ\\^IVhJi #PEjs;E7E7> E= E=4>Fsave_on_each_nodesafe_serializationc[5R[R:Xa[R "U5nU(a1[ [SS0S9n[U[5(a [U5nO[Rn[5R(aU(d U"X5 g[5R(aU(a U"X5 ggg)a Save the data to disk. Use in place of `torch.save()`. Args: obj: The data to save f: The file (or file-like object) to use to save the data save_on_each_node (`bool`, *optional*, defaults to `False`): Whether to only save on the global main process safe_serialization (`bool`, *optional*, defaults to `False`): Whether to save `obj` using `safetensors` or the traditional PyTorch way (that uses `pickle`). formatpt)metadataN)rdistributed_typerXLAxm_maybe_convert_to_cpursafe_save_filer rrrsaveis_main_processis_local_main_process)objfrr save_funcs r$rr_s$~&&/*=*==&&s+Nh5EF c; ' '237CJJ ~%%.?#  - -2C#3D -r&z2.0.0z1.25.0c p[5(aM[RR5nSU;aSUS'[RR [ 5 OUR SS5 [R"U4SU0UD6n[5(aD[RR5 W(a[RR U5 U$![5(aF[RR5 W(a [RR U5 fff=f)a Compatible drop-in replacement of `torch.load()` which allows for `weights_only` to be used if `torch` version is 2.4.0 or higher. Otherwise will ignore the kwarg. Will also add (and then remove) an exception for numpy arrays Args: f: The file (or file-like object) to use to load the data map_location: a function, `torch.device`, string or a dict specifying how to remap storage locations **kwargs: Additional keyword arguments to pass to `torch.load()`. weights_onlyTN map_location) rr serializationget_safe_globalsadd_safe_globalsTORCH_SAFE_GLOBALSrrloadclear_safe_globals)rrkwargsold_safe_globals loaded_objs r$rrs G $ & &$22CCE V+)-~&    0 01C D JJ~t ,ZZG GG $ & &    2 2 4##445EF  % & &    2 2 4##445EF  's BCAD5c[US5(d[US5(d [USU5n[US5(a UR$[US5(a UR$[ U5$)z Gets a pretty name from `obj`. __qualname____name__r/)rrsrrr)rs r$get_pretty_namersd 3 ' 'Z0H0Hc;,sN##sJ|| s8Or&cUR5H=up#[U[5(aURU05n[ X45 M9X1U'M? U$)z Recursively merges two dictionaries. Args: source (`dict`): The dictionary to merge into `destination`. destination (`dict`): The dictionary to merge `source` into. )rr dict setdefault merge_dicts)source destinationkeyvaluenodes r$rrsMlln  eT " "))#r2D  $$  % r&portcUcSn[R"[R[R5nURSU45S:HsSSS5 $!,(df  g=f)z Checks if a port is in use on `localhost`. Useful for checking if multiple `accelerate launch` commands have been run and need to see if the port is already in use. Ni> D [[F XX0$,,?NAKw'+..&wi/[\g[his s  sd3 /r&attrcDSn[X /URS5-5$)z Recursive `getattr`. Args: obj: A class instance holding the attribute. attr (`str`): The attribute that is to be retrieved, e.g. 'attribute1.attribute2'. c[X5$r<)rs)rrs r$_getattr#recursive_getattr.._getattrs s!!r&.)rr)rrrs r$recursive_getattrrs#" (EDJJsO3 44r& return_fqnscU(dUOSU4nU/n/nU(aUR5nU(aUupeUR5Hmupx[U[RR 5(dM0U(a%W(aUS-U-OUn UR X45 M\UR U5 Mo U(aUR WU45 OUR U5 U(aMUSSS2$)a)Traverse the model in bottom-up order and return the children modules in that order. Args: model (`torch.nn.Module`): the model to get the children of Returns: `list[torch.nn.Module]`: a list of children modules of `model` in bottom-up order. The last element is the `model` itself. rrN)rrrBr rr4rJr=) rQrtopstackordered_modulescurrent_modulecurrent_module_namerFr child_names r$get_module_children_bottom_uprs#%U C EEO  2@ / (779JD$00EX!4s!:T!A^bJLL*!34LL& :   " "$7#H I  " "> 2 % 4R4  r&)TTF)FFr<)F)Wrrrrcodecsrr functoolsrrtypesrtypingrnumpynprpackaging.versionr safetensors.torchr rcommands.config.defaultr loggingr stater constantsr dataclassesrimportsrrrrrmodelingrtransformer_enginerversionsrrrtorch_xla.core.xla_modelcore xla_modelrr4rJboolr%r,r7r9rKrOr\rzr|rrr_corenp_core multiarray _reconstructndarraydtyperr=dtypes UInt32DTyperrrintrrrrrrrrrMr&r$rsq #% %98 +((-& H ))HuxxH4H  T ouxxo4o  D EEHHOOE%((//EP)ehhoo)0CUXX__CC*_dR"R?CRW[Rj '  FDdH)11"((rww ##JJ HH hbii334@ $6#6$6 "s ""4$55"!!t!X\]b]e]e]l]lXm!r&