9i ddlZddlZddlZddlZddlmZddlmZddlm Z ddl Z ddl m Z ddl mZddlmZdd lmZd d lmZd d lmZd d lmZmZmZd dlmZd dlmZeeZ edr ddl!m"cm#Z$erddl%m&Z'dZ(d#de)fdZ*dZ+d$de)de)fdZ,edZ-edZ.dZ/dZ0d%de1de)fd Z2d!Z3d"Z4y)&N)contextmanager)partial) MethodType)Version)write_basic_config) get_logger) PartialState)FSDP_PYTORCH_VERSION)DistributedType)is_deepspeed_availableis_safetensors_availableis_tpu_available) convert_model)is_torch_versionF) check_device) save_filectddsttdsyt|tjj j S)zD Check whether the module was compiled with torch.compile() =)FullyShardedDataParallelforward_original_forwardN __wrapped__ _converted_to_transformer_engineF)to_transformer_engine)rnnparallelDistributedDataParallel DataParallelr _orig_modr deepspeedr#rr 2torch.distributed.fsdp.fully_sharded_data_parallelr$rrgetattr__dict__poprr'rr%r) modelr!options is_compiledcompiled_modelr#FSDPr%original_forwards rextract_model_from_parallelr:7s*xx  88%((:O:OPG$U+K-O%%23gD7 UG $  UG $ %+ >>--.A4H  ''=1!--..'=1'w6EM 5 Make sure all processes will reach this instruction otherwise one of your processes will hang forever. N)r wait_for_everyoner rr<r<ksN$$&r save_on_each_nodesafe_serializationcH|stjnttddi}t j t jk(rtj||yt jr |s |||yt jr |r |||yyy)a Save the data to disk. Use in place of `torch.save()`. Args: obj: The data to save f: The file (or file-like object) to use to save the data save_on_each_node (`bool`, *optional*, defaults to `False`): Whether to only save on the global main process safe_serialization (`bool`, *optional*, defaults to `False`): Whether to save `obj` using `safetensors` formatpt)metadataN) rsaversafe_save_filer distributed_typer TPUxmis_main_processis_local_main_process)objfr>r? save_funcs rrDrDxs{#5 '.\dfj[k:lI~&&/*=*== Q  ' '0A#q  - -2C#q3D -r c#pKtj}tt_d|t_yw)a A context manager that will cache origin `os.environ` and replace it with a empty dictionary in this context. When this context exits, the cached `os.environ` will be back. Example: ```python >>> import os >>> from accelerate.utils import clear_environment >>> os.environ["FOO"] = "bar" >>> with clear_environment(): ... print(os.environ) ... os.environ["FOO"] = "new_bar" ... print(os.environ["FOO"]) {} new_bar >>> print(os.environ["FOO"]) bar ``` N)osenvirondict)_old_os_environs rclear_environmentrSs$2jjOBJ  BJs46c+Ki}|jD]Y\}}|j}|tjvrtj|||<t |tj|<[d|D]M}|j}||vr||tj|<.tjj |dOyw)a A context manager that will add each keyword argument passed to `os.environ` and remove them when exiting. Will convert the values in `kwargs` to strings and upper-case all the keys. Example: ```python >>> import os >>> from accelerate.utils import patch_environment >>> with patch_environment(FOO="bar"): ... print(os.environ["FOO"]) # prints "bar" >>> print(os.environ["FOO"]) # raises KeyError ``` N)itemsupperrOrPstrr3)kwargs existing_varskeyvalues rpatch_environmentr\s$Mlln% Uiik "** !#CM# e* 3 %  &iik - +C0BJJsO JJNN3 % &sCCct|dst|ds t|d|}t|dr |jSt|dr |jSt |S)z( Gets a pretty name from `obj`. __qualname____name__ __class__)rr1r^r_rW)rKs rget_pretty_namerasX 3 'Z0Hc;,sN#sJ|| s8Or c|jD]9\}}t|tr|j|i}t ||5|||<;|S)z Recursively merges two dictionaries. Args: source (`dict`): The dictionary to merge into `destination`. destination (`dict`): The dictionary to merge `source` into. )rUrrQ setdefault merge_dicts)source destinationrZr[nodes rrdrdsUlln% U eT "))#r2D t $$K  % r portreturnc|d}tjtjtj5}|jd|fdk(cdddS#1swYyxYw)z Checks if a port is in use on `localhost`. Useful for checking if multiple `accelerate launch` commands have been run and need to see if the port is already in use. Ni> D [[F XX0$,,?NAwKw'+..&wi/[\g[his s  sd3 /r )T)FF)N)5rOrrrl contextlibr functoolsrtypesrrpackaging.versionrcommands.config.defaultrloggingr stater constantsr dataclassesr importsrrrtransformer_enginerversionsrr_rtorch_xla.core.xla_modelcore xla_modelrHsafetensors.torchrrErboolr:r<rDrSr\rardintrqr|rr=r rrs  % %8 +(WW-& H '))=H1$1h 'Dd.!!@ & &F $666"4r