9i($ddlZddlZddlmZddlmZddlZddlZddl m Z ddl m Z m Z mZmZmZmZmZmZmZmZedr ddlmcmZdd lmZdd lmZeeZ dd e d ee!d e"de"de"de#de de$fdZ% ddZ&dde#de$fdZ'dde#fdZ(y)N)Path)List) GradScaler) MODEL_NAMEOPTIMIZER_NAMERNG_STATE_NAME SAMPLER_NAME SCALER_NAMESCHEDULER_NAMEget_pretty_nameis_tpu_availableis_xpu_availablesaveF) check_device) get_logger) PartialState output_dir model_states optimizers schedulers dataloaders process_indexscalersave_on_each_nodec^t|D]e\}} |dk(r tdn td|d} tjj || } t | | |t jd| gt|D]u\}} | j} |dk(r tdn td|d} tjj || }t | ||t jd|wt|D]u\}}|j} |dk(r tdn td|d}tjj ||}t | ||t jd|wt|D]\}}|dk(r tdn td|d}tjj ||}dd l m }m}t|j |r0|j"j"}t||rt |||t jd |d ||b|j} tjj |t$}t'j | |t jd |i}t(d|d }t+j,|d<t.j*j1|d<t'j2|d<t5r"t&j6j9|d<n!t&j:j9|d<t=rt?j2|d<tjj ||}t'j ||t jd||S)a Saves the current states of the models, optimizers, scaler, and RNG generators to a given directory. Args: output_dir (`str` or `os.PathLike`): The name of the folder to save all relevant weights and states. model_states (`List[torch.nn.Module]`): A list of model states optimizers (`List[torch.optim.Optimizer]`): A list of optimizer instances schedulers (`List[torch.optim.lr_scheduler._LRScheduler]`): A list of learning rate schedulers dataloaders (`List[torch.utils.data.DataLoader]`): A list of dataloader instances to save their sampler states process_index (`int`): The current process index in the Accelerator state scaler (`torch.cuda.amp.GradScaler`, *optional*): An optional gradient scaler instance to save save_on_each_node (`bool`, *optional*): Whether to save on every node, or only the main node. r.bin_rzModel weights saved in zOptimizer state saved in zScheduler state saved in rIterableDatasetShardSeedableRandomSamplerzSampler state for dataloader z saved in zGradient scaler state saved in .pkl random_statenumpy_random_seedtorch_manual_seedtorch_xpu_manual_seedtorch_cuda_manual_seedxm_seedzRandom states saved in ) enumeraterospathjoinrloggerinfo state_dictrr r data_loaderr!r" isinstancedatasetsamplerr torchr randomgetstatenp get_state get_rng_staterxpuget_rng_state_allcudarxm)rrrrrrrristate weights_nameoutput_model_fileoptoptimizer_nameoutput_optimizer_file schedulerscheduler_nameoutput_scheduler_file dataloader sampler_nameoutput_sampler_filer!r"r4output_scaler_filestates states_nameoutput_states_files b/var/www/html/backtest/airagagent/rag_env/lib/python3.12/site-packages/accelerate/checkpointing.pysave_accelerator_staterQ0sl@l+C5./1f*T*ZL!D:Q GGLL\B U%9JK -.?-@AB C J'I3 45FN+40>BRRSTUSVVZ@[ " Z H U)=NO /0E/FGH I"*-I 9$$&45FN+40>BRRSTUSVVZ@[ " Z H U)=NO /0E/FGH I#;/ X :01Q,t,|nAaSPT ? ((00G'#89W1EVW 3A3jAT@UVW X!!#WW\\*kB 5,- 56H5IJK F#$Am_D9K#__.F>"$))"5"5"7F "'"5"5"7F */))*E*E*G&'+0::+G+G+I'(,,.yj+> JJv)* KK)*<)=>? c `|dvr td|d}n|dk(rtj}t|D]j\} } | dk(r tdn td| d} t j j|| } || jtj| | fi|ltjd t|D]j\} } | dk(r tdn td| d}t j j||}tj|| }|| j|ltjd t|D]c\} }| dk(r tdn td| d}t j j||}|jtj|etjd t|D]\} }| dk(r tdn td| d}t j j||}d dlm}m}t'|j(|s_|j*j*}t'||stj||j*_tjd|]t j j|t,}|jtj|tjd tjt j j|t.d|d}t1j2|dt4j0j7|dtj8|dt;r#tj<j?|dn"tj@j?|dtCrtEj8|dtjdy#tF$rtjdYywxYw)a Loads states of the models, optimizers, scaler, and RNG generators from a given directory. Args: input_dir (`str` or `os.PathLike`): The name of the folder to load all relevant weights and states. models (`List[torch.nn.Module]`): A list of model instances optimizers (`List[torch.optim.Optimizer]`): A list of optimizer instances schedulers (`List[torch.optim.lr_scheduler._LRScheduler]`): A list of learning rate schedulers process_index (`int`): The current process index in the Accelerator state scaler (`torch.cuda.amp.GradScaler`, *optional*): An optional *GradScaler* instance to load map_location (`str`, *optional*): What device to load the optimizer state onto. Should be one of either "cpu" or "on_device". load_model_func_kwargs (`dict`, *optional*): Additional arguments that can be passed to the model's `load_state_dict` method. )Ncpu on_devicezaUnsupported optimizer map location passed, please choose one of `None`, `'cpu'`, or `'on_device'`NrTrUrrr map_locationz%All model weights loaded successfullyz(All optimizer states loaded successfullyz(All scheduler states loaded successfullyrr z1All dataloader sampler states loaded successfullyz$GradScaler state loaded successfullyr#r$r%r&r'r(r)z%All random states loaded successfullyzCould not load random states)$ TypeErrorrdevicer*rr+r,r-load_state_dictr5loadr.r/rr r r1r!r"r2r3r4r r r6setstater8 set_state set_rng_staterr;set_rng_state_allr=rr> Exception) input_dirmodelsrrrrrrWload_model_func_kwargsr?modelrAinput_model_filerCrDinput_optimizer_fileoptimizer_staterFrGinput_scheduler_filerIrJinput_sampler_filer!r"r4input_scaler_filerMs rPload_accelerator_staterks@55 o    $#~,, f%u5./1f*T*ZL!D:Q 77<< <@!q !!%**-=L"Yt]stu KK78J'7345FN+40>BRRSTUSVVZ@[!ww||I~F**%9 U1 %%o6 7  KK:;"*-D 945FN+40>BRRSTUSVVZ@[!ww||I~F!!%**-A"BCD KK:;";/ L :01Q,t,|nAaSPT ? ((00G'#89-2ZZ8J-K ""* L KKCDGGLLK@uzz*;<= :; 4BGGLL~6Fa VZ4[\]~./ F#678 F#678   II ' '/F(G H JJ ( (0H)I J     VI. / ;< 4 234s6DP P-,P-indexct|d|dz }tjdt|d|t |j ||y)zL Saves the state of `obj` to `{path}/custom_checkpoint_{index}.pkl` custom_checkpoint_r#zSaving the state of z to rN)rr.r/r rr0)objr,rlr save_locations rPsave_custom_staterqsP J#5eWD!AAM KK&s';&loggingrr@r__name__r.strdictlistintboolrQrkrqrtrRrPrs  %   ')) H #VVt*VV V  V  V VV@ _4DOODOGGrR