r8?i6 ~ d dl Z d dlmZ d dlmZ d dlZd dlZd dlm Z ddl
m Z m Z m
Z
mZmZmZmZmZmZmZmZmZmZmZmZmZmZm Z m Z m Z edd rd d l m Z nd d l!m Z e r d dl"m#c m$Z% dd
l&m'Z' dd l(m)Z) e'e* Z+ dd e,d
e-e. de-de-de-de/de/dee de0de0fdZ1 d dZ2d de/de0fdZ3d de/fdZ4y) N)Path)Optional)
load_model )
MODEL_NAMEOPTIMIZER_NAMERNG_STATE_NAMESAFE_MODEL_NAMESAFE_WEIGHTS_NAME SAMPLER_NAME SCALER_NAMESCHEDULER_NAME WEIGHTS_NAMEget_pretty_nameis_cuda_availableis_hpu_availableis_mlu_availableis_musa_availableis_sdaa_availableis_torch_versionis_torch_xla_availableis_xpu_availableloadsavez>=z2.4.0)
GradScaler)
get_logger) PartialState
output_dir model_states
optimizers
schedulers dataloaders
process_indexstepscalersave_on_each_nodesafe_serializationc
t | } t | D ]f \ }
} | st nt } |
dkD r| j dd|
d } | j | }
t
| |
|| t j d|
h t | D ]g \ }
}|j } |
dk( r t dn t d|
d}| j | }t
| ||d t j d| i t | D ]g \ }
}|j } |
dk( r t dn t d|
d}| j | }t
| ||d t j d | i t | D ] \ }
}|
dk( r t dn t d|
d}| j | }d
d l
m}m} t! |j" | r+|j% }t! || rt
|||d t' |d d rD|
dk( rd
nd|
d}| j | }|j } t) j | | t j d|
d| |S|j } | j t* } t) j | | t j d| i } t, d| d} || d< t/ j0 | d< t2 j. j5 | d< t) j6 | d< t9 r!t( j: j= | d< t? r"t( j@ j= | d< nWtC r"t( jD j= | d< n+tG r!t( jH j= | d< tK r!t( jL j= | d< tO r!t( jP j= | d < tS rtU j6 | d < | j | } t) j | | t j d | | S ) aw
Saves the current states of the models, optimizers, scaler, and RNG generators to a given directory.
If `safe_serialization` is `True`, models will be saved with `safetensors` while the rest are saved using native
`pickle`.
Args:
output_dir (`str` or `os.PathLike`):
The name of the folder to save all relevant weights and states.
model_states (`List[torch.nn.Module]`):
A list of model states
optimizers (`List[torch.optim.Optimizer]`):
A list of optimizer instances
schedulers (`List[torch.optim.lr_scheduler._LRScheduler]`):
A list of learning rate schedulers
dataloaders (`List[torch.utils.data.DataLoader]`):
A list of dataloader instances to save their sampler states
process_index (`int`):
The current process index in the Accelerator state
step (`int`):
The current step in the internal step tracker
scaler (`torch.amp.GradScaler`, *optional*):
An optional gradient scaler instance to save;
save_on_each_node (`bool`, *optional*):
Whether to save on every node, or only the main node.
safe_serialization (`bool`, *optional*, defaults to `True`):
Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
r ._)r& r' zModel weights saved in .binFzOptimizer state saved in zScheduler state saved in r IterableDatasetShardSeedableRandomSampleruse_stateful_dataloaderdl_state_dict.bindl_state_dict_z Sampler state for dataloader z
saved in z Gradient scaler state saved in .pklr$ random_statenumpy_random_seedtorch_manual_seedtorch_xpu_manual_seedtorch_mlu_manual_seedtorch_sdaa_manual_seedtorch_musa_manual_seedtorch_hpu_manual_seedtorch_cuda_manual_seedxm_seedzRandom states saved in )+r enumerater r replacejoinpathr loggerinfo
state_dictr r r data_loaderr- r.
isinstancedataset get_samplergetattrtorchr
r randomgetstatenp get_state
get_rng_stater xpuget_rng_state_allr mlur sdaar musar hpur cudar xm)!r r r r! r" r# r$ r% r& r' istate weights_nameoutput_model_fileoptoptimizer_nameoutput_optimizer_file schedulerscheduler_nameoutput_scheduler_file
dataloader sampler_nameoutput_sampler_filer- r. samplerdataloader_state_dict_name!output_dataloader_state_dict_filerB output_scaler_filestates states_nameoutput_states_files! n/var/www/html/leadgen/airagagent/ocr_fallback/ocr_env/lib/python3.12/site-packages/accelerate/checkpointing.pysave_accelerator_staterk > sO X j!J l+ C5+=|CT q5'//q1X>L & / / = U%9J_qr -.?-@AB
C J' I3 45FN+40>BRRSTUSVVZ@[ * 3 3N C U)=Nchi /0E/FGH I "*- I 9$$&45FN+40>BRRSTUSVVZ@[ * 3 3N C U)=Nchi /0E/FGH I #;/ X
:01Q, t,|nAaSPT ? ,,.G'#89W 1EVkpq : 8% @@AQ)"$))"5"5"7F "'"5"5"7F */))*E*E*G&'*/))*E*E*G&' +0::+G+G+I'( +0::+G+G+I'(*/))*E*E*G&'+0::+G+G+I'( , , .y#,,[9 JJv)*
KK)*<)=>? c t }
|dvr t d |d}n|dk( rt j }|i }t | } t | D ] \ } } | dkD rd| nd}
| j
t |
d }|j rt | |fd t | i| U| j
t |
d
}t || } | j |fi | t j d t | D ]R \ } }| dk( r t d
n t d| d
}| j
| }t |fd
|i|}|| j | T t j d t | D ]M \ } }| dk( r t" d
n t" d| d
}| j
| }t |fi |}|j | O t j d t | D ] \ } }| dk( r t$ d
n t$ d| d
}| j
| }ddlm}m} t- |j. | r6|j1 } t- | | r|j3 t | } t5 |dd s| dk( rdnd| d
} | j
| } | j st | fi |}|j | t j d |F| j
t6 } t | }!|j |! t j d t | j
t8 d| d }"d|"v r|"d |
d< t; j< |"d t> j: jA |"d tC jD |"d tG r"tB jH jK |"d tM r#tB jN jK |"d n|tQ r#tB jR jK |"d nOtU r#tB jV jK |"d n"tB jX jK |"d! t[ rt] jD |"d" t j d# |
S # t^ $ r t j d$ Y |
S w xY w)%a
Loads states of the models, optimizers, scaler, and RNG generators from a given directory.
Args:
input_dir (`str` or `os.PathLike`):
The name of the folder to load all relevant weights and states.
models (`List[torch.nn.Module]`):
A list of model instances
optimizers (`List[torch.optim.Optimizer]`):
A list of optimizer instances
schedulers (`List[torch.optim.lr_scheduler._LRScheduler]`):
A list of learning rate schedulers
process_index (`int`):
The current process index in the Accelerator state
scaler (`torch.amp.GradScaler`, *optional*):
An optional *GradScaler* instance to load
map_location (`str`, *optional*):
What device to load the optimizer state onto. Should be one of either "cpu" or "on_device".
load_kwargs (`dict`, *optional*):
Additional arguments that can be passed to the `load` function.
load_model_func_kwargs (`dict`, *optional*):
Additional arguments that can be passed to the model's `load_state_dict` method.
Returns:
`dict`: Contains the `Accelerator` attributes to override while loading the state.
)Ncpu on_devicezaUnsupported optimizer map location passed, please choose one of `None`, `'cpu'`, or `'on_device'`rn ro r r* z .safetensorsdevicer+ ) map_locationz%All model weights loaded successfullyrr z(All optimizer states loaded successfullyz(All scheduler states loaded successfullyr r, r/ Fr0 r1 z1All dataloader sampler states loaded successfullyz$GradScaler state loaded successfullyr2 r$ r3 r4 r5 r6 r7 r8 r9 r; r< z%All random states loaded successfullyz Could not load random states)0dict TypeErrorr rq r r= r? r
existsr strr r load_state_dictr@ rA r r r rC r- r. rD rE rF set_samplerrG r
r rI setstaterK set_staterH
set_rng_stater rN set_rng_state_allr rP r rQ r rR rT r rU Exception)# input_dirmodelsr r! r" r# r% rr load_kwargsload_model_func_kwargsoverride_attributesrV modelendinginput_model_filerB rZ r[ input_optimizer_fileoptimizer_stater] r^ input_scheduler_filescheduler_stater` ra input_sampler_filer- r. rc rd input_dataloader_state_dict_fileinput_scaler_file scaler_staterg s# rj load_accelerator_stater s N &55 o
$#~,, YI f% H5 E1QCr$-- 0A& .VW " " $ u . cs<7H cLb c ) 1 1ZL2M N .\JJ !E ! !* G0F G H KK78 J' 7345FN+40>BRRSTUSVVZ@[ ( 1 1. A 3^,^R]^1
%%o6 7
KK:; "*- 3 945FN+40>BRRSTUSVVZ@[ ( 1 1. A 3C{C!!/2 3
KK:;";/ 7
:01Q, t,|nAaSPT ? ,,.G'#89$006H1IJ : 8% @@AQ)r s" ( 0 D'"$)))
H $( # #sst*s s s
s
s s
Z s s sz C LO OD OU Url