;i8u SSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSK r SSK J r J r SSK JrJr SSKrSSKJr SSKJr SSKJr SS KJrJr SS KJrJrJr SS KJrJrJ r J!r!J"r"J#r#J$r$J%r%J&r& SS K'J(r(J)r) SS K*J+r+J,r,J-r- SSK.J/r/J.r. SSK0J1r1 \""SS9(aSSK2r2\"SS9(aSSK3r3\$"SS9(aSSK4r4\!"SS9(aSSK5r5SSK6J7r7 SSK8J9r: Sr;\Rx"\=5r>Sr?Sr@S\\A\B44SjrCS\R4SjrES\RS\G\R\A\A44SjrISgS\RS\BS \\A\B\R4S!\\RS\\\B\R4S"\\RS#\\L\A\L\R\R44S$\MS%\M4S&jjrNShS\RS'\MS(\MS)\M4S*jjrOSiS\RS(\MS+\M4S,jjrPS-\R4S.jrQS/rRS0rSS-\R&RS\T\T\B4S1jrUS2rVS\\B\R4S\R4S3jrWSjS-\RS\\\B\R4S4\\L\B\\B\R44S5\M4S6jjrXSkS-\RS\\\B\R4S4\\L\B\\B\R444S7jjrYS8\T\G\B\R&R4S9\L\B\A4S:\T\B4S;jrZSlS<\\L\\A\B4\\A\B444S=jjr[SmS>\L\B\\A\B\R44S?\B4S@jjr\SAr]SBr^SnS-\RS<\\L\\A\B4\\A\B44S:\\T\BS\\\B\R4S4\\L\B\\B\R44SC\M4 SDjjr_S-\R&R4SEjr`SoS-\RS<\\L\\A\B4\\A\B44S:\\T\BS\\\B\R4S4\\L\B\\B\R44S\G\T\\A\B4\L\\A\B4\\A\B44\T\\A\B4\T\A\L\B\A4\T\T\B\T\B\T\G\B\R444 SFjjraS\G\A\T\B\T\R44SGjrbSkS8\T\G\B\R4S9\L\B\A4SH\\A\B4S:\\T\BSI\\T\T\BS\G\\B\\R\T\G\B\R444 SJjjrcSpS-\RS<\\L\\A\B4\\A\B44S:\\T\BS\\\B\R4S4\\L\B\\B\R44SK\MSL\MSM\MSN\M4SOjjrdS-\RS>\L\B\\A\B\R444SPjreSlSQjrfS-\R4SRjrgSqS\RS?\BSS\L\B\\B\R44ST\\A\B\R44SUjjriSrS-\RSV\\B\R4S>\\L\B\\A\B\R44SW\\\B\R4S\\\B\R4SX\MSM\MSY\\T\BSZ\MS[\MS\\MS]\M4S^jjrkSsS_\MS`\4SajjrlSlSb\4ScjjrmS\R&RS\M4Sdjrn\RSlS\R&RSe\\R4Sfjj5rpg)tN) OrderedDict defaultdict)OptionalUnion) distributed)nn)AcceleratorState)SAFE_WEIGHTS_NAME WEIGHTS_NAME)AutocastKwargs CustomDtypeDistributedType) is_hpu_availableis_mlu_availableis_mps_availableis_musa_availableis_npu_availableis_peft_availableis_sdaa_availableis_torch_xla_availableis_xpu_available)clear_device_cacheget_xpu_available_memory)load_offloaded_weightoffload_weightsave_offload_index)is_tqdm_availabletqdm)is_torch_versionF) check_device) safe_open) load_filezpytorch_model.bin.index.jsonc~SSKJn [5(aSSKJn [5=(a [ U"U5W5$)Nr )extract_model_from_parallelr) PeftModel)otherr&rpeftr' isinstance)modelr&r's i/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/accelerate/utils/modeling.py is_peft_modelr-Is-2"   \:.I%.PR[#\\c0URUR:wagURS:wa,URc[R"URSS9nURS:wa,URc[R"URSS9nX:H$)aR Utility method to check if two `torch` devices are similar. When dealing with CUDA devices, torch throws `False` for `torch.device("cuda") == torch.device("cuda:0")` whereas they should be the same Args: first_device (`torch.device`): First device to check second_device (`torch.device`): Second device to check Fcpurindex)typer2torchdevice) first_device second_devices r,check_device_samer8RsM...E!l&8&8&@||L$5$5Q? U"}':':'B ]%7%7qA  ((r.sizecDSnSUS3n[U[5(aUnGOUR5RS5(a[[ USS5S-5nGOUR5RS5(a[[ USS5S -5nGOSUR5RS 5(a[[ USS5S -5nGOUR5RS 5(a8[[ USS 5S-5nURS5(aUS-OUnOUR5RS5(a8[[ USS 5S-5nURS5(aUS-OUnO[UR5RS5(a7[[ USS 5S-5nURS5(aUS-OUnUS:a [ U5eU$![ a [ U5ef=f)a Converts a size expressed as a string with digits an unit (like `"5MB"`) to an integer (in bytes). Args: size (`int` or `str`): The size to convert. Will be directly returned if an `int`. Example: ```py >>> convert_file_size_to_int("1MiB") 1048576 ``` z`size` z] is not in a valid format. Use an integer for bytes, or a string with an unit (like '5.0GB').GIBNi@MIBiKIBiGBiʚ;bMBi@BKBir)r*intupperendswithfloat ValueError)r9mem_sizeerr_msgint_sizes r,convert_file_size_to_intrNmsH $tu " dC H ZZ\ " "5 ) )5cr+u56H ZZ\ " "5 ) )5cr+u56H ZZ\ " "5 ) )5cr+u56H ZZ\ " "4 ( (5cr+u56H(, c(:(:x1}H ZZ\ " "4 ( (5cr+u56H(, c(:(:x1}H ZZ\ " "4 ( (5cr+u56H(, c(:(:x1}H!|!! O "!!"s1H >H #>H #>H #AH ?AH AH HdtypecU[R:XagU[R:XagU[R:XagU[R :Xag[ SS5(a%U[R[R4;ag[R"S[U55nUc[SUS 35e[UR5S 5nUS -$) z Returns the size (in bytes) occupied by one parameter of type `dtype`. Example: ```py >>> dtype_byte_size(torch.float32) 4 ``` g?g?g?r >=z2.1.0z [^\d](\d+)$z`dtype` is not a valid dtype: .rrC)r4boolrINT2INT4FP8r! float8_e4m3fn float8_e5m2researchstrrJrFgroups)rO bit_searchbit_sizes r,dtype_byte_sizer_s   +"" " +"" " +// ! $ ( (Uu7J7JEL]L]6^-^>3u:6J9%BCC:$$&q)*H q=r.tensorreturnc[RS[RS[RS[RS[R S[R S[RS[RS[RS[RS0 nUR5R5nUR5R5nUR(X#4$![a UR5R5nUR5R!5XR"-nNf![$a% SnUR'5XR"-nNf=ff=f)a Unique identifier to a tensor storage. Multiple different tensors can share the same underlying storage. For example, "meta" tensors all share the same storage, and thus their identifier will all be equal. This identifier is guaranteed to be unique and constant for this tensor's storage during its lifetime. Two tensor storages with non-overlapping lifetimes may have the same id. rCr r r)r4int64float32int32bfloat16float16int16uint8int8rSfloat64untyped_storagedata_ptrnbytes Exceptionstorager9rONotImplementedErrornelementr5)r`_SIZE storage_ptr storage_sizes r,id_tensor_storagerws'  Q q Q  q Q Q A A q E C,,.779 --/668  ==+ 33  C C ..*335K!>>+002U<<5HHL" CK!??,u\\/BBL  C Cs+$RS5UlUR@RS5Ul OU"U4SU R00UD6RX'S 9nOURS;a6[RBREUU R0S9RX'S 9nO>URS;aURX'S 9nOU"UU R0S9RX'S 9nUURU'Ub"URX'S 9URUl AURFRS :XGa[URHS!S5c[URHR 5S:wa[R "U5RS":Xa [R "U5RJOSn[URHS!S5(ddUbaURLb6URLR RS:waUROU5nOURLcUROU5nOURFRS#:Xa[URHS$S5c[URHR 5S:wa[R "U5RS":Xa [R "U5RJOSn[URHS$S5(d#Ub URHROU5Ul$SSS5 U(aUS%;a [Q5 Ub@U R 5U;a,X&U R 5;aWXmR 5U'gUbFUbBUR 5U;a-X&UR 5;aWXcR 5U'ggggg!,(df  N=f)&a A helper function to set a given tensor (parameter of buffer) of a module on a specific device (note that doing `param.to(device)` creates a new tensor not linked to the parameter, which is why we need this function). Args: module (`torch.nn.Module`): The module in which the tensor we want to move lives. tensor_name (`str`): The full name of the parameter/buffer. device (`int`, `str` or `torch.device`): The device on which to set the tensor. value (`torch.Tensor`, *optional*): The value of the tensor (useful when going from the meta device to any other device). dtype (`torch.dtype`, *optional*): If passed along the value of the parameter will be cast to this `dtype`. Otherwise, `value` will be cast to the dtype of the existing parameter in the model. fp16_statistics (`torch.HalfTensor`, *optional*): The list of fp16 statistics to set on the module, used for 8 bit model serialization. tied_params_map (Dict[int, Dict[torch.device, torch.Tensor]], *optional*, defaults to `None`): A map of current data pointers to dictionaries of devices to already dispatched tied weights. For a given execution device, this parameter is useful to reuse the first available pointer of a shared weight on the device for all others, instead of duplicating memory. non_blocking (`bool`, *optional*, defaults to `False`): If `True`, the device transfer will be asynchronous with respect to the host, if possible. clear_cache (`bool`, *optional*, defaults to `True`): Whether or not to clear the device cache after setting the tensor on the device. rRNr;z has no attribute z- does not have a parameter or a buffer named metaz7 is on the meta device, we need a `value` to put in on Params4bitz Trying to set a tensor of shape z in "z" (which has shape z), this looks incorrect.)r}z torch.uintz torch.intz torch.bool)cudaxpu) Int8Params FP4Paramsrr0npu:zmlu:zsdaa:zmusa:hpurz6 is not available, you should use device="cpu" instead) requires_gradr5rrr)QTensor QBitsTensor)AffineQuantizedTensor Linear8bitLtSCBr Linear4bit quant_state)r0r))splitgetattrrJ _parameters_buffersrnr5r4r3shape__name__torOr[ startswithno_gradr*rFrrrrrrrTensorr`r8__dict__rerhCBrr Parameter __class__weightr2biasrr)rxryr5rzrOr{r|r}r~splitsr new_module is_buffer old_valueparam param_clsdevice_quantization new_valuekwargs device_indexs r,set_module_tensor_to_devicersLN k""3'CR[E /J! F8+=eWA!FGGF ! Rj ,,,OO1SF8#PQ\P]]^_``.I,I   ' NN  / enn&67 7*9..:J*KF*S;'#    O 3 i&8&8&:; ;*9:L:L:N*OPV*W;'5<<//F65<P>P/PF  { +VZEU I  ??ekk )i.@.@L.P25;;-u[MQdenetetduvNO  =HHY__E    !!8 V$))_<""&OO"( F fc " "!!x!##x"$$ )"$$ )!## CK (8(:(:x']^_ _ =! V GI V V8L/M%M9??+667`aa ) U NI 6? YbYpYp6qF&&{3 u|| , ,CI U:I  *(F +4FOOK (  &7 V8LfN`N`alNmNtNt&u&uV// <=I'' 4==F!!%NN%%5)//U]]:Z ) U]] VIU?y'9'9\'I )) e9CZCZ e^d e h hij k n not uI#,<C\\&=Q=V=VZ`=`u||F399fj v}}eT::|?W{{.6;;3E3E3J3Jf3T!'\!:,!'\!:  ))\9FMM=$?G ,,-7>C\\&=Q=V=VZ`=`u||F399fj v}}mTBB|G_$*MM$6$6|$DFM Dv_4 #    O 3 )*<*<*>? ?8A**,-f5   ' NN  / %..*:; ;4=()&1 < 0 ( ] s 7Xd77 einclude_buffersrecurseremove_non_persistentc## URUS9ShvN U(a@[5nU(a [XS9nURUS9HnUupgXd;dMUv M ggNL7f)a  A helper function that gathers all the tensors (parameters + buffers) of a given module. If `include_buffers=True` it's the same as doing `module.named_parameters(recurse=recurse) + module.named_buffers(recurse=recurse)`. Args: module (`torch.nn.Module`): The module we want the tensors on. include_buffer (`bool`, *optional*, defaults to `True`): Whether or not to include the buffers in the result. recurse (`bool`, *optional`, defaults to `False`): Whether or not to go look in every submodule or just return the direct parameters and buffers. remove_non_persistent (`bool`, *optional*, defaults to `False`): Whether or not to remove the non persistent buffer from the buffers. Useful only when include_buffers = True rN)named_parameterssetget_non_persistent_buffers named_buffers)rxrrrnon_persistent_buffers named_buffername_s r,named_module_tensorsrsm$&&w&777!$ %?%X ""000AL"GD1""B 8sA$A">A$ A$fqnscURnU(aTUR5H@upEU(a&X5RVs1sH odS-U-iM sn-nM2X5R-nMB U$s snf)a Gather all non persistent buffers of a given modules into a set Args: module (`nn.Module`): The module we want the non persistent buffers on. recurse (`bool`, *optional*, defaults to `False`): Whether or not to go look in every submodule or just return the direct non persistent buffers. fqns (`bool`, *optional*, defaults to `False`): Whether or not to return the fully-qualified names of the non persistent buffers. rR)_non_persistent_buffers_set named_modules)rxrrnon_persistent_buffers_setnmrBs r,rrsn"(!C!C((*DA*DaDa.bDaq3w{Da.bb**.K.KK* + &% /csA*r+cSnSnSnS[R"UR5Vs/sHoDRPM sn;aSnSn[ US5(a@[ UR S5(aUR R SS9O UR nUSL=(a$ [USS5=(a UR5n[ US5=(a5 [UR S S5=(a [UR S S5n[S UR555n[XU/5$s snf) z Check if there is any indication in the given model that some weights should be tied. Args: model (`torch.nn.Module`): The model to inspect Returns: bool: True if the model needs to have tied weights FPreTrainedModelNconfigget_text_configT)decodertie_word_embeddingsis_encoder_decodertie_encoder_decoderc3:# UHn[US5v M g7f) _tie_weightsN)hasattr).0rxs r, 2check_tied_parameters_in_config..s\O&gfn==Os) inspectgetmrorrrrrrget_output_embeddingsanymodules)r+has_tied_word_embeddinghas_tied_encoder_decoderhas_tied_modulecmodel_decoder_configs r,check_tied_parameters_in_configrs.$$O1PQ1PAZZ1PQQ"'# 5( # #5<<):;; ,,T,:\\ ! ! , .,.CUK .++-  E8 $ D &:EB D &;UC ! \EMMO\\ '?S TT+RsD>cX;aX$SRURS5SS5nX :Xa[SUS35e[X!5$)NrRr;z-The `device_map` does not contain the module )joinrrJ_get_param_device)r device_map parent_params r,rr sU   88EKK,Sb12LHqQRR ::r.cUH\n0nUHn[XA5X4'M [[UR555S:dMC[R SUS35 M^ g)a Check if tied parameters are on the same device Args: tied_params (`List[List[str]]`): A list of lists of parameter names being all tied together. device_map (`Dict[str, Union[int, str, torch.device]]`): A map that specifies where each submodule should go. r z*Tied parameters are on different devices: zC. Please modify your custom device map or set `device_map='auto'`. N)rlenrvaluesloggerwarning) tied_paramsr tie_paramtie_param_devicesrs r,$check_tied_parameters_on_same_devicersj! E'8'K  $ s$++-. /! 3 NN<=N The signature accepts keyword arguments, but they are for the recursive part of this function and you should ignore them. Args: model (`torch.nn.Module`): The model to inspect. Returns: List[List[str]]: A list of lists of parameter names being all tied together. Example: ```py >>> from collections import OrderedDict >>> import torch.nn as nn >>> model = nn.Sequential(OrderedDict([("linear1", nn.Linear(4, 4)), ("linear2", nn.Linear(4, 4))])) >>> model.linear2.weight = model.linear1.weight >>> find_tied_parameters(model) [['linear1.weight', 'linear2.weight']] ``` F)remove_duplicateT)rrkeysitemsappendsortedlist) r+rrrall_named_parametersno_duplicate_named_parameterstied_param_namestied_param_groupstied_param_name tied_param param_namertieds r,find_tied_parametersr*s+>[>`>`>b:cc+): !>!D!D!F J"646%1!-44_E "G,DUCZCZC\ ]C\<6FF8d3t9o- .C\ ]]+k%s" ^sDD-DcUHnSnUHhnUnURS5nUSSHn[XW5nM [XVS5nUbM@UR[R"S5:wdMfUn O UcMxUH<nUnURS5nUSSHn[XW5nM [ XVSU5 M> M g)a< Reties tied parameters in a given model if the link was broken (for instance when adding hooks). Args: model (`torch.nn.Module`): The model in which to retie parameters. tied_params (`List[List[str]]`): A mapping parameter name to tied parameter name as obtained by `find_tied_parameters`. NrRr;r)rrr5r4setattr) r+r tied_group param_to_tierrxrrrs r,retie_parametersras"  $JF%%c*F /%F2J/E#  V8L(L$ %  #( #))#.#CR[E$V3F)r L9 )"r.ct[U[5(a"URSS5n[[U5nU$)z, Just does torch.dtype(dtype) if necessary. torch.)r*r[replacerr4rOs r,_get_proper_dtypers2% h+u% Lr.special_dtypes buffers_onlycUb[U5n[U5nUb\UR5VVs0sHupVU[U5_M nnnUR5VVs0sHupVU[U5_M nnn[[5n/n U(d [ USS9n OUR SS9n U GHupUbX;aU R5WU -n OUc'U R5[U R5-n O[U R5RS5(a'U R5[U R5-n O0U R5[W[U R55-n U RS5n [[U 5S-5H"nUSRU SU5==U - ss'M$ GM U$s snnfs snnf)z6 Compute the size of each submodule of a given model. NTrrrRr )rr_rrrFrrnumelrOr[rminrrangerr)r+rOrr dtype_sizekeydtypspecial_dtypes_size module_sizes module_listrr`r9 name_partsidxs r,compute_module_sizesrs !%($U+ !HVH\H\H^_H^93#066H^_KYK_K_KabKaicsOD$99Kabs#LK *5$? ))$)7 #   %$*@<<>$7$==D ]<<>OFLL$AAD   ) )*S T T<<>OFLL$AAD<<>C OFLL4Q$RRDZZ_ Z1,-C *Tc"23 4 < 4.$ 5`bs GGc<[XUSS9nURSS5$)zG Compute the total size of buffers in each submodule of a given model. T)rOrrrr)rget)r+rOrrs r, compute_module_total_buffer_sizers&(>hlmL   B ""r.rrno_split_module_classesc Sn/nUR5n[U5S:aURS5upg[U[R R 5(a[UR55O/n[U5S:XdURRU;a&Xn X:aU nU/nO8X:XaURU5 O!UV V s/sHupUSU 3U 4PM sn n U-n[U5S:aMX44$s sn n f)a Utility function that will scan a list of named modules and return the maximum size used by one full layer. The definition of a layer being: - a module with no direct children (just parameters and buffers) - a module whose class name is in the list `no_split_module_classes` Args: modules (`List[Tuple[str, torch.nn.Module]]`): The list of named modules where we want to determine the maximum layer size. module_sizes (`Dict[str, int]`): A dictionary mapping each layer name to its size (as generated by `compute_module_sizes`). no_split_module_classes (`List[str]`): A list of class names for layers we don't want to be split. Returns: `Tuple[int, List[str]]`: The maximum size of a layer with the list of layer names realizing that maximum size. rrR) copyrpopr*r4rModulernamed_childrenrrr) rrrmax_size layer_namesmodules_to_treat module_namerxmodules_childrenr9rvs r,get_max_layer_sizer"s(HK||~  ! #.2215 n[R "S/US9n[R.RU5SX'M@ [15(aUR35R4US 'U$UR35R4US 'U$UH+n[7X[85(dM[;X5X'M- UR=5Vs/sHn[7U[>5(dMUPM nnURA5 [5(a [RR 5nGO[5(a[RR 5nO[5(a[RR 5nO[!5(a[R"R 5nOz[%5(a[R&R 5nOL[+5(a[R,R 5nO[R.R 5nUH=nX:dUS:dM[RCSUS[E[U5535 M? USVs/sHoUUR=5;dMUPM sn-n UR=5HnXY;dM [GSUS35e U Vs0sHoUX_M nnU$![a [RSUS35 GM<f=f![a [RSUS35 GMf=f![a [RSUS35 GMvf=f![a [RSUS35 GMf=f![a [RSUS35 GMf=f![a [RSUS35 GMef=f![a [RSUS35 GMf=fs snfs snfs snf)zZ Get the maximum memory available if nothing is passed, converts string to int otherwise. rNnpurzDevice z; seems unavailable, Proceeding to check subsequent devices.mlusdaamusarrmpsr0z) is not available, available devices are )r)r0diskzX is not recognized, available devices are integers(for GPU/XPU), 'mps', 'cpu' and 'disk')$psutilrr r4r% device_countr`r5 mem_get_inforprinforr&rr'rr(rrrrrrrvirtual_memory availabler*r[rNrrFsortrrrJ) r#r+irr k gpu_devices num_devicesr5 all_devicess r,get_max_memoryr7s   5991134 Qu||E1/EFA$)II$:$:1$=a$@JM5  5991134 Qu||E1/EFA$)II$:$:1$=a$@JM5 5::2245 Qu||FA/FGA$)JJ$;$;A$>q$AJM6 5::2245 Qu||FA/FGA$)JJ$;$;A$>q$AJM6  5991134 Qu||E1/EFA$q$AJM6    & 5 5 7 A AJu !' 5 5 7 A AJu  jos + +6zGJO )oo/F/:a3E1/KFii,,.   ii,,.   jj--/   jj--/   ii,,.   ii,,. jj--/   FQJ NNWVH,UVZ[`al[mVnUop q,B ],Bq:??K\F\,B ]]K __  !tu  -88KqZ]"KJ8 }!KK'!,g hi!KK'!,g hi!KK'!,g hi!KK'!,g hi!KK'!,g hi!KK'!,g hi!KK'!,g hi G*!^9sAYAZ!AZ,/A[=7\3A\-1:]?^^/^ ^^ #Y>=Y>#Z)(Z),#[[#[?>[?#\*)\*-#]]#^?^rrc^US:XaSOUS3nUR5VVs/sHup4URU5(dMUPM! nnn[[U55S:XaI[U5S:a:UVs/sHo3RU5(dMUPM snHnX M USX'UR 5Vs/sH6o3RU5(dM[U5[U5:dM4UPM8 nn[U5S:a[UR S55S-OSm[U4SjU55nUH n[ XS9 M U$s snnfs snfs snf)zU Cleans a device_map by grouping all submodules that go on the same device together. rrRr rc3j># UH(nSRURS5ST5v M* g7frRN)rr)rr3rs r,r#clean_device_map..fs.RAQA388AGGCL#$677AQs03)r)rrrrrrclean_device_map) rrprefixr3r!rchildren_moduleschildrs @r,r<r<Ws8 "$R[M*;F&,,. G.DA!,,v2Fa.F G 3v;1Vq'@Z<<+?!Z@A A"() $.??#4k#4a V8LQTUVQWZ]^iZjQj#4k-0-=-A#k$ % )qCRAQRR!7" H@ ls)EE6EE>EE5Ec Ub[U5S:XagUR5Hup4SU;aM SnSU;adURSS5UR5;a@URSS5n[ [ R RX&S35X5n[ R RX#S35n[ Xt5n[XSXS9 M g)az Loads the weights from the offload folder into the model. Args: model (`torch.nn.Module`): The model to load the weights into. index (`dict`): A dictionary containing the parameter name and its metadata for each parameter that was offloaded from the model. offload_folder (`str`): The folder where the offloaded weights are stored. Nrrrz.datr0)rzr{) rrrrrospathrr) r+r2offload_folderrmetadatar{ weight_name tensor_filers r,load_offloaded_weightsrGms }E a %  J   z !j&8&85&IUZZ\&Y$,,Xu=K3 ^}D-ABEDVOggll>\3FG &{=#EuFl!.r.c0nUH=nUS:XdSU;aMURSS5SnURUS5S-X'M? UVs/sH$o!RUS5S:XdMUS:wdM"UPM& nnU$s snf)NrrRr r)rsplitr)rmodule_childrenrxparentleavess r,get_module_leavesrMsO R<3f, sA&q)"1"5"5fa"@1"D  $0 h<3F3Fvq3QUV3Vf[aeg[gf All computation is done analyzing sizes and dtypes of the model parameters. As a result, the model can be on the meta device (as it would if initialized within the `init_empty_weights` context manager). Args: model (`torch.nn.Module`): The model to analyze. max_memory (`Dict`, *optional*): A dictionary device identifier to maximum memory. Will default to the maximum memory available if unset. Example: `max_memory={0: "1GB"}`. no_split_module_classes (`List[str]`, *optional*): A list of layer class names that should never be split across device (for instance any layer that has a residual connection). dtype (`str` or `torch.dtype`, *optional*): If provided, the weights will be converted to that type when loaded. special_dtypes (`Dict[str, Union[str, torch.device]]`, *optional*): If provided, special dtypes to consider for some specific weights (will override dtype used as default for all weights). low_zero (`bool`, *optional*): Minimizes the number of weights on GPU 0, which is convenient when it's used for other operations (like the Transformers generate function). Nr%r&r'r(rrr)rrr Fg?z(We will use 90% of the memory on device z for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk).rOrrrRg?c3h# UH(up[U[5(dMUS:dM$Uv M* g7f)rN)r*rF)r device_id device_mems r,r&get_balanced_memory.. s/ 3E/)T]_bIcIhruvhvII3Es 22 2r;)!r7rrrrrrrrr4r5r3rr*rFrr.rrtuplerrrrrrmaxrrMsumrrr )r+r#rrOrrNuser_not_set_max_memoryexpected_device_typedr5r rper_gpuno_split_childrenrr9 submodulesubmodule_name class_namebufferrL leaves_setrr! mean_leaves gpus_idx_listrr2min_zeros r,get_balanced_memoryresJ)D0 +J$   $   %   %   $   $   $%*s*Q Q0D0DH\0\qakanqrarq*stKaa "!(c3''Os*OKKB3%Hoo)(>ZL2h;?KPG&"$ /$ ? ?#:"; "#a'&,,.JDrzI"&**S/#I~> #2",,55J44K\9\04*-$))+,4K0LL/588I4JQ4N&--/0TU| ,FVJ%1%7%7%9Q%9TQQj=PDAqD%9LQ | ,FcF;Fq<?F;ZL*51O&u--#o2F!2K A U # #E # 23 u##% & ' u""5"1 2 3    1 Ks $ D;1D;c |[U5S:aU//4$/n/nUHrn[U5VVV s/sH%unupURUS-5(dM#UPM' sn nnSn URX:S5 URX:S5 Mt Un [ X5HuplXU X&- - n M XU4$s sn nnf)a Calculate the total size of a module, including its tied parameters. Args: tied_params (`List[str]`): The list of tied parameters. module_size (`int`): The size of the module without tied parameters. module_sizes (`Dict[str, int]`): A dictionary mapping each layer name to its size. modules_to_treat (`List[Tuple[str, nn.Module]]`): The list of named modules to treat. Returns: `Tuple[int, List[str], List[nn.Module]]`: The total size of the module, the names of the tied modules, and the tied modules. r rRr)r enumeraterrzip) r module_sizerrtied_module_names tied_modulesrr2rrtied_module_indexmodule_size_with_tiestied_module_names r,get_module_size_with_tiesr}ns& ;!B""L! 09:J0Kn0K91fqzOdOdefilelOmQ0Knopq  !1!DQ!GH,?BC" ((+K(K$ .>!?,BZ!ZZ)L !\ AAos "B7 B7 size_limitrrc ^[U5nUc/nUc/nUR5nSnU(GaURS5umnUVs/sH=n[ U4SjU55(dM[ U4SjU55(aM;UPM? n n[ U VV s/sH#oV s/sHn TS-U S-;dMU PM sn PM% sn n/5n [XTX5un n X::aSnO[U[R5(d[U[R5(a/O[UR55n[U5S:XdUR R"U;aGMC[UR%SS95U-nUVVs/sHunnTSU3U4PM snnU-nU(aGMU(dSSU4$UVV s/sHupUPM nnn ['T5VVs/sHunnUS:XdMUPM nnnUHnTSUnUU;dMUR)U5nUUun n[UR%SS95[UR55-nUSUUVVs/sHunnUSU3U4PM snn-UUS -S-nUVV s/sHupUPM nnn M UR)T5nURU5umnTXp4$![a SSU4s$f=fs snfs sn fs sn nfs snnfs sn nfs snnfs snnfs sn nf) a Find a module that fits in the size limit using BFS and return it with its name and the remaining modules. Args: modules (`List[Tuple[str, nn.Module]]`): The list of named modules to search in. module_sizes (`Dict[str, int]`): A dictionary mapping each layer name to its size (as generated by `compute_module_sizes`). size_limit (`Union[int, str]`): The maximum size a module can have. no_split_module_classes (`Optional[List[str]]`, *optional*): A list of class names for layers we don't want to be split. tied_parameters (`Optional[List[List[str]]`, *optional*): A list of lists of parameter names being all tied together. Returns: `Tuple[Optional[str], Optional[nn.Module], List[Tuple[str, nn.Module]]]`: A tuple containing: - The name of the module that fits within the size limit. - The module itself. - The list of remaining modules after the found module is removed. NFrc3:># UHnTS-US-;v M g7fr:rr3rs r,r$fallback_allocate..=*Q4#:S(*c3:># UHnTS-US-;v M g7fr:rrs r,rr%Itisde$QT*XY\_X_J_isrrRTrr )rNrJrrrallrWr}r*rrr4rrrrrrrrur2)rrr~rrrmodules_to_search module_foundrxrrprr{rr rr! current_namesr2rdot_idx dot_index parent_nameparent_module_idx parent_modulerJ target_idxrs @r,fallback_allocatersg8#-j9 &"$ L (,,Q/ f. - =*== FIItisItFt -  Te fTej AAtczS'@a ATe fhj '@ d+\' #q! ! .L  &",,//:fell3S3S f++-.   A %)9)9)B)BF])]   7 7 7 FGJZZ [U 5S:a!U3(d[>R@"S U2S!35 U(a?S"RCS#UR;555n7[DRGS$U7S%35 U$s snfs snfs snnfs snfs snfs snnfs snfs sn*nfs sn,nn+fs snnfs sn*nfs snnfs sn1nf)&a Compute a device map for a given model giving priority to GPUs, then offload on CPU and finally offload to disk, such that: - we don't exceed the memory available of any of the GPU. - if offload to the CPU is needed, there is always room left on GPU 0 to put back the layer offloaded on CPU that has the largest size. - if offload to the CPU is needed,we don't exceed the RAM available on the CPU. - if offload to the disk is needed, there is always room left on the CPU to put back the layer offloaded on disk that has the largest size. All computation is done analyzing sizes and dtypes of the model parameters. As a result, the model can be on the meta device (as it would if initialized within the `init_empty_weights` context manager). Args: model (`torch.nn.Module`): The model to analyze. max_memory (`Dict`, *optional*): A dictionary device identifier to maximum memory. Will default to the maximum memory available if unset. Example: `max_memory={0: "1GB"}`. no_split_module_classes (`List[str]`, *optional*): A list of layer class names that should never be split across device (for instance any layer that has a residual connection). dtype (`str` or `torch.dtype`, *optional*): If provided, the weights will be converted to that type when loaded. special_dtypes (`Dict[str, Union[str, torch.device]]`, *optional*): If provided, special dtypes to consider for some specific weights (will override dtype used as default for all weights). verbose (`bool`, *optional*, defaults to `False`): Whether or not to provide debugging statements as the function builds the device_map. clean_result (`bool`, *optional*, defaults to `True`): Clean the resulting device_map by grouping all submodules that go on the same device together. offload_buffers (`bool`, *optional*, defaults to `False`): In the layers that are offloaded on the CPU or the hard drive, whether or not to offload the buffers as well as the parameters. fallback_allocation (`bool`, *optional*, defaults to `False`): When regular allocation fails, try to allocate a module that fits in the size limit using BFS. rz Treating module rRc3:># UHnTS-US-;v M g7fr:rrs r,r(infer_auto_device_map..`rrc3:># UHnTS-US-;v M g7fr:rrs r,rr`rrz' Found the relevant tied param groups z4 So those parameters need to be taken into account r*NzPutting z and z (size=)z (available=z on c3B># UHunup#UT:XdMUv M g7fNr)rr2rrr|s r,rrs$,u@[91fq_`dt_tQQ@[s rPzNot enough space on z to put z (space available z, needed size z).Fz Splitting rr Tz?None of the tied module can be split, going to the next device.z, module size z6This module cannot be split, going to the next device.r0zCurrent model requires z bytes of buffer for offloaded layers, which seems does not fit any GPU's remaining memory. If you are experiencing a OOM later, please consider using offload_buffers=True. c38# UHupSUSUS3v M g7f)z - : z bytes requiredNr)rr5mems r,rr%s%! Emkfd6("SE 1Emsz{Based on the current allocation process, no modules could be assigned to the following devices due to insufficient memory: z These minimum requirements are specific to this allocation attempt and may vary. Consider increasing the available memory for these devices to at least the specified minimum, or adjusting the model config.)$rsrr"rrprintrr*r4rrrrrWr}nextrurrrvrrrrrrrrVrrr<warningswarnrrr.):r+r#rrOrrrrrrorqrprrrrrcurrent_devicer5device_memory_useddevice_buffer_sizes device_minimum_assignment_memorymax_layer_sizemax_layer_namesrxrrrwrrrrcurrent_max_sizecurrent_memory_reservedr{rxryoutputrzcurrent_buffer_sizesplit_happened tied_moduletied_module_childrenr!r2rr fallback_module_namefallback_moduleremaining_modulesrnon_gpu_buffer_sizeis_buffer_fit_any_gpu gpu_devicegpu_max_memorygpu_memory_used devices_inforr|s: @@r,infer_auto_device_maprs ~ $E7NWef  JN29:'&!)':')$'99IYp&q#NO  ! #'++A. f  &tfA. /&5dod11< "l 2/.@ &4 #AZ lB >0,  #'9&'ADY'Y]m'm#D6*$&7%899F }A66F#/ -=@RSY@Z-Z,[[\]]FD**f v &*? ? & &Jt %6 #6F'G6F!6F'GG(,,u JZ@[,u(u%$(():;06 +,%6#z&"))'D'D&F%'#/B.E.Efa.PSf.f#F+  { a $6v$>$LP`$`*7>+B*C8D6QVWhVij!!14Fv4N!N O~^s]ttvx#N145F 1U- +'+K,F,F,H'I$+,1[5J5J5S5SWn5nJ'7&8:;'+K,H,HQV,H,W'X[o'o$Sg'hSg41a,<+=Qqc)BA(FSg$'h8ABR8S$m8S91fq!WX\lWlQ8S$mno$p!F^$&'9(9:;*+''81'<'>?@!3E(8[(81Jq%((//+B*C8D6Qc'*>A  ! #D:L9Q9Q9S_9S+&#WZ]^W^+&#+9S_%j1 -11%;>Q>U>UV\^_>``Q %*4*:*:*< &JU"jF&:(("4"8"8Q"G!%8?%JJ,0)+= t9q=!6 MM)*=)>?()  (yy! EeEkEkEm!    %nw x  ];eX B fR(HL(i$m\J$U\B`s `  `%`%2`%0`* `* %`0`0#`0 `: #`57`5=`: a a3a a 60a *a !a0a <a 7 a$a$5`: cf[UR55nUVs/sHo3S:wdM X2;dMUPM nnU(a[R"SU3[5 UR 5R 5VVs/sHupVUPM nnnUR5HOnUS:XaUR5 O8UVs/sH'nXX:XaM URUS-5(aM%UPM) nnMQ [U5S:aSRU5n [SU 35egs snfs snnfs snf)z Checks a device map covers everything in a given model. Args: model (`torch.nn.Module`): The model to check the device map against. device_map (`Dict[str, Union[int, str, torch.device]]`): The device map to check. rzHThe following device_map keys do not match any submodules in the model: rRrz, zOThe device_map provided does not give any device for the following parameters: N) dictrrr UserWarning state_dictrrclearrrrrJ) r+rall_module_namesr3 invalid_keysrrall_model_tensorsrnon_covered_paramss r,check_device_mapr2s;E//12)Sz!"WA9RAzLS VWcVd egr .3-=-=-?-E-E-GH-G'$-GH!( "   # # % .!-D*37??;QTCT3U- !  ) !!YY'89]^p]q r  "%TI !s' D#D#D#D( D. D.(D.c LURS5(Ga+[USS9nUR5nUR5nSSS5 Wc[R SUS35 SS0nUR S5S;a[SUS 35eUSS:wa[S USS 35eUc [U5$[[UR555S :Xab[UR55S nUn[U[5(a&[!5(aSU3nO[#5(aSn[XS9$[[UR55S1- 5nSU;aUR%S5 UVs0sHoU/_M nnUR'5HPupXW;dM XR)WV s/sH%oU :XdU R+U S-5(dM#U PM' sn 5 MR USR)WV s/sH$o[-UR5/5;dM"U PM& sn 5 0n [/5(a3[1S[-UVs/sHn[X5PM sn5SS SS9n OSn UHnUn[U[5(a&[!5(aSU3nO[#5(aSn[USUS9nXHOn U b!U R3USS9 U R5U 5 UR7U 5X'U cM?U R95 MQ SSS5 M U bU R;5 U $[<R>"U[<R@"S5SS9$!,(df  GN1=fs snfs sn fs sn fs snf!,(df  GM$=f)a Load a checkpoint from a given file. If the checkpoint is in the safetensors format and a device map is passed, the weights can be fast-loaded directly on the GPU. Args: checkpoint_file (`str`): The path to the checkpoint to load. device_map (`Dict[str, Union[int, str, torch.device]]`, *optional*): A map that specifies where each submodule should go. It doesn't need to be refined to each parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the same device. z .safetensorspt) frameworkNz"The safetensors archive passed at zx does not contain metadata. Make sure to save your model with the `save_pretrained` method. Defaulting to 'pt' metadata.format)rtfflaxzf does not contain the valid metadata. Make sure you save your model with the `save_pretrained` method.z%The checkpoint passed was saved with z, we need a the pt format.r rrrrr*r0rRFw)main_process_onlytotalunit smoothingleave)rr5)devrefreshT) map_location weights_only)!rHr#rDrrrrOSErrorrJsafe_load_filerrrrr*rFrrrrextendrrWrr set_postfixset_description get_tensorupdatecloser4loadr5)checkpoint_filerfrD weight_namesr5 target_devicerodevice_weightsrr3tensors progress_barr s r,load_state_dictrTs// $ 71zz|H668L8   NN4_4EFoo !$'H << !)= =4_4EFII h 4 'DXhEWDXXrst t  !/2 23z((*+,1j//1215 & fc**'))*.vh )++(- %oLL3z0023vh>?GG#u%8??wVbjwN?'1'7'7'9# $"*11$0hLq4D U`cfUfHgLh(: 5 ! ( (\)o\cR`RgRgRikmNnEn!\)o pG ""#&+Qvs>#9:QR    $ ! & fc**'))*.vh )++(- $}UYZ-5'3(444O(88='(||C'8 '3(//1 6VU" '""$Nzz/ U8KZ^__a8 7N@i*p RVUsB!M- M?<"N "N !N *N N ANN- M< N# ch0n[5nUR5Hup4US:XaM [US5 UR5nSSS5 WH_nXVR [ R "S5:XaURUSU3-5 MBXVnURS5XSU3-'Ma M UR5HnXa;dM URU5 M U(a[RSU35 U$!,(df  N=f![a [ S5Sef=f)z Returns the state dictionary for an offloaded model via iterative onloading Args: model (`torch.nn.Module`): The offloaded model we want to save rr0Nz;Offloaded module must fit in CPU memory to call save_model!rrRzMThe following tensors were not saved because they were still on meta device: ) rralign_module_devicer MemoryErrorr5r4addrrremoverr)r+r placeholdersrrxmodule_state_dictr paramss r,get_state_dict_offloaded_modelrs+J5L++-  2:  g$VU3$*$5$5$7!4 %C %,, V0DD  !C5 !12&+F+199U+;J#i' ( %."  "      $#fgsftuv %43 g[\bf f gs( DD D D DDD1rdevice_to_put_offloadcUSURS5n[U5(dSn[X5 UR5R 5HupVUSU3-U;dMXbUSU3-'M SSS5 U$!,(df  U$=f)a Retrieve the state dictionary (with parameters) from an offloaded module and load into a specified device (defaults to cpu). Args: module: (`torch.nn.Module`): The module we want to retrieve a state dictionary from module_name: (`str`): The name of the module of interest state_dict (`Dict[str, Union[int, str, torch.device]]`): Dictionary of {module names: parameters} device_to_put_offload (`Union[int, str, torch.device]`): Device to load offloaded parameters into, defaults to the cpu. NrR)rfindhas_offloaded_paramsrrr)rxrrrrootm_keyrs r,get_state_dict_from_offloadrs* /**3/ 0D  ' ' $ V ;#..0668ME5'{"z1174AeW+-.9 <  < ; s.A<$A<< B  checkpointrCoffload_state_dictkeep_in_fp32_modulesoffload_8bit_bnbstrictfull_state_dictbroadcast_from_rank0c DU(aSSKJn [U5n [U5(a$[ U 5S:Xa[ R S5 Ub [X5 Uc"UbSUR5;a [S5eUb,Ub)SUR5;a[R"USS 9 [U[5(a"URS S 5n[[ U5nSnSn[R"R%U5(a([U5R'S 5(aUnGOU/nGO[R"R)U5(Gav[R*"U5Vs/sHnU[,:XdMUPM nn[R*"U5Vs/sHnU[.:XdMUPM nn[ U5S:Xa%[R"R1UUS5/nO[ U5S:Xa%[R"R1UUS5/nO[R*"U5Vs/sHnUR'S 5(dMUPM nn[ U5S:Xa[US[,S[.S35e[ U5S:Xa$[R"R1UUS5nO[US35e[SUS35eUb[R"R3U5Sn[5U5n[6R8"UR;55nSSS5 SW;aUSn[=[?[AUR5555nUVs/sH#n[R"R1UU5PM% nn0nU(a[BRD"5n0n[A5n[AURG5RI55nURK5VVs/sHunnUPM nnnURG5R5Vs1sH0n[U[ RL5(dM$URNiM2 nnU[ RN"S51- n UGHSn!Uc[QSS5(a[QSS5(a[ U 5S::d[ U5S::ajSSK)J*n"J+n# U [QSS5-n U (a[XRZ"5S:Xa []U!US9O0n$U#"UU$U""S-U U S.[QSS5(aSU 0O0D6S9 O[]U!US9n$UR]U$U S9 UR_[AU$RI55U- 5 GOE[]U!US9n$U$Ra5GH&un%n&S U%;aMU%U;aURcU%5 U (dM/U%n'[ U'5S:a@U'U;a:SR1U'R3S5SS!5n'[ U'5S:aU'U;aM:U'S :XaS U;a[U%S"35eUU'n(Un)Ubm[ Rd"U&5(aRUbOU[ Rf:Xa;S#n*UHn+U+U%;a U+S-U%;dU+U%:XdMSn* O U*(a[ Rhn)S$U%;aXU%RS$S 5U$RI5;a4U&Rj[ Rl:XaU$U%RS$S 5n,OSn,U(S:XaRU(dU%U;aBU)c U&Rjn)U(aW "UU&U%U)UUW,5 GM[oUU%SU)S%9 [qU&U%UUS&9 GMGMU(S':XaIU(aBU)c U&Rjn)U(aW "UU&U%U)WWW,5 GM[oUU%SU)S%9 [qU&U%WWS&9 GM[oUU%U(U&U)W,S(9 GM) A$[rRt"5 GMV U (dB[ U5S:a3[ R S)US*URvRxS+US,35 [{UU5 U(a#[}UWW5 [~R"U5 [X 5 gs snfs snfs snf!,(df  GNR=fs snfs snnfs snf).aN Loads a (potentially sharded) checkpoint inside a model, potentially sending weights to a given device as they are loaded. Once loaded across devices, you still need to call [`dispatch_model`] on your model to make it able to run. To group the checkpoint loading and dispatch in one single call, use [`load_checkpoint_and_dispatch`]. Args: model (`torch.nn.Module`): The model in which we want to load a checkpoint. checkpoint (`str` or `os.PathLike`): The folder checkpoint to load. It can be: - a path to a file containing a whole model state dict - a path to a `.json` file containing the index to a sharded checkpoint - a path to a folder containing a unique `.index.json` file and the shards of a checkpoint. - a path to a folder containing a unique pytorch_model.bin or a model.safetensors file. device_map (`Dict[str, Union[int, str, torch.device]]`, *optional*): A map that specifies where each submodule should go. It doesn't need to be refined to each parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the same device. offload_folder (`str` or `os.PathLike`, *optional*): If the `device_map` contains any value `"disk"`, the folder where we will offload weights. dtype (`str` or `torch.dtype`, *optional*): If provided, the weights will be converted to that type when loaded. offload_state_dict (`bool`, *optional*, defaults to `False`): If `True`, will temporarily offload the CPU state dict on the hard drive to avoid getting out of CPU RAM if the weight of the CPU state dict + the biggest shard does not fit. offload_buffers (`bool`, *optional*, defaults to `False`): Whether or not to include the buffers in the weights offloaded to disk. keep_in_fp32_modules(`List[str]`, *optional*): A list of the modules that we keep in `torch.float32` dtype. offload_8bit_bnb (`bool`, *optional*): Whether or not to enable offload of 8-bit modules on cpu/disk. strict (`bool`, *optional*, defaults to `False`): Whether to strictly enforce that the keys in the checkpoint state_dict match the keys of the model's state_dict. full_state_dict (`bool`, *optional*, defaults to `True`): if this is set to `True`, all the tensors in the loaded state_dict will be gathered. No ShardedTensor and DTensor will be in the loaded state_dict. broadcast_from_rank0 (`False`, *optional*, defaults to `False`): when the option is `True`, a distributed `ProcessGroup` must be initialized. rank0 should receive a full state_dict and will broadcast the tensors in the state_dict one by one to other ranks. Other ranks will receive the tensors and shard (if applicable) according to the local shards in the model. r )quantize_and_offload_8bitrrnNr*zeAt least one of the model submodule will be offloaded to disk, please pass along an `offload_folder`.T)exist_okrrz.jsonz .index.jsonz6 is not a folder containing a `.index.json` file or a z or a z filezI containing more than one `.index.json` file, delete the irrelevant ones.z`checkpoint` should be the path to a file containing a whole state dict, or the index of a sharded checkpoint, or a folder containing a sharded checkpoint or the whole state dict, but got rR weight_maprrQz2.2.0z2.7.0)StateDictOptionsset_model_state_dictz2.4.0)r)rrr)options)rrr;z doesn't have any device set.Frrr1r0)rzrOr{z(Some weights of the model checkpoint at z! were not used when initializing rz. This may or may not be an issue - make sure that the checkpoint does not have unnecessary parameters, or that the model definition correctly corresponds to the checkpoint.r)BbnbrrrrrrrrrJrAmakedirsr*r[rrr4rBisfilerHisdirlistdirr r rropenjsonloadsreadrrrtempfilemkdtemprrrrr5r!'torch.distributed.checkpoint.state_dictrrdistget_rankrrrris_floating_pointrhrerOrkrrgccollectrrrrGshutilrmtreer)-r+rrrCrOrrrrrrrrrcheckpoint_filesindex_filenamerpotential_state_binpotential_state_safetensorpotential_indexcheckpoint_folderr2 offload_indexstate_dict_folderstate_dict_indexunexpected_keys model_keysrr buffer_namest model_devicesmodel_physical_devicesrrrloaded_checkpointrrr param_device new_dtypeproceedr r{s- r,load_checkpoint_in_modelr-sz2&u-K&u--#k2Ba2G A ,[E*"8VzGXGXGZ=Z s    # (>6ZM^M^M`C` NT2% h+u%N ww~~j!! z? # #G , ,'N *|  z " "*,**Z*@V*@QADUq*@V13J1G%b1GA1PaKaa1G"%b " #q ( " Z9LQ9O PQ  + , 1 " Z9STU9V WX +-**Z*@^*@QAJJ}D]q*@O^?#q( !l"XYeXfflm~l@EF_%*!#j/!:L!M !l"kl hhrgsst v  !GGMM.9!< . !QJJqvvx(E" 5 ,'E!$s5<<>':";<HXYHX1BGGLL):A>HXYM$,,.eOU%%',,./J(-(;(;(=>(=WT1D(=L>','7'7'9'@'@'Bb'B!jQRTYT`T`FaXQXX'BMb*ell6.B-CC+   g..!$00S9O5PTU5UZ]^kZlpqZqj$(8w(GG$04==?a3G$O K" %%,(7%N^^bdkMlMl24HIrt%4OPZ$[!%%&7%G  " "3'8'='='?#@:#M N /J W %6%<%<%>! EJ&Z/#'' 3! ( +&*{*/L"%((;+<+>bkbRab b  " "o 5%:P:P     % %  % %  % %  & &  & &  % %  % %  % %     U ; >>ckcSbc c>>MkM_M M%%''r.r8c bU[R:XaSSKJn U"S 0UD6$[ SS9(aSSKJn UR"S 0UD6$[5(a*[RR R"S 0UD6$[5(a*[RR R"S 0UD6$[5(a*[RR R"S 0UD6$[5(a*[R R R"S 0UD6$[#5(a [R R"S 0UD6$[%5(a [R R"S 0UD6$['5(a<[)SS5(d [+S5e[R R"S 0UD6$[)SS 5(a [R R"S0UD6$[R,R R"S 0UD6$)aH A generic helper which will initialize the correct `GradScaler` implementation based on the environment and return it. Args: distributed_type (`DistributedType`, *optional*, defaults to None): The type of distributed environment. kwargs: Additional arguments for the utilized `GradScaler` constructor. r)ShardedGradScalerTr1NrQz2.8.0z7Grad Scaler with MPS device requires a Pytorch >= 2.8.0z2.3r)r)r)r))r)rrE*torch.distributed.fsdp.sharded_grad_scalerrKr torch_xla.ampamp GradScalerrr4r&rr'rr(rr%rrrr!rJr)r8rrKxamps r,get_grad_scalerrQ,s?///P *6**40$(((   yy}}''1&11   zz~~((2622   zz~~((2622   yy}}''1&11   yy##4V44   yy##4V44   g..VW Wyy##4V44 D% ( (99''9&9 9::>>,,6v6 6r.cSSKJn [US5=(a3 [URU5=(a URR $)a@ Checks if a module has offloaded parameters by checking if the given module has a AlignDevicesHook attached with offloading enabled Args: module (`torch.nn.Module`): The module to check for an offload hook. Returns: bool: `True` if the module has an offload hook and offloading is enabled, `False` otherwise. r )AlignDevicesHook_hf_hook)hooksrSrr*rToffload)rxrSs r,rrVs7) 6: & t:fooGW+X t]c]l]l]t]ttr.execution_devicec## [U5(a{Ub&URRnXRlURRU5 Sv URR US5 UbWURlggUbmUR SS9VVs0sHup4X4R _M nnnUHn[XU5 M Sv UR5Hup6[XU5 M gSv g!URR US5 UbWURlff=fs snnf!UR5Hup6[XU5 M f=f7f)a^ Context manager that moves a module's parameters to the specified execution device. Args: module (`torch.nn.Module`): Module with parameters to align. execution_device (`torch.device`, *optional*): If provided, overrides the module's execution device within the context. Otherwise, use hook execution device or pass NFr) rrTrW pre_forward post_forwardrr5rr)rxrWoriginal_devicerrror5s r,rrfs?F##  '$oo>>O/?OO , C OO ' ' /  OO ( ( 6+3B0,  %9?9P9PY^9P9_`9_+$4%9_` B+F:JK  '  +F&A!0  OO ( ( 6+3B0,a !(  +F&A!0sA:E)DAE)#D9;E)?D?+E)3D66 E)?'E&&E))NNNNFT)TFF)FF)NNF)NNr)r)NNNNF)NNNN)NNNNFTFF)r0) NNNFFNFFTF)FN)qrFrrr loggingrArYrrr collectionsrrtypingrrr4rrrrHr constantsr r dataclassesrrrimportsrrrrrrrrrmemoryrrrVrrrr rversionsr! torch_npu torch_mlu torch_sdaa torch_musa safetensorsr#safetensors.torchr$rWEIGHTS_INDEX_NAME getLoggerrrr-r8rFr[rNrOr_rrUr5rwr HalfTensorrrSrrrrrrrrrrrrr"r7r<rGrMrerlrsr}rrrrrr`rPathLiker-rIrQrcontextmanagerrrr.r,ros    0" %$6EE   ANN)&''%(%(!94   8 $])6)5c?)X5;;8!4ell!4uU\\35K/L!4P%)/326MQO> IIO>O> #sELL( )O> ELL ! O> E#u{{*+ , O> e../ O>d3U\\5<<-G(H#HIJO>O>O>fkp# II#(,#>B#cg#<&ryy&4&t&0%U299%UP;.4^4^d49o4^n:>U3 #45%++15DH ' 99' E#u||+, -'T#uS%,,->'?"?@A' 'X15DH # 99 # E#u||+, - #T#uS%,,->'?"?@A #$! %UXX__,- .$!>B38n$!gklogp$!NlxU38_eCHo-M(NOl^c5c5<<1G+H&H!IX[,m<DH37/3DH C 99CeCHouS#X>?@C&d3i0C E#u{{*+ , C T#uS%,,->'?"?@A C  CL%588??%(DH37/3DH = 99=eCHouS#X>?@=&d3i0= E#u{{*+ , = T#uS%,,->'?"?@A =  sCxsCx%S/ )*sCxIcNcOIsBII~ !  =@!B  3S 4 ? *+ !BP4815 i! %RYY' (i!sCx.i!c3hi!&d3i0 i! d49o. i!  8C=(299-tE#ryy.4I/J JK i!\DH37/3CG! %q 99qeCHouS#X>?@q&d3i0q E#u{{*+ , q T#uS%++-='>">?@ q  qqqqh  BII 4U3U\\CY=Z8Z3[ D\`~""))"R