;i1vSSKJrJr SSKJr SSKJr SSKJ r SSK r SSK J r SSKJr "SS5r"S S 5r"S S \ R$R&5r"S S\5r"SS\5rg))abc defaultdict)deepcopy)chain)OptionalN)sync_gpuc\rSrSrSrSrg)MockArgsc2UHn[XX5 M gN)setattr)self initial_datakeys l/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/bitsandbytes/optim/optimizer.py__init__MockArgs.__init__sC D|0 1 N)__name__ __module__ __qualname____firstlineno__r__static_attributes__rrrr r s2rr cN\rSrSrSrSrSrSr\S5r Sr S Sjr S r S r g) GlobalOptimManagerzC A global optimizer manager for enabling custom optimizer configs. Nc[S5e)NzCall get_instance() instead) RuntimeErrorrs rrGlobalOptimManager.__init__s899rcJ0Ul0UlSUlSUl/Ulg)NF) pid2config index2config optimizeruses_config_overridemodule_weight_config_tripler!s r initializeGlobalOptimManager.initialize s($)!+-(rcURc0URU5UlURR5 UR$r ) _instance__new__r))clss r get_instanceGlobalOptimManager.get_instance's6 == KK,CM MM $ $ &}}rc,[U5n[US[5(dSU0/n[U5H]up4[US5HFupV[ U5UR ;dM UR [ U5UR X54'MH M_ g)Nrparams)list isinstancedict enumerateidr$r%)rr2 param_groups group_indexgroupp_indexps rregister_parameters&GlobalOptimManager.register_parameters.sF| ,q/400%|45L"+L"9 K'h8 a5DOO+@DPRSTPU@VD%%{&<=9#:rcSUl[U[RR5(aU/n[U[R 5(aU/nUb UbUbeX#0nUbcUH\n[ U5UR;a)UR[ U5RU5 MEX@R[ U5'M^ gg)a Override initial optimizer config with specific hyperparameters. The key-values of the optimizer config for the input parameters are overridden This can be both, optimizer parameters like `betas` or `lr`, or it can be 8-bit specific parameters like `optim_bits` or `percentile_clipping`. Arguments: parameters (`torch.Tensor` or `list(torch.Tensors)`): The input parameters. key (`str`): The hyperparameter to override. value: The hyperparameter value. key_value_dict (`dict`): A dictionary with multiple key-values to override. Example: ```py import torch import bitsandbytes as bnb mng = bnb.optim.GlobalOptimManager.get_instance() model = MyModel() mng.register_parameters(model.parameters()) # 1. register parameters while still on CPU model = model.cuda() # use 8-bit optimizer states for all parameters adam = bnb.optim.Adam(model.parameters(), lr=0.001, optim_bits=8) # 2. override: the parameter model.fc1.weight now uses 32-bit Adam mng.override_config(model.fc1.weight, 'optim_bits', 32) ``` TN) r'r4torchnn ParameterTensorr7r$update)r parametersrvaluekey_value_dictr<s roverride_config"GlobalOptimManager.override_config8sJ%)! j%(("4"4 5 5$J j%,, / /$J ?u0!) ))!\N  %a5DOO+OOBqE*11.A-;OOBqE*   &rc>URRXU45 gr )r(append)rmodule param_nameconfigs rregister_module_override+GlobalOptimManager.register_module_overridems ((//V0LMr)r%r(r&r$r')NNN)rrrr__doc__r,rr) classmethodr/r=rHrOrrrrrrs>I:. W3[TU]X5 SUl0UlX@l[ R R5Ul[R5Ul 1SkUl US:XaUR5 gg)a2 Base 8-bit optimizer class. Arguments: params (`torch.Tensor`): The input parameters to optimize. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. is_paged (`bool`, defaults to `False`): Whether the optimizer is a paged optimizer or not. F> max1max2qmap1qmap2state1state2absmax1absmax2new_max1new_max2 gnorm_vec unorm_vecN) superr initialized name2qmapis_pagedFGlobalPageManagerr/page_mngrmngnon_castable_tensor_keys fill_qmap)rr2defaults optim_bitsrg __class__s rrOptimizer8bit.__init__rsn *  ++88: %224 ) % ? NN  rc[R"SS9URS'[R"SS9URS'g)NT)signeddynamicFudynamic)rhcreate_dynamic_maprfr!s rrmOptimizer8bit.fill_qmaps4$%$8$8$Ey!%&%9%9%Gz"rc$>[TU]U5 gr )rd __setstate__)rstaterps rryOptimizer8bit.__setstate__s U#rc^^^[U5nTRnUSn[U5[U5:wa [S5eSU5nSU5n[ S[ XV555(a [S5e[ [ R"SU55[ R"SU555VVs0sHupxXx_M n nnUUU4S jm[[5n US R5HupX;aXn T"X5X'MXU 'M S n[ X45VVs/sHunnU"UU5PM nnnTRU US .5 g s snnfs snnf)aLoad an optimizer state. Arguments: state_dict (`dict`): An optimizer state (should be returned from a call to `state_dict`) to load. move_to_device (`bool`, defaults to `True`): Whether to move the optimizer's state to the device. r8z# UHn[US5v M g7fr2Nlen.0gs r 0Optimizer8bit.load_state_dict..s71c!H+&&c3># UHn[US5v M g7fr~rrs rrrs= 1c!H+&& rc3.# UH upX:gv M g7fr r)rp_lens_lens rrrsN2M,%u~2Msz]loaded state dict contains a parameter group that doesn't match the size of optimizer's groupc3*# UH oSv M g7fr~rrs rrrs#FAhKc3*# UH oSv M g7fr~rrs rrrs#@AhKrcD>^[U[R5(aPTR5(a9UR[R :waUR TR5nU$[U[5(a^UR5HHup#UTR;a(T(aUR TR5X'M;M=T"TU5X'MJ U$[U[R5(a[U5"UU4SjU55$U$)zBMake a deep copy of value, casting all tensors to device of param.c36># UHnT"TU5v M g7fr r)rvcastparams rr>Optimizer8bit.load_state_dict..cast..s"A5a4q>>5s)r4r@rCis_floating_pointdtypeuint8tor5itemsrldevicecontainer_abcsIterabletype)rrFkrrmove_to_devicers` rr+Optimizer8bit.load_state_dict..casts%..**,, 1K!HHU[[1E E4((!KKMDAD999)'(ttELL'9EH*$(q> * E>#:#:;;E{"A5"AAA rrzcUSUS'U$)Nr2r)r: new_groups r update_group3Optimizer8bit.load_state_dict..update_groups"'/Ih  r)rzr8N) rr8r ValueErroranyzipr from_iterablerr5rry)r state_dictrgroups saved_groups param_lens saved_lensold_idr<id_maprzrrrrrngr8rs` ` @rload_state_dictOptimizer8bit.load_state_dictshj) ""!.1 v;#l+ +[\ \77 = = N#j2MN N No !###F#FF###@#@@   I   2D!w'--/DA{ #E~ a 0 :=V9RS9R2 Q+9R S E<HI_ \Ts E!3E'c[UR5Hup[US5Hup4X@R;dMURUnUR5Hgupg[ U[ R 5(dM&[USS5nU(aM<URUR5URUU'Mi M M g)Nr2rgF) r6r8rzrr4r@rCgetattrrr) rgindexr:pindexr<valuesrrrgs rto_gpuOptimizer8bit.to_gpus&t'8'89MF&uX7  ?!ZZ]F & %a66'.q*e'DH#+83444> 1 a 0 !/8:rclURRGHupn[X5nUce[U[R 5(d![U[R 5(deSn[UR5HupgU(a M}[US5HupU(a M,[U 5[U5:XdM)X0RR[U 5'URR[U 5URRXh4'SnM M GM g)NFr2T) rkr(rr4r@rCrBr6r8r7r$r%) rrLattrrNpmodulefoundrr:rr<s rcheck_overridesOptimizer8bit.check_overridess$(HH$H$H F&f+G& &&gu||44 7EOO8\8\ \\E!*4+<+? $"<"> %IrctSnUb%[R"5 U"5nSSS5 UR(d'UR5 UR 5 SUlSn[ UR 5HupE[ US5HrupcURcMURUn[U5S:XaURXSXF5 URU5 URXSXF5 [U5 Mt M UR(aUb [U5 U$!,(df  GN =f)zPerform a single optimization step. Arguments: closure (`Callable`, *optional*, defaults to `None`): A closure that reevaluates the model and returns the loss. NTr2r)r@ enable_gradrerrr6r8gradrzr init_stateprefetch_state update_steprrg)rclosurelossr<rr:rrzs rstepOptimizer8bit.steps  ""$y%  " KKM#D  &t'8'89MF&uX7 66> 1 u:?OOEf=##A&  6: 8: ==Q] QK 5%$s D(( D7c|0nUSUS'USUS'USUS'USUS'URSS5US'URSS5US'URS S5US 'URRUS 'URRUS 'URRUS 'URR US 'URR US'URRUS'X4URR;a)URURRX45 U$)Nbetaseps weight_decaylralphat_alphart_beta3ro min_8bit_sizepercentile_clipping block_wise max_unorm skip_zeros) getargsrorrrrrrkr%rD)rrrr:rNs r get_configOptimizer8bit.get_config,s,.we u !&~!6~T{t ))GS1w!IIi3y!IIi3y#yy33|"&))"9"9(, (E(E$%#yy33|"ii11{#yy33|  txx44 4 MM$((//0@A B rc[S5e)Nz(init_state method needs to be overriddenNotImplementedErrorrr:r<rrs rrOptimizer8bit.init_state@s!"LMMrc[S5e)Nz-The update_step method needs to be overriddenrrs rrOptimizer8bit.update_stepCs!"QRRrcZUR(aUR5S:a[R"XURS9$[ R "URX!RS.6n[ R"US5 URRRU5 U$)Ngj@rrr) rgnumelr@ zeros_likerrh get_pagedshapefillrj paged_tensorsrK)rr<rbuffs rget_state_bufferOptimizer8bit.get_state_bufferFsp}} C##A188D D;;uXXFD FF4O MM ' ' . .t 4KrcUR(acURUnUSn[USS5nU(a:[R"US5 SU;a[R"US5 gggg)Nr[rgFr\)rgrzrrhprefetch_tensor)rr<rzs1rgs rrOptimizer8bit.prefetch_statePsk ==JJqMExBr:u5H!!%/2u$%%eHo6% r)rergrkrfrlrj) F)Tr )rrrrrrmryrrrr@no_gradrrrrfloat32rrr __classcell__rps@rrTrTqsk#JH$HJT B%& ]]_##J(NS). 77rrTc^\rSrSrSS\\S\\4U4Sjjjr\R"5S5r \R"5S5r Sr U=r $) Optimizer2Statei[rrc >SU::d[SU35eSU::d[SU35e[U[5(aZURSS5RSS5R 5R S5nUVs/sHn[ U5PM nn[[U55H)nSUUs=::aS:aMO [S US UU35e SU::d[S U35e[X4XVUUUS 9n[TU]-UUX~5 Uc+0nXxS'XS'XS'XS'XS'XS'[U5Ul OXl Xlg s snf)a@ Base 2-state update optimizer class. Arguments: optimizer_name (`str`): The name of the optimizer. params (`torch.Tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-3): The learning rate. betas (`tuple`, defaults to (0.9, 0.999)): The beta values for the optimizer. eps (`float`, defaults to 1e-8): The epsilon value for the optimizer. weight_decay (`float`, defaults to 0.0): The weight decay value for the optimizer. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. max_unorm (`float`, defaults to 0.0): The maximum value to normalize each block with. skip_zeros (`bool`, defaults to `False`): Whether to skip zero values for sparse gradients and models to ensure correct updates. is_paged (`bool`, defaults to `False`): Whether the optimizer is a paged optimizer or not. alpha (`float`, defaults to 0.0): The alpha value for the AdEMAMix optimizer. t_alpha (`Optional[int]`, defaults to `None`): Number of iterations for alpha scheduling with AdEMAMix. t_beta3 (`Optional[int]`, defaults to `None`): Number of iterations for beta scheduling with AdEMAMix. rInvalid learning rate: Invalid epsilon value: (),? Invalid beta parameter at index : Invalid weight_decay value: )rrrrrrrNrorrrrr)rr4strreplacestripsplitfloatrangerr5rdrr roptimizer_name)rr r2rrrrrorrrrrrrgrrrbirnrps rrOptimizer2State.__init__\snxby6rd;< <cz6se<= = eS ! !MM#r*223;AACII#NE',-u!U1XuE-s5z"A%((S( #CA3bq !STT#l";L>JK KC%Y`jq  :@ <D!+ $1 !*=& '!+  ) !+  DII,5.sEcPURX4U5nUSS:Xa[RnO+USS:Xa[RnO[ SUS35eUR 5US:a[RnUR UnSUS'U[R:XaDURU[RS9US 'URU[RS9US 'GOeU[R:XGaPUSS:XaS UR;aUR5 URS RUR5URS 'URS RUR5URS 'URU[RS9US 'URS US 'URU[RS9US 'URS US'US(aSnUR 5n X-[X-5-n [R"U 4[RURS9US'[R"U 4[RURS9US'O[R"S[RURS9US'[R"S[RURS9US'[R"S[RURS9US'[R"S[RURS9US'USS:a"[R"SURS9US'USS:a#[R"SURS9US 'gg)!Nrorrc(Amount of optimizer bits not supported: rrrrr[r\rtrurYrZrrr]r^rWr_rXr`rdrrrarrrbrr@rrrrrzrrfrmrrboolzeros rr:r<rrrNrrz blocksizenblockss rrOptimizer2State.init_states7 , 2 %MME L !Q &KKE%(PQWXdQePf&gh h 779vo. .MME 1 f EMM !"33AU]]3KE(O"33AU]]3KE(O ekk !V}!DNN2NN$,0NN9,E,H,H,Ry)-1^^J-G-J-J188-Tz*"33AU[[3IE(O!^^I6E'N"33AU[[3IE(O!^^J7E'Nl# GGI.D,??#(;;y VWV^V^#_i #(;;y VWV^V^#_i % D ahh Wf $)KKEMMRSRZRZ$[j! % D ahh Wf $)KKEMMRSRZRZ$[j! ' (3 .!&VAHH!EE+  +  $!&T!((!CE+  %rcURR5UlURR5UlURUnURnUR X4U5nUS==S- ss'USnUSS:a#[ R "UUSUUS5upn OSn USR[R:Xa[ R"URUUUSUSS US UUS US USS[US5S :aUSSOSURSS5USU USS:aUSOSUSUSS9 gUSR[R:XaUS(d[ R"URUUUSUS USS USSUS UUS USUSUSUSUSUSUSU USS:aUSOSUSS9 USUSsUS'US'USUSsUS'US'gUSR[R:XaUS(a[ R "URUUUSUS USS USS[US5S :aUSSOSURSS5US UUS USUSUSUSUSU USS 9 ggg)!Nrrrrrarr[rrrrr\rrrrrbrrrrrYrZrWrXr_r`) gnorm_scalerbrr]r^r"r)data contiguousrrzrrhrrr@roptimizer_update_32bitr rrroptimizer_update_8bitoptimizer_update_8bit_blockwise rr:r<rrrzrrNr current_gnorm clip_valuer"s rrOptimizer2State.update_stepsN""$""$ 1 vv7 f  V} ' (3 .565J5Jk",- 6 2M{K ? EKK / $ $##hw"u t hw"&)&/&:a&?w"S 7C(~&&,[&9C&?k"T -!,/# (8_ " "ekk 1&:N # ###hhw"w"u t ggf f j!j!~&'06{0Cc0I% ,t -) 005Z/@%- ,E&M5,/4Z/@%- ,E&M5, 8_ " "ekk 1f\6J - -##hhw"w"&)&/&:a&?w"S 7C(u t ggi i ~&'!,/' 7K 1rrr )MbP?)?g+?:0yE>rrNrTrFFrNN) rrrrrintrr@rrrrrrs@rrr[s   !%!%%]-"##]-$#%]-]-~ ]]_1D1Df ]]_\\rrc^\rSrSrSU4Sjjr\R "5S5r\R "5S5rSr U=r $)Optimizer1StateiOc>SU::d[SU35eSU::d[SU35e[[U55H'nSXOs=::aS:aMO [SUSXO35e SU::d[SU35e[X4XVS9n[TU]UUX~5 Uc+0nXxS 'XS 'XS 'XS 'XS'XS'[ U5UlOXlXlg )a Base 1-state update optimizer class. Arguments: optimizer_name (`str`): The name of the optimizer. params (`torch.Tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-3): The learning rate. betas (`tuple`, defaults to (0.9, 0.0)): The beta values for the optimizer. eps (`float`, defaults to 1e-8): The epsilon value for the optimizer. weight_decay (`float`, defaults to 0.0): The weight decay value for the optimizer. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. max_unorm (`float`, defaults to 0.0): The maximum value to normalize each block with. skip_zeros (`bool`, defaults to `False`): Whether to skip zero values for sparse gradients and models to ensure correct updates. is_paged (`bool`, defaults to `False`): Whether the optimizer is a paged optimizer or not. rrrrrrr)rrrrNrorrrrr) rrrr5rdrr rr )rr r2rrrrrorrrrrrrgr rnrps rrOptimizer1State.__init__Psdby6rd;< <cz6se<= =s5z"A%((S( #CA3b !STT#l";L>JK K2O :@ <D!+ $1 !*=& '!+  ) !+  DII,rcURX4U5nUSS:Xa[RnO+USS:Xa[RnO[ SUS35eUR 5US:a[RnUR UnSUS'U[R:Xa#URU[RS9US 'GOiU[R:XGaTUSS:XaUS UR;aUR5 URS RUR5URS 'URU[RS9US 'URS US 'US (aWS nUR 5n X-[X-5-n [R"U 4[RURS9US'Ob[R"S[RURS9US'[R"S[RURS9US'USS:a"[R"SURS9US'USS:a#[R"SURS9US'gg)Nrorrcrrrrrr[rtrYrrrr]rrWr_rrrrrarrrbrrs rrOptimizer1State.init_states7 , 2 %MME L !Q &KKE%(PQWXdQePf&gh h 779vo. .MME 1 f EMM !"33AU]]3KE(O ekk !V}!DNN2NN$,0NN9,E,H,H,Ry)"33AU[[3IE(O!^^I6E'Nl# GGI.D,??#(;;y VWV^V^#_i % D ahh Wf $)KKEMMRSRZRZ$[j! ' (3 .!&VAHH!EE+  +  $!&T!((!CE+  %rcURR5UlURR5UlURUnURnUR X4U5nUS==S- ss'USnUSS:a#[ R "UUSUUS5upn OSn USR[R:Xa[[ R"URUUUSUSS US UUS SUSSS S US U USS :aUSOSUSUSS9 gUSR[R:XaUS(du[ R"URUUUSSUSS USSUS UUS USSUSSUSSUS U USS :aUSOSUSS9 USUSsUS'US'gUSR[R:Xa]US(aR[ R"URUUUSSUSS USSS S US UUS USSUSSUS U USS9 ggg)Nrrrrrarr[rrrrrrrrbrr!rrYrWr_)rr]r#)r$r%rrzrrhrrr@rr&r rr'r(r)s rrOptimizer1State.update_steps""$""$ 1 vv7 f  V} ' (3 .565J5Jk",- 6 2M{K ? EKK / $ $##hw"u t w"~&&,[&9C&?k"T -!,/# (8_ " "ekk 1&:N # ###hw"w"u t gf j!~&&,[&9C&?k"T -) .05Z/@%- ,E&M5, 8_ " "ekk 1f\6J - -##hw"w"u t gi ~&'!,/' 7K 1rr-) r.)r/rr0rrNr1rTrFF) rrrrrr@rrrrrrs@rr4r4Osi   K-Z ]]_)D)DV ]]_ZZrr4) collectionsrrrcopyr itertoolsrtypingrr@bitsandbytes.functional functionalrhbitsandbytes.utilsrr roptim OptimizerrTrr4rrrrDsk ; #'22 XNXNvg7EKK))g7TqmqhUmUr