9i>,SSKJr SSKrSSKrSSKJr SSKJrJ r J r SSK r SSK r SSK Jr SSKJr SSKJrJr SS KJrJr 0r\R0\R24\R4\R64\R8\R:4\R<\R>4\R0\R24\R4\R64S .r "S S 5r!"S S5r""SS5r#\ RH"SSS9r%\ RLRO5S:aS\ R4Sjr(OSSK)r)S\ R4Sjr(\ RT\%S.Sjr+SS\ R4Sjjr,SSjr-SSjr.SSjr/SSjr0SSjr1SS jr2SS!jr3S"\\ \ R4S#jr4S$\S%\Rj4S&jr6S\ \S%\ \Rj4S'jr7"S(S)5r8SS\ RS*\ \ RS+\ \ RS,\ \ RS%\9\ R\844 S-jjr:SS\ RS.\ \8S+\ \ RS*\ \ RS,\ \ RS/\;S%\ R4S0jjrjj5rH\"S9\ES:9SS\S*\S,\ \ RS%\4S?jj5rISS@\JSA\SB\SC\SD\KSE\KSF\;SG\KSH\ \ RSI\KSJ\KSK\KSL\KSM\KSN\ \ RSO\KS%S4"SPjjrL\"SQ\ES:9SS@\JSA\SB\SC\SH\ \ RSD\KSI\KSE\KSF\;SG\KSR\SS\ \ RST\SU\ \ RSV\SW\ \ RSL\KSM\KSN\ \ RSO\KS%S4*SXjj5rMSS@\JSA\SB\SC\SH\ \ RSD\KSI\KSJ\KSK\KSE\KSF\;SG\KSR\SS\ \ RSY\SZ\ \ RSL\KSM\KS%S4&S[jjrN\"S9\ES:9SS\\S]\SF\;S^\;4S_jj5rO\ R4S`jrQSS\Sa\S,\ \ R4SbjjrRSS\Sa\S,\ \ R4ScjjrSSS\Sa\S,\ \ R4SdjjrTS\ R4S\ RSa\ RS,\ \ R4SejjrVSS\ RSf\ RSg\ RS,\ \ RSh\ \ R4 SijjrW\"S9\ES:9SS\ RSf\ \ RSg\ \ RSj\ \ RS%\9\ R\ R\ \ R44 Skjj5rX\"S9\ES:9SS\ R4Sljj5rY"SmSn5rZ"SoSp5r["SqSr5r\Ssr]Str^\ R4Sujr`SS\ RSg\ \ RSf\ \ RSv\ \ RSw\ \ R4 SxjjraS\ RSy\ R4SzjrbSS\ R4S{jjrcSS|\ \Z\ R4Sa\ RS,\ \ R4S}jjrdSS~jreSrfg))IterableN)prod)AnyOptionalUnion)Tensor) deprecated)pack_dict_to_tensorunpack_tensor_to_dict)HIP_ENVIRONMENTlib)adammomentumrmsproplionlamblarsc>\rSrSrSrSrSr\S5rSSjr Sr g) GlobalPageManager3Nc[S5eNzCall get_instance() instead RuntimeErrorselfs g/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/bitsandbytes/functional.py__init__GlobalPageManager.__init__6899c/UlgN paged_tensorsrs r initializeGlobalPageManager.initialize9s r"cURc0URU5UlURR5 UR$r$ _instance__new__r'clss r get_instanceGlobalPageManager.get_instance<6 == KK,CM MM $ $ &}}r"cLURSSS2Hn[X!5 M g)N)r&prefetch_tensor)rto_cputs r prefetch_allGlobalPageManager.prefetch_allCs%##DbD)A A &*r"r%F) __name__ __module__ __qualname____firstlineno__r+rr' classmethodr/r7__static_attributes__r"rrr3s*I:  'r"rc:\rSrSrSrSrSr\S5rSr Sr g)CUBLAS_ContextKNc[S5errrs rrCUBLAS_Context.__init__Nr!r"c0Ulgr$contextrs rr'CUBLAS_Context.initializeQs  r"cURc0URU5UlURR5 UR$r$r*r-s rr/CUBLAS_Context.get_instanceTr1r"cURUR;a[RR 5n[RR U5 [ R"[R"55URUR'[RR U5 URUR$r$) indexrHtorchcudacurrent_device set_devicectc_void_pr get_context)rdevice prev_devices rrTCUBLAS_Context.get_context[s} <r/rTr?r@r"rrBrBKs*I: *r"rBc4\rSrSrSrSrSr\S5rSr g)Cusparse_ContextdNc[S5errrs rrCusparse_Context.__init__gr!r"c`[R"[R"55Ulgr$)rRrSr get_cusparserHrs rr'Cusparse_Context.initializejs{{3#3#3#56 r"cURc0URU5UlURR5 UR$r$r*r-s rr/Cusparse_Context.get_instancemr1r"rG) r:r;r<r=r+rr'r>r/r?r@r"rrYrYds%I:7r"rYrO)rMac@[RRU5$r$)rNrO device_ofrbs r_cuda_device_ofrf}szz##A&&r"c,[R"5$r$) contextlib nullcontextres rrfrfs%%''r"dtyperUcUR[U5-n[R"[R "U55n[R "U[R"[R55n[RRXRS9n[R"X`[U5S9RU5nSUlUR UlU$)N)shape)rkcountT)itemsizerrcget_managed_ptrrRc_size_tcastPOINTERc_intnp ctypeslibas_arrayrN frombufferviewis_pagedrM page_deviceid)rkrUrm num_bytescuda_ptrc_ptr new_arrayouts r get_pagedrse,I##BKK $:;H GGHbjj2 3E %%e%9I   9e E J J5 QCCL C Jr"FAcUR(dS5eU(aSnO URn[R"[ U5[ R "UR5[ R"U55 g)Nz%Only paged tensors can be prefetched!r3) rzr{r cprefetchget_ptrrRrqnbytesc_int32)rr5deviceids rr4r4sO ::>>>: ??MM'!*bkk!((3RZZ5IJr"c SnUR[R:Xa,[[SUS3S5n[ R "U5nOIUR[R:Xa+[[SUS3S5n[ R"U5nUc[SU35e[USS5nU(a U(a[U5 Ub [U5 U"[U5[U5W[ R"UR555 UR(dUR(a[RR!5 gg)Nc_fp32_uint8zFunction not implemented: is_managedF)rkrNfloat32getattrrrRc_floatuint8c_uint8NotImplementedErrorr4rc_int64numelrzrO synchronize) func_namerBvalueprefetchfunccvaluers relementwise_funcrs Dww%--sa {%0$7E" EKK sa {&148E" |!$>yk"JKKL%0Jh = A WQZAGGI)>?zzQZZ    r"c [SUSU5 g)Nfillr)rrrUrs rrrsVQe,r"c[SXS5 g)N_mulrr)rrrUs rrrsVQ1%r"crU(aSOSnSU-nU(dUS:aU(dSU-OSU-S- n[R"USU5nSUR5- nUS:XaU$UR5S-n[R"USUR 5S/U--XWSR 5-5$) Nr ?r)rNlinspacerrtolist)signed total_bitsadd_zerosign total_valuesvaluesgapls rcreate_linear_maprs4sDj=L:> -3q*}: 8I ^^D#| 4F   C ax LLNa ||F2AJ--/1#);fRj>O>O>QQRRr"cSSKJn U(axUR[R "USS5SS5R 5nS/S-nUR[R "USS5SS5*R 5nOwUR[R "USS5SS5R 5nS/S -nUR[R "USS5SS5*R 5nXE-U-n[R"U5nUR5RnXR5-nUR5S :XdeU$![an[S5UeSnAff=f) Nr)normzZScipy is required for `create_normal_map`. Install `bitsandbytes` with the `[test]` extra.g? r3rr) scipy.statsr ImportErrorppfrNrrrsortrmaxr) offsetuse_extra_valueriev1v2v3vrs rcreate_normal_maprsN$  XXennVS!4Sb9 : A A CSH xxvsA6s;<< D D F XXennVS!4Sb9 : A A CSH xxvsA6s;<< D D F " A \\!_F [[] ! !F jjlF <<>S  M/  h  sE E:) E55E:cvUnUnU(aSOSnXE-X6- :Xde/n[[SX- -*SX- -S55HupURSU -5 M /n [[R "SS/US95n SUS- -n [SU-5Hn U HnU S:waSOSn[[U55HunnUUSUS-*--- nM U S:Xa USU *--nOUSX- S- *--nU RU5 U(dMrU RU*5 M M [ U 5SU-:XdeU R5 US:a1S[ U 5- n[U5HnU RS5 M U R5 [R"U 5nUUR5-nU$)Nr rr)repeatrr) enumeraterangeappendlist itertoolsproductlenrrNtensorr)r exponent_bitsprecision_bitsrephas_signevaluesivalrlstbiasevalue bit_patternrpvalrcodes rcreate_fp8_maprsAAqH 5J) )) )GEA-*B$C"DaMLdFeghijq#vkF y  !Q? @C " #Dm,-K1A!E$T+%674Ah//8{T7 *v}q'8%9 99 MM% v uf%. v;!Z- '' ' KKMA~CKsA MM!  KKM << DDHHJD Kr"c/nUS- nSXA- -S- n[U5Hn[U(a SXd-U- -S-OSXd-U- S--S-5n[R"SSU[RS9nUSSUSS-S- n USUS- *U--U -R 5- nU(dMUSUS- *U--*U -R 5- nM US :a~[R"SSUS-[RS9nUSSUSS-S- n USUS- *W--U -R 5- nU(a!USUS- *U--*U -R 5- nUR S 5 UR S 5 [U5SU-:XdeS [U5- n [U 5HnUR S 5 M UR5 [R"U[RS9$) a Creates the dynamic quantiztion map. The dynamic data type is made up of a dynamic exponent and fraction. As the exponent increase from 0 to -7 the number of bits available for the fraction shrinks. This is a generalization of the dynamic type where a certain number of the bits and be reserved for the linear quantization region (the fraction). n determines the maximum number of exponent bits. For more details see (8-Bit Approximations for Parallelism in Deep Learning)[https://arxiv.org/abs/1511.04561] r rg?rkNr3@ rrr) rintrNrrrrrrr) rmax_exponent_bitsrdata non_sign_bitsadditional_itemsrfraction_items boundariesmeansrs rcreate_dynamic_maprs" DNM]>?!C $ % !#&77 81 <q(+<1S8 ",q01A56%?GGII 6 r 1A 56:;1S8 ",q01A56%?GGII  r 1A 56:;!> hhmmu, ,F KK7 8  iIPkQIPDElslsuvu}u}k~IPkQjR S   7|arRYtZRYMNu|u|~GGuHRYtZs[ \   MkQ tZs !D !D rreturncRURRS:XaG[R"[R R URR55$[R"[R RURR55$)Nxpu) rUrrRrSrN_C_xpu_getCurrentRawStreamrM_cuda_getCurrentRawStream)rs r_get_tensor_streamrssd }}U"{{588<"++C,=,=>>>> g %\\%(@"ABEEfMF/226:!"45/033F;eW^%<= F(NF|,8$''/k*%((0%!1229'2B2N%**WW-.TX  KgDs III!IcURURURUR[ UR 5R S5[UR5S.nUR(aURURRURRURRR5[ URR 5R S5URR5S.5 U(dU$UR5VVs0sH(up4[!U["R$5(dM&X4_M* nnnUR5VVs0sH(up4[!U["R$5(aM&X4_M* nnn['U5USUR-'U$s snnfs snnf)z returns dict of tensors and strings to use in serialization via _save_to_state_dict() param: packed -- returns dict[str, torch.Tensor] for state_dict fit for safetensors saving ztorch.)r rr rrkrm)rr rr r zquant_state.bitsandbytes__)r rr rstrrkstriptuplermrr rcloneritemrrrNrr )rpackedrr(rqs_packed_dictnon_tensor_dicts ras_dictQuantState.as_dictsR //kk_**844::&   ;; NN%)[[%7%7(, (=(=(, (8(8(>(>(@$' (9(9$:$@$@$J%)[[%5%5%7  N,3==?Z?41jELL>Y$!$?Z,3MMO_ODA:aQVQ]Q]C^414O_NabqNr84??JK[_s9%G "G >%G'GcURRU5UlURRU5UlUR(aURRU5UlUR RRU5UR lUR RRU5UR lggr$)rr&rrrr)rrUs rr& QuantState.tosIILL( kknnV, ;;++..0DK!%!3!3!6!6v!>DKK #{{//226:DKK  r"c0[U[5(dg[R"URURSS9=(GaO UR UR :H=(Ga. [R"UR UR SS9=(a URUR:H=(a URUR:H=(a URUR:H=(a URb&URbURUR:HOURURL=(aK URb&URbURUR:H$URURL$)NFgư>)atol) rrrNallcloserrmrrkr r rr)rothers r__eq__QuantState.__eq__"s3%,, NN4;; 4 @   ekk)  tyy%**4@  ekk) %//1   5#3#33  ;;*u||/G u||+[[ELL0 ;;*u||/G u||+ [[ELL0 r") rr rrkrrr rmr)NNNNNNNr9)r:r;r<r=__doc__valid_quant_typesrr$rrr>dictr.rrNrUr+r6r&r>r?).0xs00rrrsa&8IJ8I1N1#.8IJ M$ )*(0S#X0 000d@; kKsA#rrrrc Uc=S[;a*[5RUR5[S'[Sn[R R RRUURUR5U5upgU(aQUR5nXx-n[XtSS9up[U URURSS9UURUU S9n O,[XqRURSS9X@RS9n UbURU5OUnUb URU R5U l X;4$)aQuantize a tensor in blocks of values. The input tensor is quantized by dividing it into blocks of `blocksize` values. The the absolute maximum value within these blocks is calculated for scaling the non-linear quantization. Args: A (`torch.Tensor`): The input tensor. Supports `float16`, `bfloat16`, or `float32` datatypes. code (`torch.Tensor`, *optional*): A mapping describing the low-bit data type. Defaults to a signed 8-bit dynamic type. For more details, see (8-Bit Approximations for Parallelism in Deep Learning)[https://arxiv.org/abs/1511.04561]. absmax (`torch.Tensor`, *optional*): A tensor to use to store the absmax values. out (`torch.Tensor`, *optional*): A tensor to use to store the result. blocksize (`int`, *optional*): The size of the blocks. Defaults to 4096. Valid values are 64, 128, 256, 512, 1024, 2048, and 4096. nested (`bool`, *optional*): Whether to additionally quantize the absmax values. Defaults to False. Raises: ValueError: Raised when the input data type is not supported. Returns: `Tuple[torch.Tensor, QuantState]`: A tuple containing the quantization results. - `torch.Tensor`: The quantized tensor. - [`QuantState`]: The state object used to undo the quantization. dynamicF)r rT)copy)rrr rkrrrrr rk) name2qmaprr&rUrNops bitsandbytesquantize_blockwisedefaultmeanrrkcopy_r) rrrrr r_out_absmaxrqabsmaxrrs rrLrL:s+F | I %#5#7#:#:188#DIi #II**==EE  MD ,WRWX -''  !ggahhTg6R^govovw !_#))D/$C#\\+*<*<=   r"rr c UcUceUc@Uc=S[;a*[5RUR5[S'[SnUc[ X#U[ R S9nURnUR(a\[URUR5nX!R- nUR[ R :waUR5nUbo[ RRRR!UUUR"RUR5UR$URUS9 U$[ RRRR'UUUR"RUR5UR$UR5$)a|Dequantize a tensor in blocks of values. The input tensor is dequantized by dividing it into blocks of `blocksize` values. The the absolute maximum value within these blocks is used for scaling the non-linear dequantization. Args: A (`torch.Tensor`): The quantized input tensor. quant_state ([`QuantState`], *optional*): The quantization state as returned by [`quantize_blockwise`]. Required if `absmax` is not provided. absmax (`torch.Tensor`, *optional*): A tensor containing the scaling values. Required if `quant_state` is not provided and ignored otherwise. code (`torch.Tensor`, *optional*): A mapping describing the low-bit data type. Defaults to a signed 8-bit dynamic type. For more details, see (8-Bit Approximations for Parallelism in Deep Learning)[https://arxiv.org/abs/1511.04561]. Ignored when `quant_state` is provided. out (`torch.Tensor`, *optional*): A tensor to use to store the result. blocksize (`int`, *optional*): The size of the blocks. Defaults to 4096. Valid values are 64, 128, 256, 512, 1024, 2048, and 4096. Ignored when `quant_state` is provided. Raises: ValueError: Raised when the input data type is not supported. Returns: `torch.Tensor`: The dequantized tensor. The datatype is indicated by `quant_state.dtype` and defaults to `torch.float32`. rFrHr)rIrr&rUrrNrrrdequantize_blockwiserrrkr%rJrKrrr rM)rrrrrr rs rrUrUssR  "f&88 8 | + I %#5#7#:#:188#DIi # YV[VcVcd   F%k&8&8+:L:LM$$$ <<5== (\\^F  3377       )  ! !    8  99 ! ! 6 6 > > AHH%  r"cnUcSnSnUS:Xa/SQnO8US:Xa/SQnO-US:Xa/SQnO"US:XaUS :Xa /S QSSS 2nO [S 5eUc[S US35e[R"X1S9nURUR 5R 55 UR 5S:XdeU$)NrOr)rg 6GgfgTFٿgI4ҿg০ǿg Org__?g`\?g?g@g?g4?g`?g`v"?rr)rg?g @g(@g@g@rg@rgg g(ggggint4)rr rrr3iiaf4@)rg|8geg:Kڞ׿gH2퓊cпg}Yu-ÿgQ #(DrgF?g`_?g 0E?gL_߹E?gƶ=?ga@?gкv -?rr3z94-bit AbnormalFloats currently only support blocksize 64.z Typename z not supportedrU)rrNrdiv_absrr)typenamerUr rs r get_4bit_typerhs ~ D5  $ U l V G U  ?"d#D&&&ab b |!IhZ~"FGG << ,DIIdhhjnn ::<2   Kr"c DUc[(dSOSn[XX#USU5$Nrbrr quantize_4bitrrrr compress_statistics quant_storages r quantize_fp4rq *-oB3 C4GP] ^^r"c DUc[(dSOSn[XX#USU5$Nrbrkrrlrns r quantize_nf4ru-rrr"rc Uc[(dSOSnURn[RRR R UUUU5up[XPRS9n U(a:U R5n [X- SS9upA [U UURUU UU U S9nO[U UURUU US9nUbURU5OUnUb URUR5UlX.4$)aQuantize tensor A in blocks of 4-bit values. Quantizes tensor A by dividing it into blocks which are independently quantized. Args: A (`torch.Tensor`): The input tensor. Supports `float16`, `bfloat16`, or `float32` datatypes. absmax (`torch.Tensor`, *optional*): A tensor to use to store the absmax values. out (`torch.Tensor`, *optional*): A tensor to use to store the result. blocksize (`int`, *optional*): The size of the blocks. Defaults to 128 on ROCm and 64 otherwise. Valid values are 64, 128, 256, 512, 1024, 2048, and 4096. compress_statistics (`bool`, *optional*): Whether to additionally quantize the absmax values. Defaults to False. quant_type (`str`, *optional*): The data type to use: `nf4` or `fp4`. Defaults to `fp4`. quant_storage (`torch.dtype`, *optional*): The dtype of the tensor used to store the result. Defaults to `torch.uint8`. Raises: ValueError: Raised when the input data type is not supported. Returns: Tuple[`torch.Tensor`, `QuantState`]: A tuple containing the quantization results. - `torch.Tensor`: The quantized tensor with packed 4-bit values. - [`QuantState`]: The state object used to undo the quantization. rbrkrcr)r )rrmrkr rr rr)rrmrkr rr )r rmrNrJrKrmrMrhrUrNrLrrkrOr)rrrr ror rp input_shaperPrQrrrRrstates rrmrm:sB-oB3 ''KII**88@@  MD HH 5D,W-=M ''!  ''!  !_#))D/$C||ELL1 :r"cBUc[(dSOSn[XX#US5$rjr dequantize_4bitrrrrr s rdequantize_fp4r}&-oB3 16 5 IIr"cBUc[(dSOSn[XX#US5$rtrzr|s rdequantize_nf4rr~r"c @Uc[(dSOSnUc*UbUce[UURURUUS9nO URnUR (a\[ URUR5nX!R- nUR[R:waUR5nUb_[RRRRXUR UR"URURUS9 O`[RRRR%UUUR UR"URUR5nURSS:XaUR'5$U$)aEDequantizes a packed 4-bit quantized tensor. The input tensor is dequantized by dividing it into blocks of `blocksize` values. The the absolute maximum value within these blocks is used for scaling the non-linear dequantization. Args: A (`torch.Tensor`): The quantized input tensor. quant_state ([`QuantState`], *optional*): The quantization state as returned by [`quantize_4bit`]. Required if `absmax` is not provided. absmax (`torch.Tensor`, *optional*): A tensor containing the scaling values. Required if `quant_state` is not provided and ignored otherwise. out (`torch.Tensor`, *optional*): A tensor to use to store the result. blocksize (`int`, *optional*): The size of the blocks. Defaults to 128 on ROCm and 64 otherwise. Valid values are 64, 128, 256, 512, 1024, 2048, and 4096. quant_type (`str`, *optional*): The data type to use: `nf4` or `fp4`. Defaults to `fp4`. Raises: ValueError: Raised when the input data type or blocksize is not supported. Returns: `torch.Tensor`: The dequantized tensor. rbrk)rrmrkr r rTrr )r rrmrkrrrUrrrNrr%rJrKr{rr r rMr6)rrrrr r s rr{r{s`F-oB3 !co55 ))))!  ##%k&8&8+:L:LM$$$ <<5== (\\^F  ..22 {,,k.D.DkFWFWYdYjYjps 3 ii$$44<<   ! !  " "         wwqzQuuw Jr"zDThis function is deprecated and will be removed in a future release.)categorycUcXS[;a*[5RUR5[S'[SnURUR5n[R "U5R 5nUR[R:waUR5nX- n[XAU5nX#U44$)NrF) rIrr&rUrNrfrrkrr%quantize_no_absmax)rrrrinps rquantizers  | I %#5#7#:#:188#DIi #wwqxx  YYq\   F ||u}}$ *C S ,C  r"rxcUcUceUc[UcXS[;a*[5RUR5[S'[SnURUR5nUcX#4n[ XSU5nXAS-$)NrFr r)rIrr&rUdequantize_no_absmax)rrxrrrs r dequantizers   22 2 | I %#5#7#:#:188#DIi #wwqxx  } q(C 0C q>r"c b[U5 Uc#[R"U[RS9n[ X/5 [ R "[U5[U5[U5[R"UR555 SSS5 U$!,(df  U$=f)a Quantizes input tensor to 8-bit. Quantizes the 32-bit input tensor `A` to the 8-bit output tensor `out` using the quantization map `code`. Parameters ---------- A : torch.Tensor The input tensor. code : torch.Tensor The quantization map. out : torch.Tensor, optional The output tensor. Needs to be of type byte. Returns ------- torch.Tensor: Quantized 8-bit tensor. Nr) rfrN zeros_likerrr cquantizerrRrtr)rrrs rrrsz,   ;""1EKK8C1( gdmWQZrxx ?RS  J   Js B B B.c |[U5 Uc#[R"U[RS9n[ XU/5 [ U5n[ R"[U5[U5[U5[R"UR55U5 SSS5 U$!,(df  U$=f)ac Dequantizes the 8-bit tensor to 32-bit. Dequantizes the 8-bit tensor `A` to the 32-bit tensor `out` via the quantization map `code`. Parameters ---------- A : torch.Tensor The 8-bit input tensor. code : torch.Tensor The quantization map. out : torch.Tensor The 32-bit output tensor. Returns ------- torch.Tensor: 32-bit output tensor. Nr) rfrNrrrrr cdequantizerrRrtr)rrrstreams rrr8s,   ;""1EMM:C4C.!#A&  wqz73<!'')ATV\]  J   Js BB,, B;optimizer_namegrstate1beta1epssteplrrbeta2beta3alpha weight_decay gnorm_scale unorm_vec max_unormcSnUS:a.[R"URR55n[ XX8U/5 [R R RUUUUUUUUUU U U UU UUU U5 g)a Performs an inplace optimizer update with one or two optimizer states. Universal optimizer update for 32-bit state and 32/16-bit gradients/weights. Parameters ---------- optimizer_name : str The name of the optimizer: {adam}. g : torch.Tensor Gradient tensor. p : torch.Tensor Parameter tensor. state1 : torch.Tensor Optimizer state 1. beta1 : float Optimizer beta1. eps : float Optimizer epsilon. weight_decay : float Weight decay. step : int Current optimizer step. lr : float The learning rate. state2 : torch.Tensor Optimizer state 2. beta2 : float Optimizer beta2. beta3 : float Optimizer beta3. alpha : float Optimizer alpha. gnorm_scale : float The factor to rescale the gradient to the max clip value. unorm_vec : torch.Tensor The tensor for the update norm. max_unorm : float The maximum update norm relative to the weight norm. skip_zeros : bool Whether to skip zero-valued gradients or not (default: False). rN)rNrrr%rrJrKoptimizer_update_32bit)rrrrrrrrrrrrrrrr skip_zeros param_norms rrrXs|J3ZZ /  qVY/0 II11         %r"zyThis function is deprecated and will be removed in a future release. Please use optimizer_update_8bit_blockwise instead. qmap1qmap2max1max2new_max1new_max2cSnUS:a.[R"URR55n[ U5 [ XX4UXXX/ 5 UR [R:XGaUR [R:XGaa[US"[U5[U5[U5[U5[U5[R"U5[R"U5[R"U5[R"U5[R"U5[R"U5[R"U 5[U 5[U 5[U 5[U 5[U5[U5[R"U5[R"U5[R"UR555 GOUR [R:XGaUR [R:XGa`[US"[U5[U5[U5[U5[U5[R"U5[R"U5[R"U5[R"U5[R"U5[R"U5[R"U 5[U 5[U 5[U 5[U 5[U5[U5[R"U5[R"U5[R"UR555 O%[!SUR SUR 35eSSS5 g!,(df  g=f)a Performs an inplace Adam update. Universal Adam update for 32/8-bit state and 32/16-bit gradients/weights. Uses AdamW formulation if weight decay > 0.0. Parameters ---------- optimizer_name : str The name of the optimizer. Choices {adam, momentum} g : torch.Tensor Gradient tensor. p : torch.Tensor Parameter tensor. state1 : torch.Tensor Adam state 1. state2 : torch.Tensor Adam state 2. beta1 : float Adam beta1. beta2 : float Adam beta2. eps : float Adam epsilon. weight_decay : float Weight decay. step : int Current optimizer step. lr : float The learning rate. qmap1 : torch.Tensor Quantization map for first Adam state. qmap2 : torch.Tensor Quantization map for second Adam state. max1 : torch.Tensor Max value for first Adam state update. max2 : torch.Tensor Max value for second Adam state update. new_max1 : torch.Tensor Max value for the next Adam update of the first state. new_max2 : torch.Tensor Max value for the next Adam update of the second state. gnorm_scale : float The factor to rescale the gradient to the max clip value. unorm_vec : torch.Tensor The tensor for the update norm. max_unorm : float The maximum update norm relative to the weight norm. rrr zAGradient+optimizer bit data type combination not supported: grad z , optimizer N)rNrrr%rfrrkrrstr2optimizer8bitrrRrrrfloat16r)rrrrrrrrrrrrrrrrrrrrrs roptimizer_update_8bitrsZJ3ZZ /  1E$hab 77emm #  (C n -a 0   " 9% :& 5! 5! 3 4  2  !! <( ;' 1779%+ .WW %&,,%++*E n -a 0   " 9% :& 5! 5! 3 4  2  !! <( ;' 1779%+ 0STUT[T[S\\hioiuiuhvw g   s M4N?? O absmax1absmax2c[X!X4XX/5 [RRR UUUUUUUUUU U U U U UUUUU5 gr$)rrNrJrKoptimizer_update_8bit_blockwise)rrrrrrrrrrrrrrrrrrrs rrr:sf*qVU7DE II::           'r"grad gnorm_vec percentilec J[U5 [X/5 UR[R:Xab[ R "[U5[U5[R"U5[R"UR555 OUR[R:Xab[ R"[U5[U5[R"U5[R"UR555 O[SURS35eSSS5 [R"XS-5n[R"U5upV[R"XS5nSnXG:aXt- nXGU4$!,(df  Nh=f)zApplies percentile clipping grad: torch.Tensor The gradient tensor. gnorm_vec: torch.Tensor Vector of gradient norms. 100 elements expected. step: int The current optimization steps (number of past gradient norms). zGradient type z not supported!NrZr)rfrrkrNrrcpercentile_clipping_g32rrRrrrcpercentile_clipping_g16rsqrtr) rrrr current_gnormvalsr clip_valuers rpercentile_clippingrhs-  4#$ :: &  ( (  " 4  4::<(   ZZ5== (  ( (  " 4  4::<(  ~djj\IJ J# &JJy45M 9%IDD,-JK! 0 k 117  s D&F F"c  [RR5(d[RR5 URU:wdURU:wa%[ SURSUR35eUR nUR nUnUn Sn [U5S:Xa[U5S:XaU(d+U (d$UR SUR S:waSn GO\U(a+U (d$UR SUR S:waSn GO*U(a+U (a$UR SUR S:waSn GOU(d)U (a"UR SUR S:waSn GO[U5S:Xa[U5S:XaU(d+U (d$UR SUR S:waSn GOvU(a+U (d$UR SUR S:waSn GODU(a+U (a$UR SUR S:waSn GOU(d)U (a"UR SUR S:waSn O[U5S:Xa[U5S:XaU(d*U (d#UR SUR S:waSn OU(a*U (d#UR SUR S:waSn OaU(a*U (a#UR SUR S:waSn O0U(d)U (a"UR SUR S:waSn UbeUR n U (dP[U5S:XaA[U5S:Xa2U SUS:Xa&U SUS:XaUSUS:XaUSUS:XaSn GO[U5S:Xaw[U5S:XahU(dU (d USUS4n GOqU(aU (a USUS4n GOWU(aU (d USUS4n GO=U(dU (a USUS4n GO#[U5S:Xa[U5S:XatU(dU (dUSUSUS4n OU(aU (aUSUSUS4n OU(aU (dUSUSUS4n OU(dU (aUSUSUS4n O[U5S:Xa[U5S:XasU(dU (dUSUSUS4n OVU(aU (aUSUSUS4n O9U(aU (dUSUSUS4n OU(dU (aUSUSUS4n U (d[S US US US U S 3 5eW $) Nz3Expected torch.int8 input tensors A and B, but got  and Trr rFr\z?Tensor dimensions incorrect for matrix mulitiplication: A x B:  x z with transpose for A x B: r) rNrOis_initializedinitrk TypeErrorrmrr) rrr transposed_A transposed_B expected_typesAsBtAtBcorrectsouts r check_matmulrsz :: $ $ & & ww-177m#;MaggYV[\]\c\c[deff B B B BG 2w!|B1 "qwwqz!9G qwwqzQWWQZ7G B1771:3GqwwqzQWWQZ7G RA#b'Q,"qwwqz!9G qwwqzQWWQZ7G B1771:3GqwwqzQWWQZ7G RA#b'Q,"qwwqz!9G qwwqzQWWQZ7G B1771:3GqwwqzQWWQZ7G yy3r7aio supported for tensor contractions, but dims for A x B were: r)rrNzerosint32rUrrm batched_igemmstriderrBr/rTrrcigemmrRc_boolrr)rrrrrrrrnldbmr(ldaldcptrs rigemmr s c >D {kkt5;;qxxH 177|qS\Q. 771: # aggaj(@ s+ + B BB1 eRU^ #b'Q,eRUBqE "B1 eRU^ #b'Q,eRUBqE " 2w!| 88:a=AGGAJ & L XXZ]aggaj (L qww<1 xxz!} *$ A!''!*,# xxz!} *$ A!''!*,# r7a<1A((*,QA6C W\c"gl11 AQ%C qE qEhhj|!4e RA2w!||1A2a5BqE>_`b_ccfgifjk   qE qE qEBqEMe  % % ' 3 3AHH =CqSkJJ  , , 1  1  1     3 3 3  Jr"c [UR5S:Xa[UR5S:Xd%[SURSUR35e[XX#U5nUc.[R "U[R URS9nUR5(aUR5SnSnOUR5nUSURS:wa$UR5nUR5SnOUSURS:XaS nUR5SnO}USS:Xa$UR5nUR5SnOPUSS:Xa$UR5nUR5SnO#UR5nUR5SnUR5(aUR5SnSnOUR5nUSURS:wa&UR5nUR5SnSnOQUSURS:XaUR5SnS nO%UR5nUR5SnSnURSn URSn URSn URSn U n URSURS-nURSURS-nURSURS-n[R5RUR5n[XU/5 [R "U["R$"U5["R$"U5["R&"U 5["R&"U 5["R&"U 5[)U5[)U5[)U5["R&"U5["R&"U5["R&"U 5["R*"U5["R*"U5["R*"U5["R,"U 55 U$) Nr\z@Expected 3-dimensional tensors for bmm, but got shapes A and B: rrr FrrT)rrmrrrNrrrU is_contiguousr contiguousrBr/rTrrcbatched_igemmrRrrrc_longc_uint32)rrrrrrrsr num_batchrrr(rstrideAstrideBstrideCrs rrrpso qww<1 CLA$5[\]\c\c[ddijkjqjqirstt c >D {kkt5;;qxxHhhjm HHJ Q41771:  A((*Q-C qTQWWQZ L((*Q-CtqyLLNhhjm1LLNhhjmLLNhhjmhhjm HHJ Q41771:  A((*Q-C L qTQWWQZ ((*Q-CL A((*Q-C L I  A  A  A Cggaj1771:%Gggaj1771:%Gggaj1771:%G  % % ' 3 3AHH =C qSk  , , 1  1  1     3 3 3 ' ' ' I!$ Jr"cUb6[RRRR XU5 U$[RRRR X5$)aPerforms an 8-bit integer matrix multiplication. A linear transformation is applied such that `out = A @ B.T`. When possible, integer tensor core hardware is utilized to accelerate the operation. Args: A (`torch.Tensor`): The first matrix operand with the data type `torch.int8`. B (`torch.Tensor`): The second matrix operand with the data type `torch.int8`. out (`torch.Tensor`, *optional*): A pre-allocated tensor used to store the result. dtype (`torch.dtype`, *optional*): The expected data type of the output. Defaults to `torch.int32`. Raises: `NotImplementedError`: The operation is not supported in the current environment. `RuntimeError`: Raised when the cannot be completed for any other reason. Returns: `torch.Tensor`: The result of the operation. )rNrJrKint8_linear_matmulrrM)rrrrks rrrsO&  1155aC@ 99 ! ! 4 4 < This function is useful for training, but for inference it is advised to use [`get_row_absmax`] instead. The column-wise quantization scales are not typically needed in inference scenarios. Args: A (`torch.Tensor` with dtype `torch.float16`): Input tensor. row_stats (`torch.Tensor`, *optional*): If provided, calculation of row statistics is skipped. col_stats (`torch.Tensor`, *optional*): If provided, calculation of column statistics is skipped. nnz_block_ptr (`torch.Tensor`, *optional*): Not used. threshold (`float`, *optional*): An optional threshold for sparse decomposition of outlier features. No outliers are held back when 0.0. Defaults to 0.0. Returns: `Tuple[torch.Tensor, torch.Tensor, Optional[torch.Tensor]]`: A tuple containing quantization statistics. - `torch.Tensor` with dtype `torch.float32`: The row-wise quantization statistics. - `torch.Tensor` with dtype `torch.float32`: The column-wise quantization statistics. - `torch.Tensor` with dtype `torch.bool`, *optional*: A mask indicating the locations of outliers in the input tensor. Nr3rrF)dimkeepdim)is_floating_pointrfryrm masked_fill_get_row_absmaxamaxr%)rrrr threshold outlier_maskabsAs rget_colrow_absmaxrsD    LI-uuw||B , s?,L   lC 0  'q4I   a 7==?I  --r"c :UR[R:Xde[URSS5nURSn[R "U4[R URS9n[U/5 [U5 [R"[U5[U5[R"U5[R"U5[R"U5[!U55 SSS5 U$!,(df  U$=f)a0Determine the quantization statistics for input matrix `A` in accordance to the `LLM.int8()` algorithm. For more information, see the [LLM.int8() paper](https://arxiv.org/abs/2208.07339). Args: A (`torch.Tensor` with dtype `torch.float16`): The input matrix. threshold (`float`, *optional*): An optional threshold for sparse decomposition of outlier features. No outliers are held back when 0.0. Defaults to 0.0. Returns: `torch.Tensor` with dtype `torch.float32`: The absolute maximum value for each row, with outliers ignored. Nr3rj)rkrNrrrmemptyrrUrrfrcget_row_statsrrRrrr)rrrowscolsrs rrr@s 77emm ## #  D 772;D TG5==JI qcN    AJ I  JJy ! JJt  JJt  q !       s A3D  Dc r\rSrSrS\S\S\S\R S\R S\R 4 SjrS rg ) COOSparseTensorifrrnnzrowidxcolidxrcUR[R:XdeUR[R:XdeUR[R:XdeUR 5U:XdeUR 5U:XdeUR 5U:XdeXlX lX0lX@lXPl X`l gr$) rkrNrrrrrrrrr)rrrrrrrs rrCOOSparseTensor.__init__gs||u{{***||u{{***||u}},,,||~$$$||~$$$||~$$$     r")rrrrrrN) r:r;r<r=rrNrrr?r@r"rrrfsE"),6;llLQLLbgbnbnr"rc\rSrSrSrSrg)CSRSparseTensoriycUR[R:XdeUR[R:XdeUR[R:XdeUR 5U:XdeUR 5U:XdeUR 5US-:XdeXlX lX0lX@lXPl X`l gNr ) rkrNrrrrrrrowptrrr)rrrrr rrs rrCSRSparseTensor.__init__z||u{{***||u{{***||u}},,,||~$$$||~$$$||~)))     r")rrrr rrNr:r;r<r=rr?r@r"rrry r"rc\rSrSrSrSrg)CSCSparseTensoricUR[R:XdeUR[R:XdeUR[R:XdeUR 5U:XdeUR 5U:XdeUR 5US-:XdeXlX lX0lX@lXPl X`l gr ) rkrNrrrrrrcolptrrr)rrrrrrrs rrCSCSparseTensor.__init__r r")rrrrrrNrr@r"rrrrr"rc[R"URSS9upURS5 [R"UR S-4[R URRS9nURUR5UR5SS9 URS5 [UR URURX0RUR 5$NT return_countsr rjr)rMsrcr)rNuniqueradd_rrrrUscatter_longrcumsum_rrrrr)cooArcountsr s rcoo2csrr!s\\$++TBNF KKN [[$))a-)T[[EWEW XF OO&++-VZZ\qOA NN1 499dii6;;PTP[P[ \\r"c4[R"UR5upURUnURUn[R "USS9upVUR S5 [R"URS-4[RURRS9nURUR5UR5SS9 URS5 [UR URUR"XsU5$r)rNrrrrrrrrrrUrrrrrrr)rr col2rowidxrr colvaluesr rs rcoo2cscr%sjj-OC [[ $F [[ $F S=I NN1 [[$))a-)T[[EWEW XF OO)..* !OD NN1 499dii66 RRr"c[R"U4[RUS9n[R"U4[RUS9n[R"U4XCS9n[XX%Xg5$)Nrj)rNrrr)rrrrUrkrrrs r coo_zerosr'sS [[#u{{6 BF [[#u{{6 BF [[#u This function is useful for training, but for inference it is advised to use [`int8_vectorwise_quant`] instead. This implementation performs additional column-wise transposed calculations which are not optimized. Args: A (`torch.Tensor` with dtype `torch.float16`): The input matrix. col_stats (`torch.Tensor`, *optional*): A pre-allocated tensor to hold the column-wise quantization scales. row_stats (`torch.Tensor`, *optional*): A pre-allocated tensor to hold the row-wise quantization scales. out_col (`torch.Tensor`, *optional*): A pre-allocated tensor to hold the column-wise quantized data. out_row (`torch.Tensor`, *optional*): A pre-allocated tensor to hold the row-wise quantized data. threshold (`float`, *optional*): An optional threshold for sparse decomposition of outlier features. No outliers are held back when 0.0. Defaults to 0.0. Returns: `Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, Optional[torch.Tensor]]`: A tuple containing the quantized tensor and relevant statistics. - `torch.Tensor` with dtype `torch.int8`: The row-wise quantized data. - `torch.Tensor` with dtype `torch.int8`: The column-wise quantized data. - `torch.Tensor` with dtype `torch.float32`: The row-wise quantization scales. - `torch.Tensor` with dtype `torch.float32`: The column-wise quantization scales. - `torch.Tensor` with dtype `torch.int32`, *optional*: A list of column indices which contain outlier features. zUrow_stats must be None. int8_double_quant() does not support pre-allocated row_stats.zUcol_stats must be None. int8_double_quant() does not support pre-allocated col_stats.zQout_col must be None. int8_double_quant() does not support pre-allocated out_col.zQout_row must be None. int8_double_quant() does not support pre-allocated out_row.)r)rrNrJrKint8_double_quantrM)rrrr(r)rs rr+r+sqNpqqpqqlmmlmm 99 ! ! 3 3 ; ;A ; SSr"statsch[RRRR X5$)aADequantizes a tensor with dtype `torch.int8` to `torch.float32`. Args: A (`torch.Tensor` with dtype `torch.int8`): The quantized int8 tensor. stats (`torch.Tensor` with dtype `torch.float32`): The row-wise quantization statistics. Returns: `torch.Tensor` with dtype `torch.float32`: The dequantized tensor. )rNrJrKint8_vectorwise_dequantrM)rr,s rr.r.s% 99 ! ! 9 9 A A! KKr"ch[RRRR X5$)aGQuantizes a tensor with dtype `torch.float16` to `torch.int8` in accordance to the `LLM.int8()` algorithm. For more information, see the [LLM.int8() paper](https://arxiv.org/abs/2208.07339). Args: A (`torch.Tensor` with dtype `torch.float16`): The input tensor. threshold (`float`, *optional*): An optional threshold for sparse decomposition of outlier features. No outliers are held back when 0.0. Defaults to 0.0. Returns: `Tuple[torch.Tensor, torch.Tensor, Optional[torch.Tensor]]`: A tuple containing the quantized tensor and relevant statistics. - `torch.Tensor` with dtype `torch.int8`: The quantized data. - `torch.Tensor` with dtype `torch.float32`: The quantization scales. - `torch.Tensor` with dtype `torch.int32`, *optional*: A list of column indices which contain outlier features. )rNrJrKint8_vectorwise_quantrM)rrs rr0r0s%$ 99 ! ! 7 7 ? ? MMr"rc[U[5(dUR(aUR[R :XdS5e[UR SUR SUR5UR5SR5UR5SR5UR5S9nUcC[R"URUR S4URURS9nURnUR R#5U:XdeUR$R#5U:XdeURR#5U:XdeUR&UR S:XdeUR)5(+nUR+5U(aSOSnUR Sn[,R/5R0n[3UR 5n[3UR$5n [3UR5n [3U5n [3U5n [4R6"UR5n [4R6"UR5n[4R6"UR&5n[4R6"UR S5n[4R6"U5n[4R6"U5n[9UR UR$URX/5 [:R<"UUU U U UUUUU UU [4R>"U55 U$)Nz8Tensor must be `COOSparseTensor or a PyTorch COO tensor.rr )rrrrrrrUrk) rr is_sparselayoutrN sparse_coorm_nnzindicesrrrrrUrkrrrrrrrrYr/rHrrRrrr cspmm_coor)rrrrrrrr ptrRowidx ptrColidx ptrValuesptrBptrCcnnzcrowsAccolsAccolsBcldbcldcs rspmm_coorD s dO , ,~~$++1A1A"A F A AA <<>!$((*<<>!$((*;;=   {kk499aggaj1!((!''R ((C ;;   # %% % ;;   # %% % ;;   # %% % 99 "" "((L ((*MM       , Jr"cxUcM[R"URURS4URUR R S9nURnURR5U:XdeURR5U:XdeUR R5U:XdeURURS:Xd URSUR35e[R"URSS9upVURS5R5n[R"USS9upU R5n UR5nUSS::dS USS 35eUR [R [R"4;de[%U5n [%U5n [%U 5n [%UR5n [%UR5n[%UR 5n[%U5n[%U5n[%U5n[&R("UR55n[&R("UR5n[&R("UR5n[&R("URS5n[&R("URS5n[+U5 [-URURUR XU/5 UR [R :Xa$[.R0"U U U U UUUUUUUUUU5 OAUR [R":Xa#[.R2"U U U U UUUUUUUUUU5 SSS5 U$!,(df  U$=f) Nr r2rz vs Tr) descending z)Current max count per row is 8 but found r)rNrrrmrUrrkrrrrrrcumsumrrrint8rrRrrfrr cspmm_coo_very_sparse_naive_fp16 cspmm_coo_very_sparse_naive_int8)rr dequant_statsrr_r r max_countmax_idx ptrOffset ptrMaxCount ptrMaxIdxr9r:r;r<r=ptrDequantStats cnnz_rowsr>r?crowsBrAs rspmm_coo_very_sparserVPs {kk499aggaj1!(($++J[J[\ ((C ;;   # %% % ;;   # %% % ;;   # %% % 99 "?tyykaggY$?? " T[[=IA ]]1  ! ! #FFt<IkkmG I Q<2 Z!J9UV<.XYZZ  77u}}ejj1 11 1I)$K I $I $I $I 1:D 3*I ::dhh D ZZ "F ZZ #F ZZ #F  4;; T[[!-PQ 77emm #  0 0  WW "  0 0 ) L JM  L Js .B2N** N9g_@r9)T)NTr$)TrT)g+ew?T)TrZrr)TrXr)NNNF)NNNNrWF)Nrb)NNNN)NNNNrr) NrrrrrNrF)rrNr)rrF)rZ)NFFN)NFF)NNNr)r)NNNNr)gcollections.abcrctypesrRrmathrtypingrrrnumpyrurNrtyping_extensionsr bitsandbytes.utilsr r cextensionr rrIcadam_static_8bit_grad_32cadam_static_8bit_grad_16cmomentum_static_8bit_grad_32cmomentum_static_8bit_grad_16crmsprop_static_8bit_grad_32crmsprop_static_8bit_grad_16clion_static_8bit_grad_32clion_static_8bit_grad_16rrrBrYrUFIRST_CUDA_DEVICErO device_countrfrhrrr4rrrrrrrrrSrrrr0rLrrUrhrrqrurmr}rr{ FutureWarningrrrrr.r%rrrrrIrrrrrrrrrrrrr!r%halfr'r+r.r0rDrVCr@r"rrmsu  %'' (I,   %% %%  )) )) (( (( %% %%  %% %%  )) )) +8''0**2"LLq1  ::q '5<<'(5<<("MM2CKu||K!6-&S&:'T43n!x 67!HPvP"++P %x %HR[[$9 %n n f$(%)"&  D ||D 5<< D U\\ "D %,,  D 5<< #$DR)-%)#'"& J ||J*%J U\\ "J 5<< J %,,  J  J \\JZOh&*"&++ _ || _ U\\ " _ %,,  _&*"&++ _ || _ U\\ " _ %,,  _&*"&++N ||N U\\ "N %,, N 5<< #$Nf)-%)"&# J || J*% J U\\ " J %,,  J } J  \\ J)-%)"&# J || J*% J U\\ " J %,,  J } J  \\ J)-%)"&# J ||J*%J U\\ "J %,,  J } J \\JZ R]jk$("&  5<<  %,,  65( () l& R]jk.2%)#'"&   E&&.) * U\\ " 5<<  %,,    l( R]jk&Xell5KW]l< R]jkF&x 7MY_lP&*(,#VV VV  V  V  V V V U\\ "V V V VVV %V !V$ %Vr ; ,(,)AA AA  A U\\ " A  A A A A A A ELL !A A 5<< AA u||$!A"#A$%A& %'A()A* +A  Aj'++ ++  + U\\ " +  + + + + + + + + ELL !++ ell #!+"#+$%+( )+\ R]jk&2f&2&2s&2PS&2l&2RGLjjQn#'  " " " %,, "P#' b b b %,, bP#' ] ] ] %,, ]@X\chcncnC%,,C5<<Chu||>TC<#'#'  |||||| %,,   5<< 8 R]jk)-(,,0 6. ||6. %6. %6.ELL) 6.  5<<x '= => 6.l6.r R]jk"ell"l"J&""] S.3ZZD)-(,&*&* 0T ||0T %0T %0Tell # 0T ell # 0Tf Lu|| LELL LNU\\N0#'@ - .@ ||@ %,, @FGT r"