;iSSKrSSKJrJrJrJrJr SSKrSSKrSSKJ r J r J r J r SSK J s Jr SSKrSSKJr SSKJr SSKJr SSKJrJr \"SS S 9r"S S \RR85r"S S\RR85r"SS\RR<5rS\S4Sjr "SS\ RB5r""SS\"5r#"SS\"5r$"SS\RR<5r%Sr&"SS\ R85r'"SS \ R85r("S!S"\(5r)"S#S$\(5r*"S%S&\ RB5r+"S'S(\ RB5r,"S)S*\ RB5r-g)+N)AnyOptionalTypeVarUnionoverload)Tensordevicedtypenn)HIP_ENVIRONMENT) QuantState)GlobalOptimManager)*INVERSE_LINEAR_8BIT_WEIGHTS_FORMAT_MAPPING OutlierTracerTztorch.nn.Module)boundc^\rSrSrSrSS\S\S\\S\\S\S \S \S \\ S S4U4S jjjr SSjr SSjr S\ S \ 4Sjr SrU=r$)StableEmbeddinga Custom embedding layer designed to improve stability during training for NLP tasks by using 32-bit optimizer states. It is designed to reduce gradient variations that can result from quantization. This embedding layer is initialized with Xavier uniform initialization followed by layer normalization. Example: ``` # Initialize StableEmbedding layer with vocabulary size 1000, embedding dimension 300 embedding_layer = StableEmbedding(num_embeddings=1000, embedding_dim=300) # Reset embedding parameters embedding_layer.reset_parameters() # Perform a forward pass with input tensor input_tensor = torch.tensor([1, 2, 3]) output_embedding = embedding_layer(input_tensor) ``` Attributes: norm (`torch.nn.LayerNorm`): Layer normalization applied after the embedding. Methods: reset_parameters(): Reset embedding parameters using Xavier uniform initialization. forward(input: Tensor) -> Tensor: Forward pass through the stable embedding layer. Nnum_embeddings embedding_dim padding_idxmax_norm norm_typescale_grad_by_freqsparse_weightreturnc >[T U]UUUUUUUUU U 5 [RR X)S9Ul[ R"5RUSSS05 ga Args: num_embeddings (`int`): The number of unique embeddings (vocabulary size). embedding_dim (`int`): The dimensionality of the embedding. padding_idx (`Optional[int]`): Pads the output with zeros at the given index. max_norm (`Optional[float]`): Renormalizes embeddings to have a maximum L2 norm. norm_type (`float`, defaults to `2.0`): The p-norm to compute for the `max_norm` option. scale_grad_by_freq (`bool`, defaults to `False`): Scale gradient by frequency during backpropagation. sparse (`bool`, defaults to `False`): Computes dense gradients. Set to `True` to compute sparse gradients instead. _weight (`Optional[Tensor]`): Pretrained embeddings. r weight optim_bits N) super__init__torchr LayerNormnormr get_instanceregister_module_override) selfrrrrrrrrr r __class__s g/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/bitsandbytes/nn/modules.pyr&StableEmbedding.__init__0sn@            HH&&}&D '')BB4T`bdSefc[RRRUR5 UR 5 gNr'r initxavier_uniform_r"_fill_padding_idx_with_zeror,s r.reset_parameters StableEmbedding.reset_parameters_(  %%dkk2 ((*r0cURbG[R"5 URURR S5 SSS5 gg!,(df  g=fNrrr'no_gradr"fill_r7s r.r6+StableEmbedding._fill_padding_idx_with_zerojF    ' D,,-33A6! (  )A A$inputc b[R"UURURURUR UR UR5nUR[R"55nURU5RURR5$r2) F embeddingr"rrrrrtor'get_default_dtyper)r r,rCembs r.forwardStableEmbedding.forwardos}kk  KK    MM NN  # # KK ffU,,./yy~  !2!233r0)r))NN@FFNNNrN)__name__ __module__ __qualname____firstlineno____doc__intrfloatboolrr&r8r6rK__static_attributes__ __classcell__r-s@r.rrs:&*$(#($(-g-g-gc] -g 5/ -g  -g!-g-g&!-g -g-g^+7 4V444r0rc^\rSrSrSrSS\S\S\\S\\S\S \S \S \\ S \\ S S4U4Sjjjr SSjr SSjr S\ S \ 4SjrSrU=r$) EmbeddingzK Embedding class to store and retrieve word embeddings from their indices. Nrrrrrrrrr rc >[T U]UUUUUUUUU S9 [R"5R USSS05 gr )r%r&rr*r+) r,rrrrrrrrr r-s r.r&Embedding.__init__sV>            '')BB4T`bdSefr0c[RRRUR5 UR 5 gr2r3r7s r.r8Embedding.reset_parametersr:r0cURbG[R"5 URURR S5 SSS5 gg!,(df  g=fr<r=r7s r.r6%Embedding._fill_padding_idx_with_zerorArBrCc [R"UURURURUR UR UR5nU$r2)rErFr"rrrrrrIs r.rKEmbedding.forwardsHkk  KK    MM NN  # # KK  r0)NNrMFFNNrN)rOrPrQrRrSrTrrUrVrr r&r8r6rKrWrXrYs@r.r[r[s&*$(#($(#'*g*g*gc] *g 5/ *g  *g!*g*g&!*g *g *g*gX+7 V   r0r[cP^\rSrSrSSSSSS\R SS4 S\\RS\\S\\ S \ S \ S \RS \S S\ SS4Sjjr SrSrSrSr\S'S\RS\\ \4S\ S \S SS4 Sjj5rSrSrS(S\\\ \\ 4S\ 4SjjrS(S\\\ \\ 4S\ 4Sjjr\S)S\S\\\ \4S\\\ \ 4S\ S\4 S jj5r\S*S\S\\ \ 4S\ S\4S!jj5r\S*S\S"\S\ S\4S#jj5rU4S$jr\S+U4S%jj5rS&r U=r!$), Params4bitNFTfp4data quant_state blocksizecompress_statistics quant_type quant_storagemodule Linear4bit bnb_quantizedrc Uc[R"S5nUc[(dSOSn[RR XU5n XJlXZlXjlX:lXzl Xl Xl Xl U $)Nr@r\) r'emptyr r_make_subclassrlrmrnrkrorrrjrp) clsrj requires_gradrkrlrmrnrorprrr,s r.__new__Params4bit.__new__sn <;;q>D  "1/sI||**3mD"#6 $&**   r0cvURR5nURUS'URUS'U$)Nrjrx)__dict__copyrjrxr,states r. __getstate__Params4bit.__getstate__s6 ""$ f !%!3!3o r0cUSUlUSUlUSUlUSUlUSUlUSUlUSUlUSUlUS Ulg) Nrxrlrmrnrkrjrorrrp) rxrlrmrnrkrjrorrrpr~s r. __setstate__Params4bit.__setstate__sr"?3{+#()>#?  - /&M "?3"?3Ho r0c[U5R[U55nUR5nURU5 [R "US5Ul[R "US5UlU$)Nrkrj)typeryrrr}deepcopyrkrj)r,memo new_instancers r. __deepcopy__Params4bit.__deepcopy__sgDz))$t*5 !!#!!%(#'==}1E#F   MM%-8 r0c[U5R[U55nUR5nURU5 U$r2)rryrr)r,rrs r.__copy__Params4bit.__copy__s<Dz))$t*5 !!#!!%(r0quantized_statsrxc [RRXRU55nX7l[ R "X$S9UlURRUlURRUl URRUl SUl URUlXWlURbURURlU$)N)qs_dictr T)r'rrvrGrxr from_dictrkrlnestedrmrnrrr rorp)rwrjrrxr rpkwargsr,s r.from_prequantizedParams4bit.from_prequantized s||**3@*%//W))33#'#3#3#:#: **55!!ZZ ;; "&*&6&6DKK # r0cNURR5RU5n[RR UUR URURURS9up4X0lX@l URbX@Rl SUl U$)N)rlrmrnroT) rj contiguousrGbnb functional quantize_4bitrlrmrnrorkrprr)r,r ww_4bitrks r. _quantizeParams4bit._quantize&s II " % %f -!nn:: nn $ 8 8,, ;  & ;; "&1KK #! r0c URSS9$Ncpur!rGr7s r.rParams4bit.cpu6wwew$$r0r non_blockingc2URUcSUS9$UUS9$Ncudar rrr,r rs r.rParams4bit.cuda9'wwfQ]w^^FQ]w^^r0c2URUcSUS9$UUS9$Nxpurrrs r.rParams4bit.xpu<'wwv~eP\w]]6P\w]]r0r,r cgr2rer,r r rs r.rG Params4bit.to? r0cgr2rer,r rs r.rGrGNQr0tensorcgr2rer,rrs r.rGrJDGr0c >[RRR"U0UD6up4pVUb2URS:wa"UR (dUR U5$URbURRU5 [[TU]!X4US9URURURURURURUR S9nU$)Nmetar r r)rxrkrlrmrnrorr)r'_C_nn _parse_torrrrrkrGrgr%rxrlrmrnro) r,argsrr r rconvert_to_format new_paramr-s r.rGrMs9>9O9OQU9`Y_9`6|  &++"7@R@R>>&) )+  ##F+" &L Q"00 ,,..$($<$<??"00"00 I r0c >^^Uc0nU[R[R4;aUSm[TT]XX45n[ U[ 5(a[ UU4SjU55$T"UTRTRTRTRTRTRTRTRS9 $[TT]XX45$)Nrc 3># UHcnT"UTRTRTRTRTRTR TR TRS9 v Me g7f) rjrxrkrlrmrnrorprrN)rxrkrlrmrnrorprr).0chunkrwrs r. 0Params4bit.__torch_function__..nsj "("&,&:&:$*$6$6"("2"2,2,F,F#)#4#4&,&:&:%}}&,&:&: "(sA+A.r)r'rsplitr%__torch_function__ isinstancetuplerxrkrlrmrnrorprr)rwfunctypesrrresultrr-s` @r.rParams4bit.__torch_function__cs >F EKK- -!WFW/TJF&%(( "(   "("6"6 & 2 2$..(.(B(B%00"("6"6!=="("6"6  w)$tDDr0) rlrrrmrjrprkrornrx)FrNNF....)reN)"rOrPrQrRr'uint8rrr rTrVstrr ryrrrr classmethoddictrrrrrr rrrrrGrrWrXrYs@r.rgrgs=(,,0#'$(%*[[)-#u||$j)  C=  " {{& : &  $)- llc3h  &  2 %_8E#vs*:$;<_SW_^(5fc)9#:;^RV^03-0 sF{+,eSj)*  QQ5,QDQ1QQ GGFG$GGG,%E%Er0rgrp) Embedding4bitrqcx[URSS5bg[USS5c[R"S5 URRSS:Xde[ UR[ 5(d$[ URURSS9UlURURlg)NrkzhFP4 quantization state not initialized. Please call .cuda() or .to(device) on the LinearFP4 layer first.T)rorr) getattrr"warningswarnshaperrgrork)rps r.'fix_4bit_weight_quant_state_from_modulersv}}mT2>v}d+3 v ==  q !Q && & fmmZ 0 0"6==@T@Tdhi & 2 2FMMr0c^\rSrSrSrSSSS\R S4U4SjjrSrU4Sjr S \R4S jr S r U=r $) rqiaq This class is the base module for the 4-bit quantization algorithm presented in [QLoRA](https://arxiv.org/abs/2305.14314). QLoRA 4-bit linear layers uses blockwise k-bit quantization under the hood, with the possibility of selecting various compute datatypes such as FP4 and NF4. In order to quantize a linear layer one should first load the original fp16 / bf16 weights into the Linear4bit module, then call `quantized_module.to("cuda")` to quantize the fp16 / bf16 weights. Example: ```python import torch import torch.nn as nn import bitsandbytes as bnb from bnb.nn import Linear4bit fp16_model = nn.Sequential( nn.Linear(64, 64), nn.Linear(64, 64) ) quantized_model = nn.Sequential( Linear4bit(64, 64), Linear4bit(64, 64) ) quantized_model.load_state_dict(fp16_model.state_dict()) quantized_model = quantized_model.to(0) # Quantization happens here ``` TNric >[T U]XX85 [URRSUUUUS9UlX@lUSLUlSUlXplg)a/ Initialize Linear4bit class. Args: input_features (`str`): Number of input features of the linear layer. output_features (`str`): Number of output features of the linear layer. bias (`bool`, defaults to `True`): Whether the linear class uses the bias term as well. FrxrmrnrorpN) r%r&rgr"rj compute_dtypecompute_type_is_setrkro) r,input_featuresoutput_featuresbiasrrmrnror r-s r.r&Linear4bit.__init__s_, $G KK   3!'   +#0#< *r0cvUR[R[R4;aURUlgUR[R :XaURS[R4;aLUR 5URS:Xa+[R"S5 [R"SSS9 URS[R4;aNUR 5URS:wa,[R"S5 [R"SSS9 gggg)NzInput type into Linear4bit is torch.float16, but bnb_4bit_compute_dtype=torch.float32 (default). This will lead to slow inference.ignorez .*inference.)messagezInput type into Linear4bit is torch.float16, but bnb_4bit_compute_dtype=torch.float32 (default). This will lead to slow inference or training speed.z.*inference or training) r r'float32bfloat16rfloat16numelrrrfilterwarningsr,xs r.set_compute_typeLinear4bit.set_compute_types 77u}}enn5 5"#D  WW %!!dEMM%:: QWWUW[@X Y''.I!!dEMM%:: QWWUW[@X k'':ST AY:&r0c>[TU]XU5 [URSS5b\URRR SS9R 5H&upEU(aUOUR5XS-U-'M( gg)zK save weight and bias, then fill state_dict with components of quant_state rkNT)packedzweight.)r%_save_to_state_dictrr"rkas_dictitemsdetach)r, destinationprefix keep_varskvr-s r.rLinear4bit._save_to_state_dictsw #KC 4;; t 4 @ //77t7DJJL;Da!((* Y.23M Ar0rc[U5 URbbURRUR:wa>URRR UR5URlUR (dUR U5 SUlURnURbUR UR5nURcSO$URR UR5nURR5n[R"XX0RRS9R U5$)NT)rrk) rrr rjrGrrrr"tr matmul_4bitrk)r,r inp_dtyperr"s r.rKLinear4bit.forwards/5 99 TYY__%?!YY^^..qww7DIIN''  ! !! $'+D $GG    )T''(Ayy(tdiill4;M;M.NqtAXAXY\\]fggr0)rrrkror")rOrPrQrRrSr'rr&rrrrKrWrXrYs@r.rqrqsKH kk#+JU( Uhhhr0rqcN^\rSrSrSrSSS\R S4U4SjjrSrU=r $) LinearFP4iz Implements the FP4 data type. TNc 2>[TU]UUUUUSUU5 g) Args: input_features (`str`): Number of input features of the linear layer. output_features (`str`): Number of output features of the linear layer. bias (`bool`, defaults to `True`): Whether the linear class uses the bias term as well. riNr%r& r,rrrrrmror r-s r.r&LinearFP4.__init__*&          r0re rOrPrQrRrSr'rr&rWrXrYs@r.r r s' kk  r0r cN^\rSrSrSrSSS\R S4U4SjjrSrU=r $) LinearNF4i;a Implements the NF4 data type. Constructs a quantization data type where each bin has equal area under a standard normal distribution N(0, 1) that is normalized into the range [-1, 1]. For more information read the paper: QLoRA: Efficient Finetuning of Quantized LLMs (https://arxiv.org/abs/2305.14314) Implementation of the NF4 data type in bitsandbytes can be found in the `create_normal_map` function in the `functional.py` file: https://github.com/TimDettmers/bitsandbytes/blob/main/bitsandbytes/functional.py#L236. TNc 2>[TU]UUUUUSUU5 g)rnf4Nrrs r.r&LinearNF4.__init__Grr0rerrYs@r.rr;s'  kk  r0rc ^\rSrSrSS\\R S\\R S\\R 4SjjrU4SjrSr SS\\ \ \ \ 4S \4S jjrSS\\ \ \ \ 4S \4S jjrS r\SS \S\\ \ \ 4S\\ \\ 4S \S\4 Sjj5r\SS \S\ \\ 4S \S\4Sjj5r\SS \S\S \S\4Sjj5rU4SjrSrU=r$) Int8ParamsifrjCBSCBcUc[R"S5n[RRXU5nXFlXVlX6lU$r<)r'rurrvrrhas_fp16_weights)rwrjrxrrrobjs r.ryInt8Params.__new__gsA <;;q>Dll))#]C/ r0c>UR(a[TU] U5$URR 5RU[ R S9n[RRU5up4nX0lX0l X@l U$)Nr r ) rr%rGrjrr'rrrint8_vectorwise_quantrr)r,r Brr_r-s r.rInt8Params._quantizewso  7:f% % II " % %V5== % I^^99!<   r0c URSS9$rrr7s r.rInt8Params.cpurr0r rc2URUcSUS9$UUS9$rrrs r.rInt8Params.cudarr0c2URUcSUS9$UUS9$rrrs r.rInt8Params.xpurr0c 4[U5R[U5[R"URU5UR UR [R"URU5[R"URU5S9nU$)N)rjrxrrr) rryr}rrjrxrrr)r,rrs r.rInt8Params.__deepcopy__srDz)) Jtyy$/,,!22}}TWWd+ dhh- * r0r,r rcgr2rers r.rG Int8Params.torr0cgr2rers r.rGr1rr0rcgr2rers r.rGr1rr0c|>[RRR"U0UD6up4pVURR [R :HnU(dHUbEURS:wa5URRRS:XaURU5$[[T U]1X4US9URURS9nU(aeURUlUbQUR bDUR RRS:wa UR RU5UlU$)Nrrr)rxr)r'rrrrjr int8rr rrr%rGrxrrr) r,rrr r rr is_quantizedrr-s r.rGr1s9>9O9OQU9`Y_9`6|yy%**4  2v{{f7LQUQZQZQaQaQfQfjoQo>>&) ) GJf J M,,!22  $>>IL!dhh&:txx?S?SW]?] $ F 3 r0)rrrj)NTFNNrrr)rOrPrQrRrr'rryrrrrTr rrVrrrrrr rGrWrXrYs@r.rrfsq(,%)&* u||$ U\\ "  ell #  %_8E#vs*:$;<_SW_^(5fc)9#:;^RV^ 03-0 sF{+,eSj)*  QQ5,QDQ1QQ GGFG$GGGr0rcURUS35nUcgURUS3S5n[U[R5(aUR 5n[U[ 5(aU[;a[SU35e[U[ 5(aU[;a [UnUS:wa[SU35eg)Nr" weight_formatrowz'Expected supported weight format - got z+Only 'row' weight format is supported, got ) getpoprr'ritemrTr ValueError) state_dictrlocal_metadatastrict missing_keysunexpected_keys error_msgsr"r8s r.maybe_rearrange_weightrDs ^^vhf- .F ~NNfX]#;UCM-..%**, -%%-?i*iB=/RSS M3 ' 'M=g,gB=Q F}oVWWr0cH^\rSrSrSrS U4SjjrSrS\S\4SjrSr U=r $) Embedding8bitia This class implements [LLM.int8()](https://arxiv.org/abs/2208.07339) algorithm for embedding layer Quantization API is similar to Linear8bitLt: ```python import torch import torch.nn as nn from bitsandbytes.nn import Embedding8bit fp16_module = nn.Embedding(128, 64) int8_module = Embedding8bit(128, 64) int8_module.load_state_dict(fp16_module.state_dict()) int8_module = int8_module.to(0) # Quantization happens here ``` c>[TU]XX4S9 URRRUl[ URRSSS9Ulg)Nr#Frrx)r%r&r"rjr r)r,rrr r r-s r.r&Embedding8bit.__init__sG vS[[%%++  !1!1EY^_ r0c[S5e)Nz.Saving Embedding8bit module is not implementedNotImplementedErrorr,rrrs r.r!Embedding8bit._save_to_state_dict!"RSSr0rCrc[URS5(d [S5eURRnURRnUR UR UR4:XdeUR UR 4:Xde[R"X5n[R"XRUR S55nXES- -nURUR5$)NrzKEmbedding layer is not quantized. Please call .cuda() or .to(device) first.rg_@) hasattrr" RuntimeErrorrjrrrrrErFviewrGr )r,rCrows row_statscompressed_outputcompressed_output_statsoutputs r.rKEmbedding8bit.forwardst{{E**lm m{{KKOO zzd1143E3EFFFF4#6#6"8888KK4"#++e^^DDWDWYZ5["\"&EFyy$$r0r r")NN) rOrPrQrRrSr&rrrKrWrXrYs@r.rFrFs,&` T%V%%%r0rFcr^\rSrSrSrSS\R S4U4SjjrS\4Sjr Sr S\S \4S jr S r U=r $) ria This is the base class similar to Linear4bit. It implements the 4-bit quantization algorithm presented in [QLoRA](https://arxiv.org/abs/2305.14314) for embeddings. Quantization API is similar to Linear4bit: ```python import torch import torch.nn as nn from bitsandbytes.nn import Embedding4bit fp16_module = nn.Embedding(128, 64) quantized_module = Embedding4bit(128, 64) quantized_module.load_state_dict(fp16_module.state_dict()) quantized_module = quantized_module.to(0) # Quantization happens here ``` Nric 2>[TU]XXcS9 URRRUl[ URRSSUUUS9UlURR nX'-S:wa[R"SUSUS35 gg)Nr#FrrzEmbedding size z is not divisible by block size z#. This will lead to slow inference.) r%r&r"rjr rgrlrr) r,rrr rnror rlr-s r.r&Embedding4bit.__init__s vS[[%%++  KK   $!'   KK))  $ ) MM!-0PQZP[\44  *r0rCcURURRR-S:XdeURRR [ R5R URUR-S-S5n[ RRRUR URURS-5US9R SS5nURUR5UR-S-S4:XdeURURR-nURRRnURURU-4:Xde[ RRRUR URU5US9R S5nURUR5U-4:Xde[R "URR5nXgl[ R""/URQURP75Ul [$RR'X75nUR/URQURP7:XdeUR)UR*5$)Nrrr"rCr)rr"rkrlrjrSr'rrr rrFrrabsmaxr}rSizerdequantize_4bitrGr ) r,rC w_4bit_uint8 output_4bitblocks_per_embra output_absmaxoutput_quant_staterXs r. _forward_with_partial_dequantize.Embedding4bit._forward_with_partial_dequantize8sC!!DKK$;$;$E$EEJJJ{{'',,U[[9>>t?R?RUYUgUg?gkl?lnop hh))33$$T%8%8$:L:LPQ:QR4  $r1+   U[[]T5G5G%G1%La$PPPP++t{{/D/DD((//|| 3 3n DFFFF++55;;t22NC6  $    ""u{{}~'E&GGGG!]]4;;+B+BC$1!#(::.P .PT=O=O.P#Q // P||AAd.@.@AAAAyy$$r0c[S5e)Nz.Saving Embedding4bit module is not implementedrKrMs r.r!Embedding4bit._save_to_state_dictYrOr0rc[U5 URURRR-S:XaUR U5$[ RRURRURR5n[RRRUUS9RUR5$)Nrr`)rrr"rkrlrirrrcrjr'r rFrGr )r,rCdequantized_weights r.rKEmbedding4bit.forward\s/5    7 7 A A AQ F88? ? ^^;;DKK[TU]UUUSUUS9 g)Nrir rnror rr,rrr ror r-s r.r&EmbeddingFP4.__init__k)   '  r0re rOrPrQrRr'rr&rWrXrYs@r.rqrqj kk   r0rqcF^\rSrSrS\R S4U4SjjrSrU=r$) EmbeddingNF4i}Nc *>[TU]UUUSUUS9 g)Nrrsrrts r.r&EmbeddingNF4.__init__~rvr0rerwrYs@r.rzrz}rxr0rzc^\rSrSrSrS S\S\4U4SjjjrU4SjrU4SjrSr U4S jr S \ R4S jr S rU=r$) Linear8bitLtia This class is the base module for the [LLM.int8()](https://arxiv.org/abs/2208.07339) algorithm. To read more about it, have a look at the paper. In order to quantize a linear layer one should first load the original fp16 / bf16 weights into the Linear8bitLt module, then call `int8_module.to("cuda")` to quantize the fp16 weights. Example: ```python import torch import torch.nn as nn import bitsandbytes as bnb from bnb.nn import Linear8bitLt fp16_model = nn.Sequential( nn.Linear(64, 64), nn.Linear(64, 64) ) int8_model = nn.Sequential( Linear8bitLt(64, 64, has_fp16_weights=False), Linear8bitLt(64, 64, has_fp16_weights=False) ) int8_model.load_state_dict(fp16_model.state_dict()) int8_model = int8_model.to(0) # Quantization happens here ``` rrcR>[TU]XX75 [R"5UlX`lXPRlX@RlUS:aU(dSURl[URRXDS9Ul UR[5 g)a1 Initialize Linear8bitLt class. Args: input_features (`int`): Number of input features of the linear layer. output_features (`int`): Number of output features of the linear layer. bias (`bool`, defaults to `True`): Whether the linear class uses the bias term as well. TrHN)r%r&r MatmulLtStaterindex thresholdruse_poolrr"rj"_register_load_state_dict_pre_hookrD) r,rrrrrrr r-s r.r&Linear8bitLt.__init__s{* $G&&(  ( &6 # s?#3"&DJJ  !1!1DTu  //0FGr0c>[T U]XU5 Sn[URU5n[URU5nX$-nUS-nURR (dUbAU(aUOUR 5X'[R"S[RS9X'gUbAU(aUOUR 5X'[R"S[RS9X'ggg)Nrr8r)r ) r%rrr"rrrr'rr) r,rrrscb_nameparam_from_weightparam_from_statekey_name format_namer-s r.r Linear8bitLt._save_to_state_dicts #KC$DKK:"4::x8j). zz** ,=F(9L]LdLdLf %+0<<+M (!-[T U]UUUUUUU5 [U5nUHn U [U5Sn U S:XdMURR c [ S5eXn URR RU 5 URR b%URR URlURU 5 M g)NrzLoading a quantized checkpoint into non-quantized Linear8bitLt is not supported. Please call module.cuda() before module.load_state_dict()) r%_load_from_state_dictlistlenr"rrRcopy_rremove) r,r>rr?r@rArBrCunexpected_copykey input_name input_paramr-s r.r"Linear8bitLt._load_from_state_dicts %        /"CS[]+JU";;??*&c )o  %%k2::>>-%)[[__DJJN&&s+!#r0cURRURlURRURlSURlSURlgr2r"rrrr7s r.init_8bit_stateLinear8bitLt.init_8bit_state>     r0c>[TU]"U0UD6n[RRR "U0UD6upEpgUbUR Rb4UR RRU5UR lUR Rb4UR RRU5UR lU$r2) r%rGr'rrrrrr) r,rrrr r rrr-s r.rGLinear8bitLt.tosT,V,9>9O9OQU9`Y_9`6|  ||*"(,,//"4"4V"< ||+#)<<#3#3#6#6v#>   r0rc~URURlURRbUR 5 UR bbUR RUR:wa>UR RRUR5UR l[R"XRUR URS9nURR(d<URRb%URRURlU$N)rr) trainingr is_trainingr"rrrr rjrGrmatmulr)r,routs r.rKLinear8bitLt.forward&s!%  ;;>> %  " 99 TYY__%?!YY^^..qww7DIINjjKKdiitzzJzz**tzz}}/H#zz}}DKK  r0rrr")TTrNN)rOrPrQrRrSrTr&rrrrGr'rrKrWrXrYs@r.r~r~sfF H H H HDN0%,N r0r~c>^\rSrSrSU4SjjrSrSrSrSrU=r $)OutlierAwareLineari7cB>[TU]XX45 SUlSUlgr)r%r& outlier_dimr6)r,rrrr r-s r.r&OutlierAwareLinear.__init__8s" $G!r0c[S5e)NzJPlease override the `forward_with_outliers(self, x, outlier_idx)` functionrK)r,r outlier_idxs r.forward_with_outliers(OutlierAwareLinear.forward_with_outliers=s!"noor0c[S5e)NzEPlease override the `quantize_weights(self, w, outlier_idx)` functionrK)r,rrs r.quantize_weight"OutlierAwareLinear.quantize_weight@s!"ijjr0cURcV[R"5nUR5(d [ S5 UR UR 5nX0lUR(dSURUR UR5nUR RRU5 SUlgg)NzTPlease use OutlierTracer.initialize(model) before using the OutlierAwareLinear layerT) rrr*is_initializedprint get_outliersr"r6rrjr)r,rtracerrrs r.rKOutlierAwareLinear.forwardCs    #"//1F((**lm --dkk:K*   $$T[[$2B2BCA KK   " "1 % $D !r0)r6r)TN) rOrPrQrRr&rrrKrWrXrYs@r.rr7s" pk % %r0rcD^\rSrSrSU4SjjrSrSrSrU=r$)SwitchBackLinearBnbiRc H>[T U]XX85 [R"5UlXplX`RlX@RlXPRlUS:aU(dSURl [URRXDS9Ul g)NrTrH) r%r&rrrrrrmemory_efficient_backwardrrr"rj) r,rrrrrrrr r-s r.r&SwitchBackLinearBnb.__init__Ssv $G&&(  ( &6 #/H , s?#3"&DJJ  !1!1DTu r0cURRURlURRURlSURlSURlgr2rr7s r.r#SwitchBackLinearBnb.init_8bit_statejrr0c.URURlURRbUR 5 [ R"UR5URR5SURS9UR-$r) rrrr"rrr matmul_mixedhalfrrs r.rKSwitchBackLinearBnb.forwardpsf!%  ;;>> %  "$++*:*:*<4tzzZ]a]f]fffr0r)TTFrNN) rOrPrQrRr&rrKrWrXrYs@r.rrRs. "'v. ggr0r).r}typingrrrrrrr'rr r r torch.nn.functionalrrE bitsandbytesrbitsandbytes.cextensionr bitsandbytes.functionalr bitsandbytes.optimrbitsandbytes.utilsrrrr[r ParameterrgrLinearrqr rrrDrFrrqrzr~rrrer0r.rs\  :: ++3.1X C()g4ehh((g4TL""L^zE##zEz3E:W4X3"whwht! ! H( ( VY##YxX*,%BLL,%^aBLLaH = & = &d299dN%%6$g"))$gr0