i5SSKrSSKJrJr SSKJr SSKJr SSKJ r J r J r J r J r JrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJr SSKJr SS K J!r! SS K"J#r# SS K$J%r% SS K&J'r' SS K(J)r) SSK*J+r+ SSK,J-r- SSK.J/r/ SSK0J1r1 SSK2J3r3 SSK4J5r5 SSK6J7r7 SSK8J9r9 SSK:J;r; SSKJ?r? SSK@JArA SSKBJCrC SSKDJErE SSKFJGrG 0S\%_S\)_S\+_S \7_S!\!_S"\?_S#\A_S$\5_S%\/_S&\9_S'\;_S(\-_S)\1_S*\E_S+\'_S,\G_S-\C_\3\#\=S..ErH0S\ _S\ _S\ _S%\_S \_S!\ _S"\_S#\_S$\_S'\_S(\_S)\_S&\_S*\_S+\ _S,\_S-\_\\ \S..ErI\R"\K5rL"S/S05rM"S1S25rNS3\O4S4jrPS5\O4S6jrQS7rRg)8N)OptionalUnion) AutoConfig)logging) AqlmConfigAutoRoundConfig AwqConfigBitNetQuantConfigBitsAndBytesConfigCompressedTensorsConfig EetqConfigFbgemmFp8ConfigFineGrainedFP8Config FPQuantConfig GPTQConfig HiggsConfig HqqConfig Mxfp4ConfigQuantizationConfigMixinQuantizationMethod QuantoConfig QuarkConfig SpQRConfig TorchAoConfig VptqConfig) HfQuantizer)AqlmHfQuantizer)AutoRoundQuantizer) AwqQuantizer)BitNetHfQuantizer)Bnb4BitHfQuantizer)Bnb8BitHfQuantizer)CompressedTensorsHfQuantizer)EetqHfQuantizer)FbgemmFp8HfQuantizer)FineGrainedFP8HfQuantizer)FPQuantHfQuantizer)GptqHfQuantizer)HiggsHfQuantizer)HqqHfQuantizer)Mxfp4HfQuantizer)QuantoHfQuantizer)QuarkHfQuantizer)SpQRHfQuantizer)TorchAoHfQuantizer)VptqHfQuantizerawqbitsandbytes_4bitbitsandbytes_8bitgptqaqlmquantoquarkfp_quanteetqhiggshqqzcompressed-tensors fbgemm_fp8torchaobitnetvptqspqr)fp8z auto-roundmxfp4c@\rSrSrSr\S\4Sj5r\S5rSr g)AutoQuantizationConfigsz The Auto-HF quantization config class that takes care of automatically dispatching to the correct quantization config given a quantization config stored in a dictionary. quantization_config_dictc URS5nURSS5(dURSS5(a/URSS5(aSOSn[RU-nOUc [S5eU[;a,[SUS [ [ R5535e[UnURU5$) N quant_method load_in_8bitF load_in_4bit_4bit_8bitThe model's quantization config from the arguments has no `quant_method` attribute. Make sure that the model has been correctly quantizedUnknown quantization type, got  - supported types are: ) getrBITS_AND_BYTES ValueError AUTO_QUANTIZATION_CONFIG_MAPPINGlistAUTO_QUANTIZER_MAPPINGkeys from_dict)clsrHrJsuffix target_clss l/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/transformers/quantizers/auto.pyrY AutoQuantizationConfig.from_dictys/33NC # ' ' > >BZB^B^_motBuBu 8 < <^U S SWY`F-<>>( # #r_rFc\rSrSrSr\S\\\44Sj5r \S5r \S\\\4S\ \4Sj5r \ S5rS rg ) AutoHfQuantizerz The Auto-HF quantizer class that takes care of automatically instantiating to the correct `HfQuantizer` given the `QuantizationConfig`. rac ^[U[5(a[RU5nURnU[ R :XaUR(aUS- nOUS- nU[;a,[SUS[[R5535e[UnU"U40UD6$)NrNrMrPrQ) isinstancerprFrYrJrrSrKrWrTrVrX)rZrargrJr\s r] from_configAutoHfQuantizer.from_configs )4 0 0"8"B"BCV"W *77  -<< <"//' ' 5 51,@/44678:  ,L9 -888r_c R[R"U40UD6nURU5$)N)rFrcrw)rZrfrgras r]rcAutoHfQuantizer.from_pretraineds*4DDEbmflm233r_quantization_config_from_argsc 6UbSnOSn[U[5(aA[U[5(a[R"U5nO[RU5nUbhUR R UR R :wa:[SUR R SUR R S35e[U[[[[[[45(aXUbUUR5nUR5HupV[XU5 M US[!UR#55S3- nUS:wa-[U[5(d[$R&"U5 U$[(R+U5 U$)zt handles situations where both quantization_config from args and quantization_config from model config are present. zYou passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` from the model will be used.zThe model is quantized with z but you are passing a z| config. Please make sure to pass the same quantization config class to `from_pretrained` with different loading attributes.z"However, loading attributes (e.g. z]) will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored.)rvrpr rYrF __class__rjrTrr rr rget_loading_attributesitemssetattrrVrXwarningswarnloggerinfo)rZrar{ warning_msgloading_attr_dictattrvals r]merge_quantization_configs*AutoHfQuantizer.merge_quantization_configss ) 4y  K )4 0 07II&5&?&?@S&T#&<&F&FGZ&[# * 5#--66:W:a:a:j:jj./B/L/L/U/U.VVmoLoVoVo_o_n``FF  #YJacno  .9 = T T V .446 +377 ?EVE[E[E]@^?_`}~ ~K " Z0C[%Q%Q MM+ &#" KK $""r_c URSS5nURSS5(dURSS5(a/URSS5(aSOSn[RU-nOUc [S5eU[;a8[ R SUS [[R55S 35 gg ) NrJrKFrLrMrNrOrPrQz~. Hence, we will skip the quantization. To remove the warning, you can delete the quantization_config attribute in config.jsonT) rRrrSrTrUrwarningrVrWrX)rHrJr[s r]supports_quant_method%AutoHfQuantizer.supports_quant_methods/33NDI # ' ' > >BZB^B^_motBuBu 8 < <^U S SWY`F-<T[;a[STS35e[U[5(d [ S5eU[T'U$)NzConfig '' already registeredz*Config must extend QuantizationConfigMixin)rUrT issubclassr TypeError)rZrs r]register_config_fn8register_quantization_config..register_config_fn sK 5 5xx/CDE E#677HI I36(0 r_rb)rrs` r]register_quantization_configr s r_namec^U4SjnU$)zRegister a custom quantizer.c>T[;a[STS35e[U[5(d [S5eU[T'U$)Nz Quantizer 'rz!Quantizer must extend HfQuantizer)rWrTrr)rZrs r]register_quantizer_fn1register_quantizer..register_quantizer_fnsJ ) ){4&0DEF F#{++@A A'*t$ r_rb)rrs` r]register_quantizerrs ! r_cp[US5nU(a&[RUR5(dSnU(dUbRU(a&[R URU5UlOXl[R URUS9n OSn U bU R UUUUUS9 U RU5nU RU5nU RU5nU RU5n[U RSS5(d&U RRn [U SU 5US'XX%4$)NraF) pre_quantized)dtypefrom_tf from_flax device_map weights_only dequantizevaluequant) hasattrrsrrarrwvalidate_environment update_dtypeupdate_device_mapupdate_tp_planupdate_ep_planrdrJ) configrarrrrr user_agentr hf_quantizerrJs r]get_hf_quantizerr*s8F$9:M_BB6C]C]^^ +7 )8)S)S**,?*F &*= &&22  & &'3  ))!% * ))%0!33J? ,,V4,,V4|77uMM';;HHL"),"NJw   22r_)Srtypingrrmodels.auto.configuration_autorutilsrutils.quantization_configrr r r r r rrrrrrrrrrrrrrrbaserquantizer_aqlmrquantizer_auto_roundr quantizer_awqr!quantizer_bitnetr"quantizer_bnb_4bitr#quantizer_bnb_8bitr$quantizer_compressed_tensorsr%quantizer_eetqr&quantizer_fbgemm_fp8r'quantizer_finegrained_fp8r(quantizer_fp_quantr)quantizer_gptqr*quantizer_higgsr+ quantizer_hqqr,quantizer_mxfp4r-quantizer_quantor.quantizer_quarkr/quantizer_spqrr0quantizer_torchaor1quantizer_vptqr2rWrU get_loggerrjrrFrsstrrrrrbr_r]rs"7.+4'/22F+6@2+-)-/-+1+ <++ O   O     " O  >6&!   O!" O#$ %$ ).$ 9$+$+$ J $  J $  J $ l$ [$ $ 9$1$/$ [$}$ $  J!$" J#$$ ! )$ .   H %&#&#Rkk\   !S ! %3r_