i?.SSKJrJr SSKJrJrJrJr SSKJ r J r SSK J r J r SSKJr \(aSSKJr \ "5(a SS KrSS KJr O\r\ R,"\5r"S S \5r"S S\5rS\\ R6\ R8/S.0rg ))ABCabstractmethod) TYPE_CHECKINGAnyOptionalUnion)is_torch_availablelogging)QuantizationConfigMixinQuantizationMethod)get_module_from_name)PreTrainedModelN) ModuleListc \rSrSrSrSrSrSrS\4Sjr S8S jr S8S jr S \ \ \\4S \ \ \\44S jrS8SjrS\\S\S \\4SjrS\\S\\S \\4SjrS\\S \\4SjrSSS \ \S44SjrS\ \\\\44S \ \\\\444SjrS \4SjrSSS\S \4SjrSrS rS!rS"r S9S#jr!S9S$jr"S%r#S&r$S'r%S(r&S\S \4S)jr'\(S:SSS*\ \\S+\ \\S,\4S-jj5r)\*S \4S.j5r+\*S \4S/j5r,S;S0jr-S1r.\/S25r0\/S35r1\/S!!d&?&?*+>+K+K*LMNO '@!dtype torch.dtypereturncN[RS5 URU5$)z Deprecared in favor of `update_dtype`! Args: dtype (`torch.dtype`): The input dtype that is passed in `from_pretrained` zb`update_torch_dtype` is deprecated in favor of `update_dtype`! It will be removed in version v4.57)logger warning_once update_dtyperr#s rupdate_torch_dtypeHfQuantizer.update_torch_dtypeHs(  p   ''r"cU$)a Some quantization methods require to explicitly set the dtype of the model to a target dtype. You need to override this method in case you want to make sure that behavior is preserved Args: dtype (`torch.dtype`): The input dtype that is passed in `from_pretrained` r*s rr)HfQuantizer.update_dtypeU  r" device_mapcU$)aa Override this method if you want to pass a override the existing device map with a new one. E.g. for bitsandbytes, since `accelerate` is a hard requirement, if no device_map is passed, the device_map is set to `"auto"`` Args: device_map (`Union[dict, str]`, *optional*): The device_map that is passed through the `from_pretrained` method. r.)rr1s rupdate_device_mapHfQuantizer.update_device_mapas r"cU$)a Override this method if you want to adjust the `target_dtype` variable used in `from_pretrained` to compute the device_map in case the device_map is a `str`. E.g. for bitsandbytes we force-set `target_dtype` to `torch.int8` and for 4-bit we pass a custom enum `accelerate.CustomDtype.int4`. Args: dtype (`torch.dtype`, *optional*): The dtype that is used to compute the device_map. r.r*s radjust_target_dtypeHfQuantizer.adjust_target_dtypemr0r" missing_keysprefixcU$)z Override this method if you want to adjust the `missing_keys`. Args: missing_keys (`list[str]`, *optional*): The list of missing keys in the checkpoint compared to the state dict of the model r.)rmodelr8r9s rupdate_missing_keysHfQuantizer.update_missing_keysys r" expected_keys loaded_keyscU$)a Override this method if you want to adjust the `update_expected_keys`. Args: expected_keys (`list[str]`, *optional*): The list of the expected keys in the initialized model. loaded_keys (`list[str]`, *optional*): The list of the loaded keys in the checkpoint. r.)rr;r>r?s rupdate_expected_keys HfQuantizer.update_expected_keyss r"unexpected_keyscU$Nr.)rr;rCs rupdate_unexpected_keys"HfQuantizer.update_unexpected_keyssr"c^UR5V^Vs0sH/umn[U4SjUR55(dM,TU_M1 snn$s snnf)a returns dtypes for modules that are not quantized - used for the computation of the device_map in case one passes a str as a device_map. The method will use the `modules_to_not_convert` that is modified in `_process_model_before_weight_loading`. Args: model (`~transformers.PreTrainedModel`): The model to quantize dtype (`torch.dtype`): The dtype passed in `from_pretrained` method. c3,># UH oT;v M g7frEr.).0mnames r 8HfQuantizer.get_special_dtypes_update..sFvZuUVDyZus)named_parametersanyr)rr;r#rL_s ` rget_special_dtypes_update%HfQuantizer.get_special_dtypes_updatesN(-'='='? '?GD!3FvZ^ZuZuFvCvKD%K'?   s +AA max_memorycU$)zaadjust max_memory argument for infer_auto_device_map() if extra memory is needed for quantizationr.)rrTs radjust_max_memoryHfQuantizer.adjust_max_memoryr"cP[RS5 UR"U0UD6$)zDEPRECATED -> remove in v5z`check_quantized_param` is deprecated in favor of `param_needs_quantization`, which is a much more self.explanatory name for what the method achieves. It will be removed in v5)r'r(param_needs_quantizationrargsrs rcheck_quantized_param!HfQuantizer.check_quantized_params. ` ,,d=f==r"r;r param_namec g)zX Check whether a given param needs quantization as defined by `create_quantized_param`. Fr.)rr;r_rs rrZ$HfQuantizer.param_needs_quantizationsr"clUR(d#[SURRS35eg)z Take needed components from state_dict (those from which `param_needs_quantization` is True) and create quantized param. It usually also load the new param directly in the `model`. Note: only applicable if requires_parameters_quantization == True. zG`.create_quantized_param()` method is not supported by quantizer class .N) requires_parameters_quantizationAttributeError __class____name__r[s rcreate_quantized_param"HfQuantizer.create_quantized_params:44 YZ^ZhZhZqZqYrrst 5r"cg)a This method is used to potentially check for potential conflicts with arguments that are passed in `from_pretrained`. You need to define it for all future quantizers that are integrated with transformers. If no explicit check are needed, simply return nothing. Nr.r[s rvalidate_environment HfQuantizer.validate_environments r"cU$z"updates the tp plan for the scalesr.rconfigs rupdate_tp_planHfQuantizer.update_tp_plan r"cU$rnr.ros rupdate_ep_planHfQuantizer.update_ep_planrsr"c SUlURRUlUR(aUR U5 UR "U40UD6$)a  Setting model attributes and/or converting model before weights loading. At this point the model should be initialized on the meta device so you can freely manipulate the skeleton of the model in order to replace modules in-place. Make sure to override the abstract method `_process_model_before_weight_loading`. Args: model (`~transformers.PreTrainedModel`): The model to quantize kwargs (`dict`, *optional*): The keyword arguments that are passed along `_process_model_before_weight_loading`. T) is_quantizedrrquantization_methodr_convert_model_for_quantization$_process_model_before_weight_loadingrr;rs rpreprocess_modelHfQuantizer.preprocess_modelsM"$($<$<$I$I!     0 0 788I&IIr"c (UR"U40UD6$)aM Post-process the model post weights loading. Make sure to override the abstract method `_process_model_after_weight_loading`. Args: model (`~transformers.PreTrainedModel`): The model to quantize kwargs (`dict`, *optional*): The keyword arguments that are passed along `_process_model_after_weight_loading`. )#_process_model_after_weight_loadingr|s rpostprocess_modelHfQuantizer.postprocess_models77HHHr"c[US5(aU?[URS5(a UR?[URS5(a UR?[US5(aU?SUlg)z0 Remove the quantization config from the model. hf_quantizerr_pre_quantization_dtyperyFN)hasattrrrprrryrxrr;s rremove_quantization_config&HfQuantizer.remove_quantization_configsk 5. ) )" 5< Override this method if you want to adjust the `param_name`. r.)rr_s rget_param_nameHfQuantizer.get_param_names r" skip_moduleskeep_in_fp32_modulesadd_default_skipscSSKJn UbU(a U"U5nO/nUbURU5 UbURU5 U$)Nr )get_keys_to_not_convert) integrationsrextend)r;rrrrrs rget_modules_to_not_convert&HfQuantizer.get_modules_to_not_convert$sP ;  #4% ?%%r"cg)zUFlag indicating whether the quantized model can carry out quantization aware trainingFr.rs ris_qat_trainableHfQuantizer.is_qat_trainable:r"cg)z;Flag indicating whether the quantized model can be compiledFr.rs ris_compileableHfQuantizer.is_compileable?rr"c S04$)zcGet state dict and metadata. Useful when we need to modify a bit the state dict due to quantizationNr.)rr;safe_serializations rget_state_dict_and_metadata'HfQuantizer.get_state_dict_and_metadataDs Rxr"cU$)zEUpdate state dict with metadata. Default behaviour returns state_dictr.)r state_dictmetadatas rupdate_state_dict_with_metadata+HfQuantizer.update_state_dict_with_metadataHrXr"c grEr.r|s rr{0HfQuantizer._process_model_before_weight_loadingLsEHr"c grEr.r|s rr/HfQuantizer._process_model_after_weight_loadingOsDGr"cgrEr.)rrs ris_serializableHfQuantizer.is_serializableRs8;r"cgrEr.rs r is_trainableHfQuantizer.is_trainableUsr"cSSKJn UR5Hup4URRnU[ ;dM'UR R[ US;dMMU"5 [X5upc[ US"URR55URU'SSS5 M g!,(df  M=f)Nr)init_empty_weightsquantization_methods module_name) accelerater named_modulesrfrg!MODULES_TO_PATCH_FOR_QUANTIZATIONrrrrpget_text_config_modules)rr;rrLmodulemodule_class_name parent_modules rrz+HfQuantizer._convert_model_for_quantizationYs1!//1LD & 0 0 9 9  $EE((5545FGH^_`()*>u*K'M3TUf3ghu3v 4464M**40*) 2 *)s +AB== C )rrr)r#r$r%r$)r;r)NNF)FrE)6rg __module__ __qualname____firstlineno____doc__rrequired_packagesrdr r r+r)rdictstrrr3r6listr<rArFrRrintrVboolr]rZrhrkrqrur}rrrrrr staticmethodrpropertyrrrrrr{rrrrz__static_attributes__r.r"rrr"s|&!',$ ,C  (  HT#s(^,D RVWZ\_W_R`Ia  tCy#RVWZR[ c QUVYQZ _cdg_h T#Y4PS9 m SR_M_H` "DeCHo1E,F4PSUZ[^`c[cUdPdKe>>.?S_c J$ I #   -148"' & &tCy)&'tCy1& &&*$HHGG;;  r"rc>^\rSrSrSrU4SjrSSjrSrU=r$)SequentialLlama4TextExpertsiiz A module that implements a compressed version of a list of expert modules. This is specifically designed to work with Llama4TextExperts in MoE layers. c>SSKJn [TU] [ UR 5Vs/sH o2"U5PM sn5 UR Ulgs snf)Nr) Llama4TextMLP)*transformers.models.llama4.modeling_llama4rsuperr rangenum_local_experts num_experts)rrprrQrfs rr $SequentialLlama4TextExperts.__init__osHL v?W?W9XY9XA-/9XYZ!33ZsAcURURSURS5n[R"U5n[ UR5HnX"X5X#'M U$)N)reshapershapetorch zeros_liker)r hidden_states routed_out expert_idxs rforward#SequentialLlama4TextExperts.forwardush&--d.>.>MDWDWXZD[\ %%m4  0 01J%)%5m6O%PJ "2r")r)r torch.Tensorr%r) rgrrrrr rr __classcell__)rfs@rrris) 4 % r"rLlama4TextExperts)rr)abcrrtypingrrrrutilsr r utils.quantization_configr r quantizers_utilsrmodeling_utilsrrtorch.nnrr get_logger__file__r'rrCOMPRESSED_TENSORSBITS_AND_BYTESrr.r"rrs$66/S20#J   H %D#DN *02  1 1  - -! %!r"