;i S SK Jr S SKJrJrJr S SKrS SKJr S SK Js J
s J r SSK
Jr " S S\R 5 rS\R S \S
\\\4 S \R( 4S jr S S\R S \S
\S\S S\\ 4
S jjrS\R S\\S4 S\4S jrS\R S\\S4 4S jrS\R S \4S jr SS.S\R S\\\4 S\S \S \S \\\4 S S4S jjr g)! )partial)AnyLiteralOptionalN )
functionalc ^ \ rS rSrSrS\R 4U 4S jjr\R " 5 S\R S\R 4S j5 r Sr U =r
$ ) Bnb4bitParametrization a
A parametrization module that handles dequantization of a 4-bit quantized parameter.
The parameter data is expected to be already quantized when this parametrization is applied.
This module will dequantize the parameter data to its original floating-point representation
when the forward method is called (i.e. when the parameter is accessed).
Args:
quant_state (`F.QuantState`):
The quantization state containing the necessary information for dequantization.
quant_statec . > [ TU ] 5 Xl g N)super__init__r )selfr __class__s k/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/bitsandbytes/nn/parametrize.pyr Bnb4bitParametrization.__init__ s
& quantized_paramreturnc B [ R " XR 5 $ )z
Forward pass to dequantize the parameter.
Args:
quantized_param (`torch.Tensor`): The quantized parameter tensor (from .original)
Returns:
`torch.Tensor`: The dequantized parameter tensor in the original shape and dtype.
)Fdequantize_4bitr )r r s r forward Bnb4bitParametrization.forward s 2B2BCCr )r )__name__
__module__ __qualname____firstlineno____doc__r
QuantStater torchno_gradTensorr __static_attributes__
__classcell__)r s @r r
r
sL
'ALL ' ]]_
Du||
D
D
Dr r
module
param_nameqs_dictdevicec B [ X5 ( d [ SU S35 e[ X5 n[ U[ R
5 ( d [
SU S35 e[ R R X#S9n[ R " X[ U5 SS9 [ X5 g )N Module does not have parameter '' Parameter '$' is not an instance of nn.Parameter)r+ Tunsafe)hasattrAttributeErrorgetattr
isinstancenn Parameter TypeErrorr r" from_dictPregister_parametrizationr
_register_parametrization_hooks)r( r) r* r+ original_paramr s r #replace_parameter_4bit_prequantizedr? * s 6 & & ?
|1 MNN V0N nbll 3 3+j\1UVWW,,(((@K v3I+3V_cd $F7r compress_statistics
quant_type)nf4fp4 blocksizec [ X5 ( d [ SU S35 e[ X5 n[ U[ R
5 ( d [
SU S35 e[ R " UR UUUS9u pg[ X[ R
" USS95 A[ R " X[ U5 SS 9 [ X5 g
) a
Replace a module parameter with a 4-bit quantized version using parametrization.
This function quantizes an existing parameter in a PyTorch module to 4-bit precision
and sets up parametrization to handle automatic dequantization during forward passes.
The original parameter is replaced with quantized data, and a parametrization layer
is registered to manage the quantization state and dequantization process.
Additional, it registers a state dict post-hook to ensure that the quantization state
is saved correctly when the model's state dict is saved.
It is useful for MoE models or other scenarios where you want to quantize parameters
outside of nn.Linear layers without changing the model's architecture.
This feature is experimental and may change in future releases.
Args:
module (`nn.Module`):
The PyTorch module containing the parameter to be quantized.
param_name (`str`):
The name of the parameter within the module to quantize.
compress_statistics (`bool`, *optional*, defaults to `False`):
Whether to compress quantization statistics to reduce memory usage.
quant_type (`Literal["nf4", "fp4"]`, *optional*, defaults to `"nf4"`):
The quantization format to use.
blocksize (`int`, *optional*, defaults to `None`):
The block size for quantization. If None, uses the default block size.
Raises:
AttributeError: If the module does not have the specified parameter.
TypeError: If the specified attribute is not an instance of nn.Parameter.
r- r. r/ r0 )rD r@ rA F)
requires_gradTr1 N)r3 r4 r5 r6 r7 r8 r9 r
quantize_4bitdatasetattrr; r< r
r= )r( r) r@ rA rD r> quantized_datar s r replace_parameter_4bitrK > s P 6 & & ?
|1 MNN V0N nbll 3 3+j\1UVWW #$// / # N F ^5 QR v3I+3V_cd $F7r inputs.outputc x [ =R S- sl [ R ( d 0 [ l g g N )r; _cache_enabled_cache)r( rL rM s r _disable_parametrization_cacherS s( r c 6 [ =R S-
sl g rO )r; rQ )r( rL s r _enable_parametrization_cacherU s r c [ R S: a U R [ [ US95 U R [ 5 U R [ 5 g )N)r r) ) r# __version__ register_state_dict_post_hookr "_parametrized_state_dict_post_hookregister_forward_pre_hookrU register_forward_hookrS )r( r) s r r= r= sL F",, 2%
$$%BC
!?@r weightrX
state_dictprefixlocal_metadatakwargsr c j U SU S3nXa; a U U 3nUR U5 X' [ R " X5 ( d e[ [ S U R
U 5 S5 nUc S5 eUR n U b2 U R SS9R 5 H u pXU U SU
3' M ggg) zB
Hook to modify the state dict to include the quantization state.
zparametrizations.z .originalc " [ U [ 5 $ r )r6 r
)xs r 4_parametrized_state_dict_post_hook.. s Z+A Br Nz,Parametrization not found for the parameter.T)packed.) popr; is_parametrizednextfilterparametrizationsr as_dictitems) r( r_ r` ra r) rb original_key clean_keyparametrizationr kvs r r[ r[ s X .zl)DL! hzl+ *| <
4444 37 BFD[D[\fDg hjn3
*Z,ZZ*%11 "#++4+8>>@:;fXj\1#67 A #' "r )FrB N) functoolsr typingr r r r# torch.nnr7 torch.nn.utils.parametrizeutils parametrizer; r r Moduler
strdictr+ r? boolintrK tuplerS rU r= r[ r r r sv ) ) & & DRYY D>8 II8#&815c3h 8IN 8. !&(- # @8 II@8@8 @8 % @8
} @8F299 eCHo WZ ")) U38_ ABII A3 A.
$< II$<S#X$<
$< $<
$< 38n$<
$