;iC 6 S SK r S SKrS SKrS SKJr S SKJr S SKJrJ r S SK
r
S SK J r SSK
JrJrJrJrJrJrJr SSKJrJrJrJrJrJrJ r J r J r J r J r J!r!J"r"J#r#J$r$J%r%J&r& SSK'J(r( SS K)J*r* \ RV " \,5 r-\S4S
\\. 4S jj5 r/\S4S \
R` S
\\. 4S
jj5 r1 S5S\ Rd S\\
R` S\.S\\3\4\
Rj 4 S\\6\4 4
S jjr7 S6S\
R Rd S\\ \8\4\
R` 4 S\\ 4S jjr9 S7S\ Rd S\ \4\Rt 4 S\\
R` S\.S\\6\4 4
S jjr; S8S\ Rd S\3\4\ \4\8\
R` 4 4 S\\
R` S\\3\4\
Rj 4 S\\ \4\Rt 4 S\\3\4\44 S\.S\\ \4\6\4 4 S\\6\4 S \.4S jjr< S9S\ Rd S \ \4\Rt 4 S\\ \4\3\4\ \8\4\
R` 4 4 4 S \\3\ \8\44 \ \8\44 4 S \\6\4 S!\\ \4\Rt 4 S\.S"\\ \4\
Rz 4 S#\\. S\\ \4\6\4 4 S\\6\4 S \.S$\.S%\.S&\.4 S' jjr> S:S(\
R Rd S)\
Rz S*\
Rz S+\\ \4\?\4S,4 4 S-\\?\@\
R Rd S,4 S.\.S/S4S0 jjrA S;S(\
R Rd S)\
Rz S*\
Rz S+\\ \4\?\4S,4 4 S-\\?\@\
R Rd S,4 S.\.S1\44S2 jjrBS\ Rd 4S3 jrCg)< N)contextmanagerwraps)OptionalUnion )AlignDevicesHook
CpuOffloadLayerwiseCastingHookUserCpuOffloadHookadd_hook_to_moduleattach_align_device_hook"attach_align_device_hook_on_blocks)OffloadedWeightsLoadercheck_cuda_p2p_ib_supportcheck_device_map extract_submodules_state_dictfind_tied_parametersget_balanced_memoryinfer_auto_device_mapis_bnb_availableis_mlu_availableis_musa_availableis_npu_availableis_sdaa_availableis_xpu_availableload_checkpoint_in_modeloffload_state_dictparse_flag_from_envretie_parameters)&SUPPORTED_PYTORCH_LAYERS_FOR_UPCASTING)recursive_getattrinclude_buffersc # # U c [ SS5 n [ [ R " S5 U S9 nUv SSS5 g! , ( d f g= f7f)a
A context manager under which models are initialized with all parameters on the meta device, therefore creating an
empty model. Useful when just initializing the model would blow the available RAM.
Args:
include_buffers (`bool`, *optional*):
Whether or not to also put all buffers on the meta device while initializing.
Example:
```python
import torch.nn as nn
from accelerate import init_empty_weights
# Initialize a model with 100 billions parameters in no time and without using any RAM.
with init_empty_weights():
tst = nn.Sequential(*[nn.Linear(10000, 10000) for _ in range(1000)])
```
Any model created under this context manager has no weights. As such you can't do something like
`model.to(some_device)` with it. To load weights inside your empty model, see [`load_checkpoint_and_dispatch`].
Make sure to overwrite the default device_map param for [`load_checkpoint_and_dispatch`], otherwise dispatch is not
called.
N ACCELERATE_INIT_INCLUDE_BUFFERSFmeta)r# )r init_on_devicetorchdevice)r# fs g/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/accelerate/big_modeling.pyinit_empty_weightsr, < sB < -.OQVW V,o NRS
O N Ns .A> A
A Ar) c # ^ ^^ # Uc [ SS5 nU( a T Sv SSS5 g[ R R m U( a [ R R mU U 4S jnSU U4S jjnU( a" S Vs0 s H nU[ [ U5 _M nnO0 nU 4S jn U[ R l U( a U[ R l UR 5 H' n[ [ XF" [ [ U5 5 5 M) Sv T [ R l U( a T[ R l UR 5 H u pG[ [ XG5 M g! , ( d f g= fs snf ! T [ R l U( a T[ R l UR 5 H u pG[ [ XG5 M f = f7f) a
A context manager under which models are initialized with all parameters on the specified device.
Args:
device (`torch.device`):
Device to initialize all parameters on.
include_buffers (`bool`, *optional*):
Whether or not to also put all buffers on the meta device while initializing.
Example:
```python
import torch.nn as nn
from accelerate import init_on_device
with init_on_device(device=torch.device("cuda")):
tst = nn.Linear(100, 100) # on `cuda` device
```
Nr% Fc > T" XU5 Ubs [ U R U 5 nU R U R nUR US' U" U R U R T5 40 UD6U R U' g g )N
requires_grad)type _parameters__dict__r/ to)modulenameparam param_clskwargsr) old_register_parameters r+ register_empty_parameter0init_on_device..register_empty_parameter s vU3 V / / 56I''-66F&+&9&9F? #'01C1CD1I1L1LV1T'_X^'_F t $ c r > T" XX#S9 Ub, U R U R T5 U R U' g g )N)
persistent)_buffersr3 )r4 r5 bufferr> r) old_register_buffers r+ register_empty_buffer-init_on_device..register_empty_buffer s: F&H $*OOD$9$<$^ UU 4S jnU$ )Nc > TUS' T" U 0 UD6$ )Nr) )argsr8 r) fns r+ wrapperAinit_on_device..patch_tensor_constructor..wrapper s %F8 t&v& &r< rJ )rL rM r) s ` r+ patch_tensor_constructor0init_on_device..patch_tensor_constructor s ' r< )T)
r nnModuleregister_parameterregister_buffergetattrr( keyssetattritems)
r) r# r: rB torch_function_name tensor_constructors_to_patchrO old_torch_functionrA r9 s
` @@r+ r' r' ` s * -.OQVW
YY 9 9 ii 7 7`E E (J(
'I#
0C!D D'I % (
$
(*$ D'? $ (=BII %#?#D#D#F E.0HQVXkIl0m n $G
'= $ (;BII %7S7Y7Y7[ 3 E. C 8\a V*(
, (> $ (;BII %7S7Y7Y7[ 3 E. C 8\sC G)E4A G)F G))A0F
AG)4
F> G)
A G&&G)modelexecution_deviceoffload_buffers
state_dictpreload_module_classesc 8 Uc, [ [ U R 5 5 5 R nUcA U R 5 R 5 VVs0 s H u pVXVR
S5 _M nnn[ U [ SS9SS9 [ U USUUUS9 U $ s snnf )a>
Activates full CPU offload for a model. As a result, all parameters of the model will be offloaded and only one
copy of the state dict of the model will be kept. During the forward pass, parameters will be extracted from that
state dict and put on the execution device passed as they are needed, then offloaded again.
Args:
model (`torch.nn.Module`):
The model to offload.
execution_device (`torch.device`, *optional*):
The device on which the forward pass of the model will be executed (should be a GPU). Will default to the
model first parameter device.
offload_buffers (`bool`, *optional*, defaults to `False`):
Whether or not to offload the buffers with the model parameters.
state_dict (`Dict[str, torch.Tensor]`, *optional*):
The state dict of the model that will be kept on CPU.
preload_module_classes (`List[str]`, *optional*):
A list of classes whose instances should load all their weights (even in the submodules) at the beginning
of the forward. This should only be used for classes that have submodules which are registered but not
called directly during the forward, for instance if a `dense` linear layer is registered, but at forward,
`dense.weight` and `dense.bias` are used in some operations instead of calling `dense` directly.
cpuTio_same_deviceappendr] offloadr^ weights_mapr` )
nextiter
parametersr) r_ rX r3 r
r r )r\ r] r^ r_ r` nps r+ cpu_offloadro s 8 U%5%5%7 89@@161A1A1C1I1I1KL1Kaen1K
Lu .d CDQ
)' 5
L Ms Bprev_module_hookc D [ XS9n[ XSS9 [ X5 nX4$ )a
Offloads a model on the CPU and puts it back to an execution device when executed. The difference with
[`cpu_offload`] is that the model stays on the execution device after the forward and is only offloaded again when
the `offload` method of the returned `hook` is called. Useful for pipelines running a model in a loop.
Args:
model (`torch.nn.Module`):
The model to offload.
execution_device(`str`, `int` or `torch.device`, *optional*):
The device on which the model should be executed. Will default to the MPS device if it's available, then
GPU 0 if there is a GPU, and finally to the CPU.
prev_module_hook (`UserCpuOffloadHook`, *optional*):
The hook sent back by this function for a previous model in the pipeline you are running. If passed, its
offload method will be called just before the forward of the model to which this hook is attached.
Example:
```py
model_1, hook_1 = cpu_offload_with_hook(model_1, cuda_device)
model_2, hook_2 = cpu_offload_with_hook(model_2, cuda_device, prev_module_hook=hook_1)
model_3, hook_3 = cpu_offload_with_hook(model_3, cuda_device, prev_module_hook=hook_2)
hid_1 = model_1(input)
for i in range(50):
# model1 is offloaded on the CPU at the first iteration, model 2 stays on the GPU for this whole loop.
hid_2 = model_2(hid_1)
# model2 is offloaded to the CPU just before this forward.
hid_3 = model_3(hid_3)
# For model3, you need to manually call the hook offload method.
hook_3.offload()
```
)r] rp Tre )r
r
r )r\ r] rp hook user_hooks r+ cpu_offload_with_hookrt s- L '7 [Du40"5/I r< offload_dirc [ R R U5 ( aB [ R R [ R R US5 5 ( d [ XR
5 5 Uc, [ [ U R 5 5 5 R n[ US9n[ U [ SS9SS9 [ U USUUUS9 U $ )a@
Activates full disk offload for a model. As a result, all parameters of the model will be offloaded as
memory-mapped array in a given folder. During the forward pass, parameters will be accessed from that folder and
put on the execution device passed as they are needed, then offloaded again.
Args:
model (`torch.nn.Module`): The model to offload.
offload_dir (`str` or `os.PathLike`):
The folder in which to offload the model weights (or where the model weights are already offloaded).
execution_device (`torch.device`, *optional*):
The device on which the forward pass of the model will be executed (should be a GPU). Will default to the
model's first parameter device.
offload_buffers (`bool`, *optional*, defaults to `False`):
Whether or not to offload the buffers with the model parameters.
preload_module_classes (`List[str]`, *optional*):
A list of classes whose instances should load all their weights (even in the submodules) at the beginning
of the forward. This should only be used for classes that have submodules which are registered but not
called directly during the forward, for instance if a `dense` linear layer is registered, but at forward,
`dense.weight` and `dense.bias` are used in some operations instead of calling `dense` directly.
index.json) save_folderTrc re rg )ospathisdirisfilejoinr r_ rj rk rl r) r r
r r )r\ ru r] r^ r` ri s r+ disk_offloadr~ s 6 77== % %RWW^^BGGLLVb Sn[ TR 5 S:X aF [ R R R
" U 0 UD6S nUb [ R U5 O[ R U5 TR 5 H2 nUR [ R " S5 :X d M) [ S5 e T" U 0 UD6$ )NzEYou shouldn't move a model that is dispatched using accelerate hooks.r3 r r& zFYou can't move a model that has some modules offloaded to cpu or disk.) str__name__r( _C_nn _parse_tologgerwarningrl r) RuntimeError)rK r8 warning_msg to_devicer6 rL r\ s r+ rM 4dispatch_model..add_warning..wrapper s e r{{#t+ % 6 6 G G JI , {3NN;/ " - - /E ||u||F';; *+s tt 0 4*6**r< r )rL r\ rM s `` r+ add_warning#dispatch_model..add_warning s
2Y
+
+ Nr< zWe've detected an older driver with an RTX 4000 series GPU. These drivers have issues with P2P. This can affect the multi-gpu inference when using accelerate device_map.Please make sure to update your driver to the latest version which resolves this.znpu:zmlu:zsdaa:zmusa:zfYou are trying to offload the whole model to the disk. Please use the `disk_offload` function instead.)*r rU r lensetvaluesrX r r_
ValueErrorr} ry rz r{ r| r r r r" data_ptrr r r r r3 r npur mlur sdaar musar xpucudar list
isinstanceintdict
hf_device_map) r\ r r r_ ru r r^ r r` r dr5 r) cpu_modules disk_modulesdisk_state_dictr] offloaded_devicesrh rx ri tied_paramsr group
param_namer offloaded_devices_strr
use_multi_gpus r+ dispatch_modelr 5 s \ U' u+^<N 5-u 5 5 E. 6 6?O\d?e K C
!!# $%)k :$$&'E72c*:K:K:M6NSXZ`Ra6a # *4*;*;*= Z*=Q/AYq*= Z[\ ] % 4>4D4D4FZ4FLD&TY/44FKZ!c+&6&: :5;K;K;M{ [
1;1A1A1CX1C vQWGW1C X =#8S =NQR=R ))-<)@(AD
! %WW]];//rww~~bggllS^`lFm7n7n ;EWV": X!
$ % ) NN\]r\sstu
, $ uxx/ #EIIu5EI
#EIIu5EI
$UZZ7EJ
$UZZ7EJ
#EIIu5EI$UZZ7EJ #j6G6G6I2J l2J\kNkV2J lmpqq
!:!>> from accelerate import init_empty_weights, load_checkpoint_and_dispatch
>>> from huggingface_hub import hf_hub_download
>>> from transformers import AutoConfig, AutoModelForCausalLM
>>> # Download the Weights
>>> checkpoint = "EleutherAI/gpt-j-6B"
>>> weights_location = hf_hub_download(checkpoint, "pytorch_model.bin")
>>> # Create a model and initialize it with empty weights
>>> config = AutoConfig.from_pretrained(checkpoint)
>>> with init_empty_weights():
... model = AutoModelForCausalLM.from_config(config)
>>> # Load the checkpoint and dispatch it to the right devices
>>> model = load_checkpoint_and_dispatch(
... model, weights_location, device_map="auto", no_split_module_classes=["GPTJBlock"]
... )
```
)autobalancedbalanced_low_0
sequentialziIf passing a string for `device_map`, please choose 'auto', 'balanced', 'balanced_low_0' or 'sequential'.r r )r r r low_zero)r r r r^ r T)r r r r r^ r r r )r ru r^ r r` r )r r r r r r r r )r\ r r r r r r^ r r r r` r r r r s r+ load_checkpoint_and_dispatchr s | *c""z9m'm w
*c"" %,%(?$(88 J + !$; +
!j&<:K\K\K^A^ !
%-'' 1
"' 5 r< r4
storage_dtype
compute_dtypeskip_modules_pattern.skip_modules_classes non_blockingreturnc [ XX#XE5 g)aZ
Applies layerwise casting to a given module. The module expected here is a PyTorch `nn.Module`. This is helpful for
reducing memory requirements when one doesn't want to fully quantize a model. Model params can be kept in say,
`torch.float8_e4m3fn` and upcasted to a higher precision like `torch.bfloat16` during forward pass and downcasted
back to `torch.float8_e4m3fn` to realize memory savings.
Args:
module (`torch.nn.Module`):
The module whose leaf modules will be cast to a high precision dtype for computation, and to a low
precision dtype for storage.
storage_dtype (`torch.dtype`):
The dtype to cast the module to before/after the forward pass for storage.
compute_dtype (`torch.dtype`):
The dtype to cast the module to during the forward pass for computation.
skip_modules_pattern (`tuple[str, ...]`, defaults to `None`):
A list of patterns to match the names of the modules to skip during the layerwise casting process. If set
to `None` alongside `skip_modules_classes` being `None`, the layerwise casting is applied directly to the
module instead of its internal submodules.
skip_modules_classes (`tuple[type[torch.nn.Module], ...]`, defaults to `None`):
A list of module classes to skip during the layerwise casting process.
non_blocking (`bool`, defaults to `False`):
If `True`, the weight casting operations are non-blocking.
Example:
```python
>>> from accelerate.hooks import attach_layerwise_casting_hooks
>>> from transformers import AutoModelForCausalLM
>>> import torch
>>> # Model
>>> checkpoint = "EleutherAI/gpt-j-6B"
>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)
>>> # Attach hooks and perform inference
>>> attach_layerwise_casting_hooks(model, storage_dtype=torch.float8_e4m3fn, compute_dtype=torch.bfloat16)
>>> with torch.no_grad():
... model(...)
```
Users can also pass modules they want to avoid from getting downcasted.
```py
>>> attach_layerwise_casting_hooks(
... model, storage_dtype=torch.float8_e4m3fn, compute_dtype=torch.bfloat16, skip_modules_pattern=["norm"]
... )
```
N) _attach_layerwise_casting_hooks)r4 r r r r r s r+ attach_layerwise_casting_hooksr s p $}DXr< _prefixc ^ US L=( a [ X5 =( d US L=( a [ U4S jU 5 5 nU( a [ R ST S35 g [ U [ 5 ( a- [ R ST S35 [ U [
XUS9SS9 g U R 5 H$ u pT( a T SU 3OUn
[ U UUUUUU
S 9 M& g )
Nc 3 R ># U H n[ R " UT5 v M g 7fN)research).0patternr s r+ 2_attach_layerwise_casting_hooks.. s" 0q\pQX7G1L1L\ps $'z&Skipping layerwise casting for layer ""z%Applying layerwise casting to layer ")r r r Tre r )r ) r anyr debugr! r
r named_childrenr ) r4 r r r r r r should_skipr5 submodule
layer_names ` r+ r r s (t3`
68` D(qS0q\p0q-q =gYaHI&@AA ._self_attn_pre_forward_hook s( } ,.2M* +)-M+ &))r< self_attnT) with_kwargsprependN)
named_modulesendswithregister_forward_pre_hook)r\ r r5 r4 s r+ _attach_context_parallel_hooksr sG "* ++- == % % , ,-HVZdh , i .r< r )NFNN)NN)NFN)NNNNFNNF)
NNNNFNNNNFFTF)NNF)NNFr )Dloggingry r
contextlibr functoolsr typingr r r( torch.nnrQ hooksr r
r r r
r r utilsr r r r r r r r r r r r r r r r r utils.constantsr! utils.otherr" getLoggerr r boolr, r) r' rR r r Tensorr ro r rt PathLiker~ r r r tupler0 r r r rJ r< r+ r s % " & D *
8 $ F ID5<< ID(4. ID ID\ 04 !4826 +
99+ u||,+ + c5<< /01 +
%T#Y / +` AE59) 88??) uS#u||%; <=) 12) ^ 04 !26 +
99+sBKK'(+ u||,+ +
%T#Y / +b +/4859.2 !1526 H
99HS%S%,, 6778H %,,'H c5<< /01 H
%R[[ 012 H DcN+
H H c49n-.H %T#Y /H H\ RVCG378< !/3)-1526 !& J
99Jc2;;&'J sDeCell4J.K)K$L LMNJ eCHouS#X >?@ J
&d3i 0 J U3 #3 45
J J E#u{{*+ ,J !J c49n-.J %T#Y /J J
J J Jb CGHL
: HHOO:;;: ;;: #5eCHo)=#>? :
#5ehhoo)>)C#DE :
:
:B CGHL #
HHOO#
;;#
;;#
#5eCHo)=#>? #
#5ehhoo)>)C#DE #
#
#
L%j
99%jr<