9ia^dZddlZddlmZmZddlmZmZddlZddl m Z ddl m Z dd l mZmZdd lmZmZmZed r ddlmcmZerdd lmZdZdZdZdZdZed ddZd3dZ dZ!dZ"dZ#dZ$dZ%dZ&dZ'Gdde(Z)dZ*d Z+e*d!Z,d"efd#Z-d"efd$Z.d4d%Z/d5d&Z0e*d4d'e1fd(Z2d4d'e1fd)Z3d6d*Z4d4d+Z5e+d7d,Z6e*d8d-Z7d.Z8Gd/d0Z9d1Z:d2Z;y)9zB A set of basic tensor ops compatible with tpu, gpu, and multigpu N)update_wrapperwraps)AnyMapping) PartialState)!TORCH_DISTRIBUTED_OPERATION_TYPES)DistributedTypeTensorInformation)is_torch_distributed_availableis_torch_versionis_tpu_availableF) check_device)ReduceOpc6t|tjSN) isinstancetorchTensortensors e/var/www/html/backtest/airagagent/rag_env/lib/python3.12/site-packages/accelerate/utils/operations.pyis_torch_tensorr's fell ++c vt|tjjtjjtjj tjj tjjtjjtjjSr) rrxpu FloatTensor ByteTensor IntTensor LongTensor HalfTensor DoubleTensorBFloat16Tensorrs ris_torch_xpu_tensorr%+sm            rc"t|tSr)rr  tensor_infos ris_tensor_informationr)8s k#4 55rct|}|j}t|dk7s |dtk7ryt |dd}t |tsyt d|DS)z Checks if `x` is a `namedtuple` or not. Can have false positives, but only if a user is trying to mimic a `namedtuple` perfectly. r rF_fieldsNc3<K|]}t|tywr)rstr).0members r z is_namedtuple..Hs<6z&#&.os@ "!".7M`dj s rAzUnsupported types (z ) passed to `z?`. Only nested list/tuple/dicts of objects that are valid for `z` should be passed.) rr4r=r@rr1itemsrG TypeError__name__)rJr7rBrCrIrKkvs` ```` rrGrGVs ,$ &       D' "tDz !JJL  Aq$!".7M`dj    4D*4*6** !$t*]4==/J++4+=+=*>>Q S   K s#C c t|ttfrt|fd|DSt|tr^tt rgngt ||jDcic]\}}||vr|nt|c}}St|dr |jS|Scc}}w#t$r|jcYSwxYw)a Recursively sends the elements in a nested list/tuple/dictionary of tensors to a given device. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to send to a given device. device (`torch.device`): The device to send the data to. Returns: The same data structure as `tensor` with all tensors sent to the proper device. c3<K|]}t|yw) non_blocking skip_keysN)send_to_device)r.tdevicerTrUs rr0z!send_to_device..s!ocd^AvLT]^^osrSto)rT) rr4r=r@rr-r1rLrVhasattrrYrM)rrXrTrUrOrWs ``` rrVrVs&5$-( ohno   FG $ i %" I  ItF|#LLN AqY1N1fS_kt,uu     %99V,9? ?   %99V$ $ %s7 C ,CC$#C$c d}t||S)aK Recursively gathers the information needed to rebuild a nested list/tuple/dictionary of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to send to analyze. Returns: The same data structure as `data` with [`~utils.TensorInformation`] instead of tensors. cDt|j|jS)N)shapedtype)r r]r^rs r_get_data_structurez/get_data_structure.._get_data_structures v||6<<HHrrF)r7r_s rget_data_structurer`sI 0$ 77rc d}t||S)a: Recursively gathers the shape of a nested list/tuple/dictionary of tensors as a list. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to send to analyze. Returns: The same data structure as `data` with lists of tensor shapes instead of tensors. c,t|jSr)r=r]rs r _get_shapezget_shape.._get_shapesFLL!!rrF)r7rcs r get_shaperds" Z ..rc,d}t||tS)z Recursively initializes tensors from a nested list/tuple/dictionary of [`~utils.TensorInformation`]. Returns: The same data structure as `data` with tensors instead of [`~utils.TensorInformation`]. cTtj|jd|jiSNr^)remptyr]r^r's r_initialize_tensorz.initialize_tensors.._initialize_tensors"{{K--G[5F5FGGrrB)rGr))data_structureris rinitialize_tensorsrlsH /K` aarc6t|ttfrt|dSt|tr%|j D]}t||cSn2t|t jstdt|d|jdS)a Recursively finds the batch size in a nested list/tuple/dictionary of lists of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to find the batch size. Returns: `int`: The batch size. rz0Can only find the batch size of tensors but got .) rr4r=find_batch_sizerkeysrrrMr1r])r7rOs rroros$ &tAw'' D' " ,A"47+ + , ell +J4PT:,VWXYY ::a=rc d}t||S)aS Recursively finds tensors in a nested list/tuple/dictionary and converts them to a list of numbers. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to convert to regular numbers. Returns: The same data structure as `data` with lists of numbers instead of `torch.Tensor`. c|jj}|jtjk(r|j tj }|jSr)detachcpur^rbfloat16rYfloat32tolistrs r_convert_to_listz!listify.._convert_to_listsF$$& <<5>> )YYu}}-F}}rrF)r7rxs rlistifyrys -t 44rcPd}t||d}tj|S)Nc|jdk(r|jd}|js|j}t j |S)Nr)ndimclone is_contiguous contiguousxm all_gatherrs r_tpu_gather_onez$_tpu_gather.._tpu_gather_one sI ;;! \\^D)F##%&&(F}}V$$rTrC)rGr mark_step)rrress r _tpu_gatherr s%% OV NCLLN Jrcttddrtjjntjj fd}t ||dS)Nz>=z1.13c|jdk(r|jd}|js|j}jjdk7rtt j j|jz|jj}|||jdg|jddStjDcgc]}t j|}}t jj!||t j"|dScc}w)Nrgloo)r^rXr dim)r|r}r~rbackendrrh num_processesnumelr^rXviewsizerange empty_like distributedrcat)routput_tensors_ gather_opstates r_gpu_gather_onez$_gpu_gather.._gpu_gather_one!s  ;;! \\^D)F##%&&(F == $&)@ #[[##flln4ll||N nf -&>&&r>FKKM!",=> > AFeFYFY@Z[1e..v6[N[    ( ( @99^3 3\s*D?Tr)rrrrall_gather_into_tensor_all_gather_baserG)rrrrs @@r _gpu_gatherrsL NEf%%%<< %%66 48 _f$ OOrceZdZdZy)DistributedOperationExceptionz An exception class for distributed operations. Raised if the operation cannot be performed due to the shape of the tensors. N)rN __module__ __qualname____doc__rrrr@s   rrc.tfd}|S)zv Verifies that `tensor` is the same shape across all processes. Only ran if `PartialState().debug` is `True`. c tjtjk(stjs |i|S j d j }d|vr|d}n|d}t|}t|g}|df|j|dt|k(}|sDdjt|Dcgc] \}}d|d|c}}} td|d|  |i|Scc}}w) Nrnrrz - zProcess z: znCannot apply desired operation due to shape mismatches. All shapes across devices must be valid. Operation: `z` Input shapes: - )rdistributed_typer NOdebugrrNrd gather_objectcountr3join enumerater) rIrK operationrshapesoutputare_sameir]process_shape_strfunctions rwrapperz!verify_operation..wrapperNs > * *o.@.@ @ H\H\T,V, ,**+1X->->,?@ v H%F!WF6"x( !9 ||F1I.#f+=H$,MM[dek[l2mxqRWXaS5'3J2m$n!3''0k1GHYGZ\ ((( 3nsC7 rrrs` rverify_operationrIs"  8_))* Nrc.tfd}|S)z Checks that `verify_operation` failed and if so reports a more helpful error chaining the existing `DistributedOperationException`. c |i|S#t$r0}jdj}td|d|d}~wwxYw)NrnzError found while calling `z1`. Please see the earlier error for more details.)rrrN)rIrKerrs rrz"chained_operation..wrappermsc T,V, ,, #../q1B1B0CDI/-i[8ij  s A+?Arrs` rchained_operationrgs"  8_ Nrctjtjk(r t |Stjt vr t |S|S)a4 Recursively gather tensor in a nested list/tuple/dictionary of tensors from all devices. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. Returns: The same data structure as `tensor` with all tensors sent to the proper device. )rrr TPUrr rrs rgatherrzsF~&&/*=*==6""  ( (,M M6"" robjectcttjDcgc]}d}}tjj |||Dcgc] }|D]}| c}}Scc}wcc}}wr)rrrrrall_gather_object)rroutput_objectsyxs r_gpu_gather_objectrs`$),.*F*F$GHqdHNH ''?% 1!q 1!A 1A 11I 2s A$A)ctjtjk(r t dtjt vr t |S|S)a5 Recursively gather object in a nested list/tuple/dictionary of objects from all devices. Args: object (nested list/tuple/dictionary of picklable object): The data to gather. Returns: The same data structure as `object` with all the objects sent to every device. z&gather objects in TPU is not supported)rrr rNotImplementedErrorr r)rs rrrsG~&&/*=*==!"JKK  ( (,M M!&)) rc(dd}t||d|S)NcHtjj|||S)Nsrc)rr broadcast)rrs r_gpu_broadcast_onez*_gpu_broadcast.._gpu_broadcast_ones  ##F#4 rT)rCrrrF)r7rrs r_gpu_broadcastrs /4UX YYrc Tt|ttfrt|fdt |DSt|t rCt ||jDcic]\}}|t|d|c}}Stj|fdScc}}w)Nc3HK|]\}}t|d|yw)rnameN)_tpu_broadcast)r.rrWrs rr0z!_tpu_broadcast..s*"gTQPQ>!TF!A3-#H#H"gs"rrc|Srr)rrs rz _tpu_broadcast..s !C&r) rr=r4r@rrr1rLrr mesh_reduce)rrrrOrPs `` rrrs&4-(&"gU^_eUf"ghh FG $tF|RXR^R^R`a$!QQq$q} EEabb >>$(8 99bs%B$ from_processctjtjk(rt ||dStjt vr t ||S|S)a Recursively broadcast tensor in a nested list/tuple/dictionary of tensors to all devices. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. from_process (`int`, *optional*, defaults to 0): The process from which to send the data Returns: The same data structure as `tensor` with all tensors broadcasted to the proper device. zaccelerate.utils.broadcast)rrr)rrr rrr r)rrs rrrsM~&&/*=*==f,=YZZ  ( (,M Mf,77 rc,tjtjk(r2t |D]"\}}t j d|fd||<$|Stjtvr!tjj||S)a Broadcast a list of picklable objects form one process to the others. Args: object_list (list of picklable objects): The list of objects to broadcast. This list will be modified inplace. from_process (`int`, *optional*, defaults to 0): The process from which to send the data. Returns: The same list containing the objects from process 0. z&accelerate.utils.broadcast_object_listc|Srr)rrs rrz'broadcast_object_list..sefgsetrr) rrr rrrrr rrbroadcast_object_list) object_listrrr>s ` rrrs~&&/*=*== , vFAs^^,TVY[tuKN v   ( (,M M // /N rc"d}t|||S)aN Recursively takes a slice in a nested list/tuple/dictionary of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to slice. tensor_slice (`slice`): The slice to take. Returns: The same data structure as `data` with all the tensors slices. c ||Srr)r tensor_slices r _slice_tensorz$slice_tensors.._slice_tensors l##rrF)r7r process_indexrrs r slice_tensorsrs$ ]D, ??rctdttfr.tdfdt t dDStdt rRtddjDcic]!}|tDcgc]}|| c}#c}}StdtjstdtdtjScc}wcc}}w)a Recursively concatenate the tensors in a nested list/tuple/dictionary of lists of tensors with the same shape. Args: data (nested list/tuple/dictionary of lists of tensors `torch.Tensor`): The data to concatenate. dim (`int`, *optional*, defaults to 0): The dimension on which to concatenate. Returns: The same data structure as `data` with all the tensors concatenated. rc3dK|]"}tDcgc]}|| c}$ycc}ww)rN) concatenate)r.rdr7rs rr0zconcatenate..s.#lSTKt0D!10D#$N$N#l0Ds 0 + 0rz%Can only concatenate tensors but got )rr4r=r@rr3rr1rprrrrMr)r7rrOrs`` rrrs$q'E4=)$q'#lX]^abfghbi^jXk#lmm DGW %tDG}UYZ[U\UaUaUcdPQaD-Aqad-As!KKdee Q .?T!W OPP 99Ts ##.BdsD  C= D =D c,dd}t||d|||S)a3 Recursively pad the tensors in a nested list/tuple/dictionary of tensors from all devices to the same size so they can safely be gathered. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. dim (`int`, *optional*, defaults to 0): The dimension on which to pad. pad_index (`int`, *optional*, defaults to 0): The value with which to pad. pad_first (`bool`, *optional*, defaults to `False`): Whether to pad at the beginning or the end. c> t|jk\r|Stj|j|jd}t |j }tfd|D |jk(r|S|j t } |<|jt||z}|r)t fdtt|D}n't fdtt|D}|||<|S)N)rXc3(K|] }| ywrr)r.srs rr0zFpad_across_processes.._pad_across_processes..&s-!qv-sc3bK|]&}|k(rtz n td(ywrslice)r.rrmax_sizeold_sizes rr0zFpad_across_processes.._pad_across_processes../s6[\Q#Xh#.95QU;Vs,/c3\K|]#}|k(rtdn td%yw)rNr)r.rrrs rr0zFpad_across_processes.._pad_across_processes..3s,oUVqCxE!Xc]3U4[Pos),) r3r]rrrXrrtmaxr= new_zerosr4r) rr pad_index pad_firstrsizesnew_size new_tensorindicesrrs ` @@r_pad_across_processesz3pad_across_processes.._pad_across_processess #fll# #M||FLL?Et   "-u-- v||C( (M<<>  %%eHo6B `efijrfs`tGoZ_`cdl`mZnooG$ 7rT)rCrrrrrFrF)rrrrrs rpad_across_processesr s%"2 v4ST]ir rc*dd}t||d||S)aX Recursively reduce the tensors in a nested list/tuple/dictionary of lists of tensors across all processes by the mean of a given operation. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to reduce. reduction (`str`, *optional*, defaults to `"mean"`): A reduction method. Can be of "mean", "sum", or "none" scale (`float`, *optional*): A default scaling value to be applied after the reduce, only valied on XLA. Returns: The same data structure as `data` with all the tensors reduced. ct}|j}|jtjk(r|S|jtj k(rt jd||nJ|jjtvr.tjj|tj|dk(r||jz}|S)Nsummean)rr}rr rrr all_reducevaluer rrrSUMr)r reductionscaler cloned_tensors r_reduce_across_processesz(reduce.._reduce_across_processesNs   ! !_%7%7 7  ! !_%8%8 8 MM% 6  # # ) )-N N    ( ( E   U00 0MrT)rCrrrg?rF)rrrrs rreducer <s"$   &di_d rc*d}d}t|||S)av Recursively converts the elements nested list/tuple/dictionary of tensors in FP16/BF16 precision to FP32. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to convert from FP16/BF16 to FP32. Returns: The same data structure as `tensor` with all tensors that were in FP16/BF16 precision converted to FP32. c"|jSr)floatrs r_convert_to_fp32z)convert_to_fp32.._convert_to_fp32ls||~rcvt|dxr,|jtjtjfvSrg)rZr^rfloat16rurs r_is_fp16_bf16_tensorz-convert_to_fp32.._is_fp16_bf16_tensoros*vw'[FLLU]]ENN<[,[[rrjrF)rr rs rconvert_to_fp32r`s \ -vAU VVrc"eZdZdZdZdZdZy)ConvertOutputsToFp32ad Decorator to apply to a function outputing tensors (like a model forward pass) that ensures the outputs in FP16 precision will be convert back to FP32. Args: model_forward (`Callable`): The function which outputs we want to treat. Returns: The same function as `model_forward` but with converted outputs. c*||_t||yr) model_forwardr)selfrs r__init__zConvertOutputsToFp32.__init__s*t]+rc8t|j|i|Sr)rr)rrIrKs r__call__zConvertOutputsToFp32.__call__s 1t114B6BCCrc,tjd)NzCannot pickle a prepared model with automatic mixed precision, please unwrap the model with `Accelerator.unwrap_model(model)` before pickling it.)pickle PicklingError)rs r __getstate__z!ConvertOutputsToFp32.__getstate__s"" `  rN)rNrrrrrrrrrrrus ,D rrc6tfd}|_|S)Nc|i|Srr)rIrKrs rforwardz(convert_outputs_to_fp32..forwardsd-f--r)r __wrapped__)rr s` rconvert_outputs_to_fp32r"s!(7M.(G Nrct|tr'|jD]}t|}||cSyt|tt fr|D]}t|}||cSyt|t jr |jSy)z Finds the device on which a nested dict/list/tuple of tensors lies (assuming they are all on the same device). Args: (nested list/tuple/dictionary of `torch.Tensor`): The data we want to know the device of. N) rrvalues find_devicer4r=rrrX)r7r>rXs rr%r%s$ ;;= C %F!   D5$- ( C %F!   D%,, '{{ (r)FNr)rzbroadcast tensor)NNrr)r1sY+ 8;WW'))"#*, 6 =$4CX]0f"J8$/$ b(5. #PL I <&&2s2#&Z:C*S*@($,++\  FW*  4 r