;iNZSSKrSSKrSSKJr SSKJrJrJr SSKrSSK J r SSK J r J r JrJr SSKJr SSKJrJrJrJr SS KJrJrJrJrJrJrJrJrJ r J!r!J"r"J#r#J$r$J%r% \"\&5r'SS SSSSS S SSSSS S S S .r(SSS00r)\)RU5H'ur+r,\!"S\+5(dM\(R[\,5 M) "SS\5r."SS\ 5r/"SS\5r0"SS5r1"SS5r2"SS\2\15r3\"5(aSSK4J5s J6r7 "SS\7Rp5r9"S S!\2\15r:S"r;S8S#\ S$\\RxS%\\=S&\\=S'\>S(\>S)\\?\\@\4S*\\>S+\>S,\\S-\>S.\\=S/\>S0\>S1\ 4S2jjrA"S3S4\ 5rB"S5S6\2\15rCS9S7jrDg):N)suppress)CallableOptionalUnion)version) BatchSampler DataLoaderIterableDataset RandomSampler) get_logger)DistributedType GradientState PartialStateis_torch_xla_available)RNGType broadcastbroadcast_object_listcompare_versions concatenatefind_batch_sizeget_data_structureinitialize_tensorsis_datasets_availableis_torch_version*is_torchdata_stateful_dataloader_availablesend_to_device slice_tensorssynchronize_rng_statesF) batch_sizeshufflesampler batch_sampler num_workers collate_fn pin_memory drop_lasttimeoutworker_init_fnmultiprocessing_context generatorprefetch_factorpersistent_workerspin_memory_device2.6.0in_orderT>=cF^\rSrSrSrU4SjrU4SjrS\4SjrSr U=r $)SeedableRandomSamplerIax Same as a random sampler, except that in `__iter__` a seed can be used. Needed specifically in distributed cases, when the random generator for each GPU needs to start from the same seed and be fully reproducible on multiple iterations. If a custom `generator` is passed, it will rely on its initial seed as well as the current iteration it is on (stored in `self.epoch`). c>URSS5n[TU]"U0UD6 UbUO[RR 5UlSUlg)N data_seedr)popsuper__init__torchrandom initial_seedepoch)selfargskwargsr8 __class__s f/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/accelerate/data_loader.pyr;SeedableRandomSampler.__init__TsHJJ{D1  $)&))2)>IELLD]D]D_ c#># URch[R"[[S5(a[R"5OSS9UlURR UR 5 URUR -nURR U5 [TU]%5ShvN URURS-5 gN#7f)Nget_default_devicecpudevicer ) r-r< GeneratorhasattrrH manual_seedr>r?r:__iter__ set_epoch)r@seedrCs rDrOSeedableRandomSampler.__iter__[s >> !"__5C"?$C$r?cXlg)z*Sets the current iteration of the sampler.N)r?r@r?s rDrPSeedableRandomSampler.set_epochis rF)r?r-r>) __name__ __module__ __qualname____firstlineno____doc__r;rOintrP__static_attributes__ __classcell__rCs@rDr5r5Is# 'srFr5c j\rSrSrSrSS\S\S\S\S\4 Sjjr\ S 5r S r S r S r S rSrg)BatchSamplerShardna' Wraps a PyTorch `BatchSampler` to generate batches for one of the processes only. Instances of this class will always yield a number of batches that is a round multiple of `num_processes` and that all have the same size. Depending on the value of the `drop_last` attribute of the batch sampler passed, it will either stop the iteration at the first batch that would be too small / not present on all processes or loop with indices from the beginning. Args: batch_sampler (`torch.utils.data.sampler.BatchSampler`): The batch sampler to split in several shards. num_processes (`int`, *optional*, defaults to 1): The number of processes running concurrently. process_index (`int`, *optional*, defaults to 0): The index of the current process. split_batches (`bool`, *optional*, defaults to `False`): Whether the shards should be created by splitting a batch to give a piece of it on each process, or by yielding different full batches on each process. On two processes with a sampler of `[[0, 1, 2, 3], [4, 5, 6, 7]]`, this will result in: - the sampler on process 0 to yield `[0, 1, 2, 3]` and the sampler on process 1 to yield `[4, 5, 6, 7]` if this argument is set to `False`. - the sampler on process 0 to yield `[0, 1]` then `[4, 5]` and the sampler on process 1 to yield `[2, 3]` then `[6, 7]` if this argument is set to `True`. even_batches (`bool`, *optional*, defaults to `True`): Whether or not to loop back at the beginning of the sampler when the number of samples is not a round multiple of (original batch size / number of processes). `BatchSampler`s with varying batch sizes are not enabled by default. To enable this behaviour, set `even_batches` equal to `False` r% num_processes process_index split_batches even_batchescHU(a/URU-S:wa[SURSUS35eXlX lX0lX@lXPl[USS5Ul[USS5UlURcUR (a [S5egg) NrzDTo use `BatchSamplerShard` in `split_batches` mode, the batch size (;) needs to be a round multiple of the number of processes ().r"r)FzYou need to use `even_batches=False` when the batch sampler has no batch size. If you are not calling this method directly, set `accelerator.even_batches=False` instead.) r" ValueErrorr%rbrcrdregetattrr))r@r%rbrcrdres rDr;BatchSamplerShard.__init__s ]55 EJVWdWoWoVpqLLY?Z\^ +***(!-tD  UC ?? "t'8'8f (9 "rFc,[UR5$Nlenr%r@s rD total_lengthBatchSamplerShard.total_length4%%&&rFcUR(a[UR5$[UR5UR-S:Xa"[UR5UR-$[UR5UR-nUR(aU$UR (aUS-$UR [UR5UR-:aUS-$U$Nrr )rdror%rbr)rercr@lengths rD__len__BatchSamplerShard.__len__s   t))* * t!! "T%7%7 71 <t))*d.@.@@ @T''(D,>,>> >>M   A: "&!3!3c$:L:L6MPTPbPb6b!b6A: nhn nrFcdUR(aUR5$UR5$rm)rd_iter_with_split_iter_with_no_splitrps rDrOBatchSamplerShard.__iter__s'*.*<*>c,>,>&>% '''4+=+==!&'''4+=+=+AA'3u:+H$$!#8~~~#l"3a"7$$~&*((+~&$//9((,'$*<*H#$H#,B?H#-BH#H#H#)r%r"r)rerbrcrdN)r rFT)rVrWrXrYrZrr[boolr;propertyrqrxrOr{r|r\rFrDr`r`nsv J#! #     4''o$]i,-rFr`c Z\rSrSrSrSS\S\S\S\S\S\4 S jjrS r S r S r S r g)IterableDatasetShardi a; Wraps a PyTorch `IterableDataset` to generate samples for one of the processes only. Instances of this class will always yield a number of samples that is a round multiple of the actual batch size (depending of the value of `split_batches`, this is either `batch_size` or `batch_size x num_processes`). Depending on the value of the `drop_last` attribute of the batch sampler passed, it will either stop the iteration at the first batch that would be too small or loop with indices from the beginning. Args: dataset (`torch.utils.data.dataset.IterableDataset`): The batch sampler to split in several shards. batch_size (`int`, *optional*, defaults to 1): The size of the batches per shard (if `split_batches=False`) or the size of the batches (if `split_batches=True`). drop_last (`bool`, *optional*, defaults to `False`): Whether or not to drop the last incomplete batch or complete the last batches by using the samples from the beginning. num_processes (`int`, *optional*, defaults to 1): The number of processes running concurrently. process_index (`int`, *optional*, defaults to 0): The index of the current process. split_batches (`bool`, *optional*, defaults to `False`): Whether the shards should be created by splitting a batch to give a piece of it on each process, or by yielding different full batches on each process. On two processes with an iterable dataset yielding of `[0, 1, 2, 3, 4, 5, 6, 7]`, this will result in: - the shard on process 0 to yield `[0, 1, 2, 3]` and the shard on process 1 to yield `[4, 5, 6, 7]` if this argument is set to `False`. - the shard on process 0 to yield `[0, 1, 4, 5]` and the sampler on process 1 to yield `[2, 3, 6, 7]` if this argument is set to `True`. datasetr"r)rbrcrdcU(a US:aX$-S:wa[SUSUS35eXlX lX0lX@lXPlX`lg)Nr rzGTo use `IterableDatasetShard` in `split_batches` mode, the batch size (rgrh)rirr"r)rbrcrd)r@rr"r)rbrcrds rDr;IterableDatasetShard.__init__+se Z!^ 0Ja0OYZdYefLLY?Z\^ )0 $"***rFc~Xl[URS5(aURRU5 ggNrP)r?rMrrPrTs rDrPIterableDatasetShard.set_epoch@s/ 4<< - - LL " "5 ) .rFc<UR(a<[UR5URUR--UR-$[ R "[UR5URUR-- 5UR-$rm)r)rorr"rbmathceilrps rDrxIterableDatasetShard.__len__Esp >> %$//D Inheriters of this class should ensure that the class creates a `GradientState()` instance, stored in `self.gradient_state`. c SUlSUlgNFend_of_dataloader remainder)clsrBs rD__init_subclass__&DataLoaderStateMixin.__init_subclass__s % rFc SUlSUlgrrrps rDresetDataLoaderStateMixin.resets!&rFcHUR5 [[5 UR(d=[ UR S[ UR 55nXR-UlSSS5 URRU5 g!,(df  N*=f)z6Prepares the gradient state for the current dataloadertotal_dataset_lengthN) rr Exception _drop_lastrjrrototal_batch_sizergradient_state_add_dataloaderrvs rDbeginDataLoaderStateMixin.beginsj i ?? /Es4<$ 5 7 7 H "+U+W+Wr  # ' i.@.@.H.H.U V f$$%6VDD$T733 :g#e]#e^d#eD #-g#]]#]V\#]D 4'' 6 6!%!5!5!@!@!BD  7rFcNUS:Xa [5e[URU5$)Nr)AttributeErrorrjr)r@names rD __getattr__DataLoaderAdapter.__getattr__s' $ $ " "t++T22rFcUR$rm)rrps rDrDataLoaderAdapter.state_dicts!!!rFc:URRU5 grm)rload_state_dict)r@rs rDr!DataLoaderAdapter.load_state_dicts ,,Z8rFc.URR$)z In order to maintain backwards compatibility with other code, we need to ensure `isinstance(obj, DataLoader)` returns true. This is because some downstream code assumes that the `DataLoader` is the base class of the object. )rrCrps rDrCDataLoaderAdapter.__class__s##---rFc,[UR5$rm)rorrps rDrxDataLoaderAdapter.__len__s4''((rFc[5R[R:wa[5RS- nUR SS:aUR S==U-ss'UR SS:aUR S==U-ss'UR SbSSUR S;a?UR SSS:a(UR SS==UR U--ss'ggggg)a Adjusts the state dict for prefetching. Natively, this will adjust all of the iters yielded keys in `self.dl_state_dict` by a factor of `num_processes - 1`, however if a custom correction is needed, this can be overridden. This should modify `self.dl_state_dict` directly r _sampler_iter_yieldedr _num_yielded_index_sampler_stateNsamples_yielded)rdistributed_typerNOrbrr")r@factors rDadjust_state_dict_for_prefetch0DataLoaderAdapter.adjust_state_dict_for_prefetchs > * *o.@.@ @!^11A5F!!"9:Q>""#:;vE;!!.1A5"">2f<2!!"89E%););?PQUYUdUdgmUmmQZTF ArFc[URS5(aIURR5UlUR 5 UR URS'gg)Nr_iterator_finished)rMrrrrrrps rD_update_state_dict$DataLoaderAdapter._update_state_dictsV 4'' 6 6!%!5!5!@!@!BD   / / 17;7M7MD  3 4 7rF)rrr)FN)rVrWrXrYrZr;rrrrrCrxrrr\rrFrDrrsA C03"9..)n. NrFrc^\rSrSrSrSS\S\4U4SjjjrSrU4SjrS\ 4S jr \ S 5r \ S 5r S rS rSrU=r$)DataLoaderShardia Subclass of `DataLoaderAdapter` that will deal with device placement and current distributed setup. Args: dataset (`torch.utils.data.dataset.Dataset`): The dataset to use to build this dataloader. device (`torch.device`, *optional*): If passed, the device to put all batches on. rng_types (list of `str` or [`~utils.RNGType`]): The list of random number generators to synchronize at the beginning of each iteration. Should be one or several of: - `"torch"`: the base torch random number generator - `"cuda"`: the CUDA random number generator (GPU only) - `"xla"`: the XLA random number generator (TPU only) - `"generator"`: an optional `torch.Generator` synchronized_generator (`torch.Generator`, *optional*): A random number generator to keep synchronized across processes. skip_batches (`int`, *optional*, defaults to 0): The number of batches to skip at the beginning. use_stateful_dataloader (`bool`, *optional*, defaults to `False`): Whether to have this class adapt `StatefulDataLoader` from `torchdata` instead of the regular `DataLoader`. **kwargs (additional keyword arguments, *optional*): All other keyword arguments to pass to the regular `DataLoader` initialization. **Available attributes:** - **total_batch_size** (`int`) -- Total batch size of the dataloader across all processes. Equal to the original batch size when `split_batches=True`; otherwise the original batch size * the total number of processes - **total_dataset_length** (`int`) -- Total length of the inner dataset across all processes. r _non_blockingc  >[T U]"U4SU0U D6 X lX0lX@lXPl[ 5UlXplXl SUl g)Nrr) r:r;rK rng_typessynchronized_generator skip_batchesrrrr iteration) r@rrKrrrrrrtorch_device_meshrBrCs rDr;DataLoaderShard.__init__sP \:Q\U[\ "&<#(+o$*rFc## URb [URUR5 UR5 UR UR 5 UR R5n[U5nSnURb[X RURS9nUR5 [U5nX0R:aUv US- nUnMc![a UR5 gf=f![a- SUlUR5 X0R:aUv Of=fU=R S- slUR5 g7f)NrT non_blockingr )rrrrrPrrrOnext StopIterationrrKrrrrr)r@dataloader_iterr batch_index next_batchs rDrODataLoaderShard.__iter__.s: >> % "4>>43N3N O  t~~&..779  1M   ;;*$2=++\`\n\n$oM'')!/2 "3"33''q *    HHJ   ! )-&'')"3"33''   !  sOA3E'6 C'E'A D%E''DE'DE'4D>;E'=D>>)E'c:>[TU]5n[/USSQ7$)z Define the `__reduce__` method to ensure a `DataLoaderShard` can be pickled and unpickled. This needs to be explicitly defined since default pickling behavior is broken by `DataLoaderAdapter` messing with its `__class__` member. r N)r: __reduce__rr@rArCs rDrDataLoaderShard.__reduce__Rs& w!#+$qr(++rFr?cURU:waXl[URS5(aURRU5 [URS5(aJ[URRS5(a%URRRU5 [URS5(a[URRS5(a_[URRRS5(a0URRRRU5 g[UR S5(aUR RU5 gg)NrPr$r%)rrMr%rPr$rrTs rDrPDataLoaderShard.set_epoch[s >>U ""N 4%%{ 3 3    ( ( / 4%%y 1 1gd>P>P>X>XZe6f6f    & & 0 0 7 D&& 8 8**88)DD**88@@+NN    , , 4 4 > >u ET\\; / / LL " "5 )0rFc[UR[5(a URO URn[ USS5(a UR $UR [ USS5-$)NrdFrbr )rr$rr%rjr")r@r%s rDr DataLoaderShard.total_batch_sizensd(24<<(N(N TXTfTf }ou==  $ $  **W]OUV-WW rFc[URS5(aURR$[UR5$)Nrq)rMrrqrorps rDr$DataLoaderShard.total_dataset_lengthws2 4<< 0 0<<,, ,t||$ $rFc[U5$rm get_samplerrps rDr DataLoaderShard.get_sampler~ 4  rFc[UR[5nU(aXRlgXRl[ URS5(aXRRlggNr%rr$rr%rMr@r$sampler_is_batch_samplers rD set_samplerDataLoaderShard.set_samplerV#-dllL#I ##*LL )0   &t))?;;;B""008[TU]X5 URRUlSURlX lgrm)r:r;_loaderr _rng_typesrK)r@rrKrCs rDr;MpDeviceLoaderWrapper.__init__s2 G Z 0"ll44DO%)DLL " KrFc>URb*[URURR5 [TU]5$rm)rrrrr:rO)r@rCs rDrOMpDeviceLoaderWrapper.__iter__s3*&t 8[8[\7#% %rFr?cr[URS5(aURRU5 ggr)rMrrPrTs rDrPMpDeviceLoaderWrapper.set_epochs*t 44))%05rFc.URR$rm)rrrps rDr&MpDeviceLoaderWrapper.total_batch_sizes<<00 0rFc.URR$rm)rrrps rDr*MpDeviceLoaderWrapper.total_dataset_lengths<<44 4rFc.URR$rm)rr%rps rDr%#MpDeviceLoaderWrapper.batch_samplers<<-- -rFcUR$rm)rrps rDr MpDeviceLoaderWrapper.dataloaders << rF)rrK)rVrWrXrYrZrr<rKr;rOr[rPrrrr%rr\r]r^s@rDrrs  ! !  !  &  13 1  1  1  5  5  .  .    rFrc^\rSrSrSrSS\S\S\4U4SjjjrSrSrS \ 4S jr S r U4S jr \ S 5r\ S5rSrSrSrU=r$)DataLoaderDispatcheria Subclass of `DataLoaderAdapter` that will iterate and preprocess on process 0 only, then dispatch on each process their part of the batch. Args: split_batches (`bool`, *optional*, defaults to `False`): Whether the resulting `DataLoader` should split the batches of the original data loader across devices or yield full batches (in which case it will yield batches starting at the `process_index`-th and advancing of `num_processes` batches at each iteration). Another way to see this is that the observed batch size will be the same as the initial `dataloader` if this option is set to `True`, the batch size of the initial `dataloader` multiplied by `num_processes` otherwise. Setting this option to `True` requires that the batch size of the `dataloader` is a round multiple of `batch_size`. skip_batches (`int`, *optional*, defaults to 0): The number of batches to skip at the beginning of an iteration. use_stateful_dataloader (`bool`, *optional*, defaults to `False`): Whether to have this class adapt `StatefulDataLoader` from `torchdata` instead of the regular `DataLoader`. **Available attributes:** - **total_batch_size** (`int`) -- Total batch size of the dataloader across all processes. Equal to the original batch size when `split_batches=True`; otherwise the original batch size * the total number of processes - **total_dataset_length** (`int`) -- Total length of the inner dataset across all processes. rdrrc  h>Sn SSKJn [X5(a URn [T U]"U4SU0U D6 X lU (a1[RRRRXS9 [5Ul [5UlXPlX`lX0lXlUc[(OUUlSUlSUlSUlSUlUR&(aSUR&R4;apUR&SUlSUR&R4;aUR&SUlSUR&R4;aUR&SUlUR.(a.UR0(dUR2(a [7S 5egg) NFr)ShufflerIterDataPiper)r#tpdpfsdpz4TP + (DP/FSDP) is not yet supported in dispatch mode)-torch.utils.data.datapipes.iter.combinatoricsr.r_shuffle_enabledr:r;rdr<utilsdatagraph_settingsapply_shuffle_settingsrrrstaterrrrrslice_fnr submesh_tp submesh_dp submesh_fsdpmesh_dim_namesri) r@rrdrrrrr9rrBr#r.rCs rDr;DataLoaderDispatcher.__init__scV g 4 4..G \:Q\U[\*  KK   + + B B7 B \+o!^ $*(!2)1)9 x    ! !dd.D.D.S.S&S"44T:DOt--<<<"&"8"8">//>>>$($:$:6$B! ??43D3DST T4E?rFcSup#URRS:XGaUR(aBUR(a[R S5 UR 5 [U5nO/nUR(a6UR 5 [U5nU/URR-nOP[URR5H-nUR 5 UR[U55 M/ [USS9n[U5S/nOSUR/n[!U5 USUlUR(arUR(daUR"(dPURRS:Xa'[%U5S:a[USS9n[U5S/nOSS/n[!U5 X64$![an[S5UeSnAff=f![a SS/nNf=f) N)NNrzUse of split_batches for TP would need the dataloader to produce duplicate batches,otherwise, use dispatch_batches=True instead.dimaGYou can't use batches of different size with `dispatch_batches=True` or when using an `IterableDataset`.either pass `dispatch_batches=False` and have each process fetch its own batch or pass `split_batches=True`. By doing so, the main process will fetch a full batch and slice it into `num_processes` batches for each process.FTr )r8rcrdr:loggerwarningrrrbrrr RuntimeErrorrr_stop_iterationrrro)r@iteratorbatchesr_e batch_infos rD_fetch_batches#DataLoaderDispatcher._fetch_batchess# :: # #q ( ( *%%L++- NE!G//1 $X#('DJJ,D,D"D!&tzz'?'?!@A 335#NN4>:"A! +G ;17?  4 45Jj))!}   %%doo::++q0S\A5E'Q7E"4U";U!CJ"&J%j1  7(!*V ! !!! *"D\  *s6C,G- G G- G* G%%G**G--G>=G>c#:# UR5 URUR5 Sn[SS5(aURR 5nO4UR RS:XaURR 5nSnSUlSnURU5upESnU(Gd=XEpUR RS:wa[US5n[XpR RURS9n[USS9nUR(d[UcXUR!U[#SUR R$5UR RUR R$S9nUc['SUS 35e[)U5n XR R$-n URnU(d,URU5upEUR(aUScS nUR(d3U(a,XR R$-S:wa[+Xs/SS 9nU S - n [#UR RU -UR RS -U -5n UR!UU UR RUR R$S9nU(aS UlUR/5 XlX`R2:aUv US - nU(dGM=U=RS - slUR55 g7f) Nr3z2.0.1rFr) from_process)rcrbz"Batch does not contain any data (`zM`). At the end of all iterable data available before expected stop iteration.Tr@r )rrPrrrrOr8rcrErKrrrKrrrr9slicerbrirrrrrrr) r@ main_iteratorstop_iterationrrnext_batch_inforrrJobserved_batch_sizer" data_slices rDrODataLoaderDispatcher.__iter__Ts  t~~& D' * *!0099;M ZZ % % * 0099;M$ &*&9&9-&H#   *:zz''1,*:a=9"5***;*;$J\J\]Ee!4E??{':"mm!TZZ556"&**":":"&**":": , } 8@MN#2%"8 , 0H0HHJ!11N!/3.A.A-.P+ ''OA,>,F%)N??~:MPZPZPhPh:hlm:m#U$8a@a tzz77*DtzzG_G_bcGcgqFqrJMM"jj66"jj66 "E)-&'')!4/// 1 Kq!.r !  s K0L5&Lr?cjURU:waXl[URS5(aK[URRS5(a&URRR U5 g[UR S5(aUR R U5 gg)Nr$rP)rrMr%r$rPrrTs rDrPDataLoaderDispatcher.set_epochs >>U ""N 4%%y 1 1gd>P>P>X>XZe6f6f    & & 0 0 7 T\\; / / LL " "5 )0rFc[UR5nUR(aU$UR(aXRR -$[ R"XRR - 5$rm)rorrdrr8rbrr)r@ whole_lengths rDrxDataLoaderDispatcher.__len__sW4//0     __::#;#;; ;99\JJ,D,DDE ErFc:>[TU]5n[/USSQ7$)z Define the `__reduce__` method to ensure a `DataLoaderDispatcher` can be pickled and unpickled. This needs to be explicitly defined since default pickling behavior is broken by `DataLoaderAdapter` messing with its `__class__` member. r N)r:rr,rs rDrDataLoaderDispatcher.__reduce__s& w!#$0tABx00rFcUR(aURR$URRURR-$rm)rdrr"rbrps rDr%DataLoaderDispatcher.total_batch_sizesB(,'9'9DLL # # @D @W@WZ^ZfZfZtZt@t rFc,[UR5$rm)rorrps rDr)DataLoaderDispatcher.total_dataset_lengths4<<  rFc[U5$rmr rps rDr  DataLoaderDispatcher.get_samplerrrFc[UR[5nU(aXRlgXRl[ URS5(aXRRlggrrrs rDr DataLoaderDispatcher.set_samplerrrF)rrrErrrrrr9rdr8r;r<r:r)FrFFFNN)rVrWrXrYrZrr;rKrOr[rPrxrrrrr rr\r]r^s@rDr,r,s:$ % #5U5U  5U5U5Un@!DJX*s*F1  !!!CCrFr,c[UR[5nU(a[URSS5nU$[URSS5nU$)z Get the sampler associated to the dataloader Args: dataloader (`torch.utils.data.dataloader.DataLoader`): The data loader to split across several devices. Returns: `torch.utils.data.Sampler`: The sampler associated to the dataloader r$N)rr$rrjr%)rrr$s rDr r sP ***<* N*22ItD NrFrrKrbrcrd put_on_devicerdispatch_batchesreslice_fn_for_dispatchuse_seedable_samplerr8rrreturncUc$U(dSnO[UR[5nU(aU(d [S5e[ 5nUc UR nUc UR nU(aUR[R:Xa0SnSUR;aUSR5nUU-nUU-nOSnSnSnSnSUR;aUSR5nSUR;aUSR5nSUR;aUSR5nSUR;aUSR5nUUU--nUU-nU(aURb URnOT[URS 5(aURRnO"[S [UR5S 35eUS:a%UU-S :wa[S URSUS35eURn[U[5(d UROSn[UR ["5nSn[%U5n[U[&5(ayU (ar[)UR*UR,UR.[1US[2R4"[5(a[2R6"5OSS95U S9n[UR [&5(aUR[R8:Xa[2R4"[5(a[2R6"5OSS9n[;[2R<"S[2R>S9RA5RC55nUREU5 UUl#UUR l#US:wdUR[RH:XGaU(Gd[K5(aS SK&Jn [K5(a9[UW5(a(U(d!URNU:aURQX#S9nGOj[U[5(aR[1URSS5bURRFn[SUURURTUUUS9nGOU (d[US5(aURFc[2R4"[5(a[2R6"5OSS9Ul#[;[2R<"S[2R>S9RA5RC55nURFREU5 URFnU(a UR O URn[WUUUUUS9n/SQnUbUcSU;aURYS5 [ZVs0sH nUU;dM U[1UU[ZU5_M" n nUc;URTU S'U(aU(dURU-O URU S 'U(a0U R]S5 [_U4UUURTU U U US.U D6nOU(aQ[aU4U(a UR[R8:waUOSUURUURTU UU S.U D6nOE[aU4U(a UR[R8:waUOSUUUURTU U S.U D6n[U[(5(aU (aURcU5 UR[R8:Xa [eX5$U$s snf) aJ Wraps a PyTorch `DataLoader` to generate batches for one of the processes only. Depending on the value of the `drop_last` attribute of the `dataloader` passed, it will either stop the iteration at the first batch that would be too small / not present on all processes or loop with indices from the beginning. Args: dataloader (`torch.utils.data.dataloader.DataLoader`): The data loader to split across several devices. device (`torch.device`): The target device for the returned `DataLoader`. num_processes (`int`, *optional*): The number of processes running concurrently. Will default to the value given by [`~state.PartialState`]. process_index (`int`, *optional*): The index of the current process. Will default to the value given by [`~state.PartialState`]. split_batches (`bool`, *optional*, defaults to `False`): Whether the resulting `DataLoader` should split the batches of the original data loader across devices or yield full batches (in which case it will yield batches starting at the `process_index`-th and advancing of `num_processes` batches at each iteration). Another way to see this is that the observed batch size will be the same as the initial `dataloader` if this option is set to `True`, the batch size of the initial `dataloader` multiplied by `num_processes` otherwise. Setting this option to `True` requires that the batch size of the `dataloader` is a round multiple of `batch_size`. put_on_device (`bool`, *optional*, defaults to `False`): Whether or not to put the batches on `device` (only works if the batches are nested list, tuples or dictionaries of tensors). rng_types (list of `str` or [`~utils.RNGType`]): The list of random number generators to synchronize at the beginning of each iteration. Should be one or several of: - `"torch"`: the base torch random number generator - `"cuda"`: the CUDA random number generator (GPU only) - `"xla"`: the XLA random number generator (TPU only) - `"generator"`: the `torch.Generator` of the sampler (or batch sampler if there is no sampler in your dataloader) or of the iterable dataset (if it exists) if the underlying dataset is of that type. dispatch_batches (`bool`, *optional*): If set to `True`, the dataloader prepared is only iterated through on the main process and then the batches are split and broadcast to each process. Will default to `True` when the underlying dataset is an `IterableDataset`, `False` otherwise. even_batches (`bool`, *optional*, defaults to `True`): If set to `True`, in cases where the total batch size across all processes does not exactly divide the dataset, samples at the start of the dataset will be duplicated so the batch can be divided equally among all workers. slice_fn_for_dispatch (`Callable`, *optional*`): If passed, this function will be used to slice tensors across `num_processes`. Will default to [`~utils.slice_tensors`]. This argument is used only when `dispatch_batches` is set to `True` and will be ignored otherwise. use_seedable_sampler (`bool`, *optional*, defaults to `False`): Whether to use the [`~data_loader.SeedableRandomSampler`] instead of a `RandomSampler` for better reproducibility. Comes at a cost of potentially different performances due to different shuffling algorithms but ensures results will be the *exact* same. Should be paired with `set_seed()` at every `self.set_epoch` data_seed (`int`, *optional*, defaults to `None`): The seed to use for the underlying generator when using `use_seedable_sampler`. If `None`, the generator will use the current default seed from torch. non_blocking (`bool`, *optional*, defaults to `False`): If set to `True`, dataloader will utilize non-blocking host-to-device transfers. If the dataloader has `pin_memory` set to `True`, this will help to increase overlap between data transfer and computations. use_stateful_dataloader (`bool`, *optional*, defaults to `False`): "If set to true, the dataloader prepared by the Accelerator will be backed by " "[torchdata.StatefulDataLoader](https://github.com/pytorch/data/tree/main/torchdata/stateful_dataloader). This requires `torchdata` version 0.8.0 or higher that supports StatefulDataLoader to be installed." torch_device_mesh (`torch.distributed.DeviceMesh`, *optional*, defaults to `None`): PyTorch device mesh. Returns: `torch.utils.data.dataloader.DataLoader`: A new data loader that will yield the portion of the batches `BatchSampler`s with varying batch sizes are not enabled by default. To enable this behaviour, set `even_batches` equal to `False` NFz%> >  O(777"3D"9">">"@)_">"@(777"3D"9">">"@!2!A!AA"3N"C"H"H"J.===$5j$A$F$F$H!)o.OPM-?M  ,#-#8#8 z//>>'1'?'?'J'J$ j66788hj ! #(<}(LPQ(QQR\RgRgQhiLLY?Z\^  $$K8B;P_8`8` 00fj)**<*\rSrSrSrS SjrSr\S5rSr Sr g) SkipBatchSampleri z A `torch.utils.data.BatchSampler` that skips the first `n` batches of another `torch.utils.data.BatchSampler`. Should not be used if the original dataloader is a `StatefulDataLoader`. cXlX lgrmr%r)r@r%rs rDr;SkipBatchSampler.__init__&s*(rFc#r# [UR5HupXR:dMUv M g7frm)rr%r)r@rtsampless rDrOSkipBatchSampler.__iter__*s-'(:(:;NE))) [TU]"U4SU0UD6 X l[5Ulg)Nr)r:r;rrr)r@rrrrBrCs rDr;SkipDataLoader.__init__Es, \:Q\U[\(+orFc## UR5 [URR55H*upXR:dMUR 5 Uv M, UR 5 g7frm)rrrrOrrr)r@rtrs rDrOSkipDataLoader.__iter__JsW %d&:&:&C&C&EFLE)))'') G  s AA5 )A5cF[UR5UR- $rm)rorrrps rDrxSkipDataLoader.__len__Rs4''(4+<+<<[TU]5n[/USSQ7$)z Define the `__reduce__` method to ensure a `SkipDataLoader` can be pickled and unpickled. This needs to be explicitly defined since default pickling behavior is broken by `DataLoaderAdapter` messing with its `__class__` member. r N)r:rrrs rDrSkipDataLoader.__reduce__Us& w!#*ab**rF)rr)rF) rVrWrXrYrZr;rOrxrr\r]r^s@rDrr7s! . =++rFrc [5nUR[R:XaURnUR nUR nSn[U[5(aSnOB[UR[5nU(a URO URn[XqS9n/SQn[V s0sHn X;dM U [X [U 5_M n n UcURU S'UR U S'[U["5(a,UcXS'[#U4UR$UUR&S.U D6nO[U[(5(aVUcXS'OU(aXjS 'UR U S'OXjS '[)U4URUR*UR,S .U D6nO Uc[/U4SU0U D6nO[1U4S U0U D6nUR[R:Xa [3UW5nU$s sn f) z Creates a `torch.utils.data.DataLoader` that will efficiently skip the first `num_batches`. Should not be used if the original dataloader is a `StatefulDataLoader`. FN)rrur)r"r)rdr%rr$r%)rKrr)rrrrzrKrrrr r$rr%rrrjr)r"r,rdrrrrrr r) r num_batchesr8rKrrrr%rrrBs rDskip_first_batchesr_s NE !4!44""**   G$'?++ #-j.@.@,#O .F **JLdLd ,]UM,+A ! A7:" ")  $22+!,,     J 0 0  $%0> " % 19 #-#8#8F< &7? #$  $$ **#-#D#D      $'TkTVTJ#GW;LWPVWJ !4!44*:v> as 5 G5G5)NNNFFNNTNFNFFNr)Err contextlibrtypingrrrr< packagingrtorch.utils.datarr r r loggingr r8rrrrr4rrrrrrrrrrrrrrrVrBr%_PYTORCH_DATALOADER_ADDITIONAL_KWARGSitemsvadditional_kwargsupdater5r`rrrr%torch_xla.distributed.parallel_loader distributedparallel_loaderxplMpDeviceLoaderrr,r rKr[rliststrrrrrrrFrDrs ,, UUWW$ H #&*1:t2D(E%AGGIAa  "))*;<J "M"JY Yx`'?`'F(5(5VYNYNxTC')=TCn77/  2 2/ dOC,.BOCd(&*#'#'59'+04!&#$)yy U\\ "yC=yC= y  y  yU3<012ytnyy$H-yy}yy"y !yx ;|;.%+&(<%+PLrF