;i3SSKrSSKrSSKJr SSKJrJrJr SSKJ r J r SSK J r \(aSSK Jr \"SS55rg) N) dataclass) TYPE_CHECKINGOptionalUnion)TorchContextParallelConfigTorchTensorParallelConfig)is_torch_version) Acceleratorc\rSrSr%SrSr\\\S'Sr \\\S'Sr \\\S'Sr \\\S'Sr \ S\4\S'Sr\ S\4\S 'SrS rS r\S 5r\S 5r\S5r\S5r\S5r\S5r\S5r\S5r\S5r\S5r\S5r\S5r \S5r!S\"4Sjr#S%S\\"4Sjjr$S\%\%\S4\%\"S444Sjr&Sr'S \"S!\4S"jr(S&S#jr)S$r*g)'ParallelismConfiga, A dataclass to configure parallelisms applied to the model. Inspired by torchtitan's `ParallelDims` https://github.com/pytorch/torchtitan/blob/main/torchtitan/distributed/parallel_dims.py Args: dp_replicate_size (`int`, defaults to `1`): The size of the data parallel group. If `dp_replicate_size` is set to 1, the data parallel replication group will not be used. dp_shard_size (`int`, defaults to `1`): The size of the model shard group. If `dp_replicate_size > 1` and `tp_size > 1`, `dp_shard_size` must also be greater than 1, as composing DDP + TP is currently not supported. tp_size (`int`, defaults to `1`): The size of the tensor parallel group. If `tp_size` is set to `1`, the tensor parallel group will not be used. cp_size (`int`, defaults to `1`): The size of the context parallel group. Currently not supported, but reserved for future use and enabled for downstream libraries. tp_handler (`~utils.TorchTensorParallelConfig`, defaults to `None`): The handler for the tensor parallel group. You may obtain different distributed data parallel paradigms by configuring `dp_replicate_size` and `dp_shard_size` together: - `dp_replicate_size == 1` and `dp_shard_size > 1`, we obtain Fully Sharded Data Parallel (FSDP). - `dp_replicate_size > 1` and `dp_shard_size > 1`, we obtain Hybrid Sharded Data Parallel (HSDP). - `dp_replicate_size > 1` and `dp_shard_size == 1` is an invalid configuration, to use pure DP, use `DistributedDataParallelKwargs` instead. Ndp_replicate_size dp_shard_sizetp_sizecp_size tp_handler cp_handlercSURSURSURSURSURSUR SUR S3$) Nz'ParallelismConfig( dp_replicate_size=z, dp_shard_size=z , tp_size=z , cp_size=z, total_size=z tp_handler=z, cp_handler=z) )rrrr total_sizerrselfs m/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/accelerate/parallelism_config.py__repr__ParallelismConfig.__repr__Fsy ##'#9#9":;#1123'' OO,- OO,- OO,C 1 c SSKnS/nURURR5VVs0sH;up4X2;dM U[ US5(aURUR5OU_M= snn5 gs snnf)Nr device_mesh__dict__)copydeepcopyritemshasattr)rr_non_serializable_fieldskvs rto_jsonParallelismConfig.to_jsonRss$1?  !MM//1 1DA4N:0F0F4==,AM1   s A; 3A; cf/nUR(aUS/- nUR(aUS/- nU$)zENames of enabled dimensions across which data parallelism is applied. dp_replicatedp_shard)dp_replicate_enableddp_shard_enabledrdimss r dp_dim_namesParallelismConfig.dp_dim_names_s9  $ $ ^$ $D  ZL D rcf/nUR(aUS/- nUR(aUS/- nU$)z]Names of enabled dimensions which will receive the same batch (non-data parallel dimensions).tpcp) tp_enabled cp_enabledr-s rnon_dp_dim_names"ParallelismConfig.non_dp_dim_namesis2 ?? TFND ?? TFND rcf/nUR(aUS/- nUR(aUS/- nU$)zlNames of enabled dimensions which will be flattened into a joint mesh across which is model sharded in FSDP.r*r3)r,r5r-s rdp_shard_cp_dim_names'ParallelismConfig.dp_shard_cp_dim_namesss5  ZL D ?? TFND rc/nUR(aUS/- nUR(aUS/- nUR(aUS/- nU$)z@Names of enabled dimensions across which loss should be averagedr)r*r3)r+r,r5r-s rdp_cp_dim_names!ParallelismConfig.dp_cp_dim_names}sK  $ $ ^$ $D  ZL D ?? TFND rcD/nUR(aUS/- nUS/- nU$)z^Names of enabled dimensions across which FSDP is applied, including data parallel replication.r) dp_shard_cp)r+r-s rfsdp_dim_names ParallelismConfig.fsdp_dim_namess0  $ $ ^$ $D  rchURUR-UR-UR-$)zSThe total size of the parallelism configuration, which is the product of all sizes.)rrrrrs rrParallelismConfig.total_sizes-%%(:(::T\\IDLLXXrc4URUR-$)zhThe size of the non-data parallel dimensions, which is the product of tensor and context parallel sizes.)rrrs rnon_data_parallel_size(ParallelismConfig.non_data_parallel_sizes||dll**rc4URUR-$)z_The size of the data parallel dimensions, which is the product of data parallel replication and)rrrs rdata_parallel_size$ParallelismConfig.data_parallel_sizes%%(:(:::rc URS:$)zKTrue if data parallel replication is enabled, i.e. `dp_replicate_size > 1`.)rrs rr+&ParallelismConfig.dp_replicate_enableds%%))rc URS:$)zDTrue if data parallel sharding is enabled, i.e. `dp_shard_size > 1`.rK)rrs rr,"ParallelismConfig.dp_shard_enableds!!A%%rc URS:$)z:True if tensor parallelism is enabled, i.e. `tp_size > 1`.rK)rrs rr4ParallelismConfig.tp_enabled||arc URS:$)z;True if context parallelism is enabled, i.e. `cp_size > 1`.rK)rrs rr5ParallelismConfig.cp_enabledrQrc4URUR-$)z$Names of all active mesh dimensions.)r/r6rs ractive_mesh_dims"ParallelismConfig.active_mesh_dimss  4#8#888r device_typec[SS5(aSSKJn O [S5eUR 5n[ U5S:XagUupEU"UUUS9nUR (aX`R RS5 UR(aX`RRS 5 UR(aX`RRS 5 U$) aBuilds a device mesh for the given device type based on the parallelism configuration. This method will also create required joint meshes (e.g. `dp_shard_cp`, `dp_cp`, `dp`). Args: device_type (`str`): The type of device for which to build the mesh, e z>=z2.2.0r)init_device_meshz4Building a device_mesh requires to have torch>=2.2.0N)mesh_dim_namesdpr?dp_cp) r torch.distributed.device_meshrY RuntimeError _get_meshlenr/_flattenr9r<)rrWrYmeshrZ mesh_shapers rbuild_device_mesh#ParallelismConfig.build_device_meshs D' * * FUV V~~ t9>%)"&  )    )) * 3 3D 9  % % 22 3 < <] K    ,, - 6 6w ?rcURc'Ub"URU5UlUR$SeUb@URRU:wa&[SURRSUS35eUR$)Nz@You need to pass a device_type e.g cuda to build the device meshz4The device_mesh is already created with device type z@. However, you are trying to get a device mesh with device_type z<. Please check if you correctly initialized your device_mesh)rrdrW ValueError)rrWs rget_device_mesh!ParallelismConfig.get_device_meshs    #&#'#9#9+#F ZZ&##//;>$NtO_O_OkOkNlmmnymzzvwrreturn.c^URVs0sHoURU_M nn/SQm[UR5U4SjS9n[ [ U65$s snf)zQGenerate mesh shape and dimension names for torch.distributed.init_device_mesh().)r)r*r3r2c,>TRUS5$)Nr)index)x mesh_orders r-ParallelismConfig._get_mesh..s:++AaD1r)key)rU_sizessortedr!tuplezip)r parallelism mesh_dims sorted_itemsros @rr_ParallelismConfig._get_meshseOSNcNcdNc{$++k"::Nc d>  OO 2 S,'((esAcURc.[[RR SS55UlUR c.[[RR SS55UlUR c.[[RR SS55UlURc.[[RR SS55UlUR S:aURc[5UlURS:aURc[5Ul URS:a[SUR35eUR S:a[SUR 35eUR S:a[S UR 35eURS:a[S UR35eUR S:dURS:a+URS:aUR S:Xa [S 5eURUR UR URS .Ul g) N$PARALLELISM_CONFIG_DP_REPLICATE_SIZE1 PARALLELISM_CONFIG_DP_SHARD_SIZEPARALLELISM_CONFIG_TP_SIZEPARALLELISM_CONFIG_CP_SIZErKz.dp_replicate_size must be at least 1, but got z*dp_shard_size must be at least 1, but got z$tp_size must be at least 1, but got z$cp_size must be at least 1, but got aCTensor/Context parallelism (tp/cp_size > 1) cannot be used with pure data parallelism (dp_replicate_size > 1 and dp_shard_size == 1). Please set dp_shard_size > 1 and dp_replicate_size == 1 to compose FSDP + TP/CP for 2D parallel, or set dp_replicate_size == 1 and dp_shard_size > 1 to compose HSDP + TP/CP for 3D parallel.)r)r*r2r3)rintosenvirongetrrrrrrrrgrsrs r __post_init__ParallelismConfig.__post_init__s  ! ! )%(8^`c)d%eD "    %!$RZZ^^4VX[%\!]D  << rzz~~.JCPQDL << rzz~~.JCPQDL <  ! !A %MdNdNdMefg g    !I$J\J\I]^_ _ <RS S <RS S LL1  q 0d6L6Lq6PUYUgUgklUlo  !22**,,,,   rrwsizecXRR5;d"SURR535eX RU'[XS3U5 g)NzParallelism must be one of _size)rskeyssetattr)rrwrs r _set_sizeParallelismConfig._set_sizesVkk..00d4OPTP[P[P`P`PbOc2dd0#' K  U+T2rc[5nUR(dURS:XagURS:XaURSUR5 URUR:wa&[ SURSURS35eURS:a;UR (d*UR(d[ SURS35eURR5H7up4US:XdM [XS3S5cMURS US US 35 M9 U(a@UR(a.[R"S S RU5-[ 5 ggg)NrKr)zParallelismConfig total_size (z ) does not match num_processes (zB). Please adjust dp_replicate_size/ dp_shard_size/tp_size/cp_size.zpParallelismConfig is only compatible DistributedType.FSDP (version 2) or DistributedType.Multi{Device}, but got ._handlerzParallelismConfig.z_handler is set, but z0_size is set to 1. This handler will be ignored.z.ParallelismConfig has the following warnings:  )set multi_devicerr num_processesrgis_fsdp2distributed_typersr!getattraddis_main_processwarningswarnjoin UserWarning)r accelerator _warningsrwrs r_validate_accelerator'ParallelismConfig._validate_accelerator"stE ''DOOq,@  ??a  NN>;+D+D E ??k77 700AB""-";";!<=12  ??Q  (<(< @X@XEFQFbFbEccde "&!2!2!4 KqyWT](+CTJV ( 5J;-XHI"5 44 MMADIIiDXX 59r)rsrrrrrrr)N)rr )+__name__ __module__ __qualname____firstlineno____doc__rrr__annotations__rrrrrrrrrrr&propertyr/r6r9r<r@rrErHr+r,r4r5rUstrrdrhrur_rrr__static_attributes__rrr r s:(,x}+#'M8C='!GXc]!!GXc]!:>Jd556=:>Jd667>K      YY++;;**&&    99S<  8C=   )5sCx%S/!AB )' R3S33 rr )rr dataclassesrtypingrrraccelerate.utils.dataclassesrraccelerate.utils.versionsr accelerater r rrrrs@ !11^6& ee er