;i`7SSKrSSKrSSKrSSKrSSKJrJr SSKJrJ r J r J r J r J r JrJr SSKJr SrS SjrS Sjrg) N)AcceleratorState PartialState) PrecisionTypePrepareForLaunchare_libraries_initializedcheck_cuda_p2p_ib_support get_gpu_infois_mps_availableis_torch_versionpatch_environment)0ELASTIC_LOG_LINE_PREFIX_TEMPLATE_PYTORCH_VERSIONc[5ng)z+Verify a `PartialState` can be initialized.N)r)_s d/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/accelerate/launchers.py test_launchr#s Ac SnSn[S[RR555(aSnOAS[R ;a-S[ [R SR55;n[UR55nU(dU(a[RRS S 5c$[RRS S 5S :XaXSS KJs Jn [#[$R&5S:a [S5e[)USS9n[+S5 UR-UUSS9 g U(aQ[/5SS:a@[0R2R55(a [+S5 O [+S5 U"U6 g Uc [S5eXg:a [S5eUS:GaSSKJnJn SSKJn SSK J!n [#[$R&5S:a [S5e[ES5n[#U5S:aS nUH nUS!US"3- nM [GU5e[IUUXr-UUUS#9n[K5(d S$US%'S$US&'[MS/0UD6 [O[0S'5(a([0RPRS5RTOS(nUS):XaS*OS+n[RRS,S-5R5S.:Xa[)[VUS9nU"US/USS09 [)UUS9n[+S7US8URY5S935 U c0n US::XaXjS;'U (dUS[^5(aUUS?'U"U"S/0UD6US@9"U6 S S S 5 g [a5(aS$[RSC'[+SD5 Oi[0R2R55(a [+S5 O:[0RbR55(a [+SE5 O [+SF5 U"U6 g ![a> [SURR5S[R"5S35ef=f!Ua]nS1nS2URY5S33URZS;a [GUS4URY5S535Ue[GUS6U35UeS nAff=f!UagnS2URY5S33URZS;a.[GURY5SAURY5S535Ue[GSBU35UeS nAff=f!,(df  g =f)Ga Launches a training function, using several processes or multiple nodes if it's possible in the current environment (TPU with multiple cores for instance). To use this function absolutely zero calls to a device must be made in the notebook session before calling. If any have been made, you will need to restart the notebook and make sure no cells use any device capability. Setting `ACCELERATE_DEBUG_MODE="1"` in your environment will run a test before truly launching to ensure that none of those calls have been made. Args: function (`Callable`): The training function to execute. If it accepts arguments, the first argument should be the index of the process run. args (`Tuple`): Tuple of arguments to pass to the function (it will receive `*args`). num_processes (`int`, *optional*): The number of processes to use for training. Will default to 8 in Colab/Kaggle if a TPU is available, to the number of devices available otherwise. mixed_precision (`str`, *optional*, defaults to `"no"`): If `fp16` or `bf16`, will use mixed precision training on multi-device. use_port (`str`, *optional*, defaults to `"29500"`): The port to use to communicate between processes when launching a multi-device training. master_addr (`str`, *optional*, defaults to `"127.0.0.1"`): The address to use for communication between processes. node_rank (`int`, *optional*, defaults to 0): The rank of the current node. num_nodes (`int`, *optional*, defaults to 1): The number of nodes to use for training. rdzv_backend (`str`, *optional*, defaults to `"static"`): The rendezvous method to use, such as 'static' (the default) or 'c10d' rdzv_endpoint (`str`, *optional*, defaults to `""`): The endpoint of the rdzv sync. storage. rdzv_conf (`Dict`, *optional*, defaults to `None`): Additional rendezvous configuration. rdzv_id (`str`, *optional*, defaults to `"none"`): The unique run id of the job. max_restarts (`int`, *optional*, defaults to 0): The maximum amount of restarts that elastic agent will conduct on workers before failure. monitor_interval (`float`, *optional*, defaults to 0.1): The interval in seconds that is used by the elastic_agent as a period of monitoring workers. log_line_prefix_template (`str`, *optional*, defaults to `None`): The prefix template for elastic launch logging. Available from PyTorch 2.2.0. Example: ```python # Assume this is defined in a Jupyter Notebook on an instance with two devices from accelerate import notebook_launcher def train(*args): # Your training function here ... notebook_launcher(train, args=(arg1, arg2), num_processes=2, mixed_precision="fp16") ``` Fc3B# UHoRS5v M g7f)KAGGLEN) startswith).0keys r $notebook_launcher..{s A/@>>( # #/@sTIPythonz google.colabzUnknown mixed_precision mode: z. Choose between .TPU_NAMEN PJRT_DEVICETPUrzTo train on TPU in Colab or Kaggle Kernel, the `Accelerator` should only be initialized inside your training function. Restart your notebook and make sure no cells initializes an `Accelerator`.XLA)distributed_typez"Launching a training on TPU cores.fork)args start_methodrzLaunching training on one GPU.zLaunching training on one CPU.zfYou have to specify the number of devices you would like to use, add `num_processes=...` to your call.z4The node_rank must be less than the number of nodes.) LaunchConfigelastic_launchstart_processes)ProcessRaisedExceptionzTo launch a multi-device training from your notebook, the `Accelerator` should only be initialized inside your training function. Restart your notebook and make sure no cells initializes an `Accelerator`. bitsandbyteszCould not start distributed process. Libraries known to initialize device upon import have been imported already. Please keep these imports inside your training function to try and help with this:z * ``)nproc node_rank world_size master_addr master_portmixed_precision1nccl_p2p_disablenccl_ib_disable acceleratorcudaxpu MULTI_XPU MULTI_GPUACCELERATE_DEBUG_MODEfalsetruer%nprocsr&zQAn issue was found when verifying a stable environment for the notebook launcher.zCannot re-initialize z in forked subprocesszThis likely stems from an outside import causing issues once the `notebook_launcher()` is called. Please review your imports and test them when running the `notebook_launcher()` to identify which one is problematic and causing z to be initialized.z! The following error was raised: zLaunching training on  zs.staticrank:) min_nodes max_nodesnproc_per_noderun_id rdzv_endpoint rdzv_backend rdzv_configs max_restartsmonitor_intervalr&z>=log_line_prefix_template)config entrypointa: has been initialized before the `notebook_launcher` could create a forked subprocess. This likely stems from an outside import causing issues once the `notebook_launcher()` is called. Please review your imports and test them when running the `notebook_launcher()` to identify which one is problematic and causing z0An issue was found when launching the training: PYTORCH_ENABLE_MPS_FALLBACKzLaunching training on MPS.zLaunching training on one XPU.zLaunching training on CPU.)2anyosenvironkeyssysmodulesstr get_ipythonrlower ValueErrorr4listget)torch_xla.distributed.xla_multiprocessing distributedxla_multiprocessinglenr _shared_staterprintspawnr torchr9 is_availabletorch.distributed.launcher.apir(r)torch.multiprocessingr+torch.multiprocessing.spawnr,r RuntimeErrordictr r hasattrr8current_acceleratortyperupperr%r rr r:)functionr% num_processesr4use_portr2r0 num_nodesrLrK rdzv_confrdzv_idrNrOrPin_colab in_kagglexmplauncherr(r)r+r,problematic_importserrlib_name patched_env device_typer#elaunch_config_kwargss rnotebook_launcherr(szbHI Arzz/@ AAA ckk !!SY)?)K)K)M%NN '(=(=(?@ I  D ) 52::>>-Y[;\`e;e @? -- . 2!  $HuE 23 (F ; lnQ'!+ :: " " $ $ 2 3 2 3$  x   !ST T 1  S = J#112Q6 % #>"97CIIKvU/ N^_H d'r-^de,HGWX.}oQ{?P?P?R>SSUVWj ($& #x/,5&),/:m1XJ,GM+/"+"+'4&&3%1%.%1)9%+ ,((.^__Kc,-GH",*N9M*N[cdfjkW21r !! This function is provided for internal testing and debugging, but it's not intended for real trainings. It will only use the CPU. Args: function (`Callable`): The training function to execute. args (`Tuple`): Tuple of arguments to pass to the function (it will receive `*args`). num_processes (`int`, *optional*, defaults to 2): The number of processes to use for training. rr* 127.0.0.129500noyes)r1r2r3accelerate_mixed_precisionaccelerate_debug_rdv_fileaccelerate_use_cpuT)debugr$rAN)rjr+tempfileNamedTemporaryFiler namer)rrr%rsr+tmp_filer{s rdebug_launcherrsq&6  $ $ &($#'+&.mm$  (=H H TZ [  ' &   ' &s"A*AA* A' #A** A8)r@NrrrrrrDr Nnonerg?N)r@r')rUrXrrgstaterrutilsrrrr r r r r utils.constantsrrrrr@rrrsi  1   N   !fR!\r