;i SSKrSSKJr SSKJrJrJr SSKJr SSK J r J r J r J r JrJrJrJr SS\S\\4S jjrS rS rS rSS \\\\\4S\\\S\\\S\\\\4S\\S\\4 Sjjrg)N) MethodType)AnyOptionalUnion) PartialState)calculate_maximum_sizes convert_bytescopy_tensor_to_devicesignorant_find_batch_sizeinfer_auto_device_mapis_pippy_availablepad_input_tensorssend_to_device num_processes max_memoryc@US:Xa [XSS9$Ucy[U5upEXES-U- n[U5nURS5upx[R "[ U55S-nUSU3n[U5V s0sHoU_M nn [UUUSS9n U $s sn f)z@ Calculates the device map for `model` with an offset for PiPPy rF)no_split_module_classes clean_resultr g?)rrr)r r r splitmathceilfloatrange) modelrrr model_sizesharedmemoryvalueendingi device_maps d/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/accelerate/inference.pygenerate_device_mapr%s $Ujopp4U; ay(M9v& S) 5<(3.81VH%).})=>)=Ai)= >&  7 J ?s? BcSnUbUHn[U5nUcM O Ub+Uc(UR5Hn[U5nUcM U$ U$N)r values)argskwargsfound_batch_sizeargkwargs r$find_pippy_batch_sizer.<sl C7< +.6]]_E7> +  % cSSKJnJnJn [ 5nUV s0sHoUR _M n n U"UUUU S9n U R URURS9n U"X5n U $s sn f)a. Attaches the split points to the model based on `self.device_map` and generates a `PipelineStage`. Requires passing in needed `args` and `kwargs` as the model needs on the CPU. Users can pass in custom `num_chunks` as an optional hyper-parameter. By default will use `AcceleratorState.num_processes` r) ScheduleGPipe SplitPointpipeline)mb_args mb_kwargs split_spec)device) torch.distributed.pipeliningr1r2r3r BEGINNING build_stagelocal_process_indexr7)r split_pointsr)r* num_chunksr1r2r3state split_pointr6pipestageschedules r$build_pipelinerCKs}QP NEGST| z333|JT    D   U66u||  LEU/H OUsA(cT[5nSnURS:Xa U"U0UD6nOpUR(a?[X45nUc [ S5eXq:wa[ X7U5n[ XGU5nU"U0UD6 O UR (aU"5nOU"5 U(a [U5nU$)Nrz-Could not find batch size from args or kwargs)rris_local_main_processr. ValueErrorris_last_processr )forwardr= gather_outputr)r*r>outputr+s r$ pippy_forwardrKes NE F a$)&)  $ $0>  #LM M-(L*6ZP     '/ Mr/r<r example_argsexample_kwargsr=rIc^^^ ^ [5(d [S5e[5n[US5n[US5nTc URmUS:XaR[ UTUS9n/n[ ST5H5m UR[U 4SjUR5555 M7 Xl [XX4T5m URUl URUlT UlXl UUU 4Sjn [#X5n XlXl U$)a Wraps `model` for pipeline parallel inference. Args: model (`torch.nn.Module`): A model we want to split for pipeline-parallel inference split_points (`str` or `List[str]`, defaults to 'auto'): How to generate the split points and chunk the model across each GPU. 'auto' will find the best balanced split given any model. Should be a list of layer names in the model to split by otherwise. no_split_module_classes (`List[str]`): A list of class names for layers we don't want to be split. example_args (tuple of model inputs): The expected inputs for the model that uses order-based inputs for a *single process*. Recommended to use this method if possible. example_kwargs (dict of model inputs) The expected inputs for the model that uses dictionary-based inputs for a *single process*. This is a *highly* limiting structure that requires the same keys be present at *all* inference calls. Not recommended unless the prior condition is true for all cases. num_chunks (`int`, defaults to the number of available GPUs): The number of different stages the Pipeline will have. By default it will assign one chunk per GPU, but this can be tuned and played with. In general one should have num_chunks >= num_gpus. gather_output (`bool`, defaults to `False`): If `True`, the output from the last GPU (which holds the true outputs) is sent across to all GPUs. zEUsing `torch.distributed.pipelining` requires PyTorch 2.4.0 or later.cpuauto)rrc3<># UHupUT:XdM Uv M g7fr').0kvr"s r$ prepare_pippy..s$P3E41aQQ3Es  c:>[TRTT/UQ70UD6$r')rKstep)r)r*rIr=rAs r$rHprepare_pippy..forwards UZZ]TTTVTTr/)r ImportErrorrrrr%rappendnextitemshf_split_pointsrCrH_original_forward__call___original_call pippy_stager __wrapped__) rr<rrLrMr=rIr>r#rH model_forwardr"rAs `` @@r$ prepare_pippyrf~sB   abb NE!,6L#NE:N(( v( Tkl  q*%A   $P:3C3C3E$P P Q&( 5 j YE#mmE >>EE(U w.M'M Lr/)rNN)rPNrRNNF)rtypesrtypingrrrr>rutilsr r r r r rrrintdictr%r.rCrKstrlisttupleboolrfrRr/r$rps ''   _cMUVZ^: 465;37)+/3 $$)<5d3i01<&d3i0<5:& < T#s(^, <  <D><r/