iF~vSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKJ r J r SSK J r SSK JrJr SSKJr SSKJr SSKrSSKJrJr SSKJrJrJr SS KJr SS KJrJ r SS K!J"r"J#r# SSK$r$\RL"5S :waSSK'r'\""5(aSS K(J)r) SSK*J+r+ SSK,J-r- \#"5(a SSK.r.SSKJ/r/J0r0 \1"\Rd\Rf-5r4\1"\Rd\Rj-\Rf-S-5r6SS0SS0SS0SS0SS0SS0SS0S.r7Sr8SSRs\7Ru55S3r;"SS 5r<\"S!S"55r=S#\ 4S$jr>"S%S&\5r?\@S':Xa.\="5rAS(\AlBS)\AlB\?"\A5rC\CR5 gg!\%a GNCf=f)*N)ArgumentParser Namespace) AsyncIterator) dataclassfield)Thread)Optional)AsyncInferenceClientChatCompletionStreamOutput) AutoTokenizerGenerationConfigPreTrainedTokenizer)BaseTransformersCLICommand)ServeArguments ServeCommand)is_rich_availableis_torch_availableWindows)Console)Live)Markdown)AutoModelForCausalLMBitsAndBytesConfigz .!\"#$%&'()*+,\-/:<=>?@[]^_`{|}~textz5There is a Llama in my lawn, how can I get rid of it?zyWrite a Python function that integrates any Python function f(x) numerically over an arbitrary interval [x_start, x_end].z4How many helicopters can a human eat in one sitting?z4Count to 10 but skip every number ending with an 'e'zWhy aren't birds real?z2Why is it important to eat socks after meditating?z$Which number is larger, 9.9 or 9.11?)llamacode helicopternumbersbirdssocksnumbers2a **TRANSFORMERS CHAT INTERFACE** Chat interface to try out a model. Besides chatting with the model, here are some basic commands: - **!help**: shows all available commands (set generation settings, save chat, etc.) - **!status**: shows the current status of the model and generation settings - **!clear**: clears the current conversation and starts a new one - **!exit**: closes the interface am **TRANSFORMERS CHAT INTERFACE HELP** Full command list: - **!help**: shows this help message - **!clear**: clears the current conversation and starts a new one - **!status**: shows the current status of the model and generation settings - **!example {NAME}**: loads example named `{NAME}` from the config and uses it as the user input. Available example names: `z`, `a%` - **!set {ARG_1}={VALUE_1} {ARG_2}={VALUE_2}** ...: changes the system prompt or generation settings (multiple settings are separated by a space). Accepts the same flags and format as the `generate_flags` CLI argument. If you're a new user, check this basic flag guide: https://huggingface.co/docs/transformers/llm_tutorial#common-options - **!save {SAVE_NAME} (optional)**: saves the current chat and settings to file by default to `./chat_history/{MODEL_NAME}/chat_{DATETIME}.yaml` or `{SAVE_NAME}` if provided - **!exit**: closes the interface c\rSrSrSS\\S\\4SjjrS\\S\ \\ 44Sjr S\4S jr S r S \4S jrS \S \4SjrSS\4SjjrS\S\S\4SjrSrg) RichInterfacevN model_name user_namecf[5UlUcSUlOXlUcSUlgX lg)N assistantuser)r_consoler%r&)selfr%r&s j/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/transformers/commands/chat.py__init__RichInterface.__init__ws/   )DO(O  #DN&Nstreamreturnc# URRSURS35 [URSS9nSnUIShvN ShvN nURSR R nU(dM5[R"SSU5nX5- n/nUR5HNnURU5 URS 5(aURS 5 M=URS 5 MP [SRU5R5S S 9nURUSS9 MNN SSS5 O!,(df  O=fURR5 W$7f)Nz [bold blue]:)consolerefresh_per_secondrz <(/*)(\w*)>z\<\1\2\>z``` z z github-dark) code_themeT)refresh)r*printr%rchoicesdeltacontentresub splitlinesappend startswithrjoinstripupdate) r+r0livertokenoutputslineslinemarkdowns r, stream_outputRichInterface.stream_outputs2 l4??*;2>? $--A >$D%+||$ 4e--*0088&&gF OO-DLL&u-- T* V,.$BGGEN$8$8$:}U Hd 3I ,$ 4|? > >P  sN>E6E D; E  D?D=D?C'E =D??E  E6 E#E6cURRSURS35nURR5 U$)z!Gets user input from the console. [bold red]: )r*inputr&r:)r+rPs r,rPRichInterface.inputs9 ##k$..1A$FG  r/c8URR5 g)zClears the console.N)r*clearr+s r,rSRichInterface.clears r/rcURRSURSU35 URR5 g)z%Prints a user message to the console.rOz>:[/ bold red] N)r*r:r&)r+rs r,print_user_message RichInterface.print_user_messages7 k$..)99I$PQ r/colorczURRSUSU35 URR5 g)z,Prints text in a given color to the console.z[bold ]Nr*r:)r+rrYs r, print_colorRichInterface.print_colors1 fUG1TF34 r/minimalcURR[U(a[O[55 URR5 g)z'Prints the help message to the console.N)r*r:rHELP_STRING_MINIMAL HELP_STRING)r+r_s r, print_helpRichInterface.print_helps. HG%8UV r/generation_config model_kwargscURRSUS35 U(aURRSU35 URRSU35 URR5 g)zFPrints the status of the model and generation settings to the console.z[bold blue]Model: r7z[bold blue]Model kwargs: z [bold blue]Nr\)r+r%rerfs r, print_statusRichInterface.print_statussf 0 B?@  MM  ";L> J K k*;)<=> r/)r*r%r&)NN)F)__name__ __module__ __qualname____firstlineno__r strr-rr tupleintrLrPrSrWr]boolrcr dictrh__static_attributes__r/r,r#r#vs '8C= 'HSM ',-8R*S,X]^acf^fXg,\s s C $ s?O_cr/r#cv\rSrSr%Sr\"SSS0S9r\\\ S'\"SSS0S9r \\\ S '\"SSS 0S9r \\\ S '\"S SS 0S9r \\ S'\"SSS0S9r \\\ S'\"SSS0S9r\\ S'\"SSS0S9r\\\ S'\"SSS0S9r\\ S'\"SSS0S9r\\ S'\"SS/SQS.S9r\\\ S'\"SS /SQS.S9r\\\ S!'\"SSS"0S9r\\ S#'\"SSS$0S9r\\\ S%'\"SSS&0S9r\\ S''\"SSS(0S9r\\ S)'\"S*S+S,S*/S.S9r\\ S-'\"SSS.0S9r\\ S/'\"S0SS10S9r\\ S2'\"S3SS40S9r\\ S5'S6rS7rg)8 ChatArgumentsz Arguments for the chat CLI. See the metadata arg for each argument's description -- the medatata will be printed with `transformers chat --help` Nhelpz_Name of the pre-trained model. The positional argument will take precedence if both are passed.)defaultmetadatamodel_name_or_pathzKUsername to display in chat interface. Defaults to the current user's name.r)zSystem prompt. system_promptz./chat_history/zFolder to save chat history. save_folderz"Path to a yaml file with examples. examples_pathFz7Whether to show runtime warnings in the chat interface.verbosezPath to a local generation config file or to a HuggingFace repo containing a `generation_config.json` file. Other generation settings passed as CLI arguments will be applied on top of this generation config.remainzLSpecific model version to use (can be a branch name, tag name or commit id).model_revisionautozDevice to use for inference.devicezA`torch_dtype` is deprecated! Please use `dtype` argument instead.)rbfloat16float16float32)rxr; torch_dtypezOverride the default `torch.dtype` and load the model under this dtype. If `'auto'` is passed, the dtype will be automatically derived from the model's weights.dtypez2Whether to trust remote code when loading a model.trust_remote_codezWhich attention implementation to use; you can run --attn_implementation=flash_attention_2, in which case you must install this manually by running `pip install flash-attn --no-build-isolation`.attn_implementationzIWhether to use 8 bit precision for the base model - works only with LoRA. load_in_8bitzIWhether to use 4 bit precision for the base model - works only with LoRA. load_in_4bitnf4zQuantization type.fp4bnb_4bit_quant_typez#Whether to use nested quantization.use_bnb_nested_quant localhostz%Interface the server will listen to..host@zPort the server will listen to.portcURb`URcURUlgURUR:wa&[SURSURS35egg)z(Only used for BC `torch_dtype` argument.Nz`torch_dtype` z and `dtype` zn have different values. `torch_dtype` is deprecated and will be removed in 4.59.0, please set `dtype` instead.)rr ValueErrorrTs r, __post_init__ChatArguments.__post_init__%su    'zz!!-- !!TZZ/ $T%5%5$6mDJJ<PMM0 (r/)r) rjrkrlrm__doc__rr{r rn__annotations__r)r|r}r~rrqrerrrrrrrrrrrrrprrsrtr/r,rvrvsV). u )   ghD(3-$)IY@Z#[M8C=[%6&JhAijKj#(Im@n#oM8C=o%6;t2uvGTv(- 1  (x}  hiNC&:X1YZFCZ!&WA "K#!PA E8C=$)] ^t*/ r *#efL$efL$ %UFZhmotgu=vww!&uHm?n!o$okV=d4efD#fdf6W-XYD#Y r/rvargsc[U5$)z3 Factory function used to chat with a local model. ) ChatCommand)rs r,chat_command_factoryr2s t r/c\rSrSr\S\4Sj5rSr\S\4Sj5r \S#S\ S \ \S\4S jj5r \S#S \ \S\ \4S jj5rS \ \S\4SjrS\ S\S\\\44Sjr\S\S\S\ \S\ \S\\\ \44 Sj5r\S\ S\ \4Sj5rS\ S\S\44SjrS\S\ S\S\\\\\44S\S\S\ \S\\ \\\44SjrS rS!rS"rg)$ri9parserc[4nURSUS9nURS5nURS[SSS9 URS[SS S S 9 UR [ S 9 g) z Register this command to argparse so it's available for the transformer-cli Args: parser: Root parser to register command-specific arguments chat)dataclass_typeszPositional argumentsmodel_name_or_path_or_addressNz7Name of the pre-trained model or address to connect to.)typeryrxgenerate_flagsaFlags to pass to `generate`, using a space as a separator between flags. Accepts booleans, numbers, and lists of integers, more advanced parameterization should be set through --generation-config. Example: `transformers chat max_new_tokens=100 do_sample=False eos_token_id=[1,2]`. If you're a new user, check this basic flag guide: https://huggingface.co/docs/transformers/llm_tutorial#common-options*)rryrxnargs)func)rv add_parseradd_argument_group add_argumentrn set_defaultsr)rr chat_parsergroups r,register_subcommandChatCommand.register_subcommand:s)*'''P ../EF  +J   W      &: ;r/cURbURnURS5(d,URS5(dURS5(atSUlURS:wdURS:wa [ S5eURR SS5uUlUlURc [ S 5eOS UlURUl[5(d+[5(dUR(a [S 5e[5(d [S 5e[5(dUR(a [S 5eXl g)NhttphttpsrFruuLooks like you’ve set both a server address and a custom host/port. Please pick just one way to specify the server.:z\When connecting to a server, please specify a model name with the --model_name_or_path flag.TzYou need to install rich to use the chat interface. Additionally, you have not specified a remote endpoint and are therefore spawning a backend. Torch is required for this: (`pip install rich torch`)zHYou need to install rich to use the chat interface. (`pip install rich`)zYou have not specified a remote endpoint and are therefore spawning a backend. Torch is required for this: (`pip install rich torch`)) rrB spawn_backendrrrrsplitr{rr ImportErrorr)r+rnames r,r-ChatCommand.__init__[sC  - - 955Dv&&$//'*B*BdooVaFbFb%*"99 +tyyD/@$J (,'I'I'P'PQTVW'X$ 49**2$v3 &*"*.*L*L' "",>,@,@TEWEWx #$$hi i#%%$*<*<7   r/r1c[R"5S:Xa[R"5$[R "[R "55R$)z)Returns the username of the current user.r)platformsystemosgetloginpwdgetpwuidgetuidpw_namertr/r, get_usernameChatCommand.get_usernames: ??  );;= << ,44 4r/Nrfilenamec0n[U5US'XS'URnUcG[R"S5nURSUS3n[ R RXB5n[ R"[ R RU5SS9 [US 5n[R"X6S S 9 SSS5 [ R RU5$!,(df  N-=f) z!Saves the chat history to a file.settings chat_historyNz%Y-%m-%d_%H-%M-%Sz/chat_.jsonT)exist_okwr3)indent)varsr}timestrftimerrpathrCmakedirsdirnameopenjsondumpabspath)rrr output_dictfoldertime_strfs r, save_chatChatCommand.save_chats "&t* J&*N#!!  }}%89H<<=VH:USHww||F5H BGGOOH-= (C A IIkQ /!wwx((! s )C&& C4r|c Uc/nU$SUS./nU$)zClears the chat history.rroler=rt)r|rs r,clear_chat_historyChatCommand.clear_chat_historys*  D &-@AD r/rc "[U5S:Xa0$UVs0sH0nSURS5S-S-URS5S_M2 nnUR5VVs0sH+upEXER5S;aUR5OU_M- nnnUR5VVs0sHupEXES:XaSOU_M nnnS[S [ 4S jnUR5VVs0sHupEXF"U5(dSUS3OU_M nnnS R UR5VVs/sH upEUS U3PM snn5nS U-S-nURSS5nURSS5nURSS5nURSS5nURSS5nURSS5n[R"U5nU$s snfs snnfs snnfs snnfs snnf![Ra [S5ef=f)zUParses the generate flags from the user input into a dictionary of `generate` kwargs.r"=r)truefalseNonenullsr1cfURS5nURSSS5R5$)N-.r6r) removeprefixreplaceisdigit)rs r, is_number3ChatCommand.parse_generate_flags..is_numbers,s#A99S"a(002 2r/z, z: {}z"null"z"true"rz"false"rz"[[z]"r[rzFailed to convert `generate_flags` into a valid JSON object. `generate_flags` = {generate_flags} Converted JSON string = {generate_flags_string}) lensplititemslowerrnrqrCrrloadsJSONDecodeErrorr) r+rflaggenerate_flags_as_dictkvrgenerate_flags_stringprocessed_generate_flagss r,parse_generate_flags ChatCommand.parse_generate_flagssF ~ ! #I bp!paoY]# 3(:":S"@$**S/RSBT"Tao!p OeNjNjNl" NldaAGGI)::qwwy ANl " OeNjNjNl!mNlda!F{V"ANl!m 3 3 3 VlUqUqUs!tUsTQ!Yq\\q1Xq"HUs!t !% CYC_C_Ca*bCa41aS1#;Ca*b c!$&; ;c A!6 = =h O 5 = =h O 5 = =i Q 5 = =dC H 5 = =dC H!6 = =c3 G '+zz2G'H $('W"q" "n"u +c"## D  s)7G"2G*G'!G")G( 9G.. Hmodel_generation_configcURbSUR;ai[RRUR5n[RR UR5n[ R "X45nOL[ R "UR5nO+[R"U5nUR"S0SSS.D6 URUR5nUR"S0UD6nXW4$)zZ Returns a GenerationConfig object holding the generation parameters for the CLI command. rT) do_samplemax_new_tokensrt) rerrrbasenamer from_pretrainedcopydeepcopyrErr)r+rrrrreparsed_generate_flagsrfs r,get_generation_parameterization+ChatCommand.get_generation_parameterizations  ! ! -$000''//$*@*@A77++D,B,BC$4$D$DW$W!$4$D$DTE[E[$\! !% .E F   $ $ RTS'Q R!% 9 9$:M:M N(//H2GH !..r/ tokenizerre eos_tokens eos_token_idscURc URnO URn/nUb/URURUR S555 Ub:URUR S5Vs/sHn[ U5PM sn5 [ U5S:XaURUR5 XE4$s snf)z:Retrieves the pad token ID and all possible EOS token IDs.,r) pad_token_id eos_token_idextendconvert_tokens_to_idsrrprrA)rrerrrall_eos_token_idstoken_ids r,parse_eos_tokensChatCommand.parse_eos_tokenss  ) ) 1,99L,99L  !  $ $Y%D%DZEUEUVYEZ%[ \  $  $ $MDWDWX[D\%]D\c(mD\%] ^  !Q &  $ $%6%C%C D.. &^s<C model_argscUR(a7[SURURURURS9nU$UR (a [SS9nU$SnU$)NT)rbnb_4bit_compute_dtyperbnb_4bit_use_double_quantbnb_4bit_quant_storage)r)rrrrrr)rquantization_configs r,get_quantization_config#ChatCommand.get_quantization_configst  " ""4!'1'7'7$.$B$B*4*I*I'1'7'7 # #" $ $"4!#  #"#' ""r/rc[R"URURURS9nUR S;a UR O[ [UR 5nURU5nURURUSUS.n[R"UR4SUR0UD6n[ USS5cURUR5nXb4$)N)revisionr)rNr)r'rr device_mapr#r hf_device_map) r r model_name_or_path_positionalrrrgetattrtorchr$rrtor)r+rrrr#rfmodels r,load_model_and_tokenizer$ChatCommand.load_model_and_tokenizer)s!11  . .(("44 #jjN: tzz@Z"::4@++#'#;#; #6  %44  . . BFBXBX \h  5/4 0 8HHT[[)Er/ user_input interfaceexamplesrfrcSnUS:Xa-URUR5nUR5 GO6US:XaUR5 GOUR S5(al[ UR 55S:aOUR 5n [ U 5S:XaU Sn OSn URXrU 5n URSU S 3S S 9 GOUR S 5(aUS SR5n U R 5n U H n SU ;dM URSU S3SS 9 GO? URU 5n UR"S0U D6nUR"S0UD6 GOUR S5(a[ UR 55S:XaUR 5SnX;aBUR5 /nURXOS5 URSXOSS.5 OySUS[UR55S3nURUSS 9 OHUS:XaUR!UR"UUS9 O&SnURSUS3SS 9 UR5 XxXV4$)z Handles all user commands except for `!exit`. May update the chat history (e.g. reset it) or the generation config (e.g. set a new flag). Tz!clearz!helpz!saverNzChat saved in !green)rrYz!setr3rz(Invalid flag format, missing `=` after `z;`. Please use the format `arg_1=value_1 arg_2=value_2 ...`.red!examplerr)rzExample z* not found in list of available examples: rz!status)r%rerfF'z/' is not a valid command. Showing help message.rt)rr|rSrcrBrrrr]rDrrErWrAlistkeysrhr{)r+r1rr2r3rerfr valid_command split_inputrnew_generate_flagsrparsed_new_generate_flagsnew_model_kwargs example_name example_errors r,handle_non_exit_user_commands)ChatCommand.handle_non_exit_user_commandsDs  !**4+=+=>D OO  7 "  "  " "7 + +J4D4D4F0G!0K$**,K;1$&q>~~d(;H  ! !z'C7 ! S  " "6 * *",AB!5!5!7 !3!9!9!; *d?))FtfMAA$ *+-1,E,EFX,Y)#4#;#;#X>W#X ##7&67  " ": . .3z7G7G7I3Ja3O%++-a0L'!,,X-CF-KL V8Nv8VWX|n,VW[\d\i\i\kWlVmmno%%=%F 9 $  " "22"3) # "M  ! !:,6e'fns ! t  "$5CCr/cL[R"UR55 gN)asynciorun _inner_runrTs r,rIChatCommand.runs DOO%&r/c# UR(Ga[URRURRURR URR URRURRURRURRURRURRSS9 n[U5n[URS9nSUlUR#5 URR$S-URR&-nURRS:XaSOURRnURnUR(c[*nO4[-UR(5n[.R0"U5nSSS5 UR2cUR55n O UR2n [6R8"UR$5n UR;Xj5up[=UR$U S9n U R?5 URAURB5nU RESS 9 [GUS URR35IShvN nU RI5nURKS 5(a@US :XaOURMUUU WU U US 9unnpU(aURKS5(dMgOUROSUS.5 URQUSU RS5US.S9nU RUU5IShvN nUROSUS.5 MSSS5IShvN g!,(df  GN=fNN?![Va M4f=fN,!,IShvN (df  g=f7f)Nerror) rrrrrrrrrr log_level)targetT@rzhttp://localhost)r%r&)r_rr6z!exit)r1rr2r3rerfrr9r)r)rer.)r0 extra_bodyr(),rrrrrrrrrrrrrrrrIdaemonstartr{rr~DEFAULT_EXAMPLESryaml safe_loadr)rr r rr#rSrr|rcr rPrBrDrAchat_completionto_json_stringrLKeyboardInterrupt)r+ serve_args serve_commandthreadr.rrr3rr)rrerfr2rclientr1r=r0 model_outputs r,rJChatCommand._inner_runs3    'yy''iioo"&))"="=$(II$A$A!YY33!YY33$(II$A$A%)YY%C%CYY^^YY^^! J)4M=#4#45F FM LLN ,,s2TYY5M5MM%)YY^^{%B! yy    %'Hd(()Q>>!,* 99 $$&D99D"2"B"B4CZCZ"[*.*N*Nt*m'!T-D-DPTU &&t'9'9: T*'4&$))..1A(BCCv%!*!2J",,S11%0!SWSuSu+5%)*3)12C-9%)TvTPD-1B -J4I4I*4U4U$5V V $KL#33#1B1Q1Q1S%*$4F*3)@)@)H#HLKK NOIDCC%*)$DF$I)MDCCCsF:O<NCO.N#/O2N:5,N'!N:"6N'N:A N'%N%&N'>N: O N8 O N  O%N'' N51N:4N55N:8O:OO  O O)rrrG) rjrkrlrm staticmethodrrr-rnrrvr rr;rrrrr rorrrprrr$r r/r#rDrIrJrsrtr/r,rr9s Z##& ] uE[]jEj?k 6PDPDPD! PD sDcN*+ PD , PDPD4jPD tDz+T1 2PDh'Ur/r__main__z meta-llama/Llama-3.2-3b-Instructzhttp://localhost:8000)ErHr rrrr>stringrargparserrcollections.abcr dataclassesrr threadingrtypingr rUhuggingface_hubr r transformersr r rtransformers.commandsrtransformers.commands.servingrrtransformers.utilsrrreadlinerrr rich.consoler rich.liver rich.markdownrr,rrset ascii_letters whitespaceALLOWED_KEY_CHARSdigitsALLOWED_VALUE_CHARSrTrarCr<rbr#rvrrrjrrrrIrtr/r,rws   .)( L =FD   ?? !$& ,,v/@/@@A 6==(6+<+<?@A &YYx \\ \~yw,wt  z ?D)KD&)@D& t DHHJ M  s0F..F87F8