i53SSKrSSKrSSKJr SSKrSSKJrJr SSKJr SSK J r SSK J r \ R"\5r\"S5"S S \R 55r\"S 5"S S \R 55r\"S5"SS\R 55r\"S5"SS\R 55r\"S5"SS\R 55r\"S5"SS\R 55r"SS\R 5r"SS\R 5r"SS\R 5r"S S!\R 5r"S"S#\R 5r"S$S%\R 5r"S&S'\5r"S(S)\R 5r0S*\_S+\S,S-S..4_S/\_S0\_S1\S2S304_S4\_S5\S6S304_S7\_S8\_S9\R>_S:\_S;\_S<\_S=\R@_S>\_S?\RB_S@\RD_\\RF\RH\RJ\SA.Er&\"\&5r'SBr(\("S15r)\("S05r*\("S*5r+\("S/5r,\("S<5r-\("SC5r.\("S;5r/\("S:5r0g)DN) OrderedDict)Tensornn)use_kernel_forward_from_hub)logging)is_torchdynamo_compilingGeluTanhc\^\rSrSrSrS S\4U4SjjjrS\S\4SjrS\S\4Sjr S r U=r $) GELUTanha A fast C implementation of the tanh approximation of the GeLU activation function. See https://huggingface.co/papers/1606.08415. This implementation is equivalent to NewGELU and FastGELU but much faster. However, it is not an exact numerical match due to rounding errors. use_gelu_tanh_pythonc>[TU]5 U(aURUlg[R "[ RRSS9Ulg)Ntanh) approximate) super__init___gelu_tanh_pythonact functoolspartialr functionalgelu)selfr __class__s h/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/transformers/activations.pyrGELUTanh.__init__(s<  --DH ((););PDHinputreturnc US-S[R"[R"S[R- 5US[R "US5---5--$N??@Hm?g@torchrmathsqrtpipowrrs rrGELUTanh._gelu_tanh_python/sPs{cEJJtyytww/G5S[^c^g^ghmor^sSsKs/t$uuvvrc$URU5$Nrr-s rforwardGELUTanh.forward2xxrr1F) __name__ __module__ __qualname____firstlineno____doc__boolrrrr2__static_attributes__ __classcell__rs@rr r sJQTQQwvw&wVrr NewGELUc*\rSrSrSrS\S\4SjrSrg)NewGELUActivation6z Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 rr c SU-S[R"[R"S[R- 5US[R "US5---5--$r"r'r-s rr2NewGELUActivation.forward=sPU{cEJJtyytww/G5S[^c^g^ghmor^sSsKs/t$uuvvrNr6r7r8r9r:rr2r<rErrrArA6s wVwwrrAGeLUc\^\rSrSrSrS S\4U4SjjjrS\S\4SjrS\S\4Sjr S r U=r $) GELUActivationAa Original Implementation of the GELU activation function in Google BERT repo when initially created. For information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 use_gelu_pythonc>[TU]5 U(aURUlg[R R Ulgr0)rr _gelu_pythonrrrr)rrKrs rrGELUActivation.__init__Js/  ((DH}}))DHrrr cnUS-S[R"U[R"S5- 5--$)Nr#r$r%)r(erfr)r*r-s rrMGELUActivation._gelu_pythonQs,s{cEIIediin.D$EEFFrc$URU5$r0r1r-s rr2GELUActivation.forwardTr4rr1r5) r6r7r8r9r:r;rrrMr2r<r=r>s@rrIrIAsG***G&GVGVrrISiLUc*\rSrSrSrS\S\4SjrSrg)SiLUActivationXa See Gaussian Error Linear Units (Hendrycks et al., https://arxiv.org/abs/1606.08415) where the SiLU (Sigmoid Linear Unit) was originally introduced and coined, and see Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning (Elfwing et al., https://arxiv.org/abs/1702.03118) and Swish: a Self-Gated Activation Function (Ramachandran et al., https://arxiv.org/abs/1710.05941v1) where the SiLU was experimented with later. rr c@[RRU5$r0)rrsilur-s rr2SiLUActivation.forwardbs}}!!%((rrENrFrErrrVrVXs)V))rrVFastGELUc*\rSrSrSrS\S\4SjrSrg)FastGELUActivationfzu Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs rr c^SU-S[R"US-SSU-U---5--$)Nr#r$g3E?r&)r(rr-s rr2FastGELUActivation.forwardls:U{cEJJu|/CsXX]M]`eMeGe/f$gghhrrENrFrErrr]r]fsiViirr] QuickGELUc*\rSrSrSrS\S\4SjrSrg)QuickGELUActivationpzj Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs rr c:U[R"SU-5-$)NgZd;?)r(sigmoidr-s rr2QuickGELUActivation.forwardvsu}}UU]333rrENrFrErrrcrcps4V44rrccJ^\rSrSrSrS\S\4U4SjjrS\S\4SjrS r U=r $) ClippedGELUActivationzar Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to https://huggingface.co/papers/2004.09602. Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://huggingface.co/papers/1606.08415 minmaxch>X:a[SUSUS35e[TU] 5 XlX lg)Nzmin should be < max (got min: z, max: )) ValueErrorrrrkrl)rrkrlrs rrClippedGELUActivation.__init__s8 9=cU'#aPQ Q rxr cl[R"[U5URUR5$r0)r(cliprrkrl)rrqs rr2ClippedGELUActivation.forwards!zz$q'488TXX66r)rlrk) r6r7r8r9r:floatrrr2r<r=r>s@rririzs3 E77F77rric>^\rSrSrSrU4SjrS\S\4SjrSrU=r $)AccurateGELUActivationz Applies GELU approximation that is faster than default and more accurate than QuickGELU. See: https://github.com/hendrycks/GELUs Implemented along with MEGA (Moving Average Equipped Gated Attention) cz>[TU]5 [R"S[R- 5Ulg)N)rrr)r*r+precomputed_constantrrs rrAccurateGELUActivation.__init__s' $(IIa$''k$:!rrr c SU-S[R"URUS[R"US5---5--$)Nr#rr&)r(rr{r,r-s rr2AccurateGELUActivation.forwardsEU{a%**T-F-F%RZ]b]f]fglno]pRpJp-q"rrssr)r{) r6r7r8r9r:rrr2r<r=r>s@rrwrws);tVtttrrwcP^\rSrSrSrU4SjrS\S\4SjrS\S\4SjrSr U=r $) MishActivationz See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://huggingface.co/papers/1908.08681). Also visit the official repository for the paper: https://github.com/digantamisra98/Mish c`>[TU]5 [RRUlgr0)rrrrmishrr|s rrMishActivation.__init__s ==%%rrr cnU[R"[RR U55-$r0)r(rrrsoftplusr-s r _mish_pythonMishActivation._mish_pythons%uzz"--"8"8"?@@@rc$URU5$r0r1r-s rr2MishActivation.forwardr4rr1) r6r7r8r9r:rrrr2r<r=r>s@rrrs; &A&AVAVrrc*\rSrSrSrS\S\4SjrSrg)LinearActivationzS Applies the linear activation function, i.e. forwarding input directly to output. rr cU$r0rEr-s rr2LinearActivation.forwards rrENrFrErrrrsVrrc"\rSrSrSrSSjrSrg)LaplaceActivationz Applies elementwise activation based on Laplace function, introduced in MEGA as an attention activation. See https://huggingface.co/papers/2209.10655 Inspired by squared relu, but with bounded range and gradient for better stability cX- RU[R"S5-5nSS[R"U5--$)Nr%r#r$)divr)r*r(rP)rrmusigmas rr2LaplaceActivation.forwards:  3!78cEIIe,,--rrEN)g۞?g ^/ ?r6r7r8r9r:r2r<rErrrrs .rrc\rSrSrSrSrSrg)ReLUSquaredActivationzX Applies the relu^2 activation introduced in https://huggingface.co/papers/2109.08668v2 cp[RRU5n[R"U5nU$r0)rrrelur(square)rr relu_appliedsquareds rr2ReLUSquaredActivation.forwards)}}))%0 ,,|,rrENrrErrrrs rrc(^\rSrSrU4SjrSrU=r$)ClassInstantiercl>[TU]U5n[U[5(aUOU04up4U"S0UD6$)NrE)r __getitem__ isinstancetuple)rkeycontentclskwargsrs rrClassInstantier.__getitem__s7'%c*!+GU!;!;g'2 }V}rrE)r6r7r8r9rr<r=r>s@rrrs rrc^\rSrSrSrSSSS\R S4U4SjjrS\S \4S jr S\S \4S jr S \S \4S jr Sr U=r $)XIELUActivationz Applies the xIELU activation function introduced in https://arxiv.org/abs/2411.13010 If the user has installed the nickjbrowning/XIELU wheel, we import xIELU CUDA Otherwise, we emit a single warning and use xIELU Python g?r#gưFc P>[T U]5 [R"[R "[R "[R"XS955RS55Ul [R"[R "[R "[R"X#- US955RS55Ul URS[R"X5S95 URS[R"XES95 X`l [URR5R!5R5R#55Ul[UR&R5R!5R5R#55UlSUlSSKn[R.R0R35UlSnSSKJn U "UR85UlUS- n[>RAU5 g![<a$n USU S 3- nUR8UlSn A N?Sn A ff=f![<a)n [>RAS [CU 55 Sn A gSn A ff=f) N)dtyperbetaepszUsing experimental xIELU CUDA.)allow_in_graphz& Enabled torch._dynamo for xIELU CUDA.z+ Could not enable torch._dynamo for xIELU (z*) - this may result in slower performance.uCUDA-fused xIELU not available (%s) – falling back to a Python version. For CUDA xIELU (experimental), `pip install git+https://github.com/nickjbrowning/XIELU`)"rrr Parameterr(logexpm1tensor unsqueezealpha_palpha_nregister_bufferwith_vector_loadsrurdetachcpuitem _beta_scalarr _eps_scalar_xielu_cuda_obj xielu.opsclassesxieluXIELU torch._dynamor _xielu_cuda_xielu_cuda_fn Exceptionlogger warning_oncestr) r alpha_p_init alpha_n_initrrrrrmsgrerrrs rrXIELUActivation.__init__s ||EIIekk%,,|:a.b$c$m$mno$pq || IIekk%,,|/B%"PQ R \ \]^ _   VU\\$%DE UELL$BC!2!$))"2"2"4"8"8":"@"@"B"G"G"IJ !2!6!6!8!>!>!@!E!E!GH#  #(==#6#6#<#<#>D 2C 78&4T5E5E&F#??    $ 7DSEIstt&*&6&6## 7    jC   sB3I2 "I+I2 I/ I*%I2*I//I22 J%<J  J%rqr c [RRUR5nUR[RRUR 5-n[ R"US:X!-U-URU--[ R"[ R"XR55U- U-URU--5$)Nr) rrrrrrr(whererrkr)rrqrrs r _xielu_pythonXIELUActivation._xielu_pythons--((6))bmm44T\\BB{{ E K!Odii!m + [[1hh/ 01 4 ?$))a- O  rcURnUR5S:a'URS5nUR5S:aM'UR5S:a"URSSUR S55nX!R:wa![ R SUUR5 URRUURRUR5URRUR5URURUR5nURU5$)zDFirewall function to prevent torch.compile from seeing .item() callsrrrz_Warning: xIELU input tensor expects 3 dimensions but got (shape: %s). Reshaping to (shape: %s).)shapedimrviewsizerrrr2rtorrrrr)rrqoriginal_shaperesults rrXIELUActivation._xielu_cudaseegk AAeegk 557Q;r1affRj)A WW $   q  %%-- LLOOAGG $ LLOOAGG $        " " {{>**rrcURbFUR(a5[5(dURU5$[R S5 UR U5$)Nz:torch._dynamo is compiling, using Python version of xIELU.)ris_cudar rrrrr-s rr2XIELUActivation.forward1sN    + +--**511##$`a!!%((r)rrrrrrr)r6r7r8r9r:r(bfloat16rrrrr2r<r=r>s@rrrsd  nn)V v & +V++2)V)))rrrgelu_10i )rkrl gelu_fastgelu_new gelu_pythonrKTgelu_pytorch_tanhgelu_python_tanhr gelu_accuratelaplace leaky_relulinearr quick_gelurrelu2relu6rf)rYswishrprelurc U[;a [U$[SUS[[R5535e)Nz function z not found in ACT2FN mapping )ACT2FNKeyErrorlistkeys)activation_strings rget_activationrUsBF"'((#4"55RSWX^XcXcXeSfRghiirrY)1rr) collectionsrr(rrintegrations.hub_kernelsrutilsrutils.import_utilsr get_loggerr6rModuler rArIrVr]rcrirwrrrrrr LeakyReLUReLUReLU6SigmoidrTTanhPReLUACT2CLSrrrrrrrrYr linear_actrErrrs # A8   H %Z(ryy).Y'w w(wV$RYY%,V$ )RYY )% )Z(ii)i[)4"))4*47BII72 tRYY t RYY"ryy . .BIIk[)bii[)|  N  %s2'>? # !   N%6$=>     $:D#AB +   ",,    N %  BGG  "   RXX! "rzz# $  WW GG XX - 0  !j]+ * %f ; ' L ) ff H % r