9i bddlZddlZddlmZdZdZdZdZdZ dZ dd Z ejejjejd k\r&d Zejj j"ZeZneZeZee eee ee ejj j&ejj j(ejj j*ejj j*ejj j,d Zd Zy)N)versionc tj|}ddtjj|tjtj d|j z zz}||zS)av Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) Also see https://arxiv.org/abs/1606.08415 ??@)tfconvert_to_tensormatherfcastsqrtdtype)xcdfs e/var/www/html/backtest/airagagent/rag_env/lib/python3.12/site-packages/transformers/activations_tf.py_gelursU QA rww{{1rwwrwws|QWW'E#EFF GC s7Nc ntj|}tjtj|j }tjd|j }ddtj tjd|z ||tj|dzzzzz}||zS)z Gaussian Error Linear Unit. This is a smoother version of the GELU. Original paper: https://arxiv.org/abs/1606.0841 Args: x: float Tensor to perform activation Returns: `x` with the GELU activation applied. Hm?rrr) rr r r pirtanhr pow)rrcoeffrs r _gelu_newr"s QA !'' "B GGHagg &E rwwrwwsRx0Aq! 8L4LMNN OC s7Nrctj|}|tjtjj |zS)N)rr rr softplusrs rmishr4s5 QA rwwrww''*+ ++rctj|}tjd|j}tjd|j}d|zdtj||zd||z|zzzzzS)Nrg3E?rr)rr r rr)rcoeff1coeff2s r gelu_fastr#:sm QA WWXqww 'F WW\177 +F 7cBGGAJ# Q2F$GHH IIrctj|}tjd|j}|tjj ||zzS)NgZd;?)rr r rr sigmoid)rrs r quick_gelur&BsB QA GGE177 #E rwwuqy) ))rcBtjt|ddS)a Clip the range of possible GeLU outputs between [-10, 10]. This is especially useful for quantization purpose, as it allows mapping 2 negatives values in the GeLU spectrum. For more information on this trick, please refer to https://arxiv.org/abs/2004.09602 Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) Also see https://arxiv.org/abs/1606.08415 :param x: :return: i )r clip_by_valuerrs rgelu_10r*Hs   E!Hc2 ..rc|tj|d|\}}|tjj|zS)a Gated Linear Unit. Implementation as defined in the original paper (see https://arxiv.org/abs/1612.08083), where the input `x` is split in two halves across a dimension (`axis`), A and B, returning A * sigmoid(B). Args: `x`: float Tensor to perform activation `axis`: dimension across which `x` be split in half Returns: `x` with the GLU activation applied (with its size halved across the dimension `axis`). )axis)rsplitr r%)rr-abs rglur1Vs2 88Aqt $DAq rwwq! !!rz2.4cXtjjj|dS)NT) approximate)rkeras activationsgelurs rapproximate_gelu_wrapr7hs"xx##(((==r) r6r*r#gelu_newr1rr&relur%siluswishrc ||tvr t|Std|dttj)Nz function z not found in ACT2FN mapping )ACT2FNKeyErrorlistkeys)activation_strings rget_tf_activationrBsBF"'((#4"55RSWX^XcXcXeSfRghiir))r tensorflowr packagingrrrrr#r&r*r1parseVERSIONr7r4r5r6r8r9r%r;rr=rBrrrIs   $, J* / " 7==##$  e(<<> 88   $ $D$H DH    HH % %xx##++ HH & & XX ! ! ' ' HH % %  jr