;iSrSSKJr SSKrSSKrSSKrSSKJr SSKJ r J r SSK J r J r JrJrJrJr SSKJr SSKJr SS KJr SS KJrJrJrJr SSKrSS K J!r! SS K"J#r#J$r$J%r%J&r& \RN"\(5r)"S S\5r*"SS\SS9r+"SS\5r,"SS\SS9r-"SS5r.\\,\-\/4r0"SS\5r1\\1\/4r2"SS\.5r3S'Sjr4Sr5S(Sjr6"S S!5r7S)S"jr8S*S#jr9S+S$jr:S,S%jr;\\ \ \RL\ \RJ/\\,\-44\ \<\RL\<\RJ/\\,\-444r=S-S&jr>g!\a SS KJrJrJrJr GN+f=f).z?This module contains the evaluator classes for evaluating runs.) annotationsN)abstractmethod) AwaitableSequence)AnyCallableLiteralOptionalUnioncast) TypedDict run_helpers)schemas) BaseModelFieldValidationError validator)wraps) SCORE_TYPE VALUE_TYPEExampleRunc0\rSrSr%SrS\S'S\S'Srg) Category/z$A category for categorical feedback.Optional[Union[float, int]]valuestrlabelN__name__ __module__ __qualname____firstlineno____doc____annotations____static_attributes__r!n/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/langsmith/evaluation/evaluator.pyrr/s. &&C J&r*rcH\rSrSr%SrS\S'S\S'S\S'S\S 'S rg ) FeedbackConfig8zaConfiguration to define a type of feedback. Applied on on the first creation of a feedback_key. z0Literal['continuous', 'categorical', 'freeform']typerminmaxz%Optional[list[Union[Category, dict]]] categoriesr!Nr"r!r*r+r-r-8s* ;: $$; $$A55r*r-F)totalc\rSrSr%SrS\S'SrS\S'SrS\S 'SrS \S 'Sr S \S '\ "\ S9r S\S'Sr S\S'SrS\S'SrS\S'SrS \S'"SS5r\"S SS9S5rSrg)EvaluationResultGzEvaluation result.rkeyNrscorerrz Optional[str]commentzOptional[dict] correction)default_factorydictevaluator_infoz%Optional[Union[FeedbackConfig, dict]]feedback_configOptional[Union[uuid.UUID, str]] source_run_id target_run_idextrac\rSrSrSrSrSrg)EvaluationResult.ConfigbzPydantic model configuration.Fr!N)r#r$r%r&r' allow_extrar)r!r*r+ConfigrDbs + r*rGT)precSU;dUSc3[U[[45(a[R SU35 U$)z$Check that the value is not numeric.r8zJNumeric values should be provided in the 'score' field, not 'value'. Got: ) isinstanceintfloatloggerwarning)clsvvaluess r+check_value_non_numeric(EvaluationResult.check_value_non_numericgsH & F7O$;!c5\**C! r*r!)r#r$r%r&r'r(r8rr9r:rr<r=r>r@rArBrGrrRr)r!r*r+r5r5Gs H@E:0E:8!G]!2!%J%: 6ND65=AO:A;59M29659M29!E> ) wD! " r*r5c$\rSrSr%SrS\S'Srg)EvaluationResultsvzeBatch evaluation results. This makes it easy for your evaluator to return multiple metrics at once. zlist[EvaluationResult]resultsr!Nr"r!r*r+rUrUvs $#!r*rUcb\rSrSrSr\SSSjj5rSSSjjrSrg) RunEvaluatorzEvaluator interface class.Ncg)zEvaluate an example.Nr!)selfrunexampleevaluator_run_ids r+ evaluate_runRunEvaluator.evaluate_runsr*c^^^^^# [R"5mUUUUU4Sjn[R"5R SU5IShvN $N7f)z#Evaluate an example asynchronously.c>[R"S0TD6 TRTTT5sSSS5 $!,(df  g=f)Nr!)rhtracing_contextr`)current_contextr_r^r]r\sr+_run_with_context5RunEvaluator.aevaluate_run.._run_with_contexts3##6o6((g7GH766s 5 AN)rdget_tracing_contextasyncioget_running_looprun_in_executor)r\r]r^r_rgrfs```` @r+ aevaluate_runRunEvaluator.aevaluate_runsG002 I I--/??FWXXXXsAAAAr!NNr]rr^Optional[Example]r_Optional[uuid.UUID]return*Union[EvaluationResult, EvaluationResults]) r#r$r%r&r'rr`rmr)r!r*r+rYrYs$&*04 # ###. # 4 ##&*04 Y  Y# Y. Y 4 Y Yr*rYcP\rSrSr%SrS\S'S\S'SrS\S 'SrS \S 'S rg) ComparisonEvaluationResultzFeedback scores for the results of comparative evaluations. These are generated by functions that compare two or more runs, returning a ranking or other feedback. rr7z'dict[Union[uuid.UUID, str], SCORE_TYPE]scoresNr?r@z6Optional[Union[str, dict[Union[uuid.UUID, str], str]]]r9r!) r#r$r%r&r'r(r@r9r)r!r*r+rvrvs8 H@ 33459M296FJG CJ:r*rvc^\rSrSrSrS SSjjrSSSjjrSSjrSSjr\ SSj5r SSSjjr SSU4S jjjr S SS jjr SS jrS rU=r$)DynamicRunEvaluatoraA dynamic evaluator that wraps a function and transforms it into a `RunEvaluator`. This class is designed to be used with the `@run_evaluator` decorator, allowing functions that take a `Run` and an optional `Example` as arguments, and return an `EvaluationResult` or `EvaluationResults`, to be used as instances of `RunEvaluator`. Attributes: func (Callable): The function that is wrapped by this evaluator. c^[U5unmU(a[U5unmS U4Sjjn[U5"U5 SSKJn Ub&UR X#S9Ul[ USS5Ul[R"U5(a5Ub [S5eUR XS9Ul[ USS5UlgUR [[[[[/[ 4U5US9Ul[ USS5Ulg) zInitialize the DynamicRunEvaluator with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns a dict or `ComparisonEvaluationResult`. cf>TcU$T"URS5URS55u pU$)Nr]r^getinputs_ traced_inputsprepare_inputss r+process_inputs4DynamicRunEvaluator.__init__..process_inputss<% $2 5!6::i#8% !Q! r*rrNrr#rzFunc was provided as a coroutine function, but afunc was also provided. If providing both, func should be a regular function to avoid ambiguity.rr<rsr<)_normalize_evaluator_funcr langsmithrensure_traceableafuncgetattr_nameinspectiscoroutinefunction TypeErrorr rrr r_RUNNABLE_OUTPUTfuncr\rrrrrs @r+__init__DynamicRunEvaluator.__init__s(";4!@~ &?&F #UN ! d D)  $556DJ! 4IJDJ  & &t , , 3 %556DJ!z3HIDJ#44XsHW$568HHI4P-5DI!z3HIDJr*cv^[T[5(aTR(dUTlT$T(d[ST35eST;aU(aURTS'[ U4SjS55(a[ST35e[S0SU0TED6$![ an[ST35UeSnAff=f) NziExpected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got empty result: r7c3,># UH oT;v M g7fNr!).0kresults r+ @DynamicRunEvaluator._coerce_evaluation_result.. sJ,IqF?,I)r8rr9zrExpected an EvaluationResult object, or dict with a metric 'key' and optional 'score' or categorical 'value'; got r@z[Expected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got r!)rJr5r@ ValueErrorrallr)r\rr@ allow_no_keyes ` r+_coerce_evaluation_result-DynamicRunEvaluator._coerce_evaluation_results f. / /'''4$M  FFLXOF"| $ u J,IJJJ OOUhX$Q &P&PQ Q 44:8=  sA'B B8$B33B8cSU;aAUR5nUSVs/sHnURXBS9PM snUS'[S0UD6$UR[[U5USS9$s snf)NrW)r@T)r@rr!)copyrrUr r<)r\rWr@cprs r+_coerce_evaluation_results.DynamicRunEvaluator._coerce_evaluation_resultss  B!++A..q.N+ByM%*r* *-- w }4.  sA'c[U[5(aUR(dX!lU$[U5nUR X5$r)rJr5r@_format_evaluator_resultr)r\rr@s r+_format_result"DynamicRunEvaluator._format_result)sA f. / /'''4$M)&1..vEEr*c[US5$zCheck if the evaluator function is asynchronous. Returns: bool: True if the evaluator function is asynchronous, False otherwise. rhasattrr\s r+is_asyncDynamicRunEvaluator.is_async7tW%%r*c[US5(dU[R"5nUR5(a [ S5eUR UR X55$Uc[R"5nSUR0n[USS5(a[UR5US'URUUX5S.S9nURXc5$) aUEvaluate a run using the wrapped function. This method directly invokes the wrapped function with the provided arguments. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. rtCannot call `evaluate_run` on an async run evaluator from within an running event loop. Use `aevaluate_run` instead.NrA session_id experimentrun_idmetadatalangsmith_extra)rrjget_event_loop is_running RuntimeErrorrun_until_completermuuiduuid4idrrrrr)r\r]r^r_ running_looprrs r+r` DynamicRunEvaluator.evaluate_run@s"tV$$"113L&&(("R $66t7I7I#7WXX  ##zz| $3SVV#< 3 d + +%(%8H\ "  '7N  ""6<# [US5(d[TU] X5IShvN $Uc[R"5nSUR 0n[ USS5(a[UR5US'URUUX4S.S9IShvN nURXS5$NN7f)a|Evaluate a run asynchronously using the wrapped async function. This method directly invokes the wrapped async function with the provided arguments. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. rNrArrrr) rsuperrmrrrrrrrr)r\r]r^r_rr __class__s r+rm!DynamicRunEvaluator.aevaluate_runfs&tW%%.s<< <  ##zz| $3SVV#< 3 d + +%(%8H\ "zz  '7N"   ""6<<=  s"$B,B(A*B,B*B,*B,c$URX5$)aMake the evaluator callable, allowing it to be used like a function. This method enables the evaluator instance to be called directly, forwarding the call to `evaluate_run`. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. )r`)r\r]r^s r+__call__DynamicRunEvaluator.__call__s  ..r*c"SURS3$))Represent the DynamicRunEvaluator object.zrrs r+__repr__DynamicRunEvaluator.__repr__s&tzzl!44r*rrrr)rXCallable[[Run, Optional[Example]], Union[_RUNNABLE_OUTPUT, Awaitable[_RUNNABLE_OUTPUT]]]rzIOptional[Callable[[Run, Optional[Example]], Awaitable[_RUNNABLE_OUTPUT]]])F)rzUnion[EvaluationResult, dict]r@ uuid.UUIDrboolrsr5)rWzUnion[dict, EvaluationResults]r@rrsrt)rzMUnion[EvaluationResult, EvaluationResults, dict, str, int, bool, float, list]r@rrsrtrsrrorp)r]rr^rqr_rr)r]rr^rqrsrtrsr)r#r$r%r&r'rrrrpropertyrr`rmrrr) __classcell__)rs@r+rzrzsE, 8J 8J 8J|# -!   < / !  4  " F  F ! F 4 F&&&*04 $= $=#$=. $= 4 $=R&*04 = =#=. ==D6://!2/ 3/"55r*rzc[U5$)zeCreate a run evaluator from a function. Decorator that transforms a function into a `RunEvaluator`. )rzrs r+ run_evaluatorrs t $$r*i'c`[U5n[U5[:aUS[S- S-nU$)Nz...))reprlen_MAXSIZE)objss r+ _maxsize_reprrs1 S A 1v n1  & Hr*c\rSrSrSrS SSjjr\SSj5rS SSjjrS SSjjr S SSjjr SS jr \ SS j5r SS jrS rg)DynamicComparisonRunEvaluatoriz4Compare predictions (as traces) from 2 or more runs.Nc(^[U5unmU(a[U5unmS U4Sjjn[U5"U5 SSKJn Ub&UR X#S9Ul[ USS5Ul[R"U5(a5Ub [S5eUR XS9Ul[ USS5UlgUR [[[[[[ /["4U5US9Ul[ USS5Ulg) zInitialize the DynamicRunEvaluator with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns an `EvaluationResult` or `EvaluationResults`. cf>TcU$T"URS5URS55u pU$)Nrunsr^r~rs r+r>DynamicComparisonRunEvaluator.__init__..process_inputss<% $2 6"FJJy$9% !Q! r*rrNrr#rzrr)$_normalize_comparison_evaluator_funcrrrrrrrrrrr rrrr r_COMPARISON_OUTPUTrrs @r+r&DynamicComparisonRunEvaluator.__init__s%("Fd!K~ &J5&Q #UN ! d D)  $556DJ! 4IJDJ  & &t , , 3 %556DJ!z3HIDJ#44!#(9:*,  .5 DI!z3HIDJr*c[US5$rrrs r+r&DynamicComparisonRunEvaluator.is_asyncrr*cd[US5(dU[R"5nUR5(a [ S5eUR UR X55$[R"5nURU5nURUUXES.S9nURXdU5$)zCompare runs to score preferences. Args: runs: A list of runs to compare. example: An optional example to be used in the evaluation. rrrtagsr) rrjrrrr acompare_runsrr _get_tagsr_format_results)r\rr^rr@rrs r+ compare_runs*DynamicComparisonRunEvaluator.compare_runsstV$$"113L&&(("R $66&&t5 ~~d#  '4C  ##F4@@r*c# [US5(dURX5$[R"5nUR U5nUR UUX4S.S9IShvN nUR XSU5$N7f)anEvaluate a run asynchronously using the wrapped async function. This method directly invokes the wrapped async function with the provided arguments. Args: runs (Run): The runs to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. rrrN)rrrrrrr)r\rr^r@rrs r+r+DynamicComparisonRunEvaluator.acompare_runss| tW%%$$T3 3 ~~d#zz  '4C"   ##F4@@  sAA:!A8"A:c$URX5$)ayMake the evaluator callable, allowing it to be used like a function. This method enables the evaluator instance to be called directly, forwarding the call to `evaluate_run`. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. )r)r\rr^s r+r&DynamicComparisonRunEvaluator.__call__:s  //r*c"SURS3$)rz>r*c/nUHenURS[UR5-5 [USS5(dM>URS[UR5-5 Mg U$)zExtract tags from runs.zrun:rNz experiment:)appendrrrr)rrr]s r+r'DynamicComparisonRunEvaluator._get_tagsOs\C KKSVV, -sL$// MC,??@ r*c[U[5(aUR(dX!lU$[U[5(a:[ X15VVs0sHupEUR U_M snnUR US.nO<[U[5(aSU;aUR US'OSU<3n[U5e[S0SU0UED6$s snnf![an[SU35UeSnAff=f)N)rxr7r@r7zXExpected 'dict', 'list' or 'ComparisonEvaluationResult' result object. Received: result=r@zExpected a dictionary with a 'key' and dictionary of scores mappingrun IDs to numeric scores, or ComparisonEvaluationResult object, got r!) rJrvr@listziprrr<rr)r\rr@rr]r8msgrs r+r-DynamicComparisonRunEvaluator._format_resultsZs  f8 9 9'''4$M  % %;>t;LM;LZS3665=;LMzz!.F  % %F" $ u -%+I/ S/ ! -"M >J >J >J@&&AEA!A,=A #A@AEA!A,=A #A:AE0!0,=0 #0"?"="!" " $ "r*rc[U5$)z.Create a comaprison evaluator from a function.)rrs r+comparison_evaluatorrs ) ..r*c ^^^ ^ ^ Sm [R"T5m T RR5VVs/sH#upURUR :wdM!UPM% nnnT RR5VVs/sH0upUR [RRLdM.UPM2 snnm U(aB[U U 4SjU55(d8[UVs/sH oDT ;dM UPM sn5S:waST S3n[U5e[U U 4SjU55(aUSS/:XaTS4$[R"T5(aOSU 4S jjmSUU4S jjn[TS 5(a [TS 5O URUlUT4$SU 4S jjmSUU4S jjn[TS 5(a [TS 5O URUlUT4$s snnfs snnfs snf)N)r]r^routputsreference_outputs attachmentsc3D># UHoT;=(d UT;v M g7frr!rpnameargs_with_defaultssupported_argss r+r,_normalize_evaluator_func..& PXu^ # Bu0B'B BPX UInvalid evaluator function. Must have at least one argument. Supported arguments are . Please see https://docs.smith.langchain.com/evaluation/how_to_guides/evaluation/evaluate_llm_application#use-custom-evaluatorsc3D># UHoT;=(d UT;v M g7frr!rs r+rr%LT5>5,>#>>Hrr]r^c>UUU(a URO0UR=(d 0U(aUR=(d 0O0U(aUR=(d 0O0S.n0n/n0nTRR 5HeupgXb;dM UR UR UR4;aURX&5 OX&X6'US;a [X&5OX&XV'Mg XCU4$N)r]r^rrrr)r]r^ rrr parametersitemskindPOSITIONAL_OR_KEYWORDPOSITIONAL_ONLYrr r]r^arg_mapkwargsargsr param_nameparamsigs r+_prepare_inputs2_normalize_evaluator_func.._prepare_inputs&07gnnR"{{0b@G7#6#6#<"RBI)>Br  " ),)=)=)?%J!, ::!77!11*!KK(;<181DF. *-??*'*=>!(!4&1*@]22r*cF># T"X5up#nT"U0UD6IShvN $N7frr!r]r^r-r,rr1rs r+awrapper+_normalize_evaluator_func..awrappers-%4C$A!q!4262222 !!r#c>UUU(a URO0UR=(d 0U(aUR=(d 0O0U(aUR=(d 0O0S.n0n/n0nTRR 5HeupgXb;dM UR UR UR4;aURX&5 OX&X6'US;a [X&5OX&XV'Mg XCU4$r#r$r*s r+r1r2r3r*c*>T"X5up#nT"U0UD6$rr!r5s r+wrapper*_normalize_evaluator_func..wrappers"$3C$A!qT,V,,r*)r]rr^rqrstuple[list, dict, dict])r]rr^rqrsrr signaturer%r&r' VAR_KEYWORDdefault Parameteremptyrrrrrrr# rrpall_argsar r6r;r1rr0rs ` @@@@r+rrs9N   D !C&)nn&:&:&<X&<(%!--@W& 3 3#4 3! 3 34,,j)&&   o. . 3 3#4 3( 3> - - 4,,j)%%   _- -kYEs# G4G4-G:6G:+ H8Hc^^^ ^ ^ Sm [R"T5m T RR5VVs/sH#upURUR :wdM!UPM% nnnT RR5VVs/sH0upUR [RRLdM.UPM2 snnm U(aB[U U 4SjU55(d8[UVs/sH oDT ;dM UPM sn5S:waST S3n[U5e[U U 4SjU55(aUSS/:XaTS4$[R"T5(aOSU 4S jjmSUU4S jjn[TS 5(a [TS 5O URUlUT4$SU 4S jjmSUU4S jjn[TS 5(a [TS 5O URUlUT4$s snnfs snnfs snf)Nrr^rrrc3D># UHoT;=(d UT;v M g7frr!rs r+r7_normalize_comparison_evaluator_func..%rrrrrc3D># UHoT;=(d UT;v M g7frr!rs r+rrK3r!rrr^c>UUU(a URO0UVs/sHo"R=(d 0PM snU(aUR=(d 0O0S.n0n/n0nT RR5HeupxXs;dM URUR UR 4;aURX75 OX7XG'US;a [X75OX7Xg'Mg XTU4$s snfNrI)rr^ rrr%r&r'r(r)rr rr^r]r+r,r-rr.r/r0s r+r1=_normalize_comparison_evaluator_func.._prepare_inputs=!&07gnnR=ABTc 1r 1TBBI)>Br  " ),)=)=)?%J!, ::!77!11*!KK(;<181DF. *-@@*'*=>!(!4&1*@]22+ CC) cF># T"X5up#nT"U0UD6IShvN $N7frr!rr^r-r,rr1rs r+r66_normalize_comparison_evaluator_func..awrapper[s-%4D$B!q!4262222r8r#c>UUU(a URO0UVs/sHo"R=(d 0PM snU(aUR=(d 0O0S.n0n/n0nT RR5HeupxXs;dM URUR UR 4;aURX75 OX7XG'US;a [X75OX7Xg'Mg XTU4$s snfrNrOrPs r+r1rQjrRrSc*>T"X5up#nT"U0UD6$rr!rUs r+r;5_normalize_comparison_evaluator_func..wrappers$%4D$B!qT,V,,r*)rr r^rqrsr=)rr r^rqrsrr>rDs ` @@@@r+rrsPSN   D !C&)nn&:&:&<X&<(%!--@W&.s71:a&&sz8Expected a list of dicts or EvaluationResults. Received .rWrzZExpected a dict, str, bool, int, float, list, EvaluationResult, or EvaluationResults. Got ) rJrrLrKrrrrr<)rs r+rrs&4,--6"* M);;A( D   FD ! !7777J6(RST V$ M FC 6" M FD ! ! M &&,X /  r*c^^^^ Sm [R"T5mTRR5VVs/sHupUPM nnnTRR5VVs/sH0upUR[R R LdM.UPM2 snnmU(aB[UU 4SjU55(dH[UVs/sH oDT;dM UPM sn5S:wa!ST S3nU(a USUS3- n[U5e[U 4SjU55(aUSS /:XaT$S UU4S jjn[TS 5(a[TS 5Ul U$URUl U$s snnfs snnfs snf) Nrexamplesrrrc3D># UHoT;=(d UT;v M g7frr!rs r+r/_normalize_summary_evaluator..rrrrr^z Received arguments c3,># UH oT;v M g7frr!)rrrs r+rrcs?hU.(hrrrac $>UUUVs/sHo"RPM snUVs/sHo3R=(d 0PM snUVs/sHo"R=(d 0PM snS.n0n/nT RR5HMupxXt;dM URUR UR 4;aURXG5 MGXGXW'MO T "U0UD6n [U [5(aU $[U 5$s snfs snfs snf)Nr`) rrr%r&r'r(r)rrJr5r) rrar^r]r+r,r-r.r/rrr0s r+r;-_normalize_summary_evaluator..wrappers$9ABg>>B9=>#KK-2->KS%T8oo&;&;8%T GFD%(^^%9%9%;! (zz33--& G$78-4-@*&<4*6*F&"233 +F3 3)C>%TsDD D r#)rzSequence[schemas.Run]razSequence[schemas.Example]rsrt) rr?r%r&rArBrCrrrrrr#) rrrErFrGr r;rr0rs ` @@@r+_normalize_summary_evaluatorrgsSN   D !C&)nn&:&:&<=&<(%&EsE7(-E=E= FF)rr)rr)rr rsr)rrrsztuple[Union[Callable[[Run, Optional[Example]], _RUNNABLE_OUTPUT], Callable[[Run, Optional[Example]], Awaitable[_RUNNABLE_OUTPUT]]], Optional[Callable[..., dict]]])rrrsztuple[Union[Callable[[Sequence[Run], Optional[Example]], _COMPARISON_OUTPUT], Callable[[Sequence[Run], Optional[Example]], Awaitable[_COMPARISON_OUTPUT]]], Optional[Callable[..., dict]]])rz;Union[EvaluationResults, dict, str, int, bool, float, list]rszUnion[EvaluationResults, dict])rrrsSUMMARY_EVALUATOR_T)?r' __future__rrjrrabcrcollections.abcrrtypingrrr r r r typing_extensionsr rrrdr pydantic.v1rrrr ImportErrorpydanticlogging functoolsrlangsmith.schemasrrrr getLoggerr#rMrr-r5rUrYr<rrvrrzrrrrrrrrrrhrgr!r*r+rusE" /(' BB   8 $'y' 6Ye 6,y,^" "YY8)+s E((E>=E>