ievSrSSKJr SSKrSSKrSSKrSSKJrJr SSKJ r SSK J r J r J r SSKJr \ "\5R#5R$r\S- S - r\S- S - r\R-S S S 9 /S QrSSjrSSjrSSjrSSjrSSjrSSjrSSSjjr\S:Xa\R>"\"55 gg)z Lightweight QA regression harness for the RAG system. Usage: python -m tests.run_regression python -m tests.run_regression --categories history,finance --limit 5 ) annotationsN)datetimetimezone)Path)DictListAny) RAGSystem baselinesz qa_cases.jsonresultsT)parentsexist_ok)z based on the available documentsz"i analyzed the available documentszno documents availablezno relevant documents foundz$according to the distilled documentsc[[SSS9n[R"U5nSSS5 U(aeUVs1sH oDR 5R 5iM" nnWVs/sH+nUR SS5R 5U;dM)UPM- nnUbWSUnW$!,(df  N=fs snfs snf)Nrutf-8encodingcategory)open BASELINE_PATHjsonloadstriplowerget) categorieslimitfcasesccategories_lowercases tests/run_regression.py load_casesr%#s mS7 3q !  47ABz!GGIOO-zB" "Txx B'--/3CC U   fu  L 4 3C sB%'B6'(B;B;% B3cV^UR5m[U4Sj[55$)Nc3,># UH oT;v M g7fN).0pattern answer_lowers r$ $is_generic_answer..6sG6F7,&6Fs)ranyGENERIC_PATTERNS)answerr,s @r$is_generic_answerr24s <<>L G6FG GGc ^USnURS/5mURS/5nURS/5nURU5n[ U[5(aUSURS5USUU/0S.$URS S 5nURS /5=(d /nUV s/sH+n [ U [ 5(dMU RS 5PM- n n UR 5n UV s/sHn U R 5U ;dMU PM n n UV s/sHn U R 5U ;dMU PM nn [U4S jU 55n[U5n[UR55nSnU (dU(dU(dSnUU UUUU SSS.nUSURS5UUUUUS.$![a0nUSURS5US[U5S/0S.sSnA$SnAff=fs sn fs sn fs sn f)Nquestionpreferred_sourcesrequired_termsoptional_termsidrerror)r9rr5statusr:r1sourcesmetricsr1rr<sourcec36># UHoT;dM Uv M g7fr(r))r*srcr6s r$r- evaluate_case..gs#.?'?|s  passfail) answer_lengthmissing_required_terms optional_hitspreferred_sources_hitgeneric_detected top_sources)r9rr5r;r1r<r=) rsearch_and_answer Exceptionstr isinstancedictrr/r2lensplit)ragr#r5r7r8 raw_resultexcr1r<r@ source_namesr,termmissing_requiredrG preferred_hitgenericrEr;r=r6s @r$ evaluate_caserZ9s6JH!4b9XX.3NXX.3N  **84 *c""t*,    ^^Hb )FnnY+1rG18R#JsDL'' ::<| +  (' ::<< '  #M 'G 'M F7-'"2&!.##BQ' G4jHHZ( u   t*, X    4SsAG2G=G=:HH"H>H G: %G5/G:5G:cD[U5n[SU55nX- n[SU55n0nUH^nURS5=(d SR5nUR USSSS.5nURUSS5S-XS'M` UUUUUS .$) Nc3:# UHoSS:XdM Sv M g7f)r;rBNr)r*results r$r-summarise..sGv8,<,F  c3:# UHoSS:XdM Sv M g7f)r;r:r]Nr)r^s r$r-r`sHv8,<,Grar uncategorizedr)rBrCr:r;r]) total_casespassesfailureserrors by_category)rPsumrr setdefault) r totalrerfrgrr_catbuckets r$ summariserns LE GG GF~H HH HF,.Jzz*%8??A&&sQA,NO#)::fX.>#BQ#Fh  !  r3c[R"[R5R S5nUUUS.n[ US3- n[ USSS9n[R"X5SS9 SSS5 [ S - n[ USSS9n[R"X5SS9 SSS5 U$!,(df  NB=f!,(df  U$=f) Nz%Y%m%dT%H%M%SZ) generated_atsummaryr z.jsonwrr)indentz latest.json) rnowrutcstrftime RESULTS_DIRrrdump)r rq timestamppayload output_pathr latest_paths r$ save_resultsr~s X\\*334DEI!G 9+U 33K k3 1Q 'Q' 2 -K k3 1Q 'Q' 2  2 1 2 1 sB#B4# B14 CcUSnUSnUSnUSn[S5 [SU35 [SU35 [SU35 [S U35 [S 5 [US R55H[upV[S UR5S SUR SS5SUR SS5SUR SS535 M] g)Nrdrerfrgz === QA Regression Summary ===z Total cases: z Passes : z Failures : z Errors : z By category:rhz z<20z pass=rBrz fail=rCz error=r:)printsorteditemstitler)rqrkrerfrgrcountss r$ print_summaryrs M "E X Fz"H X F +, M% !" L !" L #$ L !" "7=#9#?#?#AB 8>>#C(vzz&!/D.EFjj+,GFJJw4J3KM NCr3c [R"SS9nURSSSSS9 URS[SSS9 URS S S S 9 UR U5n[ UR UR5nU(d[S [RS9 g[5nURR5 URR(d[S[RS9 gUR5 /nUGHn[!XF5nUR#U5 USR%5n[SUSUSSUS35 US:wdMUUR'S05n UR'S5(a[SUS35 OrU R'S5=(d /n U (a[SU 35 U R'SS5(d [S5 U R'S 5(a [S!5 UR((dGM O [+U5n [-X[5n [/U 5 [S"U 35 U S#S$:Xa U S%S$:XaS$$S$)&NzRun QA regression checks.) descriptionz --categoriescURS5Vs/sH)oR5(dMUR5PM+ sn$s snf)N,)rQr)sparts r$main..s,N  NNs AAz/Comma-separated list of categories to evaluate.)typedefaulthelpz--limitz"Optional limit on number of cases.z --fail-fast store_truez&Stop after the first failure or error.)actionrzNo cases matched the filters.)filer]z3Vector store is empty. Run setup before regression.r;[z] r9z :: r5PASSr=r:z error: rFz missing terms: rHFz. preferred sources not found in top resultsrIz generic response detectedz Detailed results saved to: rfrrg)argparseArgumentParser add_argumentint parse_argsr%rrrsysstderrr vector_storeload_existing_index documentsinitialize_modelrZappendupperr fail_fastrnr~r) argvparserargsr rRr r#r_r;r=missingrq save_paths r$mainrsB  $ $1L MF  N >     1    5   T "D t 3E  -CJJ? +C((*    % % C#**U$&Gs)v!'') &DJ?@ V jjB/Gzz'"" F7O#456!++&>?E2/y9:{{#:EBBJL;;1229:~~~%( GW.I' )) 56 #q(WX->!-C1JJr3__main__)rList[str] | Nonerz int | NonereturnList[Dict[str, Any]])r1rMrbool)rRr r#Dict[str, Any]rr)r rrr)r rrqrrr)rqrrNoner()rrrr) __doc__ __future__rrrrrrpathlibrtypingrrr rr __file__resolveparentROOT_DIRrrxmkdirr0r%r2rZrnr~r__name__exitr)r3r$rs# '"" > ! ! # * *;&8 $y0 $."H JZ*$N";K| zHHTVr3