{ildZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl m Z m Z ddl mZddlmZddlmZddlmZmZmZmZmZddlmZmZmZmZmZdd lm Z eGd d Z!d d dddddddddddddddddd dddddd dddddZ"gdZ#gdZ$gdZ%gdZ&gdZ'gdZ(gd Z)e#e$ze%ze&ze'ze(ze)zZ*d!d"d#d$d%d&d'd(Z+d)e,d*e-fd+Z.d)e,d*e/fd,Z0gd-Z1e(jee1gd.gd/gd0gd1gd2gd3gd4gd5gd6gd7d8 Z3gd9gd:gd;gd<gd=gd>gd?gd@dAZ4dBe,d*e,fdCZ5dDdEdFdGd!d&d%dHdIZ6gdJgdKgdLdMdNggdOdPZ7dBe,d*e,fdQZ8gdRgdSgdTgdUdVZ9dBe,d*e:fdWZ;gdXgdYgdZgd[gd\gd]gd^gd_d`ZgdcZ?dBe,d*e=fddZ@dBe,d*e:fdeZAdae,d*e=fdfZBdggdhfdigdjfdkgdlfdmgdnfdogdpfdqgdrfdsgdtfdugdvfgZCdwgdxgdydzdDgd{gd|dzdGgd}gd~dzdggddzdZDdDdBe,de:d*e/fdZEdDde:de:d*e/fdZFgddEdddgdddddgdddddgdddddgddGdddgddddddZGdBe,d*e=fdZHgdgdgdgddgdddggdgddgdddggdgddgdgdddggddgdgdgdgddgdgdgdgddgdgdgdgddgd¢gdâgdĢgdŢdƜdǜZIdae,d*e=fdȄZJdBe,d*e,fdɄZKdggd!d!dd!ddd˜ ZLde:d*e:fd̈́ZMdDZNd#ZO ddlPZPdZQ ddlSmTZUdZV ddlWmXZXdZY ddlZZ[dZ\ ddl]m^Z^dZ_ ddl`Z`dZa ddlbZbdZcGdӄdԫZdGdՄd֫ZeGdׄdثZfGdلdګZgGdۄdܫZheeidݬޫZj ddlkmlZlejjelende!ZpeeepZqefepZregepZsehepZtiZuddddddddde-ejjddd axe jZze jZ{e jZ}da~epjdz ZdZdZejjddZdZenderj ede,de,de-de,fdZde,dede,fdZejjddZejjddZde/d*e:fdZde/d*e:fdZde/d*e:fdZde:d*e:fdZejjddgdZejjddgdZejjddgedZejjd dgd Zejjd dged Zejjd dgedZejjddgdZejjddgdZejjddgdZejjddgdZejjdddgedZejjddgedZejjddgedZejjddgdZejjd dgd!Zejjd"dged#Zejjd$dgd%Zejjd&dgd'Zejjd(dged)Zgiige jDe jFjId*ae jZdad+Zd,Zdae,d-e:fd.Zd-e:d*e-fd/Zejjd0dgd1Zejjd2dgd3Zejjd4dgd5Zejjd6dgd7Z dd8l mZeeid:k(re-ejjd;d<Zend=eend>ead?e_d@eQendAeerjheejjkdBedϐCyy#eR$rdZQYwxYw#eR$rdZVYwxYw#eR$rdZYYwxYw#eR$rdZ\YwxYw#eR$rdZ_YwxYw#eR$rdZaYwxYw#eR$rdZcYwxYw#eR$rZoendeoYdZo[odZo[owwxYw#e$rZoend9eoYdZo[odZo[owwxYw(Ez Advanced Legal Document RAG Service Incorporates sophisticated PDF processing, FAISS vector search, BM25 hybrid scoring, and DeepSeek/Tesseract OCR fallback. Port: 5007 (configurable via RAG_PORT) N)datetimetimezone)Pathwraps) dataclass)ListDictAnyOptionalTuple)Flaskrequestjsonifysend_from_directoryrender_template)secure_filenameceZdZUdZeej Zeed<dZ eed<dZ eed<dZ eed<dZ e ed<d Ze ed <d Ze ed <d Ze ed<dZe ed<dZeed<dZeed<dZeed<d Ze ed<dZe ed<dZeed<dZeed<dZy) RAGConfigzRAG Service Configurationbase_dirNdata_dir cache_dirpdf_dir chunk_sizechunk_min_size@chunk_max_sizesentence_overlap min_sentences?hybrid_vector_weighthybrid_keyword_weight?hybrid_bm25_weightmax_chunks_per_source default_khttp://127.0.0.1:5003/ocrdeepseek_ocr_urlhttp://127.0.0.1:5002/ocrtesseract_ocr_urlc|j;ttjj d|j dz |_|j |jdz |_|j|jdz |_|j jdd|jjddy)NDATA_DIRdata rag_cachecsvT)parentsexist_ok) rrosenvirongetrrrmkdirselfs E/var/www/html/eventheodds/ai-lawyer-rag/advanced_legal_rag_service.py __post_init__zRAGConfig.__post_init__;s ==   DMMF> !!]][8DN << ==50DL TD9 4$7)__name__ __module__ __qualname____doc__r__file__parentr__annotations__rrrrintrrr!r#r%floatr&r(r)r+r-strr/r>r?r=rr s#(^**Hd*HdItGTJNCNCcM3#'%&#'5' $$!"3"Is8c78s8 8r?r rr )rrrr!r#rir"iixr) technicalresearchlegalsportsdefault)'zlegal doctrineholdingdicta precedentz stare decisisanalogydistinguishing syllogismzrule applicationz fact patternelementprongtestz daubert testzbalancing testzmulti-factor testzbright-line rulestandard of reviewde novoabuse of discretionz clear errorburden of productionzburden of persuasion presumption inferencezprima facie caseaffirmative defensegenuine issue of material fact moving partynon-moving partyzcelotex trilogyzreasonable doubtzpreponderance of evidencezclear and convincingstrict scrutinyintermediate scrutinyrational basiszmens reaz actus reus)+zcause of actionclaimmotion to dismiss12(b)(6)zrule 12motion for summary judgment discovery interrogatory depositionrequest for productionrequest for admissionz in limine voir diredirect examinationcross-examinationclosing argumentopening statementjury instructionappealwrit of certiorariremand settlement negotiation mediation arbitration injunctionztemporary restraining orderzpreliminary injunctionzpermanent injunctionspecific performancezcompensatory damageszpunitive damageszliquidated damagesznominal damagesztreble damages class actionjoinder intervention interpleaderzdeclaratory judgmentmandamus prohibitionz quo warranto)& complaintanswer counterclaim cross-claimthird-party complaintappellate brief trial briefzmemorandum of lawzmemorandum in supportzmemorandum in oppositionz reply briefz amicus briefzmajority opinionzconcurring opinionzdissenting opinionzplurality opinionz per curiamzbench memorandum demand lettercease and desistzengagement letterzretainer agreementzunilateral contractzbilateral contractzindemnity clausez hold harmlessz force majeurestatute regulation ordinancezlegislative historyzcommittee reportz floor debatezregulatory commentznotice and commentz proposed rulez final rulezconsent decree)* plaintiff defendant petitioner respondent appellantappelleemovantz non-movantzcounsel of recordz amicus curiaez fact witnesszexpert witnessz trier of factjuryz bench trialz trial courtzappellate courten bancz supreme courtz circuit courtdistrict courtzmagistrate judgezadministrative law judge arbitratormediatorzspecial masterreceivertrusteezguardian ad litemcuratorsecepaftcdojnlrbeeoccfpbosha prosecutorzpublic defenderzsolicitor generalzattorney general))justicezprocedural justicezdistributive justiceequityfairness due processsubstantive due processprocedural due processequal protectionlibertyautonomyprivacy efficiencyz kaldor-hickszpareto efficiency deterrence retributionrehabilitationincapacitation restitutionz good faithz bad faithzfiduciary duty duty of carezduty of loyaltyconfidentialityattorney-client privilegework product doctrinezconflict of interestzzealous advocacyzcandor to the tribunal jurisdictionpersonal jurisdictionsubject matter jurisdiction sovereignty legitimacyz rule of lawzseparation of powers federalism preemptionzdormant commerce clause)"zstrategic considerationzrisk assessmentzclient counselingzethical dilemmazcost-benefit analysiszprobabilistic outcomezworst-case scenarioleveragezbargaining positionzsettlement valuezlitigation riskexposurezprecedent strengthz circuit splitzemerging trendz novel issuezmatter of first impressionzpolicy argumentztextualist argumentzoriginalist argumentzpurposive interpretationlegislative intentzslippery slopezparade of horribleszfloodgates argumentzchilling effect overrulingzdistinguishing precedentzlimiting holdingzextending doctrinez case strategyzlitigation posturezforum selection choice of law)Bcontract agreementbreachdamages liability negligencetortrvenuer constitution amendmentz civil rightscriminalfelony misdemeanor arraignment indictmentverdictsentencejudgewitness testimonyevidencemotionbriefpleadingsubpoenasummonswritz habeas corpusattorneycounsellawyer privilegecustodyalimonydivorceprobatewilltrustestate corporationllc partnership shareholder fiduciarypatent trademark copyright employmentdiscrimination harassmentz real estateleasemortgagedeedeasement bankruptcycreditorlien collateral immigrationasylum deportationnaturalization?333333?g?g?ffffff?皙?r$)rMr r"rtermreturncp|j}|tDcgc]}|jc}vry|tDcgc]}|jc}vry|tDcgc]}|jc}vry|tDcgc]}|jc}vry|t Dcgc]}|jc}vry|t Dcgc]}|jc}vry|tDcgc]}|jc}vryycc}wcc}wcc}wcc}wcc}wcc}wcc}w) z;Get the tier level for a legal term (1-6, or 0 for general)rMr r"r rrr)lower LEGAL_TIER_1 LEGAL_TIER_2 LEGAL_TIER_3 LEGAL_TIER_4 LEGAL_TIER_5 LEGAL_TIER_6 LEGAL_GENERAL)r term_lowerts r= get_term_tierrsJ6Aaggi66 <8a 8 8 <8a 8 8 <8a 8 8 <8a 8 8 <8a 8 8 =9a 9 9 7888889s(DD$D D$.D)D.8D3cPt|}|dk\rtj|dSy)z7Get the boost weight for a legal term based on its tierr333333?)rLEGAL_TIER_WEIGHTSr9)rtiers r=get_term_weightr#s)  D qy!%%dC00 r?)zissue spottingzelements analysiszcounterargument anticipationzrebuttal strategyzevidentiary foundationzpreservation of errorzdeferential reviewharmless error plain errorzstructural errorwaiver forfeitureestoppellachesz clean handsz unclean handszparol evidence rulezstatute of limitationsstatute of fraudszbest evidence rulehearsayzhearsay exceptionzbusiness records exception)summary judgmentrmrdrerfRule 56zCelotex trilogyno genuine dispute material factzsummary adjudicationCelotexAnderson v. Liberty Lobby Matsushitar`) rrrrzduty of confidentialityzethics rule 1.6zclient confidencezprivileged communicationzcommon interest doctrinez joint defensezcrime-fraud exceptionzwaiver of privilegezattorney client)rnrorprqrrrzmandatory disclosureprotective order privilege logzdocument productionRule 26 spoliationESIz!electronically stored information) rrrzforum non conveniensminimum contactsspecific jurisdictiongeneral jurisdictionzlong-arm statutezdiversity jurisdictionzfederal questionzsupplemental jurisdictionremoval) rrrnoticezopportunity to be heardfundamental rightszMathews v. Eldridgerirg) offer acceptance considerationz mutual assentzmeeting of the mindszcontract formationbargainzpromissory estoppelzdetrimental reliancer*) rrzbreach of duty causationproximate causebut-for causationrforeseeabilityreasonable personzstandard of carezres ipsa loquitur)zfirst amendment free speechestablishment clause free exercisezfourth amendmentsearch and seizureprobable causewarrantzfifth amendmentzself-incriminationzdouble jeopardytakingszsixth amendmentzright to counselzconfrontation clausezfourteenth amendmentrr incorporation) r+ relevancezprobative valuezprejudicial effectzRule 403zcharacter evidencezprior bad actszRule 404zexpert testimonyDaubert foundationauthentication)ryr]r^r_zclearly erroneousr$r% preservationr oral argumentr{affirmreversevacate) summary_judgmentattorney_clientrnr due_processcontract_formationrconstitutionalr appellate)rrzpre-litigationzsettlement demandztolling agreementznotice of claimzadministrative exhaustion) rrrkzamended complaintrlzRule 8rcrrrzdefault judgmentzresponsive pleading) rorpzproduction requestzexpert disclosurer5zinitial disclosurezdiscovery disputezmotion to compelr3r4zsubpoena duces tecum) zmotion in liminer,zpretrial conferencerr-zpretrial orderz witness listz exhibit listzDaubert motion) rsrwrtrxrvruredirectzmotion for directed verdictzjudgment as a matter of lawzjury deliberationr)zpost-trial motionzmotion for new trial remittituradditurz judgment notwithstanding verdictJNOVzRule 59zRule 60)znotice of appealrrTrzzpetition for reviewzinterlocutory appealrr)zjudgment enforcement execution garnishmentz judgment liencontempt collectionzsatisfaction of judgmentzjudgment debtor) pre_filing pleadingsrn pre_trialtrial post_trialry enforcementtextc|ji}tjD]$\}}tfd|D}|dkDs |||<&|rt ||j Sy)z/Detect which litigation phase a text relates toc3,K|] }|vsd ywrMNrJ).0r text_lowers r= z*detect_litigation_phase..Ws>$4:+=A> r)keygeneral)rLITIGATION_PHASESitemssummaxr9)rl phase_scoresphasetermsscorerqs @r=detect_litigation_phaser~RskJL)//1( u>%>> 19"'L (<\%5%566 r???皙?333333?333333?)us_supreme_courtfederal_circuitfederal_district state_supremestate_appellate state_trialadministrative secondary)z"supreme court of the united statesz\d+\s+u\.?s\.?\s+\d+scotus)zcircuit court of appealsz\d+\s+f\.(?:2d|3d|4th)\s+\d+zcourt of appeals)rz\d+\s+f\.\s*suppzu\.s\. districtzsupreme court of \w+zcourt of last resort)zcourt of appealzappellate divisionzintermediate appellate)rrrrrc|j}tjD](\}}|D]}tj||s|ccS*y)z$Detect court level from text contentunknown)rCOURT_PATTERNSrwresearch)rlrqlevelpatternspatterns r=detect_court_levelrusPJ)//1x Gyy*-   r?) affirmedfollowedadoptedapprovedzcited with approval reaffirmedextendedz relied uponzconsistent with) reversed overruled disapproved criticizedlimitedrejectedzdeclined to follow abrogated superseded)cited discussed mentioned referencedzsee alsozcf.) distinguished questionedzcalled into doubtzbut seecontramodified clarifiednarrowedpositivenegativeneutralcautionc|j}ddddd}tjD]\}}|D]}||vs||xxdz cc< |S)z)Analyze how citations are treated in textrrrM)rCITATION_SIGNALSrw)rlrq treatment signal_typesignalssignals r=analyze_citation_treatmentrsgJA!JI 0 6 6 8, W ,F#+&!+& ,, r?) zChevron deferencezadministrative deferencezagency interpretationstep onestep twozambiguous statutezreasonable interpretationzChevron U.S.A. v. NRDCzAuer deferencezSkidmore deference) z Erie doctrinezsubstantive vs proceduralrzErie Railroad v. Tompkinszfederal common lawzoutcome determinativeztwin aims of EriezHanna v. Plumerzvertical choice of law) ripenessmootnessstandingzcase or controversyzinjury in factrBredressabilityzLujan v. Defenderszcapable of repetitionzpolitical questionzadvisory opinion) zqualified immunityzclearly established lawzgovernment immunityzreasonable officerzconstitutional violationzHarlow v. Fitzgerald sequencingzPearson v. Callahanzsovereign immunity) rzRule 23zclass certification commonality typicalityadequacy numerosity predominance superiorityzopt-outzsettlement classzcy presz class notice) z res judicataclaim preclusionzissue preclusionzcollateral estoppelfinal judgmentzsame cause of action same parties mutualityzoffensive collateral estoppelzdefensive collateral estoppel) rJrKwarrant requirementzexclusionary rulezfruit of the poisonous treezgood faith exceptionexigent circumstances plain viewzconsent searchz Terry stopzreasonable suspicion) rGz content-basedzcontent-neutralrgrhtime place mannerzprior restraintz public forumcommercial speechzsymbolic speechzcompelled speech)chevroneriejusticiabilityqualified_immunity class_action res_judicatafourth_amendmentfirst_amendmentqueryc|jg}tjD].\}}tfd|ddDs|j |0|S)z!Get doctrines relevant to a queryc3BK|]}|jvywNrrpr query_lowers r=rrz)get_relevant_doctrines..sAttzz|{*ANr")rLEGAL_DOCTRINESrwanyappend)rrelevantdoctriner|rs @r=get_relevant_doctrinesrsV++-KH*002&% AuRayA A OOH %& Or?))z\d+\s+U\.?S\.?\s+\d+r)z\d+\s+S\.?\s*Ct\.?\s+\d+r)z"\d+\s+L\.?\s*Ed\.?\s*(?:2d)?\s+\d+r)z\d+\s+F\.(?:2d|3d|4th)?\s+\d+r)z&\d+\s+F\.\s*Supp\.?(?:\s*2d|3d)?\s+\d+r)z!\d+\s+S\.?\s*E\.?(?:\s*2d)?\s+\d+r)z$\d+\s+S\.?\s*W\.?(?:\s*2d|3d)?\s+\d+r)z$\d+\s+N\.?\s*E\.?(?:\s*2d|3d)?\s+\d+r)z!\d+\s+N\.?\s*W\.?(?:\s*2d)?\s+\d+r)z\d+\s+P\.?(?:\s*2d|3d)?\s+\d+r)z\d+\s+A\.?(?:\s*2d|3d)?\s+\d+r)z\d+\s+So\.?(?:\s*2d|3d)?\s+\d+r)u*\d+\s+U\.?S\.?C\.?(?:\.?A\.?)?\s*§?\s*\d+r)u\d+\s+C\.?F\.?R\.?\s*§?\s*\d+r)z#Fed\.?\s*R\.?\s*Civ\.?\s*P\.?\s*\d+rule)zFed\.?\s*R\.?\s*Evid\.?\s*\d+rcg}tD]b\}}tj||tj}|D]3}|j |j ||j d5d|S)z0Extract and categorize legal citations from text)citationtypeposition)LEGAL_CITATION_PATTERNSrfinditer IGNORECASErgroupstart)rl citationsr citation_typematchesmatchs r=extract_citationsrspI"9++gtR]]; E   !KKM%!KKM   r?ct|}dddddd}|D]g}|ddk(r|dxxdz cc<|ddk(r|dxxdz cc</|dd k(r|d xxdz cc<E|dd k(r|d xxdz cc<[|d xxdz cc<i|S)z"Count citations by authority levelr)supremecircuitdistrictstateotherrrrrMrrrrrrr)r)rlrcountscits r=count_authoritative_citationsrs!$'Iqa!a PF ! v;, , 9  "  [- - 9  "  [. . : ! #  [- - 7Oq O 7Oq O ! Mr?cx|jg}tjD]Q\}}tfd|Dr|j |,|j ddvsA|j |St jD]Q\}}tfd|Dr|j |,|j ddvsA|j |St}g}|D]E}|j|vs|j|j|j|G|S)z(Expand query with related legal conceptsc3BK|]}|jvywrrrs r=rrz-expand_query_with_concepts.. =ttzz|{*=r_ c3BK|]}|jvywrrrs r=rrz-expand_query_with_concepts..rr) rLEGAL_CONCEPT_CLUSTERSrwrextendreplacersetaddr) rexpanded_terms cluster_namer|rseen unique_termsrrs @r=expand_query_with_conceptsr s++-KN5;;=) e =u= =  ! !% (  ! !#s +{ :  ! !% ( )+002)% =u= =  ! !% (   c3 '; 6  ! !% ( ) 5DL& ::xH HI &l3I>B 99;* $ v(=),7A AB r?chunk_metadata query_contextcztjj}|jd}|sy||z }|jddj }|jdg}d|vsd|vst d|Dr|d kry |d kry|d kry yt d|Dry|d kry|dkry|dkryy)z?Calculate weight based on temporal relevance of legal authorityyearr  document_typer=r(rrc3$K|]}d|v yw)rNrJ)rprs r=rrz,calculate_temporal_weight..s?]ST Q?]rrr*r rc3$K|]}|dv yw))r!r"r#NrJ)rpr@s r=rrz,calculate_temporal_weight..s G336 6 GrHr 2r r$)rnowrEr9rr)rBrC current_year chunk_year year_diffdoc_typer(s r=calculate_temporal_weightrRts<<>&&L##F+J z)I!!/26<<>H   fb )DH  8C?]X\?]<] q= ^ ^ G$ GG2~ R Rr?) ifthenwhenwherez provided thatunless thereforethushence accordingly consequently deductivezRule-based deductive reasoning)rr&r description) z on one handzon the other handbalanceweighzcompeting interestscountervailingzpolicy considerationszpublic interestzprivate rights? dialecticalzPolicy-based balancing analysis) similarlylike analogousdistinguishablezakin to comparablez in contrastunlikez parallel toffffff? comparativez#Analogical reasoning from precedent) zthree-part testz four-factorelementsprongsrrfirstsecondthirdz multi-factortotality of circumstancesg? frameworkzStructured doctrinal analysis) plain meaningzplain languagezordinary meaningzstatutory text unambiguouszliteral interpretationz express termsrtextualz Textual/statutory interpretation) historicallyz at common law traditional longstanding originallyz founding eraframerszoriginal understandingffffff? historicalzHistorical/originalist analysis)rule_synthesispolicy_balancing analogicaldoctrinal_frameworktextual_analysishistorical_analysisc |j}g}tjD]\}}d}g}|dD]}||vs|dz }|j||dkDs5t d|t |ddzz }|j|t |d|d|d|d |d d d |jd d|S)z)Identify legal reasoning patterns in textrrrMr r r r&rr^Nr)schema confidencer&rr^matched_patternsc |dS)NrrJxs r=z+detect_argument_structure..s ,r?TrtrV)rARGUMENT_SCHEMASrwrminlenroundsort) rlrqdetected schema_namerrrrrs r=detect_argument_structurersJH/557 Vj) 1G*$1  ''0 1 Q;S'S 1C-Ds-J"KLJ OO%#J2 *v%m4$4Ra$8  * MM/M> Or?)rr=zprocedural fairnessrznotice and hearing) police powerzstate interestri)rgrhzrational basis review)criminal procedurezadministrative lawz family lawr)related contrastingtests applications)rzsuspect classificationr=zlegitimate government interestr)rgrhri)z voting rights educationzemployment discrimination)rGrIrHz free presszcompelling interestzcontent-neutral regulation)rgrhr)zpolitical speechrzreligious liberty)rJrKr)rconsentrz!reasonable expectation of privacyrq)rzadministrative searcheszborder searches)rszparol evidencezcourse of dealing)zcontra proferentemzejusdem generisznoscitur a sociis)rlcontextz trade usagezcourse of performance)rr rescission reformation)rcanonsr'remedies)rrrBrrC)zcontributory negligencezcomparative faultzassumption of risk)rFrErD)zmedical malpracticezproducts liabilityzpremises liability)rr/rerf)r.zview evidence favorablyzdraw inferences)r0r1r2)r-zstatement of factsresponsereply)r standards key_cases procedures)r8zpurposeful availmentz fair play)r:r9r)zInternational Shoezstream of commercez effects test)zinternet jurisdictionzcontract disputesz tort claims)rtypesrr)rZequal_protectionrrcontract_interpretationrrXpersonal_jurisdictionc|j}g}tjD]\}}|jdd}||vrc|j |j dg|j |j dg|j |j dg|j dgD])}||vs|j |j dgt t|S)z.Get related legal concepts for query expansionrrrrr)rLEGAL_CROSS_REFERENCESrwrrr9listr)rrrrrefsdoctrine_cleanrel_terms r=get_related_conceptsrs++-KG0668 $!))#s3 [ ( NN488Ir2 3 NN488GR0 1 NN488K4 5B/ H;&txx 267   G r?c|j}tjD]I}|jdd}||vr|cSt|}|j dgddD] }||vs|ccSKy)z,Extract the primary legal doctrine from textrrrNr")rrkeysrr9)rlrqrrrrs r=extract_primary_doctriner5sJ+//1  !))#s3 Z 'O&h/HHY+BQ/ Dz!    r?r) primary_doctrinesupporting_doctrinesargument_structuresprecedential_weighttemporal_weightauthority_leveljurisdictional_relevancecitation_countrEchunkc@|jdd}|jdi}t|}|r|dd|d<|dd|d <t|||d <t||d <t ||d <t |}t ||d <td|D|d<|S)z&Add advanced legal metadata to a chunkcontentr=r:Nr"rrrprimary_argument_typerrrrc3,K|] }|ddk(yw)rrNrJrpcs r=rrz)enhance_chunk_metadata..ls.,-& ''.shas_supreme_court_citation)r9rrArRrrrr)rrr:arg_structuresrs r=enhance_chunk_metadatarUsii 2&GyyR(H/w7N'5bq'9#$)7):6)B%&$;7H#ME   9BE !9 AE "'*I!)nE *-.1:.+E &' Lr?TF extract_text) PdfReader)SentenceTransformerc xeZdZdZdZdZd dededede efdZ dededede efd Z dededede efd Z y ) LLMServicezGLLM service using local DeepSeek for enrichment, Grok for chat fallbackctjjdd|_tjjdd|_d|_tjjdxs tjjddj j d j d |_d |_d |_ |j|j s|js td yyy)NDEEPSEEK_LLM_URLz)http://127.0.0.1:5004/v1/chat/completionsDEEPSEEK_MODELz'deepseek-ai/DeepSeek-R1-Distill-Qwen-7BT GROK_API_KEY XAI_API_KEYr="'z$https://api.x.ai/v1/chat/completionszgrok-4-fast-reasoningzK[RAG] WARNING: Neither DeepSeek nor Grok available. LLM features will fail.) r7r8r9 deepseek_urldeepseek_modeldeepseek_availablestrip grok_api_keygrok_url grok_model_check_deepseekprintr;s r=__init__zLLMService.__init__sJJNN+=?jk jjnn-=?hi"& ZZ^^N;`rzz~~m]_?`ggioopstzz{~> 1 &&t/@/@ _ `0A&r?cbtsd|_y |jjdd}t j |d}|j dk(r td|jd |_ytd d|_y#t$r }td |d d|_Yd}~yd}~wwxYw) z$Check if local DeepSeek is availableFNz/v1/chat/completionsz /v1/modelsr)timeoutz [RAG] Local DeepSeek available: Tz;[RAG] Local DeepSeek not responding, will use Grok fallbackz#[RAG] Local DeepSeek check failed: z, will use Grok fallback) HAS_REQUESTSrrrrequestsr9 status_coderr Exception)r<base_urlres r=rzLLMService._check_deepseeks&+D #  ,((001GVH||Ha8H##s*89L9L8MNO*.'SU*/' , 7s:RS T&+D # # ,sA!B2B B.B))B.prompt max_tokens temperaturercn|jr|j|||}|r|S|j|||S)z9Generate text using DeepSeek (primary) or Grok (fallback))r_call_deepseek _call_grok)r<rrrresults r=generatezLLMService.generates>  " "(([IF vz;??r?c tsy tj|jddi|jd|dg||ddd }|j d k(ro|j }d |vrzt|d d kDri|d d d d}d|vr6d|vr2d dl}|jdd||jj}|Std|j d|_ y#t$r}td|d|_ Yd}~yd}~wwxYw)zCall local DeepSeek LLMN Content-Typeapplication/jsonuserrolerFmodelmessagesrrstreamxheadersjsonrrchoicesrmessagerzzz.*?r=flagsz[RAG] DeepSeek error: z[RAG] DeepSeek exception: )rrpostrrrrrrsubDOTALLrrrr) r<rrrrr2rrrs r=rzLLMService._call_deepseeks1 ,}}!!');<!00*0V!D E",#.#  H##s*}}$T)_)=)A"9oa0;IFG G+ g0E!"("&&)>GSUS\S\"]"c"c"e"N.x/C/C.DEF*/'   , .qc2 3&+D # #  ,sB6C C D)DDc :tr |js tdy td|jdt j |j d|jdd|jd|d g||d d d }|jdk(r3|j}d|vrUt|ddkDrD|ddddStd|jd|jr|jddndy#t$r}td|Yd}~yd}~wwxYw)zCall Grok API (fallback)z*[RAG] Grok API not configured (no API key)Nz[RAG] Calling Grok fallback (z)...zBearer r) AuthorizationrrrFr-rrrrrrz[RAG] Grok API error: z - rNz(empty)z[RAG] Grok API exception: ) rrrrrrrrrrrlr)r<rrrrr2rs r=rzLLMService._call_groks:4#4#4 > ? 4 1$//1B$G H}} '.t/@/@.A%B$6 "__*0V!D E",#.# H ##s*}}$T)_)=)A ?1-i8CC.x/C/C.DC_g_l_l VZWZH[r{G|}~ 4 .qc2 3 3 4sB!C96C99 DDDN)rKr$) r@rArBrCrrrIrGrHr rrrrJr?r=rrsQa",( @s @ @ @X`adXe @$S$c$$RZ[^R_$L!!#!E!hWZm!r?rc eZdZdZdefdZdedefdZdedefdZ ded edefd Z dede efd Z dede e eeeffd Zdded ede eeeffdZdede e dede efdZdedede efdZdededefdZddedede efdZy) PDFProcessorz4Advanced PDF processor with structure-aware chunkingconfigc$||_gd|_y)N)ucopyright\s+©?\s*\d{4}zall rights reservedzpage\s+\d+\s+of\s+\d+z^\s*$z^\d+$z [a-f0-9]{32,}z ^[\s\W]*$)rboilerplate_patternsr<rs r=rzPDFProcessor.__init__-s % !r? file_pathrc|jjdk(rhtr |j|St r tt|Str. t|}djd|jDSy|jjd k(r) t|d d 5}|jcdddSy#t$r}t d|Yd}~d}~wwxYw#t$r}t d|Yd}~d}~wwxYw#t$r}t d|Yd}~yd}~wwxYw#1swYyxYw#t$r8t|d d 5}|jcdddcYS#1swYYyxYwwxYw)z(Extract text using best available method.pdfzPyMuPDF failed: Nzpdfminer failed:  c3DK|]}|jxsdyw)r=Nr)rppages r=rrz,PDFProcessor.extract_text..Ns!$X4T%6%6%8%>B%>$Xs zPyPDF2 failed: r=.txtrutf-8encodingzlatin-1)suffixr HAS_PYMUPDF_extract_with_pymupdfrr HAS_PDFMINERpdfminer_extractrI HAS_PYPDF2rjoinpagesopenreadUnicodeDecodeError)r<r rreaderfs r=rzPDFProcessor.extract_text9s    ! ! #v -255i@@ 3+C N;; 1&y1F99$X6<<$XXX    # # % / $)S7;$q668$$ 9!2,QC0112!3-aS1223!1OA3/001$  & $)S9=$668$$$  $sC C1,D$E2D9 E C.C))C.1 D:D  D D6D11D69E>EEFE8, F8F =FFc4tj|}g} |D]E}|jdtjtjz}|j |G |j dj|S#|j wxYw)z&High-fidelity extraction using PyMuPDFrlrr)fitzrget_textTEXT_DEHYPHENATETEXT_PRESERVE_WHITESPACErcloser)r<r docrrrls r=rz"PDFProcessor._extract_with_pymupdf^sii "  #}}V43H3H4KhKh3h}i T" # IIKyy IIKs A BBrlfilenamec|j|j}gd}tfd|Ddk\rygd}tfd|Ddk\rygd}tfd |Ddk\ry y ) z)Detect document type for optimal chunking)bettingoddsspread moneylineparlay sportsbook handicappingc3,K|] }|vsd ywrorJrpindrqs r=rrz4PDFProcessor.detect_document_type..qsCS 1BqCrsr rR) specificationrOapiprotocol algorithmc3,K|] }|vsd ywrorJr5s r=rrz4PDFProcessor.detect_document_type..vsASsj/@qArsrO)abstract methodologyresults conclusion referencesc3,K|] }|vsd ywrorJr5s r=rrz4PDFProcessor.detect_document_type..{sES3*3DqErsrPrS)rrx)r<rlr+filename_lowersports_indicatorstech_indicatorsresearch_indicatorsrqs @r=detect_document_typez!PDFProcessor.detect_document_typejs{ZZ\ !)o C-C Cq HY AOA AQ Fa E/E E Jr?c|sgSd}tj||}|Dcgc]/}t|jdkDs |j1c}Scc}w)zSplit text into sentencesz?(?<=[.!?])\s+(?=[A-Z])|(?<=[.!?])\s*$|(?<=[.!?"\'])\s+(?=[A-Z])r*)rsplitrr)r<rlsentence_endings sentencesss r=split_into_sentencesz!PDFProcessor.split_into_sentencessOI^HH-t4 #,DaAGGI0C DDDs !AAc  g}|jd}t|D]\}}|j}|rt|dkr'|j r1t|dk\r#t|dkr|j ||dfht jd|}|std|jdd D}|jd} t| dkDs|j | |t|df|S) zDetect headings in documentrr"r PrMz^\s*(\d+)(?:\.(\d+))?\s+(.+)$c3&K|] }|sd ywrorJ)rpgs r=rrz/PDFProcessor.detect_headings..sH!aAHsNr ) rH enumeraterrisupperrrrrxgroupsrr) r<rlheadingslinesiline line_strippednumbered_matchdepth heading_texts r=detect_headingszPDFProcessor.detect_headingss 4  ' FGAt JJLM C $6$:$$&3}+=+Bs=GY\^G^1 56 XX&F VNH~'<'<'>r'BHH-33A6 |$q(OO\1c%m$DE! F$r?rQc&tj|td}|d}|d}|d}|d}|d}|rt|j|krgS|j |} | r|j || |} | r| S|j ||S)zStructure-aware chunkingrSrrrr!r#)DOCUMENT_TYPE_CONFIGSr9rrr\_chunk_by_sections_sentence_based_chunking) r<rlrQrr chunk_min chunk_maxoverlapr#rTchunkss r= chunk_textzPDFProcessor.chunk_texts&**85J95UVL) +, +, +,/ s4::<(94I''- ,,T8VDF ,,T6::r?rTcg}|jd}t|D]\}\}}} |dzt|kr ||dzdn t|} dj||dz| j } | sYt| |dk\sk|j | |} | D] } || d<| | d<|j | |S)zChunk text by detected sectionsrrMrsection section_level)rHrQrrrr`r)r<rlrTrrdrUrVr[ heading_posrend_possection_contentsection_chunksrs r=r_zPDFProcessor._chunk_by_sectionss 4 6?x5H . 1A1 k5,-ECM,Ahq1uoa(s5zG"iikAog(FGMMOO3#76BR;S#S!%!>!>PV!W+3E'3E)$-2E/*3 n- . r?c|j|}t||dkrgSg}g}d}|D]k}t|dz}||z|dkDr|rdj|j} t| |dk\r'|j | t|t| dt |dt|} | dkDr|| d ng}t d |D}|j |||z }||d k\st||dk\sdj|j} t| |dk\r'|j | t|t| dt |dt|} | dkDr|| d ng}t d |D}n|rWdj|j} t| |dk\r'|j | t|t| d|S) z$Sentence-based chunking with overlapr#rrMrrr)rsentence_count char_countr!Nc38K|]}t|dzywrorrprKs r=rrz8PDFProcessor._sentence_based_chunking..$GASVaZ$Grc38K|]}t|dzywrorqrrs r=rrz8PDFProcessor._sentence_based_chunking..rsrt)rLrrrrrrx) r<rlrrJrd current_chunkcurrent_lengthrsentence_lengthre overlap_counts r=r`z%PDFProcessor._sentence_based_chunkings--d3 y>F?3 3I ! HH!(ma/O/&9I2JJ} XXm4::< z?f-=&>>MM#-*-m*<&)*o#!$F+=$>M@R S BORSBS }no >Y[ !$$G$G!G   * o -N !55#m:LPVWfPg:g XXm4::< z?f-=&>>MM#-*-m*<&)*o# !$F+=$>M@R S BORSBS }no >Y[ !$$G$G!G? HD -0668J:&)9":: )&)-&8"%j/  r?rc|jdd}t||jddkry|j}|jD])}t j ||tj s)yt jd|}t|dkry|r tt|t|z nd}|d kryy ) zValidate chunk qualityrr=rrF\b\w+\brrr T) r9rrr rrrfindallr)r<rrr content_lowerrwords unique_ratios r=is_valid_chunkzPDFProcessor.is_valid_chunks))Ir* w<&**%5s; ;  00 Gyy-?   :}5 u:>8=s3u:U3! # r?N source_namec| |j}|j|}|rt|jdkrgS|j ||}t j |t d}|j||}g}t|D]F\}} |j| |s|| d<|| d<t|| d<|| d<|j| H|S)z.Process a file and return chunks with metadatadrSr<chunk_id total_chunksrF) namerrrrFr^r9rerQrr) r<r rrlrQr raw_chunksrdrVrs r= process_filezPDFProcessor.process_files  #..K  +s4::<(3.I,,T;?&**85J95UV__T84 !*- %HAu""5&1"-h$%j!(+Jn%)1o& e$  % r?)rSr)r@rArBrCrrrrIrrrFr rLr rGr\r r redictr_r`boolrrrJr?r=rr*s:>  y  #d#s#J  t    ,EEcECDsC}1E,F2;s;c;$tCQTH~BV;.sd5k4TXY]T^&5S5$54:5nD$40dT r?rc eZdZdZdefdZdZdedejfdZ de edejfd Z de fd Zd Zd Zd edefdZdZde efdZdede efdZddedede efdZde edeeeffdZdede ededefdZy) VectorStorez0FAISS-based vector store with BM25 hybrid searchrc ||_g|_g|_d|_d|_g|_g|_d|_|jdz |_ |jdz |_ |jdz |_ |jy)Nrzfaiss_index.indexzdocuments_metadata.jsonzembeddings.npy) r documents embeddingsindexrsearchable_texts doc_lengthsavg_doc_lengthr index_file metadata_fileembeddings_file _init_modelr s r=rzVectorStore.__init__@s   "!**-@@#--0II%//2BB r?ctr# td|_d|_t d|jd|_t dyy#t $r}t d|d|_Yd}~Cd}~wwxYw)zInitialize embedding modelzall-MiniLM-L6-v2iz&[RAG] Loaded SentenceTransformer modelz"[RAG] SentenceTransformer failed: NzT[RAG] Using hash-based embeddings (install sentence-transformers for better results))HAS_SENTENCE_TRANSFORMERSrr embedding_dimrr)r<rs r=rzVectorStore._init_modelRsu $ "01CD %(">@ :: !$D  h i   ":1#>?!  "s"A A2A--A2rlrc|jr|jj|dStj|jj }t j |t jjt j}|t jj|dzz S)zGet embedding for textT)normalize_embeddings)dtypeg& .>) rencodehashlibsha384digestnp frombufferuint8astypefloat32linalgnorm)r<rlharrs r=_get_embeddingzVectorStore._get_embeddingas ::::$$T$E Et{{}-446A--299"**EC"))..-45 5r?textsc|jr|jj|ddStj|Dcgc]}|j |c}Scc}w)z!Get embeddings for multiple textsT)rshow_progress_bar)rrrarrayr)r<rrs r=_get_embeddings_batchz!VectorStore._get_embeddings_batchksN ::::$$UY]$^ ^88UCT003CD DCsAc|jjsy t|jdd5}tj||_dddt rH|jjr.tjt|j|_ nG|jjr-tjt|j|_|jt!dt#|j dy#1swYxYw#t$$r}t!d |Yd}~yd}~wwxYw) z!Load existing index and documentsFrrrNz [RAG] Loaded documentsTz[RAG] Failed to load index: )rexistsrrloadr HAS_FAISSrfaiss read_indexrIrrrr_prepare_searchable_textsrrrr<r#rs r=rzVectorStore.loadrs!!((* d((#@ .A!%1 .T__335"--c$//.BC %%,,."$''#d.B.B*C"D  * * , M#dnn"5!6jA B . .  04 5 s/D,D CD, D)%D,, E 5EE cz t|jdd5}tj|j|dddddt r@|j 4tj|j t|jn^t|jdkDrFtjt|jtj |jt#d t|jd y#1swYxYw#t$$r}t#d |Yd}~yd}~wwxYw) zSave index and documentswrrFr  ensure_asciiindentNrz [RAG] Saved rz[RAG] Failed to save index: )rrrdumprrrr write_indexrIrrrrsaverrrrrs r=rzVectorStore.saves 6d((#@ KA $..!%J KTZZ3!!$**c$//.BCT__%)D001288DOO3LM LT^^!4 5Z@ A K K 6 04 5 5 6s.D$D CD DD D:"D55D:cg|_g|_|jD]n}|j|}|jj |j |jj t |jp|jr1t|jt |jz |_ yd|_ y)zBuild searchable text cacherN) rrr_build_searchable_textrrrrHrxr)r<r*rls r=rz%VectorStore._prepare_searchable_textss ">> 7C..s3D  ! ! ( ( 6    # #C $5 6 7 PTO_O_c$"2"23c$:J:J6KKefr?r*c g}|jdd}|rXtjdd|tj}|j |j ddj dd|jdr|j |d|jd r)|j t |jd |jd rH |j dj|jd gDcgc] }t |c}|jd rH |j dj|jd gDcgc] }t |c}|j |jd ddj|Scc}w#t$rYwxYwcc}w#t$rYVwxYw) z#Build searchable text from documentr<r=z \.(pdf|txt)$rrr-rgsummary key_pointsthemesr) r9rrrrrrIrr)r<r*partsr< clean_sourceprs r=rz"VectorStore._build_searchable_textsx2& 66/2vR]]SL LL--c37??SI J 779  LLY ( 779  LLSWWY/0 1 77<   SXXsww|R7P&Q!s1v&QRS 778   SXXswwx7L&M!s1v&MNO  SWWY+,xx'R  'N  sH *F7 F2  F79*G #G 5 G 2F77 GGG GGc~ |jj|jjdz}t|dd5}t j |j |dddddtj||jy#1swY*xYw#t$r}td |Yd}~yd}~wwxYw) zOPersist documents metadata safely without rewriting the FAISS index/embeddings..tmprrrFr rNz,[RAG] Failed to save metadata (checkpoint): ) r with_suffixrrrrrr7rrr)r<tmp_pathr#rs r=save_metadata_onlyzVectorStore.save_metadata_onlys F))55d6H6H6O6ORX6XYHhg6 K! $..!%J K JJx!3!3 4 K K F @D E E Fs0AB$B&(BBB B<$B77B<rdc2|sy|Dcgc]}|d }}|j|}tr|j$tj|j |_t j|jd}tj||jj|njt|jt jr?t|jdkDr't j|j|g|_ n||_ |j j#||D]n}|j%|}|j&j)|j+|j,j)t|j/p|j,r0t1|j,t|j,z |_t5dt|dt|j ycc}w)zAdd documents to the storeNrrrz [RAG] Added z documents. Total: )rrrr IndexFlatIPrrrr normalize_L2r isinstancerndarrayrvstackrrrrrrrrHrxrr)r<rdrcontentsnew_embeddings embeddings_nprrls r= add_documentszVectorStore.add_documentss *01QAiL1133H= zz!"..t/A/AB HH^4;;IFM   } - JJNN= )$//2::63t;ORS;S"$))T__n,M"N"0 f% 7E..u5D  ! ! ( ( 6    # #C $5 6 7   "%d&6&6"7#d>N>N:O"OD  S[M)  ? # **4+=+=>DJ2;t*j@rt|j>dkDrvt+jB|j>|}t5|d zt|j}t+jD|| dddd }|Dcgc] }|||f }}ng}g}|D]\}}|t|jk\r |j|}|t|jFkr|jF|nd:|jId:}d}g}g}|D]f}|dj7:s|jIdr|j|dn|j|d||jIddz }ht5d|}|jK||| }|jLjNtQ|z|jLjR|zz|jLjT|zz} t |}!d}"|dk7r|!|k(rd}"tW|}#tXjI|#d}$d}%|D]2}&tZjI|&g}'t]:fd|'Ds.|%dz}%4t_|}(d})|(ddkDr|)dt5|(ddzz })|(ddkDr|)dt5|(ddzz })ta|}*d}+|*d |*d!kDrd"}+n|*d!|*d dzkDrd#}+tc||},te|}-tg|}.d}/|.r|.d}0|0d$dkDr|0d}/ti|}1d}2|1dd%D](}3|3j:jvs$|2dz }2*t5d&|2}2| |"z|$z|%z|)z|+z|,z|-z|/z|2z}4|j||jkD5cic]\}}5|dk7s ||5c}5}|4| tQ|||||tm|"dtm|$dtm|%dtm|)dtm|+dtm|,dtm|-dtm|/dtm|2dd' |!|#|.r|.dndto|d(|jqd)d*g}6i}7|D]{}8|8d+jId,d-}9|7jI|9d|jLjrksB|6j|8|7jI|9dd.z|7|9<t|6k\sz|6S|6Scc}wcc}5}w)/z?Enhanced hybrid vector + keyword search with legal intelligencer{r z\br)rris_legalr"r&TNr )rrrr"r&expandedrr rr=rrrrr&r r rur{c3^K|]$}|jjv&ywrr)rprsearchable_texts r=rrz%VectorStore.search..s%Z4tzz|'<'<'>>Zs*-gQ?rg{Gz?r"r皙?rrg?g?rrrb) r{rrrr precedentialtemporalargument cross_ref)rr:r} base_score vector_score keyword_score bm25_score matched_termsexpanded_matchesboostsr{ court_levelargument_structurerc |dS)Nr}rJrs r=rz$VectorStore.search..s ajr?rr:r<rrM):rrrr|r~rr rrrrcompileescaperr#r LEGAL_TERMS _compute_idfrrrrrrrrrrrziprrrdotargsortrr9 _compute_bm25rr%rHr&r(rJURISDICTION_HIERARCHYrrrrrArRrrrwrrrr));r<rrr query_words query_phaserelevant_doctrinesr keyword_patterns added_termsword term_tierr idf_tablequery_embedding oversamplescoresindicesvector_results similarities top_indicesrV candidatesridxr* doc_contentrrrrrcombined doc_phase phase_boostrjurisdiction_boostdoctrine_boostrdoctrine_termscitation_countscitation_boostrtreatment_boostprecedential_boosttemporal_boostrargument_boostbest_argrelated_conceptscross_ref_boostconcept final_scorevr> source_countsrr<rs; @r=rzVectorStore.search s/~~Ikkm jj[9 .e4 4E:4E:e  &D4y1}[!8)$/  '' !zz%"))D/*AE*I2==Y )Q%-d3 )% & &D{"t;'>)$/  '' !zz%"))D/*AE*I2==Y $%-d3 )% &#3B' .Dzz|;.)$/  '' !zz%"))D/*AE*I2==Y $%-d3c9 $ ) - .%%&67 --e4 / hh'89@@KO    /QUC$78J"jj//LOFG!#fQi"<=N$//2::63t;ORS;S!vvdooG  QDNN(;<  jj6 {|DTrTJ @K!L1<?A"6!L!L!# !/y  L#c$..))..%CZZ"d*N + )BB"&"9c!J 7s;N7{CN N)!,L)C/%-h%7N 4E: !O+BQ/ ,==?o&;&;&==#t+O ,"$8O$k14FFW()+:;=OP()+9:.s#V4AiL%alsobeeneachevenfromhaveintojustremoremostonlysomethanthatthemtheythisverywerewhatrUrwithyouraboutbeingcouldrtheirthererVwhichwouldshouldbecausethroughr rrMc |dSNrMrJrs r=rz3EnrichmentService._extract_themes..s 1r?TrNr rrr")rr|rrr9sortedrwrrrrrrr)r<rlr\r~ stop_words word_countsr sorted_wordscountrrqlegal_themes_found tier_termstier_numrs r=r[z!EnrichmentService._extract_themess . =I   AD:%#d)q.$/OOD!$ Key Points: - - - Themes: theme1, theme2, theme3 Begin the card now:rr rrz[RAG] LLM enhancement error: )rLr_parse_llm_responserr)r<rrrrrs r=rRz#EnrichmentService._enhance_with_llms # 9  $ 7xx((CS(QH//99  7 1!5 6 6 7s0A A% A  A%c |syi}tjd|tj}|r%|jdj dd|d<g}tjd|tj}|r|jdj j d}|D]\}|j }|j dr|ddj }|s:t|d kDsI|j|dd ^|r|dd |d <tjd |}|rT|jdj d D cgc]} | j } } | D cgc]} | s|  c} dd |d<|r|SdScc} wcc} w)z%Parse LLM output into structured cardNz)Summary:\s*(.+?)(?=Key Points:|Themes:|$)rMrrz Key Points:\s*(.*?)(?=Themes:|$)rrrrZrzThemes:\s*(.+),r) rrrrrrH startswithrr) r<rlr summary_matchrkp_matchrUrW themes_matchrrs r=rz%EnrichmentService._parse_llm_responses| "NPTVXV_V_`  - 3 3A 6 < < >t DF9  99@$ R NN1%++-33D9E 2zz|??3'8>>+DCIM%%d4Cj1  2 #-bq>F< yy!2D9 )5););A)>)D)DS)IJAaggiJFJ+17aQ7;F8 v)T)K7sF+F 3F originalrUcd|jddd|jdddd|jd d g}|jd r/|jd |d D]}|jd ||jdr&|jddj|d|jd|dj|S)z$Compose enriched content for storagez [Source: r<Unknownz | Chunk r?]z Summary: rr=rz Key Points:z- rzThemes: z, z Original Content: r)r9rr)r<rrUrUpoints r=_compose_enriched_contentz+EnrichmentService._compose_enriched_contents956iUX@Y?ZZ[ \B/0 1  88L ! LL 'l+ + r%\* + 88H  LL8DIId8n$=#>? @ ,XJ78yyr?Nrdc g}t|}t|D]c\}}|j|}|j||r ||dz||dzdzdk(sBt d|dzd|d|j de|S)z(Enrich all chunks with progress trackingrMr*rz[RAG] Enriched /z chunks (LLM used: ))rrQrXrrrN)r<rdprogress_callbackrPtotalrVrrWs r=enrich_all_chunksz#EnrichmentService.enrich_all_chunks sF !&) cHAu!..u5N OON + !!a%/A|q Awaw6I$J^J^I__`ab cr?)rr)r@rArBrCrrr rXrIrQrGr r[r rRrrrrJr?r=rIrIIs0!y! $4< C   ./#C/#S/#c/#b(4.: * * *D # T c &T tTXzr?rI templates)template_folder)case_bpz,[RAG] Case management API loaded at /cases/*z%[RAG] Case management not available: idler=ENRICHMENT_CHECKPOINT_EVERYr*) statusprogressrllm_used updated_at started_atr< total_target done_targetcheckpoint_everyzenrichment_status.jsoncN tjtjdz}t|dd5}t j t |dddddtj|ty#1swY$xYw#t$r}td |Yd}~yd}~wwxYw) z4Persist enrichment status for resume after restarts.rrrrFr rNz+[RAG] Failed to persist enrichment status: ) ENRICHMENT_STATUS_FILErrrrrenrichment_statusr7rrr)rr#rs r=_save_enrichment_status_to_diskrGsA)556L6S6SV\6\] (C' 2 Ja II'q I J 834 J J A ;A3?@@As.4BA7"B7B<B B$ BB$c P tjrUttdd5}tj|}dddt t rtj|tjddtjdd tjd d tjd d tjd dtjddtjddtjdd tjdd tjdttjjddtjddk(rodtd<tjd xsdtd <t!j"t$j&j)dztd <t+yy#1swYxYw#t$r}td|Yd}~d}~wwxYw)zRRestore enrichment status on startup (and mark stale running jobs as interrupted).rrrNz([RAG] Failed to load enrichment status: rrrrrr=rrrr<rrrrr*running interrupted4Previous enrichment was interrupted. You can resume.Z)rrrrrrrrrSrr setdefaultrGr7r8r9rrMrutc isoformatr)r#r2rs r=!_load_enrichment_status_from_diskrRs> ! ( ( *,cGD $yy| $$%!((.   62  Q/  B/  Q/  t4  t4  40  3  2  !3SHegi9j5klX&)3&3(#  ! !) , FE )$+3,,x||*D*N*N*PSV*V,'')4) $ $ > 8<==>s-&HG6-H6H;H H% H  H% FLASK_API_KEYzeventheodds-flask-api-key-2025c.tfd}|S)Nctjjdd}|tk7rt ddidfS|i|S)Nz X-API-Keyr=errorzInvalid API keyi)rrr9rr)argskwargsapi_keyr#s r= decoratedz"require_api_key..decoratedxsG//%%k26 m #G%678#= =$!&!!r?r)r#rs` r=require_api_keyrws! 1X"" r?z9[RAG] Initializing Advanced Legal Document RAG Service...job_idrrrc ~t5|||tjd|t|<dddy#1swYyxYw)zUpdate upload progress)rrrupdatedN) progress_locktimeupload_progress)rrrrextras r=update_progressrsB   yy{ #  #    s#3<r r+c  t|dddtj|}|rt|j dkr#t|dddt j|}|rt|j dkrt|ddd y t|dd d t|d tj ||}tj||}|st|dddy t|D]"\}}||d<||d<t||d<||d<$t|dddt|dtj||t|dddtjt|dddt|dt||ttjy #t$rF}td|dd l} | j!t|dddt#|Yd }~y d }~wwxYw)z#Process uploaded file in background processingr*z Extracting text from document...rrIz%Text extraction failed, trying OCR...failedrz$Failed to extract text from documentN(z Extracted z characters. Chunking...z'No valid chunks extracted from documentr<rrrF<zCreated z! chunks. Generating embeddings...ZSaving index... completedzSuccessfully processed! Added z chunks.) chunks_addedrFtotal_documentsz[RAG] Error processing file: Error: )r pdf_processorrrr ocr_servicerFrerQ vector_storerrrrr traceback print_excrI) rr r+rlrQrdrVrrrs r=process_file_backgroundrs6A b2TU)))4s4::<(3. FL"6] ^++I6Ds4::<(3. FHa1W X  bJs4ykIa2bc!55dHE))$9 FHa1Z [ "&) .HAu&E(O !E* $'KE. !%-E/ "  .  bHS[MIj2kl ##Hf5 b2CD K,S[M BV" 6 67   A -aS12!ws1vh-?@@ As&BF.AF.+CF.. G=7 **r?z/admincy)z@Serve upload interface with chunk viewer and enrichment controlsuS Legal Document RAG - Document Upload

🏆 Legal Document RAG

Advanced Document Processing with FAISS + BM25 + DeepSeek LLM Enrichment

📊 Status
📤 Upload
📄 View Chunks
✨ Enrichment

📊 System Status

-
Total Chunks
-
Enriched
-
Raw
-
Files
-
Search
rJrJr?r=adminrs| r?boostcf|dk\rddddS|dk\rddd dS|d k\rd d d dS|dk\rddddSddddS)z=Calculate precedential strength confidence (0-100) with labelr_z Gold Standardz&Supreme Court or Restatement authorityr}labelr^rrNzHighly PersuasivezCircuit Court or major treatiserA Persuasivez'District Court or state court authorityr rLStandardzGeneral legal authorityrK DecliningzMay be overruled or supersededrJrs r=!calculate_precedential_confidencer] se |oFnoo #&9Jkll #lClmm #jAZ[[kBbccr?cf|dk\rddddS|dk\rddd dS|d k\rd d d dS|dk\rddddSddddS)z-Calculate jurisdictional relevance confidencerrrzBinding precedent nationwiderrrNzFederal CircuitzBinding in circuitrFz State SupremezBinding in staterrz State AppealszPersuasive in staterLGeneralzGeneral jurisdictionrJrs r=!calculate_jurisdiction_confidencerk se |oFdee #&7H\]] #oFXYY #oF[\\i@VWWr?cP|dk\rddddS|dk\rddd dS|d k\rd d d dSddddS)z/Calculate argument structure quality confidencerjrzStrong Structurez'Clear analogical or doctrinal frameworkrrKzGood StructurezWell-organized legal argumentr{rModeratezStandard argument structurerBasiczSimple presentationrJrs r=calculate_argument_confidencery sQ |&8Irss #&6Gfgg $jA^__g>STTr?rc |jdi}|jdddf|jdddf|jdddf|jd ddf|jd dd f|jd dd f|jd ddfd}d}|jD]/\}\}}tdtd|dz dz dz}|||zz }1t |}|dk\rd} d} n"|dk\rd} d} n|dk\rd} d} n|dk\rd} d} nd } d!} || | |jD cic]\} } | t | dd"c} } d#Scc} } w)$z6Calculate overall confidence score for a search resultrrr r'rg?rg333333?rrg?rrr)rrrrrrrrrr4g?Uz Very Highz!Highly authoritative and relevantrHighzStrong legal authority7rz+Relevant but verify with additional sourcesrLowz#Limited authority, use with cautionzVery Lowz%Weak authority, requires verificationr )r}rr^factors)r9rwrryrGr) rrr weighted_sumfactorrr& normalizedr}rr^rr$s r=calculate_overall_confidencer s ZZ" %F NC8$?NC8$?ZZ C0$7ZZ C0$7ZZ C0$7jjc2D9ZZ C0$7GL#*==?,c!eckS%83%>?@  V++ ,  E {9 ". "C "; = "29--/B$!QAuQqT1~%B  Cs,E z/healthGET)methodscjtdttjtt t dS)z Health checkhealthy)rr has_faisshas_sentence_transformers has_pymupdf)rrrrrrrrJr?r=healthr s0 |556%>"  r?z/statusc tdtjD}tj}d|j rdnd|j |j r |j ndt|jd}tttjt|tttrdndtrd nd |d S) zGet system statusc3@K|]}|jddywr<r=Nr9rprs r=rrzstatus.. sF!!%%"%F Grok API (grok-4-fast-reasoning)zDeepSeek (local)zGrok APIN)chat_llmenrichment_llmrrgrok_availablerz hash-basedzFAISS + BM25 HybridzCosine + BM25 Hybrid)r total_filesrrembedding_model search_typerL) rrrenrichment_servicerLrrrrrrrr)r'rLllm_infos r=rr sF|/E/EFFG  C6030F0F,J!44030F0F#,,Ds//0 H |5567|%>4M0S_09,?U r?z/uploadPOSTcdtjvrtddidfStjd}|jstddidfSt |jj j }|dvrtddidfSt|j}tj|z }|jt|ttjdd }t|d d d tj t"|||fd }|j%td |d|ddfS)zUpload and process a documentrErNo file providedrEmpty filenamerrz$Only PDF and TXT files are supportedNstartingrz%File uploaded, starting processing...T)targetrdaemonz"File uploaded. Processing started.rGrrr+)rrFrr+rrrrrrrrIuuiduuid4r threadingThreadrr)rEext safe_filenamer rthreads r= upload_filer& s8W]]"!345s:: == D ==!123S88 t}}  $ $ * * ,C ""!GHI3NN$DMM2M.IIIc)n r "FFJ+RS   &i /F  LLN 7!     r?z/progress/ct5|tvrtt|cdddS dddtdddddfS#1swYxYw)zGet upload progressNrrz Job not found)rrr)rrr)rs r= get_progressr) sX 4 _ $?62344 $4 iQ?S TVY YY44s AAz/askcz ddl}|j}tjdxsi}|jddj }|jdd}|jdd}|st d d id fS|jd d }t j||}|s't dgtt jddSg}d} t} g} |D]} | jdi} | jdd}|r|r| jdr| dz } d| jdd}| jdg}|r.| j|dd|ddj|ddz }| jdg}|r| j||j||j| ddddj|}d |dd!d"|d#}d} tj }|j#|d$d%&}|rd'}|j }n d(}d)|dd*z}g}|D]} | jdi} | jd/d0t+| d1d| ddd2| jdd| jdr| jdddd3nd| jdgd4}|rZt+| jd5| d1dt+| jd6ddt+| jd7ddt+| jd8dd| jd9g| jd:g| jd;i| jdd0| jd?| jd@dA |dB<| jd;i}t-|jdCdDt/|jdEdDt1|jdFdDt3| dG|dH<|j|t5|j|z dIz}||tt jt|| t7| ddJ| dd ||dK }|r-t9|t;|tt=|dL|dM< t?||t |S#t$$r2}t'd+|d,t)|dd-}d.|dd*z}Yd}~d}~wwxYw#t$$r"}t'dN|Yd}~t |Sd}~wwxYw)Oz2Query the RAG system with enriched context supportrNTsilentquestionr= use_enrichedinclude_analysisrzQuestion is requiredrrrrz8I don't have enough information to answer that question.)rr'chunks_searchedenriched_countr:rPFrrMz **Summary**: rr z **Key Points**: z; r"rrrNz --- zYou are an AI Legal Research Assistant. Your role is to help users understand legal concepts, case law, statutes, and legal procedures based on the retrieved legal documents. === RETRIEVED LEGAL KNOWLEDGE === i z === QUESTION === am INSTRUCTIONS: 1. Answer based on the retrieved legal documents and your legal knowledge. 2. Cite specific sources, cases, or statutes when applicable. 3. Explain legal concepts clearly for the user's understanding. 4. If the question involves specific case law or procedures, reference the relevant details. 5. If you cannot find sufficient information, acknowledge limitations and suggest areas to research. 6. Always maintain professional legal language while being accessible. IMPORTANT: This is for educational/research purposes only. Remind users to consult a licensed attorney for specific legal advice. ANSWER:rLr4rr zNone (fallback to context)zGI could not generate a specific answer. Here is the relevant context: i z[RAG] LLM generation error: rrLz/Error generating answer. Here is the context: r<rr}r)r<r}previewrPrrrrrrrrrr{rurrr) rrrrrrrr{rrranalysisrr rr)precedential_strengthrargument_qualityoverallrrr*) rr'r1chunks_returnedr2rrrresponse_time_ms)detected_phaserexpanded_terms_countquery_analysisz[Analytics] Track error: ) rrget_jsonr9rrrrrrrrrrSrrrLrrrrIrrrrrrGrr~rr  track_query) time_module start_timer2r-r.r/rr> context_partsr2 all_themesall_key_pointsrmeta is_enrichedpartrrrrrr generated_answerrrr' source_datarr: response_datas r=askrK s~!!#J   4 ( .BDxx B'--/H88ND1Lxx 2D9 !7893>> aA!!(a!0G P"<#9#9:    MNJN 5uuZ$hhz51 KDHHY,? a N"488Ir#:"; B!&quu_a'@!!D#AEE,$:A>!"!;$%EE*&'ee,@&A$%EE*<$= 'K #UU8R(F)J6::VdfiKj)k,MfjjYgilNm,n$A&**ZY\B]$^7: )K % {#I$$NK,,.;tCD|556w<(z"3B'$Ra(, M5h?"8"B$'(B8(L$M+ &'/Hm, = !!W V ,QC01SVCR[M*DwuPT~UVP / )!-.. = !!/s10AS: T T 'TT  T:T55T:z/searchc6tjdxsi}|jddj}|jdd}|st ddid fSt j || }t ||tt jd S) zSearch documentsTr+rr=rr*rzQuery is requiredrr0)r>rr) rr>r9rrrrrr)r2rrr>s r=rr s   4 ( .BD HHWb ! ' ' )E bA !456;;!!%1!-G |556 r?z /documentsc i}tjD]c}|jdd}||vrdd|jddd||<||dxxdz cc<|jdsT||d xxdz cc<ettj}t d tjD}t |j Dcgc] \}}d|i| c}}t|||t|dkDr||z d znddd Scc}}w) z)List all documents with enrichment statusr<rrrF) chunk_countr2rFrNrMrPr2c3DK|]}|jdsdywrPrMNrr s r=rrz!list_documents.. sPqaeeJ>OP  r)r total_sourcesrtotal_enrichedenrichment_percentage)rrr9rrxrrwr)r'r*r<rrSrr$s r=list_documentsrU sG%% 39-   "#!$)!DGFO  &!+& 77:  FO, - 2 - 3|--.LPL$:$:PPN 5<]]_ETQx(a(EW$(!&P\_`P`(E(Kfgij!k  Es D z/chunkscPtjjd}tjjddjdk(}tjjddjdk(}t tjjdd}t tjjdd }t j }|r$|Dcgc]}|jd|k(s|}}|r"|Dcgc]}|jds|}}n#|r!|Dcgc]}|jdr|}}t|}|dz |z}||z} ||| } g} t| | D]\} } | j| | jdd | jd | | jdd | jdr| jddddnd| jdgdd| jdgt| jdddkDr| jdddddzn| jddt| jddd t| |||||zdz |zd|||ddScc}wcc}wcc}w)z)List chunks with pagination and filteringr<rPr=truerawrrMper_pagerI)rrrFrNrrr"rr,rC) rr<rrPrrrcontent_previewcontent_length)rrYrr)r< enriched_onlyraw_only)rd paginationfilters) rrr9rrGrrrrQrr)r<r]r^rrYfilteredrrrend paginatedrdrVr*s r= list_chunksrd s\\  h 'FLL$$Z4::<FM||r*002fAs$*JJJ'J4J# J#z/chunks/c |dks|ttjk\rtddidfStj|}t||j dd|j d|j d|j d d |j d |j d g|j d g|j ddt|j dd|j d|j dd S)z$Get a single chunk with full detailsrrzChunk not foundr(r<rrrFrPFrrrrr=rgr) rr<rrFrPrrrrr\rgr)rrrrr9) chunk_indexr*s r= get_chunkrg$ sQ+\-C-C)DD!234c99   -C ''(I.GGJ'1GGJ.779%gglB/''(B'779b)cggi45779%/   r?z$/chunks/by-source/ct|}tjDcgc]}|jd|k(s|}}|st ddidfSt d|D}t |t |||Dcgc]r}|jd|jdd|jd r|jd d d d nd |jd g|jdd d d dtc}dScc}wcc}w)z$Get all chunks for a specific sourcer<rzSource not foundr(c3DK|]}|jdsdywrPrrs r=rrz'get_chunks_by_source..E s@qaeeJ.?@rQrrPFrr=Nrrr)rrPrrr[)r<renriched_chunksrd)rrrr9rrxr)r safe_sourcerrdr2rs r=get_chunks_by_sourcerl< s"+.K%// RA155?k3Qa RF R !345s::@F@@N F )   j)j%056UU95EquuY+DS14eeHb) uuY3DS9     SsC6C66A7C;z/delete/DELETEcNt|}ttj}tjDcgc]}|j d|k7s|c}t_|ttjz }|dkDrtjrtjDcgc]}|d }}tj |}t rrtjtjt_ |jd}tj|tjj|n]|t_nQt r-tjtjt_ ntj gt_tj#tj%t&j(|z }|j+r|j-t/dd|d|d|d St/d d id fScc}wcc}w) zDelete a documentr<rrrTzDeleted z (z chunks))rGrchunks_removedrzDocument not foundr()rrrrr9rrrrrrrrrrrrrrrrrunlinkr) r+r$original_countr removed_countrrrr s r=delete_documentrsU s$H-M//0N)5)?)?dA155?VcCcadL"S)?)?%@@Mq  ! !.:.D.DE) EHE%;;HEJ%*%6%6|7Q7Q%R " * 1 1) < ""=1""&&}5*4 '%*%6%6|7Q7Q%R "*,((2, '..0NN]2        !-=/J+   G12 3S 88Ie FsHH% H"z/reloadcttjtdttjdS)zReload the indexT)rGr)rrrrrrJr?r=reloadru s3 |556 r?z/enrichct5tjddk(r7t1tj rt dtddfcdddStjddk(rndtd<tjdxsd td<t jtjjd ztd <tdddtjd xsi}|jdfd}tj|d atj!t d ddddfS#1swYuxYw)z,Enrich documents with LLM-generated metadatarrNzEnrichment already running)rrirrrrrTr+r<c 4 ttjjdtjddxsd} t j rAtt jDcgc]\}}|jdk(s|}}}n+tttt j}t|}|dk(rt5dtd<dtd <d td <tjtd <td<dtd <dtd<t!j"t$j&j)dztd<|td<t+dddyt-d|D}||z }t!j"t$j&j)dz}t5dtd<|rt||z dzndtd <d|d|d|dtd <tjtd <tjdxs|td<|td<td<|td <|td<|td<t+ddd|dkrt5dtd<dtd <dtd <tjtd <t!j"t$j&j)dztd<t+dddyd}|D]} t j| } | jdr)tj/| } | j1| t!j"t$j&j)dz| d< t j3| } tt j4tt jk(r!| j7t j4| <tt j8tt jk(rt j8| } t| j;}|t j8| <tt j8}|dkDr%t j<|z| z |z|z t _|dz }||z}|rt||z dznd}t5dtd<|td <d|d|d|d|d td <tjtd <t!j"t$j&j)dztd<|td<|td<ddd||zdk(sit j?t5t+dddt j?t5dtd<dtd <d|d |dtd <tjtd <t!j"t$j&j)dztd<|td<t+dddy#t$rY wxYwcc}}w#1swYyxYw#1swYxYw#1swYyxYw#t$rYwxYw#1swY:xYw#1swYxYw#1swYyxYw#t$r}tAd!|ddl!}|jEt5d"td<d#tG|td <tjtd <t!j"t$j&j)dztd<t+dddn#1swYnxYwYd}~yYd}~yd}~wwxYw)$Nrrr*r<rrrrrzNo chunks found to enrich.rrrrrrc3fK|])}tj|jds&d+ywrP)rrr9)rprVs r=rrz;enrich_documents..run_enrichment.. s*"j @V@VWX@Y@]@]^h@i1"js'11rzResuming enrichment: rz' already enriched. Enriching remaining rCrz0No chunks need enrichment (all already enriched)rP enriched_atrMz Enriched z chunks (this run: rzSuccessfully enriched z chunks (target: z[RAG] Enrichment error: rr)$rGr7r8r9rrrrrQrrrangerenrichment_lockrrNrrMrrrrrxrXrSrrrrrHrrrrrrI)rrVrtarget_indicesralready_enriched remainingrenriched_this_run doc_indexr* enriched_docnew_textold_lennew_lenndonerrr source_filters r=run_enrichmentz(enrich_documents..run_enrichment sz 2"2::>>2OQbQfQfgy{}Q~RECE$F G  668 09,:P:P0Q!v1UVUZUZ[cUdhuUu!!v!v!%eC 0F0F,G&H!I~.Lq $ 62=%h/47%j13O%i04F4V4V%j12?%h/89%n578%m46>ll8<<6P6Z6Z6\_b6b%l3ll8<<6P6Z6Z6\_b6b%l335 6 ! +) : ",,Y777:&1>>sC  <(%-\\(,,%?%I%I%Kc%QM" +BB3GH<889SAWAW=XXCK>>CS 55i@<334LE 00; 8 89q5;G;V;VYZ;Z]d;dgn;nrs:sL7"Q&!'*;;?K3| 3s:;QR$M2;%h/4<%j15>tfAl^Sfgxfyyz|E{FFG4H%i04F4V4V%j16>ll8<<6P6Z6Z6\_b6b%l37;%m4r!r"r)r2rrs @r=enrich_documentsr sL  .   *i 7.? s3$"-66qvv>$s58rpending)r+rsizerz'[RAG] Error getting processing status: NrF)rrriterdirrrrrrrrstatst_sizer fromtimestampst_mtimerrrr)rF processedrr#s @r=get_processing_statusr5 s E= >> "^^++- 88>>#'77 #$#/#9#9$!ILL$%FF1:+ ! 0 0$,$:$:1668;L;L$M$W$W$Y "  GU# $$ = 7s;<<=sAD!BD D(D##D(z/enrich-uploadcdtjvrtddidfStjd}|jstddidfSt |jj j }|dvrtddidfSt|jtjz |jtttjdd td d d fd }tj |d}|j#tddddfS)z,Upload and process with immediate enrichmentrErrrrrz Only PDF and TXT files supportedNrrrz5File uploaded, starting processing with enrichment...c & t dddtj }|rt|j dkr#t dddt j }|rt|j dkrt ddd yt dd d tj | }tj||}|st ddd yt|D]"\}} |d <||d<t||d<||d<$t dddt|d fd}tj||}t dddtj |t dddtjt dddt|dt|tjy#t$rF}t!d|ddl}|j%t dddt'|Yd}~yd}~wwxYw)Nrr*zExtracting text...rrz Trying OCR...rrzFailed to extract textzChunking document...zNo valid chunks extractedr<rrrFrLz Enriching z chunks with LLM...c Xdt||z dzz}td|d|d|dy)NrLrrzEnriching chunk rrC)rGr)currentrpctrs r=enrich_progresszEenrich_on_upload..process_and_enrich..enrich_progress s=3%2566 c=MgYVWX]W^^a;bcr?rzAdding to vector store...rrrz$Successfully processed and enriched z chunks!)rrz [RAG] Error: r)rrrrrrrFrerQrrrrrrNrrrrrI) rlrQrdrVrrrjrrr rr$s r=process_and_enrichz,enrich_on_upload..process_and_enriche s6 E FL"6J K --i8D3tzz|,s2 b/J"// :3tzz|,s2!5MN FL"6L M$99$ NH"--dH=F!5PQ%f- 25"/h$%j!(+F n%)1o&  2 FL" 3v;-Ob6c d d1BB6?[O FL"6Q R  ' '  G FL"6G H      S6s?7K6LHU 1+;;   E M!% &     ! FHa73q6(1C D D  Es&BGA GC!G H rr0rc3@K|]}|jddyw)rr=Nrrprs r=rrzvalidate_rag.. s N!y"!5 Nr r r})rexpected_terms found_termscoverage result_count top_scorec3&K|] }|d yw)rNrJrs r=rrzvalidate_rag.. s:Q1Z=:srG)roverall_coveragerr>) rrrrVALIDATION_QUERIESrrrrrrx) r>rrsearch_resultsr combined_textrrrs r= validate_ragr s^ < ! !"a'$K   G!3 ~%,,Ua,8  N~ NNTTV " )Dzz|},""4( )>L3{#c.&99QR,&h*/7E*731    *NUs:'::S\IZ[ !"2A6|556  r?z/legal-intelligencectdtttttttt tt ttttddttjddttjddttjdd ttjd d ttd d tt jt j#Dcic] \}}||d c}}ddtt$jt$j'Dcgc]}|d c}ddtt(jdd dddddddddd dd Scc}}wcc}w)!z5Get information about the legal intelligence featuresz-Legal terms weighted by importance tier (1-6))r^ tier_1_count tier_2_count tier_3_count tier_4_count tier_5_count tier_6_count total_termsz8Related legal concepts automatically expanded in queries)r^clusterszPhase-aware search boosting)r^phasesz$Doctrine-specific search enhancement)r^ doctrineszCourt authority level boosting)r^levelsz Legal citation pattern detection)r^ pattern_countzASource authority weighting (gold standard, persuasive, declining)r&)r^rweightsz#Legal reasoning pattern recognitionr)r^schemasrz1Doctrine relationship mapping for query expansion) tiered_keywordsconcept_clusterslitigation_phaseslegal_doctrinesjurisdiction_hierarchycitation_recognitionprecedential_hierarchyargument_schemascross_referencesz2Documents from same litigation phase get 15% boostzDU.S. Supreme Court: 1.5x, Federal Circuit: 1.3x, State Supreme: 1.2xz12% boost per matched doctrinez+Up to 24% boost for Supreme Court citationsz-Positive treatment: +5%, Heavy negative: -10%z=Gold standard: 2.0x, Highly persuasive: 1.5x, Declining: 0.4xz.Recent statutes boosted, landmark cases stablez2Well-structured arguments boosted by schema weightz*Up to 25% boost for related legal concepts) phase_awarerdoctrine_specificcitation_authoritycitation_treatmentrrrcross_reference )features search_booststotal_boost_factors)rrrrrrrrrrrrrvrrrr>rwrvaluesr)rr$rKs r=legal_intelligence_infor s  O #L 1 #L 1 #L 1 #L 1 #L 1 #L 1";/  Z !7!?! =05578" F!/"6"6"89  @5::<=' B!$%z/analyze-queryc tjxsi}|jdd}|stddidfSt |}t |}t |}|j}ggggggd}tD],}||vst|}|dkDs||j|.t||||dd |jD cic] \} } | s | |  c} } t|d Scc} } w) z>Analyze a query to show how legal intelligence will process itrr=rzQuery requiredr)rMr r"r rrrNrI)rr;rr  tier_matchestotal_expansion_terms) rr>r9rr~rr rrrrrwr) r2rr{rr rrrr"rr$s r= analyze_queryrs     #D HHWb !E !123S88 $E *E&u-I/6N++-KbRB2"=L0 ;  &DaxT"))$/ 0 '("-*6*<*<*>D$!Q!AD!$^!4   Es C.C.)queries daily_statsdoctrinal_coverageresponse_quality session_startcBtjdz atjr t td5}t j |}tj|tjtjtdz j}tjdgDcgc]}|jdd|kDr|c}td<dddyycc}w#1swYyxYw#t $r}t#d |Yd}~yd}~wwxYw) z5Initialize analytics file path and load existing datazanalytics.jsonrr)daysr timestampr=Nz%[Analytics] Error loading analytics: )rrANALYTICS_FILErrrranalytics_datarSrrMrr timedeltarr9rr)r#loadedcutoffqrs r=init_analyticsrHs__'77N ?nc* Qa1%%f-",,x||4yb7IITTV8F8J8J9VX8Y-Q101k20F0O./-Qy)  Q Q-Q  Q Q ? 9!= > > ?sGC=BC1:C, C1"C=,C11C:6C=:C== DDDctsy ttd5}tjt|dddy#1swYyxYw#t $r}t d|Yd}~yd}~wwxYw)zSave analytics to diskNrz$[Analytics] Error saving analytics: )rrrrrrr)r#rs r=save_analyticsrXs^ : .# & )! IIna ( ) ) ) : 4QC899:s1A =A AA A A*A%%A*rJc Zt5tjtj}|j d}t |}t|}t|}|jt|||t|jdg|jdd||jd|jddd }tdj|ttdd kDrtdd d td<|td vrdddiiddd td |<td |}|dxxdz cc<|dxx|z cc<|d|dz |d<|dxx|jddz cc<|d|dz |d<||dvrd|d|<|d|xxdz cc<|D]L} | |dvrd|d| <|d| xxdz cc<| tdvr dtd| <td| xxdz cc<Nttddzdk(r td d d y #1swYy xYw)zTrack a query for analyticsz%Y-%m-%dr'r2rrr:) r query_lengthr{r sources_countr2 quality_scorerr:rriNr) query_count avg_quality total_qualityrdoctrines_usedavg_response_timetotal_response_timerrMrrrrrrrr*)analytics_lockrrMrrstrftimer~rcalculate_response_qualityrrr9rrr) rrJrMtodayr{rr query_recorddailyrs r=r?r?bs Bll8<<( Z((.*51 3=A J" !2!29b!AB+//0@!D*%))*5 - 1 12Da H   y!((6 ~i( )D 0(6y(A%&(IN9 % }5 5  !""$%&'(4N= )% 0}-e4 m! o-/$_5m8LLm #$ (9(9:La(PP$%*+@%AE-DX%X!" h '%&E(OE " h!#" @Hu%56645&'1 " #H - 2 -~.BCCAB34X> / 0 :a ? : @ ~i( )B .! 3  EBBBs HH!!H*cd}|jdg}|r|tdt|dzz }|jdd}|r|td|dzz }|jdd }t|d kDr|d z }t|d kDr|d z }t|d kDr|dz }|jdg}|tdt|dzz }|jdg}|td t|dzz }td |S)z0Calculate a quality score (0-100) for a responserr'rrr2rIrrr=rr*rZrNrrr"rr )r9rr)rJr}r'rPrrrs r=rrs E 2.G RW)**  !115H RA&&  x ,F 6{S   6{S   6{S    x ,F SS[1_ %%E""<4J SS_q( ))E sE?r?z /analyticscPt5tjdg}t|dkDr|ddn|}|rt d|Dt|z nd}|rt d|Dt|z nd}|rt d|Dt|z nd}i}|D],}|jd d }|j|dd z||<.t tjd ij d dd}g} |D]_\} } | j| | jddt| jddd t| jddddat tjdij dd dd} tt|t|d t|d t|dtjdd|| | D cgc] \} }| |d c}} ttjdittdttjt dtjDttdtjDddcdddScc}} w#1swYyxYw) zGet analytics dashboard datarriNc3@K|]}|jddyw)rrNrrprs r=rrz get_analytics..L!%%3Lr rc3@K|]}|jddyw)rrNrrs r=rrz get_analytics..rr c3@K|]}|jddyw)r:rNrrs r=rrz get_analytics..sU&8! <Ur r{rrMrT)rVrrr)daterrrrc |dSrrJrs r=rzget_analytics..s !A$r?rr*r) total_queriesavg_quality_scoreavg_sources_per_queryavg_response_time_msr)rr)coveredtotal_availablec3DK|]}|jdsdywrPrr s r=rrz get_analytics..s)aquuU_O`!)arQc3@K|]}|jddywrrr s r=rrz get_analytics..s)^!!%%"*=)^r )renriched_documentsunique_sources)rphase_distribution daily_trends top_doctrinesrdocument_stats)rrr9rrxrrwrrrrrrr) all_queriesrecent_queriesr avg_sourcesr phase_countsrr{ recent_daysrdaystatsrrrs r= get_analyticsrs 6 $((B7 /2;/?#/ETU+;gucL^LLsSaObbz{ ftcL^LLsSaObbz{ o}CUnUUX[\jXkkDE  AAEE'9-E"."2"25!" > @  2  !$[!1%*;%:).{A)>(-.?(C!/!3!3O!D #/(FSTda1q9T~112FKL#&#7# $'|'='=#>&))a\5K5K)a&a"%c)^|G]G])^&^"_  E6 6 XUY6 6 sGJ%J4BJJJ%z/analytics/query-logcpt5ttjj dd}ttjj dd}t j dg}t t||||z}t|t|||dcdddS#1swYyxYw)zGet recent query loglimitrLoffsetrr)rrrrN) rrGrrr9rrrrr)rrrrecents r= get_query_logr s   GLL$$Wb12W\\%%h23 $$Y3hw'(?\       s BB,,B5z/analytics/doctrinal-coveragect5tjdi}g}tj D]\}}|j|d}d}t j D]?}|jddjtfd|ddDs;|dz }A|j||||ddtd |tdtt j z d zd zd |jd d t|ttt|Dcgc] }|ddkDs |c}t|Dcgc] }|ddkDs |c}dcdddScc}wcc}w#1swYyxYw)z&Get detailed doctrinal coverage reportrrrr=c3BK|]}|jvywrr)rprrs r=rrz)get_doctrinal_coverage..'sE4tzz|w.ErNrrMrr )rrdocument_count key_termscoverage_scorec |dS)Nr#rJrs r=rz(get_doctrinal_coverage..3s !$4"5r?Trrr#)rtotal_doctrinescovered_in_queriescovered_in_documents)rrr9rrwrrrrrrryrrr) rreportrr|r doc_countr*rrs @r=get_doctrinal_coverager,s~  !%%&:B?.446 OHe",,x3KI#-- #'')R0668E5!9EENI # MM$*"+"2AY"%cIAsQTU>U(V$W   3  8'R(V;  s=BE5BE5" E+0E+4E5 E0E0 E5+ E55E>z/analytics/exportcXt5ttcdddS#1swYyxYw)zExport analytics data as JSONN)rrrrJr?r=export_analyticsr.=s$ '~&'''r)rz[Analytics] Init error: __main__RAG_PORTiz;[RAG] Starting Advanced Legal Document RAG Service on port z [RAG] FAISS: z, SentenceTransformers: z , PyMuPDF: z[RAG] Documents loaded: z0.0.0.0)hostportdebugr)rCr7sysrrrrrr(r!queuerrpathlibr functoolsr dataclassesrtypingr r r r r flaskrrrrrwerkzeug.utilsrrr^rrrrrrrrr!rIrGrrHr#STRATEGIC_ADDITIONSrrrvr~rrrrrrrrrrrrr rr>rArRrrrrrENHANCED_METADATA_FIELDSrr.r/r%r ImportErrorpdfminer.high_levelrrrPyPDF2rrnumpyr HAS_NUMPYsentence_transformersrrrrrrrrrr8rIr@appcase_apirregister_blueprintrrrrrrrrr8r9rLockrr{Queueprocessing_queuerrrrrrrrrrrouterrrrrrrrr&r)rKrrUrdrgrlrsrurrrrrrrrMrrrrrrrr?rrr r,r.rrr2rrrunrJr?r=rLs    '!33OO*  $8$8 $8P!$sdhi|}~#sdhi|}~3$dexy z!SDfgz{|"cTgh{|}               <,. "./1>?        &#% '(         i9~   A$L # #   djT-/FG] SSMNS? ST        I)V#$& C D (cdD1BD/=?179 ;=+/1";="68 :<4EN UL 38 A'6#:#d#4#SX#RR7 :8 K< B6 <9 X8 O.`CDBEKVa S8.IOS Z-/KLRV SI57RS\ LPNT  Y]M[ ZYKL MMMU Y2h.33* #  $4>   KDL J I &9 $ I LzzzMMfUUv++bPPl Hk27 7# 89 V$ 6"  &v.BJJNN+H"MN    ).."5;;=)),DDA *F 0PQ AB !#  C      c  8AC8AD8AC8A~3++8~ ~ J dU dt d XU Xt X U U4 U..$.b9ug&'9ug&'49vh'# (# L %1Z2Z6F8$h"%h"R9vh'($<%)*89ug&6'6r &89. 1E7CD0 $x.@A*9B*9Z9vh'(9vh'Z (Z z %1*2*  %1%2%2 fX.U /U v;(&)&R 5'2=3=@ fX./L!X\\(,,/99;  !? :DsD4DLdsD<%)8 *8 v !E73 4 $ *UG<  =  F 0'1'*" z rzz~~j$/ 0D Gv NO M)$<=V=c>d  d ddd8" d33d8;e ee