HCi?DdZddlZddlZddlZddlZddlZddlmZmZm Z  ddl m Z m Z dZ d Zd Z d2d ed e ed ededede ef dZdedefdZdedefdZdedefdZdeedeefdZefdeededeefdZdeedeeeeffdZdedeededefdZdedeedefdZd3ded ed!edefd"Zded#edefd$Z e!d%k(r1d&d'd(d)id*d+d,d(d)id*d-d.d(d/id*gZ"ed0e"Z#e$d1e$e#yy#e$rejddZ dZ dZ YwxYw)4zz Answer Synthesis Pipeline - 3-Layer Enhancement for RAG Quality Fixes: Answer incoherence, chunk mixing, reasoning leaks N)ListDictOptional) GROK_API_KEY GROK_MODELz$https://api.x.ai/v1/chat/completions XAI_API_KEYz grok-2-1212gffffff?prompt system_prompt max_tokens temperaturetimeoutreturncttsy dtdd}g}|r|jd|d|jd|dt|||dd }tjt ||| }|j |jd d d dS#t$r} td| Yd} ~ yd} ~ wwxYw)z4Call Grok API for synthesis (grounded final answer).NzBearer zapplication/json) Authorizationz Content-Typesystem)rolecontentuserF)modelmessagesr rstream)headersjsonrchoicesrmessagerzGrok API Error: ) rappendrrequestspost GROK_API_URLraise_for_statusr Exceptionprint) r r r rrrrpayloadresponsees 4/var/www/html/leadgen/airagagent/answer_synthesis.py call_grok_apir)s &|n5.    OOX-H IF;<  $&  ==wWV]^!!#}}y)!,Y7 BB   $%sB B B7B22B7textc|s|Sd|vr"|jddj}d|vr&d|vr"|jddj}|jddj}|jddj}gd}gd }|j}d}|D]4}|j |j}|dk7s(|dk(s||ks3|}6|dkDr$|t |d zkr||d jS|jd }g}d } d} |D]} | j| rst fd|D} tjdtjrd } | r| dz } [t dkDs| rld} d} |j| d j|j} t | dkrht |dkDrZ|jd}t|D]9jt dkDs"t fd|Dr7cS|S| r| S|S)zx Remove AI reasoning/thinking from response. Handles DeepSeek R1 model's verbose reasoning about documents. zzru<｜end▁of▁sentence｜>r z<|end_of_sentence|>)zC^(Okay|Alright|Let me|I'll|I need to|Looking at|Based on|First,? I)z6^(The user is asking|The question asks|To answer this)z-^(I see that|I notice that|I can see|I found)z)^(Checking|Analyzing|Reviewing|Searching)z.^(First, there's|Document \d|Now,? looking at)z)^(Putting it all together|So,? from this)z/^(That's about|That's not relevant|Not related)z$^(Also unrelated|Again not relevant)) z The answer isz In summary,z To summarize,zIn conclusion,z Therefore,zThe book is aboutzWilliam Cooper wrotezThe main themes arezBased on the documents:z**From N Tc3hK|])}tj|tj+ywNrematch IGNORECASE).0pattern line_strippeds r( z"strip_reasoning..s"mw288G]BMMJm/2z+^Document \d+ (is|talks|discusses|mentions)2Fd c3hK|])}tj|tj+ywr0r1)r5pparas r(r8z"strip_reasoning..s#*hPQ288AtR]]+K*hr9) splitstripreplacelowerfindlenanyr2r3r4rjoinreversed)r*reasoning_startersanswer_markers text_lower answer_startmarkerposlines clean_linesskip_reasoningconsecutive_reasoningline is_reasoningcleaned paragraphsr7rAs @@r(strip_reasoningrYAse  Tzz*%b)//1DZt3zz)$Q'--/ <<5r : @ @ BD <<-r 2 8 8 :D  NJL #ooflln- "9r!S<%7" #aL3t9>9LM"((** JJt EKN)  - mZlmm  88BMSUS`S` aL  !Q & ! =!B&n!&()%""4(-)0ii $**,G 7|cc$i#oZZ' Z( D::z*group_chunks_by_document..sgq@QrZTkeyreverseN)rlr2r`rDrsortedMAX_CHUNKS_PER_DOC)rigroupedrprzs r(group_chunks_by_documentrsG&:r*..xC("f5S)11#s;   GFOu%&v 6Q[_`atbtuv NrZdoc_namequeryc|syg}t}|D]}t|jdd}t|dkr-|ddj }t |}|r||vrR|r|j ||j|t|dk\sn|sydj|D cgc]} d|  c} } | Scc} w) z Build a compact, grounded snippet from top chunks. We prefer verbatim excerpts (trimmed) over weak extractive summaries. r r(Nir-r.z- ) rkrarlrGrCrhrmrrI) rrirexcerptsrnrprexcerptrqr'bulletss r(summarize_document_chunksrs H 5D "599Y#;< w<"  $3-%%'  ( "*  HHRL x=A    ii84a2aS45G N5s, C raw_chunksc|j}ttjd|}t}|D]8}|j dij dd}|s(|j |:i}|D]}tj dd|tj} | jddjd d} hd } tjd | D cgc]%} | j| vs| j'} } | D]} | |vrg|| <|| j|!t}|D]L}||vr|j|||jD]\}}||vs||vs|j|!Ng}|D]}|j d d }|j dij dd}d }||vrd}||z}|dk\s|d kDsOt|j dd|d<||d<|j||jddt|}|syt|dd}g}g}|j!D]K}|jt#fd|D}|r|j|;|j|M|j%|g}|D].}||}t'|||}|s|jd|d|0|sydj)|Scc} w)z LAYER 1-2: Transform raw chunks into clean, organized context. Args: query: User's question raw_chunks: Raw chunks from vector search Returns: Enhanced, organized context string z\b\w+\bryrzr r|r\r}r_r~> byandcomfororgpdfthewww oceanofpdfz \b\w{3,}\brurg@g333333?radjusted_scorecF|jd|jddS)Nrrurrrs r(rz+enhance_retrieved_context..Ps&6gq8I JrZTrzANo sufficiently relevant information found in the knowledge base.Nc3DK|]}t|dkDs|vyw)r N)rG)r5qw clean_docs r(r8z,enhance_retrieved_context..asQSWq[B)OQs  zSOURCE: r.r>)rErkr2findallrlrmr`r4rDrupdateitemsrasortrrrkeysrHextendrrI)rr query_lower query_words doc_namesrprz doc_keywordsr clean_name stop_wordswwordsword matching_docs query_wordkeyworddocsrvruboostrr ordered_docs other_docs matches_query context_parts doc_chunkssummaryrs @r(enhance_retrieved_contextr s;++-Kbjj[9:KI":r*..x<  MM& !"L 0VV,b("--P ''S199#sC Z $&JJ}j$IiqQWWY^hMhii 0D<'%' T"   % %h / 0 0EM!+  %  j!9 :)//1 +MGTW$:(=$$T* + +H# '1%:r*..x< ] "E S EAI/ )R0HIE) &4E" # OOE "#" MMJTXMYX&H R'x}5GLJLLN(NN$ QkQQ     )   h '( #M CX& +Hj%H   8H:Ry!A B C R ;;} %%Ojs $L<Lenhanced_contextuse_grokc|xrd|v}|r0tr*|r(d}d|d|d}t||ddd }|r t|S|}|S) a4 LAYER 3: Generate final coherent answer using LLM. Args: query: User's question enhanced_context: Clean, organized context from Layer 1-2 use_grok: Whether to use Grok API (falls back to simple format if False) Returns: Final synthesized answer z$No sufficiently relevant informationaYou are CashHive RAG, an expert research assistant. You MUST follow these rules: - Use ONLY the provided SOURCES and their excerpts. Do not add outside facts. - Write a single coherent answer that merges the sources. - After factual claims, add citations like [SourceName]. Multiple citations allowed. - If sources conflict, explicitly note the conflict and cite both sides. - If the sources do not contain enough info, say what is missing. - Do NOT reveal chain-of-thought or hidden reasoning. z QUESTION: z SOURCES (verbatim excerpts): z9 Write the best possible grounded answer with citations.皙?K)r r r rr)rr)rY)rrr has_contextr synthesis_prompt grok_responseanswers r(synthesize_answerrvs#e'MUe'eKL[ F  8 ;&#'   "=1 1!!F MrZ raw_responsec|jdg}|jddt||}dvxs$tdkxstfddD}|s t ||}||d d dt|d S) z Main entry point: Process RAG response through 3-layer pipeline. Args: query: Original user query raw_response: Raw response from RAG /ask endpoint Returns: Enhanced response with synthesized answer sourcesrr Chunkr;c3HK|]}|ddjvyw)Nr<)rE)r5roriginal_answers r(r8z'process_rag_response..s& `dDODS)//1 1 `s")zlet mez looking atzi seeTN)rrenhanced source_count)rlrrGrHrrY)rrrrneeds_enhancement final_answerrs @r(process_rag_responsers!!)R0J"&&x4O1 C ?" a Or! a `>_ `` D(0@A bq>J  rZ__main__z:(Chunk 72) William Cooper discusses UFOs and government...g333333?rzzWilliam_Cooper_Pale_Horse.pdf)rruryz(The secret societies have long hidden...g ףp= ?z%Hasheesh was introduced by doctors...g?zHasheesh_Eater.pdfz2What are the main themes in William Cooper's book?zEnhanced Context:)Nrr<)T)%__doc__r2rrosrctypingrrrgrok_apirrr! ImportErrorgetenvMIN_CHUNK_SCORErstrintfloatr)rYrarhrrrwrrrboolrr__name__ test_chunksresultr$rZr(rsm  ''19L $( $ $C=$$ $  $ c] $Ng(#g(#g(T33ESESE $t* d :I$t*TRVZT$ZDd4j4I(T$ZPS@f&Sf&d4jf&Sf&R/S/C/4/SV/d""4"D"L zP[_nvyXnY Z>\dgF\G H;dYacwXxyK ''[]h iF  &M} 299]B/L9LJs DDD