i PSSKrSSK7 SSKrSSKrSrSr\S4Sjr SSjr SSjr \ r SSjr S r\rSS jrS rS r\rSS jr\rSSjr"SS5rS Sjr"SS5r\S4SjrS!Sjr"SS5rSr\rSr\ r!S"Sjr g)#N)*c[R"USS9nURup#[R"X!4SS9n[R"X!4SS9n[R "5n[ U5Ul[ U5UlX&l Xl UR5 URU[ U55 UR5 XT4$)zLreturn k smallest values (and their indices) of the lines of a float32 arrayfloat32dtypeint64)npascontiguousarrayshapezerosfaissfloat_maxheap_array_tswig_ptridsvalnhkheapifyaddnreorderarrayrmnIDhas d/home/dmtnaga/Documents/work/airagagent/rag_env/lib/python3.13/site-packages/faiss/extra_wrappers.pykminr  i 8E ;;DA !w'A !y)A  $ $ &B a[BF a[BF E DJJLGGAxJJL 4Kc[R"USS9nURup#[R"X!4SS9n[R"X!4SS9n[R "5n[ U5Ul[ U5UlX&l Xl UR5 URU[ U55 UR5 XT4$)zKreturn k largest values (and their indices) of the lines of a float32 arrayrrr)r r r r r float_minheap_array_trrrrrrrrrs rkmaxr$+r r!c [R"USS9n[R"USS9nURupEURupgXW:Xde[R"XF4SS9nU[:Xa,[ XT[ U5U[ U5[ U55 U$U[:XaXR-USS&U$[XT[ U5U[ U5X#[ U55 U$)zFcompute the whole pairwise distance matrix between two sets of vectorsrrN) r r r empty METRIC_L2pairwise_L2sqrrMETRIC_INNER_PRODUCTTpairwise_extra_distances) xqxbmetric metric_argnqdnbd2diss rpairwise_distancesr5=s  b 2B  b 2B HHEB XXFB 7N7 ((B89 -C  8B<  SM  J ' 'ddA J ! 8B<   SM   Jr!cp[R"USS9n[[U5URU5 U$Nrr)r r& float_randrsizerseedress rrandr=Vs+ ((1I &Cx}chh- Jr!c[R"USS9nUc"[[U5URU5 U$[ [U5URX!5 U$Nrr)r r& int64_randrr9int64_rand_max)rr;vmaxr<s rrandintrC\sM ((1G $C |8C=#((D1 J x}chh; Jr!cp[R"USS9n[[U5URU5 U$r7)r r& float_randnrr9r:s rrandnrFhs+ ((1I &C sxx. Jr!c URS5nURS:Xa[UR[ U55$UR up[ R"USS9n[X[ U5[ U55 U$)z=compute a checksum for quick-and-dirty comparisons of arrays uint8uint64r) viewndim bvec_checksumr9rr r r bvecs_checksum)arr1css rchecksumrQnsc wAvv{QVVXa[11 77DA !8 $B1!hrl3 Ir!c^[R"X4SS9n[X[U5U5 U$r7)r r&rand_smooth_vectors_cr)rr1r;r<s rrand_smooth_vectorsrTzs) ((A6 +C! t4 Jr!c R[R"USS9n[R"USS9nURSnURSU:XdeURSURSpCSn[U5H*nU[ U[ X5U[ X55- nM, U$)z;size of intersection between each line of two result tablesrrrrI)r r r rangeranklist_intersection_sizer)I1I2rk1k2ninteris reval_intersectionr^s  b 0B  b 0B  A 88A;!   XXa["((1+ F 1X, Xbe_6 6 Mr!cd[URSURS[U55 g)NrIr)fvec_renorm_L2r rxs r normalize_L2rcs"1771:qwwqz8A;7r!c [R"USS9nUc[UR5S-5n[R"US-SS9n[R"UR SS9n[ UR [R"URS55U[R"U5[R"U5U5 X44$)aPerform a bucket sort on a table of integers. Parameters ---------- tab : array_like elements to sort, max value nbucket - 1 nbucket : integer number of buckets, None if unknown nt : integer number of threads to use (0 = use unthreaded codepath) Returns ------- lims : array_like cumulative sum of bucket sizes (size vmax + 1) perm : array_like perm[lims[i] : lims[i + 1]] contains the indices of bucket #i (size tab.size) rrrIrJ) r r intmaxr&r9 bucket_sort_cr rrK)tabnbucketntlimsperms r bucket_sortrms&  s' 2Ccggi!m$ 88GaKw /D 88CHHG ,D %..(!34%u~~d';  :r!c BURS:XdURS:XdeURup4Uc[UR5S-5n[R "US-SS9n[ X4[R"U5U[R"U5U5 U$)afPerform a bucket sort on a matrix, recording the original row of each element. Parameters ---------- tab : array_like array of size (N, ncol) that contains the bucket ids, maximum value nbucket - 1. On output, it the elements are shuffled such that the flat array tab.ravel()[lims[i] : lims[i + 1]] contains the row numbers of each bucket entry. nbucket : integer number of buckets (the maximum value in tab should be nbucket - 1) nt : integer number of threads to use (0 = use unthreaded codepath) Returns ------- lims : array_like cumulative sum of bucket sizes (size vmax + 1) int32rrIr) rr rerfr r&matrix_bucket_sort_inplace_cr r)rhrirjnrowncolrks rmatrix_bucket_sort_inplacerss, 99 399#77 7JDcggi!m$ 88GaKw /D  ENN3'%  Kr!c4\rSrSrSrS SjrSrSrSrSr g) ResultHeapz[Accumulate query results from a sliced dataset. The final result will be in self.D, self.I.cp[R"X4SS9Ul[R"X4SS9UlXsUlUlU(a [ 5nO [5nX$lXl[UR5Ul [UR5Ul UR5 X@l g)z{ nq: number of query vectors, k: number of results per query keep_max: keep the top-k maximum values instead of the minima rrrN)r r rrr0rr#rrrrrrheaps)selfr0rkeep_maxrxs r__init__ResultHeap.__init__s 2'12'3 )+E)+ETVV$ TVV$   r!cURup4[R"USS9n[R"USS9nURX44:XdeX0R:XdeURR U[ U5[ U5U5 g)z{ Add results for all heaps D, I should be of size (nh, nres) D, I do not need to be in a particular order (heap or sorted) rrrN)r r r r0rx addn_with_idsr)ryrrr0kds r add_resultResultHeap.add_resultsy   ) 4  ' 2ww2("""WW}}    QK r!c URupEU[U5:XdeURS:XaURUR:Xd#URS:XaURU4:Xde[R"USS9n[R"USS9n[R"USS9nURS:XaSOUnUR R U[U5U[U5[U5U5 g)z Add results for a subset of heaps. D, I should hold resutls for all the subset as a special case, if I is 1D, then all ids are assumed to be the same rIrrrrN)r lenrLr r rxaddn_query_subset_with_idsr)rysubsetrrnsubsetr id_strides radd_result_subsetResultHeap.add_result_subsets gg #f+%%% FFaKAGGqww. FFaKAGGv-  .  ) 4  ' 2%%fG<1A"  -- Xf%  Xa[) r!c8URR5 gN)rxr)rys rfinalizeResultHeap.finalizes r!)rrrxrr0NF) __name__ __module__ __qualname____firstlineno____doc__r{rrr__static_attributes__r!rrurus(  *r!ruc fURUR:XdeURup4n[R"XE4URS9n[R"XE4URS9nU(a[O[ nU"XEU[ U5[ U5[ U5[ U55 Xg4$)z Merge a set of sorted knn-results obtained from different shards in a dataset Dall and Iall are of size (nshard, nq, k) each D[i, j] should be sorted returns D, I of size (nq, k) as the merged result set r)r r r&rmerge_knn_results_CMaxmerge_knn_results_CMinr) DallIallrznshardrrDnewInewfuncs rmerge_knn_resultsrs :: ## #::LFq 88QF$** -D 88QF$** -D%- !3ID f :r!c&\rSrSrSrSrSrSrg)MapInt64ToInt64i1c,[[R"U55UlUSUR-:XdS5eXl[R "US4SS9Ul[R"UR[UR 55 g)Nrzneed power of 2 capacityrr) rer log2 log2_capacitycapacityr&rhr hashtable_int64_to_int64_initr)ryrs rr{MapInt64ToInt64.__init__3sq !231 2 222N4NN2 88XqM9 ++D,>,>@RSr!c URunURU4:Xde[R"UR[ UR 5U[ U5[ U55 gr)r r hashtable_int64_to_int64_addrrrh)rykeysvalsrs raddMapInt64ToInt64.add:sR ZZzzaT!!! **    2 x~x~ /r!c URun[R"U4SS9n[R"UR [ UR5U[ U5[ U55 U$r?)r r r&r hashtable_int64_to_int64_lookuprrrh)ryrrrs rlookupMapInt64ToInt64.lookupAsW ZZxxG, --    2 x~x~ / r!)rrrhN)rrrrr{rrrrr!rrr1sT/r!rch[R"USS9n[R"USS9nURupVURupxXh:Xde[R"XR4SS9n [R"XR4SS9n U[:Xa7[ [ U5[ U5XeXr[ U 5[ U 55 X4$U[:Xa7[[ U5[ U5XeXr[ U 5[ U 55 X4$[[ U5[ U5XeXsXB[ U 5[ U 55 X4$)au Compute the k nearest neighbors of a vector without constructing an index Parameters ---------- xq : array_like Query vectors, shape (nq, d) where the dimension d is that same as xb `dtype` must be float32. xb : array_like Database vectors, shape (nb, d) where dimension d is the same as xq `dtype` must be float32. k : int Number of nearest neighbors. metric : MetricType, optional distance measure to use (either METRIC_L2 or METRIC_INNER_PRODUCT) Returns ------- D : array_like Distances of the nearest neighbors, shape (nq, k) I : array_like Labels of the nearest neighbors, shape (nq, k) rrr) r r r r&r' knn_L2sqrrr)knn_inner_productknn_extra_metrics) r,r-rr.r/r0r1r2r3rrs rknnrMs2  b 2B  b 2B HHEB XXFB 7N7 "(A " *A  RL(2, 2(1+x{ 4K ' ' RL(2, 2(1+x{  4K  RL(2, 2z QK! 4Kr!c URupEURupgXW:Xde[R"XB4SS9n[R"XB4SS9n US:Xa[R"5n X*lXJl[R"U 5U l[R"U5U l [R"U [R"U5[R"U5UUS5 X4$US:Xan[R"[R"U5[R"U5XFX%[R"U5[R"U 55 X4$[e)a^ Compute the k nearest neighbors of a set of vectors without constructing an index. Parameters ---------- xq : array_like Query vectors, shape (nq, d) where d is the number of bits / 8 `dtype` must be uint8. xb : array_like Database vectors, shape (nb, d) where d is the number of bits / 8 `dtype` must be uint8. k : int Number of nearest neighbors. variant : string Function variant to use, either "mc" (counter) or "hc" (heap) Returns ------- D : array_like Distances of the nearest neighbors, shape (nq, k) I : array_like Labels of the nearest neighbors, shape (nq, k) rorrhcrImc) r r r&r int_maxheap_array_trrrrrhammings_knn_hchammings_knn_mcNotImplementedError) r,r-rvariantr0r1r2r3rrheaps r knn_hammingrs2 HHEB XXFB 7N7 "(A "(A$((*>>!$>>!$  %..$ennR&8" q  4K D  NN2 r 2BA NN1 u~~a0 4K"!r!c>\rSrSrSrSrSrS SjrS SjrSr S r g) Kmeansia_Object that performs k-means clustering and manages the centroids. The `Kmeans` class is essentially a wrapper around the C++ `Clustering` object. Parameters ---------- d : int dimension of the vectors to cluster k : int number of clusters gpu: bool or int, optional False: don't use GPU True: use all GPUs number: use this many GPUs progressive_dim_steps: use a progressive dimension clustering (with that number of steps) Subsequent parameters are fields of the Clustring object. The most important are: niter: int, optional clustering iterations nredo: int, optional redo clustering this many times and keep best verbose: bool, optional spherical: bool, optional do we want normalized centroids? int_centroids: bool, optional round centroids coordinates to integer seed: int, optional seed for the random number generator c |XlURU5 SUlSU;a[5UlO[ 5UlUR 5HUup$US:XaUS:XdUS:Xa [5nX@lM)[URU5 [URX$5 MW UR5 g)zd: input dimension, k: nb of centroids. Additional parameters are passed on the ClusteringParameters object, including niter=25, verbose=False, spherical = False Fprogressive_dim_stepsgpuTN) r1resetr"ProgressiveDimClusteringParameterscpClusteringParametersitems get_num_gpusgetattrsetattr set_index)ryr1rkwargsvs rr{Kmeans.__init__s  1  "f ,8:DG*,DGLLNDAEz9R$A#&# r!cURnURR[:Xa}URR(a[ U5UlO[U5UlUR(a/[R"UR URS9UlggUR(a[URS9nO [5nX l g)N)ngpu)r1r __class__r spherical IndexFlatIPindex IndexFlatL2rr index_cpu_to_all_gpusGpuProgressiveDimIndexFactoryProgressiveDimIndexFactoryfac)ryr1rs rrKmeans.set_indexs FF 77   4 4ww  (^ (^ xx"88$((S xx3B02Hr!NcTUb[U5UlSUlSUlSUlg)z^prepare k-means object to perform a new clustering, possibly with another number of centroids N)rer centroidsobjiteration_stats)ryrs rr Kmeans.resets* =VDF#r!c [R"USS9nURupEXPR:XdeURR [ :Xa[XPRUR5nUbDURupxX:Xde[R"UR5UR5 URXRU5 OmUbeUbeURR(ae[!XPRUR5nURU[#U5UR$5 [R&"UR5n U R)URU5Ul UR*n [-U R/55V s/sHoR1U 5PM n n [R2"U V s/sHoR4PM sn 5UlSR75n U V Vs/sHn U Vs0sHo[9X5_M snPM! snn UlUR4R.S:aUR4S$S$s sn fs sn fs snfs snn f)a|Perform k-means clustering. On output of the function call: - the centroids are in the centroids field of size (`k`, `d`). - the objective value at each iteration is in the array obj (size `niter`) - detailed optimization statistics are in the array iteration_stats. Parameters ---------- x : array_like Training vectors, shape (n, d), `dtype` must be float32 and n should be larger than the number of clusters `k`. weights : array_like weight associated to each vector, shape `n` init_centroids : array_like initial set of centroids, shape (n, d) Returns ------- final_obj: float final optimization objective rrz,obj time time_search imbalance_factor nsplitrrr)r r r r1rrr Clusteringrr copy_array_to_vectorravelrtrainrrProgressiveDimClusteringrrvector_float_to_arrayreshaperrVr9atrrsplitr)ryrbweightsinit_centroidsrr1clusncr3rstatsr]st stat_fieldsfields rr Kmeans.train s4  ) 4wwFF{{ 77   4 4a1D)'--ww**>+?+?+A4>>R JJq**g .? "?!) ))ww(( ((+Avvtww?D JJq(1+txx 0//? "**46615$$&+EJJL&9:&9!&9:88e4eVVe45DJJL  5@ @K5GB& &K @  $xx}}q0txx|9c9;4 A s$,I,I1 I;I6 1I;6I;cN[R"USS9nURcS5eURR 5 URR UR5 URR US5up#UR5UR54$)Nrrzshould train before assigningrI)r r rrrrsearchr)ryrbrrs rassign Kmeans.assignHs}  ) 4~~)J+JJ)  t~~&zz  A&wwy!'')##r!) rrr1rrrrrrr)NN) rrrrrr{rrrrrrr!rrrs"@. $::x$r!rcJ[U[RR5$r) isinstance collectionsabcSequenceras r is_sequencerUs a11 22r!c URup#[R"USS9n[U5(a[R"USS9nURU4:Xde[ UR 5S-S-5n[R "X$4SS9n[X#[U5[U5[U5U5 U$X1-S-S-n[R "X$4SS9n[X#U[U5[U5U5 U$)a" Pack a set integers (i, j) where i=0:n and j=0:M into n bitstrings. Output is an uint8 array of size (n, code_size), where code_size is such that at most 7 bits per code are wasted. If nbit is an integer: all entries takes nbit bits. If nbit is an array: entry (i, j) takes nbit[j] bits. rorrH) r r r rresumr&pack_bitstrings_cr)rOnbitrM code_sizebs rpack_bitstringsr Zs 77DA Qg.A4##D8zzaT!!!aA-. HHa^7 3 (4.(1+x{I G HX\a' HHa^7 3!hqk8A; J Hr!c URup4Uc[R"USS9n[U5n[ UR 5S-S-5nXF:de[R "X54SS9n[X5[U5[U5U[U55 U$UnXR-S-S-nXF:de[R "X54SS9n[X5U[U5U[U55 U$)ax Unpack a set integers (i, j) where i=0:n and j=0:M from n bitstrings (encoded as uint8s). Input is an uint8 array of size (n, code_size), where code_size is such that at most 7 bits per code are wasted. Two forms: - when called with (array, M, nbit): there are M entries of size nbit per row - when called with (array, nbits): element (i, j) is encoded in nbits[j] bits rorrr) r r r rrerr&unpack_bitstrings_cr)r  M_or_nbitsrrr r  min_code_sizerOs runpack_bitstringsrus77LA |##Jg> ITXXZ!^12 ))) HHaV7 + (4. QKHQK 1 H A!+ ))) HHaV7 + $ Y  = Hr!)90)rN)i)Nrr)rr)"numpyr faiss.loaderr collections.abcrrr$r'r5r=rClrandrFrQrTrSr^rcrmrgrsrprurrrrrrr rrrrr!rrs $$'0A2   , 8 > : N==@,8$3l0pS$S$t3$ 2( r!