Search

OmniData search fuses two complementary retrieval methods — vector similarity and full-text keyword search — into a single ranked result set using Reciprocal Rank Fusion (RRF). This gives you the semantic understanding of embeddings with the precision of keyword matching.

All search queries run against index.db inside the .omnidata bundle. This is the database that holds resources, chunks, embeddings, and the FTS5 index.

Two retrieval paths

Vector search (semantic)

Vector search finds chunks whose meaning is similar to the query, even when they share no exact words. The query is embedded using the same model that produced the stored embeddings, then ranked by cosine similarity against every chunk’s embedding vector.

Embeddings are stored as little-endian float32 BLOBs in the omnidata_chunks table within index.db. Cosine similarity is computed at query time — no external vector database required.

FTS5 search (keyword)

Full-text search uses SQLite’s FTS5 extension with BM25 ranking. The fts_chunks virtual table in index.db indexes all chunk text. FTS5 excels at exact phrase matching, rare terms, and proper nouns — cases where semantic similarity alone may not surface the right result.

Reciprocal Rank Fusion (RRF)

RRF combines the two ranked lists into one. The formula:

score = sum(1 / (k + rank))

Where k is a constant (default 60) that controls how much weight is given to top-ranked results versus lower-ranked ones. Each retrieval path contributes independently to the final score.

The paths are weighted:

• Vector search weight: 0.6 — Semantic similarity is the primary signal
• FTS5 weight: 0.4 — Keyword matching serves as a precision boost

A chunk that ranks highly in both paths will score significantly higher than one that ranks well in only one. This fusion handles the failure modes of each method: vector search recovers from vocabulary mismatch, FTS5 recovers from embedding blind spots.

The search flow

1. Embed the query using the same model as the stored chunks
2. Run vector search — cosine similarity against omnidata_chunks.embedding in index.db, rank results
3. Run FTS5 search — BM25 against fts_chunks in index.db, rank results
4. Fuse with RRF — combine both ranked lists using the weighted formula
5. Optional: PEFM reranking — if enabled, a Post-Embedding Fusion Model re-scores the top-N results using a cross-encoder for higher precision

PEFM reranking (optional)

PEFM (Post-Embedding Fusion Model) is an optional second pass. After RRF produces a fused ranking, PEFM takes the top candidates and re-scores them using a cross-encoder that reads both the query and the chunk text together. This is more computationally expensive than embedding similarity but produces more accurate relevance judgments.

PEFM is off by default. Enable it when precision matters more than latency — for example, when an agent needs to select the single most relevant chunk to include in a prompt.

Why RRF over alternatives?

RRF is rank-based, not score-based. This matters because vector similarity scores and BM25 scores are on incompatible scales. RRF sidesteps normalization entirely by working only with ordinal positions. It is simple, effective, and requires no training data.