RAG, retrieval-augmented generation, means that a model answers from documents or records fetched at the moment of the question, not from what it memorized in training. Retrieval decides the answer: a model that reads the wrong file answers with confidence from it. A company needs this work when its knowledge is spread over documents, databases and business tools, and when an answer from the wrong source costs more than no answer.
What I build
- Hybrid retrieval. Vector search and full-text search over the same corpus, merged in the database. On LeanManufacturing.mx this is one SQL function on Neon Postgres with pgvector, under a widget that answers from the published documentation. The embeddings are optional: without them the search falls back to the lexical index.
- Retrieval for agents. How an agent locates a document: a search tool, a curated index or both, chosen by measuring which one reaches the right file.
- Migrations, models and analytics. From collections in a document store to a relational schema with keys and constraints, dimensional models that keep history, and the dashboards that read them.
- Sync bridges that keep what they move. Integrations between business systems with persistent storage, duplicate detection and automatic recovery from errors.
Where answers go wrong: the retrieval or the data under it
- The assistant writes a fluent answer and takes it from the wrong document.
- Records about the same customer sit in separate systems with no shared key.
- A sync moves data between systems and deletes it afterwards, so no history is left.
- Duplicates, mixed formats and placeholder dates reach the model as if they were facts.
The first is a retrieval fault, and it can be measured. A common design gives the agent a hand-written index that summarizes every file and tells it what to open. In a pre-registered study on a corpus of prose documents, an agent with that index as its only locator reached the right document on about half of the questions, and an agent with plain search on four out of five. The agent with the index read more files and landed on the wrong one more often. A public replication with a released corpus pointed the same way. The result concerns that kind of index on those corpora, not an index used together with search, so on your corpus the answer is measured, not assumed.
How the work runs
- A frozen question set. We write real questions, each with the document or record that answers it, and freeze the set before anything is tuned.
- Reach before accuracy. For every question I record whether the system reached the right source, apart from whether the answer was correct. Not finding the document and misreading it are different faults.
- The simplest search that passes. Lexical search first, then vectors where they find more. Combining methods is tested, not assumed: in a companion study a better ranker found far more documents, and fusing methods made the results worse.
- The data underneath. Where the sources are not ready I model them: keys and constraints, deduplication, history, and incremental loads that process only what changed.
- Operation. Reindexing after a content change is one authenticated request. Sync jobs get a dry-run mode, switches that turn a destination off without a deployment and retries with backoff.
What you receive
The schema and its migrations, the loading and sync jobs, the index and the retrieval function, the question set with its results, and the dashboard or assistant built on top, all in your database and your cloud accounts.
What this does not include
I do not promise that a vector database fixes wrong answers, and I do not add an index, a layer of summaries or a second search method without a measurement that shows it helps. Retrieval does not repair data that is wrong at the source: duplicates and placeholder values are handled first.
Where the evidence is
For a business in the warranty management industry I built the pipeline that migrates and synchronizes data from MongoDB to PostgreSQL, the relational schema and the analytics dashboard on top. I turned an inherited sync bridge between PropertyWare and ServiceFusion into a persistent warehouse with a dimensional model and full history, while the system stayed in production. LeanManufacturing.mx shows hybrid retrieval on a public site. The retrieval study is published with its corpus, questions, harness and adjudicator.