rag-assistant

Problem

The usual way to ask questions about your documents is to ship those documents to a cloud API. I wanted the opposite: a document Q&A assistant that runs with no internet, no API key and no cloud after the initial model download, and that names the source file behind every answer. A second constraint was teachability — the repo also carries a six-week plan (docs/one_month_plan.md) and weekly exercises (examples/), so the code had to work as a running system and as course material at the same time.

Approach

I built five layers with hard file boundaries: interface (ui_streamlit.py / main.py) → application (generation.answer_query) → retrieval (retrieval.get_top_chunks) → data (SQLite rag.db via db.py) → AI (foundry_client). There are two flows. Ingest runs once: read data/*.md, split it paragraph-first through chunk_text(), embed each piece and store it in SQLite as JSON text — the whole ingest runs inside a single transaction so a crash halfway through cannot leave a half-built database. The query flow embeds the question, scores cosine similarity against every stored vector, takes the top 3 chunks, and only then builds a prompt for the on-device model. If the relevance gate fails, the model is never called at all — the cheapest defence against a fabricated answer is not asking for one.

Decisions and trade-offs

• 01I picked brute-force cosine similarity over a vector database, because for a 5-10 document knowledge base comparing the query vector against every stored vector in Python is fast enough and needs zero extra dependencies; the alternative was a dedicated vector DB or a SQLite vector extension, which the repo notes place beyond a few thousand chunks.
• 02I made one file the only thing that touches the SDK (foundry_client.py), because the Foundry Local SDK's embedding response shape varies between versions and I wanted that absorbed in a single _extract_embedding function; the alternative was importing the SDK in each module, which would have let one version bump break four files at once.
• 03I settled on qwen2.5-1.5b for chat, because per the repo notes qwen2.5-0.5b was faster but too weak in Turkish, while phi-3.5-mini took roughly two minutes per answer on CPU; those were the alternatives, and 1.5B is where speed and quality met. Both models load once at startup, so the first question is as fast as the rest.
• 04I merge short pieces into their neighbour (MIN_CHARS=150), because a short text like "## Heading" or a single code line gets a very focused embedding that ranks high while carrying no information; the alternative was plain fixed-window splitting, which filled the top retrieval slot with headings.
• 05I treat the model saying "I don't know" differently from the model parroting its instructions: an instruction echo (ECHO_MARKERS) is deemed untrustworthy and replaced with the best passage, whereas a genuine refusal (REFUSAL_MARKERS) is respected and normalised to one fallback sentence — only when retrieval is very strong (STRONG_SCORE=0.60) is the passage shown with its source instead. The alternative was overwriting every unsatisfying answer with the top chunk, which would have destroyed legitimate "I don't have that" answers too.

Outcome

There is a recorded measurement in the repo: eval/results.md, run of 2026-07-27 00:28 — 22 of 26 test questions passed, 4 failed, averaging 44.53 seconds per question. All four unanswerable questions correctly fell through to the "not in my documents" fallback, and all four edge cases (empty input, single word, very general, very long question) were handled without crashing. The nature of the four failures matters: they are keyword-matching failures; the fourth is a real defect — on question 15 the answer was in the documents but the system returned its fallback (the README lists this as a separate weakness) — the model describes hardware acceleration without saying "GPU", explains the threshold without quoting "0.45". So this is a keyword check, not human grading; the README lists exactly this as a known weak spot and proposes either multiple acceptable keywords per question or grading with a second model pass. The hardware behind the run is not recorded, so I won't turn 44.53 seconds into a hardware claim.

Stack