Skip to content
msgxr
Light theme
Contact
← All Posts
Architecture
18 Jul 2026
10 min read

Eleven Agents for Public-Sector Paperwork: Why We Chose Orchestration Over a Single LLM Call

Contents

The first scenario of the TEKNOFEST 2026 AI Language Agents competition asks for an intelligent agent system to support document handling and official correspondence in Turkish public institutions. The system we built as team AGENTRA TECH consists of eleven specialist agents coordinated by a pure-Python orchestrator. The repository is open under Apache-2.0: teknofest-2026-kamu-evrak-akilli-ajan.

This is not a product tour. I want to explain the architectural decisions, what we gave up in exchange for each, and where the system actually broke. Every code snippet below comes from the repository — abbreviated and annotated, but not invented.

The problem: paperwork is not one "understand this" task#

An incoming petition passes through a chain: read and classify it, extract structured facts, detect what is missing, find the governing legislation, compute urgency and the statutory deadline, summarise it, mask personal data if it will be shared, draft the reply, route it to the correct department, and tell the applicant what happens next.

Four properties make that chain painful. It is multi-step and interdependent — each stage consumes the previous stage's output. It is under deadline pressure — Turkish law imposes different statutory response windows (Freedom of Information Act 4982, Right of Petition Act 3071, Administrative Procedure Act 2577, and the CİMER citizen-request channel), and missing one costs the citizen a right. It is person-dependent — the same document gets read differently by different clerks. And it is privacy-sensitive — masking personal data before sharing is usually done by hand.

On top of that comes a constraint imposed both by the competition rules and by how public institutions actually work: institutional networks are frequently air-gapped, and a citizen's personal data must not leave for a third-party API. That single constraint shaped the entire architecture.

The decision: why not a single LLM call#

The short path was obvious. Feed the document text to a large model, impose a JSON schema, fill twelve fields in one shot. We didn't. Three reasons:

1. It has to work in a world with no LLM at all. The system is offline-first. The core requirements.txt contains no LangChain, no LangGraph, no OpenAI SDK, no torch — only pypdf, streamlit, pandas, altair, rich, pytest and pydantic. Even LLM access is written against stdlib urllib: if an OpenAI-compatible endpoint or a local Ollama is reachable it gets used, and if not, nothing breaks. A single-call architecture makes that impossible — no model, no system.

2. Confidence has to be measurable per step. One call gives you one confidence number. Here, classification confidence and routing confidence are tracked separately, and either one can independently push the case into a human-review queue. In a public-sector setting, the answer to "which of your decisions are you unsure about?" cannot be "roughly 70% overall".

3. The flow has to be able to stop. When a blank scanned page arrives, the correct behaviour is not to invent a document type but to halt and hand off to a human. We wanted that encoded as flow structure, not as a sentence inside a prompt.

What we gave up: flexibility. Each of the eleven agents carries its own rule set, so adding a new document type means coordinated edits across several files rather than one prompt line. In a bounded domain — 8+1 document types, 9 departments, a 15-document legislation corpus — that trade pays off. As the domain widens, maintenance cost grows.

The orchestrator: shared state and three conditional gates#

Agents never call each other. They all read and enrich a single dataclass called AgentState, and the orchestrator runs a conditional workflow over it.

text
  TXT / PDF / image
          |
          v
  (0) OCR / text reading
          |
   +------+------------------------------+
   | GATE 1: >= 30 meaningful chars?     |-- no --> flow halts
   +------+------------------------------+           type = "unknown"
          | yes                                      human review requested
   +------+------------------------------+
   | GATE 2: is the text Turkish?        |-- no --> drafting SKIPPED
   +------+------------------------------+           (analysis continues)
          | yes
          v
  TASK 1   (1) Classification --[GATE 3a: conf < 0.6 -> human review]
           (2) Info extraction
           (3) Missing-field detection
           (4) Legislation RAG
           (5) Triage / statutory deadline
           (6) Summarisation
           (7) Personal-data masking
          |
  TASK 2   (8) Draft + format validation
           (9) Department routing --[GATE 3b: conf < 0.6 -> human review]
          (10) User notification
          |
          v
  result dict with 25+ keys + per-step timings + confidence trace

The gates live in code, not in a prompt. Thresholds are defined in one place:

python
# src/agents/orchestrator.py — conditional gate thresholds
_MIN_ANLAMLI_KARAKTER = 30        # Gate 1: below this, text counts as unreadable
_INSAN_ONAYI_GUVEN_ESIGI = 0.6    # Gate 3a/3b: below this, human review
_MAX_GIRDI_KARAKTER = 200_000     # DoS bound (CWE-400 / OWASP LLM04)

def _metin_okunabilir_mi(self) -> bool:
    """Meaningful char = letter or digit; whitespace/punctuation excluded."""
    anlamli = sum(1 for ch in self.state.raw_text if ch.isalnum())
    return anlamli >= _MIN_ANLAMLI_KARAKTER

def _run_workflow(self, mode: str) -> dict:
    self._apply_girdi_siniri()
    try:
        # GATE 1 — never fabricate output for empty/too-short text
        metin_okunabilir = self._metin_okunabilir_mi()
        if not metin_okunabilir:
            self._uygula_bos_metin_kapisi(mode)

        # GATE 2 — only meaningful once the text is readable
        metin_turkce = self._metin_turkce_mi() if metin_okunabilir else True

        if mode in ("full", "classify") and metin_okunabilir:
            self._run_step("classification", "Evrak sınıflandırma")
            self._record_confidence("classification",
                                    self.state.classification.get("guven"))
            self._degerlendir_siniflandirma_guveni()      # GATE 3a
            self._run_step("info_extraction", "Bilgi çıkarma")
            self._run_step("legislation", "Mevzuat eşleştirme")
            # ... triage, summarisation, masking

        if mode in ("full", "draft"):
            if metin_okunabilir and metin_turkce:
                self._run_step("draft_writer", "Yazı taslağı oluşturma")
            elif metin_okunabilir:
                self._skip_step("draft_writer", "Yazı taslağı oluşturma",
                                "evrak dili Türkçe görünmüyor")
            # ... routing (GATE 3b), user_info
    except Exception as e:
        self.state.errors.append(str(e))
    return self._compile_results()

Skipped steps do not vanish quietly. _skip_step records every skip with its reason into processing_steps, and _run_step times every executed step with time.perf_counter(). The output dict shows which steps ran, how long they took, and why any were skipped.

Library choices: no heavy dependency in the core#

The retrieval core is pure-Python BM25-Okapi. We didn't install rank_bm25 or scikit-learn; a hundred-line file was enough:

python
# src/utils/bm25.py — IDF: non-negative variant
self.idf = {
    token: math.log((self.corpus_size - n + 0.5) / (n + 0.5) + 1.0)
    for token, n in df.items()
}

def get_scores(self, query_tokens):
    scores = [0.0] * self.corpus_size
    for token in query_tokens:
        idf = self.idf.get(token)
        if idf is None:
            continue
        for i, freqs in enumerate(self.doc_freqs):
            tf = freqs.get(token, 0)
            if tf == 0:
                continue
            norm = self.k1 * (1 - self.b + self.b * self.doc_lens[i] / self.avgdl)
            scores[i] += idf * tf * (self.k1 + 1) / (tf + norm)
    return scores

Tokenisation is Turkish-specific: it lowercases through a turkish_lower helper (the dotted/dotless i problem), keeps circumflexed vowels that appear in formal legal Turkish, and drops stopwords.

Classification follows the same philosophy — a pure-Python Multinomial Naive Bayes over TF-IDF plus character 3-grams, mixed with a rule-based scorer:

python
# src/agents/classification_agent.py
_ENSEMBLE_KURAL_AGIRLIGI = 0.6   # regulation-grounded structural anchors
_ENSEMBLE_ML_AGIRLIGI = 0.4      # lexical patterns learned on a small corpus

birlesik = {
    tur: (_ENSEMBLE_KURAL_AGIRLIGI * kural_olasiliklar.get(tur, 0.0)
          + _ENSEMBLE_ML_AGIRLIGI * ml_olasiliklar.get(tur, 0.0))
    for tur in EVRAK_TURLERI
}

The reasoning behind the weights is written into the source comments. The rule layer encodes structural anchors from the Official Correspondence Regulation — the "T.C." header, the reference and file-number blocks, salutation patterns — so it generalises to unseen documents and earns the majority weight. The Naive Bayes model was trained on 52 documents: high variance, minority weight. We chose an arithmetic mixture specifically so that no single component can contribute more than its own weight; an overconfident model emitting probability 1.0 cannot decide the outcome alone.

Optional layers exist — turkish-e5-large for dense semantic search and bge-reranker-v2-m3 for reranking, both via sentence-transformers — and both are off by default, because downloading model weights on first launch would break the offline-first promise.

The trade-off: absolute similarity instead of relative#

Most RAG demos normalise scores by dividing by the best hit. The result always reads "98% similar", no matter how weak the match actually is. We couldn't do that, because the drafting agent thresholds that score to decide whether to cite a legal provision: only matches with similarity ≥0.6 get cited in the body of the letter. An inflated score converts directly into a wrong statutory citation.

So similarity is scaled against an absolute saturation point: min(1, weighted_score / (1.5 × total_idf)). The 1.5 coefficient comes from BM25 theory — in an average-length passage a single occurrence (tf=1) contributes exactly one idf unit, and the contribution saturates toward (k1+1)=2.5·idf. The coefficient ties "full similarity" to the query terms being used centrally in a passage (tf≈2–3) rather than mentioned once. If even the best match falls below 0.5, the whole result list comes back explicitly flagged as a weak match, and the draft invents no article number — it falls back to the phrase "the relevant provisions of the applicable legislation".

The cost is that scores look modest in the UI. A 0.83 could have been displayed as 1.00 under relative normalisation. Honest scaling is a bad marketing decision and the only defensible engineering one.

Where it broke: the adversarial set#

On the 52-document development set everything looked fine. Then we built a deliberately hostile held-out set that development never touched: malformed file-number blocks, broken reference chains, dates written out in words with no digits, and documents dense with personal data that never use the words "personal data". Missing-field detection dropped to 0.667 micro-F1; legislation hit@3 fell to 0.875.

There were three distinct root causes, and none of them was fixable by "train the model a bit more".

Broken reference chains. The extractor was treating prose mentions in the body — the Turkish equivalent of "as recorded in reference (b) of your letter" — as an actual reference field. It therefore believed a reference block existed and never reported it as missing. The fix was structural: in official correspondence the reference label is always written as a colon-terminated field heading.

python
# src/agents/info_extraction_agent.py
# The colon is MANDATORY. Prose mentions inside the body are NOT the field
# label and do not count as a reference block — this is how a broken
# reference chain gets detected structurally.
_ILGI_SATIRI = re.compile(r"^\s*[İI]lgi\s*:\s*(.*)$")
_ILGI_MADDE  = re.compile(r"^\s*([a-zçğıöşü])\)\s*(.+)$")

Dates written out in words. Documents with no numeric date at all ("on the fifteenth day of June") yielded no document date, which meant no statutory deadline could be computed either. A verbal-date resolver was added.

Privacy law without the vocabulary. Turkish data protection law (KVKK, Act 6698) applies whenever a document contains a validated national ID number or IBAN, even if the words "personal" or "KVKK" never appear. Because BM25 topic activation depends on trigger words, 6698 was never suggested for exactly those documents. The fix is a data-signal bridge: the presence of a checksum-validated ID or IBAN, as confirmed by the extraction agent, feeds the legislation suggestion directly. Crucially, the injected suggestion does not claim a textual match — it is reported with similarity 0.55, deliberately below the 0.6 citation threshold, and tagged with eklenme_nedeni="kvkk_veri_sinyali". It shows up as advice in the list without forcing a citation into the draft.

The binding constraint on all three fixes: no file-specific memorisation. If you write a rule aimed at one document in a held-out set, the set stops being held out. So the fixes were written as general rules, verified to cause zero regression on the development and earlier held-out sets, and only then measured on a completely untouched new adversarial set.

Threshold calibration: 0.5 or 0.15?#

The legislation agent has a corrective retrieval loop: if the best similarity from the first search falls below a threshold, the query gets expanded with the procedural vocabulary of that document type and searched once more.

python
# src/agents/legislation_agent.py
ilk_benzerlik = matches[0]["benzerlik"] if matches else 0.0
genisletme = TUR_SORGU_GENISLETME.get(evrak_turu, [])
if ilk_benzerlik < DUZELTME_ESIGI and genisletme:
    genis_sorgu = f"{query_text} {' '.join(genisletme)}"
    yeni_matches, yeni_yontem = self._hibrit_ara(
        genis_sorgu, evrak_turu, duzeltilmis=True
    )
    yeni_benzerlik = yeni_matches[0]["benzerlik"] if yeni_matches else 0.0
    if yeni_benzerlik > ilk_benzerlik:      # adopt ONLY if it improves
        matches, yontem = yeni_matches, yeni_yontem

My instinct was to set the trigger at 0.5, the same value as the weak-match flag. Measurement killed that idea. At 0.5 the loop fired on 33 of the 35 documents in the then-current development set; the injected procedural terms pushed domain-specific statutes out of the top three, and hit@3 fell from 0.943 to 0.914. At 0.15 hit@3 stayed at 0.943 and the loop fired on only 2 borderline documents. The threshold moved to 0.15.

The lesson generalises: "if the system is unsure, work harder" is only half a policy. You also have to calibrate when the rescue mechanism must stay out of the way, or you damage the well-behaved majority to help the tail. The weak-match flag (0.5) and the correction trigger (0.15) are deliberately two different numbers — one exists for transparency, the other as a safety net.

Measurement: what the numbers do and don't say#

The data/processed/eval_report*.json files are produced only by scripts/evaluate.py, and each report embeds a provenance stamp: git commit SHA, platform, requirements hash, dataset content hash. On the untouched adversarial set (16 documents, fully offline mode, llm.kullanilabilir: false) the measured values were: classification accuracy 0.938, macro-F1 0.933, department routing 0.938, missing-field micro-F1 1.000, legislation hit@3 0.938.

It matters to be precise about what those numbers are:

  • The data is synthetic. No real public-sector documents were used — both the competition rules and data protection law forbid it. The fictional ID numbers pass the official checksum but belong to no real person.
  • N is small. Each held-out set has 16 documents, two per class. An accuracy of 0.938 means 15 out of 16. A single error moves the metric by six points.
  • These are our own measurements. They are the output of our evaluation harness, not a jury's assessment. The preliminary evaluation presentation took place on 12 July 2026; no result, score or ranking is recorded in the repository. None of the numbers on this page come from the competition jury.
  • The failure is visible. The single v4 error is in one class: one of two cover letters was read as a reply letter (recall 0.5). The confusion matrix sits in the report; nothing is hidden.

On the testing side, pytest tests/ as measured on 16 July 2026 reports 632 tests across 42 modules, with one skipped because it depends on an optional PDF library being installed. CI runs compile checks, the full suite, and a 5-document evaluation smoke test on a Python 3.9 / 3.12 matrix for every push. For throughput, the repository's benchmark report gives roughly 88 documents per second and 11.3 ms median latency on a single core offline — those figures describe the rule-based path only; with an LLM in the loop, latency belongs entirely to the provider.

Limits and where it stands#

There is no hosted demo; the system runs locally. On Windows calistir.bat and on Linux/macOS ./baslat.sh set up a virtualenv and open the Streamlit dashboard. python -m src.api starts a zero-dependency JSON API built on stdlib http.server, and python -m src.mcp_server starts a Model Context Protocol server so the whole pipeline can be called as a tool by another agent. There is also a Dockerfile based on python:3.12-slim.

The limits I know about: the legislation corpus is 15 documents, far below what a real institution needs; integration with the national standard file plan exists only as a design note; electronic document management system integration is still at the vision stage — the correspondence metadata is generated as a draft but has never been tested against a live system. Religious holidays are not in the statutory-deadline calendar, so the computed deadline is a deliberately conservative "no later than" estimate.

The most transferable lesson from this project isn't architectural, it's about measurement discipline: making a system's uncertainty visible is worth more than one more point of accuracy. The three conditional gates, the absolute similarity scale, and the human-review queue are the same idea implemented in three different places.

Sources#

Share
Let’s build something together
A project, a collaboration, or just an idea — my inbox is open. I reply quickly.
← Previous
VARIANT-GNN: TEKNOFEST 2026 AI in Health — Technical Deep Dive
Next →
On-Device RAG with Foundry Local: My Microsoft AI Innovators Project
← All PostsMuhammed Sina Gün