teknofest-2026-kamu-evrak-akilli-ajan

Problem

Inside a public institution a document runs through a chain: read it, classify it, analyse its content, check the legislation, draft a reply, route it to a unit, archive it. That chain has four concrete failure modes: the steps are repetitive and manual; missing a statutory deadline (laws 4982, 3071, 2577, or the CİMER process) costs citizens their rights; the same document gets interpreted differently by different staff; and masking personal data before sharing is done by hand, which is a data-protection risk. On top of that, institutional networks are frequently air-gapped — so a "just call a cloud LLM" design is disqualified before it starts.

Approach

As a team we modelled the chain as 11 specialist agents plus an orchestrator. Each agent owns one job; the orchestrator runs them over a shared state object (AgentState, a plain dataclass) in a conditional flow, timing each step and tracking its confidence. The flow is not a straight loop — it has three gates: gate 1 asks whether the text is readable at all (under 30 meaningful characters the process stops early), gate 2 whether it is Turkish (if not, drafting is skipped but analysis continues), gates 3a/3b whether classification and routing confidence clear 0.6 (below that the decision drops into a human-approval queue with its reasoning attached). The core intelligence is not an LLM: it is calibrated weighted rule scoring, a legislation RAG built on pure-Python BM25-Okapi, and a pure-Python Multinomial Naive Bayes, with classification blending the last two as 0.6 rules + 0.4 ML. An LLM, when present, only engages on low-confidence decisions. Layered on top is a measurement stack: ECE calibration, split conformal prediction (α=0.1), selective prediction via Chow's reject rule, metamorphic robustness and a McNemar significance test.

Decisions and trade-offs

• 01We wrote our own orchestrator on dataclass + time instead of using LangChain/LangGraph, because a public-sector setting demanded a small security surface, readable and testable steps, no dependency bloat and a guarantee of offline operation; the alternative was an off-the-shelf agent framework, which would have put a heavy dependency in the core and a flow nobody reads.
• 02We made the system offline-first with the LLM optional, because institutional networks are often air-gapped and personal data cannot travel to a third-party API; the alternative was making the LLM a hard dependency, which would have turned model access into a single point of failure. The price of this decision was having to make the rule layer genuinely good.
• 03We report legislation similarity on an absolute scale, min(1, score / 1.5·Σidf), rather than relative normalisation, because relative normalisation shows "high similarity" for every query and misleads an evaluator; on an absolute scale a weak match (<0.5) is flagged as weak and never cited in the draft body (citation threshold 0.6). The easy-looking relative score was the alternative, and we deliberately rejected it.
• 04We wrote the statistical model as a pure-Python Multinomial Naive Bayes rather than reaching for sklearn/numpy, because in a 52-document small-data regime NB is strong and transparent, its weights are readable in JSON, and it ensembles 0.6/0.4 with the rule classifier; alongside word tokens we added character 3-grams because Turkish is agglutinative — "başvuru/başvurunuz/başvurumun" are three separate features at word level but share stems as 3-grams. We also added length normalisation following Rennie et al. 2003, because NB's independence assumption double-counts dependent features and saturates probabilities toward 0/1.
• 05We made the PII regex patterns a single source of truth: they are defined in info_extraction and imported by the data-protection masking agent. If extraction and masking drift apart in their patterns, you get the worst outcome — personal data that was extracted but not masked. The alternative, each agent keeping its own patterns, made that class of leak inevitable.

Outcome

The repo contains generated, never-hand-edited measurement files, and I'm giving their framing exactly as recorded. Task metrics (README evaluation table, backed by eval_report*.json): development set (52 documents) classification 1.000 / unit routing 0.962 / missing-information micro-F1 1.000 / legislation hit@3 0.962; held-out set (16) 1.000 / 1.000 / 1.000 / 0.875; held-out v2 (16) 1.000 / 0.938 / 1.000 / 0.750; clean adversarial v4 (16) 0.938 / 0.938 / 1.000 / 0.938. The v4 report (data/processed/eval_report_heldout_v4.json) records accuracy 0.9375 and macro-F1 0.9333, with a run seal of git commit 869fe34, Python 3.12.10, Windows 11, and LLM backend "offline" and unavailable — these scores were obtained with no LLM at all, in fully rule-based mode. Performance (data/processed/benchmark_raporu.json, 2026-07-12, Darwin arm64, 8 cores, Python 3.9.6, 84 documents × 5 repeats = 420 samples): 88.1 documents/second throughput, 11.28 ms median latency, 14.19 ms p95, 15.73 ms p99, 0.21 MB peak memory (tracemalloc), and per-document time of 11.25/11.24/11.26 ms across 1x/5x/10x — linear scaling. Robustness (data/processed/dayaniklilik_raporu.json, seed 1234, 52 documents, 260 variants): type invariance 0.9923, unit invariance 0.9692, with diacritic loss the most fragile corruption (type 0.9615, unit 0.8462), flipping two files from cevap_yazisi to ust_yazi. The technical report states 632 unit and integration tests were green as of 2026-07-16. The result most worth telling, though, is a regression: when the development set grew from 35 to 52 documents on 2026-07-12, routing accuracy fell from 1.000 to 0.962 — and because the two errors were genuine functional ambiguity rather than mislabelling, they were NOT patched with file-specific rules, they were reported as-is. The same discipline holds on adversarial v3: two remaining limitations (report vocabulary pushing law 5018 out of the top three, and the second-person "yazınız" signal blurring reply-vs-cover-letter) were observed again in v4 and left honestly open. The competition evaluation has not happened yet; none of the numbers on this page is a competition result.

Stack