TruthBomb

Overview

TruthBomb is an automated fact-checker for Sri Lankan social media. You paste a post (or a screenshot of one) and it extracts the single claim the post is making, looks for evidence in a live archive of news alerts and on trusted news sites, and returns one of four verdicts: True, False, Misleading or Unverified. Each verdict comes with the evidence used, a short explanation in English, Sinhala and Tamil, and a record of every step that led to it.

We built it as a four-person university team. The first version worked end to end, but its decisions were made inside prompts and could not be explained or reproduced. Most of this case study is about the rebuild that fixed that: the same models, rearranged so that code makes the decisions and models only do the work that needs a model.

• 4input languages, including code-mixed Singlish
• 2.3 GBquantized model that runs extraction and local verification on one consumer GPU
• 125benchmark posts: 25 news events, each written five different ways
• 6pipeline steps, each one logged to an audit trail

The problem

False claims about arrests, road accidents, floods, gazette notices and exam results spread on Sri Lankan social media faster than newsrooms can correct them. Checking one by hand means working out what the post actually claims, finding a credible report of the same event, and judging whether the two agree. That is slow, and most posts are not written in a form that standard tools can work with.

The language is the first obstacle. A typical post looks like this:

Train eka derail kara Galle station ta! 😱 5 ta people injured kiyala news dakka! Track eka problem nada enna railway ta?

That is Sinhala grammar typed phonetically in Latin letters, mixed with English words, emoji and a rhetorical question. Search engines and fact-checking models trained on clean English text struggle with it. Posts in Sinhala or Tamil script, or in formal English, need to work just as well.

Timing is the second obstacle. Breaking news reaches the official Telegram channels of Sri Lankan news outlets well before articles are written and indexed by search engines, so a checker that only searches the web misses the newest events, which are the ones most likely to be rumours.

Trust is the third. A bare "False" from a model gives a reader no reason to believe it. A useful answer has to show which sources it relied on, how it weighed them, and has to say so in the reader's own language.

Requirements

From those problems we set out what the system had to do, and the conditions it had to work under.

Architecture

The system has three parts. A browser app collects the post. A FastAPI service runs the fact-checking pipeline, which is a LangGraph state graph. A separate ingestion worker keeps the local evidence store up to date. The API is deliberately thin: it validates the request, runs the graph off the event loop and maps the graph's final state onto the response. Every decision happens inside the graph.

All the steps share one state object: the raw post, its language, the extracted claim, the retrieved documents with their scores, each verifier's output, the final verdict and the audit trail. A step reads what it needs and writes only its own fields. The audit trail is the one field that is merged rather than overwritten, so each step appends a line saying what it did and whether it succeeded.

How it works

Reading the post: a fine-tuned extractor

The first step turns a noisy post into a claim that search can work with. We fine-tuned Phi-3.5-mini-instruct (3.8B parameters) with Unsloth and LoRA on Google Colab, using 4-bit training and Alpaca-style prompts. The training set has 425 examples, each a post in Sinhala, Singlish or code-mixed English paired with the plain English claim it makes:

The adapters were merged into the base model and exported as a 4-bit GGUF file (Q4_K_M, 2.3 GB), which llama.cpp runs with every layer on the GPU. At inference time the model runs at temperature 0 under a GBNF grammar, a formal grammar applied while tokens are generated, so the only output it can produce is {"language": …, "extracted_claim": …} with the language limited to four values. It cannot answer with prose, markdown or broken JSON, so nothing downstream has to clean up its output.

Finding evidence: hybrid retrieval

Retrieval combines two searches over the local store, because each misses things the other finds:

• Semantic search in ChromaDB, using the multilingual paraphrase-multilingual-MiniLM-L12-v2 embedding model, so a claim derived from a Sinhala post can match an English news alert about the same event.
• Keyword search with BM25, which catches exact names, places, numbers and acronyms that embeddings tend to blur.

An early version merged the two lists by hand and gave every BM25 hit a flat score of 1.0, which put dense distances and keyword ranks on incompatible scales. We replaced that with Reciprocal Rank Fusion (RRF, k = 60), which only looks at rank positions. The fused list is then re-weighted by source tier (trusted sources × 1.0, others × 0.6) and recency (from × 1.0 for the last week down to × 0.35 for anything over a year old), and the top four are kept. Every component score is stored with the document, so the interface can show why a source ranked where it did.

In parallel, a teammate's retrieval overhaul added an optional cross-encoder re-ranker and an offline evaluation that reports Recall@5, MRR and nDCG@5 using the benchmark's event IDs as ground truth. That meant changes to retrieval could be measured instead of judged by eye.

Choosing where to verify: score-based routing

The router takes the best semantic match from retrieval, expressed as a similarity between 0 and 1. If it reaches 0.4, the local archive has something close enough to check against, and the claim goes to the local verifier. If it does not, the claim goes to web search.

Verifying: one contract, two verifiers

Both verifiers return the same shape: a stance (SUPPORTED, REFUTED or INSUFFICIENT), a confidence from 0 to 100 and a one-sentence reason. A stance is a verdict on the evidence, not on the claim. Turning stances into a final answer is a separate step.

• The local verifier reuses the GGUF model that is already loaded for extraction, so only one copy sits in VRAM. A second grammar limits its output to the stance shape. If retrieval found nothing, it returns INSUFFICIENT without calling the model at all.
• The web verifier searches DuckDuckGo with site: filters for ten trusted Sri Lankan news and government domains, then asks Gemini 2.5 Flash-Lite, in JSON-schema mode at temperature 0, for the same stance shape plus the sources it used. If the restricted search fails, it falls back to an open search but labels those results UNRESTRICTED, and the prompt tells the model to demand official corroboration for them.

root   ::= "{" ws "\"stance\"" ws ":" ws stance ws "," ws "\"confidence\"" ws ":" ws number ws "," ws "\"reasoning\"" ws ":" ws string ws "}"
stance ::= "\"SUPPORTED\"" | "\"REFUTED\"" | "\"INSUFFICIENT\""
number ::= digit ( digit ( digit )? )?

Because both verifiers keep to this contract, the next step does not need to know where the evidence came from.

Deciding: rules first, model second

The adjudicator maps stances onto the four verdicts. Clear cases are decided by fixed rules; Gemini is called only for borderline confidence or conflicting evidence.

"Misleading" can only come from the Gemini path. That is intentional: deciding that a claim is partly true or deceptively framed takes judgement that a threshold cannot supply. Each verdict records which path produced it (rules or gemini). If Gemini fails on a borderline case, the verdict falls back to the direction of the stance; if it fails on a conflict, the verdict is Unverified.

Explaining: translation and the audit trail

The translator asks Gemini, in JSON-schema mode, for a two-sentence summary of the verdict in English, Sinhala and Tamil. It is best effort by design: if it fails, the response carries an English-only summary and the audit trail records the failure, so a translation problem never blocks a verdict.

The audit trail is what the whole redesign was aiming at. Every step appends a status and a short detail, such as which model extracted the claim and in what language, how many documents were retrieved and how strong the best match was, which verifier ran and what stance it returned, and whether the verdict came from rules or from Gemini. The API returns the trail with the verdict, so anyone can see how an answer was reached.

Keeping the evidence fresh

The ingestion worker connects to Telegram over MTProto using Telethon and polls the channels of Ada Derana, News 1st and Hiru News. Each message is stored with its source, message ID and timestamp, and IDs already in the store are skipped. Running it as its own process means a Telegram outage or a login prompt never holds up a fact-check request.

The interface

The front end is a React and TypeScript app built with Vite, Tailwind and shadcn/ui. Users can paste text or upload a screenshot. Tesseract.js reads the screenshot in the browser using its English, Sinhala and Tamil models, so the image never leaves the device. The result card colour-codes the verdict, shows a confidence bar, switches between the three language summaries on tabs, and lists each piece of evidence with its source tier, timestamp and scores.

How the design evolved

The project began as a fine-tuned model that checked claims about traffic law against the Motor Traffic Act. Generalizing it to any news topic produced a working multi-agent system: an LLM triage agent, a local and a web researcher, and a synthesis agent that wrote the verdict. It demonstrated well, but it was hard to trust. Results could change between runs, failures were silent, and no one could say why a given verdict came out the way it did.

We rebuilt it in five phases. Each phase changed one thing and was tested before the next began:

Results

TruthBomb runs end to end on a laptop with a consumer GPU and a Gemini API key. It takes a post or a screenshot in any of the four input languages and returns a verdict with its evidence, a trilingual explanation and a step-by-step audit trail.

The project includes its own evaluation:

• A 125-post benchmark: 25 news events across 11 categories, including arrests, accidents, weather alerts, court rulings, gazette notices, strikes and exam results. Each event is written in five styles: formal Sinhala, two informal Singlish variants and two code-mixed variants.
• An accuracy run that sends every post through the live API and reports per-class precision and recall and a weighted F1 score.
• A retrieval evaluation that reports Recall@5, MRR and nDCG@5, using the benchmark's event IDs as ground truth, so retrieval can be measured without hand labelling.
• Unit tests for rank fusion, which run without models or a database.

What I learned

• Put the decisions you need to explain in code, not in prompts. Routing and adjudication went from "the model said so" to a threshold you can read, test and change. The models still do the language work, which is what they are good at.
• Constrain model output while it is generated, not afterwards. Grammars for the local model and JSON schemas for Gemini removed a whole class of failures where the model replied in markdown or prose, along with the code that tried to clean that up.
• A shared contract makes components interchangeable. Once both verifiers returned the same stance shape, the adjudicator could be simple rules, and adding a third evidence source would not change it.
• Design the failure path for every step. Each node has a defined fallback and logs it, so a translation outage produces an English-only answer instead of an error page.
• Next time, agree the target architecture before branching. We built two retrieval implementations in parallel on separate branches, and later merges brought back pieces the redesign had removed. Next time I would write down the architecture and the interfaces between steps first, then split the work.
• Keep test data out of training data from the first day. Our benchmark posts were reused as fine-tuning examples, which means the benchmark cannot measure the extractor honestly. I would split off a held-out set before generating any training data.
• Calibrate before you add features. I would run the benchmark and tune the thresholds before anything else. I would also update the keyword index incrementally: at the moment new Telegram posts appear in semantic search straight away but only reach BM25 after a restart.