01 · SANITY
Structure and schema
Detects empty or truncated answers and validates JSON or YAML formats.
Evaluation · Verification
Do not trust the output. Verify it.
A system for evaluating, scoring and verifying what LLMs produce: mechanical checks first, then an LLM judge, with a gate that regenerates until the answer passes.
MITCLIGate modePython
$ git clone https://github.com/iwebbo/LLMJudge
$ lab verify # judge a document or a RAG answer
# chat --trace | chat --continue
$ _
How it works
A verdict you can audit.
02 · CODE
Syntax and lint
Checks code blocks for syntax errors, unclosed fences and TODO placeholders, with Ruff-based linting.
03 · GROUNDING
Citations and retrieval
Verifies that every [SOURCE N] points to a real, non-empty document and that retrieval was relevant.
04 · JUDGE
LLM audit and gate
An LLM looks for contradictions and gaps. If the score fails, the gate regenerates with the findings.
Every finding rolls up into one verdict: a score from 0.0 to 1.0 and a status of passed, warning or failed.
Features
Chat, verify, repair.
CHAT
Standard streaming
A continuous streaming chat with the LLM.
TRACE
Detailed debugging with --trace
Captures complete logs for every call: generator prompt, judge prompt and temperature.
CONTINUE
Continued conversation
--continue feeds the model with past conversational history.
STATIC
Static judging
Evaluate a written .md document against defined criteria.
RAG
RAG verification
Simulate verification from a question and a local knowledge base (chunks.json).
GATE
Guided regeneration
Retry up to N rounds with the findings until a passing score or the limit is reached.
Every capabilityexpand
Interaction and chat
- Standard streaming — continuous stream chat
- --trace — full logs per call
- --continue — feed past history
Verification and scoring
- lab verify — judge existing content without new generation
- Static judging — a .md document against criteria
- RAG verification — question plus chunks.json
Pipeline
- Structural integrity — sanity and schema
- Code block validation — syntax and lint
- Citation check — source grounding
- Gate mode — iterative repair
- Verdict — score 0.0–1.0 and status
- LLM judge — high-level audit
Operational profiles
Pick the reliability you need.
Trade speed for assurance per task.
| Profile | Parameters | Use case |
|---|---|---|
| Latency | annotate / auto-judge | Real-time responses, live chat |
| Balanced default | gate · T 0.7 · 2 rounds | General purpose, standard usage |
| Quality | gate · T 0.95 · 3 rounds · 2nd judge | Mission-critical, maximum reliability |
Built withPythonCLIRuff
Star or fork on GitHub ↗