llm-reliability

Preregistered Jacobian-lens reliability research on Gemma 4: 25k prompts, frozen probes, public traces, a prospective transfer miss, and an answer-identity confound.

Turn Chaos Into Structure. A Type-Safe AI Agent that extracts valid JSON from unstructured data using PydanticAI, FastHTML, and Gemini 2.5.

Interactive Phoenix LiveView demonstrations of the Crucible Framework - showcasing ensemble voting, request hedging, statistical analysis, and more with mock LLMs

Code for the ICML 2026 Main Track Paper "Cognitive Fatigue in Autoregressive Transformers: Formalization and Measurement"

Catches safety constraints and task invariants that an LLM agent's context compaction silently drops, then repairs them. Zero dependencies.

Profile README. AI engineer working on agentic LLM systems, deterministic guardrails, and how models fail in long interactions.

Reference implementation of CAAF — three-pillar agent framework with monotonic convergence.

Public artifact bundle for the preprint 'Lightweight Evaluation and Operational Scorecards for Tool-Using AI Agents'

Three small LLMs, one CPU, no cloud. Rigorous benchmarking, structured output validation, and head-to-head quality scoring via Ollama + FastAPI.

A Python to Prolog pipeline that bounds Elasticsearch LLM errors by grounding ES diagnostics analysis in verified metrics.

TypeScript eval harness for measuring whether Grok answers stay grounded in source evidence

Collection of LLM failure modes used on failmodes.com

Companion project for the TechnologyDig Academy tutorial on building reliable generative programs with Mellea.

Extracts a driver profile from a call transcript behind a validating schema gate, then screens a load board before ranking by effective rate per mile.

When the agent breaks, which layer dropped the ball? Operator-facing failure-attribution taxonomy for AI agent estates: ten-way dictionary, MAST + AgentRx crosswalks, postmortem template. Part of the Spine catalog.

CrucibleFramework: A scientific platform for LLM reliability research on the BEAM

Official implementation of Cascading Hallucination in Agentic RAG: The CHARM Framework for Detection and Mitigation.

Map where your bolted-on AI feature breaks before customers do. A free Claude Code tool: fragility map, reliability score across six dimensions, ranked gaps, and a 30-day plan. Built by a threat-intel practitioner.

Reliability and hallucination mitigation research for tool-augmented legal AI agents using QC-Sentinel verification architecture.

AI agent metacognition skills distilled from David Dunning's Self-Insight (2005): calibration, Dunning–Kruger awareness, the outside view, and feedback discipline for Claude Code, Codex, and compatible AI agents.