LLM Model Comparison & Evaluation Platform (GPT-4.1 vs Gemini 2.5)

An end-to-end evaluation platform that benchmarks OpenAI GPT-4.1 against Google Gemini 2.5-flash on large-scale generation of CEFR-leveled (A1–C2) English practice sentences from real lesson summaries — built for an online language-learning marketplace to move model and prompt selection from opinion-driven to metric-driven.

The production runs processed ~1,000 CEFR-annotated lessons, producing ~2,000 structured result files (one Gemini + one OpenAI result per lesson) with full token, latency, and quality instrumentation.

Overview

• Dual-provider generation: both models generate sentences for every lesson in parallel, each with its own prompt template.
• 8-metric evaluation harness: rule-based checks (correctness/structure, repetitiveness, length compliance, length distribution) plus offline quantitative NLP metrics (TF-IDF topic relevance, TTR lexical diversity, POS-pattern variety) plus operational metrics (latency, token usage/cost).
• Cost accounting: per-call token metering for input / output / total tokens — including Gemini thinking tokens, which turned out to be ~12% of all Gemini tokens in the final production run and are invisible without explicit instrumentation.
• Reliability at scale: file-system resume/checkpointing, exponential-backoff retries, per-provider rate limiting, and truncated-JSON repair, so 1,000-lesson batches survive interruptions with zero duplicate API spend.
• CEFR-first analysis: every quality metric is aggregated per CEFR level and per model, because a model that looks good on average but fails at A1 or C1 is not shippable for an education product.

Architecture

flowchart TD
    DS[("Lesson dataset<br/>(semicolon-delimited CSV,<br/>CEFR-annotated)")] --> ORCH["Orchestrator<br/>src/app_comparison.py"]
    ORCH --> CKPT{"Resume check:<br/>result JSON already on disk?"}
    CKPT -- "yes: skip index" --> ORCH
    CKPT -- "no" --> PAR["Parallel dual-model generation<br/>ThreadPoolExecutor(max_workers=2)"]
    PAR --> GEM["GeminiModel<br/>gemini-2.5-flash<br/>GenAI SDK (thinking_config) →<br/>Vertex AI fallback"]
    PAR --> OAI["OpenAIModel<br/>gpt-4.1<br/>OpenAI SDK"]
    GEM --> ACC["Token & latency accounting<br/>input / output / thinking / total tokens,<br/>latency_seconds per call"]
    OAI --> ACC
    ACC --> REP["JSON repair & parsing<br/>fix_truncated_json()"]
    REP --> EVAL["8-metric evaluation harness<br/>TF-IDF relevance · TTR diversity ·<br/>POS-pattern variety · correctness ·<br/>repetitiveness · structure · length ·<br/>latency & cost"]
    EVAL --> OUT[("Per-lesson result JSON<br/>gemini_result_*.json /<br/>openai_result_*.json")]
    OUT --> RPT["Reporting<br/>CSV comparisons, summary .txt,<br/>detailed_results JSON"]
    OUT --> QUANT["Offline quantitative analysis<br/>quantitative_compare/scripts/llm_analysis.py<br/>(per-CEFR × per-model aggregates)"]

Loading

Key engineering highlights

• Provider-agnostic model layer — GeminiModel and OpenAIModel share one interface, so the orchestrator never touches SDK specifics; adding a third provider is a new class, not a rewrite.
• Dual SDK path for Gemini — prefers the Google GenAI SDK (enables thinking_config with a 1,024-token thinking budget) and falls back to the Vertex AI SDK automatically.
• Thinking-token accounting — Gemini's internal reasoning tokens are extracted from usage_metadata with defensive field-name fallbacks and tracked as a separate cost dimension.
• Resume/checkpoint by construction — the presence of paired gemini_result_*.json / openai_result_*.json files is the source of truth; restarts skip completed indices and merge prior results into final statistics.
• Truncated-JSON repair — both providers occasionally emit JSON inside markdown fences or truncate at token limits; fix_truncated_json() closes incomplete structures instead of burning re-run tokens.
• Prompt engineering across five versions — the V4 Gemini prompt encodes nine hard constraints (zero vocabulary repetition, sentence independence, no lesson/meta references, natural colloquial register, CEFR-aligned lexical and structural variety), with metrics tracked per prompt version.
• Prompt-caching experiment — cache_compare/app_gemini_cache.py measures explicit Gemini context caching (static instruction block cached via caches.create(), per-lesson variables passed as contents) against a no-cache baseline, comparing latency and cached_content_token_count.
• Detailed cost methodology — see LLM_TOKEN_AND_COST_EVALUATION.md for the full token-accounting design and production-run numbers (~12M tokens across the dual-model 997-lesson run).

Tech stack

Layer	Technology
Language	Python 3.8+
LLM — Google	google-cloud-aiplatform (Vertex AI), google-generativeai (GenAI SDK), gemini-2.5-flash
LLM — OpenAI	openai SDK, gpt-4.1
NLP / metrics	NLTK (tokenization, POS tagging), scikit-learn (TF-IDF, cosine similarity), NumPy, pandas
Concurrency	concurrent.futures.ThreadPoolExecutor (parallel dual-model generation per lesson)
Config	.env + shell setup scripts, GCP Application Default Credentials

Repository structure

├── src/
│   ├── app_comparison.py            # Orchestrator: load → generate (parallel) → evaluate → persist
│   ├── models/
│   │   ├── gemini_model.py          # Gemini 2.5-flash client (GenAI SDK / Vertex fallback, thinking tokens)
│   │   └── openai_model.py          # GPT-4.1 client (token/latency tracking, retries)
│   ├── utils/
│   │   ├── env_loader.py            # .env loading
│   │   ├── dataset_loader.py        # Semicolon-delimited CSV loader, CEFR normalization
│   │   ├── prompt_loader.py         # Template loading + placeholder substitution
│   │   └── json_utils.py            # fix_truncated_json() repair
│   ├── evaluation/
│   │   └── evaluator.py             # Dynamic loading + execution of rule-based metric scripts
│   └── reporting/
│       ├── reporter.py              # Summary .txt, comparison CSV, detailed JSON
│       └── status_reporter.py       # Per-call console status (tokens, latency)
├── quantitative_compare/
│   └── scripts/
│       ├── llm_analysis.py          # Offline NLP metrics: TF-IDF, TTR, POS variety (per CEFR × model)
│       └── run_analysis.py          # Batch analysis runner
├── cache_compare/
│   └── app_gemini_cache.py          # Gemini explicit prompt-caching experiment (baseline vs cached)
├── prompts/                         # Gemini prompt iterations (original, improved, improved_v4)
├── eval_scripts_original/           # Client metric scripts excluded — see its README for the interface
├── sample_data/
│   └── dataset_sample.csv           # Synthetic 8-row sample in the original schema
├── scripts/                         # setup_gcp.sh, setup_api_keys.sh, check_env.sh, run_comparison.sh
├── examples_per_level.txt           # CEFR A1–C1 reference sentence examples
├── LLM_TOKEN_AND_COST_EVALUATION.md # Token accounting & cost methodology deep-dive
├── .env.example
└── requirements.txt

Setup

pip install -r requirements.txt
pip install -r quantitative_compare/scripts/analysis_requirements.txt

# Configure credentials
cp .env.example .env       # then fill in GOOGLE_CLOUD_PROJECT and OPENAI_API_KEY
./scripts/setup_gcp.sh     # gcloud auth + enable Vertex AI APIs (for Gemini)
./scripts/check_env.sh     # verify environment

Two pieces are intentionally not shipped (client IP) and must be supplied to run end-to-end:

1. OpenAI baseline prompt — the orchestrator loads prompts/main_prompt.txt as the GPT-4.1 ground-truth prompt. Provide your own template using the same XML-style placeholders (<topic>, <cefr_level>, <n_sentences>, <max_words>, <lesson_summary>) — see the included Gemini prompts for the format.
2. Rule-based metric scripts — see eval_scripts_original/README.md for the exact perform_eval(run_data, example) -> dict interface the evaluation runner expects.

Run

# Smoke test the setup
python src/app_comparison.py --test-only

# Small comparison on the synthetic sample dataset
python src/app_comparison.py --samples 3 --dataset sample_data/dataset_sample.csv

# Choose a Gemini prompt version
python src/app_comparison.py --samples 8 --prompt prompts/gemini_improved_v4.txt

# Offline quantitative analysis over a results folder
cd quantitative_compare/scripts
python3 llm_analysis.py \
    --json-folder ../../sentences_gen_results/gemini_improved_v4_results \
    --output-dir ../results/my_run \
    --run-name my_run

# Gemini prompt-caching experiment
python cache_compare/app_gemini_cache.py --samples 10

Results land in sentences_gen_results/{prompt_name}_results/ as paired per-lesson JSON files plus a model_comparison_summary_*.txt with latency and token statistics; analysis CSVs land in quantitative_compare/results/.

Note

Sanitized portfolio version of a production evaluation platform built during a client engagement at TELUS Digital for an online language-learning marketplace. Client identifiers, proprietary prompts, real lesson data, and credentials have been removed; a synthetic sample dataset (sample_data/dataset_sample.csv) is provided in the original schema. Model names, configuration, methodology, and code architecture are preserved as built.