l1-gec

L1-Transfer-Aware Grammatical Error Correction — A Multi-Architecture Research Framework

Note on badges: The CI badge points to the current GitHub repository. It will display once GitHub Actions is enabled for this repository.

---

Table of Contents

• Overview
• Why This Project
• Architecture
• Pipeline Workflow
• Repository Structure
• Installation
• Verification Workflow
• Quickstart
• Development Workflow
• Testing
• Project Roadmap
• Research Design
• Contributing
• License
• Citation
• Acknowledgments

---

Overview

l1-gec evaluates and adapts Grammatical Error Correction (GEC) systems with respect to L1-transfer error patterns — the systematic error tendencies a learner's native language produces in their English writing. Off-the-shelf GEC tools (Grammarly, GPT-family models, generic T5-based correctors) are trained on error distributions dominated by a narrow set of L1 backgrounds and do not explicitly model transfer effects for underrepresented populations.

This repository implements a multi-architecture comparative study for Turkish-L1 English learners — comparing a fine-tuned transformer, a GEC-specific edit-tagger, an instruction-tuned LLM, and a commercial baseline against gold-standard ERRANT-annotated error categories — and is architected for extension to additional L1 populations (Persian, Azerbaijani) in future work.

Full research motivation, formal hypotheses, and methodology: docs/RESEARCH_SPEC.md.

Why This Project

Alisoy (2025) showed that a GEC model fine-tuned on L1-specific data outperforms generic tools for Azerbaijani EFL learners. No equivalent study exists for Turkish, despite its long-standing presence in learner corpora (FCE, TOEFL11). l1-gec fills that specific, verifiable gap — and does so as reusable infrastructure rather than a single-use script, so that extending to a new L1 population is a data-layer change, not a pipeline rewrite (see ADR 0004).

Architecture

flowchart TB
    subgraph Config["Configuration Layer (versioned YAML — ADR 0004)"]
        L1P["L1 Profiles<br/>(turkish.yaml / persian.yaml / azerbaijani.yaml)"]
        EXP["Experiment Configs<br/>(finetune_t5 / eval_gector / eval_llm / eval_api)"]
    end

    subgraph Data["Data Layer"]
        BASE["base_loader.py<br/>(abstract interface)"]
        FCE["fce_loader.py<br/>(FCECorpusLoader — implemented)"]
        SCHEMA["schema.py<br/>(LearnerRecord contract)"]
        PREP["preprocessing.py<br/>(normalize + segment)"]
    end

    subgraph Annotation["Annotation Layer — Step 3"]
        ERRANT["errant_wrapper.py<br/>(edit extraction + error typing)"]
    end

    subgraph Models["Model Layer — Steps 4-5"]
        FT["finetune_encoder_decoder.py<br/>(T5-base — the ONE trained model)"]
        GECTOR["run_gector.py<br/>(inference-only baseline)"]
        LLM["run_instruct_llm.py<br/>(inference-only baseline)"]
        API["run_api_baseline.py<br/>(inference-only, optional)"]
    end

    subgraph Evaluation["Evaluation Layer — Steps 4 & 6"]
        METRICS["errant_metrics.py + bleu_chrf.py"]
        SIG["bootstrap_significance.py"]
        BREAKDOWN["error_breakdown.py"]
    end

    subgraph Reporting["Reporting Layer — Steps 6-7"]
        REPORT["report_generator.py"]
    end

    L1P --> FCE
    FCE -.implements.-> BASE
    FCE --> SCHEMA
    FCE --> PREP
    SCHEMA --> ERRANT
    EXP --> FT
    EXP --> GECTOR
    EXP --> LLM
    EXP --> API
    ERRANT --> METRICS
    FT --> METRICS
    GECTOR --> METRICS
    LLM --> METRICS
    API --> METRICS
    METRICS --> SIG
    METRICS --> BREAKDOWN
    SIG --> REPORT
    BREAKDOWN --> REPORT

    style Data fill:#e8f4ea,stroke:#4a7c59
    style Annotation fill:#fdf3e7,stroke:#c98a2c
    style Models fill:#fdf3e7,stroke:#c98a2c
    style Evaluation fill:#fdf3e7,stroke:#c98a2c
    style Reporting fill:#fdf3e7,stroke:#c98a2c
    style Config fill:#eef1f7,stroke:#4a5a7c

Loading

This diagram reflects the module layout of the executed pipeline. Every layer shown (Data, Annotation, Models, Evaluation, Reporting) has run end-to-end against the real Turkish-FCE corpus. See Project Roadmap for exactly which baselines were executed and which remain optional future work.

Pipeline Workflow

sequenceDiagram
    participant Raw as Raw FCE Corpus (licensed, not redistributed)
    participant Loader as FCECorpusLoader
    participant Records as LearnerRecord[]
    participant Annot as ERRANT
    participant Base as LanguageTool baseline
    participant FT as Fine-tuned T5
    participant Eval as Evaluation
    participant Report as Technical Report

    Raw->>Loader: XML files, filtered by <language>
    Loader->>Loader: inspect_xml_structure() [schema verification]
    Loader->>Records: sentence-level source/target pairs
    Records->>Annot: edit extraction + error-category tagging
    Records->>Base: inference (no training)
    Records->>FT: train/val/test split, fine-tuning
    Annot->>Eval: gold error categories
    Base->>Eval: predicted corrections
    FT->>Eval: predicted corrections
    Eval->>Eval: ERRANT P/R/F0.5, BLEU/ChrF
    Eval->>Report: per-category comparison table

Loading

This diagram reflects the pipeline as actually executed: LanguageTool and fine-tuned T5 are the systems that produced predictions. GECToR, the instruction-tuned LLM, and the commercial API baseline are implemented but were not run in this execution — see Project Roadmap.

Repository Structure

l1-gec/
├── configs/                    # All experiment/data configuration (ADR 0004)
│   ├── logging.yaml
│   ├── l1_profiles/            # turkish.yaml (implemented), persian.yaml / azerbaijani.yaml (stubs)
│   └── experiment_configs/     # finetune_t5, eval_gector, eval_llm, eval_api
├── src/l1_gec/                 # Installable package (`pip install -e .`)
│   ├── data/                   # ✅ Implemented: schema, base_loader, fce_loader, preprocessing
│   ├── annotation/              # ✅ ERRANT wrapper — confirmed on real hardware
│   ├── models/                  # ✅ finetune_encoder_decoder.py and the LanguageTool baseline executed against real data. GECToR, the instruction-tuned LLM, and the commercial API baseline are implemented but were not executed in this run — optional future work
│   ├── evaluation/               # ✅ errant_metrics, bleu_chrf, error_breakdown all executed against real data. bootstrap_significance.py is implemented but was not invoked in this run — optional future work
│   └── reporting/                # ✅ plot_training_curves, render_comparison_table_markdown, render_bootstrap_summary_markdown — all complete and verified
├── tests/                       # pytest suite + synthetic fixtures (real corpus not redistributed)
├── docs/
│   ├── RESEARCH_SPEC.md         # Frozen research design (hypotheses, methodology, limitations)
│   ├── report.docx              # Technical report with real evaluation results (Section 6)
│   ├── adr/                     # Architecture Decision Records
│   ├── related_work.md
│   └── results_analysis.md      # Category-level analysis of the real evaluation results
├── data/README.md               # Dataset provenance, download + schema-verification steps
├── results/                     # Real outputs from the executed pipeline (turkish_fce/baseline_comparison.csv, predictions/, cache/)
├── .github/workflows/ci.yml
├── Dockerfile
├── Makefile
├── pyproject.toml
├── requirements.txt
└── .gitignore

Installation

Prerequisites: Python 3.10+ (tested with 3.10, 3.11, and 3.13), or Docker.

Option A — Local virtual environment (pip)

git clone https://github.com/NasiDev/L1_GEC_Project.git && cd L1_GEC_Project
python -m venv .venv && source .venv/bin/activate
make setup

Option A2 — Local virtual environment (uv)

git clone https://github.com/NasiDev/L1_GEC_Project.git && cd L1_GEC_Project
uv sync

Important uv-specific note: uv sync installs the project's runtime dependencies plus the [dependency-groups] dev group (pytest, pytest-cov, coverage, ruff, black, mypy) automatically — this is a special case for the dev group specifically (see uv's dependency groups docs). Development tools declared only under [project.optional-dependencies] are not installed by uv sync without an explicit --extra dev flag. An earlier version of this repository declared dev tools only as an optional-dependency extra, which silently dropped pytest/pytest-cov/coverage after uv sync — fixed by adding the proper [dependency-groups] table in pyproject.toml. If you still don't see them after uv sync, run uv sync --all-groups to confirm, and check you're on a uv version that supports PEP 735 dependency groups.

Option B — Docker (recommended for exact reproducibility)

make docker-build
make docker-run

See ADR 0001 for why this is a single image rather than a multi-service docker-compose setup.

Verification Workflow

This repository distinguishes code-complete from verified throughout — see the Project Roadmap and IMPLEMENTATION_STATUS.md for the current status of each component. Verification scripts, in the order they become relevant:

Script	What it checks	Cost
scripts/verify_errant_wrapper.py	ERRANT integration (Step 3)	Requires errant + spaCy model, no large downloads
scripts/verify_training_pipeline.py	T5 fine-tuning pipeline (Step 4)	Uses t5-small (~250MB), ~1 minute on CPU
scripts/verify_baselines.py	Baseline runners (Step 5)	Lightweight by design — the instruction-tuned LLM check is fully mocked (unittest.mock), so it downloads nothing and runs no real inference; it verifies the code's control flow only. GECToR (optional) still requires its own package + a small vocab file if you choose to test it. The API baseline is skipped unless you pass --run-api (costs real money).

None of these scripts are a substitute for running the real pipeline on real data — they exist to catch API-usage bugs cheaply before a multi-hour (or, in the LLM case, multi-GB) real run.

Quickstart

The following example demonstrates the data-loading layer. See Project Roadmap for the full pipeline, including annotation, baseline evaluation, fine-tuning, and reporting.

from l1_gec import configure_logging
from l1_gec.data import FCECorpusLoader, inspect_xml_structure

configure_logging()

# Before running against a real download, verify this parser's tag
# assumptions match your actual corpus file (see data/README.md):
inspect_xml_structure("path/to/one_real_fce_file.xml")

# Load the Turkish-L1 subset as standardized LearnerRecord objects:
loader = FCECorpusLoader(
    source_path="path/to/fce_corpus_directory",
    l1_code="tur",
    l1_name_in_corpus="Turkish",
)
records = loader.load()

for record in records[:3]:
    print(f"[{record.record_id}]")
    print(f"  source: {record.source_text}")
    print(f"  target: {record.target_text}")
    print(f"  unchanged: {record.is_unchanged}")

Status recap: Step 8 is complete — the end-to-end pipeline has been executed against the real Turkish-FCE corpus. The LanguageTool baseline and the fine-tuned T5-base model were both evaluated on the held-out test split. The evaluation modules (errant_metrics, bleu_chrf, error_breakdown) and the reporting module (report_generator) are implemented and were used to produce the real results in docs/report.docx. Bootstrap significance analysis (bootstrap_significance.py) is implemented but optional — it was not invoked in this run. GECToR, the instruction-tuned LLM baseline, and the commercial API baseline remain optional and were not executed. See IMPLEMENTATION_STATUS.md for the full, dated record.

Development Workflow

make help      # list all commands
make lint      # ruff + black --check
make format    # auto-fix + reformat
make test      # run pytest with coverage
make ci        # lint + test (mirrors GitHub Actions exactly)

Testing

The full test suite passes: 122 passed, 1 skipped. Overall coverage is approximately 72%. CI runs the canonical pytest invocation on push.

Synthetic XML fixtures are used for FCECorpusLoader tests — the real FCE Public Corpus is not redistributed here, per its research-use license (see data/README.md).

Project Roadmap

Step	Deliverable	Status
1	Repository architecture	✅ Complete
2	Data ingestion & preprocessing	✅ Complete
3	ERRANT error annotation	✅ Confirmed — 14/14 ERRANT tests pass on real hardware (Python 3.13.14, spaCy 3.8.14, ERRANT 3.0.2)
4	T5-base fine-tuning	✅ Confirmed — ran successfully on a real machine (Python 3.13.14, torch 2.12.1, transformers 5.13.0)
5	Baseline evaluation	✅ Complete — LanguageTool baseline executed against real data
6	Statistical analysis & error breakdown tooling	✅ Complete (pure Python + numpy, no ML deps needed)
7	Technical report / paper draft	✅ Complete — Results section populated with real experimental results in docs/report.docx
8	Full pipeline execution on real Turkish-FCE data	✅ Complete

Future Extensions

The following are implemented but were not part of the executed evaluation above, and remain optional future work:

• GECToR baseline
• Instruction-tuned LLM baseline
• Commercial API baseline
• Bootstrap significance analysis on the real results
• Multilingual robustness comparison against a non-Turkish L1 subset
• Portfolio integration (CV/SOP/outreach summaries)
• Persian and Azerbaijani L1 extensions

Full week-by-week milestones with concrete "definition of done" criteria: docs/RESEARCH_SPEC.md, Section 13.

Training pipeline control-flow diagram: docs/figures/step4_training_pipeline.md.

Technical report (BEA Workshop format, 9 sections, real evaluation results in Section 6): docs/report.docx, with citations in docs/report/references.bib.

Research Design

• Formal hypotheses, experimental design, error taxonomy methodology: docs/RESEARCH_SPEC.md
• Engineering decisions and their rationale: docs/adr/
• Literature review: docs/related_work.md
• Results: docs/report.docx (Section 6) and docs/results_analysis.md

Contributing

This is currently a solo research project built toward an MSc application portfolio, but it's structured to be genuinely extensible — see CONTRIBUTING.md for coding standards, the ADR process, and how a future collaborator (e.g., a lab supervisor or co-author) would add support for a new L1 population.

License

MIT

Citation

If referencing this work before formal publication:

@misc{dadashrostamisales2026l1gec,
  author = {Dadashrostamisales, Nastaran},
  title  = {l1-gec: L1-Transfer-Aware Grammatical Error Correction for Underrepresented Learner Populations},
  year   = {2026},
  note   = {Unpublished research prototype},
  howpublished = {\url{https://github.com/NasiDev/L1_GEC_Project}}
}

Acknowledgments

This project's framing directly builds on and cites Alisoy (2025)'s work on Azerbaijani EFL learners, and uses the FCE Public Corpus (Yannakoudakis, Briscoe & Medlock, 2011) and the ERRANT toolkit (Bryant, Felice & Briscoe, 2017). Full citations: docs/related_work.md.