What happened
When an agent is given a greenfield scaffolding task, no references load from supabase-postgres-best-practices. The agent proceeds using general Postgres training data to make foundational schema decisions — primary key type, data types, connection mode — without any Supabase-specific guidance.
This matters because stale training data gets these wrong in predictable ways. Tested against claude-haiku-4.5 with no skill loaded:
@supabase/auth-helpers-nextjs — deprecated, replaced by @supabase/ssrNEXT_PUBLIC_SUPABASE_ANON_KEY — old key namecreateClient as a shared singleton — wrong pattern for Next.js App Router
Root Cause
Two compounding issues:
1. Description only triggers on optimization vocabulary
"Use this skill when writing, reviewing, or optimizing Postgres queries, schema designs, or database configurations."
An agent scaffolding a new project isn't "optimizing" — it's building from scratch. The trigger never fires.
2. Skill body has no conditional loading guidance
The model is left guessing which references to load from file titles alone (query-missing-indexes.md, conn-pooling.md). For vague prompts those titles don't match. Compare to the base supabase skill which has explicit MUST read when conditions per reference.
Test Cases
Evaluated using claude-haiku-4.5 via OpenRouter, simulating which references an agent would load per prompt. All four returned zero references before the fix.
| Prompt |
References loaded (before) |
References loaded (after) |
create a supabase backend |
❌ none |
✅ schema-primary-keys, schema-data-types, schema-constraints, security-rls-basics, security-privileges |
use supabase as the database |
❌ none |
✅ schema-primary-keys, schema-data-types, schema-constraints, security-rls-basics |
I'm building a new app with Supabase |
❌ none |
✅ schema-primary-keys, schema-data-types, security-rls-basics |
I'm new to Supabase, where do I start |
❌ none |
✅ schema-primary-keys, schema-data-types, security-rls-basics |
set up a postgres database for my app |
❌ none |
✅ schema-primary-keys, schema-data-types, schema-constraints, security-rls-basics, conn-pooling |
scaffold a new supabase project |
❌ none |
✅ schema-primary-keys, schema-data-types, schema-constraints, security-rls-basics, conn-pooling |
Fix
See PR #99. Two changes to skills/supabase-postgres-best-practices/SKILL.md:
- Extended description to include scaffolding/setup trigger phrases
- Added a Reference Loading Guide table mapping scenarios to which references should load first — so the model has explicit guidance rather than guessing from filenames
Full eval results — before vs after (97 prompts, claude-haiku-4.5)
Run against all 97 existing prompts. Total reference loads: 227 → 245 (+18). No existing prompts regressed.
| Reference |
Before |
After |
Δ |
| schema-constraints.md |
10/97 (10%) |
14/97 (14%) |
+4 |
| security-rls-basics.md |
10/97 (10%) |
14/97 (14%) |
+4 |
| conn-limits.md |
6/97 (6%) |
8/97 (8%) |
+2 |
| lock-short-transactions.md |
13/97 (13%) |
15/97 (15%) |
+2 |
| schema-data-types.md |
12/97 (12%) |
14/97 (14%) |
+2 |
| schema-lowercase-identifiers.md |
2/97 (2%) |
4/97 (4%) |
+2 |
| schema-primary-keys.md |
10/97 (10%) |
12/97 (12%) |
+2 |
| security-privileges.md |
3/97 (3%) |
5/97 (5%) |
+2 |
| security-rls-performance.md |
7/97 (7%) |
9/97 (9%) |
+2 |
| skill-feedback.md |
8/97 (8%) |
10/97 (10%) |
+2 |
| conn-prepared-statements.md |
1/97 (1%) |
2/97 (2%) |
+1 |
| data-upsert.md |
4/97 (4%) |
5/97 (5%) |
+1 |
| advanced-jsonb-indexing.md |
3/97 (3%) |
2/97 (2%) |
-1 |
| query-index-types.md |
4/97 (4%) |
3/97 (3%) |
-1 |
| monitor-explain-analyze.md |
21/97 (22%) |
19/97 (20%) |
-2 |
| monitor-pg-stat-statements.md |
14/97 (14%) |
12/97 (12%) |
-2 |
| schema-foreign-key-indexes.md |
8/97 (8%) |
6/97 (6%) |
-2 |
References with no change omitted for brevity.
Baseline variance — 3 independent clean runs (before fix)
Established to confirm results are stable, not temperature noise. Most references have σ < 1%. Several are perfectly deterministic (σ = 0.0).
| Reference |
Mean % |
σ |
Run 1 |
Run 2 |
Run 3 |
| monitor-explain-analyze.md |
22.0% |
0.6 |
21 |
22 |
21 |
| conn-pooling.md |
13.7% |
0.6 |
14 |
13 |
13 |
| monitor-pg-stat-statements.md |
13.4% |
1.0 |
12 |
13 |
14 |
| lock-short-transactions.md |
13.1% |
1.6 |
11 |
14 |
13 |
| schema-data-types.md |
11.7% |
0.6 |
11 |
11 |
12 |
| query-missing-indexes.md |
11.3% |
0.0 |
11 |
11 |
11 |
| schema-constraints.md |
11.0% |
1.2 |
10 |
12 |
10 |
| security-rls-basics.md |
11.0% |
1.2 |
12 |
10 |
10 |
| schema-primary-keys.md |
10.0% |
0.6 |
10 |
9 |
10 |
| query-composite-indexes.md |
8.2% |
0.0 |
8 |
8 |
8 |
| lock-deadlock-prevention.md |
7.9% |
0.6 |
7 |
8 |
8 |
| skill-feedback.md |
7.9% |
0.6 |
7 |
8 |
8 |
| lock-advisory.md |
7.6% |
0.6 |
7 |
7 |
8 |
| monitor-vacuum-analyze.md |
7.6% |
0.6 |
7 |
8 |
7 |
| security-rls-performance.md |
7.6% |
0.6 |
8 |
7 |
7 |
| conn-limits.md |
7.2% |
1.0 |
7 |
8 |
6 |
| schema-foreign-key-indexes.md |
7.2% |
1.0 |
7 |
6 |
8 |
| lock-skip-locked.md |
6.2% |
0.0 |
6 |
6 |
6 |
| data-n-plus-one.md |
5.5% |
0.6 |
5 |
5 |
6 |
| data-pagination.md |
5.5% |
0.6 |
6 |
5 |
5 |
| conn-idle-timeout.md |
4.8% |
0.6 |
5 |
5 |
4 |
| data-upsert.md |
4.5% |
0.6 |
5 |
4 |
4 |
| schema-partitioning.md |
4.5% |
0.6 |
5 |
4 |
4 |
| data-batch-inserts.md |
4.1% |
0.0 |
4 |
4 |
4 |
| query-index-types.md |
4.1% |
0.0 |
4 |
4 |
4 |
| security-privileges.md |
3.1% |
0.0 |
3 |
3 |
3 |
| query-partial-indexes.md |
2.7% |
0.6 |
3 |
2 |
3 |
| schema-lowercase-identifiers.md |
2.7% |
0.6 |
3 |
3 |
2 |
| advanced-jsonb-indexing.md |
2.4% |
0.6 |
2 |
2 |
3 |
| advanced-full-text-search.md |
2.1% |
0.0 |
2 |
2 |
2 |
| query-covering-indexes.md |
1.7% |
0.6 |
2 |
1 |
2 |
| conn-prepared-statements.md |
1.0% |
0.0 |
1 |
1 |
1 |
Clean vs noisy context diff (after fix)
Noisy run adds realistic mid-session context (TypeScript refactors, failing CI, stack traces) before each prompt. Tests whether references load consistently regardless of what was in the context window before the Supabase question.
Total loads: clean 245 → noisy 254 (+9). References with drift ≥ 5 flagged as context-sensitive.
| Reference |
Clean |
Noisy |
Δ |
Note |
| conn-pooling.md |
13 |
19 |
+6 |
⚠ context-sensitive |
| schema-foreign-key-indexes.md |
6 |
11 |
+5 |
⚠ context-sensitive |
| skill-feedback.md |
10 |
15 |
+5 |
⚠ context-sensitive |
| query-composite-indexes.md |
8 |
12 |
+4 |
|
| query-missing-indexes.md |
11 |
15 |
+4 |
|
| conn-idle-timeout.md |
4 |
7 |
+3 |
|
| monitor-vacuum-analyze.md |
7 |
4 |
-3 |
|
| schema-primary-keys.md |
12 |
9 |
-3 |
|
| lock-short-transactions.md |
15 |
13 |
-2 |
|
| lock-skip-locked.md |
6 |
4 |
-2 |
|
| monitor-explain-analyze.md |
19 |
21 |
+2 |
|
| schema-constraints.md |
14 |
12 |
-2 |
|
| schema-lowercase-identifiers.md |
4 |
2 |
-2 |
|
What happened
When an agent is given a greenfield scaffolding task, no references load from
supabase-postgres-best-practices. The agent proceeds using general Postgres training data to make foundational schema decisions — primary key type, data types, connection mode — without any Supabase-specific guidance.This matters because stale training data gets these wrong in predictable ways. Tested against
claude-haiku-4.5with no skill loaded:@supabase/auth-helpers-nextjs— deprecated, replaced by@supabase/ssrNEXT_PUBLIC_SUPABASE_ANON_KEY— old key namecreateClientas a shared singleton — wrong pattern for Next.js App RouterRoot Cause
Two compounding issues:
1. Description only triggers on optimization vocabulary
An agent scaffolding a new project isn't "optimizing" — it's building from scratch. The trigger never fires.
2. Skill body has no conditional loading guidance
The model is left guessing which references to load from file titles alone (
query-missing-indexes.md,conn-pooling.md). For vague prompts those titles don't match. Compare to the basesupabaseskill which has explicitMUST read whenconditions per reference.Test Cases
Evaluated using
claude-haiku-4.5via OpenRouter, simulating which references an agent would load per prompt. All four returned zero references before the fix.create a supabase backenduse supabase as the databaseI'm building a new app with SupabaseI'm new to Supabase, where do I startset up a postgres database for my appscaffold a new supabase projectFix
See PR #99. Two changes to
skills/supabase-postgres-best-practices/SKILL.md:Full eval results — before vs after (97 prompts, claude-haiku-4.5)
Run against all 97 existing prompts. Total reference loads: 227 → 245 (+18). No existing prompts regressed.
References with no change omitted for brevity.
Baseline variance — 3 independent clean runs (before fix)
Established to confirm results are stable, not temperature noise. Most references have σ < 1%. Several are perfectly deterministic (σ = 0.0).
Clean vs noisy context diff (after fix)
Noisy run adds realistic mid-session context (TypeScript refactors, failing CI, stack traces) before each prompt. Tests whether references load consistently regardless of what was in the context window before the Supabase question.
Total loads: clean 245 → noisy 254 (+9). References with drift ≥ 5 flagged as context-sensitive.