Resolve duplicate scenario cards by identity

Author: rschardosin

docs/architecture/memory.md

@@ -179,7 +179,10 @@ The Studio memory icon appears immediately for in-band `memory_save` tool calls.
 
 Platform mode now treats durable operational memory as a **scenario card** when possible. A scenario card is still a normal memory row: it is stored in the existing personal/team/org memory store, embedded, and returned by the same semantic + BM25 search pipeline. The difference is the content contract. The row category is `scenario_card/efficient_successful_path`, and the markdown body contains frontmatter plus sections for:
 
-- `canonical_key`: stable topic key used for merge/upsert
+- `canonical_key`: a human-readable alias/label for the scenario, not the durable identity boundary
+- `scenario_id`, `aliases`, `related_scenario_ids`: optional entity metadata for card evolution
+- `identity_json`: deterministic scenario anchors such as domain, system, service family, resource type, operation, environment, credentials, endpoint host family, API family, HTTP method, and URL path
+- `superseded_json`: explicit temporal/supersedes notes when an old value or path has been replaced
 - `Recommended path`: the shortest known successful recipe
 - `Conditions`: when the recipe applies
 - `Verification`: how the path was or should be checked
@@ -188,12 +191,15 @@ Platform mode now treats durable operational memory as a **scenario card** when
 
 This makes scenario cards behave like self-managed operational skills: they are generated from verified experience, searched semantically like any other memory, and improved over time by merging new evidence into the same card. They are not human-authored skills from `memory/skills`, and they do not replace explicit skills or flows. Raw memory rows are staging inputs only; after they are incorporated into a scenario card, or if they cannot form a useful card, they are deleted or discarded rather than kept as durable memory.
 
+Scenario-card upsert uses **scenario identity resolution**, not only canonical-key equality. `pkg/memory/scenario_identity.go` extracts deterministic anchors and scores candidate pairs with rarity-aware weights. A card is auto-merged only when the score crosses `DefaultScenarioAutoMergeThreshold`, there are no negative signals, and the best candidate is not ambiguous. Scores below the auto-merge threshold are not sent to an LLM on the write path; ambiguous duplicate work belongs in Memory Health, where a user can review the proposed merge. The resolver deliberately separates alias resolution from deduplication: aliases like “LBaaS” and “Octavia” can map to the same service family, but conflicting resource types, environments, or write/read operations prevent silent merges.
+
 Implementation details:
 
-- `pkg/memory/scenario_card.go` owns rendering, parsing, canonical-key normalization, draft generation, merge, upsert, and retrieval filtering.
+- `pkg/memory/scenario_card.go` owns rendering, parsing, draft generation, merge, upsert, and retrieval filtering.
+- `pkg/memory/scenario_identity.go` owns deterministic extraction, corpus statistics, pair scoring, and conservative candidate choice.
 - `MemoryMerger.SaveOrMerge` drafts/upserts a scenario card for platform memory saves and fails closed if the card cannot be saved. It must not fall back to raw-memory insertion because that would reintroduce scattered notes.
-- Promotion is upsert-based. Personal → team and team → org promotion draft a scenario card in the target scope and merge it with any existing card for the same `canonical_key`; after a successful upsert, the source raw memory is deleted.
-- Retrieval asks the underlying search for extra candidates, runs `memory.FilterPreferredScenarioResults`, prefers scenario cards, and suppresses any transitional raw memories that are explicitly listed as `source_memory_ids` or match an already-returned card key.
+- Promotion is upsert-based. Personal → team and team → org promotion draft a scenario card in the target scope and merge it with any existing card that resolves to the same scenario; after a successful upsert, the source raw memory is deleted.
+- Retrieval asks the underlying search for extra candidates, runs `memory.FilterPreferredScenarioResults`, prefers scenario cards, suppresses any transitional raw memories that are explicitly listed as `source_memory_ids`, and de-duplicates equivalent scenario cards as a safety net.
 
 The key invariant is that Astonish should save **how to do the thing efficiently**, not a transcript of exploratory dead ends. Temporary outages, timeouts, rate limits, and “X did not work” observations may appear only as conditional cautions that require re-verification before they change behavior. If a raw memory cannot produce a usable recommended path, it is not durable memory; the system can learn it again later and create a proper card.
 
@@ -206,8 +212,9 @@ Memory Health returns reviewable, actionable recommendations, not automatic writ
 - create a scenario card from any remaining raw memory when no card exists yet
 - update an existing scenario card when new raw source memories are not yet incorporated
 - clean up raw source memories that are already represented by an existing scenario card
+- merge duplicate scenario cards that deterministic identity resolution considers the same operational scenario
 
-Each recommendation contains the proposed card, target scope, source memory IDs, and the diagnostic flags that explain why it was suggested. Applying a recommendation uses the same scenario-card upsert endpoint as manual consolidation; after the card is saved or merged, incorporated raw source memories are deleted. Cleanup recommendations re-save the existing card metadata and delete the still-visible raw source rows. If the proposed card contains only the placeholder recipe, the raw inputs are discarded and no placeholder card is saved.
+Each recommendation contains the proposed card, target scope, source memory IDs, and the diagnostic flags that explain why it was suggested. Duplicate-card recommendations also include `duplicate_card_ids`, `resolver_signals`, and a `match_score`; applying one first upserts the merged card and then deletes only those explicit duplicate scenario-card rows. Applying any recommendation uses the same scenario-card upsert endpoint as manual consolidation; after the card is saved or merged, incorporated raw source memories are deleted. Cleanup recommendations re-save the existing card metadata and delete the still-visible raw source rows. If the proposed card contains only the placeholder recipe, the raw inputs are discarded and no placeholder card is saved.
 
 `GET /api/memories/map` remains available as the advanced diagnostic report behind the Memory Health UI. It groups likely related memory chunks by a canonical topic key and flags conditions that make memory feel scattered or unsafe:
 

docs/website/docs/agent/memory.md

@@ -73,6 +73,8 @@ For repeatable operational tasks, Astonish can consolidate memory into **scenari
 
 This helps the agent reuse the efficient path it learned instead of replaying trial-and-error steps. Temporary failures should be treated as cautions to re-check, not permanent “never use this” rules. Raw memory rows are temporary staging inputs: once they are incorporated into a card, or if they cannot form a useful card, they are deleted or discarded instead of staying as long-term scattered memory.
 
+Scenario cards are matched by scenario identity, not only by title or canonical key. Astonish extracts stable anchors such as system, service family, resource type, operation, environment, credential name, endpoint host family, API family, HTTP method, and URL path. This lets related labels such as “LBaaS” and “Octavia” merge into one OpenStack load-balancer card when the anchors agree, while still keeping different resources or environments separate.
+
 ## Managing Memory in Studio
 
 Studio provides a visual interface for memory management. The main organization surface is **Memory Health**: when opened, Astonish checks whether the visible memories need consolidation, deduplication, or review. The check is lazy and on demand — there is no scheduled background job. If a recent evaluation is still fresh, Studio reuses it; after five days, the next visit runs a new evaluation.
@@ -82,6 +84,7 @@ Studio provides a visual interface for memory management. The main organization
 - Use **Memory Health** to review suggested organization improvements
 - Reanalyze memory on demand; otherwise fresh evaluations are reused for five days
 - Draft and save scenario cards from actionable recommendations
+- Merge duplicate scenario cards when Memory Health identifies two cards as the same scenario; Studio shows the resolver signals and deletes only the explicit duplicate card rows after the merged card is saved
 - Open the advanced **Memory Map** only when you need low-level diagnostics for transitional raw memories
 - Publish personal memories to your team by merging them into scenario cards when possible
 - Promote team memories to org level (admin) by merging them into org scenario cards when possible

docs/website/docs/platform/three-tier-memory.md

@@ -59,16 +59,17 @@ Personal  ──publish──▶  Team  ──promote──▶  Org
 
 The agent can also save knowledge directly during conversations using the `memory_save` tool. The tier is determined by the current context.
 
-Promotion now performs a scenario-card upsert. When possible, the selected memory is distilled into an efficient successful-path card in the target tier and merged with an existing card for the same scenario. After the card is saved, the original raw source memory is deleted; if it cannot form a useful card, it is discarded rather than kept as durable memory.
+Promotion now performs a scenario-card upsert. When possible, the selected memory is distilled into an efficient successful-path card in the target tier and merged with an existing card for the same scenario. Scenario matching uses deterministic identity anchors such as system, service family, resource type, operation, environment, credential, endpoint host family, API family, HTTP method, and URL path; the canonical key is treated as an alias, not the only identity. After the card is saved, the original raw source memory is deleted; if it cannot form a useful card, it is discarded rather than kept as durable memory.
 
 ## The Learning Loop
 
 Here is how knowledge compounds in practice:
 
 1. **Alice** debugs a tricky Kubernetes networking issue. The agent saves the resolution to her personal memory via `memory_save`.
 2. Alice publishes the resolution to the **Backend team** via Studio. Now when any backend engineer hits a similar issue, the agent surfaces Alice's solution.
-3. The team admin notices this resolution is relevant org-wide and **promotes it to org level** via Studio. If an org scenario card already exists, the new evidence is merged into that card instead of creating another duplicate memory.
-4. **Dave** on the Frontend team later encounters the same networking issue. The agent finds the org-level scenario card and guides him through the efficient path — even though Dave never interacted with Alice.
+3. The team admin notices this resolution is relevant org-wide and **promotes it to org level** via Studio. If an org scenario card already exists for the same scenario identity, the new evidence is merged into that card instead of creating another duplicate memory.
+4. If two cards were already created with different labels for the same scenario, **Memory Health** can recommend a duplicate-card merge, show the resolver signals, save the merged card, and delete only the explicit duplicate card rows.
+5. **Dave** on the Frontend team later encounters the same networking issue. The agent finds the org-level scenario card and guides him through the efficient path — even though Dave never interacted with Alice.
 
 Each step is explicit. Knowledge does not leak upward automatically.
 

pkg/api/memory_consolidation.go

@@ -22,18 +22,20 @@ type MemoryConsolidationPreviewResponse struct {
 }
 
 type MemoryConsolidationApplyRequest struct {
-	Card        memory.ScenarioCard `json:"card"`
-	TargetScope string              `json:"target_scope"`
+	Card             memory.ScenarioCard `json:"card"`
+	TargetScope      string              `json:"target_scope"`
+	DuplicateCardIDs []string            `json:"duplicate_card_ids,omitempty"`
 }
 
 type MemoryConsolidationApplyResponse struct {
-	Applied        bool                        `json:"applied"`
-	Scope          string                      `json:"scope"`
-	Action         string                      `json:"action"`
-	ExistingID     string                      `json:"existing_id,omitempty"`
-	Card           memory.ScenarioCard         `json:"card"`
-	Result         memory.ScenarioUpsertResult `json:"result"`
-	DeletedSources int                         `json:"deleted_sources"`
+	Applied               bool                        `json:"applied"`
+	Scope                 string                      `json:"scope"`
+	Action                string                      `json:"action"`
+	ExistingID            string                      `json:"existing_id,omitempty"`
+	Card                  memory.ScenarioCard         `json:"card"`
+	Result                memory.ScenarioUpsertResult `json:"result"`
+	DeletedSources        int                         `json:"deleted_sources"`
+	DeletedDuplicateCards int                         `json:"deleted_duplicate_cards"`
 }
 
 // MemoryConsolidationPreviewHandler drafts a structured scenario card from a
@@ -140,15 +142,17 @@ func MemoryConsolidationApplyHandler(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 	deletedSources := deleteConsolidatedSources(r, svc, pu, req.Card.SourceMemoryIDs)
+	deletedDuplicateCards := deleteDuplicateScenarioCards(r, svc, pu, req.DuplicateCardIDs, result.ExistingID)
 
 	respondJSON(w, http.StatusOK, MemoryConsolidationApplyResponse{
-		Applied:        true,
-		Scope:          req.TargetScope,
-		Action:         result.Action,
-		ExistingID:     result.ExistingID,
-		Card:           req.Card,
-		Result:         result,
-		DeletedSources: deletedSources,
+		Applied:               true,
+		Scope:                 req.TargetScope,
+		Action:                result.Action,
+		ExistingID:            result.ExistingID,
+		Card:                  req.Card,
+		Result:                result,
+		DeletedSources:        deletedSources,
+		DeletedDuplicateCards: deletedDuplicateCards,
 	})
 }
 

pkg/api/memory_handlers.go

@@ -375,6 +375,7 @@ func MemorySearchCrossTierHandler(w http.ResponseWriter, r *http.Request) {
 			respondError(w, http.StatusInternalServerError, fmt.Sprintf("cross-tier search failed: %v", err))
 			return
 		}
+		results = memory.FilterPreferredScenarioResults(results)
 		respondJSON(w, http.StatusOK, MemoryListResponse{Results: results, Count: len(results)})
 		return
 	}
@@ -392,6 +393,7 @@ func MemorySearchCrossTierHandler(w http.ResponseWriter, r *http.Request) {
 			respondError(w, http.StatusInternalServerError, fmt.Sprintf("memory search failed: %v", err))
 			return
 		}
+		results = memory.FilterPreferredScenarioResults(results)
 		respondJSON(w, http.StatusOK, MemoryListResponse{Results: results, Count: len(results)})
 		return
 	}
@@ -933,6 +935,34 @@ func deleteSourceIDsFromStores(ctx context.Context, sourceIDs []string, stores [
 	return deleted
 }
 
+func deleteDuplicateScenarioCards(r *http.Request, svc *store.Services, pu *PlatformUser, duplicateIDs []string, preservedID string) int {
+	if len(duplicateIDs) == 0 || svc == nil || pu == nil {
+		return 0
+	}
+	deleted := 0
+	seen := make(map[string]bool)
+	for _, id := range duplicateIDs {
+		if id == "" || id == preservedID || seen[id] {
+			continue
+		}
+		seen[id] = true
+		for _, memStore := range memoryStoresForVisibleScopes(r, svc, pu) {
+			if memStore == nil {
+				continue
+			}
+			existing, err := memStore.Get(r.Context(), id)
+			if err != nil || existing == nil || !memory.IsScenarioCard(*existing) {
+				continue
+			}
+			if err := memStore.Delete(r.Context(), id); err == nil {
+				deleted++
+			}
+			break
+		}
+	}
+	return deleted
+}
+
 func memoryStoresForVisibleScopes(r *http.Request, svc *store.Services, pu *PlatformUser) []store.MemoryStore {
 	stores := make([]store.MemoryStore, 0, 3)
 	if svc.TenantRouter != nil {