Skip to content

Fix EventQueue losing state on persistence failures - #1050

Open
elnafateh wants to merge 1 commit into
lightningdevkit:mainfrom
elnafateh:fix/eventqueue-persist-ordering
Open

Fix EventQueue losing state on persistence failures#1050
elnafateh wants to merge 1 commit into
lightningdevkit:mainfrom
elnafateh:fix/eventqueue-persist-ordering

Conversation

@elnafateh

Copy link
Copy Markdown
Contributor

Fixes #1027.

Problem

EventQueue::add_event and EventQueue::event_handled mutate the in-memory queue before the mutation is durably persisted. If the persist call fails or races with another operation, in-memory and persisted state can diverge, causing:

  1. Duplicate events — a failed add_event still leaves the event in memory; when LDK retries via ReplayEvent, the event gets enqueued a second time.
  2. Silently skipped events — a failed event_handled still pops the front event from memory; a caller retrying the ack (per the documented "same event until confirmed" contract) actually acks the next event instead.
  3. Restart resurrection — a mutation that's applied in memory but not yet persisted is lost on crash/restart, so an already-acked event can reappear.
  4. Lost newer writes — without ordering, a stale in-flight persist can overwrite a newer one.

Fix

Serialize add_event/event_handled end-to-end behind a tokio::sync::Mutex (operation_lock) spanning snapshot-compute → persist → in-memory-commit. In-memory state is now only ever advanced after its snapshot is durably persisted, and persistence submissions happen strictly in mutation order.

This closes all four failure modes structurally:

  • (1)/(2): a failed persist means the in-memory mutation never happens, so retries repeat the same logical operation instead of duplicating/skipping.

  • (3)/(4): full serialization means there's no concurrent-write-submission window left to race — persisted state can never fall behind or get reordered relative to in-memory state.

I verified that SqliteStore, VssStore, and PostgresStore all allocate their write version synchronously at the top of write(), before any async work — which is what makes serializing at this layer sufficient to also close the built-in stores' residual version-ordering window. Custom KVStore implementations that defer version allocation to a later async step wouldn't get that same guarantee from this fix alone — worth flagging as a documented expectation of KVStore implementors if that's not already stated.

Testing

Added a fault-injecting KVStore wrapper (FaultingStore) that can be told to fail the next N writes, with regression tests for:

  • Failed-enqueue replay (no duplicate)

  • Failed-ack retry (no skip)

  • Concurrent enqueue/ack under load (no lost updates, persisted state matches in-memory)

  • Restart resurrection (persisted bytes never reflect an uncommitted mutation)

add_event and event_handled mutated the in-memory queue before the
mutation was durably persisted. A failed or delayed persist could
then cause duplicate events on replay, silently skipped
acknowledgements, or resurrected/lost events after a restart.

Serialize both operations end-to-end behind an operation lock, so
the in-memory queue is only advanced after its snapshot has been
persisted, and persistence writes are submitted in mutation order.

Add regression tests for failed-enqueue replay, failed-ack retries,
concurrent enqueue/ack under load, and restart behavior, using a
fault-injecting KVStore.

Fixes lightningdevkit#1027.
@ldk-reviews-bot

ldk-reviews-bot commented Aug 13, 2026

Copy link
Copy Markdown

I've assigned @tnull as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@ldk-reviews-bot
ldk-reviews-bot requested a review from tnull August 13, 2026 11:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

EventQueue persistence failures can duplicate, skip, or resurrect events

2 participants