Redelivered message is discarded inside Receive by a disposed CancellationTokenSource (10.5.0+)

[jurgiskisunas-visma]

Summary

When Azure Service Bus redelivers a message while the first delivery is still being handled — normal at-least-once behaviour after a peek lock lapses — AzureServiceBusTransport.Receive throws ObjectDisposedException and the redelivery is discarded before dispatch. No handler runs, Rebus' retry pipeline never engages, and a broker delivery attempt is burned. If MaxDeliveryCount is exhausted before the first delivery completes, the broker dead-letters the message natively, outside anything Rebus can observe.

Affected: verified on 10.5.1 and 10.7.0; the lines are unchanged from 10.5.0 (#105) through master.

Runnable reproduction against the official Service Bus emulator, no Azure subscription needed — docker compose up -d && dotnet test, about 2 minutes: https://github.com/jurgiskisunas-visma/rebus-asb-failure-proof

Cause

var renewFailedTokenSource = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, _cancellationToken);  // L620
if (!_messageRenewerTokenSources.TryAdd(message.MessageId, renewFailedTokenSource))                                   // L621
{
    // should never happen though
    renewFailedTokenSource.Dispose();                                                                                 // L624
}
...
items["asb-message-cancel-token"] = renewFailedTokenSource.Token;                                                     // L632

MessageId identifies a message; the thing being tracked is a delivery. The same MessageId is legitimately in flight twice whenever a lock lapses and the broker redelivers while the handler is still running — so TryAdd fails, the token source is disposed, and .Token is read from it.

System.ObjectDisposedException: The CancellationTokenSource has been disposed.
   at System.Threading.CancellationTokenSource.get_Token()
   at Rebus.AzureServiceBus.AzureServiceBusTransport.Receive(ITransactionContext context, CancellationToken cancellationToken)
   at Rebus.Workers.ThreadPoolBased.ThreadPoolWorker.ReceiveTransportMessage(CancellationToken token, ITransactionContext context)

Two things widen this beyond what it first looks like:

• L620 sits outside the AutomaticallyRenewPeekLock && !_prefetchingEnabled guard at L641, so it also affects users who never enabled peek lock renewal. The third test demonstrates that.
• context.OnDisposed(...) is registered at L711, after the TryAdd at L621 — so a throw in between (e.g. the RebusApplicationException at L636) orphans an entry with no cleanup registered, and neither dictionary is cleared in Dispose().

For what it's worth, OnAck/OnNack already remove from the dictionaries before Complete/Abandon, which closes the obvious redelivery race — the remaining hole is specifically the lock-lapse overlap.

Reproduction

Test	Shows
RecoverableLockLapse_...	The case that matters. LockDuration PT5S; the handler overruns the lock on the first delivery only and returns in milliseconds afterwards. A correct client settles on delivery #2 — instead every overlapping redelivery is discarded inside Receive, and the Rebus error queue stays empty throughout.
Receive_WhenSameMessageIdIsInFlightTwice_...	Minimal isolation of the failing line (~100 ms, no lock timing). Two sends sharing one explicitly-set MessageId. This is isolation, not a claim that ASB duplicates ids on its own.
...AndRenewalIsDisabled_ThenItStillThrows	Same, with AutomaticallyRenewPeekLock() never called. Still fails.

The short lock duration only makes the single lapse deterministic — it isn't what causes the failure. The message stays comfortably processable after the lapse; each redelivery had a full 5 seconds to run a handler needing milliseconds.

What a fix needs

Stating requirements rather than a patch — the implementation is your call:

1. The receive path shouldn't throw on a condition the broker produces by design.
2. The tracking key should identify a delivery, not a message — the lock token is unique per delivery, MessageId isn't. The same aliasing affects _messageLockRenewers, where one delivery's cleanup can remove another's renewer and silently leave it unrenewed.

Worth saying explicitly: simply reading the existing entry after a failed TryAdd is not a fix — it races with the other delivery's cleanup (KeyNotFoundException, or a source disposed a moment later), and sharing one token source between two deliveries means cancelling one cancels the other.

Related production observation (not reproduced)

On 10.5.1 / ASB Premium / LockDuration PT5M / MaxDeliveryCount 10 we also see "Error when renewing peek lock for message with ID {messageId}" every ~10 s for 2 h 47 min for one message id — continuing two hours after that message was dead-lettered, stopping only on host restart. 983 exceptions from one message.

We could not reproduce this (the renewer is cleaned up correctly in our tests), but it looks explainable from the code: RenewPeekLocks cancels on renewal failure without removing the entry, and MessageLockRenewer.GetTimeOfNextRenewal doesn't update _nextRenewal on failure — once LockedUntil is past it returns a past time, so IsDue latches true and the 10-second task retries forever. 2 h 47 min ÷ 10 s ≈ 1000, against 983 observed. This resembles #40, fixed in 7.0.0-a15. Flagging as an observation, not a claim.

Minor

This is logged via _log.Warn("...: {exception}", exception) rather than the Warn(Exception, ...) overload, so it arrives as rendered text with no typed exception attached — log sinks that index by exception type don't see it, which is why it took us a while to find these in Application Insights.

Environment

Rebus.AzureServiceBus 10.5.1 and 10.7.0 · .NET 10 · Azure Service Bus Premium (production), mcr.microsoft.com/azure-messaging/servicebus-emulator:latest (repro) · observed across four services sharing one namespace, 12 occurrence-days in 90 days.