Lazy suspension

[rakyll]

So far, we observed that the explicit ResumeActor/SuspendActor RPCs turn agent applications into orchestrators. Additionally, eagerly suspending an actor after every turn is expensive if another request arrives shortly after, causing unnecessary snapshot and restore latency. Because agentic applications need to suspend at turn boundaries or when they become idle due to I/O block, they sometimes need to be frequently suspended and resumed. And become the responsibility is left to the application layer, agentic loop itself is becoming a complex orchestrator itself.

Another strategy to suspension could be adopting a lazier version of the suspension behavior. so we can rely on Substrate orchestrator to make more advanced decisions over time instead of shifting this responsibility to Substrate applications. Instead of requiring explicit suspend calls, application layer can hint that it's in a suspend-able state and leave it up to Substrate to suspend the actor when needed.

In this new model:

• Applications signal when they are idle/safe to snapshot without forcing an immediate eviction.
• Substrate keeps suspendable actors warm as long as worker pool capacity permits, and lazily suspends them under pool pressure.
• Traffic arriving at atenet continue to automatically trigger resumption if the target actor is hibernating.

Pros of lazy suspension

• Agents no longer need to implement custom suspension/resumption logic to handle the tradeoff between density and resumption latency.
• Back-to-back requests to the same actor hit a warm sandbox if worker capacity is available, bypassing unnecessary snapshot/restore cycles.
• Substrate maximizes multiplexing efficiency by reclaiming worker resources only when actual resource pressure demands it. We can implement different policies here in time if one size doesn't fit it all.

Cons

• Data loss can occur at the actual suspension time which isn't transparent to the application layer.
• Under high oversubscription, a sudden burst of new incoming actor requests requires Substrate to rapidly evict and snapshot multiple "suspendable" actors to free up physical worker pods.
• Applications can marks themselves as "suspendable" but remains running, e.g. running background operations. Developers need to be careful about the execution model to avoid inconsistent behavior.

We should discuss this option before finalizing PauseActor. I think they are highly related and pausing could be unnecessary if deferring suspension was possible.