Feature subscriptions

[alowrydi]

Subscription management module, di.subscriptions

Extracts code/common/subscriptions.q from TorQ and packages it as a standalone kdb-x module. Manages tickerplant subscriptions for RDB/WDB/chained-TP consumers - subscribe with optional exactly-once log replay, resubscribe after a dropped connection, unsubscribe, and a live registry of what's currently held.

The modularisation plan places di.subscriptions in the FRAMEWORK tier with hard dependencies on di.servers and di.pubsub. getsubscriptionhandles (the legacy handle-resolution function TorQ consumers call before subscribing) is ported here rather than duplicated across every future di.proc.* module, and a new opt-in republish capability lets a chained/segmented tickerplant consumer hand its subscribed tables to di.pubsub for downstream fan-out - the actual job a chained TP exists to do. Closing the di.pubsub edge safely required three small, necessary changes to that module, included here rather than as a follow-up: its .z.pc was overwriting rather than chaining, meaning depending on it as shipped would silently destroy any other module's observer; it shipped no VERSION, so a version pin against it would have been unenforceable; and it had no read counterpart to setsubtables, which the additive handoff needs. A fourth, unrelated di.pubsub fix (a broken result-shape guard in subscribestr/subscribestrfilter) is bundled in as well - found during this due-diligence read, not required by anything here, worth knowing that distinction if this gets split later.

Trello ticket - https://trello.com/c/aUtlpkfA/133-kdb-x-subscriptions-module

---

Files created / modified

File	Status	Description
di/subscriptions/init.q	new	Loads di.servers/di.pubsub deps, loads subscriptions.q, defines export of 9 functions
di/subscriptions/subscriptions.q	new	Full implementation - module state, internal helpers, public API, init
di/subscriptions/subscriptions.md	new	Full module documentation
di/subscriptions/deps.q	new	Declares di.servers >=0.1.0 and di.pubsub >=0.2.0 as hard dependencies
di/subscriptions/test.csv	new	534 k4unit unit assertions
di/subscriptions/test_integration.csv	new	131 k4unit integration assertions, including the three-real-process republish path
di/pubsub/pubsub.q	modified	.z.pc now chains via di.handlers-style capture instead of overwriting; getsubtables added; subscribestr/subscribestrfilter's result-shape guard fixed
di/pubsub/init.q	modified	Exports getsubtables, version
di/pubsub/VERSION	new	0.2.0
di/pubsub/test.csv	modified	119 assertions; portable - no hardcoded port or path

---

How to test

k4unit:use`di.k4unit
k4unit.moduletest`di.subscriptions

===== di.subscriptions : test.csv =====
2026.08.13T00:53:08.477 start
2026.08.13T00:53:08.477 :.../di/subscriptions/test.csv 534 test(s)
2026.08.13T00:53:08.781 end
assertions : 534
failures   : 0
by action  : `s#`fail`run`true!70 250 214

===== di.subscriptions : test_integration.csv =====
2026.08.13T00:53:08.818 start
2026.08.13T00:53:08.818 :.../di/subscriptions/test_integration.csv 131 test(s)
2026.08.13T00:53:14.390 end
assertions : 131
failures   : 0
by action  : `s#`run`true!70 61

===== di.pubsub : test.csv =====
2026.08.13T00:53:14.418 start
2026.08.13T00:53:14.418 :.../di/pubsub/test.csv 119 test(s)
2026.08.13T00:53:15.550 end
assertions : 119
failures   : 0
by action  : `s#`fail`run`true!2 62 55

784 suite assertions, 0 failures. Plus 317 smoke/adversarial-harness assertions across 12 scratchpad suites (smoke.q/smoke2.q/smoke3.q, adversarial.q, iter3.q, probe7/probe8/probe10/probe11.q, gsh.q, repub.q, pssmoke.q) - 1101 assertions total, 0 failures, 0 flags.

The integration suite runs three real processes for the republish path specifically - an upstream tickerplant, a middle process using this module with republish on, and a downstream subscriber - since a subscriber that serves its own subscribers can't be tested in-process. Its tickerplant peer runs the real di.pubsub, and assertions read that module's own registry directly, so the coupling can't regress into a stand-in.

The handoff-failure warn-not-throw path was proven non-vacuous, not just exercised: the same suite run against a copy with the fix reverted to raiseerror produces 7 failures (the module throws, the completed subscription is lost, nothing is logged at warn); with the fix in place, 0.

di.depcheck:

2026.08.12D23:50:20.082403699 INFO [depcheck] dependency check complete: 0 failure(s), 0 warning(s)
depcheck: PASS

Resolved against deps.q = di.serversdi.pubsub!("0.1.0";"0.2.0"), with di.servers at 0.1.0 and di.pubsub at 0.2.0 on disk.

---

Design decisions

Protocol correctness (subscribe/replay against a real segmented tickerplant)

1. Guards that don't need the tickerplant's reply run before it - fetchdetails's underlying subdetails call is .u.sub, which registers the handle for live delivery as a side effect of just fetching schemas. Every check not needing that reply (root-upd presence, the already-held-table guard, the setschema-implies-exists guard) now runs first; only the log-integrity preflight is irreducible, since log filenames only exist in the reply itself.

2. ` (all tables) is resolved via a tablelist round trip before subdetails - a segmented tickerplant's subdetails can't accept the bare sentinel directly (throws 'rank); legacy always resolved a concrete list first. Falls back to sending ` directly if no usable tablelist is offered.

3. A shared log (segmented singular/periodic multilog mode) collapses to one whole-file replay - the same physical file is reported once per table with different counts; replaying each entry separately re-applies the file's head. Collapsing to one whole-file replay trades a bounded, diagnosable trailing duplicate for never silently under-replaying, which a naive per-table sum could do.

4. A whole-file replay is narrowed to the offered tables; a genuine per-table log is not - a segmented tickerplant logs more tables than it publishes, so a whole-file read (from the shared-log collapse or the 0Wj sentinel) can carry a table the caller never subscribed to. Narrowed via the existing table filter; the common per-table-log fast path is unaffected.

5. 0Wj means "replay everything", not "more than expected" - resolved to the log's own readable total rather than forwarded to -11!, which would otherwise replay a corrupt log's good prefix and only then throw - the exact half-populated state the preflight exists to prevent.

Framework-tier dependencies and republish

6. getsubscriptionhandles ported rather than left to each caller - real TorQ callers across rdb.q, wdb.q, chainedtp.q, sctp.q all resolve a handle the same way before subscribing; centralising it here avoids four future di.proc.* modules each reimplementing legacy's `/()/intersection-vs-union selection semantics independently.

7. republish is opt-in, default off - realises the chained/segmented-tickerplant role (subscribe upstream, serve the same tables downstream) that chainedtp.q/sctp.q split across themselves in TorQ. Off by default: a plain RDB must not silently become a publisher.

8. The handoff to di.pubsub is additive - unions with whatever the publisher already serves rather than replacing it, because setsubtables replaces its list and unsubscribe deletes registry rows; a registry-only union would drop an unsubscribed table at the next unrelated subscribe. Never shrinks toward empty either, since an empty list makes di.pubsub publish every root table.

9. A handoff failure warns, doesn't throw - it's the last step of subscribe, by which point the subscription has already fully succeeded (schemas defined, log replayed, registry committed). Throwing would discard completed work over an optional secondary step.

10. di.pubsub's .z.pc now chains - a bare .z.pc:{closesub[x]} at load silently destroyed any observer another module had already registered, with di.handlers continuing to report it as live. Depending on di.pubsub as a hard dependency without fixing this would import that risk.

---

Checklist

• 534/534 unit + 131/131 integration assertions passing (di.subscriptions)
• 119/119 assertions passing (di.pubsub)
• di.depcheck clean
• Follows consistency.md and style.md
• Follows dependency injection guidelines
• subscriptions.md documents all exported functions, config, usage examples and notes
• di.servers PR Feature server #120 and di.permissions-equivalent version conventions confirmed compatible

---

Documentation

See subscriptions.md for full reference, including the subdetails/tablelist protocol notes, the shared-log and whole-file replay behaviour, republish configuration, and known non-blocking limitations (live-feed filtering delegation is unverified end-to-end pending di.tickerplant; a shared log with exactly-equal per-table counts is indistinguishable from an ordinary reply and silently under-replays - both carried forward as requirements for di.tickerplant to address upstream).

[DI-Software-Engineering]

getsubtables always throws and never returns the table list: the trap @[{[x] t};::;{[e] symbol$()}]passes::as the argumentxto the lambda{[x] t}, but tis a bare name that — inside module code — is rewritten to.z.m.tat load. The argumentxis never used, so the lambda always tries to read.z.m.t. If .z.m.texists this works, but if the module is loaded in a context where the compile-time rewrite resolves differently, or ifthas not been set yet, the outer lambda throws and the trap always returns`symbol$(). More critically, the argument to @[f;x;g]is::, so fis called asf[::]— which is{[x] t}[::]— not as{[x] t}[]. For a zero-argument read the correct form is @[{[] .z.m.t};::;{[e] `symbol$()}](or simply reference.z.m.t` directly inside the trap).

[DI-Software-Engineering]

resubscribe removes dead rows only for the tables it successfully re-established (done), but the deletion predicate is not active, f'[tabs] — i.e. it deletes every inactive row whose full tabs list is a subset of done. A subscription to multiple tables (e.g. `trade`quote) is only deleted when both are in done; if one of them failed and only the other was re-established, the row is kept permanently, and every subsequent call to resubscribe will re-attempt the failed table and warn about it — exactly the timer-noise the comment above says was fixed. The deletion should match on individual table membership or on the handle, not on the full tabs list.

[DI-Software-Engineering]

requiretablesexist checks wanted where not wanted in tables[\.], but at the call site on line ~565 (if[not alltabs; if[not setschema; requiretablesexist[requested]]]) it is called with requested, which is the tablelist-resolved list — potentially larger than what schemalistwill actually return. The comment on the surrounding code explicitly acknowledges this for thesetschema:0bpost-reply call againstwanted(line ~575), but the pre-reply call still usesrequested. For an explicit request this means a table in tablelistbut absent fromschemalist(a valid case the code later handles withwarnmissing) will cause a spurious early failure even though setschema:0breplay would have succeeded. The pre-reply call should usetabs(the caller's explicit list, before narrowing), notrequested`.

[DI-Software-Engineering]

The middle-process readiness check 1b~MH"MIDREADY" races with the upstream sub.subscribe inside the middle process. The middle script reports its OS port and then runs sub.subscribe, which opens a socket to the peer and replays the log. readport returns as soon as the port file exists — before MIDREADY is set — so midhandle[] and the MIDREADY check can run while the subscribe is still in flight. If the peer or the log is slow, MIDREADY will be 0b or undefined and the assertion fails spuriously. A while poll on MIDREADY (like readport uses for the port file) would close the window.

[DI-Software-Engineering]

DIReview Summary

0 critical | 4 warning(s) | 0 suggestion(s)

⚠️ Spec check skipped — tracker lookup failed (NO_REF_FOUND). Standards axis only.