Feature server

[ascottDI]

di.servers — TorQ Modularisation PR

Summary

Extracts TorQ's .servers connection management (code/handlers/trackservers.q + servers.q) into a standalone kdb-x module: di.servers. The module dials a static process.csv "phone book", maintains a pool of live handles to peer processes, hands them out by proctype via a selection algorithm, and recovers dropped connections. It satisfies the di.* module contract: one-arg init[deps] dependency injection, strict validation with no fallback, a conservative exported API, getapimeta for di.api, and no hard di.* dependencies (log, timer and handlers are all injected).

---

Background

TorQ's .servers tracks connected backend processes and lets a gateway (or any process) look up a handle to a peer by type. In TorQ this was entangled with the discovery service, password/access-list files, non-TorQ process tracking, environment reads, and the global process framework.

This PR is part of the broader TorQ → kdb-x modularisation effort. di.servers is a FRAMEWORK-tier module scoped down for v1: no discovery service, no password/access-list files, no non-TorQ process tracking, no FinSpace, and no environment reads. process.csv is treated as a static phone book (who to dial), not an identity source — a process's self-identity arrives via config, injected by di.torq. This makes connection management independently loadable, testable, and drivable from resolved config rather than TorQ globals.

---

Changes

New files

File	Description
di/servers/servers.q	Core implementation — init, startup, getservers, gethandlebytype, waitfortype, getapimeta, plus internal helpers (retry, cleanup, opencon, formathp, readprocesscsv, retryrows, selector, updatestats, signalfound, raiseerror)
di/servers/init.q	Module entry point — loads servers.q and declares the export list
di/servers/test.csv	k4unit test manifest (10 areas; value + error-path asserts + live-peer integration)
di/servers/test.q	Test helper (spins up / drives real peer processes for the live-handle tests)
di/servers/servers.md	Module README

Differences from TorQ original

Aspect	TorQ .servers	di.servers
Logging	Hard-coded .lg.o / .lg.e calls	Injected log dependency, three flat vars (.z.m.loginfo/logwarn/logerr), binary {[c;m]}
Handler registration	.z.pc via .dotz.set	Injected handlers dep; .z.pc registered as a simple/observer event via register[event;phase;nm;pri;func] (fan-out, side-effect only)
Retry timer	.timer framework	Injected timer dep; timer[`addjob][`custom] 6-arg variant, a 10s (mode-1, seconds) serversretry job
Server state	Global .servers.SERVERS table	.z.m.SERVERS module-local mutable state
Self-identity	Derived from process.csv / discovery	From config (proctype/procname in deps); process.csv is a dial-only phone book
Environment	Reads env for paths	Reads no env; process.csv path arrives via configprocesscsv` (resolved by di.torq)
process.csv parsing	Full/variable TorQ layout	Strict v1 host,port,proctype,procname 4-column layout; header validated and rejected loudly (positional read would otherwise misparse silently)
Dependency wait	startupdepcycles	waitfortype[proctype;timeoutms;pollms] — "fail fast, but wait for a hard dependency"; caller decides if timeout is fatal
Scope	Discovery, access-lists, non-TorQ tracking, FinSpace	All removed for v1 (no unexercised branches shipped)
Module contract	None	kdb-x use singleton, strict init[deps] (no fallback), raiseerror log-then-signal, export: list, getapimeta

---

Exported API

svc:use`di.servers

svc.init[deps]                        / wire injectables + config; required before all other calls. idempotent
svc.startup[]                         / open connections to configured proctypes from process.csv (reads init config)
svc.getservers[proctype]              / live (non-null handle) SERVERS rows for a proctype
svc.gethandlebytype[proctype;sel]     / one live handle via `any/`roundrobin/`last selection; 0Ni if none
svc.waitfortype[proctype;tmoms;polms] / block until a proctype connects or timeout elapses; 1b/0b
svc.getapimeta[]                      / api metadata rows for di.torq to register with di.api

init/getapimeta are exported as plumbing (di.torq calls them by convention) but are deliberately not listed in getapimeta[] — the registry describes the callable API, not plumbing.

---

deps (injectables + config, assembled by di.torq)

init takes a single deps dict carrying both the injectable dependencies and this process's resolved config slice (the one-arg convention shared with di.config):

key	kind	meaning
log	injectable (required)	binary `info`warn`error {[c;m]} logger dict — di.log satisfies it directly
timer	injectable (required)	di.timer export; calls timer[`addjob][`custom] (6-arg variant)
handlers	injectable (required)	di.handlers contract; register[event;phase;nm;pri;func]
proctype / procname	config (required)	this process's own identity; used to exclude self from process.csv
connections	config (optional)	proctypes this process should dial (symbols, or strings from a .toml cascade — normalised). Default: none
processcsv	config (optional)	path to process.csv. Required only once connections is non-empty

init wires the deps, records self-identity, and installs two one-time process-global side effects — the .z.pc cleanup observer and the 10s retry job — guarded by an internal registered flag so init is idempotent (a duplicate di.timer.addjob id would otherwise throw). init does not open connections; that is startup's job.

---

process.csv format (v1)

Strict 4-column layout — header validated up front and rejected loudly if reordered or wider:

host,port,proctype,procname
localhost,5010,tickerplant,tp1
localhost,5011,rdb,rdb1
localhost,5012,hdb,hdb1

startup reads this phone book, drops this process's own row (matched on proctype+procname), connects to every row whose proctype is in connections, and records each as a SERVERS row. A failed connect is logged, not raised, and left as w:0Ni for retry to reopen. startup is idempotent — it skips procs already tracked, so a repeat call (or a grown process.csv) adds only new rows, never a duplicate or a leaked second handle.

---

Connection recovery

• Clean drops — the injected .z.pc observer marks the closed handle's row disconnected (w:0Ni, endp stamped).
• Ungraceful drops — the scheduled serversretry job (10s) runs cleanup to sweep handles that vanished from key .z.W without a clean .z.pc, then reopens every dead (null) handle via retry.
• waitfortype reuses retry to block at startup until a hard-dependency proctype comes up (or a timeout elapses).

---

Test coverage

Tests are in test.csv and run via k4unit (the live-handle tests spin up real peer processes via test.q):

k4unit.moduletest`di.servers

Areas covered

Area	Coverage
init — dependency + config validation	Rejects non-dict deps, missing log/timer/handlers, non-dict / partial log, bad timer shape (missing addjob / custom variant), non-dict handlers, missing/non-symbol proctype/procname — all with the "di.servers" error prefix (plain signal; logger not wired yet)
init — wiring and idempotency	Deps/config recorded; second init refreshes refs without re-registering the .z.pc handler or the retry job
startup — live + dead peer	Connects to a live peer, logs (does not raise) on a dead one, excludes self
gethandlebytype — live remote handle	Returns a genuinely usable handle to a running peer; stats bumped
cleanup + retry	Recover an ungracefully-killed peer (vanished from key .z.W)
waitfortype	Connected case returns 1b; timeout case returns 0b
startup idempotency	A repeat call does not duplicate rows or leak a second handle
readprocesscsv fail-loud	A reordered / wider header is rejected instead of silently misparsed
Input validation + getapimeta	Non-symbol proctype/selection etc. rejected; getapimeta[] lists exactly the callable exports (plumbing omitted) with the registry columns
Real di.log integration	Wired against the merged di.log logger itself, not just the recording mock

All green in local KDB-X runs (live-peer suite).

Notes

• di.servers has no hard di.* module dependencies — log, timer and handlers are injected and all required; init signals immediately (plain ', logger not yet wired) if any is missing.
• v1 is TCP only (formathp builds a `:host:port handle); a future SOCKETTYPE config would reintroduce tcps/unix when there is a real requirement and a test — no unexercised branches ship.
• hopen uses the single 2-item timeout form hopen (handle;timeoutms) (the dyadic form throws 'rank); default HOPENTIMEOUT is 2000ms.
• The retry period is 10s via a mode-1 (seconds) timer job — a bare 10000 would have been ~2.8h, the latent typo that made dead-handle recovery never fire in early POCs.
• Module-local SERVERS is mutated as a source-level .z.m.SERVERS (catenate-and-reassign) so it picks up the compile-time module-local rewrite; a symbol-based `.z.m.SERVERS` insert would silently target the wrong table.
• Deferred to later sprints per v1 scope: discovery service, password/access-list handling, non-TorQ process tracking. di.torq owns central di.api registration of the getapimeta[] rows.

[DI-Software-Engineering]

DIReview Summary

0 critical | 6 warning(s) | 0 suggestion(s)

⚠️ Spec check skipped — tracker lookup failed (NO_REF_FOUND). Standards axis only.

[DI-Software-Engineering]

DIReview Summary

0 critical | 1 warning(s) | 0 suggestion(s)

⚠️ Spec check skipped — tracker lookup failed (NO_REF_FOUND). Standards axis only.

[DI-Software-Engineering]

DIReview Summary

0 critical | 3 warning(s) | 0 suggestion(s)

⚠️ Spec check skipped — tracker lookup failed (NO_REF_FOUND). Standards axis only.