Skip to content

4676a8c5 - Add bank-processing backlog observer (open transactions monitoring) - #4635

Open
TaprootFreak wants to merge 16 commits into
developfrom
feat/bank-processing-observer
Open

4676a8c5 - Add bank-processing backlog observer (open transactions monitoring)#4635
TaprootFreak wants to merge 16 commits into
developfrom
feat/bank-processing-observer

Conversation

@TaprootFreak

@TaprootFreak TaprootFreak commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Rollout note: deploy this observer first and verify at least two BankProcessing state snapshot: heartbeats in the log pipeline before enabling any downstream consumer of the snapshot lines.

What

Adds a BankProcessingObserver (monitoring subdomain) that watches open bank-processing transactions across five tables — bank_tx, buy_crypto (fiat- and crypto-funded), buy_fiat, fiat_output, bank_tx_return — as 45 declarative invariant rules. Per rule it reports:

  • count / chfSum: current backlog and CHF value
  • overdueCount / overdueChf: entries older than the rule's tolerance (fixed minutes or a time-of-day/weekend-dependent tolerance in Europe/Zurich)

Every 5 minutes the observer emits the results into the monitoring SystemState and writes structured log lines (a summary heartbeat plus one JSON line per rule). The log lines are a monitoring interface: downstream dashboards parse them, and a missing heartbeat means "monitoring is down" — missing data is never treated as "all clear".

Why

This replaces a legacy spreadsheet-based ops monitor that polled /gs/db and mailed alerts through a retired host (silently broken). The rules were re-derived as invariants on the current schema; per-ID exceptions, manual offsets and stale status values from the legacy rules were deliberately not ported.

Notes for review

  • Rules are data (bank-processing.rules.ts); the SQL fragments quote camelCase identifiers for Postgres and were verified against a real Postgres 16 instance.
  • Bundled aggregation: one query per base table (CASE sums), not one query per rule. The six statements are separate snapshots; the accepted one-cycle skew is documented in the observer.
  • The job registers as a worker-scope cron with its own process gate (BankProcessingMonitoring) and an explicit finite lock timeout (1800s, sibling parity). The monitoring state reaches the API process through the persisted snapshot.
  • CHF sums for bank_tx/fiat_output are approximations via fiat.approxPriceChf; non-convertible currencies drop out of the sum (counts stay complete). CHF rounding follows the repo-wide Util.round semantics.
  • Non-finite aggregation values fail loud before anything is emitted; a partial run never produces a heartbeat.
  • Observe-only: no alerting, no health-check gate in this PR (follow-up once the legacy backlog is triaged).
  • The feature is documented for the handbook: docs/bank-processing-monitoring.md (rule model, tolerance semantics, log-line contract, accepted limitations) plus a German metadata entry; the handbook build passes with the new page (21 docs).
  • Side change: docs/cron-jobs.md gains the new job row; two pre-existing missing declarations were added and all inventory counts re-derived (142 = 142 self-check).
  • Existing aml/payment observers keep their coarser counts; this observer does not touch them.

@TaprootFreak
TaprootFreak force-pushed the feat/bank-processing-observer branch from 214a625 to 2c81905 Compare August 4, 2026 00:45
@TaprootFreak

Copy link
Copy Markdown
Collaborator Author

4 review passes until zero findings. Fixed along the way: cron-job inventory completeness (including two pre-existing missing declarations), CHF rounding via the shared Util helper, type-level enforcement of the tolerance field, fail-loud guards for non-finite aggregation values, lock-timeout parity with the sibling observers, and adoption of the new mandatory cron scope parameter after rebasing onto the current develop.

@TaprootFreak
TaprootFreak marked this pull request as ready for review August 4, 2026 01:02
@TaprootFreak
TaprootFreak marked this pull request as draft August 4, 2026 15:10
@TaprootFreak
TaprootFreak force-pushed the feat/bank-processing-observer branch from 41d4f8d to 5e7ee83 Compare August 4, 2026 15:11
@TaprootFreak

Copy link
Copy Markdown
Collaborator Author

Reworked after the handbook landed on develop: rebased, adopted the mandatory cron scope parameter, and added a handbook-discoverable docs page for the feature (rule model, tolerance semantics, log-line contract) with its metadata entry — the handbook build passes with the new page. Two further review passes on the rework brought the total to 6 passes until zero findings.

@TaprootFreak
TaprootFreak marked this pull request as ready for review August 4, 2026 15:38
Pure, unit-tested modules for a new backlog observer over open bank
transactions: ~36 invariant rules (bank_tx, buy_crypto, buy_fiat,
fiat_output, bank_tx_return) with per-rule counts, CHF sums and
age-based overdue tracking, plus a time-of-day/weekend tolerance
helper (Europe/Zurich). Observer wiring follows in the next commit.
Runs the rule catalog every 5 minutes (one bundled aggregation query
per base table), emits results into the monitoring system state and
writes the snapshot log lines consumed by downstream monitoring: a
summary heartbeat plus one JSON line per rule. Own process gate
(BankProcessingMonitoring) with an explicit timeout; a failing block
aborts the whole run so a heartbeat never covers partial data.
- add the new cron job to the docs/cron-jobs.md inventory
- use Util.round for CHF sums instead of a local helper; assert the
  repo-wide half-cent semantics in the tests
- enforce toleranceField at the type level (discriminated union)
- fail loud on non-finite aggregation values before emitting
- type getRawOne and guard against an empty aggregation row
- align the lock timeout with the sibling observers (1800s) and
  document why a finite timeout is mandatory
- document the accepted cross-block snapshot skew
- add missing return types; translate a leftover German comment
…were missing)

The self-check line claimed consistency while two source declarations
(ProcessService::resyncStaffKycClearance, StaffKycClearanceService::
syncStaffKycClearance) were absent from the table. Add both rows and
re-derive every dependent number (134 declarations / 94 files, flag
counts, interval and area tables). Also align two rule labels to pure
German for dashboard consistency.
…tory after rebase

The cron/HTTP process split made scope mandatory on @DfxCron; the
observer registers as a worker job like its siblings. The inventory
was rewritten on develop meanwhile — re-add the new job row against
the new format (142 declarations / 99 files, flag, interval, area
and self-check numbers).
…dbook

New docs page covering the rule model, the tolerance semantics, the
log-line contract (heartbeat plus per-rule JSON, null as the
display-only marker) and the accepted limitations, with a German
handbook metadata entry. The page participates in the handbook's
recursive markdown discovery.
@TaprootFreak
TaprootFreak force-pushed the feat/bank-processing-observer branch from 0105121 to 05f2c59 Compare August 6, 2026 14:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant