Submit one thread run end to end against TACC's Ensemble Manager

[mosoriob]

Question

Run submission has never completed end to end against a live Ensemble Manager. #85 and #88 each
fixed a defect on this path, and neither fix has been exercised against a real server. This is
the last item of #76's checklist that #76 did not close.

It is not the SOW acceptance criterion — §8 is the single-form model input config, which
passes (#76). This is about not shipping an unproven path.

What is already settled, so this is a short walk

Measured while resolving #76, signed in against TACC:

• The routes are exactly what ui-react submits runs to /executionEngines/localex, a route that does not exist #88 predicted. TACC's Ensemble Manager publishes
  /executionEngines/tapis, /executions and /executionsLocal — three separate mounts, no
  shared prefix. executionEnginePath maps them correctly.
• EXECUTION_ENGINE: tapis is right for TACC. Its live Lit config at
  mint.tacc.utexas.edu/config.js sets REACT_APP_EXECUTION_ENGINE = "tapis". The constraint
  ui-react submits runs to /executionEngines/localex, a route that does not exist #88 put on Flip mint.tacc.utexas.edu to ui-react #77 stands.
• The Tapis token attaches. Every Hasura call from the signed-in app carried
  Authorization: Bearer …, so the Ensemble Manager calls send no Authorization header — localStorage key mismatch #85 fix holds in the browser.
• The thread wizard renders against TACC's real problem statements, tasks and sub-tasks.

The one caveat worth knowing before you run it

TACC holds 264 executions: 263 localex and 1 tapis. So the Tapis submission path has
essentially never been used there, by Lit or anything else. A failure may well be TACC's Tapis
configuration rather than ui-react.

Why this is not done yet

Submitting creates a real job on TACC's cluster. That is a bigger side effect than the catalog
test rows #76 wrote and removed, and it was not covered by the go-ahead for those. It needs an
explicit decision on whether to fire a real run at the client's production deployment, and which
model to fire.

End to end testing / hello world file input test is the obvious candidate — it is a
sleep-and-echo fixture, it is the config #76 used, and it has a required and an optional input.

What to check when it runs

• Submit returns 2xx, not the 500 that Ensemble Manager calls send no Authorization header — localStorage key mismatch #85 showed is how an anonymous submit surfaces.
• The execution row appears with execution_engine: tapis.
• The Runs step polls status without a 401 — the other half of the Ensemble Manager calls send no Authorization header — localStorage key mismatch #85 fix.
• The Results step resolves the output location. Note MintResults reads the engine off the
  execution row rather than config, and its localex/wings path rewrite is dead code, so it
  should need nothing.

[mosoriob]

Blocked. No run was submitted. The wizard cannot reach the Runs step, so the submit path is
still unproven. Now blocked by #94.

Run signed in as mosorio@portals on localhost:3000, branch develop (4f7debf + 1d1ada9),
with ENSEMBLE_MANAGER_API pointed at TACC.

Why it is blocked

The Datasets step cannot be satisfied, and Runs stays locked until every required input has a
dataset. Two separate faults, and only the second is ours.

1. The chosen model has no data at TACC. hello world file input test needs
land_surface_terrain~left_channel~stream_reach_water__volume_flux. TACC's CKAN holds 215
packages; 33 carry annotations, covering 40 distinct standard variables. None is that one. So
picking a different model was needed anyway — this ticket's candidate cannot run at TACC.

2. The dataset lookup is broken for almost every variable — #94. ui-react searches CKAN
free text for the variable name. CKAN does not index the mint_standard_variables resource field.
Lit filters on that field instead and never sends q. Measured: ui-react returns exactly the
annotated datasets for 3 of 40 variables. 28 return zero. Some return only false positives.
Full evidence on #94.

Fault 2 means changing the model would not have rescued this ticket. Both faults had to be
found before the run could be attempted.

What was confirmed, signed in against TACC

These held, so the #85 and #88 fixes look right up to the point the run stops:

• The routes are exactly as ui-react submits runs to /executionEngines/localex, a route that does not exist #88 predicted. Probed live: /executionEngines/tapis → 401
  (exists, wants auth), /executionEngines/localex → 404, /executionsLocal → 400,
  /executions → 400. The pre-ui-react submits runs to /executionEngines/localex, a route that does not exist #88 code posted to the 404.
• EXECUTION_ENGINE: tapis is right for TACC. Its live Lit config at
  mint.tacc.utexas.edu/config.js sets REACT_APP_EXECUTION_ENGINE = "tapis". The Ensemble
  Manager is https://ensemble-manager.mint.tacc.utexas.edu/v1.
• Login works at localhost:3000. The OAuth redirect carries client_id=mint-localhost-3000
  and redirect_uri=http://localhost:3000/oauth2/callback. Tapis now enforces MFA.
• The wizard renders against TACC's real problem statements, tasks and sub-tasks, and the
  model picker finds hello world file input test.

Still unproven

Everything this ticket exists to check: the 2xx on submit, the execution row with
execution_engine: tapis, the Runs poll without a 401, and the Results output location.

Note for whoever takes this next

Pick a model whose input variable is annotated at TACC. Candidates with real data:
groundwater__hydraulic_head (6 datasets), groundwater__recharge_volume_flux (5),
aquifer__hydraulic_conductivity (4). Do this after #94, or the wizard will not find them.

Also worth knowing: the region filter defaults to California, and widening it to "any region"
changes nothing here — the constraint is the annotation, not geography.

State left behind

I created sub-task wayfinder-92 tapis submit check under problem statement test to avoid
editing Max's existing sub-tasks, then deleted it. TACC's catalog is back to its prior state. No
job was submitted to the cluster.

[mosoriob]

A local MicroK8s dev cluster is available as a rehearsal target for this ticket.

Its Ensemble Manager runs execution_engine: tapis, the same backend TACC runs. So it exercises the same submission route. Its data catalog is also TACC's CKAN, not a local one.

It also confirms #88 on real infrastructure. /v1/executionEngines/localex returns 404. /v1/executionsLocal, /v1/executions and /v1/executionEngines/tapis all exist.

It does not unblock this ticket. #94 is a client-side matching fault, so the Datasets step stays empty against any cluster. #94 must land first.

One limit on the rehearsal. The dev cluster hostnames are not publicly resolvable, so Tapis refuses to create job subscriptions against them. The values file sends callbacks to a webhook inspector instead. You can rehearse submission. Job status callbacks do not return to the Ensemble Manager.