Skip to content

fix(legacy_openedx): keep the code location loadable when Vault is down - #2526

Open
blarghmatey wants to merge 3 commits into
mainfrom
fix/sentry-legacy-openedx-vault-degraded-config
Open

fix(legacy_openedx): keep the code location loadable when Vault is down#2526
blarghmatey wants to merge 3 commits into
mainfrom
fix/sentry-legacy-openedx-vault-degraded-config

Conversation

@blarghmatey

Copy link
Copy Markdown
Member

What are the relevant tickets?

Sentry: DAGSTER-F

Description (What does it do?)

Part of a set of PRs working through the Sentry backlog from #2518.

This issue logged only one event, which badly understates it: mechanism: excepthook, handled: no, on the dagster-user-code-...-legacy-openedx pod. It is not a step failure — it is the gRPC server for the whole code location failing to start.

_job_default_config returns a default run config so the launchpad is pre-populated for ad-hoc runs, and is written to degrade gracefully when Vault is unreachable at load time. On that degraded path it returned {}:

if not vault_authenticated:
    return {}

But config={} does not mean "no default". Dagster treats any mapping as a default run config and validates it against the job's schema, so an empty one fails:

DagsterInvalidConfigError: Error in config when building job 'residential_edx_course_pipeline':
  Missing required config entries ['ops', 'resources'] at the root.

That validation runs inside build_caching_repository_data_from_listjob.partitions_def_resolve_configs, i.e. while the repository is being constructed. So the fallback meant to keep the location alive was the thing killing it — and it takes down all three jobs and all three schedules, not just the residential one named in the message.

Returning None skips default-config validation entirely and leaves the launchpad unpopulated, which is the degraded behaviour the function was written to provide.

How can this be tested?

Reproduced directly, with Vault pointed at a closed port so authenticate_vault fails fast:

cd dg_projects/legacy_openedx
VAULT_ADDR=http://127.0.0.1:1 VAULT_ADDRESS=http://127.0.0.1:1 DAGSTER_ENVIRONMENT=qa \
  uv run python -c "import legacy_openedx.definitions as d; d.defs.get_repository_def()"
  • Before: DagsterInvalidConfigError: ... missing required fields or contains invalid entries
  • After: repository OK: 3 jobs

Note the failure does not reproduce on plain import — it needs the repository construction step, which is what the gRPC server does at startup.

Both cases are covered by the new legacy_openedx_tests/test_definitions.py (this code location had no tests before):

cd dg_projects/legacy_openedx && uv run pytest legacy_openedx_tests/ -q
# 2 passed

pre-commit run --files <changed> is clean.

Additional Context

Worth confirming after deploy that the legacy_openedx location is actually up in production — the single Sentry event suggests it crashed once rather than crash-looping, which would mean it came back when Vault recovered and this is a latent trap rather than a current outage. The mitxonline_edx_course_pipeline run in DAGSTER-E four minutes earlier indicates the location was healthy immediately before.

`_job_default_config` returned `{}` when Vault could not be reached at
code-location load time, and that empty mapping was handed straight to
`to_job(config=...)`.

`config={}` does not mean "no default". Dagster treats any mapping as a
default run config and validates it against the job's schema, so an
empty one fails with

    Missing required config entries ['ops', 'resources'] at the root

The validation runs while the repository is being constructed, so the
error escaped as an unhandled exception in the gRPC server process --
`mechanism: excepthook`, `handled: no` -- taking down the entire
legacy_openedx code location: all three jobs and all three schedules,
not just the residential one named in the message.

The function exists precisely to degrade gracefully when Vault is
unavailable, so the fallback was defeating its own purpose. Returning
`None` skips the default-config validation entirely and leaves the
launchpad unpopulated, which is the intended degraded behaviour.

Reproduced against the unfixed code with Vault pointed at a closed
port: `defs.get_repository_def()` raises the DagsterInvalidConfigError
above; with the fix it returns all 3 jobs. Both cases are covered by
the new legacy_openedx_tests/test_definitions.py.

Fixes DAGSTER-F

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ssj5MFfDB33GBZod1PYst9
Copilot AI balanced review requested due to automatic review settings August 8, 2026 22:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Keeps the legacy_openedx Dagster code location loadable when Vault is unavailable.

Changes:

  • Returns None instead of an invalid empty default configuration.
  • Adds regression tests for degraded repository loading.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
legacy_openedx/definitions.py Corrects graceful default-config fallback behavior.
legacy_openedx_tests/test_definitions.py Tests repository loading without Vault.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread dg_projects/legacy_openedx/legacy_openedx_tests/test_definitions.py
…ed one

The fixture set environment variables and then imported
legacy_openedx.definitions, but everything it was setting up happens at
import time. A cached sys.modules entry would hand back a module built
under whatever environment imported it first, so the second and later
tests in a session asserted against stale module state and the suite
passed or failed on import order rather than on behaviour.

Evicts the module from sys.modules via monkeypatch, which also restores
the previous entry on teardown so the eviction cannot leak into other
tests. Adds a test that imports twice and asserts the two module objects
differ -- it fails with `assert <module ...> is not <module ...>` when
the eviction is removed.

Raised by Sentry review on #2526.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ssj5MFfDB33GBZod1PYst9
Comment thread dg_projects/legacy_openedx/legacy_openedx_tests/test_definitions.py Outdated
The previous commit fixed the module-cache problem for
legacy_openedx.definitions but not for its dependency.
ol_orchestrate.lib.constants resolves DAGSTER_ENV and VAULT_ADDRESS from
the environment at import time, and definitions imports those *values*,
so evicting only the leaf let it rebind to constants resolved by
whatever imported them first.

The consequence is worse than a stale value: VAULT_ADDRESS would fall
back to the real vault-qa host, turning an immediate connection refusal
into a network timeout and quietly no longer testing the degraded path
at all. The suite passes today only because nothing in this test session
imports constants before the fixture does -- exactly the order
dependence being removed.

Adds a test that primes the cache with constants resolved against a
different address first, the way any earlier import would, then asserts
the module under test still sees the closed port. It fails with
`- http://127.0.0.1:1 / + https://vault-qa.odl.mit.edu` when constants
is dropped from MODULES_TO_EVICT.

Raised by Sentry review on #2526.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ssj5MFfDB33GBZod1PYst9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants