Skip to content

Conductor testcase grading can read stale output from a previous testcase run #4180

Description

@Akshay-2007-1

Summary

runTestCaseConductor in src/commons/sagas/WorkspaceSaga/helpers/runTestCase.ts grades a Conductor testcase by inspecting state.workspaces[workspaceLocation].output, which is never cleared between testcase runs within the same grading session. Both the success path and the error path only look at entries from the entire accumulated output history, not just the entries produced by the current run.

const output: Array<{ type: string; consoleLogs?: string[]; errors?: any }> = yield select(
  (state: OverallState) => state.workspaces[workspaceLocation].output,
);
const lastOutput = output[output.length - 1];

let passed: boolean;
if (lastOutput?.type === 'errors') {
  yield put(actions.evalTestcaseFailure(lastOutput.errors, workspaceLocation, index));
  passed = false;
} else {
  const lastLogOutput = [...output].reverse().find(entry => entry?.consoleLogs?.length);
  ...
}
  • The error check only inspects the trailing entry, so if an errors entry from this run isn't the very last one in the array (stdout and result messages travel on separate Conductor channels with no guaranteed relative ordering, per the existing code comment), it can be missed entirely.
  • The success path searches backward through the whole history for the first entry with consoleLogs. If the current run's own code produces no console output at all (an empty/erroring testcase, or output delayed past when this saga reads the array, e.g. via set_timeout), the search can walk back into a previous testcase's leftover output and grade against that instead, silently producing a wrong pass/fail.

Suggested fix

Snapshot output.length before calling evalCodeSaga for this testcase, then only inspect the slice appended since (output.slice(baseline)) for both the error check and the printed-output search, instead of scanning the full accumulated history.

Context

Flagged by CodeRabbit during review of #4057 (Conductor-aware Run + testcase grading). Tagged "Major" / "Heavy lift" — real correctness gap, but conditional (only bites when a testcase's own run produces no new output before grading reads the array) and needs touching the shared eval/output pipeline plus new test coverage, so it's being tracked here rather than blocking that PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions