Skip to content

Exclude prepend from Conductor testcase error line numbers - #4291

Open
Shrey5132 wants to merge 2 commits into
source-academy:masterfrom
Shrey5132:fix/conductor-testcase-prepend-line-offset
Open

Exclude prepend from Conductor testcase error line numbers#4291
Shrey5132 wants to merge 2 commits into
source-academy:masterfrom
Shrey5132:fix/conductor-testcase-prepend-line-offset

Conversation

@Shrey5132

Copy link
Copy Markdown
Contributor

Summary

  • runTestCaseConductor concatenates prepend/student code/postpend/testcase into one combined file before sending it to the Conductor evaluator, but never subtracted the prepend's line count back out of a resulting error's reported line - so an error on the student's own first line was reported at prepend line count + 1.
  • This is the same off-by-N bug Assessments: In error messages, PREPEND lines are counted #4244/Correct off-by-one prepend line shift in Conductor error messages #4245 fixed for the plain Run path (evalCode.ts's toConductorSourceError/preludeLineOffset), but that fix doesn't reach this call site: it's gated on an editor Run's actionType, and runTestCaseConductor calls evalCodeSaga with EVAL_SILENT instead. Testcase/grading errors in assessments were still shifted.
  • Shifts the line back out locally in runTestCase.ts, using the same prepend value this saga already concatenates into the combined file. A location's line of 0 (toConductorSourceError's "no location info" fallback) is left untouched rather than shifted into a misleading line 1.

Test plan

  • Added a test asserting a testcase error's line number is corrected back to the student's own line (multi-line prepend case).
  • Added a test asserting a line-0 (unknown location) error is left unshifted.
  • yarn test src/commons/sagas/WorkspaceSaga.test.ts - 69 passed.
  • yarn eslint on changed files - clean.
  • tsc --noEmit - clean.

https://claude.ai/code/session_01FYm9pttcp3nixtDLKiPh2k

runTestCaseConductor concatenates prepend/value/postpend/testcase into one
file before sending it to the Conductor evaluator, but unlike evalCode.ts's
Run-path fix for the same off-by-N bug (source-academy#4244), nothing subtracted the
prepend's line count back out of a resulting error's reported line - so an
error on the student's own first line was reported at `prepend line count +
1`. evalCodeConductorSaga's own prepend-offset correction doesn't reach this
path: it's gated on an editor Run's actionType, and runTestCaseConductor
calls evalCodeSaga with EVAL_SILENT instead. Shift the line back out locally
in runTestCase.ts instead, using the same prepend value this saga already
concatenated into the combined file.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FYm9pttcp3nixtDLKiPh2k
@coderabbitai

coderabbitai Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 81a36396-3318-46cc-9eef-e849aacaad37

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: aadd7bfd-5767-4d15-a2eb-9da3ebfd3842

📥 Commits

Reviewing files that changed from the base of the PR and between e4f8be1 and d5a658e.

📒 Files selected for processing (2)
  • src/commons/sagas/WorkspaceSaga.test.ts
  • src/commons/sagas/WorkspaceSaga/helpers/runTestCase.ts

📝 Walkthrough

Summary by CodeRabbit

  • Bug Fixes
    • Corrected error line numbers reported for testcase failures when files include prepended content.
    • Preserved line 0 and other non-positive locations when no usable error location is available.
    • Improved the accuracy of error locations for Conductor testcase runs.

Walkthrough

The change corrects Conductor testcase error locations by removing non-empty prepend lines from positive line numbers. It preserves columns, location boundaries, and non-positive line values. Regression tests cover shifted and line 0 errors.

Changes

Conductor error location correction

Layer / File(s) Summary
Correct testcase error locations
src/commons/sagas/WorkspaceSaga/helpers/runTestCase.ts
runTestCase calculates non-empty prepend lines, shifts positive SourceError line values, and dispatches corrected errors.
Validate error location behavior
src/commons/sagas/WorkspaceSaga.test.ts
Tests verify prepend removal, column and boundary preservation, and unchanged line 0 locations.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to d5a65

The PR makes a localized correction to testcase error line reporting, with validation checks passing; no actionable merge-blocking risk remains.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main fix for Conductor testcase error line numbers.
Description check ✅ Passed The description explains the bug, implementation, motivation, and reproducible test results, although it omits the template's type and checklist sections.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FYm9pttcp3nixtDLKiPh2k
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant