Summary
Currently chaste-codegen's CI only checks generated C++ against frozen reference files but doesn't compile or run the generated code.
Comprehensive numerical end-to-end testing is done with the Chaste/heart/test/ionicmodels/TestCodegenLong suite, which:
- Fetches real
.cellml model files.
- Runs chaste_codegen on each model, compiling the resulting C++ into a shared library and loading it dynamically.
- Runs the generated model and compares results against a fixed reference dataset.
This testing is thorough but can take a while to run (60+ min for the slowestTestCodegenLongGeneralisedRushLarsenSecondOrder). Running the test suite through multiple versions of a dependency like sympy further increases turnaround time.
In order to get fast numerical feedback, it might be useful to supplement the existing numerical tests with relatively lighter checks that only compare results for newly generated reference files against the existing reference baseline e.g. if 5 new variants are introduced by sympy 1.14, these would be checked against the 5 reference baselines that they update. This also fills a gap by directly verifying that all the different renderings of reference models stored in the chaste-codgen repository are numerically equivalent. The fast check can run on PRs cheaply to catch numerical regressions early (possibly using the chaste/docker image which already has Chaste compiled). Once all problems are fixed at this level, testing can then move on to the more thorough end-to-end suite.
Summary
Currently chaste-codegen's CI only checks generated C++ against frozen reference files but doesn't compile or run the generated code.
Comprehensive numerical end-to-end testing is done with the
Chaste/heart/test/ionicmodels/TestCodegenLongsuite, which:.cellmlmodel files.This testing is thorough but can take a while to run (60+ min for the slowest
TestCodegenLongGeneralisedRushLarsenSecondOrder). Running the test suite through multiple versions of a dependency like sympy further increases turnaround time.In order to get fast numerical feedback, it might be useful to supplement the existing numerical tests with relatively lighter checks that only compare results for newly generated reference files against the existing reference baseline e.g. if 5 new variants are introduced by sympy 1.14, these would be checked against the 5 reference baselines that they update. This also fills a gap by directly verifying that all the different renderings of reference models stored in the chaste-codgen repository are numerically equivalent. The fast check can run on PRs cheaply to catch numerical regressions early (possibly using the chaste/docker image which already has Chaste compiled). Once all problems are fixed at this level, testing can then move on to the more thorough end-to-end suite.