Investigate the use of per-model `install_tree`s to cut down on concurrent access issues, cleaned up packages mid-install

[CodeGat]

See previous issue #255

Background

Currently we have a single install_tree for all software in Prerelease and Release.

Due to the large amount of Prereleases locking the install_tree, and the possibility that a closed PR may clean up packages that are currently being installed somewhere else in the install_tree, we should consider splitting the install locations.

The Possibilities

• Split install_trees down repository lines - $spack/../release/ACCESS-OM3, $spack/../release/model-tools, etc. This would still mean that closed PRs may still clean up packages that are currently needed by other PRs, but there will be far less DB locking. However, reuse will be down unless we use other install_trees as upstreams.
• Split install_trees down a repositories PR lines - $spack/../release/ACCESS-OM3/pr123 for Prerelease, and for Release, either:
  • $spack/../release/ACCESS-OM3/2026.01.000 (each install_tree would be a Release)
  • $spack/../release/ACCESS-OM3 (each install_tree is just a model, like in the first solution

Things to be careful of

• install_trees will almost-certainly need to be injected at the environment level. What does this mean for the ukmo-restricted-scope in spack-config, which is currently a static $spack/../restricted/ukmo/release? It would most likely need to be removed and injected at the environment level instead.
• There will be less reuse of existing packages as there is a smaller pool of packages in the install_tree, unless we have a bunch of upstreams. But then we're back in the same boat of upstreams having packages cleaned up potentially mid-install.
  • However, binary buildcaches step around this issue by being globally-accessible, but don't affect concretization decisions. So it would be essentially 'by chance' that there is reuse in this way.
• Modulefiles, our favourite. Our module magic would need to updated to search a wider variety of install_trees. But, (at least in the case of PR-based install_tree splitting), we would delete the un-needed modulefiles by deleting the install_tree - see ci-closed.yml Doesn't Clean Up Modulefiles #155.

Pinging @harshula (as we were discussing install_trees) and @aidanheerdegen for comment.

[aidanheerdegen]

There is a lot to consider.

If we're going to split might as well do it at the PR level. Otherwise we have the same problem, just hit it a bit later.

Less reuse is a problem, but if it can be mostly overcome with caching, then cool!

Module magic isn't a big deal I think, but cleaning up modules would be nice and easy.