feat(providers): Allow injection of external providers.

[tjholm]

Motivation

I maintain a custom buildkit frontend that consumes railpack as a go library and I've hit a point where I would like to augment the current set of providers and iterate on them quickly.

Happy to maintain something like this as a full fork as well if that is more appropriate and changes like this are out of scope for railpack, but thought I would raise this as a PR as provider dependency injection felt like the simpler path.

[iloveitaly]

@tjholm is it open source? Could you point me to it—would like to understand the use case more here.

Anyway we could just integrate your providers into core railpack?

[tjholm]

@tjholm is it open source? Could you point me to it—would like to understand the use case more here.

Anyway we could just integrate your providers into core railpack?

Thanks for the quick reply @iloveitaly, the buildkit frontend I've made just wraps the internals of railpack so I can use llb.Git over llb.Local as context for remote builds. The wrapper itself is here: https://github.com/nitrictech/sugapack . If this is something that would be useful for railpack let me know. (Or if its a capability that already exists in the railpack buildkit frontend already).

For the custom provider stuff, I wasn't looking at necessarily making net new providers, but wanted to implement things like decision provenance inside the existing providers by extending them. So I can surface information in outputs that can help a user determine what might be missing from their project if the output of a build does not meet their expectations. e.g. A tanstack start application missing a start command being treated as a SPA.

Would love to upstream work like this into the existing providers if its suitable, but would still find injection of custom providers useful to rapidly iterate on work like that.

[iloveitaly]

For the custom provider stuff, I wasn't looking at necessarily making net new providers, but wanted to implement things like decision provenance inside the existing providers by extending them.

This is something I'd like to improve in railpack as well, so this would be a great candidate for merging this in / working together on this.

buildkit frontend I've made just wraps the internals of railpack so I can use llb.Git

Using git as an alternative build source is really interesting. I don't see a really strong reason we wouldn't want to add this, assuming it doesn't increase railpack complexity in a big way. It seems like using git as a build source is relatively easy, but I only spent a couple minutes looking at your code. Do you think there would be a lot of downstream changes to support git as a source in railpack? Could be interesting to submit a PR adding support for this so I can gauge complexity and play with it a bit.

Would love to upstream work like this into the existing providers if its suitable, but would still find injection of custom providers useful to rapidly iterate on work like that.

Can you fix CI? Once that is green I can take a look.

[tjholm]

This is something I'd like to improve in railpack as well, so this would be a great candidate for merging this in / working together on this.

Perfect, would love to collaborate on this then.

Using git as an alternative build source is really interesting. I don't see a really strong reason we wouldn't want to add this, assuming it doesn't increase railpack complexity in a big way. It seems like using git as a build source is relatively easy, but I only spent a couple minutes looking at your code. Do you think there would be a lot of downstream changes to support git as a source in railpack? Could be interesting to submit a PR adding support for this so I can gauge complexity and play with it a bit.

Happy to look at upstreaming this, the wrapper I've written is very thin so it should be relatively trivial, when I get some time I'll take a look.

Can you fix CI? Once that is green I can take a look.

Have rebased and ensured tests are running locally, but it looks like the workflows need reapproval.

[iloveitaly]

Happy to look at upstreaming this, the wrapper I've written is very thin so it should be relatively trivial, when I get some time I'll take a look.

Awesome! Sounds good.

Tests are passing! Could you point me to where this is used in sugarpack? I'm curious to see where this is used and see why you couldn't just rewrite this single method on your end as opposed to needing this to be changed within railpack.

I'm biased towards not making these sort of changes to allow railpack to be used as a library since this is not ever going to be an officially supported use case, but if there's something basic I can do here I'm happy to help (love what you are doing, just beyond the scope of what makes sense for us to support in terms of being careful about breaking changes, etc). If the changes you need could be made within sugarpack by just copy/paste/editing a handful of small functions I'd prefer that, so seeing where/how you are hooking into railpack and where the existing implementation breaks would be really helpful.

[tjholm]

Happy to look at upstreaming this, the wrapper I've written is very thin so it should be relatively trivial, when I get some time I'll take a look.

Awesome! Sounds good.

Tests are passing! Could you point me to where this is used in sugarpack? I'm curious to see where this is used and see why you couldn't just rewrite this single method on your end as opposed to needing this to be changed within railpack.

I'm biased towards not making these sort of changes to allow railpack to be used as a library since this is not ever going to be an officially supported use case, but if there's something basic I can do here I'm happy to help (love what you are doing, just beyond the scope of what makes sense for us to support in terms of being careful about breaking changes, etc). If the changes you need could be made within sugarpack by just copy/paste/editing a handful of small functions I'd prefer that, so seeing where/how you are hooking into railpack and where the existing implementation breaks would be really helpful.

That's fair, its a relatively small amount of glue code (a small re-implementation of core.GenerateBuildPlan). So happy to maintain that and resync to avoid behavioural divergence.

Happy for this to be closed in that case, and I can look at adding llb.Git support instead, to shift some more useful code I'm maintaining as part of the sugapack wrapper to railpack.

Once I've got a design I'm happy with for recording decision provenance for providers, will reach back out and see if we can colloborate on design that would work well for railpack as well.