[WIP/RFC] Conditional dependencies and package features

[Roger-luo]

Backgrond

Conditional dependency is useful when a package provides optional enhancement based on other packages, e.g Flux.jl allows to use CuArray to accelerate the calculation with CUDA. And in the future, there may be package depends on MKL.jl (MKL has some non-BLAS/LAPACK routines, like batched_gemm).

However, current solution, the Requires package, can not provide explicit dependency in Project.toml, and it does not support multiple package at the moment: JuliaPackaging/Requires.jl#54 thus I believe this can just be a temporary work around.

And it is not convenient and does not make sense to enable the feature by explicitly using several package:

A package Foo depends on CUDAnative and CuArrays when cuda is available, but neither of the following make sense

using Foo, CUDAnative, CuArrays
using Foo, CuArrays
using Foo, CUDAnative

We found this is very in-convenient while developing CuYao.jl and BatchedRoutines.jl, since part of the implementation is implemented with CUDAnative/GPUArray/CuArray, but actually just requires to enable the cuda feature.

This is because Foo may wrap types in CuArrays/CUDAnative, or it will just use CuArray. And what we actually want to do here, is just to precompile and build the CUDA related part of Foo when the feature CUDA is enabled.

Proposal

The proposal here is inspired by rust-cargo. It will add a new field [features] in Project.toml and for each package in Manifest.toml, and also for PackageSpec type.

TOML changes

authors = ["Roger-luo <[email protected]>"]
name = "ConditionalDependency"
uuid = "09c64820-0a14-11e9-376f-03eff1a9e926"
version = "0.1.0"

[extras]
x2 = "52baf49e-96b1-11e8-23dd-2d073a3a6758"
CUDA = "11d27378-0a18-11e9-0a59-4ba122bf6828"

[features]
cuda = ["CUDA"]

[targets]
test = ["x2"]

Since there is a new field for each package, in order to allow developer use a package with specific feature, more detailed information about dependencies in Project.toml will be allowed

authors = ["Roger-luo <[email protected]>"]
name = "ConditionalDependency"
uuid = "09c64820-0a14-11e9-376f-03eff1a9e926"
version = "0.1.0"

[extras]
x2 = "52baf49e-96b1-11e8-23dd-2d073a3a6758"

[extras.CUDA]
uuid = "11d27378-0a18-11e9-0a59-4ba122bf6828"
version = "0.2.0"
features = ["feature1", "feature2"]

[features]
cuda = ["CUDA"]

[targets]
test = ["x2"]

here

CUDA = "11d27378-0a18-11e9-0a59-4ba122bf6828"

will be equivalent to

[extras.CUDA]
uuid = "11d27378-0a18-11e9-0a59-4ba122bf6828"

CLI changes

And a new key word argument for add

(pkg) > add Foo --features cuda mkl

(pkg) > add Foo --all-features

How to know which feature is enabled in the scripts?

A new environment macro @__FEATURES__ will be add to Pkg and it can be used to decide whether to precompile/use part of the code, e.g

module Foo

import Pkg

@static if "cuda" in Pkg.@__FEATURES__
     include("cuda.jl")
end

end

This macro contains code that looks up current environment and returns a vector of string. e.g

If this package is developed in env v1.0, then it will look up v1.0/Manifest.toml which contains the enabled features and put that to @__FEATURES__.

Cases

If this is implemented, we can install Flux in the following syntax to choose different platform.

add Flux --features cuda
add Flux --features tpu
add Flux --all-features # enable them all 
add Flux # default is cpu only

similar for many other packages, it will become more explicit and elegant.

Potential Problems

The extra key word --features in add will need to be the last argument, or the CLI cannot recognize multiple features.

[Roger-luo]

Or maybe just

Pkg.@feature "cuda" include("cuda.jl")

[StefanKarpinski]

Thanks for the PR, @Roger-luo. We definitely needs something like this but I need to think a bit more about the problem. Just wanted to let you know so that you know this isn't being ignored.

[Roger-luo]

@StefanKarpinski Thanks for replying. I'd love to discuss this problem with you if you want, or maybe I could join Pkg's online meeting at some point. We had a painful time while trying to release a package with gpu support. I'm actually working a little bit to make this design work tho. lol.

[ali-ramadhan]

Is this still being considered for merging? This is exactly what we need for Oceananigans.jl as it supports running on CPUs and GPUs but we don't want to add/build CUDA packages if no GPU is available.

It's fine when a user without a GPU adds the package as they just see a bunch of CUDA warnings, but running on mybinder.org/Kubernetes it seems to interpret something as a non-zero return code and it fails to run.

Thank you for working on this @Roger-luo!

[Roger-luo]

@ali-ramadhan thanks, but there's nothing here in this PR for now actually. I was mainly open this to get comments and review on the proposal and once the proposal is decided then people could implement it.

@StefanKarpinski says we need some more discussion on this, but I'm not sure what the exactly timeframe is. I'm willing to join the discussion if there's one about it.

I guess people are busy with some other on-going work at the moment, maybe we could discuss this during JuliaCon, etc.

PS. I think a lot people need this feature ASAP

[jpsamaroo]

I agree with @Roger-luo that certain parts of the ecosystem really, really need something like this (especially the GPU-powered part), but of course one should never rush something like this if proposed solutions are premature.

It would be great to lay out some other ideas for how to implement this functionality (even the not-so-great ones), as well as the upsides and downsides of each approach, so we can discuss this in the open and get some more input from the community.

[Roger-luo]

@jpsamaroo yes, as the title indicates, I'm looking for comments and other potential solutions. My proposal is mainly copied from rust cargo with limited edition myself, so it might not suits Julia very well in some cases.

But I haven't see any other comments on what my proposal itself here... Maybe people just feel this looks good?

[tkf]

Thank you @Roger-luo, for implementing this. I have a few comments/questions:

• An idea alternative to import Pkg; if "cuda" in Pkg.@__FEATURES__; ...; end is to let Pkg.jl create a special build file (say) build/flags.jl which defines some constants and is included from the source file. I can think of two benefits:

  1. People don't have to import Pkg. In the future, Pkg may depend on many more packages and introduces version conflict with your package. It would help you installing a package (with conditional dependencies) that become a legacy code at the time you want to use it.

  2. The change in the features would be detected as stale by the precompile mechanism (but see the next item).

• I think there should be a way to support enabling and disabling features of a certain package per project/environment. To do this with the build/flags.jl approach, I think the slug part in the path to the code ~/.julia/packages/ConditionalDependency/$slug/src/ConditionalDependency.jl has to depend on the set of the features. This requires a support in Base because, IIUC, this is implemented in Base.explicit_manifest_uuid_path; observe that slug only depends on the git tree sha and the version.

  • If this path depends on the enabled features, my previous point about the stale precompilation cache does not matter.

• Or, alternatively, it also makes sense to add Base.@__FEATURES__ if the support in Base is required anyway. See also How precompile files are loaded need to change if using multiple projects are going to be pleasant julia#27418 (comment)

• Question: How do I enable a feature of a certain package in a certain project? That is to say, if I want to use ConditionalDependency in the OP, what do I put in my MyProject/Project.toml? I'd have ConditionalDependency = "09c64820-0a14-11e9-376f-03eff1a9e926" in [deps], but where is the feature flag? Would it be turned on automatically as soon as I install all packages listed in the [features] of ConditionalDependency/Project.toml?

• Question (somewhat overlapping with the previous one): Is it possible to enable a feature of package A depending on CuArrays while disabling a feature of package B depending also on CuArrays?

• Regarding the REPL interface add Flux --features cuda, another possibility is add Flux[cuda] (taken from "extras" support of pip install). It is useful for installing multiple packages with features.

[Roger-luo]

@tkf thanks for commenting!

An idea alternative to import Pkg; if "cuda" in Pkg.@FEATURES; ...; end is to let Pkg.jl create a special build file (say) build/flags.jl which defines some constants and is included from the source file. I can think of two benefits:

I'm using this in BatchedRoutines.jl to support CUDA experimentally with CUDAapi.jl in my build file: https:/Roger-luo/BatchedRoutines.jl/tree/v0.1.0/deps and yes I agree with you on this, the macro here is just some kind of place holder, but I didn't think
about how to deal with those constant flags before.

I think there should be a way to support enabling and disabling features of a certain package per project/environment. To do this with the build/flags.jl approach, I think the slug part in the path to the code ~/.julia/packages/ConditionalDependency/$slug/src/ConditionalDependency.jl has to depend on the set of the features.

What about let each environment contain a set of optional features? which will be enabled by default, it is global for packages inside the environment. I discussed the global sharing issue with @StefanKarpinski , there is a problem is how these feature constant flag propagates by the dependency we haven't make clear at the moment.

Or, alternatively, it also makes sense to add Base.@FEATURES if the support in Base is required anyway.

The idea to have this macro is basically just because there's @__DIR__, I didn't design it with very deep thought. But it is a good point.

Question: How do I enable a feature of a certain package in a certain project? That is to say, if I want to use ConditionalDependency in the OP, what do I put in my MyProject/Project.toml? I'd have ConditionalDependency = "09c64820-0a14-11e9-376f-03eff1a9e926" in [deps], but where is the feature flag? Would it be turned on automatically as soon as I install all packages listed in the [features] of ConditionalDependency/Project.toml?

No, I think the behaviour of each environment should be explicit, unless you type which feature to enable explicitly by add XXX --feature=xxx which generates some constants to the Manifest.toml (or deps folder) the feature will not enabled by just install that package. e.g if I installed Flux with add Flux then I install add CuArrays won't enable the cuda feature of Flux, unless I install Flux again with add Flux --cuda

This is useful because you can have CuArrays in your global environment but you may not want to just happen to enable other features (e.g Flux happen to be broken on cuda on master branch but you need to work on the cpu code of master branch)

Question (somewhat overlapping with the previous one): Is it possible to enable a feature of package A depending on CuArrays while disabling a feature of package B depending also on CuArrays?

Yes, if the feature can be enabled explicitly, then this will just work.

Regarding the REPL interface add Flux --features cuda, another possibility is add Flux[cuda] (taken from "extras" support of pip install). It is useful for installing multiple packages with features.

do you mean I could just write

add Flux[cuda, tpu]

for multiple features?

yeah, this looks better than mine. I thought it was ugly as well. Is this compatible with shell scripts btw? since I also hope our package mode is consistent with shell, so we could have CLIs in the future.

[tkf]

I'm using this in BatchedRoutines.jl to support CUDA experimentally with CUDAapi.jl in my build file: https:/Roger-luo/BatchedRoutines.jl/tree/v0.1.0/deps

Wow, this is a super clever hack!

BTW, an alternative strategy could be something like

const BatchedRoutinesCUDA = try
    Base.require(Base.PkgId(Base.UUID(0x00000000000000000000000000000000),
                            "BatchedRoutinesCUDA"))
catch
    nothing
end

if BatchedRoutinesCUDA !== nothing
    using .BatchedRoutinesCUDA.CuArrays
    using .BatchedRoutinesCUDA.GPUArrays
end

in (say) src/BatchedRoutines.jl where the existence of the extra package BatchedRoutinesCUDA in the current Manifest.toml works as a feature flag. But I've never tried this (so I don't know if it works) and maybe it is not worth trying if you already have a working solution. (There is a potential problem with precompilation when BatchedRoutinesCUDA is added after BatchedRoutines is precompiled, but I can think of a few workarounds.)

What about let each environment contain a set of optional features?

I think we are probably speaking the same thing. I wasn't clear enough. What I was thinking was to add feature attribute to the entries in Manifest.toml like this

[[Flux]]
deps = ["..."]
git-tree-sha1 = "08212989c2856f95f90709ea5fd824bd27b34514"
uuid = "587475ba-b771-5e3f-ad9e-33799f191a9c"
version = "0.8.3"
features = ["cuda"]

Then Base's package loader would take features into account (as well as git-tree-sha1 and uuid) when deciding the precompilation cache path.

do you mean I could just write

add Flux[cuda, tpu]

for multiple features?

Yes, also something like add Flux[cuda, tpu]@0.5 BatchedRoutines[cuda, mkl]#master.

Is this compatible with shell scripts btw? since I also hope our package mode is consistent with shell, so we could have CLIs in the future.

FYI, there's already jlpkg which works very nicely. But I think current Pkg.jl syntax is already incompatible with shell scripts (in the sense that you need quotes sometimes). For example, you need to write jlpkg add 'Example#master' to use this in shell scripts. So I don't think jlpkg add 'Flux[cuda,tpu]' 'BatchedRoutines[cuda,mkl]' is such a big deal (although I think I'm a bit biased due to my experience in pip install).

[Roger-luo]

@tkf

BTW, an alternative strategy could be something like

Yeah, it seems an alternative indeed. But I currently put them to separate packages now, where CuBatchedRoutines would re-export all BatchedRoutines functions, which seems to work better and more explicit as a pretty low level package for routines (it doesn't mean we don't need conditional dependencies tho, cuz I can't make the same program run everywhere without conditional dependencies.)

FYI, there's already jlpkg which works very nicely. But I think current Pkg.jl syntax is already incompatible with shell scripts (in the sense that you need quotes sometimes). For example, you need to write jlpkg add 'Example#master' to use this in shell scripts. So I don't think jlpkg add 'Flux[cuda,tpu]' 'BatchedRoutines[cuda,mkl]' is such a big deal (although I think I'm a bit biased due to my experience in pip install).

Yeah, I tried jlpkg but I think the compatibility with shell should be considered if we are going to ship a CLI with julia in the future. Even this may result in a different syntax between REPL pkg mode since this is something maybe used by non-Julia users to manage your programs, it would be better to make it more consistent with shell.

But yeah [cuda, mkl] is not a very big deal, but I think we should provide shell compatible alternatives for CLI if we are going to ship one with Pkg in the future.

[tkf]

@Roger-luo Given that JuliaLang/julia#37595 is merged and we (will) have https:/JuliaPackaging/Preferences.jl, I think we can close it now? IIUC, combining conditional dependencies #1285 and Preferences.jl will cover your use-cases.

[IanButterworth]

I guess we can close this now that we have package extensions