[feature] Run Conan from generic CMakeToolchain file and chainload the generated

[heilkn]

Dear conan team,

I would like to propose the following feature in order to make Conan easier to use and adopt. The idea is based on a workflow implemented by vcpkg.

The idea:

• The user provides a toolchain file that is shipped with conan to CMake.
• The user may define additional CMake variables. For example, the path to a conanfile and a profile.
• The generic toolchain file runs Conan which produces a project related toolchain and chainloads it.

I have implemented a prototype for this here: https:/heilkn/conan-toolchain/blob/main/src/conan-toolchain.cmake
You can find an example project using this toolchain file in the test directory of the repository https:/heilkn/conan-toolchain/tree/main/test Please see the CMakePresets.json how the toolchain file and additional variables are passed to CMake and to the toolchain file.

Problems with the prototype

• For some reason, I have to use the cmake_find_package generator in the conanfile, because the CMakeDeps generator is making the dependency available, but the imported target does not define the include directories.

My Motivation

I am working with conan for some time already privately and in a bigger corporation with several development teams. My experience has been that I could only introduce Conan because I could make it as transparent as possible for the developers working with me on the projects. Part of the solution was cmake-conan. Unfortunately, that has its own problems and will be deprecated with Conan 2.0. Furthermore, vcpkg is getting more traction in the moment, because it seems to be even easier to use.

I have the impression that the reason for vcpkg's popularity is the ability to point CMake at a generic toolchain file that is installed together with it and rely on the toolchain to manage the everything else.
Developers have the option to pass the toolchain either on the command line when invoking CMake directly, or simply put into the CMakePresets.json. In the latter case, all a developer needs to do is open the CMake project in his IDE of choice and select a CMake configure that is pre-configured to invoke vcpkg.
In this scenario, on-boarding new developers is then simply a matter of installing the required tools, checking out a repository and opening the project. That's it.

However, I am convinced that Conan has for the moment the better solution to handle ABI. Furthermore, some of our teams do already have a collection of recipies and binaries deployed to an Artifactory instance. For that reason, I would like to explore the idea of introducing the ease of use of vcpkg via a generic toolchain file that could be shipped with a future conan installation.

Kind regards

Konstantin

• I've read the CONTRIBUTING guide.

[memsharded]

Hi @heilkn

Thanks very much for your suggestion, and for proposing some approach.

I would like to understand better a few points, for example:

Part of the solution was cmake-conan. Unfortunately, that has its own problems and will be deprecated with Conan 2.0.

Yes, we would like to deprecate it, because it has become a chicken-and-egg problem with the generation of the conan_toolchain.cmake that CMakeToolchain is creating. You haven't mentioned it, but this is an important generator, which is necessary to be able to do builds that change depending on the Conan settings, which is a very necessary thing, when the dependencies have to be built.

It would be interesting to follow the conversation and join forces in conan-io/cmake-conan#447, it is still not fully decided what will happen, if it is possible some viable alternative, of course we will explore it.

Furthermore, vcpkg is getting more traction in the moment, because it seems to be even easier to use.

Might be true for third-party open source dependencies, and for some cases, mostly the "consuming" cases it can indeed be easier to use. But for many other cases, the more challenging they become, Conan can provide more power and flexibility, and Conan is growing super fast in the enterprise world. But we are aware it has a cost for some of the simpler cases, and we would like to improve over that.

I like some of what I see in the proof of concept, like using profiles as inputs. But I wonder how the conversion of that profile to the actual CMake configuration, provided by CMakeToolchain would be implemented. How something like -o pkg:shared=True will be passed to work correctly, or other stuff like architecture, etc. Once a toolchain has been passed, it is not possible to inject that anymore.

I think the best is to transfer this to the cmake-conan repo, to centralize discussion there, the goal is exactly the same, and it is not really a feature in the Conan client, wdyt?

[heilkn]

Dear James (@memsharded),

thank you for your attention, I am glad that you have interest discussing this. Before we discuss further technical details let's clarify where this discussion should happening. I would be fine with either, but I posted in here on purpose. Please let me explain why.

It was not my intent to make a statement about what is easier to use. I wanted to report from my experience which I got talking to my colleagues. Especially from developers who did not had interest maintaining a library and support as many platforms and build systems as possible, but from developers which needed to consume our library among others in their builds for a very specific platform and integrate it into their application so they could move along working on the application code. Their workflow looks easy: Define a json with the dependencies and pass the path to the toolchain file shipped with vcpkg. Remarkably, the path is always the same, on every project and most likely on any computer as well if you follow a standard procedure to install the software.

I think that this is an important aspect which should decide if the solution should be part of the client or a separate repository. Let me elaborate on this a little bit more. You can then decide if you want to continue here or move over to cmake-conan or to another project entirely.

You were mentioning that one of the problems with cmake-conan is the chicken-and-egg dilemma. However, you were only focusing on the fact, that you need to generate the toolchain invoking the CMakeToolchain generator, at which point it is already to late. There is another important problem as well. You need to bring in the CMake script itself and that would require to put special conan related configuration into a projects CMakeLists. This is something that is a no-go for many teams I know of. We tried shipping cmake-conan as a conan package that registers itself in the CMake package registry (.cmake directory) and requiring a project configuration only to make a call to find_package(cmake-conan). However, that introduced a second level of the chicken-and-problem and brought lot's of own quirks.

I hope you found my report useful. Please let me know where to continue.

Kind regards

Konstantin

[hwhsu1231]

@heilkn

I have the same feelings as you. I do think the current usage of Conan is inconvenient to adopt (especially without cmake-conan). However, my idea is to rewrap the APIs of cmake-conan into a CMake toolchain. Therefore, I published the following project, which can solve the Chicken-And-Egg problem and integrate CMake with Conan Nonintrusively and Transparently. You can take a look.

https:/hwhsu1231/cmake-conan-setup

@memsharded

I posted an issue on cmake-conan about my willingness to transfer this project to Conan Team. Please take a look.

conan-io/cmake-conan#449

[heilkn]

Dear @hwhsu1231, thank you for your support. At the moment, I am hesitant to continue working on cmake-conan, because I basically do not agree with it's goals. It tries to circumvent regular conan files and places the configuration into the CMake configuration. To the contrary, I am in favor of making the projects CMake configuration completely oblivious to where the dependencies come from.

@memsharded I would like to move this discussion only over to cmake-conan if there's a chance to getting more attention. Otherwise, I think this should be discussed independently of the old tool.

Kind regards

Konstantin

[memsharded]

@memsharded I would like to move this discussion only over to cmake-conan if there's a chance to getting more attention. Otherwise, I think this should be discussed independently of the old tool.

Yes, I now understand a bit better the intention, and I agree it diverges enough from the cmake-conan current approach, to make it independent. I agree with the part that I don't like either contaminating the CMakeLists.txt with a bunch of things, the CMakeLists.txt should be pretty agnostic of where the dependencies come from.

Looking forward to exploring these other new ideas (though it will require some time to have some bandwidth for it, please be patience). Many thanks!

[heilkn]

Hi! I am glad you are still intersted. I have pushed an update to the linked repository. The prototype now supports building missing dependencies and I added a comparision with the vcpkg workflow.

Please provide feedback.

Kind regards

Konstantin

[heilkn]

Hi everyone, I've updated the project with a Readme file stating the proposed workflow and an example how-to build the test application. Please take a look and comment.

I am currently waiting on @calebkiage to provide a license for his project. Depending on that, I would try to copy the conanfile detection function and maybe some of the other supported parameters.

EDIT I've added a Github action, where you can see the code running: https:/heilkn/conan-toolchain/actions/runs/3624767550/jobs/6112096620

It seems that the conan toolchain forgets the compiler set in the profile. See the artifacts produced by the build: https:/heilkn/conan-toolchain/actions/runs/3624767550
@memsharded Do you think this might be a bug?

[heilkn]

On something different.

The biggest puzzle for me, when using cmake-conan in the past, was how to properly link cache variables in CMake and conan package options defined in the conanfile.

With cmake-conan, we explicitly settled on passing them from the CMakeLists file to the conan invocation. However, this does not fit well with the goal of keeping the CMake configuration files oblivious of conan.

We could parse an additional file that defines the mapping when invoking the conan CLI. However, I am not sure this is the best idea. Maybe, someone has another suggestion?