.west/config west.yml and zephyr versioning during project development

[cederom]

Describe the bug

Current west approach does not allow easy version selection of the zephyr. Either I need to install global zephyr installation and provide ZEPHYR_BASE on where is is located, or use manifest approach and use local copy of zephyr within my project source tree. The problem arises when I need to switch to a different version of zephyr i.e. from v2.4.0 to v2.5.0 or v2.6.0-rc1. I either have to git checkout version_branch on the global repository and this forces all other projects to use this version of zephyr, or I have to git checkout version_branch in my local copy of the zephyr source tree within a project.

In a commercial projects we cannot use simply master branch. We need to select one specific version, then after testing when new release comes up we may switch to that new release. But a specific zephyr version dependency is a must and coherently controlled with one simple text file and some script.

Zephyr version needs to be arbitrarily selected on project creation and then there is no easy way to change the zephyr version later on. In perfect world I would have west.yml for my project stored on a git repo, then when we change the zephyr version we change the version number in west.yml, developers pull that change, perform west update, and everyone uses now the same version of zephyr.

Zephyr version selection should be provided in some sort of configuration file that is part of the source code repository. This way version change is consistent. We do not have a problem that someone has different version of the zephyr source code so for someone code builds and work and for someone else it builds but does not work or it does not build at all. No manual intervention in the local git zephyr pull should be necessary as this leads to confusion and problems.

Documentation on project file structure needs to be updated and extended with information on how the local project depends on zephyr versioning exactly and how this can be set in a consistent way with possibility to switch later during the code development based on a text file configuration variable.

None of the examples contain west.yml file nor there is an example that shows how to build for a specific version of the zephyr based on the local project configuration (nor how to build for a different zephyr releases).

I may be missing something or this is not yet available in zephyr? :-)

To Reproduce
Steps to reproduce the behavior:

1. Create a local project, use the west init approach with a specific version provided.
2. Develop your code.
3. When new release shows up try to setup your project to use that new release.
4. See the problem :-)

The most visible scenario is when a new developer wants to clone the local project repository to work on, normally it will be git clone to obtain the project code, then west update to get all zephyr source dependencies. This is impossible at the moment. We need to manually west init a project with a specific version of zephyr. That version of zephyr does not seems to be configurable during project development, except each developer need to perform git checkout zephyr_version by hand. This also may impact current manifest coherence.

Expected behavior
Zephyr version, a critical local project dependency, is provided within a configuration file and it can be easily updated during code development in a consistent and automated way.

Impact

• Different developers work on different versions of zephyr (the version that was selected or used when repository was initialized).
• Sometimes it results in build errors when API changed.
• Sometimes code builds fine but does not work at all.
• Zephyr is a hard underlying dependency for our project and there seems no way to set or change the zephyr version after some months.
• Documentation on setting specific version of the zephyr seems to be missing.
• Documentation on local project structure seems spreaded over introduction, manifest, and west sections, but there is no place where all informations are gathered together. Also no example uses specific version of zephyr - everything is simply tied to master / commit time. Developers are confused.

Environment (please complete the following information):

• OS: FreeBSD.
• Toolchain: gcc-arm-embedded

Additional context
Presented above.

Any hints welcome :-) Thank you :-)

[cederom]

Also statement presented in https://docs.zephyrproject.org/latest/guides/west/basics.html#west-update-basics does not seem to be true:

This command makes sure your workspace contains Git repositories matching the projects in the manifest file.

To verify:

• Please create a project with provided west init -m https:/zephyrproject-rtos/zephyr --mr v2.5.0 zephyrproject.
• Then put that project on a git repository of your team.
• You do not want to store all zephyr code so you keep empty zephyr on commit.
• Then pull this repository on another computer and run west update to see there is no such file.
• Another quick-fix is to put zephyr as git submodule, then fetch it, then west update.
• Still there is no consistent way to manage your local project that depends on a specific version of zephyr.

[cederom]

When you want to use .west/config with following contents so you can control the zephyr version:

[manifest]
path = .
file = west.yml

And the west.yml

manifest:
  remotes:
    - name: zephyrproject-rtos
      url-base: https:/zephyrproject-rtos
  projects:
    - name: zephyr
      remote: zephyrproject-rtos
      revision: v2.5.0
      import: true

Then west build will always add below to the .west/config and build is impossible because or circular dependencies:

[zephyr]
base = zephyr

[cederom]

My scenario:

• I have myproject stored on a git server that is fetched to the development workstation.
• I have created myproject/.west/config:

  1 [manifest]
  2 path = .
  3 file = west.yml
  4
  5 [build]
  6 board = nrf52dk_nrf52832

• Then myproject/west.yml:

  1 manifest:
  2   remotes:
  3     - name: zephyrproject-rtos
  4       url-base: https:/zephyrproject-rtos
  5   projects:
  6     - name: zephyr
  7       remote: zephyrproject-rtos
  8       revision: v2.5.0 #v2.6.0-rc1 #master
  9       import: true #west.yml

• I am using following CMakeLists.txt:

  4 cmake_minimum_required(VERSION 3.13.1)
  5
  6 find_package(Zephyr REQUIRED HINTS $ENV{ZEPHYR_BASE})
  7 project(myproject)
  8
  9 FILE(GLOB app_sources src/*.c)
 10 target_sources(app PRIVATE ${app_sources})

• On west update zephyr is pulled into myproject/zephyr along with myproject/bootloader, myproject/modules, myproject/tools. That sounds fine and I get exactly the desired selected version from my myproject/west.yml.

• But. Whenever I run west build west seems to modify the myproject/.west/conf:

  1 [manifest]
  2 path = .
  3 file = west.yml
  4
  5 [build]
  6 board = nrf52dk_nrf52832
  7
  8 [zephyr]
  9 base = zephyr

and that results in build failure:

-- Generated device_extern.h: myproject/build/zephyr/include/generated/device_extern.h
Parsing myproject/zephyr/Kconfig
myproject/zephyr/scripts/kconfig/kconfig.py: Kconfig:8: recursive 'source' of 'Kconfig.zephyr' detected. Check that environment variables are set correctly.
Include path:
myproject/zephyr/Kconfig:8
Kconfig.zephyr:23
modules/Kconfig:6
myproject/build/Kconfig/Kconfig.modules:2
Kconfig:8
CMake Error at zephyr/cmake/kconfig.cmake:265 (message):
  command failed with return code: 1
Call Stack (most recent call first):
  zephyr/cmake/app/boilerplate.cmake:536 (include)
  zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:24 (include)
  zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:40 (include_boilerplate)
  CMakeLists.txt:6 (find_package)


-- Configuring incomplete, errors occurred!
FATAL ERROR: command exited with status 1: /usr/local/bin/cmake -DWEST_PYTHON=myproject/build -Smyproject -GNinja

What do I miss? :-)

The reason not to use global Zephyr repository shared among all zephyr related project it that I can easily switch to a particular version of Zephyr only for that specific project. Also this is the simplest way to setup working environment on a development workstation. I also have a helper script that sets up the python venv and simply puts a developer into the venv + zephyr sdk (activate for python and zephyr/zephyr-env.sh that does not get pulled by west update I just noticed).

Also the reason of this kind of approach, unlike in all sample programs, is that I want to use external modules/projects to use with my application and Zephyr (i.e. Nordic UART Service to use Shell over BLE) that can be easily controlled with my project's myproject/west.yml.

Update 1
I have moved all of my project code to myproject/myproject including cmake, conf, etc. Now I can build the firmware with west build myproject when in top dir. This setup is really painful and it took me over 3 days to make it working.. still I am not sure if this is the "correct way" :-( Documentation on this part would be highly useful :-)

Update 2
I can now build my local project. However, when adding nrfconnect that contains NUS module that I want to use in my project along with other useful stuff, then I cannot build again - probably because nrfconnect also includes zephyr and that again creates recursion / confusion to west - no matter if the import is set to true or flase. How can I disable nrfconnect/sdk-nrf/zephyr? Maybe I should use that Zephyr repo from from nrfconnect instead my own copy created by west? How can I control Zephyr version then?

  remotes:
...
    - name: nrfconnect
      url-base: https:/nrfconnect
  projects:
...
    - name: sdk-nrf
      remote: nrfconnect
      import: false #true

Update 3
I have added nrfconnect to group and made that group inactive. Now my local project builds again. However, I am not sure whether that module will be fetched by west update on other workstations.. and if I could use anything from that sub-project in my local project..? Is there any way to disable specific module from a sub-project - only Zephyr part of the nrfconnect sub-project?

  1 manifest:
  2   remotes:
  3     - name: zephyrproject-rtos
  4       url-base: https:/zephyrproject-rtos
  5     - name: nrfconnect
  6       url-base: https:/nrfconnect
  7   projects:
  8     - name: zephyr
  9       remote: zephyrproject-rtos
 10       revision: v2.6.0-rc1 #master #v2.5.0
 11       import: true
 12     - name: sdk-nrf
 13       remote: nrfconnect
 14       import: false
 15       groups:
 16         - nrfconnect
 17   group-filter: [-nrfconnect]
 18   self:
 19       path: myproject

[mbolivar-nordic]

Hi, I hope you don't mind that I've refiled this as a question instead of a bug. I think you've missed how west workspaces are supposed to be set up.

Current west approach does not allow easy version selection of the zephyr. Either I need to install global zephyr installation and provide ZEPHYR_BASE on where is is located, or use manifest approach and use local copy of zephyr within my project source tree.

I think you may be confusing "project" with "workspace" in west terms, but other than that this is correct.

In a commercial projects we cannot use simply master branch. We need to select one specific version, then after testing when new release comes up we may switch to that new release. But a specific zephyr version dependency is a must and coherently controlled with one simple text file and some script.

Right: the west.yml file is exactly this text file.

Zephyr version needs to be arbitrarily selected on project creation and then there is no easy way to change the zephyr version later on. In perfect world I would have west.yml for my project stored on a git repo, then when we change the zephyr version we change the version number in west.yml, developers pull that change, perform west update, and everyone uses now the same version of zephyr.

There are various examples showing how to do exactly this documented here:

https://docs.zephyrproject.org/latest/guides/west/manifest.html#manifest-imports

Zephyr version selection should be provided in some sort of configuration file that is part of the source code repository. This way version change is consistent. We do not have a problem that someone has different version of the zephyr source code so for someone code builds and work and for someone else it builds but does not work or it does not build at all. No manual intervention in the local git zephyr pull should be necessary as this leads to confusion and problems.

There are various updates to this issue as you made progress, but it looks like at this point in time you were assuming that zephyr has to be the manifest repository.

For the record, that's not the case: you can have your own manifest repository, as explained here:

https://docs.zephyrproject.org/latest/guides/west/basics.html#workspace-concepts

That repository can contain your own manifest file, which you did set up later, but not quite in the right way as far as I can tell.

Documentation on project file structure needs to be updated and extended with information on how the local project depends on zephyr versioning exactly and how this can be set in a consistent way with possibility to switch later during the code development based on a text file configuration variable.

None of the examples contain west.yml file nor there is an example that shows how to build for a specific version of the zephyr based on the local project configuration (nor how to build for a different zephyr releases).

As far as I can tell that is simply not true; the manifest imports section linked above has a lot of examples for various different use cases and they all explain what version of zephyr you get if you choose each approach. Everything is defined in terms of the manifest file format which is documented earlier in the same page.

I may be missing something or this is not yet available in zephyr? :-)

I believe you may have missed this page, along with the 'west basics' page describing how the workspace is set up, yes.

When you want to use .west/config with following contents so you can control the zephyr version:

[manifest]
path = .
file = west.yml

Having path = . as shown here makes no sense. The manifest repository should be a subdirectory of the workspace topdir. Please read through https://docs.zephyrproject.org/latest/guides/west/basics.html to understand what a west workspace is and how it fits together.

And the west.yml

Are you trying to put this west.yml file in the workspace topdir itself? I'm not surprised it does not work because that's not how west is designed to work.

I have created myproject/.west/config:
[...]
Then myproject/west.yml:

That's not how west is designed to be used. It's the job of west init to create the workspace, including .west/config, using data in the manifest file.

You seem to be trying to treat the entire workspace as a single git repository, which is not the mental model that west was designed with. That model is introduced in the 'west basics' page I linked above and is documented further in the page explaining the manifest format:

https://docs.zephyrproject.org/latest/guides/west/manifest.html#multiple-repository-model

On west update zephyr is pulled into myproject/zephyr along with myproject/bootloader, myproject/modules, myproject/tools. That sounds fine and I get exactly the desired selected version from my myproject/west.yml.

It's not fine; .west is not meant to be part of a git repository in the workspace. The workspace is a larger structure that contains git repositories inside.

What do I miss? :-)

Please refer to the examples in the west manifest documentation, which are linked to above.

This setup is really painful and it took me over 3 days to make it working.. still I am not sure if this is the "correct way" :-( Documentation on this part would be highly useful :-)

I'm truly sorry you had such a hard time, but I'm not seeing any gaps in the documentation on this front. West's 'workspace' concept is sometimes surprising for people who are used to putting everything in a single git repository and this can lead to misunderstanding sometimes. (It's not random, though: it's designed to work the same way Google's "repo" tool is used to manage an Android source code tree.)

Any suggestions for making the documentation I've linked to in various places above better would be appreciated.

One thing we are aware of a need for is a 'canned' application that shows you how to set up a manifest repository which also contains a custom driver, board, etc. and a west.yml file you can tweak for your own needs.

That was recently merged and is available here:

https:/zephyrproject-rtos/example-application/

You may find it to be an easier starting point than the west documentation. Any feedback on that repository would also be welcome.

Note that its west.yml is using the bleeding edge, but you can easily change the revision: master line to something else here:

https:/zephyrproject-rtos/example-application/blob/992b90172b31c120e01d98359f6329bd8602b2a9/west.yml#L15

and get a fixed version, e.g. revision: v2.5.0 for the latest release.

Hope this helps!

[cederom]

Hello @mbolivar-nordic thanks for your reply and hints :-)

• Zephyr documentation really needs to be extended here in terms of nomenclature, use cases, and examples. At the moment it is fine for simple samples but not enough for more advanced configurations. It may be clear for someone who already understands how things work but it is not clear for someone that wants to create this kind of template from scratch :-)
  • Nomenclature clarification and examples.
  • Put all west.yml in one place with graduated sections of in-depth details.
• I got things done and working so far on my own but more people will want to use this approach for more complex projects. And Zephyr is really amazing project to create amazing things! This will get popular :-)
• Thank you for sample application! This is exactly what is needed here :-)
• This sample application needs to west init at first. Then it fails at west update because no zephyr/west.yml is found. So it does not work for me.
• This sample application can also include https:/nrfconnect/sdk-nrf and use some of its features (i.e. https:/nrfconnect/sdk-nrf/tree/master/samples/bluetooth/shell_bt_nus that I need right now). That will show what I am struggling with at the moment and may save some time for the others :-)
• With more understanding and experience I will probably commit some documentation improvements.

I am working on FreeBSD Unix and this part is still missing in documentation and I will add it one day :-) Everything works perfectly fine here - this OS is real stress-tester of Open-Source quality, all dirty hacks will come out here - and I must admit Zephyr code is so elegant and versatile that I have no problems so far. Except that project structure that goes beyond simple build of a sample.. but this is work in progress I can see :-)

Thank you for creating and developing Zephyr! :-)

[mbolivar-nordic]

Thanks for your response and feedback! Tomasz CEDRO ***@***.***> writes:

Hello @mbolivar-nordic thanks for your reply and hints :-) * Zephyr documentation really needs to be extended here in terms of nomenclature, use cases, and examples. At the moment it is fine for simple samples but not enough for more advanced configurations. It may be clear for someone who already understands how things work but it is not clear for someone that wants to create this kind of template from scratch :-) * Nomenclature clarification and examples.

I am probably too close to the details to understand what you're saying, but I think the nomenclature is all properly defined and has examples, starting from the 'basics' page and moving on to the more detailed examples in the manifest documentation. Can you please give me a concrete example of what nomenclature is either unclear or lacks an example?

* Put all `west.yml` in one place with graduated sections of in-depth details.

That's exactly what is done in the manifest documentation, so I again don't really understand what you mean here.

* Thank you for sample application! This is exactly what is needed here :-)

Good, glad to hear it!

* This sample application needs to `west init` at first. Then it fails at `west update` because no `zephyr/west.yml` is found.

The documentation here for using the sample application works for me, with no errors on 'west update': https:/zephyrproject-rtos/example-application/#getting-started I continue to suggest to you that you are not understanding west's documented behavior.

* This sample application can also include https:/nrfconnect/sdk-nrf and use some of its features (i.e. https:/nrfconnect/sdk-nrf/tree/master/samples/bluetooth/shell_bt_nus that I need right now).

That is not an upstream Zephyr issue. Remember that the nRF Connect SDK / NCS is a downstream project and it would not be appropriate to include vendor downstream details anywhere in the zephyrproject-rtos GitHub organization. However, since you mention you are an NCS user, there is a dedicated page of workflows for managing your own code for downstream users of NCS: https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/dm_adding_code.html Please let's not discuss that further in this upstream issue, though. Devzone remains the right place to look for NCS-specific support, not the upstream Zephyr project.

* With more understanding and experience I will probably commit some documentation improvements.

I think that'd be much better than continuing this discussion here, as it will give me the details on what you're seeing that I'm missing. Thanks!

I am working on FreeBSD Unix and this part is still missing in documentation and I will add it one day :-) Everything works perfectly fine here - this OS is real stress-tester of Open-Source quality, all dirty hacks will come out here - and I must admit Zephyr code is so elegant and versatile that I have no problems so far. Except that project structure that goes beyond simple build of a sample.. but this is work in progress I can see :-)

I am glad you are finding that things work in BSD.

Thank you for creating and developing Zephyr! :-)

Thanks for your feedback!

[cederom]

Yes you seem to be "to close to the details" to the point where you don't see the big picture or understand simple question. Will not bother anymore no worries :-)

The thing is really simple:

• Our project is already placed on a git repo so git clone and git pull is the way to get it because we already work that way. We cannot change it to west init repo. We have different workflow and you (or west design) dont seem to see that.
• If you go beyond building a sample you need to create a project that may need to contain more that one external "workspace" and probably this whole project will be located on a git repo. I am sure this will jump into the documentation sooner or later.
• example-application can should be also initialized with west init -l when already pulled with git (no mention in doc you showed). If not possible that way then why not possible? Why not open for this possibility? We need to be able to perform init on an existing repository.
• We need to use shell_bt_nus in our project. Zephyr does not provide shell over BLE sample. As simple as that. I hope it will one day become sample in Zephyr repo. Because it is extremely useful in development firmware on production devices with no UART connectors.
• sample-application seems to be based on (or organized the same way as) sdk-nrf so I will stick to the later. Yes I need to include it as building block of our solution along with other building blocks.
• I am happy this sample-application (maybe sample-workspace-template or sample-project-template would fit better?) showed up and I hope it will be direct part of Zephyr repo samples one day. If this needs to be an external repo and you cannot include it into another project or "workspace" or whatever you name it then here is the difference we are talking about.

Thank you for your time :-)

[mbolivar-nordic]

* We cannot change it to `west init repo`. We have different workflow and you (or west design) dont seem to see that.

You are free to avoid using west entirely if you like. It's just harder to do so.

That is in fact an option documented in the NCS guide I linked earlier as well as here:

https://docs.zephyrproject.org/latest/guides/west/without-west.html

If you are going to use west, though, you need to do things the way west expects.

* If you go beyond building a sample you need to create a project that may need to contain more that one external "workspace" and probably this whole project will be located on a git repo. I am sure this will jump into the documentation sooner or later.

This is already documented in several different use cases in the links I have given already.

* `example-application` can should be also initialized with `west init -l` when already pulled with git (no mention in doc you showed). If not possible that way then why not possible? Why not open for this possibility? We need to be able to perform init on an existing repository.

Yes, of course it works. This question continues to reflect a lack of understanding about west.

* We need to use `shell_bt_nus` in our project. Zephyr does not provide shell over BLE sample.

I'm glad you've got what you need from NCS.

[cederom]

* `example-application` can should be also initialized with `west init -l` when already pulled with git (no mention in doc you showed). If not possible that way then why not possible? Why not open for this possibility? We need to be able to perform init on an existing repository.

Yes, of course it works. This question continues to reflect a lack of understanding about west.

Further discussion is pointless. We don't understand each other completely. I have what I need from nRF SDK it is far better documented and itself is a working example of what I need. But I had this all before writing here. Thanks. Take care :-)