Support binary blobs / libraries and glue code in vanilla upstream Zephyr

[carlescufi]

This issue here is to describe what to do about Zephyr (i.e. the upstream vanilla distribution) containing either of the following:

• Binary objects of any sort (firmware images, linkable libraries)
• Accompanying OSS build, header or source files to the binary objects (typically a CMakeLists.txt custom-made for Zephyr and then any additional .c and .h files that are compiled when using the library)
• Accompanying non-OSS source code to the binary objects
• OSS glue code and Kconfig for a binary object in the main zephyr repository (typically changes in existing in-tree code to accomodate for the binary blob and a new in-tree Kconfig file for the module)

There are many types of binary blobs with different levels of integration requirements:

• External library with its own set of APIs not overlapping with existing Zephyr APIs (eg. a 3rd-party physics library)
• External library that replaces the implementation of existing libraries with a binary one (eg. replacing mbedtls with a binary library that implements the same APIs)
• External library that replaces an existing in-tree driver (eg. a replacement of a driver with a propietary version in binary form that makes use of hidden hardware)
• External library that implements a new driver (eg. a new sensor)
• External library that implements a new subsystem, but that interacts with existing subystems (e.g. Segger RTT)

The following decisions have been found to be orthogonal to each other:

1. Should binary objects be hosted in the zephyrproject-rtos GitHub organization?
2. Should accompanying OSS build, header or source files be hosted in the zephyrproject-rtos GitHub organization?
3. Should accompanying non-OSS header or source files be hosted in the zephyrproject-rtos GitHub organization?
4. Should OSS glue code and Kconfig (Apache v2) for a binary object be present in the main zephyr repository?
5. Should the vanilla manifest contain links to 3rd-party repositories with binary blobs and all the accompanying build files, headers, source and glue code?
6. Should the user explicitly have to enable binary blobs and/or non-OSS source code via a command? (eg. west config manifest.enable-blobs)

Proposals:

• a. Host binary blobs and accompanying build, headers and source code (OSS or non-OSS) in the zephyrproject-rtos GitHub organization, just like any other external project we host today, include the glue code (Apache v2) in the zephyr repository and include the projects in the vanilla manifest
• b. Same as a, but only with OSS accompanying source code and headers
• c. Same as a, but with the user having to explicitly enable binary blobs and/or non-OSS source
• d. Same as a, but with the user having to explicitly enable binary blobs
• e. Host only glue code and Kconfig (Apache v2) in the zephyr repository, and require users to pull binary blobs and accompanying build, headers and source code from a 3rd-party repository, not including any reference to it in the vanilla manifest
• f. Not host anything at all, require users to use their own manifest and 3rd-party repositories to integrate with binary blobs.

[galak]

I think this is something we need to support in Zephyr. We probably need to figure out the rules and document them around what is considered an acceptable binary. For example:

• is the binary licensed in such a way that allows re-distribution by the Zephyr project.
• are the interface/headers to the binary licensed permissively/OSI licensed (ie, something like Apache or BSD).

[carlescufi]

We probably need to figure out the rules and document them around what is considered an acceptable binary

I agree. Where do you think this document should live?

Also, @galak I guess you are assuming option 3. and the multi-repo being there?

[SebastianBoe]

This sounds somewhat similar to what is done here:

https:/zephyrproject-rtos/zephyr/blob/master/ext/hal/esp/CMakeLists.txt

[galak]

I agree. Where do you think this document should live?

I assume the doc would be part of our contributing guidelines, so something under doc/contribute/

Also, @galak I guess you are assuming option 3. and the multi-repo being there?

Yes, I think something like option 3.

[carlescufi]

Depends on #6770

[carlescufi]

This sounds somewhat similar to what is done here:

Yes, indeed. I guess that flew under the radar.

[dleach02]

+1, I'm starting down this path right now to figure out how to integrate support for BLE on the NXP KW41 platforms. The BLE controller code is a binary blob that comes from NXP and won't be able to be hosted on the Zephyr repository due to legal restrictions.

[erwango]

As an output of this ticket it would be nice to have a checklist that can be can be used to check easily if the component we want to introduced is compatible with zephyr policy and if not what is missing.

[nashif]

is the binary licensed in such a way that allows re-distribution by the Zephyr project.

this only applies if we actually are going to support that library as part of the project, in most cases I would say, no, we should not have any binaries included in the project itself, however we should enable this like with the ESP32 case.
Once we have multi-git repo support his would be easier obviously but regardless, a policy is still required.

[microbuilder]

While I would personally argue that no binary objects should be included in the core repo, it's a lot more ambiguous how to handle additional repositories hosted under zephyrproject-rtos.

Ignoring the debate of open source or not, from a practical point of view it seems like this would place a significant legal burden on the Zephyr Project to validate that we can host and distribute said binary blob(s) and associated IP or files.

The opinion of engineer X on Slack or over email saying "sure, it's fine to host blob Y" probably isn't sufficient for blobs that have EULA's associated with them, and would likely require written sign-off by appropriate IP owners. If IP owners are distributing binary blobs, there are almost certainly distribution terms attached.

Some cases may be clear cut, but it seems like there would still need to be some sort of legal validation process or paper trail here to host binaries in zephyrproject-rtos. Whether the benefit is worth the additional effort and legal exposure is obviously open for debate.

[pabigot]

Please retain the distinction between "Apache-2.0" and "OSS" in this discussion. They are very much not the same.

I'm not clear on what "OSS" is intended to identify. It would perhaps be better to use the term OSI-approved license where open source beyond Apache-2.0 is being addressed. Note that GPL is OSI-approved.

[pabigot]

Repeating here the position I gave in TSC chat on 2019-06-12:

Zephyr bills itself as Apache-2.0, which I believe to be true of the main repository. Through west it pulls in material from other repositories; at the moment these are hosted by zephyrproject-rtos, but in principle they could be external. It's not clear to me whether all material in those repositories is also Apache-2.0.

As an open source developer I may need to protect against claims of potential IP theft by doing development on a clean system, where in theory I know the licensing of every bit on the machine.

Question 0 does not appear in the description of this issue:

0. Should a default-installed Zephyr installation be restricted to content that is licensed under Apache-2.0?

I personally prefer the default answer to this to be "Yes." Whether it is "yes" or "no", the answer (and its rationale) will inform the process leading to the remaining decisions, including whether/how non-Apache-2.0 content can be exposed as an opt-in post-installation step from the default installation, and how the licensing of any retrieved material is recorded and accessed.

When I do west update to fetch everything, I have no idea what's just been installed on my system. This is somewhat concerning to me.

[marc-hb]

When I do west update to fetch everything, I have no idea what's just been installed on my system. This is somewhat concerning to me.

+1 and I believe you've been considering only the default zephyr/west.yml. West should have a west list --format={license} field that works for any manifest. From a developer (and lawyer) perspective it's much more important to know the licenses that are going into a product than which server or mirror it's hosted on.

Examples:
pip3 install --user pip-licenses && pip-licenses
https://www.yoctoproject.org/docs/2.8/dev-manual/dev-manual.html#working-with-licenses
https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/#license-specification
https://rpm-packaging-guide.github.io/#preparing-source-code-for-packaging
etc.

[aescolar]

Zephyr bills itself as Apache-2.0, which I believe to be true of the main repository.

(Being a bit pedantic here, but I'd think we need to be much more precise in this discussion)
@pabigot The main repo would +contain+ be Apache-2.0 compatible +code+. There is BSD-3 clause & MIT code in ext/ and in some other places in-tree.
I guess that is what you meant to say? and therefore a "compatible" should be added to your question 0?

@carlescufi would it be a good idea to rephrase the top comment, removing the "OSS" term, and stating clearly where it was meant to say "Apache-2 compatible" and where it refers to any other form of open source licensed SW? (if the intent was to include GPL or MPL in the discussion that would change things).

[pabigot]

Zephyr bills itself as Apache-2.0, which I believe to be true of the main repository.

(Being a bit pedantic here, but I'd think we need to be much more precise in this discussion)
@pabigot The main repo would be Apache-2.0 compatible. There is BSD-3 clause & MIT code in ext/ and in some other places in-tree.
I guess that is what you meant to say? and therefore a "compatible" should be added to your question 0?

No, I meant to say Apache-2.0. The LICENSE file at the top says that's it's license.

The following command:

git grep SPDX-License | grep -v Apache-2.0 | sed -e 's@^.*:@@' | awk '{print $1}' | sort | uniq

shows Zephyr, as of a recent master, has these licenses in its SPDX-License-Identifier tags:

• BSD-3-Clause
• BSD-5-Clause-Nordic
• CC0-1.0
• GPL-2.0
• ISC

I am informed by this.

[aescolar]

@pabigot ( sorry, I should have phrased my statement as "The main repo would contain Apache-2.0 compatible code" (irrespectively of the project overall being licensed as Apache2.0). It may have caused some confusion )

@pabigot Did you mean to say that you would like to restrict the main repo, and repos pulled by the default manifest, to contain only, Apache2.0 code?. And bar non-Apache2.0, but Apache2.0 compatible code?

(Note, AFAIK, GPL are only some development scripts used, and some of the QEMU coverage code)

[pabigot]

I presented as a naive new user who assumed the LICENSE file gave me a correct description of the licensing that applied to the content of the tree it's in. I had only intended to make a clear statement that preserved the existing state.

Since Zephyr does not accurately describe the licensing of its contained material (at least in the place a naive user would expect to find it), perhaps that should be addressed. I am not a lawyer, but at least one previous employer would see those GPL scripts and say "No Zephyr on our networks", at least until somebody in upper management approved an exception, and I'd probably be in trouble if I'd gotten initial approval on the basis of claiming it to be Apache-2.0 (not "-compatible", since the company policy isn't concerned with somebody else's assessment of "compatible": they want a list of licenses).

[jfischer-no]

I am against it to host any binary objects or source code licensed under non-OSI approved (rather: "popular and widely-used or with strong communities") licenses in the zephyrproject-rtos GitHub organization, which could be compiled and linked to an Zephyr RTOS based application (firmware). I think it is up to vendor to host it, specially from legal perspective, and the project should not be easy-peasy-barrier-free regarding code licensed under non-OSI-"popular and widely-used or with strong communities" or incompatible license because of respect for those who bring and maintain their drivers mainline.

Since Zephyr does not accurately describe the licensing of its contained material (at least in the place a naive user would expect to find it), perhaps that should be addressed.

naive user is subjective, it is described here and here(but outdated)

[carlescufi]

The TSC approved this and then a series of PRs have been bringing in support into the tree:

• doc: Document binary blobs #47678
• modules: Basic binary blob infrastructure #49037
• Binary blobs: Multiple tweaks and documentation #49194