cmake: Improve library dependencies by removing whole-archive, start/end-group

[tejlmand]

Zephyr subsystems creates multiple (sub-)sublibraries, especially bluetooth and
net. Those subsystems also has internal dependencies to each other.

Multiple subsystems in Zephyr has a lot of sub libraries, which I believe
should be minimized, especially a KConfig is good for configuring the libraries.

Subsystem Bluetooth contains:
subsys_bluetooth, subsys__bluetooth__common, subsys__bluetooth__controller,
subsys__bluetooth__host__mesh, subsys__bluetooth__host

Subsystem net contains:
subsys__net, subsys__net_ip, subsys__net_ip_l2, subsys__net_ip_l2_ethernet,
subsys__net_ip_l2_ieee802154, subsys__net_ip_l2_openthread,
subsys__net__lib__app, subsys__net__lib__dns, subsys__net__lib__http,
subsys__net__lib__lwm2m, subsys__net__lib__mqtt, subsys__net__lib__websocket

Zephyr links everything together using -Wl,--start-group
-Wl,--end-group, as well as Zephyr build system uses
-Wl,--whole-archive.

This principle has several drawbacks;
The -Wl,--whole-archive drawback is also described in: #6961

Creating executables in this manner requires extra build scripts and
is non-standard for a CMake project. Instead of using the CMake API
to link together libraries into an executable with
target_link_libraries() one must have build scripts that ensures the
linker is passed this list of linker flags.

It introduces two kinds of libraries, instead of having just one kind.
Libraries which must be whole-archived, and libraries that should not.

There is also a theoretical code size performance impact. If
whole-archive is used and there exists an object file in a library
that happens to not be in use (dead code) the object file will still
be included in the executable, instead of being pruned as one would
want it to be. But if one links without '--whole- archive' only the
object files that have symbols needed by the link will be used.
In Zephyr, the handling of library content is well handled using
KConfig, and doesn't seem to suffer from this risk.
However, that doesn't mean the problem may not be present in certain
parts of Zephyr.

Finally, using a flat linking model where all libraries are included
means that we do not have readily available information about how
libraries depend on each other.
CMake list ${ZEPHYR_LIBS_PROPERTY} is furthermore a global variable
to which libraries are added and later used in the linking stage.
Use of global variables should be minimized whenever possible.

EDIT: This is part of umbrella issue: #8827
EDIT: Removed text covered by #8826

[SebastianBoe]

Would you mind splitting the "many libraries causes noise in IDE's" out into it's own issue?

EDIT: If we split that out then this issue will solely be about the problems that arise from the
mechanism we use to link (--whole-archiving a global list of libraries).

[zephyrbot]

Hi @tejlmand,

This issue, marked as an Enhancement, was opened a while ago and did not get any traction. It was just assigned to you based on the labels. If you don't consider yourself the right person to address this issue, please re-assing it to the right person.

Please take a moment to review if the issue is still relevant to the project. If it is, please provide feedback and direction on how to move forward. If it is not, has already been addressed, is a duplicate, or is no longer relevant, please close it with a short comment explaining the reason.

@tejlmand you are also encouraged to help moving this issue forward by providing additional information and confirming this request/issue is still relevant to you.

Thanks!