Extensible Source Inclusion System, Smart Building for Marlin

[quiret]

Description

Please watch the video to get the full idea:
https://www.youtube.com/watch?v=aSdFR7pnAtg

The Marlin Firmware has a too complicated source file inclusion layout. In this PR I want to help reduce the technical burden of source code repository maintenance. There are some issues in the default way that PlatformIO does things. Notorious example is the inclusion of all font files when graphical LCDs are used.

Benefits

The amount of lines in the platformio.ini has been drastically reduced.
Shorter build times (small impact?)
Better maintainability.
Backwards compatible with old layout.

Related Issues

#24911

[jmz52]

TFT font optimization requires new line in features.ini file for each family type increasing amount of work needed to support it.
Also it is made on assumptions that only one font can be used, meaning it will be obsolete the moment UI will start to use multiple fonts.
Do you have numbers that show how much time it saves during compilation?

[quiret]

(...) meaning it will be obsolete the moment UI will start to use multiple fonts.

Sorry but you are mistaken. If multiple fonts get supported then you have a configuration option. You can use that to enable multiple features at once, like each font feature.

Do you have numbers that show how much time it saves during compilation?

Launching the GCC compiler for a disabled source file is an unnecessaringly big task. This wastes both time and energy that could be spent on other computational activities.

[jmz52]

This is exactly my point - with the proposed granularity level even small change in UI will render this system outdated and it will fail to build firmware.
The need to constantly update this system will make development process more time consuming.

On the other hand the time saving on excluding fonts file is insignificant at best.
I've run number of clean compilation on my 14-year old i7 with generic 7200 RPM HDDs and here are the results:
Best compilation time with all font files included - 127.28s (yes, it was faster than without fonts)
Best compilation time without Helvetica and NotoSans (heaviest of the all) fonts - 128.74s
Build time deviation between builds - 11.7s
Conclusion: potential time saving on excluding font files from compilation is less than mean statistical error caused by build time deviation.

So again - proposed level of granularity is not an improvement over the one we already have.

[quiret]

So again - proposed level of granularity is not an improvement over the one we already have.

You have purposefully tested on a fast computer, excluding build speedups on people who just want to build Marlin on their old computers. Do not ignore the people stuck with their AVR processors or using old laptops. While I cannot speak for every computer out there I claim that there are special situations in which build exclusions speed up for good.

Also you are ignoring the fact that the build process is much cleaner without those additional source files in the compilation stage. Your assumption that the build files do not result in a larger binary is randomly coincident to the fact that the compiler is smart enough to optimize your definitions away. In the object graph your font data does persist until linking stage. I wouldn't go as far as to rely on compiler optimization too much.

Another small advantage of cleaning up the build stage is a cleaner build log. You only see the source files being built that matter, improving on the human maintainability of the Marlin project. If everyone were to include their optional sources... Just imagine what a mess it would become!

Anyway, I do not understand your ramsack obstruction over this (maybe minor) software development improvement.

[jmz52]

Your assumption that the build files do not result in a larger binary is randomly coincident to the fact that the compiler is smart enough to optimize your definitions away. In the object graph your font data does persist until linking stage. I wouldn't go as far as to rely on compiler optimization too much.

There is absolutely no coincidence or optimization done by compiler.
No including unused code and data is literally a core functionality of a linker and it is being that way for decades.

I do not understand your ramsack obstruction over this (maybe minor) software development improvement.

"The need to constantly update this system will make development process more time consuming."

[EvilGremlin]

What is actual quantifiable benefit of changed filtering? Does it shave off at least 5% of build time? Guess i'll go get my old netbook to test.

[quiret]

@jmz52 Since you are so convinced that your multi-font stuff is so necessary for Marlin let me explain to you that many boards already hit their RAM/ROM limit with just one font (MKS Robin E3D V1.1 for example) Do you really it is a good idea? It is not a good idea for everybody. The memory is better spent increasing the framebuffer! Note that multi-font support is possible by making use of the TFT_FONT_* defines set out in this PR. You probably thought it something sinister but you are just mistaken. Please calm down and think, ok?

I am not the one making build-speed improvement this big argument. This @jmz52 guy just felt triggered about it. Please look at the software model improvement by proper features.ini parsing instead. I think this is way more worthwhile.

Also imagine code that references all the font data, by chance. By disabling the source files we are giving the user no wrong impressions that font data is overcrowding his printer firmware. The optional inclusion as set out by features.ini is giving good developer guidelines for firmware size minimalization. It simply does not appear in the build log, leading to a clean impression of the Marlin firmware building process.

Granularity has always been a goal of Marlin firmware. In this discussion a point was made to the contrary but just think about it. There are many GCodes in Marlin that you can enable or disable based on what kind of printer you want. The additional configuration resolution for the firmware is aligned with Marlin's current strategy.

@EvilGremlin Please spare yourself the extra work over this minor detail. It resulted from a user making a fuss about something changing related to what he worked on. I have corrected a detail in my post to not raise any wrong expectations! Most of the users will not see a GIANT impact on build speed improvement. But some will appreciate it, even if not always visible.

[quiret]

https://www.youtube.com/watch?v=3pTu-lXoK6M

Here is my statistical proof that Extensible Source Inclusion does save meaningful build time. The test target was the inclusion of the font files: on one end just helvetica was included and on the other all of them. The test results speak for themselves:

with minimal fonts:

1. 132.64
2. 134.43
3. 125.02
4. 134.27
5. 133.52
   mean-time = 131.98
   max-deviation = 9.41 seconds

with maximal fonts:

1. 128.24
2. 147.98
3. 140.84
4. 142.95
5. 145.14
   mean-time = 141.03
   max-deviation = 19.74 seconds

mean-saving = abs(mean-time(minimal) - mean-time(maximal)) = 9.05 seconds
-> good time saving thanks to ESIS!

[quiret]

I am sorry but I have decided to halt my activities on this PR for an indetermined amount of time. You are free to decide whether to continue it yourself. Thank you to the Marlin firmware community for providing this interesting playing ground for testing my ideas and vision. I hope that Marlin will continue to go towards the best 3D printing solution! Hopefully my ideas have inspired you on how to continue this software!

[thinkyhead]

It's been a busy few months trying to keep up with the Joneses, especially in the area of smoother and faster motion, and of course we want the followup to 2.1.2 to fix any extant issues and be a net improvement, so of course I got obsessed with refactoring. There are still various issues to work out with probing, leveling, and Fixed-Time Motion, but we're getting there.

On that note, this PR is one of those helpful refactors that simplifies things, which is exactly the kind of refactor I like. I've just been reviewing the methodology applied in the improved dependencies script, and it all seems clean and robust. So, at last this will get merged for inclusion in 2.1.3. Sorry for the long delay. There are never enough hours in the day… week… month….

Overall, this is a very simple idea and well-implemented, patching a deficiency in PlatformIO that was making it a lot more tedious to do source filtering. I appreciate that you made a video. I would like to make more of those to share my excitement over things I have been building too.

I've only made a couple of small tweaks here, notably moving the font dependencies to common-dependencies.h so that the items in features.ini work as expected and no custom handling is needed for TFT_FONT in the script. I might try cleaning up the filtering even more, but anyway, it should be merged shortly.

[thinkyhead]

Thanks again! There are all kinds of elements in this codebase that can benefit from this kind of care and attention, and we have a shortage of capable developers, so your contribution is much appreciated.

Lately I have been in the mood to address various deficits and the stratification of ad hoc solutions, and this is starting to bear some fruit. For example, I have a major LCD pins reorganization underway which is beginning to take shape, and I just managed to get rid of those annoying USE_*_PLUG settings that were a classic example of that stratification.

Meanwhile I'm working on improving the configuration documentation at the MarlinDocumentation repo by encapsulating all the settings in YML format, and stealing time to extend the Auto Build Marlin extension with the eventual goal of integrating that configuration tool with the website documentation, and even automating migration.

The newest technologies represented by Klipper, Bambu Labs, Prusa Research, and Creality3D are inspiring and challenging us to achieve a higher standard, and that begins with shoring up the foundations of this project, so that is where my focus is increasingly centered. Rather than being overwhelmed by the challenges ahead, I am feeling more exhilarated than ever. And it is gratifying to encounter others who recognize the continuing relevance and potential of venerable old Marlin in a field of shiny new contenders.

Onward and upward!