Build system woes and ideas

[algernon]

Way back when, we had #497, with the goal of rebuilding kaleidoscope-builder, which was concluded with us not wanting to rebuild it - we'd rather replace it. Now, the question is: with what?

We basically want two major things: compile sketches, and flash them. In addition that, we want to be able to run tests on virtual, simulated hardware. This latter is basically compiling for the virtual arch, and then running the result.

The simplest way would be to use arduino-cli, which is official, and supports pretty much all we want and need, and we can wrap it very, very thinly, if need be. It has a number of incompatibilities compared to arduino-builder: it requires the sketch name to be the same as the containing directory, which is sometimes a bit awkward, but easily solvable. It also lacks support for passing certain type of compiler flags, namely #defines that have a string argument (see here). Not an issue for the sketches we ship, but we have a few custom sketches by users that do make use of string defines. This can be worked around by prepping a header file first, and putting the defines there, but that's a bit more work on the part of the user.

But, stepping back: as a user, I would like to be able to: kaleidoscope-builder compile examples/Devices/Keyboardio/Model01, and have it do the right thing. I also would like to be able to kaleidoscope-builder compile examples/Devices/Keyboardio/Atreus, and have that work out of the box as well, with upload (or flash) likewise. For this, we need to figure out the arch of a sketch. Or we may require the user to specify that: kaleidoscope-builder compile --board keyboardio:avr:model01 examples/Devices/Keyboardio/Model01 - but that's quite a handful.

Flashing is the harder part, too, because we have a number of complicated scripts that help us find the right port, which we can't even integrate into the arduino system, because arduino-cli simply requires us to give it a port. So we'd need a wrapper for that.

In short, it looks like arduino-cli could be a reasonable replacement. We'd get rid of a whole host of custom code that barely works. What we do need to pay attention to, is tool paths: arduino-cli will, by default, use the tools specified in boards.txt and platform.txt. But sometimes, we want to override those - that must remain possible.

[algernon]

Having had a good nights sleep pondering about this, I have a few things in mind now, that would be amazingly nice to have, and all of which are quite simple to do with a thin wrapper around arduino-cli.

• It would be amazing, if we could easily tell the builder where 3rd party plugins are, in a way that makes sense to the kaleidoscope-builder user: kaleidoscope-builder --add-plugin-dir /some/path compile some/Sketch. Adding multiple plugin dirs would, of course, do the right thing. This'd internally translate to --libraries /some/path. We could just let extra options pass through, too, in which case we'd have kaleidoscope-builder --libraries /some/path compile some/Sketch, but I feel our custom name for the option makes things nicer.
• We'd like to be able to override the compiler tools - this is especially important for virtual builds. The easiest solution here is to grab all the tools we need, the defaults, and the overrides, and symlink those to a temporary directory, under the names the platform specification expects, and then just set compiler.path.

[per1234]

It also lacks support for passing certain type of compiler flags, namely #defines that have a string argument

I'm not knowlegeable enough to know what it would take, but my hope is that this issue will be fixed. This certainly isn't a limitation that was added intentionally.

I also would like to be able to kaleidoscope-builder compile examples/Devices/Keyboardio/Atreus, and have that work out of the box as well, with upload (or flash) likewise. For this, we need to figure out the arch of a sketch. Or we may require the user to specify that: kaleidoscope-builder compile --board keyboardio:avr:model01 examples/Devices/Keyboardio/Model01

Arduino CLI supports defining the FQBN via a sketch.json metadata file placed in the sketch folder:
https://arduino.github.io/arduino-cli/latest/sketch-specification/#metadata
With that file present, the --fqbn option is not needed.

arduino-cli simply requires us to give it a port.

It's true that arduino-cli upload does require a port. However, it doesnt' need to be a valid port. It's only used to set the {serial.port} build property, which the author of the boards platform is welcome to use, or welcome to not use. You can also define the port in the sketch metadata file's cpu.port key. The port is also not required if you specify a programmer via the --programmer option.

[algernon]

Arduino CLI supports defining the FQBN via a sketch.json metadata file placed in the sketch folder

Oooh, nice! This works like a charm, thank you!

It's only used to set the {serial.port} build property

To be honest, requiring a port is fine. We have the tooling to discover which port we need to specify, so arduino-cli upload --port $(tools/find-port) isn't too bad, and we can easily wrap that in a Makefile or something along those lines.

[algernon]

Had a quick talk with @obra this morning too, and he suggested an alternative to symlink-farming the compilers: just ship a set with our virtual platform, the same way Arduino does for AVR and SAMD. That way, we never have to care about where compilers are, nor about subtle differences, nor anything like that. We'll just package them up, and we're good to go.

Inspired by this, I converted my own sketch to use arduino-cli, and it went pretty darn well! The one problem I could only solve with a hackish solution is making the keyboardio:avr core available to arduino-cli. With arduino-builder, I could put them in lib/hardware/keyboardio, and tell the builder to use that. I don't see a similar option for arduino-cli, so I symlink that dir to ~/Arduino/hardware/keyboardio instead. It works, but it isn't exactly nice.

[per1234]

I could put them in lib/hardware/keyboardio, and tell the builder to use that. I don't see a similar option for arduino-cli, so I symlink that dir to ~/Arduino/hardware/keyboardio instead. It works, but it isn't exactly nice.

I'm not at all familiar with your project, so forgive me if I provide irrelevant information. ~/Arduino is Arduino CLI's default "user directory" (the equivalent to the Arduino IDE's "sketchbook"). You can configure the user directory via the directories.user configuration key. This can be done either by setting the ARDUINO_DIRECTORIES_USER environment variable, or writing to the directories.user key of the configuration file. This system is documented here:
https://arduino.github.io/arduino-cli/latest/configuration/

So you can configure directories.user as you like, and then Arduino CLI will pick up any cores that are under {directories.user}/hardware

[algernon]

Much appreciated, ARDUINO_DIRECTORIES_USER does the trick!