Introduce Debian Bookworm support #24

WhoAmI0501 · 2023-07-19T12:41:40Z

This should fix #23 and introduce support for the new Debian Bookworm release.

Currently, my tests of the implementation are quite poor:

Compiling and starting piaware on a Arm64-based Hetzner CAX11 worked
Compiling piaware on my x86_64-based workstation worked
for more details, see comments

My actual feeder, an Odroid XU4 with Armbian, is currently still on bullseye.

May somebody can review my changes and test them on their hardware?
Additionally, i do not have a Jenkins instance, which is why cannot check whether compiling via CI works.

WhoAmI0501 · 2023-07-19T14:02:19Z

I currently built a wireguard tunnel between the CAX11 and my productive feeder. Via socat I am simulating a local readsb etc. - the data is coming from my real feeder via the tunnel.

So far so good. Normal ADS-B as well as faup is working flawless. Only the MLAT client is not starting. It fails with the following error:

root@debian-4gb-fsn1-1:~/piaware_builder# /usr/lib/piaware/helpers/fa-mlat-client --input-connect localhost:30005 --input-type dump1090 --results beast,connect,localhost:30004 --results beast,listen,30105 --results ext_basestation,listen,30106 --udp-transport 2620:13d:c000:11::197:17881:123416086
Traceback (most recent call last):
  File "/root/piaware_builder/package-bookworm/debian/venv/lib/python3.11/site-packages/cx_Freeze/initscripts/__startup__.py", line 130, in run
    module_init.run(name + "__main__")
  File "/root/piaware_builder/package-bookworm/debian/venv/lib/python3.11/site-packages/cx_Freeze/initscripts/console.py", line 16, in run
    exec(code, module_main.__dict__)
  File "/root/piaware_builder/package-bookworm/debian/venv/bin/fa-mlat-client", line 10, in <module>
    from mlat.client.coordinator import Coordinator
  File "/root/piaware_builder/package-bookworm/debian/venv/lib/python3.11/site-packages/mlat/client/coordinator.py", line 26, in <module>
    import _modes
ImportError: /usr/lib/piaware/helpers/lib/_modes.cpython-311-aarch64-linux-gnu.so: undefined symbol: _PyFloat_Unpack4
root@debian-4gb-fsn1-1:~/piaware_builder#

WhoAmI0501 · 2023-07-19T14:43:50Z

The mentioned error was caused by an incompatibility between the mlat-client and Python 3.11. See also this issue in the upstream repository.

In fact, i took the patch from the mentioned issue and applied it via a patch in our build process. On my test system, this is working flawless. Many thanks to @wiedehopf at that point! :)

as365n4 · 2023-07-19T16:39:38Z

Just successfully build your PR and can confirm that MLAT-Client is now working on Bookworm.

Linux 6.1.0-10-arm64 Debian 6.1.37-1 (2023-07-03) aarch64 GNU/Linux
Pine64 Rock64 running Debian Bookworm.

Is it possible not to use PIP? As from my experience it creates more of a Headache, but I am not a progammer by any means.

Processing /home/adsb/piaware_builder/package-bookworm/mlat-client
Preparing metadata (setup.py) ... done
Installing collected packages: MlatClient
DEPRECATION: MlatClient is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at pypa/pip#8559
Running setup.py install for MlatClient ... done
Successfully installed MlatClient-0.2.12<

WhoAmI0501 · 2023-07-19T16:47:56Z

but I am not a progammer by any means.

I am also just a DevOps engineer lol

Is it possible not to use PIP?

I really don't know. What I know is the fact, that we need to go away from the setup.py, since this is deprecated. From my work today I must say, that pip worked actually quite good.

The MlatClient is problematic, yes - but not because of pip. It's because it's in fact not maintained anymore / a legacy project. The patch that i needed to apply... yeah, i don't think that I need to write anything more about that.

In my opinion, the mlat-client should be replaced with the fork from wiedehopf. But thats more something like a long term task.

as365n4 · 2023-07-19T16:53:06Z

I had used PIP in the past too, but the constant changes and requirements it puts on the user, I found frustrating and it is another package manager which needs maintaining and depedencies sorting out.
I prefer to have just apt and be done with it. ;-)

WhoAmI0501 · 2023-07-19T16:57:00Z

I prefer to have just apt and be done with it. ;-)

This is absolutely understandable. In this case, i just worked with the given structure in this project, which in fact installs all python3 depdencies from source - and i wouldn't change that behavior completely.

as365n4 · 2023-07-19T17:01:36Z

Yeah would be nice if the team at FlightAware could be more closer to the way Debian is doing things, so that in the end just some .deb files need distributing..... :-)

And thank you very much indeed for spending your time on this issue and pushing a PR for this.

WhoAmI0501 · 2023-07-28T12:11:29Z

From today, Armbian is not delivering any Debian Bullseye updates anymore for my Odroid XU4. I am currently doing the Bookworm upgrade.

Nine days are gone, since I created my PR and there is still absolutely no reaction from the maintainers. Appreciation looks slightly different. @eric1tran @mutability

mutability · 2023-07-28T13:14:11Z

Unfortunately I'm not willing to blindly apply this PR without digging into the details, and it suffers from the "doing more than one thing" problem which makes it a larger task to review:

a bunch of shell quoting changes?
a patch to mlat-client which would be better served as a PR to mlat-client itself?
a complex cxfreeze update (this is always going to be a pain, I know) -- in particular I'd like to look at if the packaged bookworm python infrastructure is recent enough that cxfreeze can be installed against system packages rather than having to install a bunch of versioned dependencies.

Our support generally tracks what the Pi Foundation supports, and they're still on bullseye, so it's hard to justify prioritizing a review of this at the moment.

But I certainly do appreciate the effort you made here -- thanks! -- and I'll do a review as soon as I can find the time.

If you're willing to break it out into one-functional-change-per-PR then that would make it easier to apply the more obviously-correct parts.

mutability · 2023-07-28T13:15:54Z

so that in the end just some .deb files need distributing..... :-)

That's exactly what we do for Raspbian.

WhoAmI0501 · 2023-07-28T13:37:20Z

Unfortunately I'm not willing to blindly apply this PR without digging into the details

I also do not want you to do that! Its just... nine days of hearing nothing (also not a "I will take a look on it, but i need a few days or weeks") feels like no appreciation. But now we clarified that - thanks for your statement, makes me very happy!

a bunch of shell quoting changes?

This is actually just some beauty stuff that was suggested by shellcheck. Idk whether this is worth a extra PR, but if you want it that way, i will create another PR.

a patch to mlat-client which would be better served as a PR to mlat-client itself?

I guess that we don't achieve anything with creating an PR there... i have written something according to that in my comment above.

in particular I'd like to look at if the packaged bookworm python infrastructure is recent enough that cxfreeze can be installed against system packages rather than having to install a bunch of versioned dependencies.

I already tried that, since resolving all that dependencies by hand was my last option. But there was no success on my side. There is no cx-freeze in the system repositories (see screenshot) and mixing apt and pip dependencies together... hmm, i don't think that thats a good idea.

What I personally don't understand is the fact, that the internet access at built time is forbidden/prevented. I just accepted that fact and kept that, since I was very sure that theres a good reason for doing it that way.
Normally, a simple pip3 install with a pinned cx-freeze version should help us out of this dependency hell, but thats only possible with internet access.

According to splitting the Debian Bookworm things and the mlat-client patch in two different PR's - the mlat-client patch is required to get the Debian Bookworm package working. In my mind, this should be in one PR since it belongs together.

as365n4 · 2023-07-28T14:51:22Z

Our support generally tracks what the Pi Foundation supports, and they're still on bullseye, so it's hard to justify prioritizing a review of this at the moment.

You are aware that Raspberry Pi OS for arm64 and armhf is just Debian, where Raspberry Pi Foundation only provides firmware blobs, rpi-update etc. and only for Legacy Raspbian (armhf) they provide a full repo.

http://raspbian.raspberrypi.org/raspbian/dists/
http://raspbian.raspberrypi.org/raspbian/dists/bookworm/Release

Origin: Raspbian
Label: Raspbian
Suite: testing
Codename: bookworm
Date: Fri, 28 Jul 2023 10:35:12 UTC
Architectures: armhf
Components: main contrib non-free rpi firmware
Description: Debian armhf testing distribution for Raspberry Pi

https://archive.raspberrypi.org/debian/dists/
https://archive.raspberrypi.org/debian/dists/bookworm/Release

Origin: Raspberry Pi Foundation
Label: Raspberry Pi Foundation
Suite: testing
Codename: bookworm
Date: Mon, 30 Jan 2023 14:54:26 UTC
Architectures: armhf arm64 i386 amd64
Components: main untested
Description: Packages for the Raspberry Pi

https://raspberrypi.stackexchange.com/questions/137600/what-is-up-with-the-different-repositories-and-urls

WhoAmI0501 · 2023-07-28T15:54:31Z

I guess that we don't achieve anything with creating an PR there... i have written something according to that in #24 (comment).

I just saw, that you, @mutability, are the maintainer of the mlat-client. Didn't seen that before. I created an PR with the fix there and will remove the patch now from this PR.

WhoAmI0501 · 2023-07-28T16:19:42Z

I created an PR with the fix there and will remove the patch now from this PR.

Done. I assumed, that you will release the mlat-client fix as version 0.2.13.

mutability · 2023-08-01T09:04:15Z

Thanks! I hope to find some time to look at this in the next couple of weeks.

mixing apt and pip dependencies together... hmm, i don't think that thats a good idea.

It's not pip depenencies so much as "can you install cxfreeze from source while satisfying its dependencies via the system python packaging". In the past the answer was "no" because the system packages were too old for the version of cxfreeze we needed, so you ended up going down the rabbit-hole of satisfying all the transitive dependencies by hand. But maybe now the answer is "yes"?

What I personally don't understand is the fact, that the internet access at built time is forbidden/prevented.

It's mostly from trying to follow Debian policy:

For packages in the main archive, required targets must not attempt network access, except, via the loopback interface, to services on the build host that have been started by the build.

There are various good reasons for wanting this (stability/reproducibility of builds, privacy, etc) so I try to at least get to the point where you can prepare a standalone package build tree that can be (re-)built on a build machine predictably with no network access.

mutability · 2023-08-01T09:11:07Z

You are aware that Raspberry Pi OS for arm64 and armhf is just Debian

(a) rpi armhf binary packages are not the same as upstream Debian packages (as you will discover if you try to use a Debian armhf binary package on a Pi Zero / ZeroW)

(b) we target support for the Pi OS images which include the Pi Foundation specific packages and system configuration, rather than Debian directly. The changes in this image versus upstream Debian are not merely hardware support.

The litmus test for OS support is "is it on the download page as compatible-with-all-Raspberry-Pi-Models"

mutability · 2023-08-01T09:20:15Z

cxfreeze

One more note here -- the whole reason why cxfreeze is in the mix is that, at the time, the Raspbian/Debian python dependencies weren't stable enough to reliably package against; I had a constant stream of bug reports where a binary package with apparently-correct dependencies would be broken by an upstream Python update.

Maybe that has improved enough in current versions that we don't even need cxfreeze any more, which would make life easier..

WhoAmI0501 · 2023-08-04T21:34:07Z

It's mostly from trying to follow Debian policy

That absolutely makes sense. Never had an reason to read that policy before. :)

jaredmauch · 2023-08-17T16:32:42Z

I've used this to build for my pi and it's back online and reporting to FA, would like to see a bookworm package released

mutability · 2023-09-19T04:22:38Z

I'm working through this now and cxfreeze looks like it should be fine building against the system python packages in bookworm, so I'll probably go with that approach rather than pulling in all the dependencies.

mutability · 2023-09-19T14:18:09Z

dev branch now has tentative bookworm support, if you want to try it out.

(32-bit arm userspace with 64-bit kernel may be broken; I fixed it up for bullseye but haven't had a chance to test that combination on bookworm yet)

as365n4 · 2023-09-19T16:17:47Z

Build on arm64/Bookworm and experienced no issues.
But apt won't install the package. :-(
Reading package lists... Done
E: Unsupported file /home/test/flightaware/piaware_builder/piaware_9.0~dev_arm64.deb given on commandline

as365n4 · 2023-09-19T16:28:00Z

GNU nano 7.2 piaware_9.0-dev_arm64.buildinfo
Format: 1.0
Source: piaware
Binary: piaware piaware-dbgsym
Architecture: arm64
Version: 9.0-dev
Checksums-Md5:
8bf12d6f77fa3cf7e50233954c5293b5 3240368 piaware-dbgsym_9.0-dev_arm64.deb
1b016fe140a874b0295e36f0d3ebfa3f 5536636 piaware_9.0-dev_arm64.deb
Checksums-Sha1:
37c4d6ab54c62625ba40b69c074a9e4aaf6e22e9 3240368 piaware-dbgsym_9.0-dev_arm64.deb
692efa18b7ddb1cd258dc783acb9c7cb3e9e0f69 5536636 piaware_9.0~dev_arm64.deb
Checksums-Sha256:
406d6c6b7694e11dd5d2f9dc3bbe3d6eaf8cb36a6f9a798fbc9f9d9349e31959 3240368 piaware-dbgsym_9.0-dev_arm64.deb
dcc843a45218d59acda114fd25cc91db9acd2d1747b27a8b266ebad6ea9ac76c 5536636 piaware_9.0-dev_arm64.deb
Build-Origin: Debian
Build-Architecture: arm64
Build-Date: Tue, 19 Sep 2023 17:10:37 +0100
Build-Tainted-By:
merged-usr-via-aliased-dirs
usr-local-has-libraries
usr-local-has-programs
Installed-Build-Depends:
:
Environment:
DEB_BUILD_OPTIONS="parallel=4"
LANG="en_GB.UTF-8"
SOURCE_DATE_EPOCH="1695098811"

NB: replaced "~" with "-" in package name for readabilty.

mutability · 2023-09-19T16:40:21Z

Unsupported file /home/test/flightaware/piaware_builder/piaware_9.0~dev_arm64.deb given on commandline

Check your path, this apt error typically means "I can't read the file / file doesn't exist"

Failing that try installing with dpkg -i

as365n4 · 2023-09-19T16:45:20Z

Thanks, idiot me hadn't updated it's notes. All working now :-)

mutability · 2023-12-08T14:54:13Z

9.0 was released with bookworm support.

WhoAmI0501 mentioned this pull request Jul 19, 2023

Debian 12 Bookworm - bump dependencies to uptodate versions of cx_Freeze, zipp and importlib_metadata #23

Closed

WhoAmI0501 force-pushed the master branch 2 times, most recently from 9db908c to 219425d Compare July 19, 2023 12:49

WhoAmI0501 mentioned this pull request Jul 19, 2023

Debian 11 Dependency Issue #17

Open

WhoAmI0501 force-pushed the master branch from 83481b4 to 68fa6ac Compare July 19, 2023 14:33

WhoAmI0501 force-pushed the master branch from a44e4ff to 275562a Compare July 19, 2023 14:56

WhoAmI0501 mentioned this pull request Jul 28, 2023

Fix issues with python 3.11 according to PyFloatUnpack4 mutability/mlat-client#37

Open

WhoAmI0501 and others added 3 commits July 28, 2023 18:15

Introduce debian bookworm support

cdf235d

Do not prefer any IP version

45be757

Cleaning up sensible-build.sh using shellcheck

99a5220

WhoAmI0501 force-pushed the master branch from 275562a to 99a5220 Compare July 28, 2023 16:18

mutability closed this Dec 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce Debian Bookworm support #24

Introduce Debian Bookworm support #24

WhoAmI0501 commented Jul 19, 2023 •

edited

Loading

WhoAmI0501 commented Jul 19, 2023

WhoAmI0501 commented Jul 19, 2023

as365n4 commented Jul 19, 2023 •

edited

Loading

WhoAmI0501 commented Jul 19, 2023 •

edited

Loading

as365n4 commented Jul 19, 2023

WhoAmI0501 commented Jul 19, 2023

as365n4 commented Jul 19, 2023

WhoAmI0501 commented Jul 28, 2023

mutability commented Jul 28, 2023

mutability commented Jul 28, 2023

WhoAmI0501 commented Jul 28, 2023

as365n4 commented Jul 28, 2023 •

edited

Loading

WhoAmI0501 commented Jul 28, 2023

WhoAmI0501 commented Jul 28, 2023

mutability commented Aug 1, 2023

mutability commented Aug 1, 2023

mutability commented Aug 1, 2023

WhoAmI0501 commented Aug 4, 2023

jaredmauch commented Aug 17, 2023

mutability commented Sep 19, 2023

mutability commented Sep 19, 2023

as365n4 commented Sep 19, 2023 •

edited

Loading

as365n4 commented Sep 19, 2023 •

edited

Loading

mutability commented Sep 19, 2023 •

edited

Loading

as365n4 commented Sep 19, 2023

mutability commented Dec 8, 2023

Introduce Debian Bookworm support #24

Introduce Debian Bookworm support #24

Conversation

WhoAmI0501 commented Jul 19, 2023 • edited Loading

WhoAmI0501 commented Jul 19, 2023

WhoAmI0501 commented Jul 19, 2023

as365n4 commented Jul 19, 2023 • edited Loading

WhoAmI0501 commented Jul 19, 2023 • edited Loading

as365n4 commented Jul 19, 2023

WhoAmI0501 commented Jul 19, 2023

as365n4 commented Jul 19, 2023

WhoAmI0501 commented Jul 28, 2023

mutability commented Jul 28, 2023

mutability commented Jul 28, 2023

WhoAmI0501 commented Jul 28, 2023

as365n4 commented Jul 28, 2023 • edited Loading

WhoAmI0501 commented Jul 28, 2023

WhoAmI0501 commented Jul 28, 2023

mutability commented Aug 1, 2023

mutability commented Aug 1, 2023

mutability commented Aug 1, 2023

WhoAmI0501 commented Aug 4, 2023

jaredmauch commented Aug 17, 2023

mutability commented Sep 19, 2023

mutability commented Sep 19, 2023

as365n4 commented Sep 19, 2023 • edited Loading

as365n4 commented Sep 19, 2023 • edited Loading

mutability commented Sep 19, 2023 • edited Loading

as365n4 commented Sep 19, 2023

mutability commented Dec 8, 2023

WhoAmI0501 commented Jul 19, 2023 •

edited

Loading

as365n4 commented Jul 19, 2023 •

edited

Loading

WhoAmI0501 commented Jul 19, 2023 •

edited

Loading

as365n4 commented Jul 28, 2023 •

edited

Loading

as365n4 commented Sep 19, 2023 •

edited

Loading

as365n4 commented Sep 19, 2023 •

edited

Loading

mutability commented Sep 19, 2023 •

edited

Loading