ci: switch to core20 for snap

[deepak1556]

Fixes #116690
Fixes #119436
Fixes #147166
Fixes #155094
Fixes #168427
Fixes #133856
Fixes #146890

vscode-encrypt fails to load with GLIBC_2.25 not found error.

ldd vscode-encrypt.node
  libstdc++.so.6 => /snap/core/current/usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc887a3a000)
  libgcc_s.so.1 => /snap/core/current/lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc887824000)
  libpthread.so.0 => /snap/core/current/lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc887607000)
  libc.so.6 => /snap/core/current/lib/x86_64-linux-gnu/libc.so.6 (0x00007fc88723d000)
  libm.so.6 => /snap/core/current/lib/x86_64-linux-gnu/libm.so.6 (0x00007fc886f34000)
  /lib64/ld-linux-x86-64.so.2 (0x00007fc8882bc000)

libc.so.6 from ubuntu16.04 core is not sufficient for support.
Switching to core18 matches with the build environment used for the native module
https:/microsoft/vscode-linux-build-agent/blob/main/bionic-x64/Dockerfile

[joaomoreno]

Relevant: #119436

[deepak1556]

Hi @anonymouse64, I was informed that this change might be of interest to you. Thoughts ? I chose core18 to match the build environment used today.

Side note: Should I revert #119478 in favor of #118116 in this PR, since we are using base: ?

[TylerLeonhardt]

this makes sense to me but would like to wait for someone else to sign off as well

[anonymouse64]

If the snap that is built here is working correctly, then +1 from me, I'm happy to test a .snap file built from this PR if someone can make it available for download, I don't have the time or expertise to build VS code unfortunately. I do use the VS code snap on a daily basis however :-)

[deepak1556]

@anonymouse64 here is the snap package for testing https://az764295.vo.msecnd.net/insider/cebe99f55eee750bfa53ef644c00762ad2d21e22/code-insider-x64-1625157141.snap

[anonymouse64]

unfortunately I don't think this is ready/working yet:

$ snap run code-insiders
*** stack smashing detected ***: terminated
/snap/code-insiders/x1/electron-launch: line 29: 324320 Aborted                 (core dumped) "$SNAP/usr/lib/$ARCH/gdk-pixbuf-2.0/gdk-pixbuf-query-loaders" > "$GDK_PIXBUF_MODULE_FILE"
/snap/code-insiders/x1/usr/share/code-insiders/bin/../code-insiders: relocation error: /snap/code-insiders/x1/usr/share/code-insiders/libffmpeg.so: symbol pthread_cond_destroy version GLIBC_2.3.2 not defined in file libpthread.so.0 with link time reference

did anyone manage to run this version of VS code successfully when built as a snap with core18 base ? seems like a $LD_LIBRARY_LOAD path issue where it is using the wrong glibc from the host and not from the base snap

[deepak1556]

Can confirm the crash and that the dynamic libraries are linked to the host OS, @anonymouse64 any thoughts what can cause this to happen, we are not messing with ld paths in the build.

[anonymouse64]

@deepak1556 I will defer to @sergiusens or @igorljubuncic for more help on that, I think that it might actually be that previously snapcraft was setting some paths for you, and it stopped doing this when you switched to using a base in the snapcraft.yaml.

[deepak1556]

Thanks @anonymouse64 , in case it helps we are using snapcraft:stable container to build our snaps.

[ghost]

I successfully built vscode with core20 - but you need to build it in a 20.04 container. Launching the 16.04 docker container will lead to the core dump issue as you see above. Here's the yaml that I used - but you need to edit the CI workflow to create a different snapcore container.

https:/snapcore/snapcraft/blob/master/docker/Dockerfile

Similarly, if you want to use 18.04, you need an 18.04 build env.

The yaml (ignore the version, as I built it locally):

version: "1.0"
summary: Code editing. Redefined.
description: |
  Visual Studio Code is a new choice of tool that combines the
  simplicity of a code editor with what developers need for the core
  edit-build-debug cycle.

architectures:
  - build-on: amd64
    run-on: amd64

grade: stable
confinement: classic
base: core20

parts:
  code:
    plugin: dump
    source: ./code_1.54.3-1615806378_amd64.deb
    source-type: deb
    stage-packages:
      - libx11-xcb1
      - libx11-6
      - libxcomposite1
      - libxdamage1
      - libxext6
      - libxfixes3
      - libxrandr2
      - libasound2
      - libatk1.0-0
      - libatk-bridge2.0-0
      - libatspi2.0-0
      - libcairo2
      - libdrm2
      - libgbm1
      - libgdk-pixbuf2.0-0
      - libgtk-3-0
      - libnspr4
      - libnss3
      - libpango-1.0-0
      - libsecret-1-0
      - libxcb1
      - libxkbcommon0
      - libxkbfile1

apps:
  code:
    command: usr/share/code/bin/code
    common-id: code.desktop
    environment:
      DISABLE_WAYLAND: 1
      GSETTINGS_SCHEMA_DIR: usr/share/glib-2.0/schemas

  url-handler:
    command: usr/share/code/bin/code --open-url
    environment:
      DISABLE_WAYLAND: 1
      GSETTINGS_SCHEMA_DIR: usr/share/glib-2.0/schemas```

[deepak1556]

@igorljubuncic to clarify, the snap packaging is done in two steps:

First we build vscode application on a custom 18.04 container https:/microsoft/vscode-linux-build-agent/blob/main/bionic-x64/Dockerfile and create a ci artifact of the build. This is the reason I chose core18 as we cannot move to 20.04 here yet.

Next in a separate pipeline we use snapcraft:stable container to run the snap commands for creating the package with the above artifact https:/microsoft/vscode/blob/main/build/azure-pipelines/linux/snap-build-linux.yml

Launching the 16.04 docker container will lead to the core dump issue as you see above.

If I am reading https:/snapcore/snapcraft/blob/master/docker/README.md correctly, snapcraft:stable is a 16.04 container ? Is there a 18.04 version published or should we build a custom version based on https:/snapcore/snapcraft/blob/master/docker/Dockerfile ?

[ghost]

Yes, it is 16.04 - I am not aware of an existing 18.04 image. But you can spin off your own - just change the image declaration to download the right image.

[deepak1556]

Thanks for the clarification, that helps!

[deepak1556]

@igorljubuncic I am trying to switch to use gnome-3-34 for core18 but looks like the extension does not work with classic confinement. What is the way forward here ?

[ghost]

One thing you can do is expand a snapcraft.yaml that has a declared extension - and then you can manually declare all the pieces that are automatically hidden behind the scenes.

snapcraft expand-extensions will look for snapcraft.yaml and expand it on the standard output, but you can also redirect the output to a file, and compare the two declarations, with and without. Please try that and see if that helps.

[anonymouse64]

note that you can't use the gnome extension with a classic snap, and furthermore you also can't use the gnome content interface providing snaps with a classic snap as well. Has anyone tried changing this snap to just use base: core in the snapcraft.yaml? that may be enough to switch from building with the "legacy" mode snapcraft and to using new snapcraft which understands the LZO compression option and will have more improvements. It's not as nice as fully upgrading to a newer base, but it could be a stepping stone.

[deepak1556]

@igorljubuncic thanks for the pointer, expanding it shows that the extension adds some plugs to use the gnome content interface which I don't think will work with classic snaps as mentioned by @anonymouse64 ?

plugs:
  desktop:
    mount-host-font-cache: false
  gnome-3-34-1804:
    default-provider: gnome-3-34-1804
    interface: content
    target: $SNAP/gnome-platform
  gtk-3-themes:
    default-provider: gtk-common-themes
    interface: content
    target: $SNAP/data-dir/themes
  icon-themes:
    default-provider: gtk-common-themes
    interface: content
    target: $SNAP/data-dir/icons
  sound-themes:
    default-provider: gtk-common-themes
    interface: content
    target: $SNAP/data-dir/sounds

@anonymouse64 we definitely want to use base:core18 as the application is being built on ubuntu:18.04 container. The main motivation for the move was some native dependencies didn't get required glibc version #127320 (comment)

Does the above findings imply that we need strict confinement to use core18 base ? Or is there a way to use gnome-3-34 without snap interface ?

[anonymouse64]

@deepak1556 well not necessarily, you can keep using classic and use a core18 base, but the gnome extension is our preferred way of setting up the environment variables and other such things so that most strictly confined applications "just work" out of the box without special fiddling in the YAML. You can of course setup everything that is setup by the extension manually, and without the content interface connections. The content interface simply shares the files from these other snaps with your snap so that your snap does not need to ship those files and there is less duplication across different graphical gnome apps.

There are a few options to move forward if you determine that you do indeed need the libraries and files from these gnome content snaps.

1.) You can simply copy the files from these other snaps into your own snap and put them in equivalent locations, for example the files that gnome-3-34-1804 content plug shares are put at $SNAP/gnome-platform - you can add a part to your snapcraft.yaml which copies these files into $SNAPCRAFT_PART_INSTALL/gnome-platform and it will work the same as if you had the content interface connected, albeit your snap will be slightly larger. Another disadvantage of this is that you need to handle updates to these other snaps yourself when you build these snaps, i.e. normally if there is an update/fix to the gnome-3-34-1804 snap, it can be pushed out and dependent strict applications can get it immediately without needing to rebuild their snap like gnome-calculator for example, but here since you are copying the files into your snap you need to rebuild the snap to benefit from the updates. this may not be an issue for VS code, since updates to the snap seem to be relatively recent judging by at least the updates I get on my machine...
2.) You can in fact use files from other snaps in a classic snap like with plugs for a strict snap, by doing the mounts that correspond to the implementation of the content interface manually, i.e. mount --bind /snap/gnome-3-34-1804/current/gnome-platform $SNAP/gnome-platform will do the same thing as the content interface plug mentioned above, your snap just needs to make sure that gnome-3-34-1804 is installed, which it can actually do since it is a classic snap and not strictly confined. The issue with this is that you need to be careful to make sure that snaps you depend on are installed, and that you (maybe) monitor for changes to the source bind mounts, because for example if you do the bind mount as I mentioned above in a shell script that is part of the launch of VS code, then a refresh happens to gnome-3-34-1804, your snap will still have the previous revision mounted, since bind mounts don't follow new symlink targets IIRC. This may not be a big deal since your application will just continue to use the old revisions of the dependent snaps until the user reboots their machine or logs out or something like that and you perform the mounts again.
3. We (snapd developers) are aware of this limitation and have it generally as a TODO to make content interfaces work with classic snaps the same way they do with strict snaps, and the biggest part of that work was actually already done to support parallel instances of classic snaps, but there is still a bit of work left to do to fully enable this use-case. So you could just wait until we enable the content interface for classic snaps, but it's not on our immediate roadmap so it could be a few months (or maybe even a year or two) before this is available to use, but the VS code snap is not the only one that suffers from this issue, so the more snaps that are suffering from this situation the more likely we are to fix it sooner I suppose.

I would recommend experimenting a bit with 1 as a start

[deepak1556]

Thanks @anonymouse64 , I will give 1) a try looks like it might get this PR moving again.

[dverbru]

This blocks #168427.

[deepak1556]

This iteration I tried to revamp this PR and update our snap base to core20, although overall build is looking good there are a couple of issues that needs to be addressed before getting to the finish line. Following test was made on Fedora 36

• dlopen calls from the binary end up loading libraries from the system and not the ones bundled with the snap
  • Some example failures, first when trying to load EGL for software rendering, second when trying to load curl for uploading crash reports. There are more scenarios in the runtime, just these two are caught on startup,

Failed to load EGL library: libEGL.so.1: /snap/core20/current/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /lib64/libEGL.so.1)

 dlopen:libcurl.so: cannot open shared object file: No such file or directory
[0223/173421.249105:ERROR:http_transport_libcurl.cc(118)] dlopen:libcurl-gnutls.so.4: cannot open shared object file: No such file or directory
[0223/173421.249113:ERROR:http_transport_libcurl.cc(118)] dlopen:libcurl-nss.so.4: cannot open shared object file: No such file or directory
[0223/173421.249118:ERROR:http_transport_libcurl.cc(118)] dlopen:/snap/core20/current/lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.33' not found (required by /lib64/libcurl.so.4)

• Library is available at following locations inside /snap/code-insiders/current

$ cd /snap/code-insiders/current/
[parallels@fedora current]$ find . -name libEGL*
./gnome-platform/usr/lib/x86_64-linux-gnu/libEGL.so
./gnome-platform/usr/lib/x86_64-linux-gnu/libEGL.so.1
./gnome-platform/usr/lib/x86_64-linux-gnu/libEGL.so.1.1.0
./gnome-platform/usr/lib/x86_64-linux-gnu/libEGL_mesa.so.0
./gnome-platform/usr/lib/x86_64-linux-gnu/libEGL_mesa.so.0.0.0
./usr/share/code-insiders/libEGL.so

[parallels@fedora current]$ find . -name libcurl*
./usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.3
./usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
./usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.6.0
./usr/lib/x86_64-linux-gnu/libcurl-nss.so.3
./usr/lib/x86_64-linux-gnu/libcurl-nss.so.4
./usr/lib/x86_64-linux-gnu/libcurl-nss.so.4.6.0
./usr/lib/x86_64-linux-gnu/libcurl.so.4
./usr/lib/x86_64-linux-gnu/libcurl.so.4.6.0

• Reading up on how snapcraft adjusts RPATH for classic confinement to be able to load the correct libraries https://snapcraft.io/docs/classic-confinement, I have added a step in the prime stage to output the final adjusted RPATH of the binary and this is the current output

+ patchelf --print-rpath /__w/1/s/.build/linux/snap/x64/code-insiders-x64/prime/usr/share/code-insiders/code-insiders
 $ORIGIN/../../lib/x86_64-linux-gnu:$ORIGIN:/snap/core20/current/lib/x86_64-linux-gnu

• Given the presence of $ORIGIN/../../lib/x86_64-linux-gnu:$ORIGIN, I would expect both EGL and curl libraries to be loaded from within the snap, dlopen search order https://man7.org/linux/man-pages/man3/dlopen.3.html should have first attempted the RPATH, so why is it loaded from the system ?

• Currently the following library locations https:/snapcore/snapcraft/blob/ce9c24ee02e4d6ace4502eb8f2fb63095b40764d/extensions/desktop/common/desktop-exports#L68-L71 from gnome-platform are not part of the final RPATH, should this be patched in as part of the prime step ?

• Gpu process crashes on launch with SIGSEGV

  • Chromium gpu process tries to establish gpu memory buffers via libgbm and then attempts to load libgallium_dri.so which eventually crashes inside this library. Quick check confirms that libgdm.so.1 is available inside the snap package but the mesa libraries is not available either in the gnome snap or core20 snap. So looks like we ended up loading a system installed library. Should I bundle the mesa content snap https:/MirServer/mesa-core20 to address this issue ?

[deepak1556]

@joaomoreno this is now good to go 🚀

Build for verification: Insiders snap x64

[joaomoreno]

This is great @deepak1556, thanks for all the hard work! After my round of testing I didn't hit any blockers, so let's go ahead with this. 💪