Build full chromium binary #1

piercefreeman · 2023-08-26T15:39:20Z

The goal of this PR is to:

Switch chromium builder to github actions instead of ansible/aws
Build full chromium bundle to support headless v2 instead of the old headless v1 target

Current status:

The chromium build is succeeding on amazonlinux:2023, which has the new version of glib that chromium requires. However, since the lambda docker image is still on amazonlinux:2, this can't be copied over explicitly. We also need to bump or bundle the grpc dependencies of the lambda environment. This is currently failing. See notes below.

Notes:

The glib libraries in lambda are not modern enough to run chromium; They are still based on amazon-linux2 with ldd (GNU libc) 2.26 and Chromium needs at least 2.27 to build. This works fine to build the binary but the dynamic linking of glib won't be able to resolve on the lambda container at runtime.

libgthread-2.0 is an example of one of these dependencies that is required by chromium but not otherwise bundled in the Lambda runtime environment. This is the full list of dependencies as resolved in the build container.

bash-5.2# ldd chromium-116.0.5845.0
	linux-vdso.so.1 (0x00007ffd9ef7d000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fd068779000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fd068774000)
	libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007fd068631000)
	libgobject-2.0.so.0 => /lib64/libgobject-2.0.so.0 (0x00007fd0685d0000)
	libnss3.so => /lib64/libnss3.so (0x00007fd068490000)
	libnssutil3.so => /lib64/libnssutil3.so (0x00007fd06845d000)
	libsmime3.so => /lib64/libsmime3.so (0x00007fd06842f000)
	libnspr4.so => /lib64/libnspr4.so (0x00007fd0683ea000)
	libatk-1.0.so.0 => /lib64/libatk-1.0.so.0 (0x00007fd0683c0000)
	libatk-bridge-2.0.so.0 => /lib64/libatk-bridge-2.0.so.0 (0x00007fd068387000)
	libcups.so.2 => /lib64/libcups.so.2 (0x00007fd0682e7000)
	libgio-2.0.so.0 => /lib64/libgio-2.0.so.0 (0x00007fd068104000)
	libdrm.so.2 => /lib64/libdrm.so.2 (0x00007fd0680ec000)
	libdbus-1.so.3 => /lib64/libdbus-1.so.3 (0x00007fd068099000)
	libexpat.so.1 => /lib64/libexpat.so.1 (0x00007fd068069000)
	libxcb.so.1 => /lib64/libxcb.so.1 (0x00007fd06803d000)
	libxkbcommon.so.0 => /lib64/libxkbcommon.so.0 (0x00007fd067ff5000)
	libatspi.so.0 => /lib64/libatspi.so.0 (0x00007fd067fb8000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fd067edb000)
	libX11.so.6 => /lib64/libX11.so.6 (0x00007fd067d8f000)
	libXcomposite.so.1 => /lib64/libXcomposite.so.1 (0x00007fd067d8a000)
	libXdamage.so.1 => /lib64/libXdamage.so.1 (0x00007fd067d85000)
	libXext.so.6 => /lib64/libXext.so.6 (0x00007fd067d6f000)
	libXfixes.so.3 => /lib64/libXfixes.so.3 (0x00007fd067d67000)
	libXrandr.so.2 => /lib64/libXrandr.so.2 (0x00007fd067d57000)
	libgbm.so.1 => /lib64/libgbm.so.1 (0x00007fd067d46000)
	libpango-1.0.so.0 => /lib64/libpango-1.0.so.0 (0x00007fd067cf3000)
	libcairo.so.2 => /lib64/libcairo.so.2 (0x00007fd067bd5000)
	libasound.so.2 => /lib64/libasound.so.2 (0x00007fd067ac4000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fd067aaa000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fd0678a0000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fd075d55000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007fd067802000)
	libffi.so.6 => /lib64/libffi.so.6 (0x00007fd0677f8000)
	libplc4.so => /lib64/libplc4.so (0x00007fd0677f1000)
	libplds4.so => /lib64/libplds4.so (0x00007fd0677ec000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fd067794000)
	libavahi-common.so.3 => /lib64/libavahi-common.so.3 (0x00007fd067786000)
	libavahi-client.so.3 => /lib64/libavahi-client.so.3 (0x00007fd067772000)
	libgnutls.so.30 => /lib64/libgnutls.so.30 (0x00007fd06754a000)
	libz.so.1 => /lib64/libz.so.1 (0x00007fd067530000)
	libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x00007fd067529000)
	libmount.so.1 => /lib64/libmount.so.1 (0x00007fd0674e2000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fd0674b5000)
	libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007fd0673da000)
	libXau.so.6 => /lib64/libXau.so.6 (0x00007fd0673d4000)
	libXi.so.6 => /lib64/libXi.so.6 (0x00007fd0673c0000)
	libXrender.so.1 => /lib64/libXrender.so.1 (0x00007fd0673b1000)
	libwayland-server.so.0 => /lib64/libwayland-server.so.0 (0x00007fd06739b000)
	libfribidi.so.0 => /lib64/libfribidi.so.0 (0x00007fd06737b000)
	libthai.so.0 => /lib64/libthai.so.0 (0x00007fd067370000)
	libharfbuzz.so.0 => /lib64/libharfbuzz.so.0 (0x00007fd067272000)
	libpixman-1.so.0 => /lib64/libpixman-1.so.0 (0x00007fd0671b9000)
	libfontconfig.so.1 => /lib64/libfontconfig.so.1 (0x00007fd067169000)
	libfreetype.so.6 => /lib64/libfreetype.so.6 (0x00007fd067094000)
	libpng16.so.16 => /lib64/libpng16.so.16 (0x00007fd067058000)
	libxcb-shm.so.0 => /lib64/libxcb-shm.so.0 (0x00007fd067053000)
	libxcb-render.so.0 => /lib64/libxcb-render.so.0 (0x00007fd067044000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fd066f68000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fd066f4f000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fd066f48000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fd066f37000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fd066f30000)
	libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007fd066afa000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fd066ae4000)
	libp11-kit.so.0 => /lib64/libp11-kit.so.0 (0x00007fd0669b0000)
	libidn2.so.0 => /lib64/libidn2.so.0 (0x00007fd06698f000)
	libunistring.so.2 => /lib64/libunistring.so.2 (0x00007fd066809000)
	libtasn1.so.6 => /lib64/libtasn1.so.6 (0x00007fd0667f2000)
	libnettle.so.8 => /lib64/libnettle.so.8 (0x00007fd06679e000)
	libhogweed.so.6 => /lib64/libhogweed.so.6 (0x00007fd066759000)
	libgmp.so.10 => /lib64/libgmp.so.10 (0x00007fd0666af000)
	libblkid.so.1 => /lib64/libblkid.so.1 (0x00007fd066676000)
	libcap.so.2 => /lib64/libcap.so.2 (0x00007fd06666c000)
	liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fd066641000)
	libdatrie.so.1 => /lib64/libdatrie.so.1 (0x00007fd066636000)
	libgraphite2.so.3 => /lib64/libgraphite2.so.3 (0x00007fd066614000)
	libxml2.so.2 => /lib64/libxml2.so.2 (0x00007fd06649a000)
	libbz2.so.1 => /lib64/libbz2.so.1 (0x00007fd066486000)
	libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007fd066478000)
	libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007fd066453000)

Default lambda dependencies (logged by a simple exec("ldd ${chromiumPath}") in the lambda function):

43-5d38-4676-8998-100767f0cf48	INFO	ldd stdout: 	linux-vdso.so.1 (0x00007ffe8b756000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f36d4735000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f36d4517000)
	libglib-2.0.so.0 => not found
	libgobject-2.0.so.0 => not found
	libnss3.so => not found
	libnssutil3.so => /lib64/libnssutil3.so (0x00007f36d42e8000)
	libsmime3.so => not found
	libnspr4.so => /lib64/libnspr4.so (0x00007f36d40ab000)
	libatk-1.0.so.0 => not found
	libatk-bridge-2.0.so.0 => not found
	libcups.so.2 => not found
	libgio-2.0.so.0 => not found
	libdrm.so.2 => not found
	libdbus-1.so.3 => not found
	libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f36d3e7b000)
	libxcb.so.1 => not found
	libxkbcommon.so.0 => not found
	libatspi.so.0 => not found
	libm.so.6 => /lib64/libm.so.6 (0x00007f36d3b3b000)
	libX11.so.6 => not found
	libXcomposite.so.1 => not found
	libXdamage.so.1 => not found
	libXext.so.6 => not found
	libXfixes.so.3 => not found
	libXrandr.so.2 => not found
	libgbm.so.1 => not found
	libpango-1.0.so.0 => not found
	libcairo.so.2 => not found
	libasound.so.2 => not found
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f36d3925000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f36d3578000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f36e1f06000)
	libplc4.so => /lib64/libplc4.so (0x00007f36d3373000)
	libplds4.so => /lib64/libplds4.so (0x00007f36d316f000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f36d2f67000)

We can find these in the original /lib64 directory in the docker builder:

bash-5.2# ls -lah /usr/lib64 | grep libglib
lrwxrwxrwx  1 root root   16 Jun  9 21:50 libglib-2.0.so -> libglib-2.0.so.0
lrwxrwxrwx  1 root root   23 Jun  9 21:50 libglib-2.0.so.0 -> libglib-2.0.so.0.7400.7
-rwxr-xr-x  1 root root 1.3M Jun  9 21:52 libglib-2.0.so.0.7400.7

bash-5.2# ls -lah /usr/lib64 | grep libcups
lrwxrwxrwx  1 root root   17 Jun 22 22:36 libcupsimage.so -> libcupsimage.so.2
-rwxr-xr-x  1 root root  16K Jun 22 22:36 libcupsimage.so.2
lrwxrwxrwx  1 root root   12 Jun 22 22:36 libcups.so -> libcups.so.2
-rwxr-xr-x  1 root root 644K Jun 22 22:36 libcups.so.2

Running with these in the /lib folder will result in warnings across the board:

[pid=23][err] /tmp/chromium: relocation error: /tmp/aws/lib/libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference

This is because the runtime linker on the lambda (ld-linux) is still tied to the old version of glib. We can try copying over the new version and make sure our environment is pulling that. However it's important to be careful here; the rest of the system dependencies rely on LD_LIBRARY_PATH pointing to the system-wide version of glib. We can't overwrite this version or we'll see errors across the board. We can however patch the versions we're using in the executable to make sure we're finding the right ones:

patchelf --set-interpreter /tmp/aws/lib/ld-linux-x86-64.so.2 --set-rpath /tmp/aws/lib ${localPath}

This gets us some of the way there but the runtime still only resolves some of the dependencies with our desired versions (see the /tmp/aws/lib/ links). Many others are the environment default in /lib64 or otherwise not found.

2023-08-29T05:57:11.112Z	7a293fa4-73dd-4029-9e90-6fdc5c6f5ab3	INFO	ldd stdout: /tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /tmp/aws/lib/libglib-2.0.so.0)
/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /tmp/aws/lib/libglib-2.0.so.0)
/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/aws/lib/libglib-2.0.so.0)
/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /tmp/aws/lib/libglib-2.0.so.0)
/tmp/chromium: /lib64/libnssutil3.so: version `NSSUTIL_3.82' not found (required by /tmp/aws/lib/libnss3.so)
/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/aws/lib/libcups.so.2)
/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /tmp/aws/lib/libcups.so.2)
/tmp/chromium: /lib64/libselinux.so.1: no version information available (required by /tmp/aws/lib/libgio-2.0.so.0)
/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /tmp/aws/lib/libgio-2.0.so.0)
/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/aws/lib/libgio-2.0.so.0)
/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /tmp/aws/lib/libgio-2.0.so.0)
/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/aws/lib/libdrm.so.2)
/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/aws/lib/libdbus-1.so.3)
/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /tmp/aws/lib/libdbus-1.so.3)
/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/aws/lib/libxkbcommon.so.0)
/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/aws/lib/libX11.so.6)
/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /tmp/aws/lib/libX11.so.6)
/tmp/chromium: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /tmp/aws/lib/libgbm.so.1)
/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/aws/lib/libgbm.so.1)
/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /tmp/aws/lib/libgbm.so.1)
/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /tmp/aws/lib/libcairo.so.2)
/tmp/chromium: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /tmp/aws/lib/libcairo.so.2)
/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/aws/lib/libasound.so.2)
/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /tmp/aws/lib/libasound.so.2)
/tmp/chromium: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /tmp/aws/lib/libasound.so.2)
	linux-vdso.so.1 (0x00007ffe7d5a1000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fe724483000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe724265000)
	libglib-2.0.so.0 => /tmp/aws/lib/libglib-2.0.so.0 (0x00007fe731d42000)
	libgobject-2.0.so.0 => /tmp/aws/lib/libgobject-2.0.so.0 (0x00007fe731ce1000)
	libnss3.so => /tmp/aws/lib/libnss3.so (0x00007fe724125000)
	libnssutil3.so => /lib64/libnssutil3.so (0x00007fe723ef6000)
	libsmime3.so => /tmp/aws/lib/libsmime3.so (0x00007fe731cb3000)
	libnspr4.so => /lib64/libnspr4.so (0x00007fe723cb9000)
	libatk-1.0.so.0 => /tmp/aws/lib/libatk-1.0.so.0 (0x00007fe731c89000)
	libatk-bridge-2.0.so.0 => /tmp/aws/lib/libatk-bridge-2.0.so.0 (0x00007fe723c80000)
	libcups.so.2 => /tmp/aws/lib/libcups.so.2 (0x00007fe723be0000)
	libgio-2.0.so.0 => /tmp/aws/lib/libgio-2.0.so.0 (0x00007fe7239fd000)
	libdrm.so.2 => /tmp/aws/lib/libdrm.so.2 (0x00007fe7239e5000)
	libdbus-1.so.3 => /tmp/aws/lib/libdbus-1.so.3 (0x00007fe723992000)
	libexpat.so.1 => /lib64/libexpat.so.1 (0x00007fe723762000)
	libxcb.so.1 => /tmp/aws/lib/libxcb.so.1 (0x00007fe723736000)
	libxkbcommon.so.0 => /tmp/aws/lib/libxkbcommon.so.0 (0x00007fe7236ee000)
	libatspi.so.0 => /tmp/aws/lib/libatspi.so.0 (0x00007fe7236b1000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fe72336f000)
	libX11.so.6 => /tmp/aws/lib/libX11.so.6 (0x00007fe723223000)
	libXcomposite.so.1 => /tmp/aws/lib/libXcomposite.so.1 (0x00007fe72321e000)
	libXdamage.so.1 => /tmp/aws/lib/libXdamage.so.1 (0x00007fe723219000)
	libXext.so.6 => /tmp/aws/lib/libXext.so.6 (0x00007fe723203000)
	libXfixes.so.3 => /tmp/aws/lib/libXfixes.so.3 (0x00007fe7231fb000)
	libXrandr.so.2 => /tmp/aws/lib/libXrandr.so.2 (0x00007fe7231eb000)
	libgbm.so.1 => /tmp/aws/lib/libgbm.so.1 (0x00007fe7231da000)
	libpango-1.0.so.0 => /tmp/aws/lib/libpango-1.0.so.0 (0x00007fe723187000)
	libcairo.so.2 => /tmp/aws/lib/libcairo.so.2 (0x00007fe723069000)
	libasound.so.2 => /tmp/aws/lib/libasound.so.2 (0x00007fe722f58000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fe722d42000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fe722993000)
	/tmp/aws/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fe731c64000)
	libpcre2-8.so.0 => not found
	libffi.so.6 => /lib64/libffi.so.6 (0x00007fe722789000)
	libplc4.so => /lib64/libplc4.so (0x00007fe722584000)
	libplds4.so => /lib64/libplds4.so (0x00007fe722380000)
	librt.so.1 => /lib64/librt.so.1 (0x00007fe722178000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fe721f2a000)
	libavahi-common.so.3 => not found
	libavahi-client.so.3 => not found
	libgnutls.so.30 => not found
	libz.so.1 => /lib64/libz.so.1 (0x00007fe721d15000)
	libgmodule-2.0.so.0 => not found
	libmount.so.1 => not found
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fe721aec000)
	libsystemd.so.0 => not found
	libXau.so.6 => not found
	libXi.so.6 => not found
	libXrender.so.1 => not found
	libwayland-server.so.0 => not found
	libfribidi.so.0 => not found
	libthai.so.0 => not found
	libharfbuzz.so.0 => not found
	libpixman-1.so.0 => not found
	libfontconfig.so.1 => /lib64/libfontconfig.so.1 (0x00007fe7218a7000)
	libfreetype.so.6 => /lib64/libfreetype.so.6 (0x00007fe7215f1000)
	libpng16.so.16 => not found
	libxcb-shm.so.0 => not found
	libxcb-render.so.0 => not found
	libXrender.so.1 => not found
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fe72130b000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fe7210da000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fe720ed6000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fe720cc5000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fe720ac1000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fe7208ab000)
	libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fe720649000)
	libuuid.so.1 => /lib64/libuuid.so.1 (0x00007fe720444000)
	libbz2.so.1 => /lib64/libbz2.so.1 (0x00007fe720234000)
	libpng15.so.15 => /lib64/libpng15.so.15 (0x00007fe720009000)

Without media remote enabled, full builds fail on Chromium. Other code paths rely on CastRemotingConnector::ConnectWithMediaRemoter whether or not enable_media_remoting is enabled, but the ninja configuration will only include the chrome/browser/chrome_content_browser_client.cc if this flag is truthy.

Sparticuz · 2023-08-28T17:31:40Z

This is great 😃!

I was thinking about having GH Actions do the compilation, but I was worried about how long the compilation takes. Does Github have usage limits on free repos? I would love to get the Ansible requirement removed as it's basically a bottleneck on me running the compilation, and ansible fails a number of times because my dev box loses connection to the ec2 instance.

Second, you'll need to take a look at .github/workflows/aws.yml. It's probably mis-named because it basically runs the tests. The tests, if you haven't looked at it yet, takes a md5 of a screenshot and compares it. It if matches, that's great. If not, what you can do is output a base64 of the screenshot so you can see it outside of the sam container.

All in all, I agree with you that I think the majority of the issues are because it's using the headless_shell example. Moving away from that I think will be a great step.

piercefreeman · 2023-08-28T17:47:26Z

@Sparticuz Good points! Will take a look at those test cases once I get a working lambda deployed.

I was thinking about having GH Actions do the compilation, but I was worried about how long the compilation takes. Does Github have usage limits on free repos?

I tried with the vanilla CI machines, but they were indeed too slow / didn't have enough available disk space to clone the chromium codebase and its dependencies. I'm instead using Github's large runners - specifically the 16CPU/64GB memory box, which runs a full build in about 4 hours. That's priced at $0.064/min so roughly $15 for each compilation. It's more than a similarly configured VM from AWS, but given the rest of our CI pipeline and avoiding some of the ansible connection issues seemed worth it to me.

The only snag for an OSS project is these runners can only be launched into corporate accounts. We have some lambdas running at MonkeySee, so we're happy to keep building these periodically, but that might be a limitation for other branches.

Sparticuz · 2023-09-01T19:30:18Z

I posted about the glibc problem a year ago :/ tracked here: aws/aws-lambda-base-images#59

piercefreeman · 2023-09-01T20:36:57Z

Unfortunate this has been blocking for so long! The two most promising avenues here seem to be:

Build the full chromium codebase, but with use_glib = false (like they do in headless v1). This intuitively feels like it wouldn't work because there are glib dependencies in the main codebase, but would be easy and might be worth a short.
Wrangle the new glib dependencies to copy over from amazonlinux2023 to amazonlinux2. I've gotten close in this PR with the dependency sniffing and patchelf, but for some reason it still isn't resolving the packages even though some of the libraries are getting picked up.

Any thoughts on the above? FWIW we're temporarily moved over to just building a separate docker image based on Ubuntu and running the lambdas on there. Bootup times are indeed slower but workable to avoid this dependency hell. Still shooting to get this PR up and running though.

Sparticuz · 2023-09-01T20:53:22Z

Indeed it is frustrating. My Issue on the AWS repo was originally for AL2022, which never came out. I'm skeptical they will even update the node18 image to be AL2023 at all and might just start it with the node20 image.

My build currently builds with the headless.gn build options, but I guess the headless_shell build target may do more to skip glibc?, or maybe headless_shell might work with older versions? I have no idea? An old glibc was also the reason why I could never get an arm version of chromium to run on lambda, which has been a feature I've wanted ever since the graviton instances came out.

I would not wish the dependency hell on anyone 😆. Personally, I don't see much value in headless v2, maybe because I'm not scraping websites pretending to be a normal user agent. I understand that v2 has much better compatibility, but I've not run into those issues.

As far as a separate docker image goes, my opinion is that the package is meant for running on the stock lambda containers. If you are using/want to use a separate container, then why not just install chromium at the system level instead...

iwaduarte · 2023-10-06T14:38:14Z

Thanks for all the info on this PR it led me to a journey I had to do to understand the whole problem.

I am still looking for patchy solutions and dependency hell fix but it seems that we could use custom runtimes through AL2023 and considering that Nodejs 20 LTS is around the corner I think that should be an interesting solution:

aws/aws-lambda-base-images#92 (comment)

How about we offer a full build instead of headless.gn with the disclaimer that currently only custom runtimes are supported (AL2023 specifically)? Considering that it took approx. 1 month from LTS to being supported in AWS runtimes and they are fully migrating now that would be a solution that would integrate with a new runtime anyway.

Thoughts? @Sparticuz @piercefreeman

Sparticuz · 2023-10-06T15:00:31Z

Wow, I didn't see that comment! I will be looking into quite a few options now that al2023 is available. There has also been some changes to @puppeteer/browsers that I've been interested in taking advantage of. Stay tuned!

piercefreeman · 2023-10-06T15:03:02Z

@iwaduarte Thanks for linking to that thread, hadn't seen the announcement about AL2023 yet. They also mentioned they're aiming to get GA out within the next couple of weeks. Fingers crossed.

Right now you'd still need to use this approach within a custom container on lambda, so I'm not sure using AL2023 as your docker base image really gains you that much. Docker sizes seem to be somewhat invariant with the size of the payload. But yes, given the approaching support for this in a format runtime, it does seem like a good idea to bump the base version of this builder to the new preview. The brotli compressed artifact should then just work-as is when they get to a final release.

@Sparticuz - do you want to take it from here? Or anything that would be useful on my side?

Sparticuz · 2023-10-06T15:06:16Z

Just leave this fork up for a bit in case I need to cherry pick any of your commits, otherwise I'll ping you if I need anything. Thanks @piercefreeman

piercefreeman added 3 commits August 26, 2023 08:38

Basic github action build pipeline

67919b7

Pull chromium version number

4f83fa8

Update ownership of artifact path

f47238b

piercefreeman force-pushed the feature/github-actions branch from 1a0d979 to 0886f21 Compare August 26, 2023 16:05

Switch to yum

d766091

piercefreeman force-pushed the feature/github-actions branch from 0886f21 to d766091 Compare August 26, 2023 16:08

Update entrypoint

cbe9ca1

piercefreeman force-pushed the feature/github-actions branch from 900c165 to cbe9ca1 Compare August 26, 2023 16:14

piercefreeman added 2 commits August 26, 2023 09:18

Add python 3

1113e91

Add swiftshader

bace20d

piercefreeman force-pushed the feature/github-actions branch from 6cf1128 to 8c9266c Compare August 26, 2023 17:39

Improve patching commands

e1e8b04

piercefreeman force-pushed the feature/github-actions branch from 8c9266c to e1e8b04 Compare August 26, 2023 17:41

Use src directory

46c241e

piercefreeman changed the title ~~Switch builder to github actions~~ Build full chromium binary Aug 26, 2023

piercefreeman mentioned this pull request Aug 26, 2023

[BUG] Headless 'new' mode uses a headless user agent Sparticuz/chromium#90

Open

piercefreeman added 4 commits August 26, 2023 11:50

Generate build files post-mount

d4ab52f

Upgrade to amazonlinux 2023

292d113

Update awk

cef9c44

piercefreeman force-pushed the feature/github-actions branch from 01a9563 to d590c62 Compare August 28, 2023 01:01

Test artifact upload and attachment

7cb0793

piercefreeman force-pushed the feature/github-actions branch from d590c62 to 7cb0793 Compare August 28, 2023 01:03

Bundle dependencies from builder

b755149

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build full chromium binary #1

Build full chromium binary #1

piercefreeman commented Aug 26, 2023 •

edited

Loading

Sparticuz commented Aug 28, 2023

piercefreeman commented Aug 28, 2023

Sparticuz commented Sep 1, 2023

piercefreeman commented Sep 1, 2023

Sparticuz commented Sep 1, 2023

iwaduarte commented Oct 6, 2023

Sparticuz commented Oct 6, 2023

piercefreeman commented Oct 6, 2023

Sparticuz commented Oct 6, 2023

Build full chromium binary #1

Are you sure you want to change the base?

Build full chromium binary #1

Conversation

piercefreeman commented Aug 26, 2023 • edited Loading

Sparticuz commented Aug 28, 2023

piercefreeman commented Aug 28, 2023

Sparticuz commented Sep 1, 2023

piercefreeman commented Sep 1, 2023

Sparticuz commented Sep 1, 2023

iwaduarte commented Oct 6, 2023

Sparticuz commented Oct 6, 2023

piercefreeman commented Oct 6, 2023

Sparticuz commented Oct 6, 2023

piercefreeman commented Aug 26, 2023 •

edited

Loading