feat!: Rewrite build system and third-party dependencies

This work was done over ~80 individual commits in the `cmake` branch.

BEGIN_COMMIT_OVERRIDE
feat!: Replace gyp build system with CMake

Closes shaka-project#1047

feat!: Update all dependencies
fix: Fix Python 3.10+ compatibility in scripts
feat!: Drop Python 2 support in all scripts
feat!: Replace glog with absl::log, tweak log output and flags
feat: Add install target to build system
feat: Respect the file mode for HttpFiles (shaka-project#1081)
feat: Move all third-party deps into git submodules (shaka-project#1083)
fix: Fix local files with UTF8 names (shaka-project#1246)
fix: Fix build on FreeBSD (shaka-project#1287)
fix: Fix clang build (shaka-project#1288)
fix: Fix various build issues on macOS
fix: Fix various build issues on Windows
END_COMMIT_OVERRIDE

.github/workflows/README.md

@@ -1,39 +1,63 @@
 # GitHub Actions CI
 
-## Actions
- - `custom-actions/lint-packager`:
- Lints Shaka Packager. You must pass `fetch-depth: 2` to `actions/checkout`
- in order to provide enough history for the linter to tell which files have
- changed.
- - `custom-actions/build-packager`:
- Builds Shaka Packager. Leaves build artifacts in the "artifacts" folder.
- Requires OS-dependent and build-dependent inputs.
- - `custom-actions/test-packager`:
- Tests Shaka Packager. Requires OS-dependent and build-dependent inputs.
- - `custom-actions/build-docs`:
- Builds Shaka Packager docs.
-
-## Workflows
- - On PR:
- - `build_and_test.yaml`:
- Builds and tests all combinations of OS & build settings. Also builds
- docs.
- - On release tag:
- - `github_release.yaml`:
- Creates a draft release on GitHub, builds and tests all combinations of OS
- & build settings, builds docs on all OSes, attaches static release binaries
- to the draft release, then fully publishes the release.
- - On release published:
- - `docker_hub_release.yaml`:
- Builds a Docker image to match the published GitHub release, then pushes it
- to Docker Hub.
- - `npm_release.yaml`:
- Builds an NPM package to match the published GitHub release, then pushes it
- to NPM.
- - `update_docs.yaml`:
- Builds updated docs and pushes them to the gh-pages branch.
+## Reusable workflows
+ - `build.yaml`:
+ Build and test all combinations of OS & build settings. Also builds docs on
+ Linux.
+
+ - `build-docs.yaml`:
+ Build Packager docs. Runs only on Linux.
+
+ - `build-docker.yaml`:
+ Build the official Docker image.
+
+ - `lint.yaml`:
+ Lint Shaka Packager.
+
+ - `publish-docs.yaml`:
+ Publish Packager docs. Runs on the latest release.
+
+ - `publish-docker.yaml`:
+ Publish the official docker image. Runs on all releases.
+
+ - `publish-npm.yaml`:
+ Publish binaries to NPM. Runs on all releases.
+
+ - `test-linux-distros.yaml`:
+ Test the build on all Linux distros via docker.
+
+## Composed workflows
+ - On PR (`pr.yaml`), invoke:
+ - `lint.yaml`
+ - `build.yaml`
+ - `build-docs.yaml`
+ - `build-docker.yaml`
+ - `test-linux-distros.yaml`
+
+## Release workflow
+ - `release-please.yaml`
+ - Updates changelogs, version numbers based on conventional commits syntax
+ and semantic versioning
+ - https://conventionalcommits.org/
+ - https://semver.org/
+ - Generates/updates a PR on each push
+ - When the PR is merged, runs additional steps:
+ - Creates a GitHub release
+ - Invokes `publish-docs.yaml` to publish the docs
+ - Invokes `publish-docker.yaml` to publish the docker image
+ - Invokes `build.yaml`
+ - Attaches the binaries from `build.yaml` to the GitHub release
+ - Invokes `publish-npm.yaml` to publish the binaries to NPM
+
+## Common workflows from shaka-project
+ - `sync-labels.yaml`
+ - `update-issues.yaml`
+ - `validate-pr-title.yaml`
 
 ## Required Repo Secrets
+ - `RELEASE_PLEASE_TOKEN`: A PAT for `shaka-bot` to run the `release-please`
+ action. If missing, the release workflow will use the default
+ `GITHUB_TOKEN`
  - `DOCKERHUB_CI_USERNAME`: The username of the Docker Hub CI account
  - `DOCKERHUB_CI_TOKEN`: An access token for Docker Hub
  - To generate, visit https://hub.docker.com/settings/security
@@ -47,3 +71,12 @@
  - `NPM_PACKAGE_NAME`: Not a true "secret", but stored here to avoid someone
  pushing bogus packages to NPM during CI testing from a fork
  - In a fork, set to a private name which differs from the production one
+
+## Repo Settings
+
+Each of these workflow features can be enabled by creating a "GitHub
+Environment" with the same name in your repo settings. Forks will not have
+these enabled by default.
+
+ - `debug`: enable debugging via SSH after a failure
+ - `self_hosted`: enable self-hosted runners in the build matrix

.github/workflows/build-docker.yaml

@@ -0,0 +1,37 @@
+# Copyright 2022 Google LLC
+#
+# Use of this source code is governed by a BSD-style
+# license that can be found in the LICENSE file or at
+# https://developers.google.com/open-source/licenses/bsd
+
+# A workflow to build the official docker image.
+name: Official Docker image
+
+# Runs when called from another workflow.
+on:
+ workflow_call:
+ inputs:
+ ref:
+ required: true
+ type: string
+
+# By default, run all commands in a bash shell. On Windows, the default would
+# otherwise be powershell.
+defaults:
+ run:
+ shell: bash
+
+jobs:
+ official_docker_image:
+ name: Build
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v3
+ with:
+ ref: ${{ inputs.ref }}
+ submodules: recursive
+
+ - name: Build
+ shell: bash
+ run: docker buildx build .

.github/workflows/build-docs.yaml

@@ -0,0 +1,77 @@
+# Copyright 2022 Google LLC
+#
+# Use of this source code is governed by a BSD-style
+# license that can be found in the LICENSE file or at
+# https://developers.google.com/open-source/licenses/bsd
+
+# A reusable workflow to build Packager docs. Leaves docs output in the
+# "gh-pages" folder. Only runs in Linux due to the dependency on doxygen,
+# which we install with apt.
+name: Build Docs
+
+# Runs when called from another workflow.
+on:
+ workflow_call:
+ inputs:
+ ref:
+ required: true
+ type: string
+
+ # If true, start a debug SSH server on failures.
+ debug:
+ required: false
+ type: boolean
+ default: false
+
+jobs:
+ docs:
+ name: Build docs
+ runs-on: ubuntu-latest
+
+ steps:
+ - name: Install dependencies
+ run: |
+ sudo apt install -y doxygen
+ python3 -m pip install \
+ sphinx==7.1.2 \
+ sphinxcontrib.plantuml \
+ recommonmark \
+ cloud_sptheme \
+ breathe
+
+ - name: Checkout code
+ uses: actions/checkout@v3
+ with:
+ ref: ${{ inputs.ref }}
+
+ - name: Generate docs
+ run: |
+ mkdir -p gh-pages
+ mkdir -p build
+
+ # Doxygen must run before Sphinx. Sphinx will refer to
+ # Doxygen-generated output when it builds its own docs.
+ doxygen docs/Doxyfile
+
+ # Now build the Sphinx-based docs.
+ make -C docs/ html
+
+ # Now move the generated outputs.
+ cp -a build/sphinx/html gh-pages/html
+ cp -a build/doxygen/html gh-pages/docs
+ cp docs/index.html gh-pages/index.html
+
+ # Now set permissions on the generated docs.
+ # https:/actions/upload-pages-artifact#file-permissions
+ chmod -R +rX gh-pages/
+
+ - name: Upload docs artifacts
+ uses: actions/upload-pages-artifact@v2
+ with:
+ path: gh-pages
+
+ - name: Debug
+ uses: mxschmitt/[email protected]
+ with:
+ limit-access-to-actor: true
+ if: failure() && inputs.debug

.github/workflows/build-matrix.json

@@ -0,0 +1,38 @@
+{
+ "comment1": "runners hosted by GitHub, always enabled",
+ "hosted": [
+ {
+ "os": "ubuntu-latest",
+ "os_name": "linux",
+ "target_arch": "x64",
+ "exe_ext": "",
+ "generator": "Ninja"
+ },
+ {
+ "os": "macos-latest",
+ "os_name": "osx",
+ "target_arch": "x64",
+ "exe_ext": "",
+ "generator": "Ninja"
+ },
+ {
+ "os": "windows-latest",
+ "os_name": "win",
+ "target_arch": "x64",
+ "exe_ext": ".exe",
+ "generator": ""
+ }
+ ],
+
+ "comment2": "runners hosted by the owner, enabled by the 'self_hosted' environment being created on the repo",
+ "selfHosted": [
+ {
+ "os": "self-hosted-linux-arm64",
+ "os_name": "linux",
+ "target_arch": "arm64",
+ "exe_ext": "",
+ "generator": "Ninja",
+ "low_mem": "yes"
+ }
+ ]
+}