Switch to CMake and git submodules

[joeyparrish]

The use of our ancient chromium build infrastructure has been increasingly problematic. We use old versions of chromium components so that they are compatible with gyp (gyp has been removed upstream), but this means also an older version of depot_tools, and lack of support for newer OSes, SDKs, and Linux distros.

We should replace gyp with CMake, gclient with submodules, and depot_tools should go away completely. Dependencies (like protobuf, etc) should be updated to the latest versions, and if possible, system versions should be used instead of building from source.

I'm not sure what order these should be done in, or how much work they will be. I'll experiment with it a bit to figure that out, but I may or may not be the one to finish the work. Volunteers are welcome!

[joeyparrish]

Oh, and of course, some of the old Python build scripts in our old forks of Chromium repos will only run with Python 2, which is not even installed by default any more in many places.

[joeyparrish]

In a pure python3-environment, such as my workstation at the office, I can't even run gyp any more. So replacing gyp is a high priority in this list.

Replacing gclient with submodules is relatively easy and straightforward. See joeyparrish@f4d560e Some of those deps could be dropped by the time this work is complete (such as gyp, packager/build).

[zdanek]

@joeyparrish I managed to fix some problems with generating CMakeLists.txt from gyp.
It was discussed some time ago at #774
I wished to make a PR to fix issues with generation of CMakeLists.txt but I failed when discovered that some tools are part of Chromium ecosystem and is very hard to fix because those tools were from ancient revision of Chromium tools.

But still - I can tell how to fix locally some files in order to generate CMakeLists.txt.

And by the way - I use Python3 with Shaka gyp and it works (AFAIK I had to fix only about two places regarding py2->py3)

[joeyparrish]

@zdanek, thank you! You shouldn't need to make permanent changes to gyp. You could make temporary changes, generate CMakeLists.txt from gyp, then clean up CMakeLists.txt by hand and commit it. The goal is to get rid of gyp, so it should be fine to hack it up as needed if it is easier than writing CMakeLists.txt completely from scratch.

[zdanek]

@joeyparrish I got it.

I was using generated CMakeLists.txt for CLion to edit files and have includes working properly. It was OK but I never compiled Shaka with make. I always used command line and ninja like everybody. Now I tried to builld it and it failed.

There are a few problems:

• it's very very messy
• it doesn't compile
• release is quite different than debug - but we need both (merged into one CMakeLists.txt)
• it's flat - I mean, that every directory / library / third party lib has it's own section put into a single CMakeLists.txt - it should be divided into smaller files, probably for directories / modules

Biggest problem is that it does not compile. That means, some paths are messed up and some includes are bad defined so inside the source files includes doesn't work. If there's no other volunteer I might spend some evenings trying to clean it up and/or rethink better approach. Maybe it will need more people to help to split it into sub-works to speed it up and not to be so boring and arduous task :)

I will try to dig into it tomorrow. First task is to build Shaka using cmake.

[joeyparrish]

I see. Thank you for the update!

It seems there isn't a better solution right now than manually creating the build files for cmake. I agree that doing it one library/dependency at a time is probably best.

[joeyparrish]

I have begun the process of writing CMake files by hand for each library.

[joeyparrish]

As part of this work, C++14 C++17 will now be required to build Shaka Packager. This will reduce our dependencies, and we will rely more on the standard library.

[zdanek]

@joeyparrish regarding to write a new CMake files by somebody from the community - I was thinking about it as I wrote 3 months ago but as you currently see it's quite absorbing and in my opinion must be done by somebody who is backed by bigger company that pays for the time spent. I wouldn't have enough time to do it. The time that I need to spend on my current job.
The transition is hard. I'm glad that you decided to move it forward. Thank you on behalf of all developers depending on Shaka Project :)

[joeyparrish]

Well, I only get to spend one day a week on it, so it will take a long time. Thankfully, @kqyang has offered to do code reviews as I make progress.

I'll start merging partial build system work to a branch in stages. If anyone is interested, they can follow along here. I'll tag this issue with each PR, and you can play with the branch I'm creating and provide feedback.

So far, I've got 4 internal libraries building, and I've ripped out Chromium build tools, Chromium base, and gflags. I've updated curl, boringssl, gmock, and gtest. I've added abseil and glog.

Those 4 internal libraries build, including their tests, but I'm stuck on crashing tests in the file library.

[zdanek]

Although local problems (crashing tests) this is good news. Getting rid of Chromium build tools is my dream since 1.5 years :)
Great decisions. I will take a look at your branch. Thank you.

[joeyparrish]

Everything we have so far in PR #1072 is building and passing all tests. If anyone wants to help write CMakeLists for additional libraries once that PR lands, I would be very happy to have the help! I think we could easily divide up the work by folder. Please let me know if you are interested!

[Romantic-LiXuefeng]

Hello everyone, Thanks for the update. I want to know what's the progress now !

[joeyparrish]

We're getting very close to merging cmake back into main. Just a few more PRs, we think.

[kevleyski]

That's great! Well done and thanks all

[joeyparrish]

The cmake branch fully builds now! Once we clean up documentation, logging, and some automated workflows, we can merge the cmake branch into main.

[gghua858]

When will the cmake branch be available? Thanks!

[joeyparrish]

You can check out the cmake branch now. The documentation is still out of date, but you can initialize the submodules (recursively) and build with CMake. @cosmin and I are working on finishing everything and merging back into main.

[joeyparrish]

Documentation in the cmake branch has been updated, and we are working on static linking, release workflows, and integration tests.

[kevleyski]

Great effort thanks!

[M4urici0GM]

I was able to build shaka-packager with cmake, but i'm getting a weird behavior:
running shaka with

packager \
in=in=d16ead10625946ff81892d5980485f83.mp4,stream=audio,lang=en,roles=main,init_segment=audio/init.mp4,segment_template=audio/audio_'$Number$'.mp4 \
in=in=d16ead10625946ff81892d5980485f83.mp4,stream=video,init_segment=video/init.mp4,segment_template=video/video_'$Number$'.mp4 \
--generate_static_live_mpd \
--mpd_output manifest.mpd \

doesnt seem to generate the mpd file containing the segments:

<?xml version="1.0" encoding="UTF-8"?>
<!--Generated with https:/shaka-project/shaka-packager version 2038339926-release-->
<MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" profiles="urn:mpeg:dash:profile:isoff-live:2011" minBufferTime="PT2S" type="static" mediaPresentationDuration="PT2806.48S">
  <Period id="0">
    <AdaptationSet id="0" contentType="video" width="2560" height="1440" frameRate="90000/3600" segmentAlignment="true" par="16:9">
      <SupplementalProperty schemeIdUri="urn:mpeg:mpegB:cicp:TransferCharacteristics" value="1"/>
      <Representation id="0" bandwidth="4785100" codecs="hvc1.1.2.L150" mimeType="video/mp4" sar="1:1"/>
    </AdaptationSet>
    <AdaptationSet id="1" contentType="audio" lang="en" segmentAlignment="true">
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
      <Representation id="1" bandwidth="167855" codecs="mp4a.40.2" mimeType="audio/mp4" audioSamplingRate="48000">
        <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/>
      </Representation>
    </AdaptationSet>
  </Period>
</MPD>

may you guys check if it is reproducible on your side? Thanks