Recipes failing due to format not found

[Popolechien]

Recipes are https://farm.openzim.org/recipes/telmidtice_ar_all and
https://farm.openzim.org/recipes/telmidtice_fr_all

See https://farm.openzim.org/pipeline/df41745d-b5bc-4869-b035-169f74a4129a/debug and
https://farm.openzim.org/pipeline/7326e3cd-b24b-427e-bb73-23d20321d8c2/debug

[benoit74]

Problem is that thumbnail fails to download now for some reason. Subtitle download seems to suffer the same flaw.

[benoit74]

I think analysis (and proposed PR is wrong). Problem is maybe significantly bigger.

It looks like Youtube is not proposing webm format anymore on all servers

Subsequent calls to yt-dlp --list-formats https://www.youtube.com/watch\?v\=7_N0yozUnWY gives varying results. Sometimes there is no format with webm extension at all. Sometimes we have only audio. Sometimes we have one format with video and audio, and one with only audio.

Example 1 with no webm at all:

> yt-dlp --list-formats https://www.youtube.com/watch\?v\=7_N0yozUnWY
[youtube] Extracting URL: https://www.youtube.com/watch?v=7_N0yozUnWY
[youtube] 7_N0yozUnWY: Downloading webpage
[youtube] 7_N0yozUnWY: Downloading ios player API JSON
[youtube] 7_N0yozUnWY: Downloading web creator player API JSON
[youtube] 7_N0yozUnWY: Downloading m3u8 information
[info] Available formats for 7_N0yozUnWY:
ID  EXT   RESOLUTION FPS CH │   FILESIZE   TBR PROTO │ VCODEC          VBR ACODEC      ABR ASR MORE INFO
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
sb2 mhtml 48x27        0    │                  mhtml │ images                                  storyboard
sb1 mhtml 80x45        0    │                  mhtml │ images                                  storyboard
sb0 mhtml 160x90       0    │                  mhtml │ images                                  storyboard
233 mp4   audio only        │                  m3u8  │ audio only          unknown             [fr] Default
234 mp4   audio only        │                  m3u8  │ audio only          unknown             [fr] Default
139 m4a   audio only      2 │    4.63MiB   49k https │ audio only          mp4a.40.5   49k 22k [fr] low, m4a_dash
140 m4a   audio only      2 │   12.28MiB  129k https │ audio only          mp4a.40.2  129k 44k [fr] medium, m4a_dash
269 mp4   256x144     25    │ ~ 11.27MiB  119k m3u8  │ avc1.4D400C    119k video only
160 mp4   256x144     25    │    3.56MiB   38k https │ avc1.4D400C     38k video only          144p, mp4_dash
230 mp4   640x360     25    │ ~ 38.99MiB  411k m3u8  │ avc1.4D401E    411k video only
134 mp4   640x360     25    │   14.17MiB  150k https │ avc1.4D401E    150k video only          360p, mp4_dash
605 mp4   640x360     25    │ ~ 28.83MiB  304k m3u8  │ vp09.00.21.08  304k video only
232 mp4   1280x720    25    │ ~109.26MiB 1153k m3u8  │ avc1.64001F   1153k video only
136 mp4   1280x720    25    │   39.45MiB  416k https │ avc1.64001F    416k video only          720p, mp4_dash
270 mp4   1920x1080   25    │ ~172.02MiB 1815k m3u8  │ avc1.640028   1815k video only
137 mp4   1920x1080   25    │   69.28MiB  731k https │ avc1.640028    731k video only          1080p, mp4_dash

Exemple 2 with only audio webm:

yt-dlp --list-formats https://www.youtube.com/watch\?v\=7_N0yozUnWY
[youtube] Extracting URL: https://www.youtube.com/watch?v=7_N0yozUnWY
[youtube] 7_N0yozUnWY: Downloading webpage
[youtube] 7_N0yozUnWY: Downloading ios player API JSON
[youtube] 7_N0yozUnWY: Downloading web creator player API JSON
[youtube] 7_N0yozUnWY: Downloading m3u8 information
[info] Available formats for 7_N0yozUnWY:
ID  EXT   RESOLUTION FPS CH │   FILESIZE   TBR PROTO │ VCODEC          VBR ACODEC      ABR ASR MORE INFO
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
sb2 mhtml 48x27        0    │                  mhtml │ images                                  storyboard
sb1 mhtml 80x45        0    │                  mhtml │ images                                  storyboard
sb0 mhtml 160x90       0    │                  mhtml │ images                                  storyboard
233 mp4   audio only        │                  m3u8  │ audio only          unknown             [fr] Default
234 mp4   audio only        │                  m3u8  │ audio only          unknown             [fr] Default
139 m4a   audio only      2 │    4.63MiB   49k https │ audio only          mp4a.40.5   49k 22k [fr] low, m4a_dash
140 m4a   audio only      2 │   12.28MiB  129k https │ audio only          mp4a.40.2  129k 44k [fr] medium, m4a_dash
251 webm  audio only      2 │    9.84MiB  104k https │ audio only          opus       104k 48k [fr] medium, webm_dash
269 mp4   256x144     25    │ ~ 11.27MiB  119k m3u8  │ avc1.4D400C    119k video only
160 mp4   256x144     25    │    3.56MiB   38k https │ avc1.4D400C     38k video only          144p, mp4_dash
230 mp4   640x360     25    │ ~ 38.99MiB  411k m3u8  │ avc1.4D401E    411k video only
134 mp4   640x360     25    │   14.17MiB  150k https │ avc1.4D401E    150k video only          360p, mp4_dash
605 mp4   640x360     25    │ ~ 28.83MiB  304k m3u8  │ vp09.00.21.08  304k video only
232 mp4   1280x720    25    │ ~109.26MiB 1153k m3u8  │ avc1.64001F   1153k video only
136 mp4   1280x720    25    │   39.45MiB  416k https │ avc1.64001F    416k video only          720p, mp4_dash
270 mp4   1920x1080   25    │ ~172.02MiB 1815k m3u8  │ avc1.640028   1815k video only
137 mp4   1920x1080   25    │   69.28MiB  731k https │ avc1.640028    731k video only          1080p, mp4_dash

Exemple 3 with only 1 webm audio and 1 webm video:

yt_dlp yt-dlp --list-formats https://www.youtube.com/watch\?v\=7_N0yozUnWY
[youtube] Extracting URL: https://www.youtube.com/watch?v=7_N0yozUnWY
[youtube] 7_N0yozUnWY: Downloading webpage
[youtube] 7_N0yozUnWY: Downloading ios player API JSON
[youtube] 7_N0yozUnWY: Downloading web creator player API JSON
[youtube] 7_N0yozUnWY: Downloading m3u8 information
[info] Available formats for 7_N0yozUnWY:
ID  EXT   RESOLUTION FPS CH │   FILESIZE   TBR PROTO │ VCODEC          VBR ACODEC      ABR ASR MORE INFO
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
sb2 mhtml 48x27        0    │                  mhtml │ images                                  storyboard
sb1 mhtml 80x45        0    │                  mhtml │ images                                  storyboard
sb0 mhtml 160x90       0    │                  mhtml │ images                                  storyboard
233 mp4   audio only        │                  m3u8  │ audio only          unknown             [fr] Default
234 mp4   audio only        │                  m3u8  │ audio only          unknown             [fr] Default
139 m4a   audio only      2 │    4.63MiB   49k https │ audio only          mp4a.40.5   49k 22k [fr] low, m4a_dash
140 m4a   audio only      2 │   12.28MiB  129k https │ audio only          mp4a.40.2  129k 44k [fr] medium, m4a_dash
251 webm  audio only      2 │    9.84MiB  104k https │ audio only          opus       104k 48k [fr] medium, webm_dash
269 mp4   256x144     25    │ ~ 11.27MiB  119k m3u8  │ avc1.4D400C    119k video only
160 mp4   256x144     25    │    3.56MiB   38k https │ avc1.4D400C     38k video only          144p, mp4_dash
230 mp4   640x360     25    │ ~ 38.99MiB  411k m3u8  │ avc1.4D401E    411k video only
134 mp4   640x360     25    │   14.17MiB  150k https │ avc1.4D401E    150k video only          360p, mp4_dash
18  mp4   640x360     25  2 │ ≈ 26.38MiB  278k https │ avc1.42001E         mp4a.40.2       44k [fr] 360p
605 mp4   640x360     25    │ ~ 28.83MiB  304k m3u8  │ vp09.00.21.08  304k video only
243 webm  640x360     25    │    8.90MiB   94k https │ vp9             94k video only          360p, webm_dash
232 mp4   1280x720    25    │ ~109.26MiB 1153k m3u8  │ avc1.64001F   1153k video only
136 mp4   1280x720    25    │   39.45MiB  416k https │ avc1.64001F    416k video only          720p, mp4_dash
270 mp4   1920x1080   25    │ ~172.02MiB 1815k m3u8  │ avc1.640028   1815k video only
137 mp4   1920x1080   25    │   69.28MiB  731k https │ avc1.640028    731k video only          1080p, mp4_dash

However, with use format: "best[ext=webm]/bestvideo*[ext=webm]+bestaudio*[ext=webm]/best" which should fallback to best and hence always find a solution. I don't get it. I've opened upstream issue: yt-dlp/yt-dlp#11189

[benoit74]

And I confirm that indeed problem is not only thumbnails and subtitles, it also hit video download, #353 is definitely a duplicate of this issue (when disable S3 cache for video 7_N0yozUnWY, it sometimes fails to download the video from Youtube.

[benoit74]

See yt-dlp/yt-dlp#11189 (comment) for upstream explanation. We have to fix.

I propose to change format setting (see yt-dlp documentation) to bestvideo*[ext={vidext}]+bestaudio[ext={audext}]/bestvideo*+bestaudio/best where {vidext} and {audext} are the computed video / audio extensions:

• I drop the best[ext={vidext}] which was the preferred solution in the past because now yt_dlp is capable to paralelize download of two streams, so it is more favorable to find proper video and audio format, even if they are in two different streams ; and most platforms are now offering separate audio and video streams, mostly no more are providing bundled streams according to upstream answer of "experts"
• I add * to bestvideo to allow download of the stream even if it matches only the video constraint and has audio embedded which does not match, audio is very small so it does not change much download size
• I do not add * to bestaudio because it would make the stream significantly bigger and platforms are regularly offering audio-only streams, so it is not a big constraint at all
• I still leave best as last resort, just like default yt-dlp (in fact bestvideo*+bestaudio/best is new yt_dlp default)

[benoit74]

But please note another important point in upstream comment:

The web_creator client yt-dlp currently still uses as a default, is currently getting a change rolled out that requires sign in.

Which means Youtube is continuing to close doors, and we are getting closer and closer to the point where yt_dlp will not be able to work on Youtube anymore.