Shallow clone breaks build

[stephenfin]

Details

• Read the Docs project URL: https://readthedocs.org/projects/patchwork/
• Build URL (if applicable): https://readthedocs.org/projects/patchwork/builds/8314815/
• Read the Docs username (if applicable):

Expected Result

The build, which utilizes the reno.sphinxext module, should work.

Actual Result

The build fails due to missing tags. These are presumably missing due to the shallow cloning of repos introduced in #4939.

Additional Information

The original issue that #4939 purported to resolve was #1888, Optional use of shallow clone for git repos (emphasis mine). While shallow clone might be useful at large, it should be possible to disable it for projects that rely on git history to build their docs.

[stsewd]

I'm guessing this is because the reno package needs the git history https://docs.openstack.org/reno/.

[stephenfin]

I'm guessing this is because the reno package needs the git history docs.openstack.org/reno.

Indeed 😞

[humitos]

@stsewd I think we should add a Feature flag for these cases to omit shallowing.

[stsewd]

We have to options here, feature flag (this can only be added by the core team), or an option in the web interface (this can be handled by the user).

[stephenfin]

Yeah, I've been looking at this. I see both approaches were adopted for submodules, with bd36494 providing the feature flag approach and the later bfa6bdb providing a config file based approach. I'd been adding a git.shallow_clone setting using the latter approach based on the idea that pretty much everyone using reno or similar tools would be affected. I'm not sure how the config file ties into the web interface though and I don't see submodule configuration there. Is this a third option?

[stsewd]

We can't have this option in the config file because the config file is parsed after the clone step. And we don't have the submodules option on the web because we are moving all per-version settings to the config only, all per-project settings will be available on the web.

[stephenfin]

Oh, of course 🤦 I guess web UI is the way to go so, though I'm not sure where to nestle that in.

[humitos]

feature flag (this can only be added by the core team), or an option in the web interface (this can be handled by the user).

Feature flag is the way to go. Admin option is overkill I'd say. I don't expect to have this problem too often.

[stephenfin]

See above. I've a pull request which should resolve this using the feature flag.

[humitos]

We can't have this option in the config file because the config file is parsed after the clone step

Weird idea: I think that in the future we could do a simple/plain first clone of the repo to get the config file and make more decisions after parsing the YAML file (like running another git command to get the full history of the repo). This way will help us to create a "more fully configuration YAML".

[stsewd]

@humitos I was thinking that we can use the providers API to fetch only the yaml, I think Travis does that.

[ericholscher]

This is now deployed behind a feature flag, if anyone needs this functionality, just ask an admin.

[stephenfin]

This is now deployed behind a feature flag, if anyone needs this functionality, just ask an admin.

How does one do this? I'd like this enabled for git-pw.readthedocs.io and patchwork.readthedocs.io, and I've been told by colleagues that infrared.readthedocs.io is also broken.

[humitos]

@stephenfin done for git-pw, patchwork and infrared. Please, if you have any issue open a new one to continue the discussion there. Thanks!

I triggered a build manually in git-pw and it worked ;)

[tylerbutler]

Feature flag is the way to go. Admin option is overkill I'd say. I don't expect to have this problem too often.

Is that the final decision? I think anyone who includes Git commit info in their docs using something like sphinx-git will hit this problem. Moreover, it's a time bomb. Builds will work fine for a while, then one day, once commits you are referencing have fallen out of the shallow clone, builds will start failing, complaining that commit hashes you're referring to can't be found.

I understand prioritizing other work - but this seems like it represents a lot of ongoing maintenance work for support, and also is confusing to users IMO.

[amercader]

For anyone stumbling onto this issue, you can now add extra steps in the .readthedocs.yaml file to get an unshallow clone:

https://docs.readthedocs.io/en/latest/build-customization.html#unshallow-git-clone