check-manifest fails when setup.py uses setuptools_scm for versioning

[philipsd6]

check-manifest copies the source files to a temporary directory, and then executes python2 setup.py sdist -d ... from there, but since that is not the git working directory, setuptools_scm can't find the git tag for versioning, resulting in:

$ check-manifest -v
listing source files under version control: 10 files and directories
building an sdist: hello-0.1.2.dev1+ng5ebdc38.tar.gz: 10 files and directories
copying source files to a temporary directory
building a clean sdist
['/home/philipsd6/devel/test_proj/venv/bin/python2', 'setup.py', 'sdist', '-d', '/tmp/philipsd6/check-manifest-DpRbmQ-sdist'] failed (status 1):
Traceback (most recent call last):
  File "setup.py", line 49, in <module>
    'Programming Language :: Python :: 3',
  File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
    _setup_distribution = dist = klass(attrs)
  File "/home/philipsd6/devel/test_proj/venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 272, in __init__
    _Distribution.__init__(self,attrs)
  File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
    self.finalize_options()
  File "/home/philipsd6/devel/test_proj/venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 327, in finalize_options
    ep.load()(self, ep.name, value)
  File "/home/philipsd6/devel/test_proj/venv/local/lib/python2.7/site-packages/setuptools_scm/integration.py", line 19, in version_keyword
    dist.metadata.version = get_version(**value)
  File "/home/philipsd6/devel/test_proj/venv/local/lib/python2.7/site-packages/setuptools_scm/__init__.py", line 104, in get_version
    version = _do_parse(root, parse)
  File "/home/philipsd6/devel/test_proj/venv/local/lib/python2.7/site-packages/setuptools_scm/__init__.py", line 82, in _do_parse
    "use git+https:/user/proj.git#egg=proj" % root)
LookupError: setuptools-scm was unable to detect version for '/tmp/philipsd6/check-manifest-CP9IEQ-sources'.

Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

For example, if you're using pip, instead of https:/user/proj/archive/master.zip use git+https:/user/proj.git#egg=proj

How can we use both setuptools_scm for versioning from our git tags, but also validate our MANIFEST.in with check-manifest?

[mgedmin]

I've no idea!

check-manifest tries to prevent problems in various scenarios

• you have the setuptools-git plugin installed and make releases that rely on git metadata; then you reinstall your machine or something and make a new release while forgetting to install setuptools-git -> you silently get an incomplete release
• you make releases from a subversion checkout while relying on setuptools's builtin svn support, but then a new svn version comes up, you run svn upgrade, and setuptools no longer understands the working tree metadata format -> you silently get an incomplete release
• you make releases and upload tarballs to PyPI, then someone wants to make a local patch so they download the tarball, unpack it, patch it, add a local part to the version number in setup.py and build a new tarball with python setup.py sdist -> the new tarball is incomplete since there's no git metadata

Now considering your use case:

• you use setuptools_scm, which fails hard when git metadata is not present -- so failure case 1 from the above seems to be impossible
• svn is completely irrelevant, which takes care of failure case 2
• you don't want to support people running 'setup.py sdist' from an extracted sdist (or you wouldn't be using setuptools_scm), which makes failure case 3 irrelevant

Why do you even need check-manifest? Is there any situation under which a package relying on setuptools_scm might produce an incomplete sdist?

[mgedmin]

One scenario just occurred to me:

• you make releases for various packages (some use setuptool-scm and some don't)
• you use a tool like zest.releaser that automatically runs check-manifest for you
• you want to continue using the same release workflow for all the packages

In this situation you (I'm using the general 'you' here, not you in specifically) don't need check-manifest to catch errors, but you'd like to prevent it aborting the process with false positives. It may be helpful to define a setup.cfg option to allow check-manifest to skip these projects, or perform a weaker test (just compare git metadata with sdist, in effect verifying that setuptools_scm works like it's supposed to -- and here I'm making assumptions about what setuptools_scm does, since I've nevert actually used it).

Externally the user interface would be check-manifest --weak, or, more likely, a setup.cfg with

[check-manifest]
weak = 1

This would skip the "copy to empty temporary directory without git metadata" part, in effect reverting to the behavior of check-manifest before bug #1 was fixed (with the corresponding disadvantages). Or -- thinking out loud here -- maybe I want to copy the VCS metadata too?

[philipsd6]

Well, I'm using setuptools_scm for the first time myself, and the situation I find myself in is, I have files checked into my git repo that should not be part of my distributions. Files like my .bumpversion.cfg, and .travis-ci, but it appears that setuptools_scm is working as advertised and pulling everything in.

I'm used to using a MANIFEST.in file to add things that should be included:

    include *.md
    recursive-include tests *.py

But I guess I should switch to explicitly excluding files that are part of the repo but shouldn't be part of the distribution. I'm not actually sure what role check-manifest could play in this scenario, if any.

[philipsd6]

I'll leave it up to you @mgedmin, but I'd say this can be closed, except maybe you'd like to make a note that check-manifest isn't useful when using setuptools_scm on the README. After all, the worst that can happen with setuptools_scm is having a dirty distribution with unnecessary files from the (presumably public) repo.

[mgedmin]

After all, the worst that can happen with setuptools_scm is having a dirty distribution with unnecessary files from the (presumably public) repo.

Ooh, that's a good reason to allow check-manifest to work with this kind of package!

I'll see what I can do (if I can manage to find the time).

[jbarlow83]

I suggest the following fix:

• In the user's source directory, try to import setuptools_scm and call get_version(). If this succeeds, save the version it reported
• When creating the clean sdist (run([python, 'setup.py', 'sdist', '-d', tempdir])) call this with the environment variable SETUPTOOLS_SCM_PRETEND_VERSION set to the previously reported version. If this variable is defined setuptools always answers its value and exits.

I believe this is correct behavior because copying the sources to a directory with no version control and no PKG-INFO is a synthetic situation that check-manifest creates for testing, so it makes sense to shim setuptools_scm so that it will report the same answer that the user's working directory does.

[mgedmin]

Ironically now I have a usecase for check-manifest on a package that uses setuptools_scm myself: I'm building an sdist and then I'm building wheels from that sdist in a Docker container. The setup.py here reads install_requires from requirements/prod.in, which was omitted from my MANIFEST.in. Oops.

[mgedmin]

Workaround: SETUPTOOLS_SCM_PRETEND_VERSION=0.0 check-manifest

[RonnyPfannschmidt]

i just wanted to open a issue about it myself,

mainly because i need it for setuptools_scm itself

it should be noted that setuptools_scm automatically removes the need for a MANIFEST.in

[mgedmin]

Does the workaround I suggested apply to your use case?

Should check-manifest set the SETUPTOOLS_SCM_PRETEND_VERSION environment variable automatically?

[RonnyPfannschmidt]

the suggested workaround does work, it may help to set it for the check, but it should print some notification that it had to

it shoudl also be noted that setuptools_scm adds file finders and thus makes a manifest file typically irrelevant, however it cant self-consume due to bootstrap issues on travis´

[mgedmin]

I've released check-manifest 0.37 with this fix.

[RonnyPfannschmidt]

fabulosu solution thanks