Replace custom pip freeze script with pip-tools

[lfdebrux]

In the past we've avoided using out-of-the-box solutions for Python dependency resolution because a) they haven't been very mature and b) we've had lots of issues with version conflicts. See [1], [2] for details. Instead, we've been using a custom Python script that under-the-hood runs pip freeze and saves the output to requirements.txt.

This script works well for us, but it doesn't integrate well with other tools. On the other hand pip-tools as of 2020 seems to be well-supported by its maintainers and other tools; for instance, GitHub's automated update service Dependabot supports requirements.in files.

At the same time, we've been better at keeping our dependencies up-to-date and fixing version conflicts, so there is no longer any issue caused by using pip-tools to generate our requirements.txt file. I think it's time to give pip-tools another chance :)

This PR is a straw-person to explore using pip-tools instead. It replaces our script dmutils.repoutils.freeze_requirements with pip-compile, and changes our requirements-app.txt file to be requirements.in to meet pip-tools convention. There are also separate requirements-dev.in and requirements-dev.txt files for dev requirements, following the layered requirements example in the pip-tools README.

As part of this, I've also loosened/removed the constraints in the parent requirements files, so we end up with something like the spec/lockfile pattern in tools such as Bundler and NPM. This appears to be the best way of using pip-tools.

In my mind the advantages are a cleaner interface for maintaining requirements files, and better integration with tools in the future. We could also remove our own custom code! There are a couple of extra steps we could take, such as asking pip-compile to save hashes, and using pip-sync to install requirements in our local development virtualenvs.

[katstevens]

I like this proposal!

One annoying problem we had with freeze-requirements was it getting confused when trying to pull in an update to the dmutils freeze function itself. Would we run into any issues updating pip-tools?

[lfdebrux]

I like this proposal!

One annoying problem we had with freeze-requirements was it getting confused when trying to pull in an update to the dmutils freeze function itself. Would we run into any issues updating pip-tools?

That's a good question, and I'm not sure of the answer. Looking through the open issues for pip-tools I can't see anything like this.

However, I do note that the README for pip-tools says under Example usage for pip-sync that

pip-sync will not upgrade or uninstall packaging tools like setuptools, pip, or pip-tools itself. Use pip install --upgrade to upgrade those packages.

which is interesting.

That said, pip-sync does follow semantic versioning, and the requirements file format is specified by pip, so I can't see that changing in a breaking way. I suspect in practice that if we do run into problems we can always delete the existing requirements.txt and regenerate it without too much hassle.

[lfdebrux]

One thing to note, I don't know how well PyUp works with pip-tools (this open issue from 2017 seems to suggest not at all). So I think makes more sense to adopt pip-tools and Dependabot together.

Alternatively we could do what we were doing before and run pip-tools when a PyUp PR comes in (so similar to what we do before).

[risicle]

WFM. Only downside is that it doesn't keep the direct dependencies at the top of the file verbatim, which is something I liked.

[risicle]

Am I right in thinking we no longer have the test step that forces a re-freeze if the .in files are out of sync? Guess that's another downside.

[lfdebrux]

Am I right in thinking we no longer have the test step that forces a re-freeze if the .in files are out of sync? Guess that's another downside.

That's correct. We could add it using a Make rule that looks at the timestamps of requirements.txt and requirements.in.

[risicle]

Please god no not timestamps.

[risicle]

We could write the hash of the last-frozen .in files somewhere and check it matches at make test time...

[lfdebrux]

We could write the hash of the last-frozen .in files somewhere and check it matches at make test time...

Is this a feature that lots of people want? @katstevens @benvand what are your thoughts?

[katstevens]

We could write the hash of the last-frozen .in files somewhere and check it matches at make test time...

Is this a feature that lots of people want? @katstevens @benvand what are your thoughts?

Sounds good to me, but we can introduce it in a separate PR?

[risicle]

Yeah I don't think it's immediately critical.