Promote pinning sub-dependencies in constraints.txt

[m-aciek]

Promote separating top- and sub-dependencies over generating requirements from pip freeze

Using pip freeze to generate requirements may be harmful as one may have other packages installed before.
Separating pinned versions of top-level dependencies from sub-dependencies should help in maintaining and keeping control over dependencies.

[pradyunsg]

Thanks for the PR @m-aciek! I'm on board for this change.

Let's wait for a bit, to see if other folks have any thoughts or concerns. :)

[uranusjr]

I am not against the change, but also not sure what exactly the benefits are. It would probably be a good idea to add some discussion on exactly what this would help with what situations. Probably not directly with the freeze instructions (that would confuse newcomers), but a new section further down the page discussing the differences between constraint and requirement files, with a reference pointing to it added to the freeze section.

[pfmoore]

+1 on what @uranusjr said. I don't really have a clear understanding of what a constraint file is in general (specifically, how it differs from a requirements file) so I think before we promote constraint files as standard workflow, we should improve the docs on constraint files in general, to avoid confusing users.

[chrahunt]

I don't like that this puts the burden on users to distinguish between top-level dependencies and sub-dependencies.

This will probably be something that we have to resolve as part of the resolver work, so I would suggest waiting until then to see if a more reasonable workflow can be built on top of the machinery used for that.

[pfmoore]

That's a very good point. I'm not sure the new resolver will make much difference here, though - it sounds much more likely that the discussions around a standard lockfile format are going to be the key to a good user experience here.

I'm still +1 on documenting the difference in purpose between constraint and requirement files better, regardless. And I'm +0 on subsequently adding something like this change to document what works now, if you want to spend the effort manually making the distinction between direct requirements and dependencies.

[chrahunt]

If we have anything like a "prune" capability, it will require tracking installed top-level dependencies in an environment. That would make the task of outputting something like a Pipfile or requirements + constraints actually possible.

[chrahunt]

Hi @m-aciek. As mentioned above, as it is currently documented this isn't likely to delight users, who will have to make up for pip not being able to differentiate between the top-level and sub-dependencies. Let's follow #7811 and #6925, as they will provide some basic machinery that we can build on top of to provide a better user workflow.

Any issues with closing this, in favor of following those issues and seeing where we are afterwards?

[m-aciek]

Thank you all for your comments. My original message was missing some motivation background, which I'll express now below. And I will close the PR for now.

Critique of generating requirements.txt from pip freeze

My main concern expressed in this documentation change was to not promote putting pip freeze as requirements, because

• this way one is losing information about distinction between top- and sub-dependencies,
• one can include in a requirements.txt package he didn’t intend to, e.g. having polluted environment with some other packages.

In bigger projects with multiple top-dependencies, generating requirements.txt this way may produce set which is very hard to maintain (project I worked on in the past at work experienced this, and for me working with it's requirements.txt was a pain).

I would suggest in documentation creating such file by hand, with help of sections Version and Depends of pip show [package].

Using constraints

Constraints may be better than requirements in case of upgrading some dependencies. Let assume django will drop sqlparse as dependency. If someone will miss that fact while upgrading pinned version of django in requirements.txt, constraints for sqlparse will not install that unnecessary package.

But I would say constraints are an extra to the main concern.

Let me close this pull request and open another one, with a bit less “aggresive” documentation change.

[m-aciek]

For the record, I've opened #7820 with a second take on this part of documentation.