Added cross-environment usage documentation to README.rst #651
Conversation
|
@tuukkamustonen @barrywhart @cancan101 @taion @JoergRittinger |
|
Yeah, giving Python packaging limitations this does make sense. This is incompatible with checking in |
|
@taion Good point about source control. That depends on the user's case I guess. If they deploy an app/service through CI/CD on a single defined Python environment (ex: Only Py3.6 on linux), then committing I'll figure a way to point that out w/o having to write a dissertation on CI/CD. |
|
It's tricky, though. For example, we mostly develop on OS X, but our CI runs in a Linux Docker container. |
|
@taion Argh, yeah I see. If you don't mind me asking: what are you currently doing to deal with that? |
|
Nothing – just running pip compile on OS X gives good-enough results. Not perfect, but even when there are mismatches, it’s good enough. This may well be due to specific pecularities with our setup, though (we now get |
|
(Pushed but I still need to take care of the source control recommendations) |
I'm not sure you should cristallise in doc not commiting requirements.txt to source control. Not doing so opens the door to non deterministic builds (because a compile phase is needed to re-create). If you want to talk about this, I would instead mention that the files on different platforms should have different annotation (a platform suffix) corresponding to the source environment. That way you keep deterministic behavior for the different platforms. |
|
To be more concrete, now that I'm at a laptop, in the one repo where this even comes up for us, we do: On Linux we use a Docker image with TensorFlow pre-installed. It so happens to work, but it's obviously far from perfect. In practice, though, with the frequency with which people develop on OS X but deploy on Linux, it's not clear to me that the recommendation here taken literally would improve DX. Really this is the fault of Python packaging; it's not clear there's much that's possible here |
|
Looks good to me. |
|
@davidovich @tuukkamustonen @taion Updated, with a distinct section regarding commiting. |
|
@vphilippon This was a handy read. Are you game to rebase so that this can be merged? If not, should somebody else open a new PR? I'd like to confirm that the recommendation is to generate and commit a requirements file for each environment. For example, I'm currently in the process of migrating a Django application from Python 2 to Python 3. We have base and dev requirements. Based on this PR, here's a tox configuration that I came up with: [tox]
envlist = {2.7,3.7}
skipsdist = true
[testenv]
basepython =
2.7: python2.7
3.7: python3.7
envdir =
{toxworkdir}/requirements-{envname}
deps =
pip-tools
commands =
pip-compile -o requirements-{envname}.txt \
requirements.in
pip-compile -o requirements-{envname}-dev.txt \
requirements-{envname}.txt requirements-dev.inResulting in these files: If I understand correctly, I should commit all of those files? |
|
For anyone interested, I put together an example repo of how to manage cross-environment requirements files using Tox or Nox: https:/bhrutledge/requirements-py2-py3 Feedback welcome. |
atugushev
added
docs
atugushev
force-pushed
the
document-cross-environment-usage
branch
from
24fd03a
to
2b6892f
Compare
Codecov Report
@@ Coverage Diff @@
## master #651 +/- ##
=======================================
Coverage 99.42% 99.42%
=======================================
Files 34 34
Lines 2443 2443
Branches 302 302
=======================================
Hits 2429 2429
Misses 8 8
Partials 6 6 Continue to review full report at Codecov.
|
|
Thanks to @vphilippon and all reviewers! |
... parametrized by python version. We have a bunch of dependencies that are only needed on specific python versions (e.g. backports like dataclasses, importlib_resources, etc). We use conditional environment markers for those in the top-level requirements.in files. pip-compile does not yet support generating a combined requirements.txt file where differences across python versions/platforms/archs are reconciled using environmet markers: jazzband/pip-tools#826 They currently recommend running pip-compile on each targeted python environment generating as many concrete requirements.txt files jazzband/pip-tools#651 So this is what I have done here.
... containing all top-level (*.in) and concrete requirements.txt, the latter parametrized by python version. We have a bunch of dependencies that are only needed on specific python versions (e.g. backports like dataclasses, importlib_resources, etc). We use conditional environment markers for those in the top-level requirements.in files. pip-compile does not yet support generating a combined requirements.txt file where differences across python versions/platforms/archs are reconciled using environmet markers: jazzband/pip-tools#826 They currently recommend running pip-compile on each targeted python environment generating as many concrete requirements.txt files jazzband/pip-tools#651 So this is what I have done here.
... containing all top-level (*.in) and concrete requirements.txt, the latter parametrized by python version. We have a bunch of dependencies that are only needed on specific python versions (e.g. backports like dataclasses, importlib_resources, etc). We use conditional environment markers for those in the top-level requirements.in files. pip-compile does not yet support generating a combined requirements.txt file where differences across python versions/platforms/archs are reconciled using environmet markers: jazzband/pip-tools#826 They currently recommend running pip-compile on each targeted python environment generating as many concrete requirements.txt files jazzband/pip-tools#651 So this is what I have done here.
Added some documentation regarding cross-environment usage of
requirements.txt.In short: Run
pip-compileon each targeted environment.Resolves #585
Resolves #563
Changelog-friendly one-liner: Add cross-environment usage documentation to
README.Contributor checklist
Provided the tests for the changes.