Isolating conda environments from ~/.local

[ghost]

I think, I am seeing a bug in conda create command where I am not able to create an isolated conda environment, which doesn't inherit packages from the base environment, in Ubuntu 16.04.

The reproducible steps for the bug are as follows.

1. Create a new Ubuntu 16.04 VM or any Ubuntu 16.04 machine.

2. Install azure-cli
   pip install azure-cli

Let's assume the python version is 2.7 here, as that is the default python in Ubuntu 16.04

This command installs packages ~/.local/lib/python2.7

3. Install miniconda in Ubuntu 16.04

4. conda create -n testenv python=2.7

5. pip freeze

In pip freeze, I see the azure-cli packages from ~/.local/lib/python2.7 , which indicates to me that the testenv conda environment is not isolated, as it inherited the packages from the base environment.

I only checked this for azure-cli, but my guess is that it is true for other packages too.

For python virtual env, this is not an issue, I get an isolated python virtual environment. I also don't get this issue on windows. I only get this on Linux and Mac machines.

Please take a look at the issue and let me know if I am doing something wrong or it is a genuine bug.

Thanks

[kalefranz]

@mingwandroid What are your latest thoughts here? CPython has explicit cutouts and code paths to handle virtualenv. Maybe we should patch our cpython to basically always be in virtualenv mode? Of course, that's going to make people mad who are accustomed to the current behavior. Either way, I think there will be a portion of people unhappy. I'm not sure how to thread this needle.

[ghost]

One way I figured out to create an isolated conda environment is to 1) Create a python virtualenv, which is isolated. 2) Then clone that python virtualenv using conda create --clone option. Clone option doesn't have this problem.

[kalefranz]

Yeah the virtualenv you created is hooking into that hard-coded cpython behavior. I think the difference something related to sys.prefix vs sys.exec_prefix. Don't remember the exact details without digging into it a bit more.

[ghost]

https:/pypa/pip/blob/master/src/pip/_internal/utils/misc.py#L283 is the line where pip has special condition for python virtual envs, which is not true for conda. In case that helps.

[mingwandroid]

@adyada, can we clarify a point here please, when you said:

In pip freeze, I see the azure-cli packages from ~/.local/lib/python2.7 , which indicates to me that the testenv conda environment is not isolated, as it inherited the packages from the base environment

Do you not mean that it inherited the packages from your system Python instead?

base environment means the packages in your conda base environment, to me, at least.

[ghost]

Yeah, I mean that the testenv conda environment inherited the packages from the system python. I didn't check for the base conda environment.

[mingwandroid]

@kalefranz, these deliberate exclusions are part of pip, as @adyada mentions, but really it's this part that seems to do the avoid user_site stuff (which is what ~/.local is called in the code AFAIK).

We could patch our pip to ignore user_site, but half the pip using world would maybe prefer we didn't? Or do you both contend that any exclusion done in pip for a virtualenv env should also always apply to a conda env? What about when freezing the conda base env? Should it still look in user_site?

Then there are those who say that conda should never ever look in user_site, and I have at least one foot in that camp (maybe we should invent an env var called CONDA_ECOSYSTEM_ISOLATION for this purpose? I face the same thing in the R world FWIW).

Whatever happens it should be consistent when freezing as when running and I think that's an uncontroversial statement?

[kalefranz]

Then there are those who say that conda should never ever look in user_site

I personally have a foot there too. And I'd go so far as saying this should be the default for any conda-installed python (given that any conda-install python will by definition always be in a conda environment). The environment variable I'd propose then would be to open up the interpreter to look in user_site if people actually want that behavior.

That's my personal opinion though, and why this issue is marked as a discussion topic. If we make any type of change here, we need to make sure it's thoroughly discussed and we have consensus on the behavior change.

[ghost]

I think, including user_site packages also caused a bug for us, when some people tried installing and using some packages and got errors. That's how the bug was reported to me. I will reproduce that bug and post here by today or tomorrow.

[ghost]

The error that I get when user_site packages are also included in an conda environment is that the import of some modules fail, even if that module shows up in pip freeze.
I am just pasting the output of pip freeze in the conda env and a command that shows the import failure.

The important lines are shown as bold lines.

So, backports.tempfile shows up in pip freeze, but then its import fails.

Apologies for pasting such a long output. Actually, I didn't install most of these packages in the conda environment, these were inherited from the user_site location, that is ~/.local/python2.7

I don't see this issue when I am able to create an isolated python environment using python virtual env

So, I am also on the side that we should not be including user_site packages in a conda env. Probably, might be better to have the conda behavior same as the python virtual env behavior in terms of isolation of environments.

(testenv) testuser@testVMadyada2: $ python -m pip freeze
adal==0.5.1
applicationinsights==0.11.2
argcomplete==1.9.4
asn1crypto==0.24.0
azure-batch==4.1.1
azure-batch-extensions==1.0.1
azure-cli==2.0.31
azure-cli-acr==2.0.23
azure-cli-acs==2.0.31
azure-cli-advisor==0.5.1
azure-cli-appservice==0.1.31
azure-cli-backup==1.1.1
azure-cli-batch==3.2.0
azure-cli-batchai==0.2.0
azure-cli-billing==0.1.8
azure-cli-cdn==0.0.14
azure-cli-cloud==2.0.13
azure-cli-cognitiveservices==0.1.12
azure-cli-command-modules-nspkg==2.0.1
azure-cli-configure==2.0.15
azure-cli-consumption==0.3.0
azure-cli-container==0.1.22
azure-cli-core==2.0.31
azure-cli-cosmosdb==0.1.20
azure-cli-dla==0.0.19
azure-cli-dls==0.0.21
azure-cli-eventgrid==0.1.12
azure-cli-eventhubs==0.1.2
azure-cli-extension==0.0.12
azure-cli-feedback==2.1.1
azure-cli-find==0.2.9
azure-cli-interactive==0.3.19
azure-cli-iot==0.1.19
azure-cli-keyvault==2.0.21
azure-cli-lab==0.0.21
azure-cli-monitor==0.1.5
azure-cli-network==2.0.28
azure-cli-nspkg==3.0.2
azure-cli-profile==2.0.22
azure-cli-rdbms==0.2.1
azure-cli-redis==0.2.12
azure-cli-reservations==0.1.2
azure-cli-resource==2.0.27
azure-cli-role==2.0.22
azure-cli-servicebus==0.1.2
azure-cli-servicefabric==0.0.12
azure-cli-sql==2.0.25
azure-cli-storage==2.0.31
azure-cli-vm==2.0.30
azure-common==1.1.9
azure-datalake-store==0.0.19
azure-graphrbac==0.40.0
azure-keyvault==0.3.7
azure-mgmt-advisor==1.0.1
azure-mgmt-authorization==0.40.0
azure-mgmt-batch==4.1.0
azure-mgmt-batchai==1.0.0
azure-mgmt-billing==0.2.0
azure-mgmt-cdn==1.0.0
azure-mgmt-cognitiveservices==1.0.0
azure-mgmt-compute==4.0.0rc1
azure-mgmt-consumption==2.0.0
azure-mgmt-containerinstance==0.4.0
azure-mgmt-containerregistry==1.0.1
azure-mgmt-containerservice==3.0.1
azure-mgmt-cosmosdb==0.3.1
azure-mgmt-datalake-analytics==0.2.0
azure-mgmt-datalake-nspkg==2.0.0
azure-mgmt-datalake-store==0.2.0
azure-mgmt-devtestlabs==2.0.0
azure-mgmt-dns==2.0.0rc1
azure-mgmt-eventgrid==0.4.0
azure-mgmt-eventhub==1.2.0
azure-mgmt-iothub==0.4.0
azure-mgmt-iothubprovisioningservices==0.1.0
azure-mgmt-keyvault==0.40.0
azure-mgmt-marketplaceordering==0.1.0
azure-mgmt-monitor==0.5.0
azure-mgmt-msi==0.1.0
azure-mgmt-network==2.0.0rc2
azure-mgmt-nspkg==2.0.0
azure-mgmt-rdbms==1.1.0
azure-mgmt-recoveryservices==0.1.0
azure-mgmt-recoveryservicesbackup==0.1.1
azure-mgmt-redis==4.1.0
azure-mgmt-reservations==0.1.0
azure-mgmt-resource==1.2.1
azure-mgmt-servicebus==0.4.0
azure-mgmt-servicefabric==0.1.0
azure-mgmt-sql==0.8.6
azure-mgmt-storage==1.5.0
azure-mgmt-trafficmanager==0.40.0
azure-mgmt-web==0.35.0
azure-multiapi-storage==0.2.0
azure-nspkg==2.0.0
azure-storage==0.34.3
azureml-core==0.1.0.806005
backports.tempfile==1.0
backports.weakref==1.0.post1
bcrypt==3.1.4
certifi==2018.4.16
cffi==1.11.5
chardet==3.0.4
colorama==0.3.9
configparser==3.5.0
cryptography==2.2.2
entrypoints==0.2.3
enum34==1.1.6
funcsigs==1.0.2
futures==3.2.0
humanfriendly==4.10
idna==2.6
ipaddress==1.0.22
isodate==0.6.0
jmespath==0.9.3
jsonpickle==0.9.6
keyring==12.0.1
knack==0.3.2
mock==2.0.0
monotonic==1.4
msrest==0.4.27
msrestazure==0.4.26
ndg-httpsclient==0.4.4
oauthlib==2.0.7
paramiko==2.4.1
pathlib==1.0.1
pathlib2==2.3.0
pathspec==0.5.6
pbr==4.0.2
prompt-toolkit==1.0.15
pyasn1==0.4.2
pycparser==2.18
pydocumentdb==2.3.1
Pygments==2.2.0
PyJWT==1.6.1
PyNaCl==1.2.1
pyOpenSSL==17.5.0
python-dateutil==2.7.2
pytz==2018.4
PyYAML==3.12
requests==2.18.4
requests-oauthlib==0.8.0
ruamel.ordereddict==0.4.13
ruamel.yaml==0.15.37
scandir==1.7
scp==0.10.2
SecretStorage==2.3.1
six==1.11.0
sshtunnel==0.1.3
tabulate==0.8.2
urllib3==1.22
vsts-cd-manager==1.0.1
wcwidth==0.1.7
websocket-client==0.47.0
Whoosh==2.7.4
xmltodict==0.11.0
(testenv) testuser@testVMadyada2: $
(testenv) testuser@testVMadyada2: $
(testenv) testuser@testVMadyada2: $ python
Python 2.7.14 |Anaconda, Inc.| (default, Mar 27 2018, 17:29:31)
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.

from azureml.core.project import Project
Traceback (most recent call last):
File "", line 1, in
File "/home/testuser/miniconda3/envs/testenv/lib/python2.7/site-packages/azureml/core/project.py", line 15, in
from azureml.execution.execution_api import ExperimentExecution
File "/home/testuser/miniconda3/envs/testenv/lib/python2.7/site-packages/azureml/execution/execution_api.py", line 6, in
from azureml.execution import _commands
File "/home/testuser/miniconda3/envs/testenv/lib/python2.7/site-packages/azureml/execution/_commands.py", line 37, in
from backports import tempfile as temp_dir_back
ImportError: cannot import name tempfile

[kalefranz]

Is pip freeze what's reaching into ~/.local here then? I'm guessing conda list doesn't show the packages... because those packages aren't actually in site-packages within the environment prefix, correct?

[ghost]

yeah, conda list doesn't show those packages.

This is the output of conda list.

(testenv) testuser@testVMadyada2:~$ conda list

packages in environment at /home/testuser/miniconda3/envs/testenv:

Name Version Build Channel

azureml-core 0.1.0.806005
backports.tempfile 1.0
backports.weakref 1.0.post1
ca-certificates 2018.03.07 0
certifi 2018.1.18 py27_0
jsonpickle 0.9.6
libedit 3.1 heed3624_0
libffi 3.2.1 hd88cf55_4
libgcc-ng 7.2.0 hdf63c60_3
libstdcxx-ng 7.2.0 hdf63c60_3
ncurses 6.0 h9df7e31_2
ndg-httpsclient 0.4.4
openssl 1.0.2o h20670df_0
pathspec 0.5.6
pip 9.0.3 py27_0
python 2.7.14 h1571d57_31
readline 7.0 ha6073c6_4
ruamel.ordereddict 0.4.13
ruamel.yaml 0.15.37
setuptools 39.0.1 py27_0
sqlite 3.22.0 h1bed415_0
tk 8.6.7 hc745277_3
wheel 0.31.0 py27_0
zlib 1.2.11 ha838bed_2
(testenv) testuser@testVMadyada2:~$

[kalefranz]

I'd be in favor of by default putting the kibosh on all the .local stuff for any python in a conda environment (and, to reiterate, that's ALL conda-installed python), but I'm certain there'll be some vocal griping if we do.

[ghost]

So, I could reproduce this behavior on windows too.

I think, there could be some people who would be depending on user_site packages in a conda environment. So, a solution is to have an argument like "--no-user-site" in the conda create command, which then doesn't include user_site packages. This way conda won't break the behavior for existing users and there would "--no-user-site" argument to not include any user_site packages if someone wants it.

[mingwandroid]

I think, there could be some people who would be depending on user_site packages in a conda environment. So, a solution is to have an argument like "--no-user-site" in the conda create command, which then doesn't include user_site packages.

The problem is interfacing this conda environment parameter to Python itself which is what is in control of whether ~/.local is added to the sys.path. Using an env. var is sub-optimal for external tooling that does not use the correct conda activate procedure. I guess we could put this in a file in the root of the environment and then modify our python executable to always look for this? Changing the behaviour of the python executable isn't something to do lightly though!

[kalefranz]

How does the python executable know if it’s been called from a virtualenv? Maybe we should be flipping that switch for all conda envs too? But then I think that’s going to have other ramifications than just removing .local from sys.path. Definitely something we’d need to be extremely careful about. Actually should probably just reach out to some cpython core maintainers for their thoughts.

…

Sent from my iPhone

On Apr 21, 2018, at 5:27 AM, Ray Donnelly ***@***.***> wrote: think, there could be some people who would be depending on user_site packages in a conda environment. So, a solution is to have an argument like "--no-user-site" in the conda create command, which then doesn't include user_site packages. The problem is interfacing this conda environment parameter to Python itself which is what is in control of whether ~/.local is added to the sys.path. Using an env. var is sub-optimal for external tooling that does not use the correct conda activate procedure. I guess we could put this in a file in the root of the environment and then modify our python executable to always look for this? Changing the behaviour of the python executable isn't something to do lightly though! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[mbargull]

xref: pypa/pip#5060

[mingwandroid]

I think we would like to be able to configure the root env to also not look in ~/local, and if so in that case you just want to check a file relative to sys.prefix and virtual/base shouldn't matter.

[ghost]

It looks like python virtualenv after changing the sys.prefix, sets a sys.base_prefix attribute, which is then used to determine in pip that there is a virtualenv active.
https:/pypa/pip/blob/master/src/pip/_internal/locations.py#L39

[jakirkham]

Please don’t disable ~/.local. 😟

[takluyver]

I'd support disabling user site packages for Python in conda. I always do this in my own conda environments, and I've certainly seen it cause confusion for users, e.g. when a package they just now conda-installed is shadowed by a version they installed with pip ages ago. User site packages is valuable when using system Python, to install packages without sudo, but conda is typically installed in your user account anyway, so it's just an extra complication.

I would do this by setting ENABLE_USER_SITE = False in site.py (this is what I set in conda envs locally):

https:/python/cpython/blob/v3.6.5/Lib/site.py#L79-L81

I think that with this set, pip should refuse to do a --user install, though I haven't checked whether it currently does so.

Edit: My views on this got more complex, see my comment below.

[kalefranz]

xref: #448

[RafalSkolasinski]

Is there any current plan to deal with this? Every now and then this issue hits one by surprise.

I believe that if one creates an environment, conda or virtualenv - whichever, they expect a full isolation from other python installations. If that is not provided this lead to hard to reproduce errors:

• one may think that the environment definition is sufficient and complete
• behaviour may be different on other machines
• unpredictable behaviour

Most of my use-cases is to create environments using conda (I basically use it as easy environment manager to create / activate them) and then I do all python install inside using pip.

I really thought that I finally solved issue with conda environments being polluted by site-packages by adding

create_default_packages:
  - conda-ecosystem-user-package-isolation

to my ~/.condarc but just now for some reason on one machine it did not work as expected and lead to long debugging...

[mfansler]

@RafalSkolasinski didn't know that package existed! 🤩

I notice it doesn't do anything to insulate against use of PYTHONPATH, which can also be a source of alternative modules. It might be worth adding this to the conda-ecosystem-user-package-isolation package as another environment variable to mask (issue or PR would go to the feedstock). Alternatively, the python -E flag will ignore PYTHONPATH at invocation.

[RafalSkolasinski]

@RafalSkolasinski didn't know that package existed! star_struck

:) Yes, I don't even remember how I found it - I think it may have been recommended to me on gitter / reddit when I complained about the issue looong time ago.

I notice it doesn't do anything to insulate against use of PYTHONPATH,

Interesting, it may be what was hitting me now. Cannot say for sure - it was quite a fresh install so I went on rampage of removal / reinstall of things until it stopped happening...

[less-and-less-bugs]

In fact, I met this question when I used tmux at a Linux server. I indeed tried some methods on GitHub but it did not work. At last, I found use "conda deactivate " first to leave the base python environment provided by conda, and then went into tmux session. It was successful!

[mfansler]

I notice it doesn't do anything to insulate against use of PYTHONPATH,

Interesting, it may be what was hitting me now. Cannot say for sure - it was quite a fresh install so I went on rampage of removal / reinstall of things until it stopped happening...

@RafalSkolasinski FWIW, not knowing about the aforementioned package, I had previously created my own personal version of it that only sets the PYTHONNOUSERSITE=1. Since it's basically a template for setting an environment variable, I added a similar one for masking PYTHONPATH.

Both are available via Anaconda Cloud, and have the MatchSpec:

merv::envvar-pythonnousersite-true
merv::envvar-pythonpath-null

The latter could be a possible alternative if the conda-ecosystem-user-package-isolation doesn't want to include PYTHONPATH masking.

[kousu]

@less-and-less-bugs wrote:

3. vim path of site.py
   Then set ENABLE_USER_SITE False. Then all the packages in ./local would not be found by used environment now

This is exactly what we used to do:

https:/spinalcordtoolbox/spinalcordtoolbox/blob/e6eb4299fa7d9e4a2fb8f6362c26908847a49ec8/install_sct#L608

But then I figured out (spinalcordtoolbox/spinalcordtoolbox#3606) that this was a little more reliable:

https:/spinalcordtoolbox/spinalcordtoolbox/blob/7198415f186385c9d3d2e4246d58da4b0bf7e262/install_sct#L605

So maybe that'll help you.

I still am surprised every time that conda doesn't do this by default, since it's main feature (at least, as seen by the zeitgeist) is to provide reproducibility.

[niander]

This creates issues when using tox-conda plugin since it will use conda to create an environment but pip to install dependencies. Unfortunately, I am not sure how to inject a pyvenv.cfg when tox is handling the environments.

[mfansler]

@niander what is the problem exactly? Using pip is fine - it defaults to installing into the active Python's prefix. The issue is pip install --user installs. Is that what tox is using?

[niander]

No, they use regular pip install. From logs I can see that first tox-conda create a conda environment. Then, it installs pytest because that is one of the dependencies in my testenv. pytest have attrs >= 19.3.0 as dependency but pip doesn't install it because there is already a newer version of attrs installed in the user site-packages location. Since this is Windows, the location is %APPDATA%\roaming\python\python37\site-packages.

However, when later tox calls pytest, it will fail with ModuleNotFoundException.
So pip found the package in site-packages but when calling pytest it can't find it anymore.

I don't think it is script pointing to a different python issue because I can see that pytest and pip are executed from the same created conda environment.

[mfansler]

@niander Okay, makes sense. Have you tried including conda-ecosystem-user-package-isolation package under conda_deps? That should isolate the Python installation from user space.

[niander]

@mfansler I tried but then later I got an issue with calling pip install directly inside a toxenv (I do this to install a local package in dev mode). pip complains it can't find ssl library.

Nonetheless, I am still failing to understand why when pytest executes it can't find attr if this is installed in the site-packages. If I start the python inside the conda environment, I can do import attr, but the pytest executed by tox doesn't. Also, if I start pytest from the conda environment created by tox I don't get the same error. I am confused by that.

[niander]

I figured it out. It is because of the way tox executes commands. The subprocess that tox creates to execute the test commands doesn't include the environment %APPDATA%.

The code in site.py looks for the user site-packages and if that exists adds that to the python path (unless it is a venv, but not applicable here since it is a conda env and there is no pyvenv.cfg). The logic in site.py for Windows is to see if %APPDATA% exists, if not, uses ~ (%USERPROFILE%).

So, in this case, running inside tox, because the real user site-packages exists only inside %APPDATA% and not ~, site.py will assume that there is no user site-packages and won't add it to the python path.

What I am not sure is why then when tox (or tox-conda) is just installing dependencies the user site-packages is found and added to python path.

Edit: Adding setenv =, APPDATA = {env:APPDATA:~} to tox.ini resolved the situation. Maybe this is a tox issue since APPDATA is an important OS env?

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   - What OS and version you reproduced the issue on
   - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!

[itcarroll]

So this issue has been ongoing for a long time, and github bot keeps closing issues that are not resolved:

• Missing --no-site-packages option #394
• conda incompatible with local installations? #448
• make python user site-packages isolation in .local directory configurable #7707
• maybe others?

Workarounds described include the conda-ecosystem-user-package-isolation package or setting PYTHONNOUSERSITE manually. Here is a case where this workaround is not useful.

A command line tool installed outside a conda environment, using pip install --user, goes in ~/.local/bin/ and includes as its first line #!/path/to/python. THAT Python is outside the conda environment. With PYTHONNOUSERSITE however, site packages are disabled for ALL Python interpreters. This breaks the command line tool.

New workaround.

Although conda does not create a virtual environment, a pyvenv.cfg file is checked by the site module regardless. So, do what venv does. Create a pyvenv.cfg file for your conda environment with the line include-system-site-packages = false.

$ conda activate <name>
$ cat > $CONDA_PREFIX/pyvenv.cfg
include-system-site-packages = false
$

I don't know what other repercussions that has involving sys.prefix, but so far so good. Your mileage may vary.

I strongly feel the conda dev team should provide the --no-site-packages option requested a decade ago in #394.

[mfansler]

Although conda does not create a virtual environment, a pyvenv.cfg file is checked by the site module regardless. So, do what venv does. Create a pyvenv.cfg file for your conda environment with the line include-system-site-packages = false.

This sounds like an excellent solution. The code for that mechanism has been stable since Python v3.3.

This could also be implemented as a Conda Forge package that literally just delivers a pyvenv.cfg to the $PREFIX with that line. I'd be happy to support it (feel free to tag me on staged-recipes to review).

[jezdez]

@itcarroll Please open a new ticket with your summary, it's no use piling on this 5-year-old ticket with yet more discussion where most of the current maintainers were not around.

[jezdez]

Full disclosure, I've opened conda/infrastructure#850 to lock older issue sooner, to make sure we're able to encourage actionable issues instead of reviving old ones.

[itcarroll]

@jezdez Done! You never know whether maintainers are more averse to duplication or re-activation, so thank you for the clear guidance.