non-root requirement seems too much for an ordinary user of rules_python

[qzmfranklin]

🐞 bug report

Affected Rule

The issue is caused by the rule:

During the loading phase.

Is this a regression?

I don't think so.

Description

Per #713 , rules_python cannot be used by root.

CI systems such as CircleCI and BuildBuddy uses root user by default.

Setting up non-root in those systems aren't very straightforward.

Also, it does not look like a responsibility of an ordinary user of rules_python to have to do this.

🔬 Minimal Reproduction

Example build event:

https://app.buildbuddy.io/invocation/55238174-54c3-459c-8d80-72722f2d00f9

🔥 Exception or Error

Relevant error message:

Error in fail: The current user is root, please run as non-root when using the hermetic Python interpreter. See https:/bazelbuild/rules_python/pull/713.

🌍 Your Environment

Operating System:

  
Linux, CentOS8, WSL2-Ubuntu22.04
  

Output of bazel version:

  
6.1.1
  

Rules_python version:

  
1e869d8d945f0b406da65817cf70b4fa3105a0de
  

Anything else relevant?

[nsubiron]

You can disable the error

python_register_toolchains(
    name = ...,
    python_version = ...,
    ignore_root_user_error = True,
)

[faximan]

It is good there is an option to choose but you are choosing between two evils: either you get the pyc cache misses or you have to go out of your way to support non-root builds (e.g. when building in a docker container).

We have opted for the second evil, but it creates so much extra work.

I really wish there was a way to solve the cache issue without this extra requirement.

[phlax]

we hit the same issues - my feeling is that this is just not the responsibility of rule_python to enforce this

[qzmfranklin]

To be fair, it's probably the CPython to blame, as they programmed to produce the .pyc files on the fly.

Not sure if they are willing to make a flag to NOT generate .pyc files on the stdlib on the fly just for this use case, though.

[daixiang0]

You can disable the error

python_register_toolchains(
    name = ...,
    python_version = ...,
    ignore_root_user_error = True,
)

Works for me!

[tsaarni]

When it is not feasible to run as non-root, currently one needs to modify the build files of the project. This is bit clumsy. Could the ignore_root_user_error option be exposed to the user who runs the build of some random project "xyz", which happens to use rules_python internally? What would be the standard approach to do so?

[learnsomethingtoday]

You can disable the error

python_register_toolchains(
    name = ...,
    python_version = ...,
    ignore_root_user_error = True,
)

Works for me!

@daixiang0 / @nsubiron Could you please let me know how and where did u make this edit?

[cpsauer]

^ I think this workaround doesn't suffice for build tools seeking to use rules_python in their implementations, since they aren't the root module. More context over in hedronvision/bazel-compile-commands-extractor#166

It'd be quite valuable to have this just work out of the box. That is, if running as root, fall back to the best behavior you can rather than erroring.

[calebzulawski]

Could this setting at least be overridden with an environment variable? I have a rules_$company module that is used by maybe a dozen separate repos that establishes uniform toolchains and CI scripts among other things. Since the toolchain can only be set by the root module, I have no way of overriding this other than separately tracking the toolchain in each repo.

[rickeylev]

Could this setting at least be overridden with an environment variable?

Yes, I think thats fine. Send a PR?

re: the issue in general: Unfortunately, I don't see any great solutions to this problem.

Fall back to best behavior you can

I am somewhat inclined to agree. The reason being: the problem a writable install causes is spurious rebuilds. When it's an error to have a writable install, and people have to opt out, then they're just going to opt out, because that going to mostly work, rather than entirely not work. And they just have to live with spurious rebuilds.

I don't see any interpreter options or environment variable to use hashed-based pycs instead of timestamps. Which is too bad; it would have been a nice fix to this. Nor do I see any options to control where pycs are read-from/written-to for just for the stdlib.

When a runtime is initially downloaded and extracted, we could run a compile step to pre-populate the pyc files. However, this only works if the downloaded runtime runs on the host. If we could somehow ensure that there was a host-runnable python of the same version available, then that could be used to perform the compilation; or if we had a prebuilt tool to perform this. This does seem like one of the better options, though. If the host can't run the Python, then we just move along.

Alternatively, the pyc could be generated at build time. We can do this for regular libraries now; I don't see why we couldn't also do it for the py files coming from the runtime. The downside is it'll add some build overhead, since there's a few thousand files to compile.

Maybe if the stdlib was in a zip file we'd have more options? This idea was floated as a way to reduce the number of files in the runtime, too. I'm not sure where Python will write pyc files if it reads a file from a zip.

It'd be most ideal if the upstream python-standalone packages came with the pyc files already. That would make this go away entirely.