x.py: allow configuring the build directory #71994

jyn514 · 2020-05-07T20:55:59Z

This allows configuring the directory for build artifacts, instead of having it always be ./build. This means you can set it to a constant location, letting you reuse the same cache while working in several different directories.

The configuration lives in config.toml under build.build-dir. By default, it keeps the existing default of ./build, but it can be configured to any relative or absolute path. Additionally, it allows making outputs relative to the root of the git repository using $ROOT.

r? @Mark-Simulacrum

rust-highfive · 2020-05-07T20:59:33Z

The job mingw-check of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

========================== Starting Command Output ===========================
[command]/bin/bash --noprofile --norc /home/vsts/work/_temp/9be20770-6cda-4585-924a-7d5f48fb8332.sh

##[section]Finishing: Disable git automatic line ending conversion
##[section]Starting: Checkout rust-lang/rust@refs/pull/71994/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
---
##[command]git remote add origin https:/rust-lang/rust
##[command]git config gc.auto 0
##[command]git config --get-all http.https:/rust-lang/rust.extraheader
##[command]git config --get-all http.proxy
##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/71994/merge:refs/remotes/pull/71994/merge
---
 ---> 3adb0605cc65
Step 6/7 : ENV RUN_CHECK_WITH_PARALLEL_QUERIES 1
 ---> Using cache
 ---> 28dbc326cb7f
Step 7/7 : ENV SCRIPT python3 ../x.py test src/tools/expand-yaml-anchors &&            python3 ../x.py check --target=i686-pc-windows-gnu --host=i686-pc-windows-gnu &&            python3 ../x.py build --stage 0 src/tools/build-manifest &&            python3 ../x.py test --stage 0 src/tools/compiletest &&            python3 ../x.py test src/tools/tidy &&            python3 ../x.py doc --stage 0 src/libstd &&            /scripts/validate-toolstate.sh
 ---> 537a01811900
Successfully built 537a01811900
Successfully tagged rust-ci:latest
Built container sha256:537a018119009dc218456238dec90b5530050db1e2a1e166550c218003f6159d
---
configure: 
Updating only changed submodules
Submodules updated in 0.02 seconds
Traceback (most recent call last):
  File "../x.py", line 11, in <module>
    bootstrap.main()
  File "/checkout/src/bootstrap/bootstrap.py", line 961, in main
    bootstrap(help_triggered)
  File "/checkout/src/bootstrap/bootstrap.py", line 927, in bootstrap
    build.download_stage0()
  File "/checkout/src/bootstrap/bootstrap.py", line 377, in download_stage0
    self._download_stage0_helper(filename, pattern, tarball_suffix)
  File "/checkout/src/bootstrap/bootstrap.py", line 426, in _download_stage0_helper
    os.makedirs(rustc_cache)
  File "/usr/lib/python3.5/os.py", line 231, in makedirs
    makedirs(head, mode, exist_ok)
  File "/usr/lib/python3.5/os.py", line 231, in makedirs
    makedirs(head, mode, exist_ok)
  File "/usr/lib/python3.5/os.py", line 241, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/checkout/build'
  local time: Thu May  7 20:59:14 UTC 2020
  network time: Thu, 07 May 2020 20:59:14 GMT
== end clock drift check ==


##[error]Bash exited with code '1'.
##[section]Finishing: Run build
##[section]Starting: Checkout rust-lang/rust@refs/pull/71994/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
Author       : Microsoft
Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
==============================================================================
Cleaning any cached credential from repository: rust-lang/rust (GitHub)
##[section]Finishing: Checkout rust-lang/rust@refs/pull/71994/merge to s
Cleaning up task key
Start cleaning up orphan processes.
Terminate orphan process: pid (4378) (python)
##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

jyn514 · 2020-05-07T21:05:23Z

Hmm, it looks like the repository is read-only on CI. Could we make the build directory configurable so that it defaults to $rust_root/build but CI passes another temporary directory (probably ./build)?

petrochenkov · 2020-05-07T21:10:01Z

Out of source builds worked before, they just regress regularly due to not being tested.
I used them for some time (year ago? two?), but got tired of the breakage.

Mark-Simulacrum · 2020-05-07T21:11:53Z

I would prefer not to change the default here but would not object to exposing an option to choose the build dir. I'm not sure that a change as minimal as this PR suggests will be able to do it, though.

jyn514 · 2020-05-07T21:17:39Z

I would prefer not to change the default here

What's the use case for not changing it? Right now all it does is rebuild the entire compiler when you change directories.

I would not object to exposing an option to choose the build dir

I pushed a change that allows configuring the build directory, I'm not sure how to pass that to CI though. I guess that if we're not changing the default it won't matter.

jyn514 · 2020-05-07T21:18:26Z

Out of source builds worked before, they just regress regularly due to not being tested.
I used them for some time (year ago? two?), but got tired of the breakage.

If we changed the default, we'd have out-of-source builds always be tested on CI, and the normal use case of not passing a flag would be tested by every developer working on rustc.

rust-highfive · 2020-05-07T21:21:19Z

The job mingw-check of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

========================== Starting Command Output ===========================
[command]/bin/bash --noprofile --norc /home/vsts/work/_temp/560cc307-bfa9-4768-b580-7f86045be5d3.sh

##[section]Finishing: Disable git automatic line ending conversion
##[section]Starting: Checkout rust-lang/rust@refs/pull/71994/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
---
##[command]git remote add origin https:/rust-lang/rust
##[command]git config gc.auto 0
##[command]git config --get-all http.https:/rust-lang/rust.extraheader
##[command]git config --get-all http.proxy
##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/71994/merge:refs/remotes/pull/71994/merge
---
 ---> 3adb0605cc65
Step 6/7 : ENV RUN_CHECK_WITH_PARALLEL_QUERIES 1
 ---> Using cache
 ---> 28dbc326cb7f
Step 7/7 : ENV SCRIPT python3 ../x.py test src/tools/expand-yaml-anchors &&            python3 ../x.py check --target=i686-pc-windows-gnu --host=i686-pc-windows-gnu &&            python3 ../x.py build --stage 0 src/tools/build-manifest &&            python3 ../x.py test --stage 0 src/tools/compiletest &&            python3 ../x.py test src/tools/tidy &&            python3 ../x.py doc --stage 0 src/libstd &&            /scripts/validate-toolstate.sh
 ---> 537a01811900
Successfully built 537a01811900
Successfully tagged rust-ci:latest
Built container sha256:537a018119009dc218456238dec90b5530050db1e2a1e166550c218003f6159d
---
configure: 
Updating only changed submodules
Submodules updated in 0.02 seconds
Traceback (most recent call last):
  File "../x.py", line 11, in <module>
    bootstrap.main()
  File "/checkout/src/bootstrap/bootstrap.py", line 962, in main
    bootstrap(help_triggered)
  File "/checkout/src/bootstrap/bootstrap.py", line 928, in bootstrap
    build.download_stage0()
  File "/checkout/src/bootstrap/bootstrap.py", line 377, in download_stage0
    self._download_stage0_helper(filename, pattern, tarball_suffix)
  File "/checkout/src/bootstrap/bootstrap.py", line 426, in _download_stage0_helper
    os.makedirs(rustc_cache)
  File "/usr/lib/python3.5/os.py", line 231, in makedirs
    makedirs(head, mode, exist_ok)
  File "/usr/lib/python3.5/os.py", line 231, in makedirs
    makedirs(head, mode, exist_ok)
  File "/usr/lib/python3.5/os.py", line 241, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/checkout/build'
  local time: Thu May  7 21:20:34 UTC 2020
  network time: Thu, 07 May 2020 21:20:34 GMT
== end clock drift check ==


##[error]Bash exited with code '1'.
##[section]Finishing: Run build
##[section]Starting: Checkout rust-lang/rust@refs/pull/71994/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
Author       : Microsoft
Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
==============================================================================
Cleaning any cached credential from repository: rust-lang/rust (GitHub)
##[section]Finishing: Checkout rust-lang/rust@refs/pull/71994/merge to s
Cleaning up task key
Start cleaning up orphan processes.
Terminate orphan process: pid (3777) (python)
##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

jyn514 · 2020-05-07T21:30:31Z

It looks like changing this on CI is non-trivial since it imports the bootstrap python module in src/bootstrap/configure.py instead of running x.py directly. Trying to figure it out now but it will take me a while without help.

Mark-Simulacrum · 2020-05-07T21:35:10Z

One easy answer is "don't change directories", and it can also be seen as a feature. But OTOH it's not what cargo does I believe and maybe we should match that. I'm not sure.

The main argument against doing the default change is it's somewhat likely it'll break people's workflows -- if we have an option exposed that says "I want the old behavior" (possibly via --build-dir=$(cwd) or something), though, that's less of a concern.

configure just creates config.toml so we should be able to mv that if needed?

jyn514 · 2020-05-07T21:41:32Z

configure just creates config.toml so we should be able to mv that if needed?

Hmm so the idea is instead of making it an argument it's set in config.toml? I don't think config.toml is parsed until the Rust code, does CI run before or after bootstrap.rs is built?

if we have an option exposed that says "I want the old behavior" (possibly via --build-dir=$(cwd) or something), though, that's less of a concern.

Awesome, yeah my change would allow this with --build-dir=build. Assuming that config.toml parsing happens early enough I could also figure out to let it be set there.

jyn514 · 2020-05-07T21:56:37Z

Ok I think I have an idea what's going on here ...

src/ci/azure-pipelines/steps/run.yml calls src/ci/scripts/run-build-from-ci.sh
run-build-from-ci calls src/ci/run.sh
run.sh calls configure with lots of options, then calls make
configure parses config.toml.example, adds some custom options, then outputs it to config.toml
make calls x.py, which calls src/bootstrap/bootstrap.py
bootstrap.py calls get_toml whenever it wants to know a config parameter.

So the way I would make this configurable is, instead of adding a flag to x.py, reading a section from config.toml. Then for CI I'd add a custom field to the toml file so that it builds in the current directory instead of $SRC/build.

Mark-Simulacrum · 2020-05-07T21:57:33Z

CI runs, well, I'm not sure I follow you -- bootstrap.rs is built on CI, so "both"?

Okay if --build-dir=build to x.py works then I'm fine with this (in the theoretic sense).

I believe what you suggested makes sense but I didn't think too much about it.

jyn514 · 2020-05-07T22:05:03Z

CI runs, well, I'm not sure I follow you -- bootstrap.rs is built on CI, so "both"?

My question was unclear, I meant to ask whether config.toml is parsed before or after downloading the stage0 artifacts. Since it's parsed in bootstrap.py it looks like the answer is before.

jyn514 · 2020-05-07T22:43:49Z

Ok, so now what happens is build-dir is a new entry in config.toml under [build] that defaults to build. If it contains $ROOT, that gets replaced with the top level of the git repository. Additionally, if config.toml doesn't exist in the current directory, it will use the top-level config.toml, just like Cargo.toml.

jyn514 · 2020-05-07T22:45:12Z

I'd still like to change the default just because I think it will be what new developers will expect but I understand that could break workflows. Maybe there's a way to ask for feedback on this somehow? It seems a shame to always be locked into the current defaults.

rust-highfive · 2020-05-07T22:46:58Z

The job mingw-check of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

========================== Starting Command Output ===========================
[command]/bin/bash --noprofile --norc /home/vsts/work/_temp/fb2c286c-e45a-42ef-bb86-6b43dbfb2270.sh

##[section]Finishing: Disable git automatic line ending conversion
##[section]Starting: Checkout rust-lang/rust@refs/pull/71994/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
---
##[command]git remote add origin https:/rust-lang/rust
##[command]git config gc.auto 0
##[command]git config --get-all http.https:/rust-lang/rust.extraheader
##[command]git config --get-all http.proxy
##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/71994/merge:refs/remotes/pull/71994/merge
---
 ---> 3adb0605cc65
Step 6/7 : ENV RUN_CHECK_WITH_PARALLEL_QUERIES 1
 ---> Using cache
 ---> 28dbc326cb7f
Step 7/7 : ENV SCRIPT python3 ../x.py test src/tools/expand-yaml-anchors &&            python3 ../x.py check --target=i686-pc-windows-gnu --host=i686-pc-windows-gnu &&            python3 ../x.py build --stage 0 src/tools/build-manifest &&            python3 ../x.py test --stage 0 src/tools/compiletest &&            python3 ../x.py test src/tools/tidy &&            python3 ../x.py doc --stage 0 src/libstd &&            /scripts/validate-toolstate.sh
 ---> 537a01811900
Successfully built 537a01811900
Successfully tagged rust-ci:latest
Built container sha256:537a018119009dc218456238dec90b5530050db1e2a1e166550c218003f6159d
---
   Compiling cc v1.0.50
    Checking core v0.0.0 (/checkout/src/libcore)
thread 'main' panicked at '

 failed to run "build/x86_64-unknown-linux-gnu/stage0/bin/rustc" "--crate-name" "cc" "--edition=2018" "/cargo/registry/src/github.com-1ecc6299db9ec823/cc-1.0.50/src/lib.rs" "--error-format=json" "--json=diagnostic-rendered-ansi" "--crate-type" "lib" "--emit=dep-info,metadata,link" "-C" "opt-level=3" "-C" "debuginfo=0" "-C" "metadata=9e7bf81b9fe8db6d" "-C" "extra-filename=-9e7bf81b9fe8db6d" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/release/deps" "-L" "dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/release/deps" "--cap-lints" "allow" "-Zbinary-dep-depinfo"', src/bootstrap/bin/rustc.rs:205:54
error: could not compile `cc`.

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
warning: build failed, waiting for other jobs to finish...
thread 'main' panicked at '

 failed to run "build/x86_64-unknown-linux-gnu/stage0/bin/rustc" "--crate-name" "core" "--edition=2018" "src/libcore/lib.rs" "--error-format=json" "--json=diagnostic-rendered-ansi" "--crate-type" "lib" "--emit=dep-info,metadata" "-C" "opt-level=3" "-C" "debuginfo=0" "-C" "metadata=c9663c65a085fa4c" "-C" "extra-filename=-c9663c65a085fa4c" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps" "--target" "x86_64-unknown-linux-gnu" "-L" "dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps" "-L" "dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/release/deps" "--cfg=bootstrap" "-Zmacro-backtrace" "-Clink-args=-Wl,-rpath,$ORIGIN/../lib" "-Wrust_2018_idioms" "-Wunused_lifetimes" "-Dwarnings" "-Cprefer-dynamic" "-Zbinary-dep-depinfo" "--sysroot" "build/x86_64-unknown-linux-gnu/stage0-sysroot" "-C" "debug-assertions=n" "-Z" "force-unstable-if-unmarked"', src/bootstrap/bin/rustc.rs:205:54
error: could not compile `core`.

To learn more, run the command again with --verbose.
command did not execute successfully: "build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "check" "--target" "x86_64-unknown-linux-gnu" "-Zbinary-dep-depinfo" "-j" "2" "--release" "--color" "always" "--features" "panic-unwind backtrace compiler-builtins-c" "--manifest-path" "/checkout/src/libtest/Cargo.toml" "--message-format" "json-render-diagnostics"
---
  local time: Thu May  7 22:46:27 UTC 2020
  network time: Thu, 07 May 2020 22:46:27 GMT
== end clock drift check ==

##[error]Bash exited with code '1'.
##[section]Finishing: Run build
##[section]Starting: Checkout rust-lang/rust@refs/pull/71994/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
Author       : Microsoft
Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
==============================================================================
Cleaning any cached credential from repository: rust-lang/rust (GitHub)
##[section]Finishing: Checkout rust-lang/rust@refs/pull/71994/merge to s
Cleaning up task key
Start cleaning up orphan processes.
Terminate orphan process: pid (4669) (python)
##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

rust-highfive · 2020-05-07T23:03:00Z

The job mingw-check of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

========================== Starting Command Output ===========================
[command]/bin/bash --noprofile --norc /home/vsts/work/_temp/2510c3ee-d9df-4726-b813-59453672ecc5.sh

##[section]Finishing: Disable git automatic line ending conversion
##[section]Starting: Checkout rust-lang/rust@refs/pull/71994/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
---
##[command]git remote add origin https:/rust-lang/rust
##[command]git config gc.auto 0
##[command]git config --get-all http.https:/rust-lang/rust.extraheader
##[command]git config --get-all http.proxy
##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/71994/merge:refs/remotes/pull/71994/merge
---
 ---> 3adb0605cc65
Step 6/7 : ENV RUN_CHECK_WITH_PARALLEL_QUERIES 1
 ---> Using cache
 ---> 28dbc326cb7f
Step 7/7 : ENV SCRIPT python3 ../x.py test src/tools/expand-yaml-anchors &&            python3 ../x.py check --target=i686-pc-windows-gnu --host=i686-pc-windows-gnu &&            python3 ../x.py build --stage 0 src/tools/build-manifest &&            python3 ../x.py test --stage 0 src/tools/compiletest &&            python3 ../x.py test src/tools/tidy &&            python3 ../x.py doc --stage 0 src/libstd &&            /scripts/validate-toolstate.sh
 ---> 537a01811900
Successfully built 537a01811900
Successfully tagged rust-ci:latest
Built container sha256:537a018119009dc218456238dec90b5530050db1e2a1e166550c218003f6159d
---
   Compiling cc v1.0.50
    Checking core v0.0.0 (/checkout/src/libcore)
thread 'main' panicked at '

 failed to run "build/x86_64-unknown-linux-gnu/stage0/bin/rustc" "--crate-name" "cc" "--edition=2018" "/cargo/registry/src/github.com-1ecc6299db9ec823/cc-1.0.50/src/lib.rs" "--error-format=json" "--json=diagnostic-rendered-ansi" "--crate-type" "lib" "--emit=dep-info,metadata,link" "-C" "opt-level=3" "-C" "debuginfo=0" "-C" "metadata=9e7bf81b9fe8db6d" "-C" "extra-filename=-9e7bf81b9fe8db6d" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/release/deps" "-L" "dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/release/deps" "--cap-lints" "allow" "-Zbinary-dep-depinfo"', src/bootstrap/bin/rustc.rs:205:54
thread 'main' panicked at '


 failed to run "build/x86_64-unknown-linux-gnu/stage0/bin/rustc" "--crate-name" "core" "--edition=2018" "src/libcore/lib.rs" "--error-format=json" "--json=diagnostic-rendered-ansi" "--crate-type" "lib" "--emit=dep-info,metadata" "-C" "opt-level=3" "-C" "debuginfo=0" "-C" "metadata=c9663c65a085fa4c" "-C" "extra-filename=-c9663c65a085fa4c" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps" "--target" "x86_64-unknown-linux-gnu" "-L" "dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps" "-L" "dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/release/deps" "--cfg=bootstrap" "-Zmacro-backtrace" "-Clink-args=-Wl,-rpath,$ORIGIN/../lib" "-Wrust_2018_idioms" "-Wunused_lifetimes" "-Dwarnings" "-Cprefer-dynamic" "-Zbinary-dep-depinfo" "--sysroot" "build/x86_64-unknown-linux-gnu/stage0-sysroot" "-C" "debug-assertions=n" "-Z" "force-unstable-if-unmarked"', src/bootstrap/bin/rustc.rs:205:54
error: could not compile `core`.

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
---
  local time: Thu May  7 23:02:01 UTC 2020
  network time: Thu, 07 May 2020 23:02:01 GMT
== end clock drift check ==

##[error]Bash exited with code '1'.
##[section]Finishing: Run build
##[section]Starting: Checkout rust-lang/rust@refs/pull/71994/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
Author       : Microsoft
Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
==============================================================================
Cleaning any cached credential from repository: rust-lang/rust (GitHub)
##[section]Finishing: Checkout rust-lang/rust@refs/pull/71994/merge to s
Cleaning up task key
Start cleaning up orphan processes.
Terminate orphan process: pid (4180) (python)
##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

jyn514 · 2020-05-07T23:40:20Z

Hmm, the failure message looks like it tried to run a rustc command that didn't exist, but I'm not sure why since I didn't change any defaults.

Also, while debugging that I noticed that there is already an out field for Config, which seems like it could be trouble when combined with a separate build_dir field for Build.

src/bootstrap/bootstrap.py

Mark-Simulacrum · 2020-05-08T20:56:07Z

It's pretty surprising to me that this didn't require any changes to the Rust code in bootstrap -- can you at least briefly explain why?

jyn514 · 2020-05-08T22:19:01Z

It's pretty surprising to me that this didn't require any changes to the Rust code in bootstrap -- can you at least briefly explain why?

As best I understand, it's because the Rust code gets the build directory from bootstrap.py, and that was the case even before this PR. bootstrap.py sets BUILD_DIR to build.build_dir here: https:/rust-lang/rust/blob/6665be6268d47e4c3dfd718872d5a26df2d31a78/src/bootstrap/bootstrap.py#L946

jyn514 · 2020-05-08T22:20:15Z

If you like I can try to run a full x.py test locally with a different build-dir set in config.toml, but it will take me a while.

Mark-Simulacrum · 2020-05-08T23:20:46Z

No, that's sufficient, I had forgotten about our custom logic there.

Could you add a comment to that effect on the field added to the Build struct?

jyn514 · 2020-05-08T23:44:51Z

Sure, I added a comment.

Mark-Simulacrum · 2020-05-09T00:18:24Z

Please update the PR description to reflect the current implementation and squash the commits into one; once that's done I think we should be able to land this.

ollie27 · 2020-05-09T00:30:01Z

If the old default defined at

rust/src/bootstrap/bootstrap.py

Line 335 in 7ebd87a

self.build_dir = os.path.join(os.getcwd(), "build")

is no longer used then I think it would make sense to replace that with an empty string like the other fields.

This allows configuring the directory for build artifacts, instead of having it always be ./build. This means you can set it to a constant location, letting you reuse the same cache while working in several different directories. The configuration lives in config.toml under build.build-dir. By default, it keeps the existing default of ./build, but it can be configured to any relative or absolute path. Additionally, it allows making outputs relative to the root of the git repository using $ROOT.

jyn514 · 2020-05-09T00:34:10Z

Squashed the commits and removed the unused default

Mark-Simulacrum · 2020-05-09T00:50:18Z

Thanks!

@bors r+

bors · 2020-05-09T00:50:19Z

📌 Commit df36ec0 has been approved by Mark-Simulacrum

Dylan-DPC-zz · 2020-05-09T01:12:09Z

@bors p=1

bors · 2020-05-09T06:27:02Z

⌛ Testing commit df36ec0 with merge 34d6b44...

bors · 2020-05-09T09:50:51Z

☀️ Test successful - checks-actions, checks-azure
Approved by: Mark-Simulacrum
Pushing 34d6b44 to master...

rust-highfive assigned Mark-Simulacrum May 7, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 7, 2020

pickfire reviewed May 8, 2020

View reviewed changes

src/bootstrap/bootstrap.py Show resolved Hide resolved

jyn514 force-pushed the path-independent branch from ee41958 to 01315fb Compare May 8, 2020 13:39

jyn514 changed the title ~~Allow running x.py without being in the top-level directory~~ x.py: allow configuring the build directory May 9, 2020

jyn514 force-pushed the path-independent branch from 40de856 to df36ec0 Compare May 9, 2020 00:34

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 9, 2020

bors added the merged-by-bors This PR was explicitly merged by bors. label May 9, 2020

bors merged commit 34d6b44 into rust-lang:master May 9, 2020

jyn514 deleted the path-independent branch May 9, 2020 11:38

Mark-Simulacrum mentioned this pull request Sep 14, 2020

De-couple Python and bootstrap slightly #76544

Merged

jyn514 added the T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) label Sep 27, 2020

jyn514 mentioned this pull request Oct 5, 2020

Add a command to install a git hook to automatically run x.py test tidy --bless #76356

Merged

jyn514 mentioned this pull request Mar 10, 2022

Fallback to top-level config.toml if not present in current directory, and remove fallback for env vars and CLI flags #94592

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x.py: allow configuring the build directory #71994

x.py: allow configuring the build directory #71994

jyn514 commented May 7, 2020 •

edited

Loading

rust-highfive commented May 7, 2020

jyn514 commented May 7, 2020

petrochenkov commented May 7, 2020 •

edited

Loading

Mark-Simulacrum commented May 7, 2020

jyn514 commented May 7, 2020

jyn514 commented May 7, 2020 •

edited

Loading

rust-highfive commented May 7, 2020

jyn514 commented May 7, 2020 •

edited

Loading

Mark-Simulacrum commented May 7, 2020

jyn514 commented May 7, 2020 •

edited

Loading

jyn514 commented May 7, 2020

Mark-Simulacrum commented May 7, 2020

jyn514 commented May 7, 2020 •

edited

Loading

jyn514 commented May 7, 2020

jyn514 commented May 7, 2020

rust-highfive commented May 7, 2020

rust-highfive commented May 7, 2020

jyn514 commented May 7, 2020

Mark-Simulacrum commented May 8, 2020

jyn514 commented May 8, 2020

jyn514 commented May 8, 2020

Mark-Simulacrum commented May 8, 2020

jyn514 commented May 8, 2020

Mark-Simulacrum commented May 9, 2020

ollie27 commented May 9, 2020

jyn514 commented May 9, 2020

Mark-Simulacrum commented May 9, 2020

bors commented May 9, 2020

Dylan-DPC-zz commented May 9, 2020

bors commented May 9, 2020

bors commented May 9, 2020

x.py: allow configuring the build directory #71994

x.py: allow configuring the build directory #71994

Conversation

jyn514 commented May 7, 2020 • edited Loading

rust-highfive commented May 7, 2020

jyn514 commented May 7, 2020

petrochenkov commented May 7, 2020 • edited Loading

Mark-Simulacrum commented May 7, 2020

jyn514 commented May 7, 2020

jyn514 commented May 7, 2020 • edited Loading

rust-highfive commented May 7, 2020

jyn514 commented May 7, 2020 • edited Loading

Mark-Simulacrum commented May 7, 2020

jyn514 commented May 7, 2020 • edited Loading

jyn514 commented May 7, 2020

Mark-Simulacrum commented May 7, 2020

jyn514 commented May 7, 2020 • edited Loading

jyn514 commented May 7, 2020

jyn514 commented May 7, 2020

rust-highfive commented May 7, 2020

rust-highfive commented May 7, 2020

jyn514 commented May 7, 2020

Mark-Simulacrum commented May 8, 2020

jyn514 commented May 8, 2020

jyn514 commented May 8, 2020

Mark-Simulacrum commented May 8, 2020

jyn514 commented May 8, 2020

Mark-Simulacrum commented May 9, 2020

ollie27 commented May 9, 2020

jyn514 commented May 9, 2020

Mark-Simulacrum commented May 9, 2020

bors commented May 9, 2020

Dylan-DPC-zz commented May 9, 2020

bors commented May 9, 2020

bors commented May 9, 2020

jyn514 commented May 7, 2020 •

edited

Loading

petrochenkov commented May 7, 2020 •

edited

Loading

jyn514 commented May 7, 2020 •

edited

Loading

jyn514 commented May 7, 2020 •

edited

Loading

jyn514 commented May 7, 2020 •

edited

Loading

jyn514 commented May 7, 2020 •

edited

Loading