How to cross-compile cryptography with rust support?

[holzschu]

Hi,
I'm trying to compile the new version of cryptography, with rust support, for iOS and iOS simulator, on a Mac.

Cross-compiling the C files works by setting CFLAGS and CC inside the environment:

env CC=clang CXX=clang++ \
CPPFLAGS="-arch arm64 -miphoneos-version-min=14.0 -isysroot $IOS_SDKROOT -I$PREFIX  -I$PREFIX/Frameworks_iphoneos/include/ -DCRYPTOGRAPHY_OSRANDOM_ENGINE=CRYPTOGRAPHY_OSRANDOM_ENGINE_DEV_URANDOM" \
CFLAGS="-arch arm64 -miphoneos-version-min=14.0 -isysroot $IOS_SDKROOT -I$PREFIX  -I$PREFIX/Frameworks_iphoneos/include/  -DCRYPTOGRAPHY_OSRANDOM_ENGINE=CRYPTOGRAPHY_OSRANDOM_ENGINE_DEV_URANDOM " \
CXXFLAGS="-arch arm64 -miphoneos-version-min=14.0 -isysroot $IOS_SDKROOT -I$PREFIX  -I$PREFIX/Frameworks_iphoneos/include/  -DCRYPTOGRAPHY_OSRANDOM_ENGINE=CRYPTOGRAPHY_OSRANDOM_ENGINE_DEV_URANDOM" \
LDFLAGS="-arch arm64 -miphoneos-version-min=14.0 -isysroot $IOS_SDKROOT -F$PREFIX/Frameworks_iphoneos -framework ios_system -L$PREFIX/build/lib.darwin-arm64-3.9 -L$PREFIX/Frameworks_iphoneos/lib/" \
LDSHARED="clang -v -undefined error -dynamiclib -isysroot $IOS_SDKROOT -F$PREFIX/Frameworks_iphoneos -framework ios_system -L$PREFIX/build/lib.darwin-arm64-3.9 -lz -lpython3.9 -L$PREFIX/Frameworks_iphoneos/lib/" \
PLATFORM=iphoneos python3.9 setup.py build

(I'll admit the command line is a bit scary, but it's basically just setting flags).

However, setting flags for rustc does not work in the current release. I've tried adding RUSTFLAGS="--target=aarch64-apple-ios" to the previous command line, but all I get is an error:

(...)
running build_rust
cargo rustc --lib --manifest-path src/rust/Cargo.toml --features pyo3/abi3-py36 pyo3/extension-module --release --verbose -- --crate-type cdylib -C link-arg=-undefined -C link-arg=dynamic_lookup
error: cannot produce cdylib for `cryptography-rust v0.1.0 (/Users/holzschu/src/Xcode_iPad/Carnets/cpython/packages/cryptography-3.4.4/src/rust)` as the target `x86_64-apple-darwin` does not support these crate types

As you can see, the target in the error message is not the target specified on the --target flag, which is confusing. Adding the "--target" flag manually to the compile command works for a while longer:

> cargo rustc --lib --manifest-path src/rust/Cargo.toml --features pyo3/abi3-py36 pyo3/extension-module --release --verbose -- --crate-type cdylib -C link-arg=-undefined -C link-arg=dynamic_lookup --target=aarch64-apple-ios
   Compiling proc-macro2 v1.0.24
   Compiling unicode-xid v0.2.1
   Compiling syn v1.0.60
   Compiling libc v0.2.85
   Compiling cfg-if v1.0.0
   Compiling scopeguard v1.1.0
   Compiling inventory v0.1.10
   Compiling smallvec v1.6.1
   Compiling unindent v0.1.7
   Compiling pyo3 v0.13.1
   Compiling paste v1.0.4
   Compiling instant v0.1.9
   Compiling lock_api v0.4.2
   Compiling indoc v1.0.3
   Compiling quote v1.0.8
   Compiling parking_lot_core v0.8.2
   Compiling parking_lot v0.11.1
   Compiling pyo3-macros-backend v0.13.1
   Compiling inventory-impl v0.1.10
   Compiling ctor v0.1.19
   Compiling ghost v0.1.2
   Compiling pyo3-macros v0.13.1
   Compiling cryptography-rust v0.1.0 (/Users/holzschu/src/Xcode_iPad/Carnets/cpython/packages/cryptography-3.4.4/src/rust)
error: Unrecognized option: 'release'

error: could not compile `cryptography-rust`

To learn more, run the command again with --verbose.

My question is: what's the recommended method to cross-compile cryptography, and how to set flags for the rust compiler?

Python: 3.9
cffi: 1.14.2
pip: 21.0.1
setuptools: 53.0.0
setuptools_rust: 0.11.6

[alex]

Can you try setting the env var CARGO_BUILD_TARGET=... to the target you're trying to cross-compile for?

[holzschu]

The error message is less puzzling, but it crashed at the same place:

running build_rust
cargo rustc --lib --manifest-path src/rust/Cargo.toml --features pyo3/abi3-py36 pyo3/extension-module --target aarch64-apple-ios --release --verbose -- --crate-type cdylib -C link-arg=-undefined -C link-arg=dynamic_lookup
error: cannot produce cdylib for `cryptography-rust v0.1.0 (/Users/holzschu/src/Xcode_iPad/Carnets/cpython/packages/cryptography-3.4.4/src/rust)` as the target `aarch64-apple-ios` does not support these crate types

[alex]

Looks like this is a platform limitation: https://users.rust-lang.org/t/target-aarch64-apple-ios-does-not-support-these-cdylib-crate-type/55255

That thread suggests that no .so would ever have worked on iOS, at least historically, if I'm reading correctly.

[holzschu]

Thanks. I've read the post you link. That is a very dated view; I am definitely using dynamic libraries in iOS (I have to convert them to frameworks, but that's a straightforward process; I am already doing it with cryptography/hazmat/bindings/_padding.abi3.so and cryptography/hazmat/bindings/_openssl.abi3.so). I'll keep investigating.

[alex]

I think the answer here is simply "Rust's view of what's possible on iOS is out of date, and you should file an issue with rustc asking them to fix this" :-) If you file an issue please do link it here!

…

On Thu, Feb 11, 2021 at 12:41 PM Nicolas Holzschuch < ***@***.***> wrote: Thanks. I've read the post you link. That is a very dated view; I am definitely using dynamic libraries in iOS (I have to convert them to frameworks, but that's a straightforward process; I am already doing it with cryptography/hazmat/bindings/_padding.abi3.so and cryptography/hazmat/bindings/_openssl.abi3.so). I'll keep investigating. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5814 (comment)>, or unsubscribe <https:/notifications/unsubscribe-auth/AAAAGBED6LLZC37HMCQOWMTS6QJF5ANCNFSM4XPG7EPA> .

-- All that is necessary for evil to succeed is for good people to do nothing.

[holzschu]

Done. That issue has been open since 2015, though.

[jameshilliard]

FYI I'm in the process of porting the rust setuptools infrastructure to buildroot, this is roughly what my package makefile looks like:

################################################################################
#
# python-cryptography
#
################################################################################

PYTHON_CRYPTOGRAPHY_VERSION = 3.4.4
PYTHON_CRYPTOGRAPHY_SOURCE = cryptography-$(PYTHON_CRYPTOGRAPHY_VERSION).tar.gz
PYTHON_CRYPTOGRAPHY_SITE = https://files.pythonhosted.org/packages/27/5a/007acee0243186123a55423d49cbb5c15cb02d76dd1b6a27659a894b13a2
PYTHON_CRYPTOGRAPHY_SETUP_TYPE = setuptools
PYTHON_CRYPTOGRAPHY_LICENSE = Apache-2.0 or BSD-3-Clause
PYTHON_CRYPTOGRAPHY_LICENSE_FILES = LICENSE
PYTHON_CRYPTOGRAPHY_DEPENDENCIES = host-python-setuptools-rust host-python-cffi host-rustc
PYTHON_CRYPTOGRAPHY_ENV = CARGO_BUILD_TARGET="$(RUSTC_TARGET_NAME)" PYO3_CROSS_LIB_DIR="$(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)" CARGO_HOME="$(HOST_DIR)/share/cargo"

$(eval $(python-package))

Note that this currently depends on the following unmerged patches:
setuptools update
add semantic-version
update setuptools-scm
add setuptools-rust

I should add that the rust requirement appears somewhat likely to cause serious security issues for embedded Linux if there is no long term support plan for the many architectures without full rust toolchain support as rust toolchain support is currently extremely limited compared to our supported architectures. I would highly recommend some sort of longer term migration plan such as a LTS branch or non-rust fallbacks until the rust toolchain support matures more.

I do think moving to rust long term will be beneficial to security, however for a critical library like cryptography making rust mandatory without a sufficiently long migration period has the potential to be highly counterproductive as the likely result will be pinning old unsupported versions. In any case it's not a major issue for any of my projects using buildroot as those do happen to have rust full toolchain support but I can see it being a serious issue for others.

I think compared to the gentoo issue the issues with rust cross compilation might be even more severe due to the toolchain bootstrapping issues(which from my understanding is why buildroot only supports rust for platforms with prebuilt toolchains). Normally buildroot bootstraps all required cross toolchains by building them from source, rust is basically the only exception there from my understanding.

[tiran]

I should add that the rust requirement appears somewhat likely to cause serious security issues for embedded Linux if there is no long term support plan for the many architectures without full rust toolchain support as rust toolchain support is currently extremely limited compared to our supported architectures.

Personally I consider Rust as a litmus test for platforms. Rust 1.0 was released more than half a decade ago. It is like that any platform without Rust support simply does not attract sufficient attention from researcher and developers. Which by itself commonly means that these platforms have a small user base. OSS projects have limit resources and have to make compromises how do spend their resources. To quote Spock: "The needs of the many outweigh the needs of the few"

I recommend that your work with Rust team and GCC's gccrs team to improve support for your platforms instead.

[jameshilliard]

It is like that any platform without Rust support simply does not attract sufficient attention from researcher and developers. Which by itself commonly means that these platforms have a small user base.

Embedded systems in my experience are quite different there, you will often(most of the time probably) have a very large user base that doesn't even know what OS they are running. For example, how many users even know their home router is likely Linux based, and how many users have a Linux based router, one is vastly larger than the other. And exotic architectures tend to be relatively common for embedded systems. It's also difficult as is getting vendors to stay up to date with security, throw in a bunch more toolchain issues(on top of the ones they already are fighting half the time) and the issues are likely to get worse.

OSS projects have limit resources and have to make compromises how do spend their resources.

Sure, I'm just pointing out that if the decision is being made based on the assumption that the vast majority of users(which is quite different from developers) have a rust supporting platform, then there's a good chance the current situation is quite a bit different than assumed, simply due to a library like cryptography being part of a huge amount of infrastructure software that often isn't very visible.

I recommend that your work with Rust team and GCC's gccrs team to improve support for your platforms instead.

Sure, but that's not something that will really help much in the short term, things are improving and I'm doing what I can to help but these things take time and I think for a change this big it would benefit users a lot to have a longer migration period, even if a non-rust version would be just receiving security fixes only. At least that would mitigate the bulk of the near term security issues on the embedded Linux side of things.

The gccrs port looks interesting, seems like it could bring rust to a lot more architectures a lot easier than trying to port LLVM, I'm not really a compiler developer myself but I'll def try and help with packaging and testing once it gets more mature.

[tiran]

(Disclaimer: I'm not a core developer of PyCA cryptography, only a contributor)

And exotic architectures tend to be relatively common for embedded systems. It's also difficult as is getting vendors to stay up to date with security, throw in a bunch more toolchain issues(on top of the ones they already are fighting half the time) and the issues are likely to get worse.

Your words could be interpreted as guilt-pressuring unpaid Open Source developers into supporting these embedded platforms, as otherwise "humans may die a horrible death when embedded controller of an elevator fails". A bit unfair, don't you think? </exaggeration>

I got your point of view. The decision to use Rust puts your in an uncomfortable position. You want to archive what's best for your work, product and customers. Please also understand that PyCA cryptography is a literally a two-man show. Alex and Paul are doing 99% of all work. I'm honestly worried that this week's shit storm and following peer pressure is going to burn them out.

Ultimately security updates and software maintenance is a social, legal, and political issue that is out of scope for a little OSS project as PyCA cryptography. For example the EU is working on laws in regards of right-to-repair and mandatory security updates during the life cycle of a product.

[reaperhulk]

And exotic architectures tend to be relatively common for embedded systems. It's also difficult as is getting vendors to stay up to date with security, throw in a bunch more toolchain issues(on top of the ones they already are fighting half the time) and the issues are likely to get worse.

This is an unfortunate situation, but it's also unfair to ask others to shoulder the burden of support which companies are unwilling to appropriately invest in. Domains like embedded systems push the externalities of supporting them on everyone else. Some of that is inevitable, but it's also important to keep pushing forward the state of our industry even if it results in some pain.

We would love for Rust (and our project) to be available everywhere! However, ubiquity is created through demand. In most of the cases we've seen so far the challenge has been about getting the right people talking to each other instead of any specific technical obstacle. This makes me hopeful that for many of these platforms the problem will be solved sooner rather than later.

We also had an LTS discussion in #5799. The outcome was that while we are not going to explicitly maintain an LTS (see the thread for why this is such a commitment), we've made a small change to how we handle CVEs to make backports easier for packagers (#5799 (comment)). This should make the runway longer for those who absolutely need it.

[jameshilliard]

The decision to use Rust puts your in an uncomfortable position. You want to archive what's best for your work, product and customers.

It's not an issue for me personally, since the platforms I currently support(x86_64 and aarch64) have rust cross toolchains, but I noticed the extremely limited rust toolchain architecture support when working on bringing up setuptools-rust for buildroot.

Please also understand that PyCA cryptography is a literally a two-man show. Alex and Paul are doing 99% of all work. I'm honestly worried that this week's shit storm and following peer pressure is going to burn them out.

I get that, but one should realize that this is a hugely disruptive change for packagers/integrators. Maybe I'm underestimating the maintenance burden of providing better migration paths here but I find backwards compat to be extremely important for users of infrastructure projects like this, mostly since I've seen what usually happens when updating a dependency becomes difficult(it either won't get updated or will be replaced with a different dependency/fork). Generally fixing CVE's in a LTS branch as they come in is not particularly time consuming unless there are serious fundamental design flaws that require reworking. Maybe some of the distro maintainers would be willing to help maintain a backwards compatible version.

Ultimately security updates and software maintenance is a social, legal, and political issue that is out of scope for a little OSS project as PyCA cryptography. For example the EU is working on laws in regards of right-to-repair and mandatory security updates during the life cycle of a product.

IMO a big reason embedded systems have these security update issues is due to infrastructure projects in their dependency trees not providing good migration paths, I'd say this falls under the social issue there. The Linux kernel has been improving here a lot at least with extended LTS support periods but the userspace situation is not very good overall.

This is an unfortunate situation, but it's also unfair to ask others to shoulder the burden of support which companies are unwilling to appropriately invest in. Domains like embedded systems push the externalities of supporting them on everyone else. Some of that is inevitable, but it's also important to keep pushing forward the state of our industry even if it results in some pain.

I'd say for the most part the reality of the situation is the opposite, embedded systems manufacturers typically handle most maintenance/updates themselves, which means they will typically just version pin or hack around dependencies that have problematic updates, sure some work with their upstreams but that's generally not the most common case. I usually find it easier to push things forward by providing better migration paths, just look at the python 2 to python 3 migration, which didn't seriously take off until features/shims that made single source universal python2/python3 code workable were introduced.

We would love for Rust (and our project) to be available everywhere! However, ubiquity is created through demand. In most of the cases we've seen so far the challenge has been about getting the right people talking to each other instead of any specific technical obstacle. This makes me hopeful that for many of these platforms the problem will be solved sooner rather than later.

I agree that rust should be available in more places, but this is a long tail sort of issue due to the extremely extended support periods for many platforms/architectures and toolchain complexity. Supporting rust on a lot of these platforms is essentially an unsolvable issue in the near term unfortunately.

[jameshilliard]

Now that buildroot has merged most of the patches I sent in adding the rust setuptools build infrastructure needed for building python-cryptography I've now sent in the patch which updates cryptography here:
package/python-cryptography: bump to version 3.4.6
This does still need this setuptools update patch which is in the same series:
package/python3-setuptools: bump to version 54.1.2

[jameshilliard]

I'm running into some bad longstanding bugs/missing config options in regards to how rust is handling cross toolchain build script linking that seems to be blocking cryptography cross compilation from being workable for upstream buildroot at the moment even for normal x86_64 targets.
See:
#3349
#1513

[saurik]

@jameshilliard You might also be interested in rust-lang/cargo#8147. I have been doing a lot of what I would want to claim is "simple" cross-compiling with Rust over the past year and a half--notably, I am not even working with niche targets!--but they seem to only particularly care about a handful of specific host/target configurations (and even some of those are a bit broken, such as macOS -> iOS)... host==target has never gotten any real interest, sadly. I hope you make progress on this, as my workarounds are getting more and more hilarious (I have been known to just tell Rust "my linker is /bin/true" so I can get it to stop messing me up ;P).

[jameshilliard]

I have been doing a lot of what I would want to claim is "simple" cross-compiling with Rust over the past year and a half--notably, I am not even working with niche targets!--but they seem to only particularly care about a handful of specific host/target configurations (and even some of those are a bit broken, such as macOS -> iOS)... host==target has never gotten any real interest, sadly.

Yeah, I could somewhat understand if there was a viable workaround for this but I have yet to find anything that looks even remotely maintainable, we maintain a lot of cross compilation workarounds/hacks in buildroot but this situation seems totally ridiculous to me.

I hope you make progress on this, as my workarounds are getting more and more hilarious (I have been known to just tell Rust "my linker is /bin/true" so I can get it to stop messing me up ;P).

Maybe with the Linux kernel working on a rust integration the situation might improve...I really had no idea rust cross compilation was this immature/broken when I started working on adding python rust support to buildroot. Best I can tell the main reason this is still broken seems to be bike-shedding of the issue endlessly. I'm not familiar with the rust/cargo codebase but if I had to guess it's probably only a few lines of code to add a build script linker override env variable or something that would at least let us override this broken behavior in some way. At a certain point it becomes more important to just provide a way for others to make it work in some way rather than to try and bike-shed the most ideal solution. Normally I could use the buildroot patch system to workaround this but since it's not really all that viable to build a rust toolchain from source due to the rust requires rust to build bootstrapping issue...that's not even workable.

[jameshilliard]

I'm running into some bad longstanding bugs/missing config options in regards to how rust is handling cross toolchain build script linking that seems to be blocking cryptography cross compilation from being workable for upstream buildroot at the moment even for normal x86_64 targets.
See:
#3349
#1513

So for a status update on getting these target==host architecture cross compilation bugs fixed, the pyo3 fix has been merged in #1514, in addition I have a pending pull request fixing the related cargo bug in #9322.

[alex]

Thank you for your work on this!

…

On Mon, Apr 5, 2021 at 2:18 PM James Hilliard ***@***.***> wrote: I'm running into some bad longstanding bugs/missing config options in regards to how rust is handling cross toolchain build script linking that seems to be blocking cryptography cross compilation from being workable for upstream buildroot at the moment even for normal x86_64 targets. See: #3349 <rust-lang/cargo#3349> #1513 <PyO3/pyo3#1513> So for a status update on getting these target==host architecture cross compilation bugs fixed, the pyo3 fix has been merged in #1514 <PyO3/pyo3#1514>, in addition I have a pending pull request fixing the related cargo bug in #9322 <rust-lang/cargo#9322>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5814 (comment)>, or unsubscribe <https:/notifications/unsubscribe-auth/AAAAGBCVO5TQQQPXDSEOBXTTHH5HHANCNFSM4XPG7EPA> .

-- All that is necessary for evil to succeed is for good people to do nothing.

[jameshilliard]

So cross compilation seems to mostly work on nightly rust if you set -Z target-applies-to-host in the cli and CARGO_TARGET_APPLIES_TO_HOST=false in the environment now that my initial fix is merged, although I don't think my fixes for pyo3 have landed in a release yet so things are probably going to be a bit broken there still.

In general rust cross-compilation seems to be pretty marginal even with that fix but at least it's making progress...a lot of the rust cross compilation tooling and meta buildsystem integrations(ie buildroot) are fairly immature still though so I would expect a lot of tooling bugs until things stabilize a bit more.

[jameshilliard]

Looks like pyo3 just had a release(0.14.0) that contains some of the cross compilation bug fixes required for say cross compiling for x86_64 targets from x86_64 hosts, I've updated it in #6154.

[alex]

Based on the latest comments indicating that this seems to be working, I'm going to close this. Thanks to everyone who helped on the upstream work.