CMake, scikit-build, and Apple Silicon

[YannickJadoul]

After a few hours of wasted time learning opportunities, I managed to strangle some arm64 and universal2 wheels out of my setup. But this was less straightforward than hoped for.

Thing I had to change:

• Added MACOSX_DEPLOYMENT_TARGET to CIBW_ENVIRONMENT. Could be fixed by a solution to set MACOSX_DEPLOYMENT_TARGET for all stages? #563.
• Added CMAKE_OSX_ARCHITECTURES to CIBW_ENVIRONMENT. It is kind of unfortunate that CMake doesn't seem to pick up on ARCHFLAGS (or maybe it would, but it gets overridden by CMAKE_OSX_ARCHITECTURES passed by scikit-learn). We could of course set this environment variable (and I'd be happy with that), but this is also tricky to add something very build-system specific?
  • ARCHFLAGS/CMAKE_OSX_ARCHITECTURES is also harder/impossible to fix for CIBW_BEFORE_ALL in set MACOSX_DEPLOYMENT_TARGET for all stages? #563, btw, since it's dependent on the Python version.
• Teach scikit-build about the universal2 <-> CMAKE_OSX_ARCHITECTURES=x86_64;arm64 link, and teach it to respect _PYTHON_HOST_PLATFORM if that's already set (so, yeah, scikit-build is to blame here, @joerick): YannickJadoul/scikit-build@vs2019...YannickJadoul:vs2019-apple-silicon

@henryiii, how do you judge about the scikit-build changes? They seem reasonably ad-hoc and very brittle, so not I'm wondering what else I missed. Also, worth a PR on scikit-build, according to you?

And what should we do in cibuildwheel? Just some docs?

Main commit with the changes: YannickJadoul/Parselmouth@f3b61e0

See discussion on #579 (comment) thread.

[henryiii]

I'm rather thinking (though I haven't been thinking about it too heavily) that CIBW_ENVIRONMENT should likely affect the entire CIBW process, rather than just the build phase. At least the BEFORE_ALL part; adding it to TEST could cause issues.

I wouldn't add anything specific to CMake, I'd just note it in the docs (and it should be fixed in scikit-build; ideally scikit-build should read and use everything from setuptools).

I'll look at the changes soon. scikit-build is going to need some work I expect.

[YannickJadoul]

I'm rather thinking (though I haven't been thinking about it too heavily) that CIBW_ENVIRONMENT should likely affect the entire CIBW process, rather than just the build phase.

CIBW_ENVIRONMENT is already happening, even for CIBW_BEFORE_ALL. It just so happens that MACOSX_DEPLOYMENT_TARGET gets added separately, but not for CIBW_BEFORE_ALL (though that would make sense to me, I think).

I wouldn't add anything specific to CMake, I'd just note it in the docs (and it should be fixed in scikit-build; ideally scikit-build should read and use everything from setuptools).

Not the easiest solution for me, but I agree that that's the better principle, yes :-)

[henryiii]

I think I'd make that PR to scikit-build, wouldn't hurt, and needs to start somewhere. :)

[YannickJadoul]

Good point! I'd almost forgotten. Done at scikit-build/scikit-build#530.

[henryiii]

I think this is closable now that Scikit-Build 0.12 is out. Everything should work similarly to setuptools now.

[joerick]

Agreed!