Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build wasm32-wasi without WASIX #95174

Closed
tiran opened this issue Jul 23, 2022 · 0 comments
Closed

Build wasm32-wasi without WASIX #95174

tiran opened this issue Jul 23, 2022 · 0 comments
Labels
3.11 only security fixes 3.12 bugs and security fixes OS-wasi type-feature A feature request or enhancement

Comments

@tiran
Copy link
Member

tiran commented Jul 23, 2022

Feature or enhancement

I propose to remove dependency on WASIX. Instead Python should omit missing features or provide stubs when a feature cannot be disabled.

wasm32-wasi builds are depending on WASIX. WASIX provides stubs for POSIX functions that are not part of current WASI API, for example several socket functions and pthreads. The stubs are no-ops or return an error. WASIX was the simplest approach to boot strap WASI support.

Pitch

Removal of WASIX has several benefits for us:

  • It is one less dependency that must be downloaded, compiled, installed, and injected into the build environment with CFLAGS and LDFLAGS.
  • Missing features and functions are easier to detect. With WASIX an unsupported feature raises an exception at runtime. Without WASIX the function is not wrapped and missing features can be detected with hasattr().
  • Feature detection and stubs are useful for people that try to port Python to other operating systems with limited POSIX compatibility.

Missing features in WASI that need attention:

  • pthread API, these will be stubbed with static inline functions
  • netdb functions getservbyname getservbyport gethostbyname gethostaddr getprotobyname. They will be detected by configure.ac and handled with #ifdef HAVE_ checks.
  • socket functions inet_ntoa inet_pton getpeername getsockname bind connect listen recvfrom sendto setsockopt socket. They will be handled like missing netdb functions.
  • some missing constants like POLLPRIO and ESHUTDOWN

WASI does not support DNS lookup and NSS. Sockets are limited to read/write/close from a socket FD or accept call.

@tiran tiran added type-feature A feature request or enhancement 3.11 only security fixes 3.12 bugs and security fixes labels Jul 23, 2022
tiran added a commit to tiran/cpython that referenced this issue Jul 23, 2022
WASI has no process or netdb API. The code will fail with AttributeError
when we remove the stubs from WASIX.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jul 24, 2022
miss-islington added a commit that referenced this issue Jul 24, 2022
(cherry picked from commit 8184f0f)

Co-authored-by: Christian Heimes <[email protected]>
tiran added a commit to tiran/cpython that referenced this issue Jul 25, 2022
WASI can only create sockets from an existing file descriptor. Hostname
looks are not available.

- check for missing ``sys/socket.h`` functions
- handle missing ``netdb.h`` functions
tiran added a commit to tiran/cpython that referenced this issue Jul 25, 2022
- check for ``dup()`` libc function
- handle missing ``F_DUPFD`` in ``dup2()`` replacement function
- add workaround for WASI libc bug in MSG_TRUNC
- ESHUTDOWN is missing, use EPIPE instead
- POLLPRI is missing, define as 0 (no-op)
tiran added a commit that referenced this issue Jul 26, 2022
- check for ``dup()`` libc function
- handle missing ``F_DUPFD`` in ``dup2()`` replacement function
- add workaround for WASI libc bug in MSG_TRUNC
- ESHUTDOWN is missing, use EPIPE instead
- POLLPRI is missing, define as 0 (no-op)
tiran added a commit to tiran/cpython that referenced this issue Jul 27, 2022
tiran added a commit to tiran/cpython that referenced this issue Jul 27, 2022
wasi-env now sets WASIX flags. This allows us to control all build
parameter for wasm32-wasi buildbot from CPython repository.

Also export and improve SYSROOT parameter.
tiran added a commit that referenced this issue Jul 27, 2022
wasi-env now sets WASIX flags. This allows us to control all build
parameter for wasm32-wasi buildbot from CPython repository.

Also export and improve SYSROOT parameter.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jul 27, 2022
wasi-env now sets WASIX flags. This allows us to control all build
parameter for wasm32-wasi buildbot from CPython repository.

Also export and improve SYSROOT parameter.
(cherry picked from commit 2833f37)

Co-authored-by: Christian Heimes <[email protected]>
tiran added a commit to tiran/cpython that referenced this issue Jul 27, 2022
Neither WASI nor default builds with Emscripten have support for
threading. While Emscripten comes with pthread stubs, WASI-SDK 16 and
earlier are lacking stubs.

Python initially used WASIX stubs to provide pthread stubs. This
changeset introduces stubs modelled after Emscripten's stubs.

Emscripten without pthread emulation now has ``sys.thread_info.name``
``pthread-stubs``, too.
miss-islington added a commit that referenced this issue Jul 27, 2022
wasi-env now sets WASIX flags. This allows us to control all build
parameter for wasm32-wasi buildbot from CPython repository.

Also export and improve SYSROOT parameter.
(cherry picked from commit 2833f37)

Co-authored-by: Christian Heimes <[email protected]>
tiran added a commit that referenced this issue Jul 27, 2022
tiran added a commit to tiran/cpython that referenced this issue Jul 31, 2022
…thonGH-95229)

- check for ``dup()`` libc function
- handle missing ``F_DUPFD`` in ``dup2()`` replacement function
- add workaround for WASI libc bug in MSG_TRUNC
- ESHUTDOWN is missing, use EPIPE instead
- POLLPRI is missing, define as 0 (no-op).
(cherry picked from commit 0d35a59)

Co-authored-by: Christian Heimes <[email protected]>
pablogsal pushed a commit that referenced this issue Jul 31, 2022
tiran added a commit that referenced this issue Jul 31, 2022
…95308)

Co-authored-by: Christian Heimes <[email protected]>
Co-authored-by: Pablo Galindo Salgado <[email protected]>
tiran added a commit to tiran/cpython that referenced this issue Jul 31, 2022
Co-authored-by: Brett Cannon <[email protected]>.
(cherry picked from commit 0fe645d)

Co-authored-by: Christian Heimes <[email protected]>
pablogsal pushed a commit that referenced this issue Aug 1, 2022
Co-authored-by: Brett Cannon <[email protected]>.
(cherry picked from commit 0fe645d)

Co-authored-by: Christian Heimes <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.11 only security fixes 3.12 bugs and security fixes OS-wasi type-feature A feature request or enhancement
Projects
No open projects
Status: Done
Development

No branches or pull requests

2 participants