Prefer depending on NtDll rather than kernel32 or other higher level DLLs

[daurnimator]

Proposal Clarification

---

"windows" as most people think of it is a stack of

• the nt kernel
• ntdll.dll
• kernel32.dll
• user32.dll

You don't have to use all elements of this stack, and there are situations where you want to avoid or are not able to use kernel32.dll and/or user32.dll.

The Zig standard library should not assume that kernel32.dll and user32.dll are unconditionally available.

[andrewrk]

there are situations where you want to avoid or are not able to use kernel32.dll and/or user32.dll.

What situations are these? Does anyone know?

This is an important proposal if we have some use cases to guide it.

[suirad]

Some background for anyone not familiar: ntdll.dll is the only constant that is required for a windows userland process, as its what does the syscalls. kernel32.dll mostly just wraps ntdll and provides what is colloquially known as Winapi. user32 is essentially the kernel32 of Windows gui stuff.
The windows userland stack ends up looking like:
winlibc (aka fopen()) -> kernel32 (aka CreateFileA()) -> ntdll (aka NtCreateFile()).

---

A couple examples of when some of these may not be available/wanted:

• In Windows Kernel/Driver development those kernel32.dll, ntdll.dll and user32.dll aren't available.
• In some position independent code that is loaded/unpacked/de-obfuscated from elsewhere

---

I would also like to propose having a way to tell what version of windows is being used. Something like builtin.windows_xp / builtin.windows_vista because each changes how some winapi are expressed.

An example of how this would be immediately useful is in creating std.mutex where windows xp and above has CriticalSections available to use, windows vista+ enables use of SRWLock and windows 8+ has WaitOnAddress available, which is the closest thing to a futex windows has. This would result in the mutex itself resolving which of these would be best to use at compile time rather than at runtime.

[jayschwa]

Windows XP

Microsoft stopped maintaining XP nearly 5 years ago. What is Zig's policy on supporting old operating systems?

[andrewrk]

What is Zig's policy on supporting old operating systems?

I don't think it's relevant to the language. But the standard library does not support Windows XP, since it's not supported by Microsoft.

However anyone is free of course to create a third party package for interfacing with Windows XP. And note that much of the standard library has no OS dependencies, and thanks to lazy analysis, you can therefore use the parts of the standard library that have no OS dependencies even when targeting an unsupported OS.

[PavelVozenilek]

There's one very specific use case, software hardened against tampering, where people "include" these standard DLL's (their code sections) inside the executable.

However, this (and driver development) is something very rare, and there's always present danger of feature creep, under-documentation, untested/untestable functionality.

[andrew-boyarshin]

Proposal

OS

• nt: kernel-mode/drivers
• winnative: native images (like smss.exe, csrss.exe, autochk.exe), user-mode with NTDLL only
• win32: user-mode Win32 subsystem images, with guaranteed existence of NTDLL, KERNEL32 and KERNELBASE (since Win10).
• windows => win32

libc

• msvc: Windows-only
• ros: ReactOS-based
• gnu: MinGW-w64
• wine (with wine-staging)

Compatibility

libc might impose additional dependencies.

• wine - msvcrt depends on KERNEL32, so supported only with win32 OS
• ros - full support
• gnu - not sure about nt and winnative

[andrewrk]

I would also like to propose having a way to tell what version of windows is being used. Something like builtin.windows_xp / builtin.windows_vista because each changes how some winapi are expressed.

This is #1907.

---

I'm accepting this proposal, and to provide further clarification of what such an acceptance looks like, it means:

When the standard library needs to interact with the Windows kernel, if there is ntdll API which provides the necessary components to perform the task correctly, then that is the preferred API to use. Generally, if another DLL such as kernel32 is wrapping ntdll functionality, Zig std lib should prefer NtDll directly.

Once we shave down the non-NtDll dependencies in the std lib, let's see what we have left and re-evaluate from there.

One strategy that can be used to find this information out is ProcMon. ProcMon can reveal when a higher level DLL is calling into a lower level one.

[daurnimator]

Generally, if another DLL such as kernel32 is wrapping ntdll functionality, Zig std lib should prefer NtDll directly.

To me this is like saying that on linux, we should prefer the raw syscall over libc functionality: should we be doing that too?

Also, I'm don't think all ntdll functions work correctly under wine; so that might make things harder for e.g. developers on linux to test.

[andrewrk]

To me this is like saying that on linux, we should prefer the raw syscall over libc functionality: should we be doing that too?

In some places this is true, for example with futexes. But generally, Zig users have a choice of which layer to target, by choosing whether to link libc. When linking libc, it makes sense to target the libc layer, for a few reasons:

• saving code size in some cases, since the C code must be linked anyway
• necessary for features such as glibc nss to work
• linux users might expect to be able to use LD_PRELOAD to override libc functions. This is how valgrind integrates with std.heap.c_allocator, for example.

However, not targeting the libc layer is also a primary use case, when not linking libc.

Also, I'm don't think all ntdll functions work correctly under wine; so that might make things harder for e.g. developers on linux to test.

The Zig project has already managed to get one bug fix upstreamed into Wine. Let's start with the optimistic attitude of improving the open source communities around us, before we give up and make compromises.

Now, if there are reasons to target kernel32 or other higher level DLLs, let's hear those reasons out. If they are compelling enough, then it probably makes sense to add an additional target configuration option, which is available to observe in std.builtin, and the std lib can decide which layer to prefer. But I'm not yet convinced this is desirable.

[adontz]

I am sorry, if I'll sound a bit harsh, but I have found a few false statements in this thread.

Linking to ntdll instead of kernel32 (as I see, now it is a weird mix) provides no benefits. ntdll is available in kernel mode and user mode, but exports different set of functions for different modes. For kernel mode, i.e. from driver, one will call ZwCreateFile and not NtCreateFile, so exporting user mode ntdll functions for means of driver development is simply a mistake. Such std will not be more useful, useful in more scenarios or something liek that. It's still user-mode only regular application case. Not a kernel driver, not early boot service, nothing like that.

kernel32 is always available to user mode application, it is loaded into process address space even if not linked at all. There are no reasons to avoid using kernel32, because it is always available. kernel32 is not a simple wrapper, function signatures are quite different and calls do not always map one to one. Those who link directly to ntdll will have to reimplement all kinds of argument conversion logic, or deal with all kinds of weird behavior.

Also, while Microsoft is champion of backwards compatibility (no irony, they are great), reading documentation gives us pretty straightforward directions

https://docs.microsoft.com/en-us/windows/win32/api/winternl/nf-winternl-ntcreatefile

Note Before using this function, please read Calling Internal APIs.

https://docs.microsoft.com/en-us/windows/win32/devnotes/calling-internal-apis

The functions and structures in Winternl.h are internal to the operating system and subject to change from one release of Windows to the next, and possibly even between service packs for each release. To maintain the compatibility of your application, you should use the equivalent public functions instead. Further information is available in the header file, Winternl.h, and the documentation for each function.

I do not know how can it be more clear that Nt-prefixed functions should not be used by regular user mode applications. Even if signatures match, parameter meaning or set of valid values may change. Drivers will use different (Zw-prefixed) functions anyway. So there are literally no reasons to avoid kernel32.

Also, paths are not compatible and it's not about maximum allowed length or some fancy prefix. "\Device\Harddisk0" is a valid NT path, but not a valid Win32 path with or without any prefix. There are many obvious and even more subtle differences between NT and Win32 behavior, so user mode std depending on NT calls is a really weird choice.

I can't remember a single language standard library (C, rust, D, .Net) to use these functions, and I've dissected many of them.

[andrew-boyarshin]

@adontz

1. Kernel mode can also call Nt* versions. Moreover, it is preferred to when data comes from user mode and cannot be trusted.
2. NTDLL is not used in kernel mode, because NTOSKRNL provides most of the NTDLL's API. Technically there is a common part (RTL) linked into both, difference part (for the function with Nt* name NTDLL usually does syscall, NTOSKRNL actually implements the function), plus a vast number of unique API's in both.
3. KERNEL32 and KERNELBASE are not present in Native Executable image scenario, although it's hard to deny that scenario is rare.
4. About DOS and NT paths: it depends on the function really. Some NTDLL functions expect DOS paths and convert them to NT paths internally before doing syscall, some expect already converted NT path.

Honestly, the proposal I've written might be good, but I've already decided against using Zig for low-level Windows development, so I don't really care if Zig stays user-mode only.

upd:
5. .NET depended on a lot of internal functions around the 3.0-3.5 timeframe. Later they migrated to stable APIs, or, where applicable, just started to bring the code from Windows into their own and adapting to their needs.

[daurnimator]

kernel32 is always available to user mode application, it is loaded into process address space even if not linked at all

Ah, appears they added that in XP for the windows subsystem. Most of the books on windows internals are from the Win 2000 era :(
Still, writing native subsystem programs is useful and interesting. midipix has a few examples.

kernel32 is not a simple wrapper, function signatures are quite different and calls do not always map one to one

That's sort of the issue: lots of kernel32 wrappers don't expose the full functionality of the NT API. From the basic stuff like missing flags, to different concepts of paths (nt paths are not null terminated, but length delimited resulting in end users needing tools like RegDelNull) to just plain missing functions (e.g. did you know nt supports fork... but kernel32 doesn't).

Also, while Microsoft is champion of backwards compatibility (no irony, they are great), reading documentation gives us pretty straightforward directions

https://docs.microsoft.com/en-us/windows/win32/api/winternl/nf-winternl-ntcreatefile

Note Before using this function, please read Calling Internal APIs.

https://docs.microsoft.com/en-us/windows/win32/devnotes/calling-internal-apis

The functions and structures in Winternl.h are internal to the operating system and subject to change from one release of Windows to the next, and possibly even between service packs for each release. To maintain the compatibility of your application, you should use the equivalent public functions instead. Further information is available in the header file, Winternl.h, and the documentation for each function.

I do not know how can it be more clear that Nt-prefixed functions should not be used by regular user mode applications. Even if signatures match, parameter meaning or set of valid values may change.

There have only been 2 notable function changes I know of since windows 2000, and they have coincided with major releases: despite Microsoft's statements, they are quite stable. What is unstable is the actual syscall numbers, which change even between monthly windows updates.

[andrewrk]

@adontz, I appreciate your input.

Linking to ntdll instead of kernel32 (as I see, now it is a weird mix) provides no benefits.

There is one very real benefit: the APIs are more powerful. For example, NtCreateFile has the ability to open a sub-directory path based on an open directory handle. The kernel32 APIs do not expose this. It has meaningful consequences in terms of avoiding file system race conditions.

When I first started working on Windows implementations of cross platform abstractions, I took the Microsoft documentation very seriously. But then I found out that pretty much every standard library uses SystemFunction036 from advapi32.dll for getting cryptographically secure random bytes, while Microsoft's docs says to use CryptGenRandom. The docs for CryptGenRandom now say

Important This API is deprecated. New and existing software should start using Cryptography Next Generation APIs. Microsoft may remove this API in future releases.

What I learned is that it's more important to consider ABIs rather than APIs.

Microsoft won't break RtlGenRandom and they won't break NtCreateFile. If they do, they have a lot more programs to worry about breaking than Zig binaries.

The bottom line here is that NtDll is lower-level ABIs to the kernel, and kernel32 is higher level code that wraps NtDll. This is clear when you look at calls in ProcMon. And frankly, Zig's open-source code that wraps NtDll is more robust and reliable than Microsoft's cruft inside kernel32.dll. I don't remember the specifics, but off the top of my head, there are functions in kernel32 that wrap NtDll functions that allocate heap memory, whereas the zig code that wraps NtDll functions does not. That can be the difference between deterministic latency or nondeterministic latency.

At this point I'm convinced that NtDll is an entirely appropriate abstraction layer for windows applications to target. My mind is open to counter-examples, but calls to NtDll functions have been in the zig std lib for quite some time now, and I'm not aware of a single issue it has caused. On the other hand, it has allowed us to unify the std.fs API across POSIX and Windows, since with NtCreateFile they both support operating on an open directory handle.

@andrew-boyarshin,

I've already decided against using Zig for low-level Windows development

Is there anything the Zig project can learn from your decision?

[andrew-boyarshin]

@andrewrk note that the following is purely about Windows kernel drivers, ReactOS kernel and NTDLL, user-mode Zig is cool, I play with it (but yet to use it in any real project).

1. I'm not sure Zig would allow me to develop kernel driver code (standard library)
2. I'm not sure Zig would allow me to throw and handle SEH exceptions properly (ExceptionCode, user buffer probing, exception filters)
3. I'm not sure about calling conventions being compatible, but that might well be false. I've never tried Clang/LLVM for low-level code, and Zig is 2 layers where mistakes can exist. Clang+LLVM is 1.5 layer (Clang is very coupled with LLVM, so it's difficult to count them separately). I just feel MSVC is a safer (as in error-proof) bet (single layer monolith used by Windows team themselves).
4. I'm not sure I can rely on Zig features being safe (as in secure) and fast (as fast as they can theoretically be when implemented in C using kernel APIs to the fullest). I'm sure writing C-like Zig will be ok, but when it comes to coroutines and complete kernel asynchrony (can they survive preemption? how will they play with IRQL?)...

I feel like most of these issues can be eliminated by providing a good sample of e.g. FS/registry filter, and showcase what works and what does not.

upd:
5. Status codes. Zig has an awesome concept of static errors and obligatory explicit handling. Unfortunately it's impossible to use that since even MS Docs don't specify all possible NTSTATUS codes, not to mention non-documented functions. So, Zig offers no advantage in this respect.

[daurnimator]

I feel like most of these issues can be eliminated by providing a good sample of e.g. FS/registry filter, and showcase what works and what does not.

Is this something you'd like to take on @andrew-boyarshin ?

[andrew-boyarshin]

@daurnimator not in the foreseeable future. Certainly not in the next 5 months.

[expikr]

Should page_allocator on Windows perhaps use the various Nt***VirtualMemory functions instead of kernel32.Virtual*** ones currently being used?

[ChanceNCounter]

This appears to have been in progress for six years. Every few months somebody posts it to Reddit or tweets it or etc, and it gets a lot of laughs, which appears to have led to a couple of testy exchanges over the years.

My question is, given the number of times that this issue has made this project a laughing stock, has anyone on the project actually tried reaching out to a Microsoft contact for feedback on what the rest of the internet clearly thinks is an unfathomably bizarre choice?

[alexrp]

Responding to this point specifically:

This appears to have been in progress for six years. Every few months somebody posts it to Reddit or tweets it or etc, and it gets a lot of laughs, which appears to have led to a couple of testy exchanges over the years.

It is easy, especially on the internet, to settle for the status quo and laugh at any attempt to change it. One of the defining features of Zig - not just as a project, but also as a community - is the willingness to challenge conventional wisdom to push the state of computing forward. That doesn't always pan out, but it seems to me that it works more often than not. For an easy example, consider Zig's glibc cross-compilation support: Just 5-10 years ago, the mere idea of this would have gotten you laughed out of a room - it's a mess, it's just clearly too impractical. Yet here we are. People were similarly skeptical about the viability of Zig rolling its own compiler backends and moving towards LLVM independence, yet we have multiple backends being actively worked on, with the x86-64 backend in particular being close to usability.

Which is why, IMO, this:

what the rest of the internet clearly thinks is an unfathomably bizarre choice?

really doesn't carry much weight. Technical arguments are more than welcome, but what "the internet" thinks often turns out to be misguided or wrong.

[ChanceNCounter]

I am surprised you regard the aforementioned impressive technical undertakings as equivalent to the decision to target a lower level of Windows than other systems languages think is reasonable or wise.

[dongle-the-gadget]

challenge conventional wisdom to push the state of computing forwardr

Trying to change the state of affairs of things you don’t control (which is not the case for all examples you have given, since you have direct control over the Zig toolchain, but not Windows) is not a wise idea.

I heavily doubt that Microsoft would lean down on providing their internal APIs to other consumers. Instead it’s simpler for them to simply block execution of popular Zig applications if the NT functions change in a breaking manner.

[ChanceNCounter]

Well, you can continue to doubt what Microsoft would suggest, or you could heed my original suggestion (rather than defensively responding to everything else I wrote) and ask Microsoft. Six years, nobody seems to have thought of that. I tried to make that clear as politely as I was prepared to interact with this project. Good luck to you all.

[alexrp]

Trying to change the state of affairs of things you don’t control (which is not the case for all examples you have given, since you have direct control over the Zig toolchain, but not Windows) is not a wise idea.

I heavily doubt that Microsoft would lean down on providing their internal APIs to other consumers. Instead it’s simpler for them to simply block execution of popular Zig applications if the NT functions change in a breaking manner.

I don't think anyone here has suggested that we can apply pressure on Microsoft to do anything.

What has been suggested is that we can rely on functionality exported from ntdll.dll that, in practice, is unlikely to ever change for the lifetime of Windows. It's not just for fun that projects like Wine and ReactOS try to faithfully replicate the ntdll.dll API surface and functionality; enough real applications rely on these details for it to matter.

Microsoft may prefer not to acknowledge this officially, but they absolutely know it, and they know that backwards compatibility is the killer feature that has kept Windows on top as essentially the default consumer OS. It's also worth noting that Microsoft themselves have slowly but surely started documenting some of the more popular ntdll.dll functions because, again, they do know what goes on in practice.

Also, there's literally no evidence to suggest that the Zig project wouldn't change course if this approach turns out to truly be unworkable.

---

I am surprised you regard the aforementioned impressive technical undertakings as equivalent to the decision to target a lower level of Windows than other systems languages think is reasonable or wise.

I don't. The point I was making is that it's not generally useful to measure the difficulty of an engineering challenge based on the internet hive mind's opinion. The internet is full of doomers who are wrong more often than they're right - especially so in tech communities.

(Also, a lot of what we're doing with glibc cross-compilation is relying on implementation details of glibc; it's actually not too dissimilar. But again, not actually the point.)

Well, you can continue to doubt what Microsoft would suggest, or you could heed my original suggestion and ask Microsoft. [...] Six years, nobody seems to have thought of that.

As someone who worked at Microsoft, I can tell you that, even if you work there, it can be nearly impossible to get through to the relevant people unless you specifically know who they are. So I think it would be helpful to suggest the right point of contact here; gesturing in the general direction of a huge multinational corporation doesn't arm us with any information that we didn't already have.

(rather than defensively responding to everything else I wrote)

I tried to make that clear as politely as I was prepared to interact with this project.

I would encourage you to put yourself in the shoes of a maintainer, contributor, or even just ordinary community member of a large OSS project, then read your original comment back to yourself, and see if you truly believe that it comes off as polite. I can personally think of multiple ways to rewrite that comment in a considerably more constructive (and non-inflammatory) tone. I think my response was actually quite level-headed and, dare I say, polite.

And to be clear, I'm just one contributor of many; I don't represent the project. That's why I made it clear that I was only responding to that point.

[teo-tsirpanis]

While I believe it's highly unlikely that Microsoft will break even its undocumented ntdll APIs, I think this this is the wrong question to ask.

It's not bad to use NT APIs to do things not possible with Win32 APIs, but defaulting to NT APIs and avoiding as many Win32 APIs as possible should require a very strong technical justification which IMHO am not seeing after reading the whole discussion; on the contrary there are reported problems like lack of WOW64 redirection.

If Zig plans to target the underlying NT system, I believe it should be as a new platform distinct from Windows.