SGX signal handling flows

[dimakuv]

Current SGX signal handling flows

Below explanations apply to Gramine v1.7 and (to the best of my knowledge) were like this already in v1.0.

Synchronous signals (SIGFPE, SIGSEGV, SIGBUS, SIGILL)

Synchronous signals are SIGFPE, SIGSEGV, SIGBUS and SIGILL. They are mapped to Gramine-internal signal names PAL_EVENT_ARITHMETIC_ERROR, PAL_EVENT_MEMFAULT, PAL_EVENT_ILLEGAL. See this code snippet.

These signals are generated by the Linux kernel as an immediate response to the hardware exception that happens inside the SGX enclave. The flow is like this (on the example of Divide-By-Zero exception):

Some notes:

1. On a hardware exception, SGX enclave thread execution is interrupted and the currently executing context is automatically saved (in the SSA0 frame of the enclave thread). This hardware flow is called Asynchronous Enclave eXit (AEX). As per x86-64 arch, AEX passes control to the Asynchronous Exit Pointer (AEP). AEP typically ends in the ERESUME instruction (more specifically, enclu instruction with RAX register already set to the ERESUME value). AEP in Gramine looks like this.
2. While in untrusted-runtime signal handler, Gramine blocks all asynchronous signals. This means that nested signals are impossible (if another synchronous signal arrives while this thread is in the signal handler, it is treated as a fatal error). See code snippet here.
3. Entering the enclave for the stage-1 handler involves rather complex assembly, but start looking from here, here and here.
4. Stage-1 handler is implemented in ~100 instructions of pure x86-64 assembly. This handler executes in SSA1 frame. This handler does not upcall into LibOS/application but instead prepares SSA0 frame for the real (stage-2) handler: the interrupted enclave context in SSA0 is copied on a Gramine-specific stack and the stage-2 signal handling context is overwritten into SSA0 frame. See this corresponding code.
5. Exiting the enclave's stage-1 handler is trivial. See code here and here. After handle_sync_signal() function returns, the signal handling context is over and Linux resumes normal Gramine execution (jumping to AEP).
6. ERESUME resumes execution of the enclave by restoring the context from the SSA0 frame. Since SSA0 frame was rewired to point to the stage-2 handler in note 4, the enclave thread starts executing _PalExceptionHandler(). See this function here.

Asynchronous signals (SIGTERM, SIGCONT)

Asynchronous signals are SIGTERM and SIGCONT. They are mapped to Gramine-internal signal names PAL_EVENT_QUIT and PAL_EVENT_INTERRUPTED. See this code snippet.

These signals are generated by the Linux kernel as a (delayed) response to the send-signal-to-Gramine-process request from another, untrusted process like a Bash terminal. There are two possible flows.

Enclave thread was executing and got interrupted (AEX event)

This can happen in the case of e.g. a timer interrupt. Enclave thread is interrupted and automatically saves its state in the SSA0 frame. The Linux kernel decides that this is a good opportunity to inject a pending signal, e.g. SIGTERM.

Notes 1-6 from the previous section generally apply. Some additional notes:

1. The relevant async signal handling code in untrusted runtime of Gramine is here.
2. You may notice that this case has exact same flows as the case of synchronous signals.

Enclave thread exited for an OCALL

An asynchronous signal may arrive while the enclave thread exited for an OCALL, and the untrusted runtime is executing the corresponding host system call. This is especially relevant if the enclave thread exited for a blocking host-syscall operation, such as a blocking read() on a network socket.

Some notes:

1. In this case, there is no SSA0 frame. Instead, Gramine saved the enclave-thread context on the Gramine-specific stack right before EEXITing. Thus, we cannot use the same flows as in previous sections.
2. Instead, untrusted runtime async signal handler simply rewires the context-to-be-restored of the OCALL return path. In particular, the RSI register is set to the arrived signal number. RSI will be examined inside of the SGX enclave, to learn that there is a need to execute stage-2 signal handler before returning from the OCALL. The relevant untrusted runtime code is here.
3. The relevant OCALL return path in untrusted runtime is written in assembly and can be checked here.
4. The relevant OCALL return path inside the enclave is written in assembly and can be checked here.
5. The stage-2 signal handler is the same _PalExceptionHandler() as used in other flows, see here.
6. The enclave-thread context on the Gramine-specific stack, mentioned in note 1, is not touched or modified in any way. Instead, it is directly given to _PalExceptionHandler() as the context-to-be-restored argument, see here and here.
7. In the current implementation, in case of several async signals arriving at the same time, only the last arriving signal is delivered into the enclave and the other signals are ignored. It is a feature not a bug. The reasoning is that there are only two supported async signals in Gramine: SIGTERM and SIGCONT. SIGTERM can be delivered to the enclave application only once anyway, and SIGCONT is an "optional hint" signal so can be dropped. See the comment and implementation here.
8. Not really relevant to this topic, but if you are interested what these checks in the code snippet mean, the answer lies in this code. Basically, if an async signal arrives right-before or during executing the host syscall (the syscall instruction), the syscall return value is rewired to -EINTR and the RIP is rewired to jump over the syscall and immediately return to the enclave.

[dimakuv]

Proposal for AEX-Notify-amenable SGX signal handling flows

Previous post explained the current SGX signal handling flows. Unfortunately, these flows are not amenable to AEX-Notify hardware feature.

The issue with the current Gramine flows is that they assume a strong coupling between the untrusted-runtime context and trusted-enclave context. In particular:

• Stage-1 trusted-enclave handler (in SSA 1) always executes in the signal-handling context of the untrusted runtime.
• Stage-2 trusted-enclave handler (in SSA 0) always executes in normal context of the untrusted runtime.

AEX-Notify breaks the strong coupling of contexts: AEX-Notify works in conjunction with the EDECCSAA instruction, that atomically switches the context from SSA 1 to SSA 0, without exiting the enclave. This implies that if the signal-handling context of the untrusted runtime would EENTER (for stage-1 handler), then the AEX-Notify-enabled flows inside the enclave would at some point execute EDECCSSA, execute the stage-2 handler, and continue normal execution of the enclave; all without exiting the enclave. Thus, the untrusted runtime becomes "stuck" in signal-handling context, which is plain wrong.

Therefore, to be amenable to AEX-Notify, the signal handling flows in Gramine must be first modified: the stage-1 handler must execute in normal context of the untrusted runtime.

Note that the described flows are not directly related to AEX-Notify. These flows also work without AEX-Notify. The point of this post is to show a preliminary step of changing the current design, so that in the next step AEX-Notify flows can be applied on top.

Synchronous signals (SIGFPE, SIGSEGV, SIGBUS, SIGILL)

The proposed flow is like this (on the example of Divide-By-Zero exception):

Some notes:

1. Main difference: instead of immediately delivering the signal into the enclave, the untrusted runtime's signal handler memorizes the signal in a new thread-local variable sync_signal and returns. When Linux returns back to normal context from the signal handler, it jumps to the AEP landing pad, which is augmented with a new logic: checking whether there is a sync signal pending (variable sync_signal is not zero). If there is a pending sync signal, the new AEP logic performs EENTER, so that stage-1 handler executes inside the enclave. After the stage-1 handler is done, the AEP logic finalizes with ERESUME as usual. At this point the flow is the same as previously: the enclave is resumed at stage-2 handler.
2. Main caveat: now stage-1 handler execution can be interrupted by new signals arriving, because stage-1 handler executes in normal context of untrusted runtime. We argue that this is benign:
   • In benign execution, sync signal cannot arrive, because the stage-1 handler is specially crafted to never trigger any exceptions (that would translate into sync signals).
   • In benign execution, async signal can arrive. In this case, an async signal flow is triggered (see next diagram), and the AEP landing pad after the async signal will try to EENTER, but since there's already SSA 1 executing inside the enclave and SSA 2 is forbidden by SGX hardware, this (nested) EENTER will raise a #GP fault which translates into SIGSEGV and is delivered to the untrusted runtime's signal handler. We augment the SIGSEGV signal handler to catch this particular case and ignore it: the async signal is anyhow memorized in tcb->async_signal variable and cannot be delivered right now. This async signal will be delivered on a next AEX.
   • In malicious execution, the host can inject any sync/async signals at any time. This is fine, since Gramine enclave doesn't care what happens in untrusted runtime; the stage-1 and stage-2 handlers are hardened against any unexpected/spurious signals. Thus, no matter how untrusted runtime is restructured, this cannot break security of the in-enclave logic.
3. The diagram depicts XCHG to highlight that tcb->sync_signal must be atomically read-and-modified: the current value is read into a local variable, and a "no signal" value is atomically placed in this variable. The local variable is then used as an argument for EENTER (signal number). This atomic exchange operation is not strictly required for sync signals, but it is required for async signals (see below), so we show it on this diagram too, for uniformity.
4. Performance must be fine. The AEP landing pad now has a bit more logic, but in the no-signal case, it is ~5 instructions. So we don't expect any performance hit from this refactoring.

Asynchronous signals (SIGTERM, SIGCONT)

As in the current flows in the previous post, async signals are generated by the Linux kernel as a (delayed) response to the send-signal-to-Gramine-process request from another, untrusted process like a Bash terminal. There are two possible flows.

Enclave thread was executing and got interrupted (AEX event)

The proposed flow is like this (on the example of SIGTERM):

Notes 1-4 from the previous section generally apply. Some additional notes:

1. Async signal is memorized in a separate variable tcb->async_signal. Technically, since there can be many async signals arriving at the same time, this variable should have been a queue of pending async signals, not a single slot. Luckily, Gramine has a very restricted async-signal model: only two signals are recognized, SIGCONT (just a hint to continue execution) and SIGTERM (is injected only once and is sticky). Therefore, it's enough to have a single slot which is populated with SIGCONT or SIGTERM, with SIGTERM having a higher priority.
2. It is possible that injecting the async signal via EENTER doesn't succeed (EENTER fails with #GP, see note 2 in previous section). In this case, this async signal must be returned into the tcb->async_signal variable, so that it will be triggered on some next AEX. This "return" must be atomic, just like all other operations on this variable.
3. Alternative but not working solution: We thought about a different, more structured approach to async signals. We could let Gramine logic inside the enclave decide when to consume async signals, instead of injecting async signals forcibly. This would get rid of the problems like "#GP on nested EENTER". However, this may lead to hangs -- an application inside Gramine may run an infinite loop without any syscalls, thus Gramine logic would never trigger, and a SIGTERM would never be delivered. This alternative solution doesn't support this case, which we consider realistic.

Enclave thread exited for an OCALL

No changes required to this flow. That's because the flow doesn't involve any AEX events.

[scottconstable]

Thank you so much for the detailed design, Dmitrii. I have one question. CSSA >= NSSA is just one among several reasons that EENTER can trigger #GP(0). Have you considered all of the other reasons?

I want to make sure that we don't introduce the following scenario. In the current signal handling design, any #GP(0) triggered by EENTER should kill the application (right?). With the new design, a #GP(0) triggered by EENTER would not kill the enclave (but only if EENTER is delivering an asynchronous signal, if I understand correctly). Suppose that Gramine receives and asynchronous signal and delivers it to the enclave and EENTER triggers #GP(0). Gramine will assume that CSSA >= NSSA and try again later, but if the #GP(0) had a different root cause then I think Gramine would enter an infinite loop.

[dimakuv]

CSSA >= NSSA is just one among several reasons that EENTER can trigger #GP(0). Have you considered all of the other reasons?

Based on my analysis of other reasons for #GP (https://www.felixcloutier.com/x86/eenter), there's no other legitimate reason that can happen in Gramine flows.

I want to make sure that we don't introduce the following scenario. In the current signal handling design, any #GP(0) triggered by EENTER should kill the application (right?). With the new design, a #GP(0) triggered by EENTER would not kill the enclave (but only if EENTER is delivering an asynchronous signal, if I understand correctly). Suppose that Gramine receives and asynchronous signal and delivers it to the enclave and EENTER triggers #GP(0). Gramine will assume that CSSA >= NSSA and try again later, but if the #GP(0) had a different root cause then I think Gramine would enter an infinite loop.

Yes, you're right that if EENTER (that is used to inject an async signal) is failing for other reasons other than CSSA >= NSSA, then Gramine may enter an infinite loop.

First of all, we will make sure to single out that particular use of EENTER. The code will only ignore #GP if it happened on the EENTER for injecting the async signal. (You can see one way to single out such EENTER in the current state of #1531). I.e., if a #GP happens on EENTER used for ECALL or on EENTER after OCALL is done, then the #GP will result in killing the application (not in a hang).

TLDR: On a benign host, #GP on that EENTER should happen only in the case of such a nested async signal.

[scottconstable]

First of all, we will make sure to single out that particular use of EENTER. The code will only ignore #GP if it happened on the EENTER for injecting the async signal. (You can see one way to single out such EENTER in the current state of #1531). I.e., if a #GP happens on EENTER used for ECALL or on EENTER after OCALL is done, then the #GP will result in killing the application (not in a hang).

Yes, I agree that this particular invocation of EENTER should be treated differently, in the way you described.

TLDR: On a benign host, #GP on that EENTER should happen only in the case of such a nested async signal.

A bug in Gramine could also cause a #GP, for example, if one thread tries to EENTER using the same TCS as another thread that is already running.

[dimakuv]

A bug in Gramine could also cause a #GP, for example, if one thread tries to EENTER using the same TCS as another thread that is already running.

First, bugs in Gramine's untrusted runtime can generally lead to hangs (and led in the past), so that would be just another thing to root cause and fix. Second, this particular bug should be rather trivial to debug. So I wouldn't care much about this potential.

[dimakuv]

Proposal for AEX-Notify-enabled SGX signal handling flows

Previous post explained the AEX-Notify-amenable signal handling flows. That post can be considered a preparation for the actual AEX-Notify flows.

If the signal handling flows are implemented as in the previous post, then adding AEX-Notify flows is trivial: only the EDECCSSA instruction and its associated "context switching" must be implemented. Interestingly, the untrusted runtime logic does not need to be modified at all.

Synchronous signals (SIGFPE, SIGSEGV, SIGBUS, SIGILL)

The proposed flow is like this (on the example of Divide-By-Zero exception):

Some notes:

1. Main difference: instead of EEXIT from SSA 1 and then ERESUME into SSA 0, we perform EDECCSSA. This has a side effect of better performance (no enclave exit required).
2. All notes from the previous post "AEX-Notify-amenable flows" are applicable.

Asynchronous signals (SIGTERM, SIGCONT)

As in the current flows in the previous post, async signals are generated by the Linux kernel as a (delayed) response to the send-signal-to-Gramine-process request from another, untrusted process like a Bash terminal. There are two possible flows.

Enclave thread was executing and got interrupted (AEX event)

The proposed flow is like this (on the example of SIGTERM):

Notes are same as above.

Enclave thread exited for an OCALL

No changes required to this flow. That's because the flow doesn't involve any AEX events.

No signals (but AEX-Notify mitigations must execute)

AEX-Notify's main purpose is to execute the signal/exception handler even if there is no signal to be reported to Gramine or the application. The signal/exception handler is not supposed to do any Gramine-specific operations but it only must apply the mitigations and then continue normal execution of the in-enclave app. These mitigations must be triggered after every AEX (even if AEX doesn't result in an app-visible signal), for example, on page faults (#PF hardware exceptions).

Thus, AEX-Notify introduces this new flow of "no signals". In this case, we propose to inject a dummy SIGCONT signal to re-use the existing Gramine flows without any side effects (recall that SIGCONT is really just a hint to Gramine).

The proposed flow is like this (on the example of a page fault):

Some notes:

1. Notice how untrusted runtime's signal handler of Gramine is not called at all. Instead, Linux kernel resolves the #PF internally and resumes execution. Resumed execution starts at the AEP landing pad.
2. AEP landing pad notices there are no real pending signals (tcb->sync_signal == 0 and tcb->async_signal == 0). So for uniformity, AEP logic injects a dummy SIGCONT, just to trigger stage-1 handler inside of the enclave, which will in turn trigger stage-2 handler and finally the mitigations, before passing control to the normal execution of the enclave app.
3. Alternative design: instead of injecting SIGCONT, we could introduce a new "dummy" signal. This would instruct Gramine to not try any signal-handling in LibOS (SIGCONT still needs some handling in LibOS, even if it's dummy), and it would save us some CPU cycles. We currently don't propose this as it would require modifications in the brittle PAL exception categorization logic.
4. Performance overhead of this is unclear, but can be significant. The academic paper mentions anywhere from 0.4% to 58% slowdown.

[scottconstable]

The 58% figure mentioned in the paper is a microbenchmark where the enclave is being interrupted so frequently that it never makes any forward progress. Workloads running on a moderately loaded system might be interrupted between once every 1 million cycles and once every 10 million cycles, in which case the performance overhead would range from roughly 0.4% to 0.04%.

[dimakuv]

In the descriptions above, I used the term "normal context" as a counter-part to the "signal-handling context".

In my new PRs, I am using instead "regular context", as this term seems less ambiguous, and I try to avoid having two synonyms for the same thing. Interestingly, I didn't find any ultimate source of truth which of "normal" vs "regular" terms to use.

[dimakuv]

I submitted the new series of PRs to add AEX-Notify support:

• [PAL/Linux-SGX] AEX-Notify 1/5: Prepare AEX handler to call C code in release mode #2025
• [PAL/Linux-SGX] AEX-Notify 2/5: Inject signals into enclave in regular context #2032
• [PAL/Linux-SGX] AEX-Notify 3/5: Add AEX-Notify enabling code #2034
• [PAL/Linux-SGX] AEX-Notify 4/5: Do not clobber RBX reg in stage-1 signal handler #2036
• [PAL/Linux-SGX] AEX-Notify 5/5: Add AEX-Notify flows in exception handling #2037

This series supersedes #1530 and #1531, refactoring their code, as well as fixing bugs and data races in those ones.

UPDATE 22. October 2024: Rebased this series of PRs to the latest master branch, aka post v1.8 release.