[release/5.0] Take SuppressGCTransition into account in IL stub caching

[elinor-fung]

Issue: #46184
Fix in master: #47320

This is a stripped down version of the fix that went into master and the change it depended on: adds the NDIRECTSTUB_FL_SUPPRESSGCTRANSITION flag, updates the P/Invoke stub flags based on SuppressGCTransition, and uses the updated flags in the IL stub cache.

Customer Impact

When this issue results in the runtime suppressing the GC transition for P/Invokes that should not have them suppressed, users may hit GC starvation, data corruption, or runtime termination. When the runtime doesn't suppress the transition when it should, users will miss getting the performance benefits expected from suppressing the transition.

This is difficult to diagnose and does not have a reasonable workaround (only options would be to never use SuppressGCTransition or to re-work P/Invokes such that they do not have the same signature).

Example based on the customer-reported issue:

private class NativeLib
{
    [SuppressGCTransition]
    [DllImport(nameof(NativeLib), CallingConvention = CallingConvention.Cdecl)]
    public static extern bool set_callback(void* callback);

    [DllImport(nameof(NativeLib), CallingConvention = CallingConvention.Cdecl)]
    public static extern bool invoke_callback(void* args);
}

[UnmanagedCallersOnly(CallConvs = new[] { typeof(CallConvCdecl) })]
private static void Callback(IntPtr args) { ... }

static void Main(string[] _)
{
    NativeLib.set_callback((delegate* unmanaged[Cdecl]<IntPtr, void>)&Callback);

    // IL stub from set_callback is reused. GC transition is suppressed. Fatal error occurs trying to invoke callback. 
    NativeLib.invoke_callback(IntPtr.Zero.ToPointer());
}

typedef void (__cdecl *CALLBACK_FN)(void *args);
CALLBACK_FN g_cb;

extern "C" BOOL __declspec(dllexport) set_callback(CALLBACK_FN cb)
{
    g_cb = cb;
    return TRUE;
}

extern "C" BOOL __declspec(dllexport) invoke_callback(void* args)
{
    if (g_cb == nullptr)
        return FALSE;

    g_cb(args);
    return TRUE;
}

Testing

Tests added to verify that P/Invokes with the same signature, differing only in SuppressGCTransition, do not re-use the same IL stub.

Risk

This is a fairly targeted fix, but it does affect IL stub caching and the stub flags used during generation,

Regression

No. SuppressGCTransition was introduced in 5.0.

[jeffschwMSFT]

Approved. We will take for consideration in .NET 5.0.x