Allow ref struct to contain ref fields

[Sergio0694]

The current plan is to add ref fields as first class construct to the type system. See proposal at dotnet/csharplang#3936 for more details.

---

Original proposal

Overview

Make the System.ByReference<T> type (here) public instead of internal.

Rationale

It's already possible to achieve the same behavior of ByReference<T>, namely to store a ref T to some variable as a field in a ref struct, by leveraging the Span<T> and MemoryMarshal types, like so:

public readonly ref struct FakeByReference<T>
{
    private readonly Span<T> span;

    public ByReference(ref T value)
    {
        span = MemoryMarshal.CreateSpan(ref value, 1);
    }

    public ref T Value => ref MemoryMarshal.GetReference(span);
}

This can either be used as is, as a field in some other ref struct type, or the same technique (storing a 1-length Span<T> in place of the missing ByReference<T> type) can be used in other ref struct types to reproduce this same behavior: storing a managed reference as a field in a ref struct type.

Given that this is already possible with existing APIs, as shown above, but it's less performant than the original ByReference<T>, more verbose and more error prone, why not just make the original ByReference<T> type public so everyone can use it with no need to reinvent the wheel?

It'd be a neat API to have, and at basically no cost, since it's already in the .NET Core BCL anyway. 😄

[jkotas]

We would prefer this to be exposed as ref fields in C# language: dotnet/csharplang#1147 . This would be even faster, leaner, and less error prone than ByReference<T>.

[Sergio0694]

Hi @jkotas - thanks for linking that issue, I had missed that! 😊

Is there an ETA for that though? By the looks of it, and the fact it has been opened for over two years already, it looks like it still has a way to go, plus all the additional difficulties caused by the fact that that would require at least some Roslyn updates to support that, if not some CLR changes too.

My rationale here was that making the ByReference<T> type public wouldn't allow new scenarios not achievable today, but simply make them more convenient for devs that were interested in using them (with the assumption that those devs would already be aware of all the various implications of using unsafe APIs in that case). Even if a ref T field feature was to be implemented one day, wouldn't this still be a nice and easy to implement stopgap for the time being? Again, given that the same exact feature can still be achieved today anyway, just less efficiently and with more room for mistakes (and the additional verbosity of having to reimplement that fake ByReference<T> type).

[jkotas]

We are here for the long haul and want to do things properly. We try hard to avoid exposing temporary workarounds that we are stuck forever even once the more appropriate solution is available.

[Sergio0694]

Sure thing, that's fair enough 😄
Also, just to be clear, it wasn't my intention to suggest this change as a quick and sloppy solution instead of doing things properly, as I said my rationale was more of an "iterative" improvement for a scenario that's technically supported already, but with an even worse implementation. But I can see how it'd be preferable to jump straight to the optimal and built-in solution, absolutely 🚀

Is there any approximate ETA on that proposal? I see the issue has been open for over two years already and it's not currently assigned to any milestone, not even X.0 or X.X. Is that a feature that's still being evaluated and not yet decided on, or is it something the various teams have already decided to implement, just not with an exact timeframe for it?

Thanks again for your time! 😊

[jkotas]

It did not make it into the list of C# features we are looking at for .NET 5.

[DaZombieKiller]

Wouldn't exposing this type make it possible to create ref fields in languages other than C#?
To me that sounds like something worth considering.

[jkotas]

This type as it exists today is not safe construct. It is very easy to create dangling pointers with it, without guardrails in the language. If we were to expose it, we would likely mark it with the special ObsoleteAttribute to block its use in languages that are not aware of it (e.g. similar to how ref structs are blocked).

[GrabYourPitchforks]

On a related note, Jan, you and I previously talked about a syntax that would allow operating with gcrefs as naturally as operating with pointers. Things that would allow natural syntax like the + operator without bouncing through Unsafe.Add. Would something like that ever make sense to add as a language feature, or as a ByRef-like type, or as something else?

[Sergio0694]

Hey @jkotas - I have a follow up question on this, specifically regarding the general criteria being used to expose APIs or add explicit support for "unsafe" operations. You mentioned how with the ByReference<T> type is't possible to create dangling pointers, which is certainly true. My question though is about the seemingly arbitrary decision not to make the type public, despite the BCL having a ton of potentially dangerous APIs already. Consider this snippet:

public static Span<T> GetFaultySpan<T>()
{
    T value = default;
    return MemoryMarshal.CreateSpan(ref value, 1);
}

This compiles fine, but results in a dangling pointer just like the one you mentioned. It's not perfectly clear to me why is this API public, but ByReference<T> is not. Don't get me wrong, this API is super useful (and I love the fact that it's available!), but I don't get why the same rationale doesn't allow to ByReference<T> as well: "here's a potentially dangerous but useful tool, use at your own risk!".

Or, just like the BCL already includes dedicated code to properly handle devs doing stuff that's technically not valid (eg. Memory<T> wrapping strings as mutable), then why not just use the same rationale here as well, and provide a tool that enables devs to do exactly the same thing they can already do with MemoryMarshal.CreateSpan, but just in a more efficient and less error prone way?

I just can't stress enough how useful the ByReference<T> type would be, and with the native support for ref T fields not making the cut for .NET 5 I'd say it'll be at least another year (if not more) until that eventually makes it to a public release? As you said, you could also mark it as [Obsolete] just like Span<T>, to avoid issues with other .NET languages.

Just though I'd share my thoughts on this, thank you in advance for your time! 😊

P.S. Just so I'm perfectly clear, I'm not criticizing your work in any way, the .NET team is doing an incredible job and I really love where the whole platform is going! Just trying to make my case here as to why I think making ByReference<T> public could be a win-win for everyone. 😄

[tannergooding]

@jaredpar had some reasons on why this wasn't safe from the language perspective and would be extremely likely to cause bugs without proper language support for tracking the lifetime.

[jkotas]

This compiles fine, but results in a dangling pointer just like the one you mentioned

Methods in System.Runtime.InteropServices are unsafe. They can produce dangling pointers, corrupt memory, etc. You have to know what you are doing when using them and they should be avoided where possible.

extremely likely to cause bugs without proper language support for tracking the lifetime.

Correct. We would like ref fields to have proper language that guarantees its safety.

[Sergio0694]

Ah, yeah that's a fair point, having ByReference<T> under System.* would definitely be a problem - I hadn't noticed that before and I had just assumed it was under System.Runtime.* somewhere like Unsafe or MemoryMarshal. I guess it could very well be moved there since right now it's internal, but I can see how that would require more work/refactoring than just making it public 👍

And yeah I agree that just having language-level support for ref fields would be even better, absolutely no dispute there. I was just hoping for a better workaround for the time being, seeing as that feature at least for now seems to be pretty far from completion, is all.

Thank you both again for your time, I really appreciate the conversation here!

[jaredpar]

@tannergooding

had some reasons on why this wasn't safe from the language perspective and would be extremely likely to cause bugs without proper language support for tracking the lifetime.

The reasons are pretty straight forward: the lifetime safety rules, as written today, are predicated on such a constructor not existing. It's explicitly called out in the design.

That's not to say we can't do ref fields, just that the current design was written explicitly assuming they didn't exist. The initial designs of ref struct actually had support for them but we removed them because it introduced some heavy complexity into our lifetime rules. Now we think we have a simpler way of doing this and hope to bring ref fields into a future version of C#.

[john-h-k]

+1 on this. While the span workaround, well, works, it isn't ideal. Particularly where you have an interop scenario where you are using ByReference<T> in place of a T* in the native code, and want it to be blittable so it can be passed when pinned

[john-h-k]

On a related note, Jan, you and I previously talked about a syntax that would allow operating with gcrefs as naturally as operating with pointers. Things that would allow natural syntax like the + operator without bouncing through Unsafe.Add. Would something like that ever make sense to add as a language feature, or as a ByRef-like type, or as something else?

Having it actually added to refs would be a huge breaking change wouldn't it. As currently, + refers to the underlying object the ref points to

[Sergio0694]

Personally I'd already be more than happy just to have the ref field C# feature that was mentioned, especially if not having to also work out a new syntax to allow offsetting over ref-s could make the implementation of these ref fields quicker. Even just with that, we could write something like:

public readonly ref struct Ref<T>
{
    public readonly ref T Value;

    public Ref(ref T value)
    {
        Value = value;
    }

    public ref T this[int i]
    {
        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        get => Unsafe.Add(ref Value, i);
    }
}

Which personally I'm very looking forward to, as it'd be a nice and fast abstraction over having to manually use Unsafe.Add every time you need to offset to some location 😄

[john-h-k]

Personally I'd already be more than happy just to have the ref field C# feature that was mentioned,

well yeah, that is just better, but it is far away, will take a lot of compiler work, new language rules, etc. i just want the dangerous scary thing exposed that if you f#@! up then it doesn't help you

[GrabYourPitchforks]

~~If you reeeeeeeally wanted ByReference<T>, an enterprising individual could write a fully out-of-band implementation. 😁 You'd have to write it in IL instead of C#, but there's nothing stopping anybody from doing it. Said assembly would need to target netcoreapp2.1+ and wouldn't be compatible with netstandard or netfx.

I don't think such an out-of-band type would be blittable or would play well with the interop system. I don't even think the internal in-box type fulfills these characteristics. You'd need to pin the T& to a T* if you wanted it to be blittable.~~

This idea is shot. See below.

[jkotas]

@GrabYourPitchforks I am not sure what you have in mind, but I do not think it would work.

[john-h-k]

@GrabYourPitchforks ByReference<T> is not an S.R.CS.Unsafe kind of type which is written in IL. It is an intrinsic'y one that has a special meaning to the runtime afaik

[GrabYourPitchforks]

@jkotas I thought as of netcoreapp2.1 the runtime itself had support for byref members of structs, even though the C# compiler doesn't? I was thinking you could write a class entirely in IL that looked something like:

// pseudo-code
public ref struct ByReference<T>
{
    private readonly T& _value;
    public ByReference(ref T value) { _value = ref value; }
    public ref T Value => ldfld(_value);
}

The public API surface uses only constructs that the C# compiler can handle. Admittedly I haven't actually tried this. Just spitballing.

[jkotas]

I thought as of netcoreapp2.1 the runtime itself had support for byref members of structs

That is not correct. You should see type load exception if you try to run your example.

[john-h-k]

@GrabYourPitchforks nope, invalid IL. I believe the current way it works is by having a dummy IntPtr in the ByReference<T> type (see here), and specially instructing the runtime that in reality is a tracked ref

Won't even assemble, iirc, @jkotas - FieldSig has no BYREF attribute (unlike LocalVarSig/Param blob) so it can't be encoded.

[GrabYourPitchforks]

Well, there goes that idea. :)

[ENikS]

This was discussed for C# 9 but just fell off the priority list. It will be considered as part of C# 10. I'm the champion for the feature, have the rough design for it and related features, and once we get past the .NET 5 ship cycle will likely get it polished up for review.

Is there a good way to make sure it will not 'fell off the priority list' next time?

[jaredpar]

Is there a good way to make sure it will not 'fell off the priority list' next time?

Keep an eye on dotnet/csharplang#1147. That is the main issue I'm using to track the request at the moment (there are actually many overlapping issues for this feature). Once the proposal is finished that issue will have an update to point at it and from there you can see the LDM discussions on it.

Probably won't make it up there for at least a few more weeks though.

Q: Well what if @jaredpar never actually writes a proposal for this?
A: Likely several people on the team will hunt me down, even during covid, and make me write it.

[ENikS]

@Sergio0694 I guess it is safe to close this issue now

@jaredpar

Well what if @jaredpar never actually writes a proposal for this?

Now, that we know where you live on GitHub, we will be haunting you as the shadow of Hamlet's father... :).

[tannergooding]

Likely several people on the team will hunt me down, even during covid, and make me write it.

Several of the features I want in are dependent on the feature, so I poke @jaredpar about it every know and again, as he can attest 😄

[terrajobst]

Surveillance footage from when we had a team room:

[jaredpar]

FYI: the initial proposal for allowing ref fields in C# is up for review on csharplang

dotnet/csharplang#3936

[GrabYourPitchforks]

Given that Jared is driving this from the C# side, is there anything left for this runtime issue to track? Will close this issue after a few days if there's no remaining work.

[tannergooding]

How discoverable are closed vs open issues?

I know, for example, they don't show by default in issue search so that might be a point towards keeping it open until the C# feature is implemented.

[tannergooding]

(We do have the tracking-external-issue label for such occasions)

[AaronRobinsonMSFT]

One of the items in #63768 is to delete the ByReference<T> type. Additionally, PR #63985 added support for ref fields and now it is up to Roslyn to provide the C# in order to perform the operation. Therefore, we can close this issue in lieu of #63768.