Cleanup VARIANT marshalling and convert to managed (#102498)

* Handle simple cases of MarshalComVariantForOleVariant * Pseudo code for MarshalOleVariantForComVariant * Complete and setup managed call for MarshalObjectForOleVariant * Setup MarshalHelperConvertObjectToVariant except record * Setup MarshalHelperCastVariant * Cleanup Variant struct and FCall * Handle record case * Cleanup VariantOleToCom/VariantComToOle routines * Remove VariantData definition at unmanaged side * Eliminate a dead branch for ref * Remove CVTypes in native * Share GetComIPFromObjectRef * Add tests for managed to native side * Add test for native to managed side * Add test for byref returning to BYREF * Fix BYREF handling * Fix array marshalling * Fix record marshalling * Fix VT_VARIANT case * Cleanup and fix IUnknown case * Add more test for VT_UNKNOWN * Cleanup for VT_VARIANT * Cleanup SR and comment * Fix contract in GetTypeHandleForVarType * Move VariantChangeTypeEx to coreclr * Add unmanaged fast path for (U)Int64 * Make ComVariant immutable * Add GCPROTECT in ConvertSystemColorToOleColor * Fix variant init * Fix BYREF|EMPTY * Initialize ComVariant * Move GCPROTECT to the QCall * Pass UnknownWrapper through SetFieldsObject * Apply suggestions from code review Co-authored-by: Aaron Robinson <[email protected]> * Suggestions at native side * Rename ComIP * Add comments for ConvertWrappedObject * Try convert to RECORD for all cases * Dispose Variant on failure * Update formatting * Fix NO_MAPPING definition --------- Co-authored-by: Aaron Robinson <[email protected]>
dotnet · Aug 28, 2024 · c1a9f26 · c1a9f26
1 parent 2df9935
commit c1a9f26
Show file tree

Hide file tree

Showing 26 changed files with 941 additions and 2,101 deletions.
diff --git a/src/coreclr/System.Private.CoreLib/System.Private.CoreLib.csproj b/src/coreclr/System.Private.CoreLib/System.Private.CoreLib.csproj
@@ -285,6 +285,9 @@
  <Compile Include="$(CommonPath)Interop\Windows\OleAut32\Interop.VariantClear.cs">
  <Link>Common\Interop\Windows\OleAut32\Interop.VariantClear.cs</Link>
  </Compile>
+ <Compile Include="$(CommonPath)Interop\Windows\OleAut32\Interop.VariantChangeTypeEx.cs">
+ <Link>Common\Interop\Windows\OleAut32\Interop.VariantChangeTypeEx.cs</Link>
+ </Compile>
  </ItemGroup>
  <ItemGroup Condition="'$(FeatureObjCMarshal)' == 'true'">
  <Compile Include="$(BclSourcesRoot)\System\Runtime\InteropServices\ObjectiveCMarshal.CoreCLR.cs" />

diff --git a/src/coreclr/System.Private.CoreLib/src/Microsoft/Win32/OAVariantLib.cs b/src/coreclr/System.Private.CoreLib/src/Microsoft/Win32/OAVariantLib.cs
@@ -151,19 +151,10 @@ private static ComVariant ToOAVariant(object input)
  null => default,
  Missing => throw new NotSupportedException(SR.NotSupported_ChangeType),
  DBNull => ComVariant.Null,
- _ => GetComIPFromObjectRef(input) // Convert the object to an IDispatch/IUnknown pointer.
+ _ => Variant.GetIUnknownOrIDispatchFromObject(input) // Convert the object to an IDispatch/IUnknown pointer.
  };
  }
 
- private static ComVariant GetComIPFromObjectRef(object? obj)
- {
- IntPtr pUnk = GetIUnknownOrIDispatchForObject(ObjectHandleOnStack.Create(ref obj), out bool isIDispatch);
- return ComVariant.CreateRaw(isIDispatch ? VarEnum.VT_DISPATCH : VarEnum.VT_UNKNOWN, pUnk);
- }
-
- [LibraryImport(RuntimeHelpers.QCall, EntryPoint = "MarshalNative_GetIUnknownOrIDispatchForObject")]
- private static partial IntPtr GetIUnknownOrIDispatchForObject(ObjectHandleOnStack o, [MarshalAs(UnmanagedType.Bool)] out bool isIDispatch);
-
  private static object? FromOAVariant(ComVariant input) =>
  input.VarType switch
  {