In order for my question to make sense, it needs some setup. Forgive me if you know some of this.
Mostly custom marshaling is pretty straight forward. When calling a function with a parameter marked as 'UnmanagedType.CustomMarshaler' in addition to either [In] or [Out]:
- GetInstance() gets called and returns an instance of the custom marshaler.
- When .Net needs to marshal from native to managed it calls MarshalManagedToNative, which takes an object as a parameter and returns an IntPtr.
- When .Net needs to marshal from managed to native it calls MarshalNativeToManaged, which takes an IntPtr as a parameter and returns an object.
However, things get a little tricky for parameters that are [In, Out].
- If the marshaling starts by managed calling unmanaged, then MarshalManagedToNative gets called first and it returns an IntPtr (as per normal).
- When the unmanaged code is complete, the second half of the call comes (MarshalNativeToManaged) to send the value back into managed code. The IntPtr you returned from MarshalManagedToNative is sent in as the parameter (again, as per normal).
- But in this case, the return value from MarshalNativeToManaged is ignored.
In order to complete the marshaling, you have to modify the the object that was sent in MarshalManagedToNative. Since MarshalNativeToManaged doesn't pass this value again, you have to have saved it during the original MarshalManagedToNative call.
While it's unfortunate that the docs don't describe this requirement, it doesn't *seem* like it should be too hard to work around. You just create a member variable on your custom marshaler class to save the object from the first call, and modify it in the second call.
However.
Threads
-------
If your custom marshaler is called from multiple threads, .Net *may* create multiple instances of the marshaler. But even if it does create multiple instance, it doesn't use them. .Net picks one, and uses it for every call from every thread. As you might expect, this results in chaos if you are storing data members in the custom marshaler.
My first thought on how to solve this was to use ThreadStatic variables. But is that safe? The docs I've read for how managed threadids (which I assume are the basis for Threadstatic) get assigned are a little vague. And given that we are alternating between managed code and unmanaged code, I'm even more concerned.
Am I guaranteed that both MarshalNativeToManaged and MarshalManagedToNative will always get called such that the ThreadStatic values will match? It *seems* to, but anecdotal observation is not the same as "the spec guarantees that it will."
And even if it does, that brings us to the next problem.
Nesting
-------
InOut calls come in pairs. You either have MarshalManagedToNative followed by MarshalNativeToManaged, or MarshalNativeToManaged followed by MarshalManagedToNative. Which pair you get depends on whether you start in managed or unmanaged.
But what if you start with managed calling unmanaged (MarshalManagedToNative), but then the unmanaged code makes a call to managed code during its operation? Perhaps it is forwarding the call. Or maybe it just needs some information from a managed object to satisfy the first request. It turns out that this too uses the same (single) instance of the custom marshaler.
Normal:
MarshalManagedToNative
MarshalNativeToManaged
Nested:
C1: MarshalManagedToNative - 1
C2: MarshalNativeToManaged - 2 (nested call)
C3: MarshalManagedToNative - 2
C4: MarshalNativeToManaged - 1
So when a custom marshaler starts with a MarshalManagedToNative, which is followed by a MarshalNativeToManaged, there is no way to know whether the MarshalNativeToManaged is the second half of a 'normal' call, or the first half of a nested call. You can try to guess. If you see that the IntPtr you received in C2 is the same one you just returned from C1, that might indicate it's the end of a normal call. But that could also mean that the call is being forwarded to managed to respond.
If you work the problem long enough, you can find some solutions.
I've written an implementation using ThreadStatic. It seems to work, but it depends on ThreadStatic working the way I hope it does, along with some hopes about how custom marshaling works (and will continue to work). As you might guess, the code is complex, if brief.
I also have an alternative that was proposed using ConcurrentDictionary. This solves some problems over ThreadStatic, but generates others.
These are both rather complex and feel like clumsy work-arounds. Is there an 'official' or commonly used solution? It would be great if there were some trick I'm just missing here.
I have some (ugly) sample code if anyone wants to experiment (tested with 2.0 and 4.0.3). But I'm really hoping that someone else has some code they can share with me.
Mostly custom marshaling is pretty straight forward. When calling a function with a parameter marked as 'UnmanagedType.CustomMarshaler' in addition to either [In] or [Out]:
- GetInstance() gets called and returns an instance of the custom marshaler.
- When .Net needs to marshal from native to managed it calls MarshalManagedToNative, which takes an object as a parameter and returns an IntPtr.
- When .Net needs to marshal from managed to native it calls MarshalNativeToManaged, which takes an IntPtr as a parameter and returns an object.
However, things get a little tricky for parameters that are [In, Out].
- If the marshaling starts by managed calling unmanaged, then MarshalManagedToNative gets called first and it returns an IntPtr (as per normal).
- When the unmanaged code is complete, the second half of the call comes (MarshalNativeToManaged) to send the value back into managed code. The IntPtr you returned from MarshalManagedToNative is sent in as the parameter (again, as per normal).
- But in this case, the return value from MarshalNativeToManaged is ignored.
In order to complete the marshaling, you have to modify the the object that was sent in MarshalManagedToNative. Since MarshalNativeToManaged doesn't pass this value again, you have to have saved it during the original MarshalManagedToNative call.
While it's unfortunate that the docs don't describe this requirement, it doesn't *seem* like it should be too hard to work around. You just create a member variable on your custom marshaler class to save the object from the first call, and modify it in the second call.
However.
Threads
-------
If your custom marshaler is called from multiple threads, .Net *may* create multiple instances of the marshaler. But even if it does create multiple instance, it doesn't use them. .Net picks one, and uses it for every call from every thread. As you might expect, this results in chaos if you are storing data members in the custom marshaler.
My first thought on how to solve this was to use ThreadStatic variables. But is that safe? The docs I've read for how managed threadids (which I assume are the basis for Threadstatic) get assigned are a little vague. And given that we are alternating between managed code and unmanaged code, I'm even more concerned.
Am I guaranteed that both MarshalNativeToManaged and MarshalManagedToNative will always get called such that the ThreadStatic values will match? It *seems* to, but anecdotal observation is not the same as "the spec guarantees that it will."
And even if it does, that brings us to the next problem.
Nesting
-------
InOut calls come in pairs. You either have MarshalManagedToNative followed by MarshalNativeToManaged, or MarshalNativeToManaged followed by MarshalManagedToNative. Which pair you get depends on whether you start in managed or unmanaged.
But what if you start with managed calling unmanaged (MarshalManagedToNative), but then the unmanaged code makes a call to managed code during its operation? Perhaps it is forwarding the call. Or maybe it just needs some information from a managed object to satisfy the first request. It turns out that this too uses the same (single) instance of the custom marshaler.
Normal:
MarshalManagedToNative
MarshalNativeToManaged
Nested:
C1: MarshalManagedToNative - 1
C2: MarshalNativeToManaged - 2 (nested call)
C3: MarshalManagedToNative - 2
C4: MarshalNativeToManaged - 1
So when a custom marshaler starts with a MarshalManagedToNative, which is followed by a MarshalNativeToManaged, there is no way to know whether the MarshalNativeToManaged is the second half of a 'normal' call, or the first half of a nested call. You can try to guess. If you see that the IntPtr you received in C2 is the same one you just returned from C1, that might indicate it's the end of a normal call. But that could also mean that the call is being forwarded to managed to respond.
If you work the problem long enough, you can find some solutions.
I've written an implementation using ThreadStatic. It seems to work, but it depends on ThreadStatic working the way I hope it does, along with some hopes about how custom marshaling works (and will continue to work). As you might guess, the code is complex, if brief.
I also have an alternative that was proposed using ConcurrentDictionary. This solves some problems over ThreadStatic, but generates others.
These are both rather complex and feel like clumsy work-arounds. Is there an 'official' or commonly used solution? It would be great if there were some trick I'm just missing here.
I have some (ugly) sample code if anyone wants to experiment (tested with 2.0 and 4.0.3). But I'm really hoping that someone else has some code they can share with me.