Hi all,
We are experiencing an access violation in the .NET 4.0 runtime, in a piece of code that uses NHibernate heavily. None of our own code interops with native code directly, nor does the 3rd party code we are using (to the best of our knowledge). We cannot identify the memory pattern that is appears to be corrupting the GC heap, and we don't know whether the problem is in the GC itself or in some other piece of code that is corrupting it.
I've included below the stack trace of the offending thread, and the list of loaded modules.
We need some help debugging this issue. What should we do next?
Thanks,
Adam Smith
Director, Technology
Hubbard One
Thomson Reuters
0:000> !EEStack
---------------------------------------------
Thread 0
Current frame: clr!WKS::gc_heap::find_first_object+0x64
ChildEBP RetAddr Caller, Callee
0012bf30 791fa5b8 clr!WKS::gc_heap::mark_through_cards_for_segments+0x116, calling clr!WKS::gc_heap::find_card
0012bf38 791fa618 clr!WKS::gc_heap::mark_through_cards_for_segments+0x563, calling clr!WKS::gc_heap::find_first_object
0012bfc4 791faaa1 clr!WKS::gc_heap::relocate_phase+0x5b, calling clr!WKS::gc_heap::mark_through_cards_for_segments
0012c000 791f7a52 clr!WKS::gc_heap::plan_phase+0x851, calling clr!WKS::gc_heap::relocate_phase
0012c038 791a1be6 clr!HndScanHandlesForGC+0x125, calling clr!_EH_epilog3
0012c03c 791a2424 clr!Ref_ScanDependentHandlesForClearing+0x61, calling clr!HndScanHandlesForGC
0012c044 791a2291 clr!SyncBlockCache::GCWeakPtrScan+0x61
0012c0dc 791f73d1 clr!WKS::gc_heap::gc1+0x140, calling clr!WKS::gc_heap::plan_phase
0012c100 791f7972 clr!WKS::gc_heap::garbage_collect+0x3ae, calling clr!WKS::gc_heap::gc1
0012c118 791f6edd clr!WKS::gc_heap::soh_try_fit+0x16b, calling clr!WKS::gc_heap::a_fit_segment_end_p
0012c184 791f771f clr!WKS::GCHeap::GarbageCollectGeneration+0x17b, calling clr!WKS::gc_heap::garbage_collect
0012c1a8 791f6b68 clr!WKS::gc_heap::try_allocate_more_space+0x23a, calling clr!WKS::gc_heap::allocate_small
0012c1b0 791f878a clr!WKS::gc_heap::try_allocate_more_space+0x162, calling clr!WKS::GCHeap::GarbageCollectGeneration
0012c1d4 791f6b82 clr!WKS::gc_heap::allocate_more_space+0x13, calling clr!WKS::gc_heap::try_allocate_more_space
0012c1e8 791f6e65 clr!WKS::GCHeap::Alloc+0x3d, calling clr!WKS::gc_heap::allocate_more_space
0012c208 7919c953 clr!Alloc+0x8d
0012c224 7916089d clr!SlowAllocateString+0x42, calling clr!Alloc
0012c258 7919c876 clr!HelperMethodFrame::LazyInit+0x17, calling (JitHelp: CORINFO_HELP_GET_THREAD)
0012c264 79160973 clr!FramedAllocateString+0xc9, calling clr!SlowAllocateString
0012c2ac 791608f6 clr!FramedAllocateString+0x18, calling clr!LazyMachStateCaptureState
0012c2e0 0703bec0 (MethodDesc 06ba8e80 +0xf0 NHibernate.Engine.Cascade.DeleteOrphans(System.String, NHibernate.Collection.IPersistentCollection)), calling clr!JIT_IsInstanceOfInterface
0012c300 79b3781c (MethodDesc 798fbdcc +0x7c System.String.Concat(System.String, System.String)), calling 00972350
0012c318 06bcadf0 (MethodDesc 06ba8e74 +0x2f0 NHibernate.Engine.Cascade.CascadeCollectionElements(System.Object, NHibernate.Type.CollectionType, NHibernate.Engine.CascadeStyle, NHibernate.Type.IType, System.Object, Boolean)), calling (MethodDesc 798fbdcc +0
System.String.Concat(System.String, System.String))
0012c34c 06bcaab9 (MethodDesc 06ba8e5c +0x99 NHibernate.Engine.Cascade.CascadeCollection(System.Object, NHibernate.Engine.CascadeStyle, System.Object, NHibernate.Type.CollectionType)), calling (MethodDesc 06ba8e74 +0 NHibernate.Engine.Cascade.CascadeCollectionElements(System.Object,
NHibernate.Type.CollectionType, NHibernate.Engine.CascadeStyle, NHibernate.Type.IType, System.Object, Boolean))
0012c380 06bcaa07 (MethodDesc 06ba8e50 +0x77 NHibernate.Engine.Cascade.CascadeAssociation(System.Object, NHibernate.Type.IType, NHibernate.Engine.CascadeStyle, System.Object, Boolean)), calling (MethodDesc 06ba8e5c +0 NHibernate.Engine.Cascade.CascadeCollection(System.Object,
NHibernate.Engine.CascadeStyle, System.Object, NHibernate.Type.CollectionType))
0012c3a0 06bc7f7d (MethodDesc 06ba8e2c +0x4d NHibernate.Engine.Cascade.CascadeProperty(System.Object, NHibernate.Type.IType, NHibernate.Engine.CascadeStyle, System.Object, Boolean)), calling (MethodDesc 06ba8e50 +0 NHibernate.Engine.Cascade.CascadeAssociation(System.Object,
NHibernate.Type.IType, NHibernate.Engine.CascadeStyle, System.Object, Boolean))
0012c3c4 06bc7c99 (MethodDesc 06ba8e20 +0x159 NHibernate.Engine.Cascade.CascadeOn(NHibernate.Persister.Entity.IEntityPersister, System.Object, System.Object)), calling (MethodDesc 06ba8e2c +0 NHibernate.Engine.Cascade.CascadeProperty(System.Object, NHibernate.Type.IType,
NHibernate.Engine.CascadeStyle, System.Object, Boolean))
0012c40c 06bcb8a4 (MethodDesc 04b699a0 +0x64 NHibernate.Event.Default.AbstractFlushingEventListener.CascadeOnFlush(NHibernate.Event.IEventSource, NHibernate.Persister.Entity.IEntityPersister, System.Object, System.Object)), calling (MethodDesc 06ba8e20 +0 NHibernate.Engine.Cascade.CascadeOn(NHibernate.Persister.Entity.IEntityPersister,
System.Object, System.Object))
0012c440 06bcb793 (MethodDesc 04b69994 +0xf3 NHibernate.Event.Default.AbstractFlushingEventListener.PrepareEntityFlushes(NHibernate.Event.IEventSource)), calling (MethodDesc 04b699a0 +0 NHibernate.Event.Default.AbstractFlushingEventListener.CascadeOnFlush(NHibernate.Event.IEventSource,
NHibernate.Persister.Entity.IEntityPersister, System.Object, System.Object))
0012c47c 06bcb388 (MethodDesc 04b69968 +0x88 NHibernate.Event.Default.AbstractFlushingEventListener.FlushEverythingToExecutions(NHibernate.Event.FlushEvent)), calling (MethodDesc 04b69994 +0 NHibernate.Event.Default.AbstractFlushingEventListener.PrepareEntityFlushes(NHibernate.Event.IEventSource))
0012c4b0 06bcb24d (MethodDesc 04b69b74 +0x2d NHibernate.Event.Default.DefaultFlushEventListener.OnFlush(NHibernate.Event.FlushEvent))
0012c4c4 06bcb195 (MethodDesc 02af3330 +0xd5 NHibernate.Impl.SessionImpl.Flush()), calling 057f5e72
Base TimeStamp Module
400000 48ebcdd3 Oct 07 17:00:03 2008 X:\ContactNet\bin\NAnt.exe
7c800000 49900d60 Feb 09 06:02:56 2009 C:\WINDOWS\system32\ntdll.dll
79000000 4af3af84 Nov 06 00:09:24 2009 C:\WINDOWS\system32\mscoree.dll
77e40000 49c51f0a Mar 21 13:08:26 2009 C:\WINDOWS\system32\KERNEL32.dll
7d1e0000 4a61f120 Jul 18 11:58:24 2009 C:\WINDOWS\system32\ADVAPI32.dll
77c50000 49f5889a Apr 27 06:27:38 2009 C:\WINDOWS\system32\RPCRT4.dll
76f50000 4a3742b4 Jun 16 02:59:00 2009 C:\WINDOWS\system32\Secur32.dll
603b0000 4ba1d8a9 Mar 18 03:39:21 2010 C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\mscoreei.dll
77da0000 45d70ac0 Feb 17 09:01:36 2007 C:\WINDOWS\system32\SHLWAPI.dll
77c00000 4900637a Oct 23 07:43:54 2008 C:\WINDOWS\system32\GDI32.dll
77380000 45e7c676 Mar 02 01:38:46 2007 C:\WINDOWS\system32\USER32.dll
77ba0000 45d70b06 Feb 17 09:02:46 2007 C:\WINDOWS\system32\msvcrt.dll
76290000 45d70a5f Feb 17 08:59:59 2007 C:\WINDOWS\system32\IMM32.DLL
79140000 4ba1d9ef Mar 18 03:44:47 2010 C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\clr.dll
79060000 4ba1dbf2 Mar 18 03:53:22 2010 C:\WINDOWS\system32\MSVCR100_CLR0400.dll
79880000 4ba1da6f Mar 18 03:46:55 2010 C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\mscorlib\246f1a5abb686b9dcdf22d3505b08cea\mscorlib.ni.dll
77670000 45d70aa5 Feb 17 09:01:09 2007 C:\WINDOWS\system32\ole32.dll
4b3c0000 45d70ab2 Feb 17 09:01:22 2007 C:\WINDOWS\system32\MSCTF.dll
60340000 4ba1d8aa Mar 18 03:39:22 2010 C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\culture.dll
60930000 4ba1d8ae Mar 18 03:39:26 2010 C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\nlssorting.dll
79810000 4ba1da36 Mar 18 03:45:58 2010 C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\clrjit.dll
68000000 45d69786 Feb 17 00:49:58 2007 C:\WINDOWS\system32\rsaenh.dll
76b70000 45d70ab5 Feb 17 09:01:25 2007 C:\WINDOWS\system32\PSAPI.DLL
32e0000 45d69418 Feb 17 00:35:20 2007 C:\WINDOWS\system32\xpsp2res.dll
77b90000 45d70ac8 Feb 17 09:01:44 2007 C:\WINDOWS\system32\VERSION.dll
7a820000 4ba1dff4 Mar 18 04:10:28 2010 C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System\964da027ebca3b263a05cadb8eaa20a3\System.ni.dll
60c90000 4ba1e04b Mar 18 04:11:55 2010 C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.Configuration\ac18c2dcd06bd2a0589bac94ccae5716\System.Configuration.ni.dll
69720000 4ba1dfec Mar 18 04:10:20 2010 C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.Xml\e997d0200c25f7db6bd32313d50b729d\System.Xml.ni.dll
6f350000 4a4eaae7 Jul 03 21:05:43 2009 C:\WINDOWS\system32\urlmon.dll
77d00000 4760e409 Dec 13 02:49:29 2007 C:\WINDOWS\system32\OLEAUT32.dll
40a90000 4a4eaae8 Jul 03 21:05:44 2009 C:\WINDOWS\system32\iertutil.dll
77420000 45d70a05 Feb 17 08:58:29 2007 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.3790.3959_x-ww_D8713E55\comctl32.dll
7c8d0000 485819a8 Jun 17 16:08:08 2008 C:\WINDOWS\system32\SHELL32.dll
44f20000 4ba1e740 Mar 18 04:41:36 2010 C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\Microsoft.Build.Fra#\11ef4be6ee227fce3725d6df534297a4\Microsoft.Build.Framework.ni.dll
60e50000 4ba1dfee Mar 18 04:10:22 2010 C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.Core\713647b987b140a17e3c4ffe4c721f85\System.Core.ni.dll
67160000 4ba1e0c5 Mar 18 04:13:57 2010 C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.Web\a70842538614699d690561ef5f43598b\System.Web.ni.dll
5e0d0000 4ba2183b Mar 18 08:10:35 2010 C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\diasymreader.dll
60650000 4ba1db8f Mar 18 03:51:43 2010 C:\WINDOWS\Microsoft.Net\assembly\GAC_32\ISymWrapper\v4.0_4.0.0.0__b03f5f7f11d50a3a\ISymWrapper.dll
61750000 4ba1e064 Mar 18 04:12:20 2010 C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.Data\92cccedc7cda413ff6fc6492cb256b58\System.Data.ni.dll
4e00000 4ba1e064 Mar 18 04:12:20 2010 C:\WINDOWS\Microsoft.Net\assembly\GAC_32\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
71c00000 45d70ae9 Feb 17 09:02:17 2007 C:\WINDOWS\system32\WS2_32.dll
71bf0000 45d70aea Feb 17 09:02:18 2007 C:\WINDOWS\system32\WS2HELP.dll
761b0000 45d70a80 Feb 17 09:00:32 2007 C:\WINDOWS\system32\CRYPT32.dll
76190000 45d70aac Feb 17 09:01:16 2007 C:\WINDOWS\system32\MSASN1.dll
766d0000 45d70abc Feb 17 09:01:32 2007 C:\WINDOWS\system32\shfolder.dll
666d0000 4ba1e09d Mar 18 04:13:17 2010 C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.Transactions\dd9dbf82e44454689976a49a9e4ddb6d\System.Transactions.ni.dll
66680000 4ba1e09d Mar 18 04:13:17 2010 C:\WINDOWS\Microsoft.Net\assembly\GAC_32\System.Transactions\v4.0_4.0.0.0__b77a5c561934e089\System.Transactions.dll
65d70000 4ba1df90 Mar 18 04:08:48 2010 C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.EnterpriseSe#\8b6e9d6171aad3561263ce2cd05c57df\System.EnterpriseServices.ni.dll
10020000 4ba1db80 Mar 18 03:51:28 2010 C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.EnterpriseSe#\8b6e9d6171aad3561263ce2cd05c57df\System.EnterpriseServices.Wrapper.dll
10000000 4ba1db80 Mar 18 03:51:28 2010 C:\WINDOWS\Microsoft.Net\assembly\GAC_32\System.EnterpriseServices\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.EnterpriseServices.Wrapper.dll
71f60000 3e8024bc Mar 25 05:43:24 2003 C:\WINDOWS\system32\security.dll
76750000 4a3742b4 Jun 16 02:59:00 2009 C:\WINDOWS\system32\schannel.dll
76920000 45d70ac8 Feb 17 09:01:44 2007 C:\WINDOWS\system32\USERENV.dll
71c40000 48f7bdc3 Oct 16 18:18:43 2008 C:\WINDOWS\system32\NETAPI32.dll
48060000 434f63fa Oct 14 03:53:30 2005 C:\Program Files\Microsoft SQL Server\90\Shared\instapi.dll
78130000 4889d619 Jul 25 09:33:13 2008 C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.3053_x-ww_B80FA8CA\MSVCR80.dll
68100000 45d6978b Feb 17 00:50:03 2007 C:\WINDOWS\system32\dssenh.dll
66230000 4ba1dfda Mar 18 04:10:02 2010 C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.Numerics\b07f0d26a34ad53fc369248f289d1126\System.Numerics.ni.dll
66510000 4ba1df78 Mar 18 04:08:24 2010 C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.Security\09a97525ae5583cc2685e2c39a3078bd\System.Security.ni.dll
7b1d0000 4ba1e086 Mar 18 04:12:54 2010 C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.Drawing\dd57bc19f5807c6dbe8f88d4a23277f6\System.Drawing.ni.dll
68df0000 4ba1e1a5 Mar 18 04:17:41 2010 C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.Web.Services\149f2dcb9c9706e592d1980a945850c2\System.Web.Services.ni.dll
51860000 4ba1f498 Mar 18 05:38:32 2010 C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.ServiceModel\250b525aa8c17327216e102569c0d766\System.ServiceModel.ni.dll
66610000 4ba1e143 Mar 18 04:16:03 2010 C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.ServiceProce#\6e7f1bdc845816dfc797f8002b76b5e8\System.ServiceProcess.ni.dll
52d30000 4ba1f585 Mar 18 05:42:29 2010 C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.ServiceModel#\51c60db370e050d9cdcac17060aaac53\System.ServiceModel.Web.ni.dll
51130000 4ba1f437 Mar 18 05:36:55 2010 C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.Runtime.Seri#\e9f8a45b1063d6c6a62718c88a5623d1\System.Runtime.Serialization.ni.dll
63d00000 4ba1e082 Mar 18 04:12:50 2010 C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.Data.OracleC#\db33744fb49e77c7233adb50f07fe62a\System.Data.OracleClient.ni.dll
63c80000 4ba1e082 Mar 18 04:12:50 2010 C:\WINDOWS\Microsoft.Net\assembly\GAC_32\System.Data.OracleClient\v4.0_4.0.0.0__b77a5c561934e089\System.Data.OracleClient.dll