If you are interested in the topic see the talk page
Recently I was debugging high memory usage in application written in .NET 4 using WCF. After few hours application was eating almost 8 GB of memory. I made a memory dump and started digging into it.
First, statistics of heap (sorted by total memory usage per type)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
!dumpheap -stat MT Count TotalSize Class Name 000007fecdf6df78 46895258 1500648256,00 System.ServiceModel.Security.MessagePartSpecification 000007fee98b5b80 14351165 1249823520,00 System.Object[] 000007fecdf7fea0 253208 1197563136,00 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.ServiceModel.Security.MessagePartSpecification, System.ServiceModel]][] 000007fee98cfdd0 1220610 588263408,00 System.Byte[] 000007fee8169780 11719237 468769480,00 System.Collections.Generic.List`1[[System.Xml.XmlQualifiedName, System.Xml]] 000007fee98ceeb0 392174 215851704,00 System.Int32[] 000007fee98c7d90 1917846 186874224,00 System.String 000007fee815be60 1623002 64920080,00 System.Xml.NameTable+Entry 000007feccd07a70 538933 60360496,00 System.Xml.XmlBaseReader+XmlElementNode 000007fee8160860 787684 56713248,00 System.Xml.XmlName 000007feccd07700 1285725 51429000,00 System.Xml.StringHandle 000007feccd07680 1285725 51429000,00 System.Xml.PrefixHandle 000007feccd07ae8 538933 47426104,00 System.Xml.XmlBaseReader+XmlEndElementNode 000007fee98cf8b0 102864 35425872,00 System.Collections.Hashtable+bucket[] 000007fee8160298 598925 33539800,00 System.Xml.XmlElement 000007fecdf7f8b8 378220 33283360,00 System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[System.ServiceModel.Security.MessagePartSpecification, System.ServiceModel]] 000007feccd07780 814117 32564680,00 System.Xml.ValueHandle 000007fecdf34d40 756104 30244160,00 System.Collections.Generic.List`1[[System.ServiceModel.Security.Tokens.SecurityTokenParameters, System.ServiceModel]] 000007fecd04c3d8 62954 29714288,00 System.IdentityModel.SamlDictionary ... |
So we can see, that there are over 45M of System.ServiceModel.Security.MessagePartSpecification objects. Let’s see one of them:
|
1 2 3 4 |
!dumpheap -type System.ServiceModel.Security.MessagePartSpecification ... 000000010a5745f0 000007fecdf7fea0 4752 ... |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 |
0:046> !gcroot 000000010a5745f0 Note: Roots found on stacks may be false positives. Run "!help gcroot" for more info. Scan Thread 7 OSTHread 2dcc Scan Thread 29 OSTHread 1084 Scan Thread 4 OSTHread 2c60 Scan Thread 30 OSTHread eb0 Scan Thread 31 OSTHread 21c4 Scan Thread 19 OSTHread 2e60 Scan Thread 32 OSTHread 2d24 Scan Thread 33 OSTHread 2840 Scan Thread 34 OSTHread 788 Scan Thread 35 OSTHread 145c Scan Thread 36 OSTHread 1b80 Scan Thread 37 OSTHread 1898 Scan Thread 38 OSTHread 2600 Scan Thread 40 OSTHread 216c Scan Thread 41 OSTHread 2bec Scan Thread 42 OSTHread fa8 Scan Thread 43 OSTHread 1ebc Scan Thread 44 OSTHread e28 Scan Thread 45 OSTHread 2ef8 Scan Thread 47 OSTHread 2418 Scan Thread 46 OSTHread 2934 DOMAIN(00000000067B1EA0):HANDLE(Pinned):2dc13e0:Root:00000001ffdb5b08(System.Object[])-> 00000000ffddc9b8(System.ServiceModel.ChannelFactoryRefCache`1[[MySystem.IService, App_WebReferences.kj6s2mli]])-> 00000001dfe63928(System.ServiceModel.ChannelFactoryRef`1[[MySystem.IService, App_WebReferences.kj6s2mli]])-> 00000001dfdebe28(System.ServiceModel.ChannelFactory`1[[MySystem.IService, App_WebReferences.kj6s2mli]])-> 00000001dff06140(System.ServiceModel.Channels.ServiceChannelFactory+ServiceChannelFactoryOverRequestSession)-> 00000001dff05b50(System.ServiceModel.Channels.SecurityChannelFactory`1[[System.ServiceModel.Channels.IRequestSessionChannel, System.ServiceModel]])-> 00000001dff05c50(System.ServiceModel.Channels.HttpChannelFactory)-> 00000001dff05d78(System.ServiceModel.Channels.CommunicationObjectManager`1[[System.ServiceModel.Channels.IChannel, System.ServiceModel]])-> 00000001dff05db0(System.Collections.Hashtable)-> 000000020fda8130(System.Collections.Hashtable+bucket[])-> 000000010a555568(System.ServiceModel.Channels.HttpChannelFactory+HttpRequestChannel)-> 000000010a589108(System.EventHandler)-> 000000010a5556b8(System.ServiceModel.Channels.ReliableChannelBinder`1+ChannelSynchronizer[[System.ServiceModel.Channels.IRequestChannel, System.ServiceModel]])-> 000000010a5554e0(System.ServiceModel.Channels.ClientReliableChannelBinder`1+RequestClientReliableChannelBinder[[System.ServiceModel.Channels.IRequestChannel, System.ServiceModel]])-> 000000010a555490(System.ServiceModel.Channels.ChannelParameterCollection)-> 000000010a555258(System.ServiceModel.Security.SecuritySessionClientSettings`1+SecurityRequestSessionChannel[[System.ServiceModel.Channels.IRequestSessionChannel, System.ServiceModel]])-> 000000010a556450(System.ServiceModel.Security.SecuritySessionSecurityTokenProvider)-> 000000010a564e20(System.ServiceModel.Channels.SecurityChannelFactory`1[[System.ServiceModel.Channels.IRequestChannel, System.ServiceModel]])-> 000000010a5578d8(System.ServiceModel.Security.SymmetricSecurityProtocolFactory)-> 000000010a557a00(System.ServiceModel.Security.ChannelProtectionRequirements)-> 000000010a557b88(System.ServiceModel.Security.ScopedMessagePartSpecification)-> 000000010a570a10(System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[System.ServiceModel.Security.MessagePartSpecification, System.ServiceModel]])-> 000000010a5745f0(System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.ServiceModel.Security.MessagePartSpecification, System.ServiceModel]][]) |
So we can see that it is ultimately rooted in some pinned array of objects. What’s that? This array holds static variables from app domain. It usually looks as follows:
|
1 2 3 4 5 6 7 8 9 |
0:046> !do 00000001ffdb5b08 Name: System.Object[] MethodTable: 000007fee98b5b80 EEClass: 000007fee94ceb88 Size: 8192(0x2000) bytes Array: Rank 1 Number of elements 1020 Type CLASS Element Type: System.Object Fields: None |
So if you spot an array similar to this, you might suspect that it is for holding static references. But how do we figure out what static reference?
In early days of .NET (I mean around .NET 1.0 and 1.1) application had address of this element hardcoded in some part of code. So you could find out actual address of reference and sweep the heap with that address.
First, address of our object 00000000ffddc9b8(System.ServiceModel.ChannelFactoryRefCache1[[MySystem.IService, App_WebReferences.kj6s2mli]]) must be somewhere in the array. Let’s find it:
|
1 2 |
0:046> s -d 00000001ffdb5b08 L?0x1000 00000000ffddc9b8 00000001`ffdb64b0 ffddc9b8 0 ffddca90 0 ................ |
We look for 4-bytes integer (even though we work with x64 application), pass address of our array (second parameter) and range. What we get in return is the address of element in an array which holds the value. In our case it is 00000001`ffdb64b0.
Now we can look for this address in whole application:
|
1 2 |
0:046> s-q 0 L?0x7FFFFFEFFFF 00000001`ffdb64b0 000007ff`002e0120 00000001`ffdb64b0 00000000`00000000 |
We look in user mode partition of our application and get one hit. Unfortunately, it is not the thing we are looking for:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
0:046> !U 000007ff`002e0120 Unmanaged code 000007ff`002e0120 b064 mov al,64h 000007ff`002e0122 db ??? 000007ff`002e0123 ff01 inc dword ptr [rcx] 000007ff`002e0125 0000 add byte ptr [rax],al 000007ff`002e0127 0000 add byte ptr [rax],al 000007ff`002e0129 0000 add byte ptr [rax],al 000007ff`002e012b 0000 add byte ptr [rax],al 000007ff`002e012d 0000 add byte ptr [rax],al 000007ff`002e012f 00d8 add al,bl 000007ff`002e0131 ff ??? |
If this was a .NET 1 application, you would probably get code of some static constructor which you could then examine in code.
So, what can we do next? Well, we can use !dumpdomain to find out which domain holds this array (check start and end address of domain) and then trawl for information in codebase. In my case it was:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
namespace System.ServiceModel { /// <summary> /// Provides the base implementation used to create Windows Communication Foundation (WCF) client objects that can call services. /// </summary> /// <typeparam name="TChannel">The channel to be used to connect to the service.</typeparam> [__DynamicallyInvokable] public abstract class ClientBase<TChannel> : ICommunicationObject, IDisposable where TChannel : class { private static ChannelFactoryRefCache<TChannel> factoryRefCache = new ChannelFactoryRefCache<TChannel>(32); ... } } |
One of my services inherited from this base class. So the question now is: how do I change my application to clean this cache or not use it at all (just to confirm that this is the reason of memory usage). Let’s check MSDN and see that it has CacheSettings which we can use to switch off the cache. Unfortunately, this is accessible only in .NET 4.5 (and I am using .NET 4). So what can we do? Well, use reflection to clear the cache:
|
1 2 3 4 5 6 |
var type = typeof (ClientBase<IService>); var field = type.GetField("factoryRefCache", BindingFlags.Static | BindingFlags.NonPublic); var cache = field.GetValue(null); cache.GetType().GetMethod("Clear").Invoke(cache, new object[0]); GC.Collect(2); |
We need to run this code manually and verify whether the memory usage drops. And it looks like its working.