Debugging WCF high memory usage

afish — Sat, 15 Apr 2017 08:00:20 +0000

If you are interested in the topic see the talk page

Recently I was debugging high memory usage in application written in .NET 4 using WCF. After few hours application was eating almost 8 GB of memory. I made a memory dump and started digging into it.

First, statistics of heap (sorted by total memory usage per type)

!dumpheap -stat										
MT	Count	TotalSize	Class	Name						
000007fecdf6df78	46895258	1500648256,00	System.ServiceModel.Security.MessagePartSpecification							
000007fee98b5b80	14351165	1249823520,00	System.Object[]							
000007fecdf7fea0	253208	1197563136,00	System.Collections.Generic.Dictionary`2+Entry[[System.String,	mscorlib],[System.ServiceModel.Security.MessagePartSpecification,	System.ServiceModel]][]					
000007fee98cfdd0	1220610	588263408,00	System.Byte[]							
000007fee8169780	11719237	468769480,00	System.Collections.Generic.List`1[[System.Xml.XmlQualifiedName,	System.Xml]]						
000007fee98ceeb0	392174	215851704,00	System.Int32[]							
000007fee98c7d90	1917846	186874224,00	System.String							
000007fee815be60	1623002	64920080,00	System.Xml.NameTable+Entry							
000007feccd07a70	538933	60360496,00	System.Xml.XmlBaseReader+XmlElementNode							
000007fee8160860	787684	56713248,00	System.Xml.XmlName							
000007feccd07700	1285725	51429000,00	System.Xml.StringHandle							
000007feccd07680	1285725	51429000,00	System.Xml.PrefixHandle							
000007feccd07ae8	538933	47426104,00	System.Xml.XmlBaseReader+XmlEndElementNode							
000007fee98cf8b0	102864	35425872,00	System.Collections.Hashtable+bucket[]							
000007fee8160298	598925	33539800,00	System.Xml.XmlElement							
000007fecdf7f8b8	378220	33283360,00	System.Collections.Generic.Dictionary`2[[System.String,	mscorlib],[System.ServiceModel.Security.MessagePartSpecification,	System.ServiceModel]]					
000007feccd07780	814117	32564680,00	System.Xml.ValueHandle							
000007fecdf34d40	756104	30244160,00	System.Collections.Generic.List`1[[System.ServiceModel.Security.Tokens.SecurityTokenParameters,	System.ServiceModel]]						
000007fecd04c3d8	62954	29714288,00	System.IdentityModel.SamlDictionary							
...

So we can see, that there are over 45M of System.ServiceModel.Security.MessagePartSpecification objects. Let’s see one of them:

!dumpheap -type System.ServiceModel.Security.MessagePartSpecification		
...
000000010a5745f0	000007fecdf7fea0	4752
...

0:046> !gcroot 000000010a5745f0											
Note: Roots found on stacks may be false positives. Run "!help gcroot" for											
more info.											
Scan Thread 7 OSTHread 2dcc											
Scan Thread 29 OSTHread 1084											
Scan Thread 4 OSTHread 2c60											
Scan Thread 30 OSTHread eb0											
Scan Thread 31 OSTHread 21c4											
Scan Thread 19 OSTHread 2e60											
Scan Thread 32 OSTHread 2d24											
Scan Thread 33 OSTHread 2840											
Scan Thread 34 OSTHread 788											
Scan Thread 35 OSTHread 145c											
Scan Thread 36 OSTHread 1b80											
Scan Thread 37 OSTHread 1898											
Scan Thread 38 OSTHread 2600											
Scan Thread 40 OSTHread 216c											
Scan Thread 41 OSTHread 2bec											
Scan Thread 42 OSTHread fa8											
Scan Thread 43 OSTHread 1ebc											
Scan Thread 44 OSTHread e28											
Scan Thread 45 OSTHread 2ef8											
Scan Thread 47 OSTHread 2418											
Scan Thread 46 OSTHread 2934											
DOMAIN(00000000067B1EA0):HANDLE(Pinned):2dc13e0:Root:00000001ffdb5b08(System.Object[])->											
00000000ffddc9b8(System.ServiceModel.ChannelFactoryRefCache`1[[MySystem.IService, App_WebReferences.kj6s2mli]])->											
00000001dfe63928(System.ServiceModel.ChannelFactoryRef`1[[MySystem.IService, App_WebReferences.kj6s2mli]])->											
00000001dfdebe28(System.ServiceModel.ChannelFactory`1[[MySystem.IService, App_WebReferences.kj6s2mli]])->											
00000001dff06140(System.ServiceModel.Channels.ServiceChannelFactory+ServiceChannelFactoryOverRequestSession)->											
00000001dff05b50(System.ServiceModel.Channels.SecurityChannelFactory`1[[System.ServiceModel.Channels.IRequestSessionChannel, System.ServiceModel]])->											
00000001dff05c50(System.ServiceModel.Channels.HttpChannelFactory)->											
00000001dff05d78(System.ServiceModel.Channels.CommunicationObjectManager`1[[System.ServiceModel.Channels.IChannel, System.ServiceModel]])->											
00000001dff05db0(System.Collections.Hashtable)->											
000000020fda8130(System.Collections.Hashtable+bucket[])->											
000000010a555568(System.ServiceModel.Channels.HttpChannelFactory+HttpRequestChannel)->											
000000010a589108(System.EventHandler)->											
000000010a5556b8(System.ServiceModel.Channels.ReliableChannelBinder`1+ChannelSynchronizer[[System.ServiceModel.Channels.IRequestChannel, System.ServiceModel]])->											
000000010a5554e0(System.ServiceModel.Channels.ClientReliableChannelBinder`1+RequestClientReliableChannelBinder[[System.ServiceModel.Channels.IRequestChannel, System.ServiceModel]])->											
000000010a555490(System.ServiceModel.Channels.ChannelParameterCollection)->											
000000010a555258(System.ServiceModel.Security.SecuritySessionClientSettings`1+SecurityRequestSessionChannel[[System.ServiceModel.Channels.IRequestSessionChannel, System.ServiceModel]])->											
000000010a556450(System.ServiceModel.Security.SecuritySessionSecurityTokenProvider)->											
000000010a564e20(System.ServiceModel.Channels.SecurityChannelFactory`1[[System.ServiceModel.Channels.IRequestChannel, System.ServiceModel]])->											
000000010a5578d8(System.ServiceModel.Security.SymmetricSecurityProtocolFactory)->											
000000010a557a00(System.ServiceModel.Security.ChannelProtectionRequirements)->											
000000010a557b88(System.ServiceModel.Security.ScopedMessagePartSpecification)->											
000000010a570a10(System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[System.ServiceModel.Security.MessagePartSpecification, System.ServiceModel]])->											
000000010a5745f0(System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.ServiceModel.Security.MessagePartSpecification, System.ServiceModel]][])

So we can see that it is ultimately rooted in some pinned array of objects. What’s that? This array holds static variables from app domain. It usually looks as follows:

0:046>	!do	00000001ffdb5b08						
Name:	System.Object[]							
MethodTable:	000007fee98b5b80							
EEClass:	000007fee94ceb88							
Size:	8192(0x2000)	bytes						
Array:	Rank	1	Number	of	elements	1020	Type	CLASS
Element	Type:	System.Object						
Fields:								
None

So if you spot an array similar to this, you might suspect that it is for holding static references. But how do we figure out what static reference?

In early days of .NET (I mean around .NET 1.0 and 1.1) application had address of this element hardcoded in some part of code. So you could find out actual address of reference and sweep the heap with that address.

First, address of our object 00000000ffddc9b8(System.ServiceModel.ChannelFactoryRefCache1[[MySystem.IService, App_WebReferences.kj6s2mli]]) must be somewhere in the array. Let’s find it:

0:046>	s -d	00000001ffdb5b08	L?0x1000	00000000ffddc9b8					
00000001`ffdb64b0	ffddc9b8	0	ffddca90	0	................

We look for 4-bytes integer (even though we work with x64 application), pass address of our array (second parameter) and range. What we get in return is the address of element in an array which holds the value. In our case it is 00000001`ffdb64b0.

Now we can look for this address in whole application:

0:046> s-q 0 L?0x7FFFFFEFFFF 00000001`ffdb64b0						
000007ff`002e0120  00000001`ffdb64b0 00000000`00000000

We look in user mode partition of our application and get one hit. Unfortunately, it is not the thing we are looking for:

0:046> !U 000007ff`002e0120				
Unmanaged code				
000007ff`002e0120 b064            mov     al,64h				
000007ff`002e0122 db              ???				
000007ff`002e0123 ff01            inc     dword ptr [rcx]				
000007ff`002e0125 0000            add     byte ptr [rax],al				
000007ff`002e0127 0000            add     byte ptr [rax],al				
000007ff`002e0129 0000            add     byte ptr [rax],al				
000007ff`002e012b 0000            add     byte ptr [rax],al				
000007ff`002e012d 0000            add     byte ptr [rax],al				
000007ff`002e012f 00d8            add     al,bl				
000007ff`002e0131 ff              ???

If this was a .NET 1 application, you would probably get code of some static constructor which you could then examine in code.

So, what can we do next? Well, we can use !dumpdomain to find out which domain holds this array (check start and end address of domain) and then trawl for information in codebase. In my case it was:

namespace System.ServiceModel
{
  /// 
  /// Provides the base implementation used to create Windows Communication Foundation (WCF) client objects that can call services.
  /// 
  /// The channel to be used to connect to the service.
  [__DynamicallyInvokable]
  public abstract class ClientBase : ICommunicationObject, IDisposable where TChannel : class
  {
    private static ChannelFactoryRefCache factoryRefCache = new ChannelFactoryRefCache(32);
    ...
  }
}

One of my services inherited from this base class. So the question now is: how do I change my application to clean this cache or not use it at all (just to confirm that this is the reason of memory usage). Let’s check MSDN and see that it has CacheSettings which we can use to switch off the cache. Unfortunately, this is accessible only in .NET 4.5 (and I am using .NET 4). So what can we do? Well, use reflection to clear the cache:

var type = typeof (ClientBase);
var field = type.GetField("factoryRefCache", BindingFlags.Static | BindingFlags.NonPublic);
var cache = field.GetValue(null);
cache.GetType().GetMethod("Clear").Invoke(cache, new object[0]);

GC.Collect(2);

We need to run this code manually and verify whether the memory usage drops. And it looks like its working.

WCF – Random IT Utensils

Debugging WCF high memory usage