Debugging WCF high memory usage

If you are interested in the topic see the talk page

Recently I was debugging high memory usage in application written in .NET 4 using WCF. After few hours application was eating almost 8 GB of memory. I made a memory dump and started digging into it.

First, statistics of heap (sorted by total memory usage per type)

So we can see, that there are over 45M of System.ServiceModel.Security.MessagePartSpecification objects. Let’s see one of them:

So we can see that it is ultimately rooted in some pinned array of objects. What’s that? This array holds static variables from app domain. It usually looks as follows:

So if you spot an array similar to this, you might suspect that it is for holding static references. But how do we figure out what static reference?

In early days of .NET (I mean around .NET 1.0 and 1.1) application had address of this element hardcoded in some part of code. So you could find out actual address of reference and sweep the heap with that address.

First, address of our object 00000000ffddc9b8(System.ServiceModel.ChannelFactoryRefCache1[[MySystem.IService, App_WebReferences.kj6s2mli]]) must be somewhere in the array. Let’s find it:

We look for 4-bytes integer (even though we work with x64 application), pass address of our array (second parameter) and range. What we get in return is the address of element in an array which holds the value. In our case it is 00000001`ffdb64b0.

Now we can look for this address in whole application:

We look in user mode partition of our application and get one hit. Unfortunately, it is not the thing we are looking for:

If this was a .NET 1 application, you would probably get code of some static constructor which you could then examine in code.

So, what can we do next? Well, we can use !dumpdomain to find out which domain holds this array (check start and end address of domain) and then trawl for information in codebase. In my case it was:

One of my services inherited from this base class. So the question now is: how do I change my application to clean this cache or not use it at all (just to confirm that this is the reason of memory usage). Let’s check MSDN and see that it has CacheSettings which we can use to switch off the cache. Unfortunately, this is accessible only in .NET 4.5 (and I am using .NET 4). So what can we do? Well, use reflection to clear the cache:

We need to run this code manually and verify whether the memory usage drops. And it looks like its working.