Creating instance of Entity Framework Context slows down under load

asp.net c# entity-framework entity-framework-6 performance

Question

A few very minor web service calls took far longer than we had anticipated. We conducted some research, set up several timers, and eventually settled on generating an instance of our Entity Framework 6 DbContext. merely the process of creating the context, not the query itself. Since then, I've added some logging to check how long it really does take to actually construct an instance of our DbContext, and it seems to be approximately 50ms.

Context creation is not sluggish after the program has warmed up. Following an app recycle, it begins at 2-4ms (which is what we see in our dev environments). Over time, context generation seems to slacken. It will gradually increase to the 50-80 ms range during the following several hours before leveling out.

Around 300 entities make up our context, which is a sizable code-first context with some quite intricate interactions between some of the elements. Our version of EF is 6.1.3. Although we follow the rule of "one context per request," the majority of our web API requests only execute one or two queries. A 1ms query execution time after a context creation time of 60+ ms is a little disappointing. We get roughly 10,000 inquiries each minute, therefore our website isn't seldom visited.

Here's a quick overview of what we observe. The major drop in the MS is a deploy that regenerated the app domain. There are 4 distinct web servers, one per each line. You'll see that it's not always the same server.

enter image description here

I conducted a memory dump to attempt to figure out what was going on, and the heap statistics are as follows:

00007ffadddd1d60    70821      2266272 System.Reflection.Emit.GenericFieldInfo
00007ffae02e88a8    29885      2390800 System.Linq.Enumerable+WhereSelectListIterator`2[[NewRelic.Agent.Core.WireModels.MetricDataWireModel, NewRelic.Agent.Core],[System.Single, mscorlib]]
00007ffadda7c1a0     1462      2654992 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Object, mscorlib],[System.Object, mscorlib]][]
00007ffadd4eccf8    83298      2715168 System.RuntimeType[]
00007ffadd4e37c8    24667      2762704 System.Reflection.Emit.DynamicMethod
00007ffadd573180    30013      3121352 System.Web.Caching.CacheEntry
00007ffadd2dc5b8    35089      3348512 System.String[]
00007ffadd6734b8    35233      3382368 System.RuntimeMethodInfoStub
00007ffadddbf0a0    24667      3749384 System.Reflection.Emit.DynamicILGenerator
00007ffae04491d8    67611      4327104 System.Data.Entity.Core.Metadata.Edm.MetadataProperty
00007ffadd4edaf0    57264      4581120 System.Signature
00007ffadd4dfa18   204161      4899864 System.RuntimeMethodHandle
00007ffadd4ee2c0    41900      5028000 System.Reflection.RuntimeParameterInfo
00007ffae0c9e990    21560      5346880 System.Data.SqlClient._SqlMetaData
00007ffae0442398    79504      5724288 System.Data.Entity.Core.Metadata.Edm.TypeUsage
00007ffadd432898    88807      8685476 System.Int32[]
00007ffadd433868     9985      9560880 System.Collections.Hashtable+bucket[]
00007ffadd4e3160    92105     10315760 System.Reflection.RuntimeMethodInfo
00007ffadd266668   493622     11846928 System.Object
00007ffadd2dc770    33965     16336068 System.Char[]
00007ffadd26bff8   121618     17335832 System.Object[]
00007ffadd2df8c0   168529     68677312 System.Byte[]
00007ffadd2d4d08   581057    127721734 System.String
0000019cf59e37d0   166894    143731666      Free
Total 5529765 objects
Fragmented blocks larger than 0.5 MB:
            Addr     Size      Followed by
0000019ef63f2140    2.9MB 0000019ef66cfb40 Free
0000019f36614dc8    2.8MB 0000019f368d6670 System.Data.Entity.Core.Query.InternalTrees.SimpleColumnMap[]
0000019f764817f8    0.8MB 0000019f76550768 Free
0000019fb63a9ca8    0.6MB 0000019fb644eb38 System.Data.Entity.Core.Common.Utils.Set`1[[System.Data.Entity.Core.Metadata.Edm.EntitySet, EntityFramework]]
000001a0f6449328    0.7MB 000001a0f64f9b48 System.String
000001a0f65e35e8    0.5MB 000001a0f666e2a0 System.Collections.Hashtable+bucket[]
000001a1764e8ae0    0.7MB 000001a17659d050 System.RuntimeMethodHandle
000001a3b6430fd8    0.8MB 000001a3b6501aa0 Free
000001a4f62c05c8    0.7MB 000001a4f636e8a8 Free
000001a6762e2300    0.6MB 000001a676372c38 System.String
000001a7761b5650    0.6MB 000001a776259598 System.String
000001a8763c4bc0    2.3MB 000001a8766083a8 System.String
000001a876686f48    1.4MB 000001a8767f9178 System.String
000001a9f62adc90    0.7MB 000001a9f63653c0 System.String
000001aa362b8220    0.6MB 000001aa36358798 Free

That looks like a lot of type use and information.

Attempts we've made:

  1. making a straightforward test harness to duplicate. My best opinion is that it failed since neither the traffic nor the kind of queries being conducted were varied. The time didn't rise after just loading the context and doing a few queries again.
  2. The context has been drastically reduced from 500 entities to only 300. didn't affect speed in any way. We weren't utilizing any of those 200 entities, in my estimation.
  3. (Edit) To generate our "context per request," we use SimpleInjector. I created a Context instance by just new'in it up to make sure it isn't a SimpleInjector. same sluggish creation times
  4. (Edit) EF has been ngen'd. failed to have any effect.

What more can we look into? I am aware that EF uses a large cache to speed up operations. Does the construction of context take longer when there is more in the cache? Is there a method to determine what is included in the cache precisely so that any oddities can be explained? Does anybody have any particular suggestions for accelerating context creation?

Revision: 5/30/17

I created our own version using the EF6 code to include certain timings. Since we operate a quite busy site, gathering a ton of timing information is challenging, and I didn't go as far as I had hoped, but in general, we discovered that all of the slowing is caused by this approach.

public void ForceOSpaceLoadingForKnownEntityTypes()
    {
        if (!_oSpaceLoadingForced)
        {
            // Attempting to get o-space data for types that are not mapped is expensive so
            // only try to do it once.
            _oSpaceLoadingForced = true;

            Initialize();
            foreach (var set in _genericSets.Values.Union(_nonGenericSets.Values))
            {
                set.InternalSet.TryInitialize();
            }
        }
    }

For each entity specified by a DBSet in our context, that foreach loop iterates once. Each repeat lasts just a few seconds. 1-.3 ms, however adding the 254 entities we had makes a difference. We still don't understand why it starts off quickly and then slows down.

1
23
6/9/2017 8:41:45 AM

Popular Answer

I would start here, without switching to a more enterprise-friendly solution, to solve the issue.

Our context is a fairly large code-first context with around 300 entities 

Despite the fact that EF has significantly improved over time, I would still begin to consider breaking things up once you reach 100 entities (really, I would start much earlier, but that appears to be a figure that many people have indicated is magical; is there a consensus?). Why not think of it as designing for "domains" as opposed to "contexts"? In this approach, might you persuade senior executives that you are using "domain driven design" to improve the application? You may be building for "microservices" in the future if you utilise two buzzwords in one paragraph.

I prefer to stay away from EF for big scale or high performance applications since I am not a great fan of it in the enterprise area. Your mileage may vary. It is probably just fine for SMB. However, I do encounter customers who make advantage of it.

The following concepts may not be entirely current, but they are some others I would take into account based on my experience.

  • Plan out your opinions. They are the component of the query that costs the most. With huge models, this will be much more helpful.
  • Put your model in a different assembly. My pet annoyance with regard to the arrangement of code is more of a performance issue.
  • Examine the caching potential of your application and model. Query plan caching often allows for significant time savings.
  • CompileQuery is used.
  • Use NoTracking when it is practical. If you don't need the functionality, you will save a tonne of money.

I'm going to presume that because you seem to be running some kind of application profiler, you have also examined your SQL queries and any potential speed improvements. Yes, I am aware it is not the issue you are trying to address, but from the user's viewpoint, it may add to the overall problem.

In regard to @WiktorZichia's remark that he didn't address the performance issue, I would say that eliminating Entity Framework is the greatest solution for solving issues of this kind in an Enterprise System. Every choice has trade-offs. The excellent abstraction known as EF facilitates development. However, it has some pointless overhead that might harm large-scale systems. Technically speaking, however, I still did not provide a solution to the issue of "how can I solve this problem the way I am attempting to solve it," therefore this might still be considered a failure.

3
5/9/2017 4:40:57 PM


Related Questions





Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow