Entity framework remove object from context, but not from database

c# entity-framework memory-management objectcontext out-of-memory

Question

I'm working on a batch process that transfers around 800,000 records from a sluggish old database onto MySQL, which can operate a bit more quickly. I have started loading all of the MySQL entries into RAM in order to optimize this, which reduces use to roughly 200MB. Then I begin updating the records and dumping data from the old database.

Initially, once this finished updating the information, I would use SaveContext, which caused my memory to increase from 500MB to 800MB to 1.5GB. The virtual machine this is operating on has 2GB of RAM, so very quickly I would run into out of memory problems. Even if I were to give it additional RAM, 1.5–2GB is still a touch excessive, and that would only be a temporary fix. In order to fix this, I started calling SaveContext every 10,000 records, which sped up the process a little. Since I was using delegates to fetch the data from the legacy database and update it in MySQL, I didn't experience too much of a performance hit because, after the approximate 5-second wait while it was saving, it would then run through the update in memory for the roughly 3000 backed-up records. The memory use, however, continues increasing.

I may have the following problems:

  • I am unable to chunk the changes and sporadically release the ObjectContext since the data is retrieved from the old database in any order.
  • It is exceedingly sluggish if I don't get all of the data out of MySQL beforehand and instead look it up record-by-record as I update the database. Instead, I gather it all at once, cast it to a dictionary that is indexed by the main key, and then delete entries from the dictionary as I update the data.

One potential fix that came to me was to somehow release the memory being used by entities that I know I will never touch again since they have already been changed (like emptying the cache, but just for a single item), but I'm not sure whether that is even feasible with Entity Framework.

Has anybody had any ideas?

1
7
6/29/2012 12:19:09 AM

Accepted Answer

The object you no longer need may be sent to the context's Detach function by calling http://msdn.microsoft.com/en-us/library/system.data.objects.objectcontext.detach%28v=vs.90%29.aspx.

6
6/29/2012 1:15:04 AM

Popular Answer

I wonder whether using another tool, as was previously advised, or completely forgoing the usage of Entity Framework would be your best option. Instead, if you write the code without using an ORM, you can:

  1. Enhance the SQL statements' performance.
  2. To get the greatest performance, you may easily manage and modify the scope of transactions.
  3. The changes may be done in batches so that you don't have to contact the server more than once to complete them all.


Related Questions





Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow