I'm working on a batch process that transfers around 800,000 records from a sluggish old database onto MySQL, which can operate a bit more quickly. I have started loading all of the MySQL entries into RAM in order to optimize this, which reduces use to roughly 200MB. Then I begin updating the records and dumping data from the old database.
Initially, once this finished updating the information, I would use SaveContext, which caused my memory to increase from 500MB to 800MB to 1.5GB. The virtual machine this is operating on has 2GB of RAM, so very quickly I would run into out of memory problems. Even if I were to give it additional RAM, 1.5–2GB is still a touch excessive, and that would only be a temporary fix. In order to fix this, I started calling SaveContext every 10,000 records, which sped up the process a little. Since I was using delegates to fetch the data from the legacy database and update it in MySQL, I didn't experience too much of a performance hit because, after the approximate 5-second wait while it was saving, it would then run through the update in memory for the roughly 3000 backed-up records. The memory use, however, continues increasing.
I may have the following problems:
One potential fix that came to me was to somehow release the memory being used by entities that I know I will never touch again since they have already been changed (like emptying the cache, but just for a single item), but I'm not sure whether that is even feasible with Entity Framework.
Has anybody had any ideas?
The object you no longer need may be sent to the context's Detach function by calling http://msdn.microsoft.com/en-us/library/system.data.objects.objectcontext.detach%28v=vs.90%29.aspx.
I wonder whether using another tool, as was previously advised, or completely forgoing the usage of Entity Framework would be your best option. Instead, if you write the code without using an ORM, you can: