Entity framework large data set, out of memory exception

c# entity-framework

Question

I'm working with a pretty sizable data set that has about 2 million records. The code below processes around three batches and 600,000 records before throwing an out of memory exception. I can see that since it goes through each batch of lazy loads from the entity framework, which then tries to load all 2 million records into memory. I processed a batch; is there a way to unload it?

ModelContext dbContext = new ModelContext();
IEnumerable<IEnumerable<Town>> towns = dbContext.Towns.OrderBy(t => t.TownID).Batch(200000);
foreach (var batch in towns)
{
    SearchClient.Instance.IndexMany(batch, SearchClient.Instance.Settings.DefaultIndex, "Town", new SimpleBulkParameters() { Refresh = false });
}

Note: This project, https://code.google.com/p/morelinq/, is where the Batch technique is from.

This is the search client: https://github.com/Mpdreamz/NEST

1
29
7/9/2016 8:05:08 AM

Accepted Answer

ZZZ_tmp
74
8/11/2013 7:53:37 AM


Related Questions





Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow