Exception of type 'System.OutOfMemoryException' was thrown

Question

Exception of type 'System.OutOfMemoryException' was thrown

c# c#-4.0 ef-code-first entity-framework sql-server

Question

Basically I use Entity Framework to query a huge database. I want to return a string list then log it to a text file.

List<string> logFilePathFileName = new List<string>();
var query = from c in DBContext.MyTable where condition = something select c;
foreach (var result in query)
{
    filePath = result.FilePath;
    fileName = result.FileName;
    string temp = filePath + "." + fileName;
    logFilePathFileName.Add(temp);
    if(logFilePathFileName.Count %1000 ==0)
        Console.WriteLine(temp+"."+logFilePathFileName.Count);
}

However I got an exception when logFilePathFileName.Count=397000. The exception is:

Exception of type 'System.OutOfMemoryException' was thrown.

A first chance exception of type 'System.OutOfMemoryException' occurred in System.Data.Entity.dll

UPDATE:

What I want to use a different query say: select top 1000 then add to the list, but I don't know after 1000 then what?

user1108948

Popular Answer

Why are you collecting the data in a List<string> if all you need to do is write it to a text file?

You might as well just:

Open the text file;
Iterate over the records, appending each string to the text file (without storing the strings in memory);
Flush and close the text file.

You will need far less memory than now, because you won't be keeping all those strings unnecessarily in memory.

Roy Dictus

View more on Stack Overflow

Tigran · Accepted Answer · 1/30/2013 3:16:46 PM

Most probabbly it's not about a RAM as is, so increasing your RAM or even compiling and running your code in 64 bit machine will not have a positive effect, in this case.

I think it's related to a fact that .NET collections are limited to maximum 2GB RAM space (no difference either 32 or 64 bit).

To resolve this, split your list to much smaller chunks and most probabbly your problem will gone.

Just one possible solution:

foreach (var result in query)
{
    ....
    if(logFilePathFileName.Count %1000 ==0) {
        Console.WriteLine(temp+"."+logFilePathFileName.Count);
        //WRITE SOMEWHERE YOU NEED 
        logFilePathFileName = new List<string>(); //RESET LIST !|
    }
}

EDIT

If you want fragment a query, you can use Skip(...) and Take(...)

Just an explanatory example:

var fisrt1000 = query.Skip(0).Take(1000);
var second1000 = query.Skip(1000).Take(1000);

... and so on..

Naturally put it in your iteration and parametrize it based on bounds of data you know or need.

Exception of type 'System.OutOfMemoryException' was thrown

Question

Fastest Entity Framework Extensions

Accepted Answer

Popular Answer

Performance

Expression Evaluator

Related