Large # of objects - memory problem.

Posts   
 
    
miloszes
User
Posts: 222
Joined: 03-Apr-2007
# Posted on: 16-Jul-2010 12:46:12   

Hi,

we have a LLBLGen memory issue, when trying to fetch 200 000 it consumes a large amout of memory (more then 4 GB RAM). In that case a System page file is used ad it also consumes a lot of time to execute.

I had a similar problem with V2, but find out that the same issue exists in V3. I've tried to define a excluded fields lists but it seams that memory is still consumed at high level.

1) Why it consumes so much memory?

2) Is it possible to fetch some POCO objects?

3) Is it possible to access direclty a datareader ?

Kind Regards, MiloszeS

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39903
Joined: 17-Aug-2003
# Posted on: 16-Jul-2010 12:53:42   

Do you fetch things in a transaction? If so, try to fetch outside a transaction.

The memory consumed by an object is bigger than the data it holds, as other objects are contained in it, some meta-data it keeps around etc. In general we have kept memory consumption to a minimum in v2.6 and kept that in v3.0. Still things can eat a lot of memory, e.g. when you have large string fields which are unique. We do re-use string objects btw if they're duplicates.

What I'd advice you is to use paging instead, and use pages to consume parts of the resultset in your system. This is more efficient as you don't have a large resultset on the server and also not in memory.

To disable the string cache, try to set StringCacheForFetcher in the dataaccessadapter (it's a protected property) to null.

You can read data using a datareader, see: http://www.llblgen.com/documentation/3.0/LLBLGen%20Pro%20RTF/hh_goto.htm#Using%20the%20generated%20code/Adapter/gencode_datareadersprojections_adapter.htm

Frans Bouma | Lead developer LLBLGen Pro