I've a collection with 330000 entities in it. Sometimes when the server is busy to do several things (= things who are consuming also memory) so that the free memory is limited, my process is eating to much memory when saving, causing a out of memory

this.Context.Adapter.SaveEntityCollection(ec, false, false)

--> They are no related entities, so I did set the boolean recursive already to false.

A solution is to do a foreach loop and to use the SaveEntity method : on memory level, I see it takes about the half of the memory taken by the savecollection, but on time level it is more then doubled...

            if (ec.Count <= 200000)
            {
                return this.Context.Adapter.SaveEntityCollection(ec, false, false);          }
            else
            {
                int counter = 0;
                foreach (IEntity2 entity in ec)
                {
                    if (this.Context.Adapter.SaveEntity(entity))
                        counter++;
                }
                return counter;
            }

When the proces is taken out, I receive following error : (it is a printscreen, so I will try to attach it to this topic)

Because of the original design, I can't limit the number of entities in the collection. If I want to do that, I must rewrite completely this module. So I was thinking about a paging mechanism in the savecollection. That will also consuming memory, but maybe less then now.

Any thoughts?

Bulk inserting or bulk update is very memory consuming when using ADO.NET. Do you need a transaction to save the 330,000 entities in the collection?

A better solution would be to save he entities in chunks, 1000 entity at a time. So extract from the big collection the number of entities you need into a new collection and save that one, repeat this till you empty the big collection.

v2.6 has several changes which lower the memory footprint of data and has much lower memory consumption during transactions. So if you can't change the code you're working with, it might be possible to upgrade to v2.6 (which is free for v2.x customers). Please read the v2.6's documentation's migrating your code first to check whether you will run into the breaking changes.

(from the v2.6 what's new page) .Net 2.0+: Much lower memory consumption during transactions. When entities are saved, their state is preserved during the transaction they participate in so if a transaction is rolled back, the entity state is rolled back to the state before it started participating in the transaction. This consumed a lot of memory overhead in previous versions (60% of the memory taken by an entity was necessary for this state preservation). This has been brought down with 90%. (example: 500 test entities in memory occupy 1.6MB of memory. In v2.5, after save they occupy 2.4MB of memory during the transaction. In v2.6, 1.7MB, just 100KB more.)

(but I also would work on doing more in batches, so the total # of entities in the collection is much lower than 330K)

It will be not directly possible to go to the 2.6 version, so I will do the paging mechanism. I thought about it already and Walaa confirmed my thougths

Walaa, before I add the paging mechanism, I do not need a transaction. Can I desactivate the transaction activation and will it help to lower the memory consumption?

I was asking this to make sure batching is easily implemented. Just trying batching them and don't worry about the implicit transaction.