I have a customer who wants a 20,000-record XML file to be imported on a 'all-or-nothing' basis, and therefore I have to put all 20,000 entity objects into a transaction (either directly or adding a collection containing those entities) before I know if I can commit.

The 20,000 records are all for one table - the max db recordsize is in the order of about 3k, which should give me a total memory requirement of 60 Meg plus LLBL and transaction overhead. The process is actually getting up to 900 meg before causing out-of-memory errors to be raised.

Is there any way that I can reduce the amount of memory being used- e.g. is there a switch on the entity that says 'I am just data, don't bother with all the functionality normally needed for an entity'.

It does not seem to make much difference whether I add-transaction and save on each entity, or collect and add-transaction and save on a collection.

Any ideas ?

Many thanks, Paul.

While a bulk insert (bulk copy) would be the normal way to dosuch task, specially if it's a one time shot.

There are some things that you might do to improve the speed of inserts. But first we need to know how were you doing it. (code snippet please.)

If you are using an EntityCollection, you might want to disable DoNotPerformAddIfPresent, also you should use one connection (if you are using a transaction then it holds the connection open already)

Please refer to the following thread for some hints: http://www.llblgen.com/tinyforum/Messages.aspx?ThreadID=3415

The code is similar to the following:

dim dalHoldPosition as HoldPositionEntity '** The entity dim fred as new dataset

fred.readxml(.....)

dim trans as new transaction(IsolationLevel.ReadCommitted, "Import")

for each dr as datarow in fred.tables(0).rows dalHoldPosition = new HoldPositionEntity

if dalHoldPosition.FetchUsingPK(Convert.ToInt64(pidrData("HPOS_IDENTIFIER")), _ Convert.ToDateTime(pidrData("HPOS_VERSION_START_DATE"))) Then Else dalHoldPosition.HPOS_IDENTIFIER = Convert.ToInt64(pidrData("HPOS_IDENTIFIER")) dalHoldPosition.HPOS_VERSION_START_DATE = Convert.ToDateTime(pidrData("HPOS_VERSION_START_DATE"))) End If

if convert.tostring(dr.item("HPOS_TITLE")) = "" then dalHoldPosition.HPOS_TITLE = "Not passed" else dalHoldPosition.HPOS_TITLE = convert.tostring(dr.item("HPOS_TITLE")) end if

'** .... other statements that pass each data item value, or a manipulated value, or a default value into the entity

trans.add(dalHoldPosition) dalHoldPosition.save

next

Instead of the last 3 lines above, have tried with little difference:

colHoldPositions.add(dalHoldPosition) next

trans.add(colHoldPositions)

colHoldPositions.savemulti

Didn't bother with the recurse on either save as the target table has no child tables.

I forgot to mention that I am attacking an Oracle db through ODP, and using SelfService.

The main problem is that the persistence info and field info is in every entity, instead of it being shared among all instances. This is overhead which is currently present in the entities and which will be changed in v2. For now, therefore, the memory footprint can be higher than anticipated, especially in selfservicing. (Adapter has much lower footprint). Normally you won't notice this, but during bulk imports of very large number of objects in a single transaction, you will, also because the entities have to stay in memory before the transaction has been completed.

QUIT IT!!!

You keep teasing us with just enough 2.0 stuff to make us crazy!

Can we expect a beta and/or more formal announcements by the end of the month?

Wayne

WayneBrantley wrote:

QUIT IT!!! You keep teasing us with just enough 2.0 stuff to make us crazy! Can we expect a beta and/or more formal announcements by the end of the month? Wayne

I hoped so but asp.net 2.0 datasourcecontrol work and a reworked code generator configurator took more time than anticipated so we'll slip into april, but not far into april I think.