Possible in any mode (Adapter or Self...) fetching big collection (~50K records) like DataReader or DB_Cursor ? Not loading all records into array - but get continious access forward-only like cursors. Somethink like this:

MyEntityCollection forwardOnlyEntitiesCollection = ......;
foreach(MyEntity entity in forwardOnlyEntitiesCollection)
{
    if (entity.Field1 == xxxx)
    {
        someEntityProcessing(entity);
        entity.Save(); // or adapter.Save(entity);  
    }
}

I can't find simple example like that in Doc's - only examples with reader.Get(0) - without entity binding. I'm try execute similar code with entityParent.GetMultiEntity(xxx) - but programm hang on this line when size of collection was about 50K records - seems LLBLgen try load all records into memory - present different metod for read lot of data ?

P.S. Sorry for my English.

Hi there,

You first need to fetch with FetchDataReader (http://llblgen.com/TinyForum/Messages.aspx?ThreadID=10650&StartAtMessage=0&#59414)

then project the results onto a entityCollection(http://llblgen.com/TinyForum/Messages.aspx?ThreadID=13935&StartAtMessage=0&#77616)

I'ts almost what need - but always when need I must write 10-20 lines of code instead of simple 1-2 like

foreach(... in DataReader)
{
}

or

reader = myObject.GetDataReaderCollection();
do
....
while(reader.read())

? I just want read simple [select * from T_MY_OBJECT] like a cursor or datareader with mapping each record to associated by default class. No another way ?

You may also use paging to fetch page by page, while setting the page size according to your needs..

You need to specify which fields you want in the resultset. If you want all fields of an entity, use: (adapter) List<IEntityPropertyProjector> projectors = EntityFields2.ConvertToProjectors(entityFactory.CreateFields()); and then the few lines to setup the projection, which you can wrap with a utility method you write yourself.

I'm almost solve my question

            IRetrievalQuery query = RetrievalProcedures.GetGetMyObjectCallAsQuery();
            IDataReader reader = adapter.FetchDataReader(query, CommandBehavior.Default);
            while (reader.Read())
            {
                Console.WriteLine("Row: {0} | {1} | {2} | {3} |", reader.GetValue(0),
                    reader.GetValue(1), reader.GetValue(2), reader.GetValue(3));
                
            }

work great - exactly as need (I'm no need fetch al collection to memory - need only current element), but how easy project current record - [reader] in loop - into my SINGLE object ObjectEntity ? Something like

            MyObjectEntity currentObject;
            while (reader.Read())
            {
                currentObject = ProjectReaderToEntity<MyObjectEntity>(reader);
                currentObject.Field1 = ..........
            }

Use FetchProjection() Please check the example posted in the docs, Projecting Stored Procedure resultset onto entity collection

eg.

        using(IDataReader reader = adapter.FetchDataReader(query, CommandBehavior.CloseConnection))
        {
            // first resultset: Customers.
            List<IDataValueProjector> valueProjectors = new List<IDataValueProjector>();
            // project value on index 0 in resultset row onto CustomerId
            valueProjectors.Add( new DataValueProjector( CustomerFieldIndex.CustomerId.ToString(), 0, typeof( string ) ) );
            // project value on index 1 in resultset row onto CompanyName
            valueProjectors.Add( new DataValueProjector( CustomerFieldIndex.CompanyName.ToString(), 1, typeof( string ) ) );
            // resultset contains more rows, we just project those 2. The rest is trivial.
            DataProjectorToIEntityCollection2 projector = new DataProjectorToIEntityCollection2( customers );
            adapter.FetchProjection( valueProjectors, projector, reader );

            // second resultset: Orders. 
            valueProjectors = new ArrayList();
            valueProjectors.Add( new DataValueProjector( OrderFieldIndex.OrderId.ToString(), 0, typeof( int ) ) );
            valueProjectors.Add( new DataValueProjector( OrderFieldIndex.CustomerId.ToString(), 1, typeof( string ) ) );
            valueProjectors.Add( new DataValueProjector( OrderFieldIndex.OrderDate.ToString(), 3, typeof( DateTime ) ) );
            // switch to the next resultset in the datareader
            reader.NextResult();
            projector = new DataProjectorToIEntityCollection2( orders );
            adapter.FetchProjection( valueProjectors, projector, reader );
            reader.Close();
        }

I see this example - but present conceptual difference: in example main action is fetching dataReader into some collection - and as result we will have all records in memory as collection with random-access. If I have about 1 million records in result of query - no need store all records in memory at one moment - need continuous access.

Open [dataReader], read next record, map to myObject (myObject - it's entity, mapped to some table), process [myObject], maybe save [myObject], call reader.ReadNext(), repeat.

I'ts need for processing lot of data in continuous mode - process records one-by-one. It's possible ?

Then I guess the answer is using a DataReader with paging (you may set pagesize = 1).

Additionally, if you want to persist the saved data, you might want to do that in a transaction? (i.e. roll back persisted rows if something fails).

I'd then do it batch-wise: fetch 1000 rows with the reader, process them, persist the entities, and repeat with the next 1000 (using paging by specifying pagesize = 1000 and pagenumber is 2 etc.

The projector used, DataProjectorToIEntityCollection2, is just a simple class, as it implements an interface which is easy to implement yourself. I have copied the projector code below. You can base your projector onto this code and for example call out to your process logic by passing the entity, and let it handle it from there. (source is from the runtime lib sourcecode you get access to under a modified BSD2 license once you're a licensee). Our linq provider also uses this class in special cases.

/// <summary>
/// Projector engine which projects raw projection result data onto new entities which are added to a single entitycollection.
/// </summary>
/// <remarks>Adapter specific</remarks>
public class DataProjectorToIEntityCollection2 : IEntityDataProjector, IGeneralDataProjector    
{
    #region Class Member Declarations
    private IEntityCollection2 _destination;
    private Dictionary<string, PropertyDescriptor> propertyDescriptors;
    #endregion

    /// <summary>
    /// Initializes a new instance of the <see cref="DataProjectorToIEntityCollection2"/> class.
    /// </summary>
    /// <param name="destination">The destination of the data. It's required that the destination collection has a factory set.</param>
    public DataProjectorToIEntityCollection2(IEntityCollection2 destination )
    {
        if( destination == null )
        {
            throw new ArgumentNullException( "destination", "destination can't be null" );
        }
        if( destination.EntityFactoryToUse == null )
        {
            throw new ArgumentException( "destination doesn't have an EntityFactory set.", "destination" );
        }

        _destination = destination;
        IEntity2 newInstance = _destination.EntityFactoryToUse.Create();
        PropertyDescriptorCollection properties = TypeDescriptor.GetProperties( newInstance );

        propertyDescriptors = new Dictionary<string, PropertyDescriptor>( properties.Count );
        foreach( PropertyDescriptor descriptor in properties )
        {
            propertyDescriptors.Add( descriptor.Name, descriptor );
        }
    }

    
    /// <summary>
    /// Adds a new projection result to the container contained into this instance. The container has to be set in the constructor.
    /// </summary>
    /// <param name="propertyProjectors">List of property projectors used to create the projection result</param>
    /// <param name="rawProjectionResult">The raw projection result.</param>
    void IEntityDataProjector.AddProjectionResultToContainer( List<IEntityPropertyProjector> propertyProjectors, object[] rawProjectionResult )
    {
        this.AddProjectionResultToContainer( propertyProjectors, rawProjectionResult );
    }


    /// <summary>
    /// Adds a new projection result to the container contained into this instance. The container has to be set in the constructor.
    /// </summary>
    /// <param name="valueProjectors">List of value projectors used to create the projection result</param>
    /// <param name="rawProjectionResult">The raw projection result.</param>
    void IGeneralDataProjector.AddProjectionResultToContainer( List<IDataValueProjector> valueProjectors, object[] rawProjectionResult )
    {
        this.AddProjectionResultToContainer( valueProjectors, rawProjectionResult );
    }


    /// <summary>
    /// Performs the actual projection 
    /// </summary>
    /// <param name="projectors">List of projectors used to create the projection result</param>
    /// <param name="rawProjectionResult">The raw projection result.</param>
    private  void AddProjectionResultToContainer( IList projectors, object[] rawProjectionResult )
    {
        IEntity2 newInstance = _destination.EntityFactoryToUse.Create();
        bool fieldSet = false;
        // set fields. This is done through property descriptors. 
        for( int i = 0; i < projectors.Count; i++ )
        {
            IProjector projector = (IProjector)projectors[i];
            IEntityField2 field = newInstance.Fields[projector.ProjectedResultName];
            object value = rawProjectionResult[i];
            if((value != null) && value.Equals(DBNull.Value))
            {
                value = null;
            }
            if(field == null)
            {
                // set via property descriptor
                PropertyDescriptor property = null;
                if( propertyDescriptors.TryGetValue( projector.ProjectedResultName, out property ) )
                {
                    property.SetValue(newInstance, value);
                    fieldSet = true;
                }
            }
            else
            {
                // set through field object.
                if( field.IsReadOnly )
                {
                    // just set the value, don't set dirty flags.
                    field.ForcedCurrentValueWrite(value);
                    fieldSet = true;
                }
                else
                {
                    fieldSet = newInstance.SetNewFieldValue(field.FieldIndex, value);
                }
            }
        }


        if(!fieldSet && (rawProjectionResult.Length == 1) && rawProjectionResult[0].GetType().IsAssignableFrom(newInstance.GetType()))
        {
            // special case: m:1/1:1 related entity is added. This ends up in this projection routine because the end result is
            // a collection of entities. Typical example: from o in md.Orders select o.Customer;
            _destination.Add((IEntity2)rawProjectionResult[0]);
        }
        else
        {
            _destination.Add(newInstance);
        }
    }
}