We have a utility which fetches and processes records. We process the entities and any related child entities in chunks of 500; each chunk is fetched/processed on its own thread and each thread has its own adapter. We're sporadically seeing the error below - but interestingly, only when using multiple threads. Any ideas on what could be causing it?

Some of the child entities are setup with inheritance. We have the SimpleRetryRecoveryStrategy implemented. We're using LLBLGen 5.10.1.

The following exception occurred at Int32 CalculateHashForEntityBasedOnRelation(SD.LLBLGen.Pro.ORMSupportClasses.IEntityCore, SD.LLBLGen.Pro.ORMSupportClasses.IEntityRelation, Boolean, Boolean ByRef): Object reference not set to an instance of an object.

Stack Trace:
   at SD.LLBLGen.Pro.ORMSupportClasses.PersistenceCore.CalculateHashForEntityBasedOnRelation(IEntityCore entity, IEntityRelation relation, Boolean forPkSide, Boolean& hashRejected)
   at SD.LLBLGen.Pro.ORMSupportClasses.PersistenceCore.MergeNormal(IEntityCollectionCore rootEntities, IPrefetchPathElementCore currentElement, Boolean rootEntitiesArePkSide)
   at SD.LLBLGen.Pro.ORMSupportClasses.PrefetchPathFetcher.FetchPrefetchPath(QueryParameters rootNodeParameters, Boolean forceParameterizedPPath, ITransaction transactionToUse, Int32 parameterisedPrefetchPathThreshold, Action`1 fetchNodeFunc, Action`4 mergeManyToManyFunc)
   at SD.LLBLGen.Pro.ORMSupportClasses.PersistenceCore.FetchPrefetchPath(QueryParameters rootNodeParameters, Boolean forceParameterizedPPath, ITransaction transactionToUse, Int32 parameterisedPrefetchPathThreshold, Action`1 fetchNodeFunc, Action`4 mergeManyToManyFunc)
   at SD.LLBLGen.Pro.ORMSupportClasses.PrefetchPathFetcher.FetchPrefetchPath(QueryParameters rootNodeParameters, Boolean forceParameterizedPPath, ITransaction transactionToUse, Int32 parameterisedPrefetchPathThreshold, Action`1 fetchNodeFunc, Action`4 mergeManyToManyFunc)
   at SD.LLBLGen.Pro.ORMSupportClasses.PersistenceCore.FetchPrefetchPath(QueryParameters rootNodeParameters, Boolean forceParameterizedPPath, ITransaction transactionToUse, Int32 parameterisedPrefetchPathThreshold, Action`1 fetchNodeFunc, Action`4 mergeManyToManyFunc)
   at SD.LLBLGen.Pro.ORMSupportClasses.DataAccessAdapterCore.FetchEntityCollection(QueryParameters parameters)
   at SD.LLBLGen.Pro.ORMSupportClasses.DataAccessAdapterBase.<>c__DisplayClass10_0.<FetchEntityCollection>b__0()
   at SD.LLBLGen.Pro.ORMSupportClasses.RecoveryStrategyBase.<>c__DisplayClass7_0.<Execute>b__0()
   at SD.LLBLGen.Pro.ORMSupportClasses.RecoveryStrategyBase.Execute[TResult](Func`1 toExecute)

The following exception occurred at TResult Execute[TResult](System.Func1[TResult])`: Recovery failed: Maximum number of retries of 5 exceeded.

Stack Trace:
   at SD.LLBLGen.Pro.ORMSupportClasses.RecoveryStrategyBase.Execute[TResult](Func`1 toExecute)
   at SD.LLBLGen.Pro.ORMSupportClasses.DataAccessAdapterCore.FetchEntityCollection(IEntityCollection2 collectionToFill, IRelationPredicateBucket filterBucket, Int32 maxNumberOfItemsToReturn, ISortExpression sortClauses, IPrefetchPath2 prefetchPath, ExcludeIncludeFieldsList excludedIncludedFields)
   at Grb.Platform.DatabaseUtility.Plugins.Encryption.Encryptor.ProcessGeneralChunk(...)

Do you reuse the prefetch path across threads? Or any other object across the threads? The crash happens in this function:

private static int CalculateHashForEntityBasedOnRelation(IEntityCore entity, IEntityRelation relation, bool forPkSide, out bool hashRejected)
{
    int hashValue = -1;
    hashRejected = false;
    for(int i = 0; i < relation.AmountFields; i++)
    {
        string fieldName = forPkSide ? relation.GetPKEntityFieldCore(i).Name : relation.GetFKEntityFieldCore(i).Name;
        IEntityFieldCore field = entity.GetFieldByName(fieldName);
        if(field == null)
        {
            // probably a supertype and the prefetch path was for a subtype 
            hashRejected = true;
            break;
        }
        if(i == 0)
        {
            hashValue = field.GetHashCode();
        }
        else
        {
            hashValue ^= field.GetHashCode();
        }
    }
    return hashValue;
}

and as it's a nullref, it could only be that the relation is null or the entity is null. It's called here:

for(int j = 0; j < fkSideCollection.Count; j++)
{
    bool fkSideRejected;
    int fkHash = CalculateHashForEntityBasedOnRelation(fkSideCollection[j], currentElement.Relation, false, out fkSideRejected);
    if(fkSideRejected)
    {
        continue;
    }
//...

currentElement is a prefetch path element. The fkSideCollection is the set in the prefetch path element into which the data is stored for that node.

So to fetch prefetch paths in multiple threads, re-create the path for each thread. This isn't obvious, but it's essential. Also, in the installation folder, frameworks\llblgen pro\runtime libraries, there's a pdb for the ormsupportclasses dll. Could you place that in your utility's bin folder so next time the crash happens, it will produce a line number with the stack trace?

Otis wrote:

So to fetch prefetch paths in multiple threads, re-create the path for each thread.

Gotcha! Yes, I was reusing the prefetch path. Thanks!

Otis wrote:

Also, in the installation folder, frameworks\llblgen pro\runtime libraries, there's a pdb for the ormsupportclasses dll. Could you place that in your utility's bin folder so next time the crash happens, it will produce a line number with the stack trace?

Will do!