Hello, After upgrading to from 5.4 to 5.9, we are seeing UnhandledException when multiple threads are accessing data are fetching the data from database. Its happening on multiple tables.

 FATAL - UnhandledException
SD.LLBLGen.Pro.ORMSupportClasses.ORMQueryConstructionException: The property 'Name' isn't mapped to a field or database construct of entity type 'AssayCalibratorSetEntity'. Did you mean to call an extension method instead? ('Count' vs. 'Count()') ?
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.MemberAccessEvaluator.HandleMemberIsPartOfEntity(MemberExpression expressionToHandle, Expression memberContainer)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.MemberAccessEvaluator.HandleMemberExpression(MemberExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleBinaryExpression(BinaryExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleLambdaExpression(LambdaExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleAllAnyExpression(AllAnyExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleLambdaExpression(LambdaExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleWhereExpression(WhereExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProProviderBase.HandleExpressionTree(Expression expression)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProProviderBase.System.Linq.IQueryProvider.Execute(Expression expression)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProQuery`1.System.Collections.Generic.IEnumerable<T>.GetEnumerator()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)

It looks like it came from this here:

private System.Linq.Expressions.Expression HandleMemberIsPartOfEntity(
      MemberExpression expressionToHandle,
      System.Linq.Expressions.Expression memberContainer)
    {
      EntityExpression entityExpression = (EntityExpression) memberContainer;
      IEntityCore instanceFromEntityType = LinqUtils.CreateEntityInstanceFromEntityType(entityExpression.EntityType, this.GeneratedCodeElementCreator);
      System.Linq.Expressions.Expression expression;
      if (LinqUtils.GetFieldObject(instanceFromEntityType, expressionToHandle.Member.Name) == null)
      {
        expression = LinqUtils.HandleMemberMappedOnRelationIsPartOfEntity(expressionToHandle, memberContainer, this.TrackedMappings, this.GeneratedCodeElementCreator);
        if (expression == null)
          throw new ORMQueryConstructionException(string.Format("The property '{0}' isn't mapped to a field or database construct of entity type '{1}'. Did you mean to call an extension method instead? ('Count' vs. 'Count()') ?", (object) expressionToHandle.Member.Name, (object) instanceFromEntityType.LLBLGenProEntityName));
      }
      else
      {
        SetAlias reachableAlias = this.TrackedMappings.FindReachableAlias(entityExpression.Alias);
        expression = (System.Linq.Expressions.Expression) new EntityFieldExpression(LinqUtils.GetFieldObjectAndSetObjectAlias(instanceFromEntityType, expressionToHandle.Member.Name, reachableAlias), reachableAlias, entityExpression.AliasOfRealSource);
        ((IRelationPropagator) expression).AddRelations(entityExpression.RelationsToUse);
      }
      return expression;
    }

If you see exceptions on multiple threads, please check if you're sharing the DataAccessAdapter object that's set into the LinqMetaData object. You can't share that among threads, you have to create a new one every time. (This is fine, the creation of a new adapter is very fast).

Could you please show the code you use in multiple threads that crashes?

(ps: please don't mark a thread as done if it's not answered yet as it's then removed from the support queue , thanks!)

Otis wrote:

If you see exceptions on multiple threads, please check if you're sharing the DataAccessAdapter object that's set into the LinqMetaData object. You can't share that among threads, you have to create a new one every time. (This is fine, the creation of a new adapter is very fast).

Could you please show the code you use in multiple threads that crashes?

(ps: please don't mark a thread as done if it's not answered yet as it's then removed from the support queue , thanks!)

We are creating DataAccessAdapter everytime. This issue happens randomly. Please let me know if this helpful. Below is the code

        using (var adapter = new DataAccessAdapter() { CommandTimeOut = DefaultDataAccessCommandTimeout })
        {
            var metaData = new LinqMetaData(adapter);

            return metaData.AssaySpecification.WithPath(AssaySpecificationWithCalibratorSetsPrefetch)
                           .Where(assaySpecEntity => assaySpecEntity.AssayCalibratorSets.Any(calSet => calSet.Name == calSetNameUpperCase)).ToList();
        }

Not sure if the contents in AssaySpecificationWithCalibratorSetsPrefetch is shared, but the error seems to originate from the Any() handler.

As you migrated from 5.4 to 5.9, you did take into account the move of the csproj files so there's no chance there is code running the older mappings vs. code that's running the newer mappings? All instances running use the new generated code from 5.9... ?

The code fails to return a valid field in entityInstance.GetFieldByName("Name") when this happens. The entityInstance is created by the caller at that location locally so per-thread. It'll simply do entityInstance.Fields["Name"] when this happens.

Very strange that this fails. We use static, shared readonly objects for mapping data so it should be thread safe. No idea if you do dynamic field definitions for entities in your application. (something that's not supported out of the box, but perhaps your application was augmented to support that)

Otis wrote:

Not sure if the contents in AssaySpecificationWithCalibratorSetsPrefetch is shared, but the error seems to originate from the Any() handler.

As you migrated from 5.4 to 5.9, you did take into account the move of the csproj files so there's no chance there is code running the older mappings vs. code that's running the newer mappings? All instances running use the new generated code from 5.9... ?

The code fails to return a valid field in entityInstance.GetFieldByName("Name") when this happens. The entityInstance is created by the caller at that location locally so per-thread. It'll simply do entityInstance.Fields["Name"] when this happens.

Very strange that this fails. We use static, shared readonly objects for mapping data so it should be thread safe. No idea if you do dynamic field definitions for entities in your application. (something that's not supported out of the box, but perhaps your application was augmented to support that)

This is the prefetch is used by the query. It is not shared by any other method and specifically it is used for this query. private PrefetchPath2 AssaySpecificationWithCalibratorSetsPrefetch => new PrefetchPath2((int) EntityType.AssaySpecificationEntity) { AssaySpecificationEntity.PrefetchPathAssayDisplayOrder, AssaySpecificationEntity.PrefetchPathAssayCalibratorSets }

I checked the project file. Files are generated are referenced with 5.9 <PackageReference Include="SD.LLBLGen.Pro.ORMSupportClasses"> <Version>5.9.0</Version> </PackageReference>

We are not using dynamic field definitions for entities in application.

Please let me know if there is a way I can enable LLblGenPro logging and provide you the detailed logging.

I honestly don't know what it could be caused by. Logging won't do anything here I think as the runtime clearly detects something's fundamentally wrong but why that happens is unclear (as in: it shouldn't happen), however you could enable various tracers, see: https://www.llblgen.com/Documentation/5.10/LLBLGen%20Pro%20RTF/Using%20the%20generated%20code/gencode_troubleshootingdebugging.htm#netstandard-2.0. As with multi-threading issues, the problem already occurred earlier. The Fields object is obtained from a singleton (ModelProvider) and as the query works when it's called in a single threaded scenario, all information is there.

Could you create a unit test with just this query and run it with multiple threads for a longer period of time to see if you can reproduce it? Your app is a webapp or a service or other app?

Also, if you rewrite it to:

metaData.AssaySpecification.WithPath(AssaySpecificationWithCalibratorSetsPrefetch)
                           .Where(assaySpecEntity => assaySpecEntity.AssayCalibratorSets.Count(calSet => calSet.Name == calSetNameUpperCase) > 0).ToList();

(it's slower in the DB, but it's just for testing) does it still occur? (In a different area probably)

Otis wrote:

I honestly don't know what it could be caused by. Logging won't do anything here I think as the runtime clearly detects something's fundamentally wrong but why that happens is unclear (as in: it shouldn't happen), however you could enable various tracers, see: https://www.llblgen.com/Documentation/5.10/LLBLGen%20Pro%20RTF/Using%20the%20generated%20code/gencode_troubleshootingdebugging.htm#netstandard-2.0. As with multi-threading issues, the problem already occurred earlier. The Fields object is obtained from a singleton (ModelProvider) and as the query works when it's called in a single threaded scenario, all information is there.

Could you create a unit test with just this query and run it with multiple threads for a longer period of time to see if you can reproduce it? Your app is a webapp or a service or other app?

Also, if you rewrite it to:

metaData.AssaySpecification.WithPath(AssaySpecificationWithCalibratorSetsPrefetch)
                           .Where(assaySpecEntity => assaySpecEntity.AssayCalibratorSets.Count(calSet => calSet.Name == calSetNameUpperCase) > 0).ToList();

(it's slower in the DB, but it's just for testing) does it still occur? (In a different area probably)

I will try both requests and update you. Here is the another scenario where same issue happened. I hope this can we give some insight. Query was straight forward. I also noticed that if happens during first few threads after starting the application, if the crash does not happen then everything is fine. We have desktop application.

 using (var adapter = new DataAccessAdapter() { CommandTimeOut = DefaultDataAccessCommandTimeout })
            {
                var metaData = new LinqMetaData(adapter);

                var masterLotCalStatus = from calStatus in metaData.ReagentMasterLotCalStatus
                                         where calStatus.ModuleId == moduleId &&
                                               calStatus.ReagentAssay.AssayNumber == assayNumber &&
                                               calStatus.ReagentAssay.AssayVersion == assayVersion &&
                                               calStatus.ReagentMasterLot.MasterLotNumber == masterLotNumber
                                         select calStatus;
                return masterLotCalStatus.IsNotNullOrEmpty() ? masterLotCalStatus.First() : null;
            }

Stack Trace.
SD.LLBLGen.Pro.ORMSupportClasses.ORMQueryConstructionException: The property 'ModuleId' isn't mapped to a field or database construct of entity type 'ReagentMasterLotCalStatusEntity'. Did you mean to call an extension method instead?
 ('Count' vs. 'Count()') ? at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.MemberAccessEvaluator.HandleMemberIsPartOfEntity(MemberExpression expressionToHandle, Expression memberContainer) at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.MemberAccessEvaluator.HandleMemberExpression(MemberExpression
 expressionToHandle) at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle) at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleUnaryExpression(UnaryExpression
 expressionToHandle) at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle) at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleBinaryExpression(BinaryExpression
 expressionToHandle) at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle) at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleBinaryExpression(BinaryExpression
 expressionToHandle) at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle) at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleBinaryExpression(BinaryExpression
 expressionToHandle) at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle) at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleBinaryExpression(BinaryExpression
 expressionToHandle) at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle) at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleLambdaExpression(LambdaExpression
 expressionToHandle) at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle) at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleWhereExpression(WhereExpression
 expressionToHandle) at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle) at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProProviderBase.HandleExpressionTree(Expression expression) at
 SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProProviderBase.System.Linq.IQueryProvider.Execute(Expression expression) at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProQuery`1.System.Collections.Generic.IEnumerable.GetEnumerator() at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source)

Quick Update: I executed below query. It also crashed.

metaData.AssaySpecification.WithPath(AssaySpecificationWithCalibratorSetsPrefetch)
                           .Where(assaySpecEntity => assaySpecEntity.AssayCalibratorSets.Count(calSet => calSet.Name == calSetNameUpperCase) > 0).ToList();

Stack Trace:
FATAL - UnhandledException
SD.LLBLGen.Pro.ORMSupportClasses.ORMQueryConstructionException: The property 'Name' isn't mapped to a field or database construct of entity type 'AssayCalibratorSetEntity'. Did you mean to call an extension method instead? ('Count' vs. 'Count()') ?
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.MemberAccessEvaluator.HandleMemberIsPartOfEntity(MemberExpression expressionToHandle, Expression memberContainer)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.MemberAccessEvaluator.HandleMemberExpression(MemberExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleBinaryExpression(BinaryExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleLambdaExpression(LambdaExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleAggregateExpression(AggregateExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleBinaryExpression(BinaryExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleLambdaExpression(LambdaExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleWhereExpression(WhereExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProProviderBase.HandleExpressionTree(Expression expression)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProProviderBase.System.Linq.IQueryProvider.Execute(Expression expression)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProQuery`1.System.Collections.Generic.IEnumerable<T>.GetEnumerator()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)

We can't reproduce it. Attached is a console app which starts 5 threads (also tried with 20, same result: it works fine), ran it a couple of times, but it doesn't fail and works as it should. Each thread fetches 100 Salesorderheader rows and their sales order detail rows from AdventureWorks.

The main issue is that the singletons have to be initialized, which use static constructors so this is taken care of. But as you gave little info what you're doing at startup and even if you use .net framework or .net 6 or higher, we have to assume you did basically what we're doing in the repro case.

If it only happens on startup and only sometimes (so not always), try the following: fetch 1 entity on the main thread before the threads are started. Also, if you use dynamic loading of the assemblies with generated code, make sure they're loaded and initialized before the threads are started. Doing that on threads could make the static constructors run on a later time than what might be needed.

In .NET static constructors are run before all other code, so singletons are initialized properly, but if you dynamically load the assemblies (not sure you do, let's assume you do) then the .net runtime can't decide when to run the static constructors (at least I think so).

Otis wrote:

We can't reproduce it. Attached is a console app which starts 5 threads (also tried with 20, same result: it works fine), ran it a couple of times, but it doesn't fail and works as it should. Each thread fetches 100 Salesorderheader rows and their sales order detail rows from AdventureWorks.

The main issue is that the singletons have to be initialized, which use static constructors so this is taken care of. But as you gave little info what you're doing at startup and even if you use .net framework or .net 6 or higher, we have to assume you did basically what we're doing in the repro case.

If it only happens on startup and only sometimes (so not always), try the following: fetch 1 entity on the main thread before the threads are started. Also, if you use dynamic loading of the assemblies with generated code, make sure they're loaded and initialized before the threads are started. Doing that on threads could make the static constructors run on a later time than what might be needed.

In .NET static constructors are run before all other code, so singletons are initialized properly, but if you dynamically load the assemblies (not sure you do, let's assume you do) then the .net runtime can't decide when to run the static constructors (at least I think so).

Could you please attach the database backup files. It looks like AdventureWorks from microsoft is different from your database. In regards to your statement on "startup /singletons have to be initialized", Please let me know if anything is changed from V5.4 to 5.9 in this area. Crash can happen after startup is completed like 20 minutes after the application starts. We are using .Net framework 4.8, SQL server 2019, Vistual studio 2022.

Lokesh wrote:

Otis wrote:

We can't reproduce it. Attached is a console app which starts 5 threads (also tried with 20, same result: it works fine), ran it a couple of times, but it doesn't fail and works as it should. Each thread fetches 100 Salesorderheader rows and their sales order detail rows from AdventureWorks.

The main issue is that the singletons have to be initialized, which use static constructors so this is taken care of. But as you gave little info what you're doing at startup and even if you use .net framework or .net 6 or higher, we have to assume you did basically what we're doing in the repro case.

If it only happens on startup and only sometimes (so not always), try the following: fetch 1 entity on the main thread before the threads are started. Also, if you use dynamic loading of the assemblies with generated code, make sure they're loaded and initialized before the threads are started. Doing that on threads could make the static constructors run on a later time than what might be needed.

In .NET static constructors are run before all other code, so singletons are initialized properly, but if you dynamically load the assemblies (not sure you do, let's assume you do) then the .net runtime can't decide when to run the static constructors (at least I think so).

Could you please attach the database backup files. It looks like AdventureWorks from microsoft is different from your database. In regards to your statement on "startup /singletons have to be initialized", Please let me know if anything is changed from V5.4 to 5.9 in this area and how should we initialize these singletons? ot could you please provide more information.

We are using .Net framework 4.8, SQL server 2019, Vistual studio 2022. I also enabled the tracing at verbose level. I ran about 100 times and each time 150 threads were spawned. I could not reproduce the issue after tracing is enabled.

After I disabled the tracing. Among first few thread(file attached), thread 76 crashed. Other threads completed successfully. I tried following : "fetch 1 entity on the main thread before the threads are started" : We didnt not see the crash. We are trying to understand if we have to do for all assemblies?

Lokesh wrote:

Otis wrote:

We can't reproduce it. Attached is a console app which starts 5 threads (also tried with 20, same result: it works fine), ran it a couple of times, but it doesn't fail and works as it should. Each thread fetches 100 Salesorderheader rows and their sales order detail rows from AdventureWorks.

The main issue is that the singletons have to be initialized, which use static constructors so this is taken care of. But as you gave little info what you're doing at startup and even if you use .net framework or .net 6 or higher, we have to assume you did basically what we're doing in the repro case.

If it only happens on startup and only sometimes (so not always), try the following: fetch 1 entity on the main thread before the threads are started. Also, if you use dynamic loading of the assemblies with generated code, make sure they're loaded and initialized before the threads are started. Doing that on threads could make the static constructors run on a later time than what might be needed.

In .NET static constructors are run before all other code, so singletons are initialized properly, but if you dynamically load the assemblies (not sure you do, let's assume you do) then the .net runtime can't decide when to run the static constructors (at least I think so).

Could you please attach the database backup files. It looks like AdventureWorks from microsoft is different from your database.

I've attached a new AWMT above which is on the MS adventure works

In regards to your statement on "startup /singletons have to be initialized", Please let me know if anything is changed from V5.4 to 5.9 in this area. Crash can happen after startup is completed like 20 minutes after the application starts. We are using .Net framework 4.8, SQL server 2019, Vistual studio 2022.

Please provide more information, I have no idea what you're doing, keep that in mind, we don't have your code in front of us. So give us an overview what you're doing, as we don't know. And please not only answer my questions but think about details that might be important for us to know what it is that might cause this. We can't reproduce it with a heavy multithreaded app, so it's something you do do in your software that triggers a state that makes this occur.

You say the crash happens only at startup but now you say it can crash after 20 minutes as well. But that doesn't make any sense. Our singletons are then already initialized. As they're singletons, they live in the startup appdomain. Are you loading the generated code in separate appdomains which you're unloading?

We refactored a lot internally, in the period 5.4-5.9. We haven't had a single report about this issue. Our own tests also don't report any multi-threading issue, so it's something you're doing that is apparently non-standard that triggers it. WHat that is is unclear.

Lokesh wrote:

I also enabled the tracing at verbose level. I ran about 100 times and each time 150 threads were spawned. I could not reproduce the issue after tracing is enabled.

150 threads where? In my test app or in your app? That you can't reproduce it after tracing is enabled suggests a multi-threading issue somewhere as these are heavily timing related (as they only occur when two threads try to access a shared resource). Our singletons are started by static constructors which are run at app startup as is by design in .NET. This information is readonly after that and therefore threadsafe (hence my question earlier if you change this meta data).

After I disabled the tracing. Among first few thread(file attached), thread 76 crashed. Other threads completed successfully. I tried following : "fetch 1 entity on the main thread before the threads are started" : We didnt not see the crash. We are trying to understand if we have to do for all assemblies?

I'm not sure what you mean with 'for all assemblies'? You have multiple generated code assemblies, like multiple databases or multiple projects? Please elaborate about 'thread 76 crashed', in which application?

I can ask another 1000 questions, but I don't know what's going on in your app anyway so I better wait for more information

Sorry for the confusion. I haven't used your app due to database mismatch. Let me explain 20 minute scenario and application setup. We have more than 50 feature in desktop app. We have Server and Client apps. Crash is happening on server side. Each Server feature has 3 projects=> Feature1(Services, Interfaces, business logic), Feature1.Common(LLblGenPro auto generated entities, common code with Client) and Feature1UnitTest. In Feature.Common we have auto generated LLBLgen Pro files. Business logic and repository are in Features. Below is the scenario where issue was originally noticed. It took about 20 minutes to reach from Step 1 to Step 4.

1. User starts the application
2. Creates 4 Orders which are Pending state
3. Navigates through different UI
4. When these 4 Orders are in InProcesss,  on server side Feature1 publishes  4 Async events.
5. Feature2 service subscribes to Async event.
6. Feature2 further calls Feature3.
7. Feature3 calls its DataAccessManager which  calls repository where queries are defined. 

1. We are seeing the crash when these 4 tries to run queries at same time at repository query.

To mimic the above scenario which takes lots of time, we instrumented code to publishes 150 Async events from Feature1 after 5 seconds of Feature1 startup is completed. To simplify further, we removed Feature 1 and Feature 2 calls, now we are firing 150 threads from Feature3 service startup completes(after 5 seconds). Feature3 service(singleton) calls its DataAccessManager which calls Repository.
Here is the sample code For Feature 3 with DataAccessManager and Repository. Both DataAccessManager and Repository are Interface.

      //Service OnStarupComplete
       protected override void OnStartupComplete()
        {
            var aTimer = new Timer(5000);
            aTimer.Elapsed += OnTimedEvent;
            aTimer.Enabled = true;

        }

        private void OnTimedEvent(object sender, ElapsedEventArgs e)
        {
            for (var i = 0; i < 150; i++)
            {
                Task.Factory.StartNew(() => DataAccessManager.GetData(name));
            }
        }
        
        // DataAccessManager
        public IList<TAssay> GetData<TAssay>(string name)
            where TAssay : AssaySlim, new()
        {
            LogDebug($"Entry");
            var test = Repository.GetCalInfo<TAssay>(name).SelectList(GetAssay<TAssay>);
            LogDebug($"Exit");
            return test;
        }
        
        // Repository Query
         using (var adapter = new DataAccessAdapter() { CommandTimeOut = DefaultDataAccessCommandTimeout })
            {
                var metaData = new LinqMetaData(adapter);
               

                return metaData.AssaySpecification.WithPath(AssaySpecificationWithCalibratorSetsPrefetch)
                               .Where(assaySpecEntity =>
                                          assaySpecEntity.AssayCalibratorSets.Any(calSet => calSet.Name ==
                                              name))
                               .ToList();
            }

This crash does not happen on every startup. I have logging in Feature3 DataAccessManager which indicate about threads entry and exit. Thread 76 was crashed in my application and I didnt see any exit. If crash needs to happen it happens during first few threads. All feature services and interfaces are started as part of startup process in my application. We are using One Database with multiple projects/features. We haven't change this code( Step 4-8 ) for many years. Last year we updated .NetFrame work, SQL server and LLBLGen pro as same time.

Please let me know if it helps, I can setup quick meeting with you and walk you over the crash.

Hmm, ok. The server starts all features at the same time and they're not unloaded? Are the dll's in the features dynamically loaded? Or are they compiled into the server so when the server starts they're loaded by .net? Are they also unloaded at some point? If you load them dynamically, they are loaded into the same appdomain? (of the server, so the main app domain) ? If you dynamically load the assemblies, make sure all features load the same generated code assemblies if they use the same generated code (so to make sure feature 32 for instance doesn't load the older 5.4 assembly).

Which thread crashes is random I think, so if it's 1, 12 or 76, that's not important. The main issue is that the meta data with the field information for an entity type isn't available for some threads and it then crashes.

The features you have, they all have the same generated entity project? The calls between teh services through event subscribing, does that mean they are between processes or is this in the same process?

The Feature3 service singleton, how do you initialize that? It's important to do that in the following setup:

/// <summary>Singleton implementation of the PersistenceInfoProvider. This class is the singleton wrapper through which the actual instance is retrieved.</summary>
internal static class PersistenceInfoProviderSingleton
{
    private static readonly IPersistenceInfoProvider _providerInstance = new PersistenceInfoProviderCore();

    /// <summary>Dummy static constructor to make sure threadsafe initialization is performed.</summary>
    static PersistenceInfoProviderSingleton() { }

    /// <summary>Gets the singleton instance of the PersistenceInfoProviderCore</summary>
    /// <returns>Instance of the PersistenceInfoProvider.</returns>
    public static IPersistenceInfoProvider GetInstance() { return _providerInstance; }
}

/// <summary>Actual implementation of the PersistenceInfoProvider. Used by singleton wrapper.</summary>
internal class PersistenceInfoProviderCore : PersistenceInfoProviderBase
{
//...
}

(this is in the generated code). So you define a private static instance with initialization and you also provide a static constructor, which can be empty, but you have to have that. Just to make sure you have that too.

Looking into the history of the code changes, the main work in this area was done in 5.4. As you said you migrated from 5.4 to 5.9, you already had these changes in the previous version.

To confirm, if you prior to step 1, first fetch an entity (doesn't matter which), it'll always work and no crash occurs? (This means the main thread can properly initialize the meta data store using the static constructor path as intended, and all future queries will thus be able to utilize the already initialized data stores)

Please let me know if its helpful. I answered in bold.

Hmm, ok. The server starts all features at the same time and they're not unloaded? They are not unloaded. Are the dll's in the features dynamically loaded? Or are they compiled into the server so when the server starts they're loaded by .net? **They are compiled into the server applicatiom and loaded when application starts up. ** Are they also unloaded at some point? If you load them dynamically, they are loaded into the same appdomain? (of the server, so the main app domain) ? They are not unloaded. If you dynamically load the assemblies, make sure all features load the same generated code assemblies : We are not loading dlls dynamically. if they use the same generated code (so to make sure feature 32 for instance doesn't load the older 5.4 assembly). Checked that we dont have old instance of assemblies.

Which thread crashes is random I think, so if it's 1, 12 or 76, that's not important. The main issue is that the meta data with the field information for an entity type isn't available for some threads and it then crashes. Yes

The features you have, they all have the same generated entity project? No ,each feature has its own database tables and associated entities. The calls between teh services through event subscribing, does that mean they are between processes or is this in the same process? Same process.

(this is in the generated code). So you define a private static instance with initialization and you also provide a static constructor, which can be empty, but you have to have that. Just to make sure you have that too. We are using DataAccessAdapter and during its construction, the static class PersistenceInfoProviderSingleton's constructor is called.

Below is the code:

using (var adapter = new DataAccessAdapter() { CommandTimeOut = DefaultDataAccessCommandTimeout })

/// <summary>CTor</summary>
        public DataAccessAdapter() : this(ReadConnectionStringFromConfig(), false, null, null) { }

/// <summary>CTor.</summary>
        /// <param name="connectionString">The connection string to use when connecting to the database.</param>
        /// <param name="keepConnectionOpen">when true, the DataAccessAdapter will not close an opened connection. Use this for multi action usage.</param>
        /// <param name="catalogNameUsageSetting"> Configures this data access adapter object how to threat catalog names in persistence information.</param>
        /// <param name="catalogNameToUse"> The name to use if catalogNameUsageSetting is set to ForceName. Ignored otherwise.</param>
        /// <remarks>For backwards compatibility.</remarks>
        public DataAccessAdapter(string connectionString, bool keepConnectionOpen, CatalogNameUsage catalogNameUsageSetting, string catalogNameToUse) 
                : base(PersistenceInfoProviderSingleton.GetInstance())
        {
            InitClassPhase2(connectionString, keepConnectionOpen, catalogNameUsageSetting, SchemaNameUsage.Default, catalogNameToUse, string.Empty, null, null);
        }

Looking into the history of the code changes, the main work in this area was done in 5.4. As you said you migrated from 5.4 to 5.9, you already had these changes in the previous version. There are no changes in our code base.

To confirm, if you prior to step 1, first fetch an entity (doesn't matter which), it'll always work and no crash occurs? If i call same entity then its fine. If I used different entity then it crashed. (This means the main thread can properly initialize the meta data store using the static constructor path as intended, and all future queries will thus be able to utilize the already initialized data stores)

We also tried below code to force the construction of static PersistenceInfoProviderSingleton on main thread but it didnt help.

 protected override void OnStartupComplete()
        {
            using (var adapter = new DataAccessAdapter())
            {
            }

            var aTimer = new Timer(8000);
            aTimer.Elapsed += OnTimedEvent;
            aTimer.Enabled = true;

        }

**Could you please help us to understand: How is query constructed?  For instance, SQL can pre-compile queries to speed up query execution.  Does LLBLGenPro caches its query construction? We checked and verified that the LLBLGenPro dll's are loaded when application starts up complete . However application crashed after confirming that dll was loaded. ** Code that confirm assembly was loaded before crash.

protected override void OnStartupComplete()
        {
            var assemblies = AppDomain.CurrentDomain.GetAssemblies();
            foreach (var assembly in assemblies)
            {
                LogDebug($"{assembly.GetName().FullName}{assembly.GetName().Version}");
            }

            var aTimer = new Timer(8000);
            aTimer.Elapsed += OnTimedEvent;
            aTimer.Enabled = true;
        }

I am also adding stacktrace when I execute tasks from feature 3 not sure if its helpful. A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property.

2023-02-07 21:02:39,482 [2] FATAL - UnhandledException
System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. ---> SD.LLBLGen.Pro.ORMSupportClasses.ORMQueryConstructionException: The property 'Name' isn't mapped to a field or database construct of entity type 'AssayCalibratorSetEntity'. Did you mean to call an extension method instead? ('Count' vs. 'Count()') ?
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.MemberAccessEvaluator.HandleMemberIsPartOfEntity(MemberExpression expressionToHandle, Expression memberContainer)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.MemberAccessEvaluator.HandleMemberExpression(MemberExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleBinaryExpression(BinaryExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleLambdaExpression(LambdaExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleAllAnyExpression(AllAnyExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleLambdaExpression(LambdaExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleWhereExpression(WhereExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProProviderBase.HandleExpressionTree(Expression expression)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProProviderBase.System.Linq.IQueryProvider.Execute(Expression expression)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProQuery`1.System.Collections.Generic.IEnumerable<T>.GetEnumerator()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at ..

Thank you for your help!!

     using (var adapter = new DataAccessAdapter() { CommandTimeOut = DefaultDataAccessCommandTimeout })
        {
            var metaData = new LinqMetaData(adapter);

            return metaData.AssaySpecification.WithPath(AssaySpecificationWithCalibratorSetsPrefetch)
                           .Where(assaySpecEntity =>
                                      assaySpecEntity.AssayCalibratorSets.Any(calSet => calSet.Name ==
                                          name))
                           .ToList();
        }

For the sake of testing, can you try the following, please:

1- replace the Linq query up there with a QuerySpec alternative and see if the exceptions still show.

2- Can you reproduce the problem with another model (adventure works or northwind for example), using the AWMT example attached to a post from Otis above? (It's been updated for the new aw)

Additionally, I've augmented the runtime dll a bit, to eliminate a possible cause (although I have no idea why that would fail), the runtime is attached. It's a small change in 5.9.4. You can drop this dll into the server binary folder where you start the app so it's loaded by all assemblies.

The change: in the runtime we use instances of entities to get to information, read field characteristics etc. We used to create dummy instances for these but these were created with every query. We introduced (in 5.9) a singleton using a ConcurrentDictionary which re-uses these dummy instances (as they're used purely in a readonly fashion anyway). The runtime attached to this post will always create a new instance in the method where the crash happens.

This is to see if this fixes things. It might now work but still contain a bug somewhere (in your code or in ours) but it won't trigger it anymore due to timing; these multi-threaded issues are very hard to debug/trace down as the only situation in which they occur relies on a specific situation which only occurs perhaps due to specific timing (hence why with tracing on it works fine because the runtime was slower).

For the sake of testing, can you try the following, please:

1- replace the Linq query up there with a QuerySpec alternative and see if the exceptions still show. I replaced below query not sure if this was the right way to do. I did not see the crash.

 using (var adapter = new DataAccessAdapter() { CommandTimeOut = DefaultDataAccessCommandTimeout })
            {
                var qf = new QueryFactory();
                var q = qf.AssaySpecification.WithPath(AssaySpecificationEntity.PrefetchPathAssayCalibratorSets)
                          .WithPath(AssaySpecificationEntity.PrefetchPathAssayDisplayOrder)
                          .Where(AssayCalibratorSetFields.Name == calSetNameUpperCase);

                return new List<AssaySpecificationEntity>();
            }

2- Can you reproduce the problem with another model (adventure works or northwind for example), using the AWMT example attached to a post from Otis above? (It's been updated for the new aw): I was not able to reproduce the issue with Adventure works DB.

Thank you providing DLL to test the scenario. I agree that these timing issues are very difficult to reproduce. I confirm that with your new dll 5.9.4, we do not have the crash issue. If we have to proceed with 5.9.4 including this change, could you please below confirm below questions. - When 5.9.4 will be officially available as a NuGet package. This is important for us to meet our release dates. - Could you please provide us release notes between 5.9.0 to 5.9.4. This is important to determine regression cases for internal testing. - In 5.9.4, have you reverted to the 5.4 mechanism when it comes to use of instances of entities to get to information. - Do we need to regenerate the code in 5.9.4 package.

Thanks again for your support!

Thanks for testing this. We have released 5.9.4 hotfix which is available on nuget (as a hotfix build, so you need to enable prereleases). If this solves your problem properly we will release 5.9.4 officially next week with the release of 5.10.

The code is fully tested with our unit tests so if nothing changes this will be the code that's released as 5.9.4.

We've analyzed the code paths around the area where we made the change and other code which utilizes the cache for dummy instances and we can't identify a spot where it might be things could go wrong with the results you saw. What we suspect is that a customization in your code base ran fine for years (because this change was made in 5.9) and now could cause a problem. With customization I mean code added to a factory, entity etc. which should work fine but could influence the overall outcome in this particular edge case. As that's likely very hard to track down, I think this workaround is sufficient to make it work and keep it working. It'll create some more pressure for the GC but not a lot.

I'm glad at least we found a workaround for your situation

Thank you very much for fixing the issue so fast and releasing 5.9.4-hotfix-20230209. Could you please confirm it.

Could you please help us to get below information: - Could you please provide us release notes between 5.9.0 to 5.9.4-hotfix-20230209. This is important to determine regression cases for risk assessment and internal testing. - Do we need to regenerate the code with 5.9.4-hotfix-20230209. - Could please provide official date of 5.9.4-hotfix-20230209.

Thanks again for very quick turnaround.

Lokesh wrote:

Thank you very much for fixing the issue so fast and releasing 5.9.4-hotfix-20230209. Could you please confirm it.

That is indeed the hotfix I meant

Could you please help us to get below information: - Could you please provide us release notes between 5.9.0 to 5.9.4-hotfix-20230209. This is important to determine regression cases for risk assessment and internal testing.

The latest official release is 5.9.3, so if you downloaded the latest 5.9 installer then you're currently working with 5.9.3. If you are on 5.9.0, you can see the changes since 5.9.0 here: https://www.llblgen.com/Pages/ChangelogBrowser.aspx (use the dialog to select the runtimes for 5.9)

If you log into the website you'll see the release notes for 5.9.4, but for convenience I've copied them below:

• [LLBLMAIN-2490] ( Runtime Libraries, ) [Fixed] - Linq, typedview/list/tvf call: A where clause comparing a nullable enum typed field with an enum value could crash in the built-in value converter
• [LLBLMAIN-1627] ( Runtime Libraries, ) [Fixed] - Linq: Using the PredicateBuilder with a predicate that uses a comparison operator with one side the value null could lead to a crash because the expression tree formed by the PredicateBuilder was unexpected.
• [LLBLMAIN-2492] ( Runtime Libraries, ) [Implemented] - Linq: Rare race condition in edge case system when re-using dummy instance in linq provider could cause a query conversion to crash. The workaround implemented should mitigate this issue, which we ourselves haven't been able to reproduce

• Do we need to regenerate the code with 5.9.4-hotfix-20230209.

No, not if you're on 5.9.3

• Could please provide official date of 5.9.4-hotfix-20230209.

Somewhere next week, it's not yet set in stone.

Otis wrote:

Lokesh wrote:

Thank you very much for fixing the issue so fast and releasing 5.9.4-hotfix-20230209. Could you please confirm it.

That is indeed the hotfix I meant

Could you please help us to get below information: - Could you please provide us release notes between 5.9.0 to 5.9.4-hotfix-20230209. This is important to determine regression cases for risk assessment and internal testing.

The latest official release is 5.9.3, so if you downloaded the latest 5.9 installer then you're currently working with 5.9.3. If you are on 5.9.0, you can see the changes since 5.9.0 here: https://www.llblgen.com/Pages/ChangelogBrowser.aspx (use the dialog to select the runtimes for 5.9)

If you log into the website you'll see the release notes for 5.9.4, but for convenience I've copied them below:

• [LLBLMAIN-2490] ( Runtime Libraries, ) [Fixed] - Linq, typedview/list/tvf call: A where clause comparing a nullable enum typed field with an enum value could crash in the built-in value converter
• [LLBLMAIN-1627] ( Runtime Libraries, ) [Fixed] - Linq: Using the PredicateBuilder with a predicate that uses a comparison operator with one side the value null could lead to a crash because the expression tree formed by the PredicateBuilder was unexpected.
• [LLBLMAIN-2492] ( Runtime Libraries, ) [Implemented] - Linq: Rare race condition in edge case system when re-using dummy instance in linq provider could cause a query conversion to crash. The workaround implemented should mitigate this issue, which we ourselves haven't been able to reproduce

• Do we need to regenerate the code with 5.9.4-hotfix-20230209.

No, not if you're on 5.9.3

• Could please provide official date of 5.9.4-hotfix-20230209.

Somewhere next week, it's not yet set in stone.

Thank you so much! Is this fix would be part of 5.10 also?

Of course