Quick question. I'm trying to deploy a .Net Core 3.0 app to an Ubuntu instance, and I'm getting the following exception. This app has run on Linux in the past, so I'm not sure what's changed, but I'm not sure where this issue is coming from:

Application startup exception: SD.LLBLGen.Pro.ORMSupportClasses.ORMConfigurationException: The static field 'Instance' of type 'System.Data.SqlClient.SqlClientFactory' is null and therefore this DbProviderFactory can't be used.
   at SD.LLBLGen.Pro.ORMSupportClasses.DQEConfigurationBase.StoreDbProviderFactoryRegistration(Type factoryType, String invariantName)
   at SD.LLBLGen.Pro.DQE.SqlServer.SQLServerDQEConfiguration.AddDbProviderFactory(Type factoryType, String invariantName)
   at QuestMetrics.Insight.Server.Web.AppHostSurvey.<>c.<Configure>b__1_1(SQLServerDQEConfiguration c) in C:\git\insight-server\SurveyServer\Server\Insight.Server.Web\AppHostSurvey.cs:line 62
   at SD.LLBLGen.Pro.ORMSupportClasses.RuntimeConfiguration.ConfigureDQE[TDQEConfig](Action`1 configureFunc)

The config code is simply

            RuntimeConfiguration.AddConnectionString(DataAccessAdapter.ConnectionStringKeyName,Globals.Config.DatabaseConnectionString);
            RuntimeConfiguration.ConfigureDQE<SQLServerDQEConfiguration>(c =>
                {
                    c.AddDbProviderFactory(typeof(System.Data.SqlClient.SqlClientFactory)); <== Exception thrown here
                    var compat = Enum.Parse<SqlServerCompatibilityLevel>(Globals.Config.DatabaseCompatibility);
                    c.SetDefaultCompatibilityLevel(compat);
                });

Any suggestions?

Hmm, that's a static property that should be filled by a static ctor. You deploy the default SqlClient assembly with the app? Or the newer Microsoft.Data.SqlClient?

Otis wrote:

Hmm, that's a static property that should be filled by a static ctor. You deploy the default SqlClient assembly with the app? Or the newer Microsoft.Data.SqlClient?

System.Data.SqlClient, Version=4.6.1.1, .NETStandard Version=v2.0

Ok, and if you, before that code, retrieve the value from the static property 'Instance' of the SqlClientFactory into a variable and test it for null is it indeed null? As that would be bizarre. The CLR has to run the static ctors before the type is used...

Otis wrote:

Ok, and if you, before that code, retrieve the value from the static property 'Instance' of the SqlClientFactory into a variable and test it for null is it indeed null? As that would be bizarre. The CLR has to run the static ctors before the type is used...

It is indeed very strange. I assumed it was framework related, updated everything to Core 3.1/ Standard 2.1, and added some debugging info:

                var factoryType = typeof(System.Data.SqlClient.SqlClientFactory);
                Console.WriteLine($"factoryType = {factoryType.AssemblyQualifiedName}");
                Console.WriteLine($"factoryType.BaseType = {factoryType.BaseType.AssemblyQualifiedName}");
                System.Reflection.FieldInfo field = factoryType.GetField("Instance", BindingFlags.DeclaredOnly | BindingFlags.Static | BindingFlags.Public);
                Console.WriteLine($"field for factoryType.Instance = {field?.Attributes}");
                Console.WriteLine($"field.FieldType = {field.FieldType}");
                Console.WriteLine($"field.FieldType.BaseType = {field.FieldType.BaseType}");
                Console.WriteLine($"field.FieldType.IsSubclassOf = {field.FieldType.IsSubclassOf(typeof (DbProviderFactory))}");
                Console.WriteLine($"field.GetValue typecheck = {field.GetValue((object) null) is DbProviderFactory dbProviderFactory}");
                Console.WriteLine($"field.GetValue = {field.GetValue((object) null)}");
                Console.WriteLine($"field.GetValue FullName = {field.GetValue((object) null)?.GetType()?.FullName}");
                
                c.AddDbProviderFactory(factoryType);

The weird thing is that I still get the error:

factoryType = System.Data.SqlClient.SqlClientFactory, System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
factoryType.BaseType = System.Data.Common.DbProviderFactory, System.Data.Common, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
field for factoryType.Instance = Public, Static, InitOnly
field.FieldType = System.Data.SqlClient.SqlClientFactory
field.FieldType.BaseType = System.Data.Common.DbProviderFactory
field.FieldType.IsSubclassOf = True
field.GetValue typecheck = False
field.GetValue = 
field.GetValue FullName = 
Application startup exception: SD.LLBLGen.Pro.ORMSupportClasses.ORMConfigurationException: The static field 'Instance' of type 'System.Data.SqlClient.SqlClientFactory' is null and therefore this DbProviderFactory can't be used.
   at SD.LLBLGen.Pro.ORMSupportClasses.DQEConfigurationBase.StoreDbProviderFactoryRegistration(Type factoryType, String invariantName)
   at SD.LLBLGen.Pro.DQE.SqlServer.SQLServerDQEConfiguration.AddDbProviderFactory(Type factoryType, String invariantName)

Ok, so running the code to obtain the factory instance in your own code also fails (as it returns null, not an instance).

Which means SqlClientFactory's static field isn't initialized:

https://github.com/dotnet/SqlClient/blob/master/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlClientFactory.cs#L13

I don't know of a situation where this might happen.

In any case, for testing, if you use Microsoft.Data.SqlClient instead (use "Microsoft.Data.SqlClient" as invariant name), do you see the same result?

Can you also reproduce this behavior with the testcode you have in a console app not using our runtime? (so a simple console app on .net core, reading the sqlclient instance).

Otis wrote:

Ok, so running the code to obtain the factory instance in your own code also fails (as it returns null, not an instance).

Which means SqlClientFactory's static field isn't initialized:

https://github.com/dotnet/SqlClient/blob/master/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlClientFactory.cs#L13

I don't know of a situation where this might happen.

In any case, for testing, if you use Microsoft.Data.SqlClient instead (use "Microsoft.Data.SqlClient" as invariant name), do you see the same result?

Can you also reproduce this behavior with the testcode you have in a console app not using our runtime? (so a simple console app on .net core, reading the sqlclient instance).

I already tried

                var xx = System.Data.SqlClient.SqlClientFactory.Instance;
                xx.CreateCommand();

and that fails on the Linux instance. There's no doubt it's unrelated to LLBLGen, so don't waste time chasing it, but it is a fascinatingly bizarre issue. I only found one reference to this effect, and that developer rewrote their solution and the problem vanished.

I did just try using Microsoft.Data.SqlClient, but I get the following error, so that's something to chase up tomorrow.

SD.LLBLGen.Pro.ORMSupportClasses.ORMGeneralOperationException: DbProviderFactory information for compatibilitylevel SqlServer2012 wasn't properly initialized. Please use the RuntimeConfiguration.ConfigureDQE method to register the SqlClient DbProviderFactory with this DynamicQueryEngine. See the RuntimeConfiguration documentation for examples. at at SD.LLBLGen.Pro.DQE.SqlServer.SqlServerSpecificCreator.ObtainDbProviderFactoryInfo(SqlServerCompatibilityLevel compatibilityLevel) at at SD.LLBLGen.Pro.DQE.SqlServer.SqlServerSpecificCreator.SetDefaultDbProviderFactoryParameterData(SqlServerCompatibilityLevel compatibilityLevel) at at SD.LLBLGen.Pro.DQE.SqlServer.DynamicQueryEngine.Configure(SQLServerDQEConfiguration configuration) at at SD.LLBLGen.Pro.DQE.SqlServer.SQLServerDQEConfiguration.Configure() at at SD.LLBLGen.Pro.ORMSupportClasses.RuntimeConfiguration.ConfigureDQE[TDQEConfig](Action`1 configureFunc) at QuestMetrics.Insight.Server.Web.AppHostSurvey.Configure(Container container) in

I did set the compatibility to 2012:

                var compat = Enum.Parse<SqlServerCompatibilityLevel>(Globals.Config.DatabaseCompatibility);
                c.SetDefaultCompatibilityLevel(compat);

But now it's bedtime here in Oz.

JSobell wrote:

Otis wrote:

Ok, so running the code to obtain the factory instance in your own code also fails (as it returns null, not an instance).

Which means SqlClientFactory's static field isn't initialized:

https://github.com/dotnet/SqlClient/blob/master/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlClientFactory.cs#L13

I don't know of a situation where this might happen.

In any case, for testing, if you use Microsoft.Data.SqlClient instead (use "Microsoft.Data.SqlClient" as invariant name), do you see the same result?

Can you also reproduce this behavior with the testcode you have in a console app not using our runtime? (so a simple console app on .net core, reading the sqlclient instance).

I already tried

                var xx = System.Data.SqlClient.SqlClientFactory.Instance;
                xx.CreateCommand();

and that fails on the Linux instance. There's no doubt it's unrelated to LLBLGen, so don't waste time chasing it, but it is a fascinatingly bizarre issue. I only found one reference to this effect, and that developer rewrote their solution and the problem vanished.

It's bizarre. The code above is that in a standalone console app? The reason I asked is that it might be related to the order in which static constructors are running, and doing this inside a startup of asp.net might (but grasping at straws here) be differently, but frankly I have no idea.

I did just try using Microsoft.Data.SqlClient, but I get the following error, so that's something to chase up tomorrow.

SD.LLBLGen.Pro.ORMSupportClasses.ORMGeneralOperationException: DbProviderFactory information for compatibilitylevel SqlServer2012 wasn't properly initialized. Please use the RuntimeConfiguration.ConfigureDQE method to register the SqlClient DbProviderFactory with this DynamicQueryEngine. See the RuntimeConfiguration documentation for examples. at at SD.LLBLGen.Pro.DQE.SqlServer.SqlServerSpecificCreator.ObtainDbProviderFactoryInfo(SqlServerCompatibilityLevel compatibilityLevel) at at SD.LLBLGen.Pro.DQE.SqlServer.SqlServerSpecificCreator.SetDefaultDbProviderFactoryParameterData(SqlServerCompatibilityLevel compatibilityLevel) at at SD.LLBLGen.Pro.DQE.SqlServer.DynamicQueryEngine.Configure(SQLServerDQEConfiguration configuration) at at SD.LLBLGen.Pro.DQE.SqlServer.SQLServerDQEConfiguration.Configure() at at SD.LLBLGen.Pro.ORMSupportClasses.RuntimeConfiguration.ConfigureDQE[TDQEConfig](Action`1 configureFunc) at QuestMetrics.Insight.Server.Web.AppHostSurvey.Configure(Container container) in

I did set the compatibility to 2012:

                var compat = Enum.Parse<SqlServerCompatibilityLevel>(Globals.Config.DatabaseCompatibility);
                c.SetDefaultCompatibilityLevel(compat);

I made a mistake up above, where I said you have to specify "Microsoft.Data.SqlClient" as invariant name, that's wrong. You can just use the same code as you had and use

c.AddDbProviderFactory(typeof(MicrosoftData.SqlClient.SqlClientFactory)); 

instead of

c.AddDbProviderFactory(typeof(System.Data.SqlClient.SqlClientFactory)); 

Using a different invariant name will cause a problem indeed later on. This isn't documented well, we'll fix that a.s.a.p.

Strange, but still no.

factoryType = Microsoft.Data.SqlClient.SqlClientFactory, Microsoft.Data.SqlClient, Version=1.12.20106.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5
factoryType.BaseType = System.Data.Common.DbProviderFactory, System.Data.Common, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
field for factoryType.Instance = Public, Static, InitOnly
field.FieldType = Microsoft.Data.SqlClient.SqlClientFactory
field.FieldType.BaseType = System.Data.Common.DbProviderFactory
field.FieldType.IsSubclassOf = True
field.GetValue typecheck = False
field.GetValue = 
field.GetValue FullName = 

If I find out what it is, I'll let you know. It's very odd.

I'd create a console app that reproduces the behavior and open an issue at the SqliClient repo on github (https://github.com/dotnet/SqlClient/ ) to get to the bottom of it.

I finally got to the bottom of the issue (after many hours). The solution being used contained a "Deployment" project that had dependencies on three different services. The output folder therefore contained all three services, and those were published to the server. But... MSBuild only builds a *.deps.json in the output folders of each service and doesn't copy them with the files when referenced as dependencies. That means we had a Dependencies.deps.json, and running dotnet Dependencies.dll worked fine, but running dotnet Service1.dll gave these obscure error messages. It turns out that without the deps.json specifying the exact sqlclient.dll to use, it defaults to a placeholder version with no implementation. But instead of raising an exception, it just does nothing, hence the null returned from the static constructor. Thanks Microsoft! 10/10 for appropriate use of exceptions, and why would anyone ever want to construct an instance of an SQLConnection with no implementation?

Oh, and this was secondary issue after the first that I didn't mention. Whereas older version of .NET Core weren't overly fussy about the app.config XML structure, the new Microsoft.Data.SQLClient bombs out if the app.config is not perfect with an exception complaining about the connectionstring.

Make sure you don't add any sections that are not defined in <configSections>, otherwise it throws an unrelated exception. This occurs even if your code doesn't have any connections defined in the app.config, so it's effectively acting as an unrequested app.config XML validator.

The most annoying thing is that the app.config XSD schema doesn't enforce this structure, so you get no warnings at design time.

For example, this breaks SQLClient:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <configSections>
    <sectionGroup name="common">
      <section name="logging" type="Common.Insight.Logging.ConfigurationSectionHandler, Common.Insight.Logging"/>
    </sectionGroup>
  </configSections>

  <appSettings>
    <add key="ConfigFolder" value="./config" />
  </appSettings>

  <common>
    <logging>
      <factoryAdapter type="Common.Insight.Logging.Simple.ConsoleOutLoggerFactoryAdapter, Common.Insight.Logging">
        <arg key="level" value="ALL"/>
        <arg key="showLogName" value="true"/>
        <arg key="showDataTime" value="true"/>
        <arg key="dateTimeFormat" value="yyyy/MM/dd HH:mm:ss:fff"/>
      </factoryAdapter>
    </logging>
  </common>

  <system.diagnostics>
    <trace autoflush="true">
      <listeners>
        <add name="consoleListener" type="System.Diagnostics.ConsoleTraceListener"/>
      </listeners>
    </trace>
  </system.diagnostics>

</configuration>

You now have to add this to the configSections:

    <section name="system.diagnostics"
             type="System.Configuration.SingleTagSectionHandler" />

I know this has always been a documented requirement, but it never used to be enforced, and definitely not by the SQLClient. I posted the hopelessly inappropriate exception as an issue, and it should be fixed in a future release.

Holy crap, that it's some serious issue indeed, glad you found it! (and that there's a way around it!)

Thanks for sharing, so others who run into this can avoid the mess.