Hi.

We are running an asp.net v4.7.2 in an azure app service environment. We are running up against an azure mssql database. We are using runtime Version v4.0.30319 of llbl.

Every 3 months or so we get this error for all our clients using the system.

System.ArgumentException: Object must be of type String.
   at System.Globalization.CompareInfo.InternalGetGlobalizedHashCode(IntPtr handle, IntPtr handleOrigin, String localeName, String source, Int32 length, Int32 dwFlags, Boolean forceRandomizedHashing, Int64 additionalEntropy)
   at System.Globalization.CompareInfo.GetHashCodeOfString(String source, CompareOptions options, Boolean forceRandomizedHashing, Int64 additionalEntropy)
   at System.CultureAwareComparer.GetHashCode(String obj)
   at SD.LLBLGen.Pro.ORMSupportClasses.FieldUtilities.CalculateHashCodeForValue(Object value, Boolean caseSensitiveStringHashCode)
   at SD.LLBLGen.Pro.ORMSupportClasses.EntityField2.GetHashCode()
   at System.Collections.Generic.ObjectEqualityComparer`1.GetHashCode(T obj)
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at SD.LLBLGen.Pro.ORMSupportClasses.DynamicQueryEngineBase.AppendFieldToValueFragmentsForInsert(IActionQuery query, Dictionary`2 fieldToParameter, QueryFragments valueFragments, IEntityFieldCore field, IFieldPersistenceInfo persistenceInfo)
   at SD.LLBLGen.Pro.DQE.SqlServer.DynamicQueryEngine.CreateSingleTargetInsertDQ(IEntityFieldCore[] fields, IFieldPersistenceInfo[] fieldsPersistenceInfo, IActionQuery query, Dictionary`2 fieldToParameter)
   at SD.LLBLGen.Pro.ORMSupportClasses.DynamicQueryEngineBase.CreateInsertDQ(IEntityFieldCore[] fields, IFieldPersistenceInfo[] fieldsPersistenceInfo, DbConnection connectionToUse)
   at SD.LLBLGen.Pro.ORMSupportClasses.DataAccessAdapterCore.CreateInsertDQ(IEntity2 entityToSave, IFieldPersistenceInfo[] persistenceInfoObjects)
   at SD.LLBLGen.Pro.ORMSupportClasses.DataAccessAdapterCore.CreateQueryForEntityToSave(Boolean insertActions, IEntity2 entityToSave, IPredicateExpression updateRestriction, InheritanceHierarchyType typeOfHierarchy, IFieldPersistenceInfo[] persistenceInfoObjects)
   at SD.LLBLGen.Pro.ORMSupportClasses.DataAccessAdapterCore.ConstructSaveQueryForEntity(Boolean insertActions, ActionQueueElement`1 element, EntityBase2 entityToSave)
   at SD.LLBLGen.Pro.ORMSupportClasses.DataAccessAdapterCore.PersistQueue(List`1 queueToPersist, Boolean insertActions)
   at SD.LLBLGen.Pro.ORMSupportClasses.UnitOfWork2.HandleInserts(DataAccessAdapterBase adapterToUseAsBase)
   at SD.LLBLGen.Pro.ORMSupportClasses.UnitOfWork2.CommitImpl(IDataAccessAdapter adapterToUse, Boolean autoCommit)
   at SD.LLBLGen.Pro.ORMSupportClasses.UnitOfWork2.<>c__DisplayClass47_0.<Commit>b__0()
   at SD.LLBLGen.Pro.ORMSupportClasses.UnitOfWork2.Commit(IDataAccessAdapter adapterToUse, Boolean autoCommit)
   at SD.LLBLGen.Pro.ORMSupportClasses.DataAccessAdapterCore.SaveEntityCollection(IEntityCollection2 collectionToSave, Boolean refetchSavedEntitiesAfterSave, Boolean recurse)
   at SD.LLBLGen.Pro.ORMSupportClasses.DataAccessAdapterBase.<>c__DisplayClass20_0.<SaveEntityCollection>b__0()
   at SD.LLBLGen.Pro.ORMSupportClasses.RecoveryStrategyBase.Execute[TResult](Func`1 toExecute)
   at SD.LLBLGen.Pro.ORMSupportClasses.DataAccessAdapterBase.ExecuteWithActiveRecoveryStrategy[T](Func`1 toExecute)
   at SD.LLBLGen.Pro.ORMSupportClasses.DataAccessAdapterBase.SaveEntityCollection(IEntityCollection2 collectionToSave, Boolean refetchSavedEntitiesAfterSave, Boolean recurse)
   at SD.LLBLGen.Pro.ORMSupportClasses.DataAccessAdapterCore.SaveEntityCollection(IEntityCollection2 collectionToSave)
   at Flyware.eLSManagers.Components.BatchProcess.Commit(IDataAccessAdapter adapter) in C:\Dev\Solution\source\eLS-Solution\Flyware.eLSManagers\Components\BatchProcess.cs:line 67

 public int Commit(IDataAccessAdapter adapter)
        {
            _logger.Debug("Commit");

            if (adapter == null)
            {
                _logger.Error("Commit: adapter == null throw ArgumentNullException");
                throw new ArgumentNullException("adapter", "You have to specify a valid adapter object");
            }

            int totalAmountAffected = 0;
            try
            {
                if (!adapter.IsTransactionInProgress)
                    adapter.StartTransaction(System.Data.IsolationLevel.ReadCommitted, "bp");

                // Delete the entites in reverse order
                _logger.Debug("Commit: Delete Collection");
                for (int i = _collectionsToDelete.Count - 1; i >= 0; i--)
                {
                    EntityBase2 entity = (EntityBase2)_collectionsToDelete[i];

                    if (adapter.DeleteEntity(entity))
                        totalAmountAffected++;
                }

                if (_collectionsToSave.Count > 0)
                {
                    _logger.Debug("Commit: Save Collection");
                    totalAmountAffected += adapter.SaveEntityCollection(_collectionsToSave); // line 67 that crashes
                }

                _logger.Debug("Commit: adapter.Commit");
                adapter.Commit();
                Reset();
            }
            catch (Exception ex)
            {
                _logger.Error("Commit:",ex);
                adapter.Rollback();
                IExceptionMapping exMapping = ExceptionMappingFactory.GetMapper<IExceptionMapping>();
                exMapping.HandleDalException(ex);
            }

I am not sure why something would stop being a string in an entity and give the error "Object must be of type String."

Thanks Ole

A strange indeterministic issue indeed.

I highly suggest that you use the latest build of v.4.0, or better the latest of v.4.2. In either case, please report back if this issue ever re-occurred.

Also please check the real version of the runtime you're using. 4.0.30319 sounds like the .NET Framework version as we never released a version with that fileversion. Please see: https://www.llblgen.com/Documentation/5.9/LLBLGen%20Pro%20RTF/Using%20the%20generated%20code/gencode_compiling.htm#requesting-the-runtime-libraries-build-number-and-version-number (code is the same for all our runtimes)

Looking at the code it goes wrong in a piece of code which first casts the value to string and if that's not null it ends up in the code path where it crashes. So this is very very weird.

Otis wrote:

Also please check the real version of the runtime you're using. 4.0.30319 sounds like the .NET Framework version as we never released a version with that fileversion. Please see: https://www.llblgen.com/Documentation/5.9/LLBLGen%20Pro%20RTF/Using%20the%20generated%20code/gencode_compiling.htm#requesting-the-runtime-libraries-build-number-and-version-number (code is the same for all our runtimes)

Looking at the code it goes wrong in a piece of code which first casts the value to string and if that's not null it ends up in the code path where it crashes. So this is very very weird.

ok I can see that the file version is 5.3.1.0 .

I don't see a fix in this area in newer versions of the runtime. (I also don't see how the current code can fail in this situation to be honest). This is for all the users on the system at one point, so no matter what is saved, it will cause this problem, and it keeps popping up till you recycle the app domain?

Otis wrote:

I don't see a fix in this area in newer versions of the runtime. (I also don't see how the current code can fail in this situation to be honest). This is for all the users on the system at one point, so no matter what is saved, it will cause this problem, and it keeps popping up till you recycle the app domain?

It is only when a particular object (FlightMessageDocOutput ) is going to be saved that the system fails. After a recycle the system runs normal again.

code where the FlightMessageDocOutput is saved.

 public List<FlightMessageDocOutput> SaveFlightMessageDocOutput(Guid flightId, List<FlightMessageDocOutput> flightMessageDocOutputs)
        {
            Logger.Info(string.Format("SaveFlightMessageDocOutput: flightId={0},count={1}", flightId, (flightMessageDocOutputs != null) == true ? flightMessageDocOutputs.Count.ToString() : "null"));
            LogAction("SaveFlightMessageDocOutput");

            EntityCollection<CommonEntityBase> deleteList = new EntityCollection<CommonEntityBase>();
            EntityCollection<CommonEntityBase> saveList = new EntityCollection<CommonEntityBase>();

            foreach (FlightMessageDocOutput fmdo in flightMessageDocOutputs)
            {
                Validate(fmdo, "FlightMessageDocOutput");
                saveList = FlightMessageDocOutputConverter.GetSaveList(fmdo, saveList, flightId, string.Empty);
                deleteList = FlightMessageDocOutputConverter.GetDeleteList(fmdo, deleteList, string.Empty);
            }

            Save(saveList, deleteList);  // line that crashes.

            List<FlightMessageDocOutput> boDelete = new List<FlightMessageDocOutput>();
            foreach (FlightMessageDocOutput fmdo in flightMessageDocOutputs)
            {
                if (fmdo.ObjectState == BaseObjectState.Deleted)
                    boDelete.Add(fmdo);
                else
                    FlightMessageDocOutputConverter.Cleanup(fmdo, string.Empty);
            }

            foreach (FlightMessageDocOutput fmdo in boDelete)
                flightMessageDocOutputs.Remove(fmdo);

            return flightMessageDocOutputs;

        }

I honestly have no idea why it crashes there, Ole, I'm sorry. As it's every 3 months or so, it's also hard to find the root cause I think: logging every FlightMessageDocOutput's field data in a file when they're saved (to see if a value is wrong) is not really doable I think.

What you could try perhaps is to override CreateInsertDQ() in a partial class of the generated DataAccessAdapter class (this method has been moved in v5.8 to a separate class so introducing this will cause a breaking change when you migrate to the last version so make a note of that, but it's primarily meant here for determining what the root cause might be). In the override, wrap the call to the base class method in a try/catch and catch the ArgumentException. In that catch clause you know the entity (as it's passed in), and you can write out/log the values of the fields of the entity to see why it fails. To mimic what the internal code does, you can loop over the fields in the entity (in the Fields property) and for each EntityField2 instance in the Fields property, call GetHashCode(). It should crash again with the same exception.

My suspicion however is that it might be memory related as it crashes after 3 months, but that's just guesswork, as I have no idea why the code crashes so rarely.

Unfortunately it is starting to look like a more often event , it has crashed now 3 times within the last week.

Otis wrote:

I honestly have no idea why it crashes there, Ole, I'm sorry. As it's every 3 months or so, it's also hard to find the root cause I think: logging every FlightMessageDocOutput's field data in a file when they're saved (to see if a value is wrong) is not really doable I think.

What you could try perhaps is to override CreateInsertDQ() in a partial class of the generated DataAccessAdapter class (this method has been moved in v5.8 to a separate class so introducing this will cause a breaking change when you migrate to the last version so make a note of that, but it's primarily meant here for determining what the root cause might be). In the override, wrap the call to the base class method in a try/catch and catch the ArgumentException. In that catch clause you know the entity (as it's passed in), and you can write out/log the values of the fields of the entity to see why it fails. To mimic what the internal code does, you can loop over the fields in the entity (in the Fields property) and for each EntityField2 instance in the Fields property, call GetHashCode(). It should crash again with the same exception.

My suspicion however is that it might be memory related as it crashes after 3 months, but that's just guesswork, as I have no idea why the code crashes so rarely.

Do you mean something like this ?.

namespace Flyware.eLSDal.DatabaseSpecific
{
    public partial class DataAccessAdapter
    {
        //protected override RecoveryStrategyBase CreateRecoveryStrategyToUse()
        //{
        //    return new SqlAzureRecoveryStrategy();
        //}

        protected override IActionQuery CreateInsertDQ(IEntity2 entityToSave, IFieldPersistenceInfo[] persistenceInfoObjects)
        {
            try
            {
                return base.CreateInsertDQ(entityToSave, persistenceInfoObjects);
            }
            catch (ArgumentException ex)
            {
                foreach (EntityField2 item in entityToSave.Fields)
                {
                    try { item.GetHashCode(); }
                    catch (Exception hex)
                    {
                        string exmesagge = "llbl gen error in field: " + item.Name + " with value: " + item.CurrentValue;
                        throw new Exception(exmesagge,hex);
                    }

                }
                throw ;
            }
        }

    }
}

Yes something like that, with the exception perhaps that I'd log more information regardless, you're now doing a throw at the bottom, which will give you the exception you're currently running into and that's not telling you which entity with which PK value and e.g. other info you might need to reproduce the problem (e.g. which values are in the entity. Serialize the entity to XML perhaps or read the values using CurrentValue on the field... )