I have a serious problem in my application with the JSON deserialization ..my subentities are "doubled" during the deserialization. I've tried everything but can't solve it, and since it's past 4.30AM for the second night in a row, I give up looking myself, so here goes ...

In attach you'll find a repro VS2013 solution, it's the solution I used for thread http://www.llblgen.com/TinyForum/Messages.aspx?ThreadID=23476 but adapted for this case. (that's why the solution is still called SampleConcurrency, but it's not about concurrency)

In a DB called "Sample" I have now 2 tables, "Sample" and "SampleParam", with a Foreign Key :

CREATE TABLE [dbo].[Sample](
    [Id] [int] IDENTITY(1,1) NOT NULL,
    Code [nvarchar](50) NOT NULL,
    [Description] [nvarchar](50) NULL,
    [Timestamp] [timestamp] NOT NULL,
CONSTRAINT [PK_Sample] PRIMARY KEY CLUSTERED 
(
    [Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]


CREATE TABLE [dbo].[SampleParam](
    [Id] [int] IDENTITY(1,1) NOT NULL,
    [SampleId] [int] NOT NULL,
    [Param] [nvarchar](50) NOT NULL,
    [Value] [nvarchar](50) NOT NULL,
 CONSTRAINT [PK_SampleParam] PRIMARY KEY CLUSTERED 
(
    [Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]

GO

ALTER TABLE [dbo].[SampleParam]  WITH CHECK ADD  CONSTRAINT [FK_SampleParam_SampleParam] FOREIGN KEY([SampleId])
REFERENCES [dbo].[Sample] ([Id])
GO

ALTER TABLE [dbo].[SampleParam] CHECK CONSTRAINT [FK_SampleParam_SampleParam]
GO

In each table I've just added one record (values don't matter):

Sample:
Id  Code    Description Timestamp
1   test    test    0x00000000000007D1
SampleParam:
Id  SampleId    Param   Value
1   1   param1  value1

I've changed the my WebAPI Controller to :

  public class SampleController : ApiController
    {
        [HttpGet]
        public SampleEntity GetSample()
        {
            using (var adapter = new DataAccessAdapter())
            {
                var metaData = new LinqMetaData(adapter);
                var result = metaData.Sample.WithPath(x=>x.Prefetch<SampleParamEntity>(pp => pp.SampleParams)).First();
                return result;
            }
        }

    }

Now when I fetch the sample-entity with the Prefetch path to also fetch the sample param, after serialization I got TWO (identical) sampleParam subobjects During debug I see the fetch is ok, also the JSON is correct:

{"$id":"1",
"Code":"test",
"Description":"test",
"Id":1,
"Timestamp":"AAAAAAAAB9E=",
"SampleParams":[{
        "$id":"2",
        "Id":1,
        "Param":"param1",
        "SampleId":1,
        "Value":"value1",
        "Sample":{"$ref":"1"}
        }]
}

I'm using the following JSON settings :

    public static class LLBLgenJsonSerializerSettings
    {
        public static JsonSerializerSettings Settings = new JsonSerializerSettings()
        {
            MissingMemberHandling = MissingMemberHandling.Ignore,
            NullValueHandling = NullValueHandling.Include,
            DefaultValueHandling = DefaultValueHandling.Include,
            //settings for LLBLGen
            PreserveReferencesHandling = PreserveReferencesHandling.Objects,
            ContractResolver = new DefaultContractResolver() { IgnoreSerializableInterface = true, IgnoreSerializableAttribute = true }
        };
    }

I created also a quick unit test, and the last line fails because it has 2 sampleparams instead of 1:

  [TestMethod]
        public void TestSubojects()
        {

            SampleEntity entity = new SampleEntity() {Code = "testcode", Description = "testdescription"};
            entity.SampleParams.Add(new SampleParamEntity() {Param = "param1", Value = "value1"});

            string json = JsonConvert.SerializeObject(entity, LLBLgenJsonSerializerSettings.Settings);
            SampleEntity deserialized = JsonConvert.DeserializeObject<SampleEntity>(json, LLBLgenJsonSerializerSettings.Settings);

            Assert.IsTrue(entity.Code == deserialized.Code);
            Assert.IsTrue(entity.Description == deserialized.Description);
            Assert.IsTrue(entity.SampleParams.Count == deserialized.SampleParams.Count);//fails because 1 != 2 ....


        }

So it has actually nothing to do with the whole WepAPI stuff, but purely the deserialization. The serialization is fine, in the json there is only 1 sample param entity ..

So I suspect it has something to do during the deserialization with events firing twice or so, or code being wrongfully called twice when instantiating entities ? ... I've tested with 2 sample param records in my database, and then I have 4 after deserialisation, 3 makes 6, etc ...

This also only happens when it's a 1-n relation. For n-1 relation, it's ok, the single subobject is not "doubled" then. That's why I didn't noticed this until this far in my project But it's absolutely crucial I get this to work...

Please help ...

Edit: I've created the same test, with simple POCO's instead of Entities and this test succeeds, with both counts are 1 Here's the full code for this test ( since it's not in the attachment I attached before this edit):

  public class Sample
    {
        public Sample()
        {
            SampleParams = new List<SampleParam>();
        }


        public int Id { get; set; }
        public string Code { get; set; }
        public string Description { get; set; }
        public IList<SampleParam> SampleParams { get; set; }
    }

    public class SampleParam
    {
        public int Id { get; set; }
        public int SampleId { get; set; }
        public string Param { get; set; }
        public string Value { get; set; }
    }


    
    [TestClass]
    public class UnitTest_NoEntity
    {
        [TestMethod]
        public void Same_Test_with_POCOs()
        {

            Sample entity = new Sample() {Code = "testcode", Description = "testdescription"};
            entity.SampleParams.Add(new SampleParam() {Param = "param1", Value = "value1"});

            string json = JsonConvert.SerializeObject(entity, LLBLgenJsonSerializerSettings.Settings);
            Sample deserialized = JsonConvert.DeserializeObject<Sample>(json, LLBLgenJsonSerializerSettings.Settings);

            Assert.IsTrue(entity.Code == deserialized.Code);
            Assert.IsTrue(entity.Description == deserialized.Description);
            Assert.IsTrue(entity.SampleParams.Count == deserialized.SampleParams.Count);// succeeds with 1 == 1 !!
        }



    }
}

So the bug (?) is probably in the llblgen part of the code, and not the newtonsoft.json part

Since it's quite easy reproducable, I hope it's just as easy fixable ...

(Haven't looked at your repro yet) I'm wondering: how can a bug be in our code if the deserialization is done by an external piece of code? I don't even know where to look when thinking about it. If json.net deserializes using the ISerializable API, our code does the deserialization and wiring up of the related elements. If the deserialization is done using reflection, our code isn't doing much. The thing I can think of is that it might introduce duplicates because of cyclic references (Order references customer, customer references order again through customer.Orders). It likely creates a new instance instead of re-using the existing one, but other than that... no idea.

Besides, deserialization using our tests which I documented worked fine without duplicates.

We don't raise events during graph building. I have no idea what code creates the two Sample instances. If you set a breakpoint in the InitClassEmpty() (or in all the constructors) of SampleEntity, can see the stacktrace of what code triggers the instantiation of the instances, the first is likely the right one, the second shouldn't happen and both are likely originating from another origin, one being wrong.

Hey,

I just have put a breakpoint on the constructors and InitClassEmpty method for the SampleParam entity, and they indeed only get called once in my Repro (if there is only one SampelParamEntity ofcourse). But the ParamEntity.SampleParams.Count == 2, not 1, after deserialization . So I've set the ParamEntity.SampleParams[0].Value to some teststring, and surely, the ParamEntity.SampleParams[1].Value became the same teststring, so they point to the same object.

I've also set a breakpoint on the SetRelatedEntity, where the subobjects gets added to the collection, and also only 1 call ...

Hey,

after bit more searching, and trying Newtonsof.Json older versions (6.0.1 & 5.0.1) which had the same behaviour, I've ended up dowloading the Json sourcecode from GitHub, adding references to my SampleConccurency.dll and ORMSupportClasses.dll to the NewtonSoft.Json.Tests project and adding the following test :

   [TestFixture]
    public class LLBLGenTest
    {

        [Test]
        public void TestSubojects()
        {

            SampleEntity entity = new SampleEntity() { Code = "testcode", Description = "testdescription" };
            entity.SampleParams.Add(new SampleParamEntity() { Param = "param1", Value = "value1" });

            string json = JsonConvert.SerializeObject(entity, LLBLgenJsonSerializerSettings.Settings);
            SampleEntity deserialized = JsonConvert.DeserializeObject<SampleEntity>(json, LLBLgenJsonSerializerSettings.Settings);

            Assert.IsTrue(entity.Code == deserialized.Code);
            Assert.IsTrue(entity.Description == deserialized.Description);
            Assert.IsTrue(entity.SampleParams.Count == deserialized.SampleParams.Count);


        }
    }

    public static class LLBLgenJsonSerializerSettings
    {
        public static JsonSerializerSettings Settings = new JsonSerializerSettings()
        {

            MissingMemberHandling = MissingMemberHandling.Ignore,
            NullValueHandling = NullValueHandling.Include,
            DefaultValueHandling = DefaultValueHandling.Include,
            //settings for LLBLGen
            PreserveReferencesHandling = PreserveReferencesHandling.Objects,
            ContractResolver = new DefaultContractResolver() { IgnoreSerializableInterface = true, IgnoreSerializableAttribute = true }
        };
    }

So now I can debug it and step through the code and watch my Entities.

The bulk of the work is going on in the JsonSerializerInternalReader and that's indeed where I encountered the behaviour.

After lots of stepping through code, suddenly at line 916 in JsonSerializerInternalReader, the following code is executed:

                property.ValueProvider.SetValue(target, value);

At a certain point in time, when the SampleParamEntity is being deserialized, its "Sample"-property is set on this line to the parent SampleEntity, and at that point the Sample.SampleParams.Count ==1, which is correct.

But a bit later, when the serialization of the SampleParamEntity is finished, the resulted object is added to the list of SampleParams on the parent entity, on line 1405.

         list.Add(value);

And so the object is in the list twice...

So basically what happens is that Json.NET deserializes the items of the collection when it encounters a collection type, and due to the 2-way parent/child relation, when setting the parent on the child the child gets added to the parent's childcollection. But then Json.NET itself adds the childs to the parent's childcollection.

That also explains why I didn't encounter it on n-1 relation (no collection) and I also didn't encounter it on my POCO clases unittest (no 2-way parent/child relation)

How could this be solved ? I see that the "list" object on line 1405 when the 2nd add happens is of type ((SampleConcurrency.HelperClasses.EntityCollection<SampleConcurrency.EntityClasses.SampleParamEntity>)(list)) so it's a class in my generated code that I can change. Perhaps I could adapt the "Add()" method or something to prevent adding objects that are already in the collection ? But I'm afraid by changing things in such a base class could break stuff later on in unpredictable ways ...

I've quickfixed it in my local copy of Json.NET for the moment and submitted an issue for it : https://github.com/JamesNK/Newtonsoft.Json/issues/658

Edit : I thought about the "fix" I proposed to json.net and don't think they will accept it ..what if you actually want to serialize/deserialize an object with a list of subobjects that has duplicates ..then this would break it.

So that brings me back to LLBLGen ...would it be possible to somehow notify the entities "hey, now we are in serialization mode, so you don't have to maintain the 2way parent-child relations yourself"

EDIT:
Ha, but there is already something in the entities that changes the behavior if it is deserializing that I might use.

In my generated code of SampleParamEntity I have its parent property :

        /// <summary> Gets / sets related entity of type 'SampleEntity' which has to be set using a fetch action earlier. If no related entity is set for this property, null is returned..<br/><br/></summary>
        [Browsable(false)]
        [DataMember]
        public virtual SampleEntity Sample
        {
            get { return _sample; }
            set
            {
                if(this.IsDeserializing)
                {
                    SetupSyncSample(value);
                }
                else
                {
                    SetSingleRelatedEntityNavigator(value, "SampleParams", "Sample", _sample, true); 
                }
            }
        }

The problem is that the IsDeserializing value is false ... if I put a breakpoint there, and set it to true, the SampleEntity.SampleParams collection doesn't get duplicate references anymore and the unittest succeeds (with the original Json.net, which I prefer)

But the IsDeserializing is defined in the EntityCore so I don't see its logic .. how and where should I set this property ?

Ah, I think I know what happens.

If you have myOrder and myCustomer, and you do: myOrder.Customer = myCustomer;

then this is true too: myCustomer.Orders.Contains(myOrder);

So if you then do: myCustomer.Orders.Add(myOrder)

it's in the list twice. There's a property on the collection, it's false by default: DoNotPerformAddIfPresent. If true, adding an entity object twice will make the second add fail. It's false by default because it's doing a linear search, which can be slow if there are many objects in the collection.

So the behavior you're seeing is actually 'by design' as the deserializer is simply adding objects to collections and setting associations like it's external code.

The IsDeserializing is indeed a way to solve it too, as it simply prevents myCustomer.Orders.Add(myOrder) after you do myOrder.Customer = myCustomer;

I'll see what it is made of and get back to you.

ok, super, thnx !!

in the meantime, I've just finished a quick extension method to remove the duplicates right after i deserialize them, so for now I can continue:

Something like :

     public static void RemoveDuplicates<T>(this T entity) where T : IEntity2
        {
            ObjectGraphUtils oga = new ObjectGraphUtils();
            foreach (var e in oga.ProduceTopologyOrderedList((IEntity2) entity))
                foreach (var coll in e.GetMemberEntityCollections())
                  RemoveDuplicatesFromCollection(coll);
        }

        private static void RemoveDuplicatesFromCollection(IEntityCollection2 coll)
        {
            for (int i = coll.Count-1; i >= 0; i--)
            {
                IEntity2 entity = coll[i];
                int count = 0;
                foreach (var e in coll)
                    if (ReferenceEquals(e, entity))
                        count++;
                if (count>1)
                   coll.RemoveAt(i);
            }
            
        }

still have to test it though, literally typed the last letter code when the notification of a new message on this thread came in

Removing things after the fact is a stopgap and using that will stick into the codebase forever, so I'd not do that

The IsDeserializing is set when we know we're deserializing (e.g. in xml deserializing or binary serializing through ISerializable), and with this that's not the case. It's a bit of a pain as there's no code inside the entities which knows deserialization is going on, it simply looks like any other code which adds/removes entities from the collections.

Is there a way to intercept the entity construction using Json.net? THen it would be possible to set/reset the flag.

Well, in my solution it's only happening in 1 place luckily, the following extension methods :

     /// <summary>
        /// Serializes an entity (+sub-entities) to JSON
        /// </summary>
        /// <typeparam name="T"></typeparam>
        /// <param name="entity"></param>
        /// <returns></returns>
        public static string ToJson<T>(this T entity) where T : IEntity2
        {
            string result = JsonConvert.SerializeObject(entity, EntityJsonSerializer.Settings);
            return result;
        }

        /// <summary>
        /// Creates an entity (+ subentities) from a JSON string and indicates it as non-dirty
        /// </summary>
        /// <typeparam name="T"></typeparam>
        /// <param name="json"></param>
        /// <returns>The entity (graph), made non-dirty</returns>
        public static T FromJsonToEntity<T>(this string json) where T : IEntity2
        {

            var result = JsonConvert.DeserializeObject<T>(json, EntityJsonSerializer.Settings);
            return result.RemoveDuplicates().MakeNonDirty();
        }

        /// <summary>
        /// Creates a list of entities (+ subentities) from a JSON string and indicates it as non-dirty
        /// </summary>
        /// <typeparam name="T"></typeparam>
        /// <param name="json"></param>
        /// <returns>The entity (graph), made non-dirty</returns>
        public static IList<T> FromJsonToEntityList<T>(this string json) where T : IEntity2
        {

            var result = JsonConvert.DeserializeObject<IList<T>>(json, EntityJsonSerializer.Settings);
            foreach (var e in result)
                e.RemoveDuplicates().MakeNonDirty();
            return result;
        }

(the makenondirty extension method is for the fact that after deserialization all fields are "changed" and all entities are "isnew" )

So on the client side, it's no problem. I know perfectly when I'm serializing/deserializing.

The server side might be a bit more difficult... by the time I enter my WebAPI controller, my objects are already deserialized by the framework. So I should investigate where that's happening and how I can intercept it ...

I think I might have a solution. You can force Json to use a specific constructor with the JsonConstructorAtrribute

I've tested this in my sample project by adding to SampleEntity and SampleParamEntity the follwing constructors :

     [JsonConstructor]
        public SampleEntity(bool callViaJson) : base("SampleEntity")
        {
          //callViaJson is ignored, was just to have a unique constructor signature
             this.IsDeserializing = true;
            InitClassEmpty(null, null);
        }


  [JsonConstructor]
        public SampleParamEntity(bool callViaJson) : base("SampleParamEntity")
        {
            this.IsDeserializing = true;
            InitClassEmpty(null, null);
        }

And it works !! on the plus side, this should work with WebAPI too (have to test it though)

Maybe this could be something to add to my templates ?

Only thing remaining is setting the IsDeserializing back to false, but for that I could do it in my "MakeNonDirty" routine ?

Oh, and also the IsDeserializing flag will be set during serialization too then, but that doesn't matter I supposev nvm, stupid me

EDIT: I can confirm it works in WebAPI too !!

I've added a bit of code to my testclient, and a HttPost handler to the WebAPI controller so I immediately post back the item I retrieved :

        private void button1_Click(object sender, EventArgs e)
        {
            string server = "http://localhost.fiddler:1470/";
            string controller = "/api/sample";
            var client = new RestClient(server);
            var request = new RestRequest(controller, Method.GET);
            var response = client.Execute<SampleEntity>(request);
            var json = response.Content;
            var sample = JsonConvert.DeserializeObject<SampleEntity>(json, LLBLgenJsonSerializerSettings.Settings);
            MessageBox.Show("number of params :" + sample.SampleParams.Count);
            //send it back to the server !
            request = new RestRequest(controller, Method.POST);
            request.JsonSerializer = new EntityJsonSerializer();
            request.AddJsonBody(sample);
            client.Execute<SampleEntity>(request);
        }

//WebAPI controller : 

   [HttpPost]
        public void PostSample(SampleEntity sample)
        {
            Debug.WriteLine(sample.SampleParams.Count);
        }

Now if I comment out my JsonConstructor's, I got 2 SampleParams after the first HttpGet, and I got 3 SampleParams when arriving back on the server upon HttpPost. (And I'm sure the count goes up by 1 on every transmit) But when I uncomment the JsonConstructors, the magic works and the SampleParams.Count == 1, as well on the client as after posting back ... so the WebAPI picks it up too !!

Pfew ..that was a tricky one ..my weekend can start (actually I got some Xamarin work to do now )

Glad it's working! One reminder: if you keep the deserialized entities around, be sure to reset the IsDeserializing to false again.

yup, I have not forgotten

        /// <summary>
        /// Creates an entity (+ subentities) from a JSON string and indicates it as non-dirty
        /// </summary>
        /// <typeparam name="T"></typeparam>
        /// <param name="json"></param>
        /// <returns>The entity (graph), made non-dirty</returns>
        public static T FromJsonToEntity<T>(this string json) where T : IEntity2
        {

            var result = JsonConvert.DeserializeObject<T>(json, EntityJsonSerializer.Settings);
            result.IsDeserializing = false;
            return result.MakeNonDirty();
        }

Had the same problem, here's the lpt template I wrote, nothing special:

// C#
using System.Runtime.Serialization;

namespace <%=_executingGenerator.RootNamespaceToUse %>.EntityClasses
{
    public partial class <%= ((EntityDefinition)_activeObject).Name %>Entity
    {
        [OnDeserializing]
        internal void OnDeserializingMethod(StreamingContext context)
        {
            IsDeserializing = true;
        }

        [OnDeserialized]
        internal void OnDeserializedMethod(StreamingContext context)
        {
            IsDeserializing = false;
        }
    }
}

Edited to be logically more correct and to handle it setting to false after deserialization is complete.

Thanks for the feedback.