Data Member Ordering

Posts   
 
    
TopDog74
User
Posts: 40
Joined: 27-Apr-2012
# Posted on: 02-Dec-2016 17:19:22   

Is it possible to add ordering to the DataMember attribute in each property in my DTO contracts.

i.e.


[DataMember(Order=1)]
public string SomeProperty {get; set;}

[DataMember(Order=2)]
public string SomeOtherProperty {get; set;}

I'd like to ensure that the column ordering is not automatically changed if i add a new property to the contract.

Ideally, what i am trying to achieve is to ensure that when i add new properties to a contract, any clients using the last version of the contract do not automatically hit a serialization error. i.e. it would be acceptable for newly added properties to be ignored.

daelmo avatar
daelmo
Support Team
Posts: 8245
Joined: 28-Nov-2005
# Posted on: 03-Dec-2016 02:50:54   

AFAIK, this is not possible in general. Normally you would add macros to the attribute in the project settings using macros. However there is not macro's keyword for that property (field order). Ref: http://www.llblgen.com/documentation/5.0/Designer/Functionality%20Reference/ElementDefinitionMacros.htm

What you could do is: 1. Edit your Derived Model 2. Go to Code Generation Info sub-tab 3. Click on the first field 4. Go to Attributes sub-tab 5. Add the Attribute definition manually. DataMember(Order=1) 6. Do (3-5) for the rest of the fields.

I know, it could be tedious but it should work for what you want to achieve. I have to check whether or not the 'Search for elements' tool could help here to do the automated work.

David Elizondo | LLBLGen Support Team
Otis avatar
Otis
LLBLGen Pro Team
Posts: 39588
Joined: 17-Aug-2003
# Posted on: 03-Dec-2016 10:21:16   

Regarding serialization error: when does that occur exactly? As e.g. json serialization should be able to work with differently ordered name-value pairs?

Frans Bouma | Lead developer LLBLGen Pro
TopDog74
User
Posts: 40
Joined: 27-Apr-2012
# Posted on: 05-Dec-2016 10:25:52   

Thanks for the responses guys. My understanding was that if i changed the ordering of fields, or added new fields etc. then i would get a serialization exception on clients with an out of date version of the contract.

Im using WCF service with standard DataContractSerializer at the moment.

I'll run a few tests and to confirm my assumption was wrong.

Cheers, Iain

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39588
Joined: 17-Aug-2003
# Posted on: 05-Dec-2016 11:39:11   

To my knowledge the contract is about which fields are present, not the order (e.g. xml also has no ordering even though we think it does). Please report back into this thread if you find the contracts fail if the fields are ordered differently simple_smile

Frans Bouma | Lead developer LLBLGen Pro
TopDog74
User
Posts: 40
Joined: 27-Apr-2012
# Posted on: 05-Dec-2016 12:31:21   

Hi Guys,

So i ran a few tests and here is what i found: (Using WCF test method and the standard DataContractSerializer)

The order of the DataMember is crucial to the serilaization/deserialization of a call.

For e.g. Imagine a contract like this:


   [DataContract]
    public class TesterContract
    {
        [DataMember]
        public int Identifier { get; set; }

        [DataMember]
        public string Description { get; set; }

        [DataMember]
        public DateTime? UpdatedDate { get; set; }

        [DataMember]
        public List<string> MiscProperties { get; set; }

    }

If i have a client which expects this contract it works fine, if however i update the contract and add a property without specifying an order for e.g.:


  [DataMember]
  public string DescriptionNumber2 { get; set; }

The ordering of the members of the contract is changed, and it throws off the deserialization at the client (which expects the 1st version of the contract). It does not throw an exception, however, any fields with a different order do not have their value deserialized correctly: i.e. in this example: the field 'Description' will have its value, but all others will have a value of null (the int property 'Identifier' will be 0)

If i run a similar test using ordering of the Data Members, the problem is avoided, as long as i ensure that when adding a new property i give it an incremented order number and do not mess with the original ordering.

For example, in my original contract if i do this:


[DataContract]
    public class TesterContract
    {
        [DataMember(Order=1)]
        public int Identifier { get; set; }

        [DataMember(Order=2)]
        public string Description { get; set; }

        [DataMember(Order=3)]
        public DateTime? UpdatedDate { get; set; }

        [DataMember(Order=4)]
        public List<string> MiscProperties { get; set; }
    }

This is the contract my client will expect.

I then add a new property and ensure it is numbered correctly:


        [DataMember(Order=5)]
        public string DescriptionNumber2 { get; set; }

then the client, although not recognizing the existence of the new field, still deserializes correctly all the original fields.

It would be great if i could achieve this scenario with my DTOs being generated. i.e. Order them using some macro or have a project setting which just does it, and ensure that when new properties are added, the existing ordering is not altered.

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39588
Joined: 17-Aug-2003
# Posted on: 06-Dec-2016 13:46:06   

There's a problem though: there's no ordering defined in the derived element fields: they're sorted alphabetically because their properties aren't ordered as well. So there's no way to define a fieldindex macro as there's no field index on the field.

Adding a new field will simply make it be placed in an alphabetically sorted set of fields, so using the current 'index' into the set of fields isn't sufficient, it will change with a new field.

I'm surprised you get serialization issues with json, see: http://stackoverflow.com/questions/3948206/json-order-mixed-up as json frameworks don't rely on ordering of elements. Do you use another protocol perhaps, e.g. binary?

So if it would be as easy as adding a simple macro I'd have added it, but there's no value to replace it with: there's no ordering of fields defined, so adding the macro won't help: it won't resolve to a value. The alternative is you manually specify DataMember(order=1) etc. attributes per field of a DTO, on the DTO editor's Code generation info tab, which can be tedious.

Yet another alternative is to version your interfaces more strictly, i.e. adding fields means a new version of the interface. But I realize versioning of interfaces is a hot debate like tabs vs. spaces so there's no common ground there disappointed

Frans Bouma | Lead developer LLBLGen Pro
TopDog74
User
Posts: 40
Joined: 27-Apr-2012
# Posted on: 12-Dec-2016 13:01:26   

Would it be possible to provide a project setting to turn off the automatic alphabetical sorting of the DTO properties on the designer? i.e. just order them in the order in which they are added via designer.

That way i would be able to provide the order number a little more easily in the designer if necessary, Or maybe even with this setting selected, it would then be possible to generate an order number?

At the moment, when adding a new property, i need to inspect the properties one by one to find the last order number. If a derived model contract has gone through a few versions already then it can be quite awkward to identify what the order number should be for my new property (As it is not necessarily the last in the list of alphabetically ordered properties).

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39588
Joined: 17-Aug-2003
# Posted on: 13-Dec-2016 10:47:32   

TopDog74 wrote:

Would it be possible to provide a project setting to turn off the automatic alphabetical sorting of the DTO properties on the designer? i.e. just order them in the order in which they are added via designer.

That way i would be able to provide the order number a little more easily in the designer if necessary, Or maybe even with this setting selected, it would then be possible to generate an order number?

At the moment, when adding a new property, i need to inspect the properties one by one to find the last order number. If a derived model contract has gone through a few versions already then it can be quite awkward to identify what the order number should be for my new property (As it is not necessarily the last in the list of alphabetically ordered properties).

Let's first get back to the root reason you need this: serialization goes wrong. Nor serialization using binary, json or xml have problems with order, so I wonder how you run into this. The addition of order can be done of course but if it solves a non-problem it's a waste of time to add it (besides, it's not said it will be added soon, and you need it now).

Could you give a clear example where things go wrong? Do you use some sort of custom deserialization other than normal binary / json / xml ?

Frans Bouma | Lead developer LLBLGen Pro
TopDog74
User
Posts: 40
Joined: 27-Apr-2012
# Posted on: 13-Dec-2016 12:35:19   

ok maybe i did not explain this very clearly.

So i have a standard WCF Service (Not a RESTful WCF Service), and it is using the standard out the box serialization. i.e the DataContractSerializer. This seems to put all data members in alphabetical order by default. That is the cause of the problem.

There is no custom serialization happening. Everything is standard out the box functionality.

Maybe these links explains in a better way:

http://stackoverflow.com/questions/2519240/wcf-datacontract-some-fields-do-not-deserialize

http://stackoverflow.com/questions/3816478/wcf-datacontract-datamember-order

I want to ensure that when a contract is updated on my service side, and the client is not updated immediately, that the client continues to successfully deserialize the DataMambers which it does know about.

In my current setup this will not happen in cases where a change to the contract on the service side causes the ordering of the DataMembers to be changed.

Therefore it would be great if could get my Contract Data Members just automatically incrementing the order with each property addition.

I think it would be a really good addition to the Derived Model functionality and would prove useful for anyone sending these contracts over WCF.

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39588
Joined: 17-Aug-2003
# Posted on: 13-Dec-2016 16:02:57   

I see the point now. The xml spec doesn't state the order is required, but the DataContractSerializer does enforce it, so indeed when that's used, you're out of luck.

I found this thread: https://social.msdn.microsoft.com/Forums/vstudio/en-US/a891928b-d27a-4ef2-83b3-ee407c6b9187/order-of-data-members-in-the-xml-string-influences-deserialization-datacontractserializer?forum=wcf which talks about using the XmlSerializer, which gives the same results but doesn't require the ordering.

The thing is that I can't add the system you want today, and it will likely not arrive till next year some time, something you can't wait for. So to work around this, it might be best to switch to another serializer, like xmlserializer, or go for json which also doesn't have this problem. the thread linked above, last post describes (albeit a bit briefly) how to get the classes to behave themselves the same way with the xmlserializer. The xmlserializer is also not a nice thing to work with, but it might help you out today.

At the moment there's no 'ordering' defined in the fields, so there's nothing you can base your order on. Of course, there is an order in the container the fields are in, but removing a field and then re-adding it, will mess things up already, so that's not a good source for the ordering you're after. You might also revert to a designer plugin which enumerates the fields and adds a datamember attribute with an order property. Using designer events you can bind the plugin to an event which e.g. always runs if the project is generated, loaded or saved. See the sourcecode archive for a couple of plugin sources and the SDK (http://www.llblgen.com/Documentation/5.1/SDK/gui_implementingplugin.htm) to get started.

Frans Bouma | Lead developer LLBLGen Pro
TopDog74
User
Posts: 40
Joined: 27-Apr-2012
# Posted on: 13-Dec-2016 16:21:55   

ok , thanks i'll take a look at the plug ins and see.

Understand that you cant add the functionality right now.

TopDog74
User
Posts: 40
Joined: 27-Apr-2012
# Posted on: 14-Dec-2016 18:35:29   

Just in case anyone else needs something like this, I ended up writing a plug in like this (could be improved i imagine, but it does the job)


public class DataMemberOrderingPlugIn : PluginBase
    {
        private const string _build = "1.0.0";
        private const string _version = "1.0.0";

        public override PluginDescription Describe()
        {
            PluginDescription toReturn = base.Describe();

            toReturn.Build = _build;
            toReturn.Description = "General plug-in to add ordering to Derived Model Data Members";
            toReturn.Id = new Guid("{9FF8F454-69B9-4E35-B158-96E13C1F3F40}");
            toReturn.Name = "Add Data Member Ordering Plug-in";
            toReturn.ShowProgressViewerDuringExecution = true;
            toReturn.TargetType = PluginTargetType.DerivedModel | PluginTargetType.RootDocument;
            toReturn.TypeOfPlugin = PluginType.DirectRun;
            toReturn.Vendor = "Top Dog 74";
            toReturn.Version = _version;
        
            return toReturn;
        }

        public override void Execute()
        {
            base.ProgressTaskStart("Processing Derived Models");
            base.ProgressSubtaskInit(this.RootDocuments.Count);
            
            //get all the derived models
            foreach (var derivedModel in this.RootDocuments)
            {
                base.ProgressSubtaskStart("Processing Class: " + derivedModel.Name);
                ProcessEntityRecursively(derivedModel);
                base.LogLineToApplicationOutput("Added Data Member Ordering to class: " + derivedModel.Name, "Add Data Member Ordering PlugIn", true);
                base.ProgressSubtaskComplete();

            }

            base.ProgressTaskComplete();
        }

        private void ProcessEntityRecursively(DocumentDefinitionBase derivedModel)
        {

            int currentOrderNumber = 0;

            //create a list of members which have the DataMember[order=*] set
            var orderedFields = new List<DocumentFieldDefinition>();
            //create a list of members which still needthe DataMember[order=*] set
            var unOrderedFields = new List<DocumentFieldDefinition>();

            //check which fields already have a DataMemer(Order=*)] attribute
            foreach (var dataMember in derivedModel.Fields)
            {
                var attributes =
                    dataMember.OutputSettingValues.AttributesWithRules.Where(x => x.Value1.Contains("DataMember(Order"))
                        .ToList();

                if (attributes.Any())
                {
                    //already ordered, add it to the order List
                    orderedFields.Add(dataMember);
                }
                else
                {
                    //not ordered, add it to the unorderedList
                    unOrderedFields.Add(dataMember);
                }
            }

            currentOrderNumber = orderedFields.Count + 1;

            //set the DataMemer(Order=*)] attribute for each member which does not already have it set
            foreach (var dataMember in derivedModel.Fields)
            {
                if (unOrderedFields.Contains(dataMember))
                {
                    var settingRule = new SettingRule();
                    var attributeText = "DataMember(Order=" + currentOrderNumber.ToString() + ")";
                    var pair = new Pair<string, SettingRule>(attributeText, settingRule);
                    var list = new List<Pair<string, SettingRule>>();
                    list.Add(pair);
                    dataMember.OutputSettingValues.SetAttributesWithRules(list);
                    currentOrderNumber += 1;

                    //run the process again for any member which are complex types
                    if (dataMember.TargetElementType == FrameworkSettingTargetElementType.DerivedElementFieldSetOfElements
                        || dataMember.TargetElementType == FrameworkSettingTargetElementType.DerivedElementFieldSingleElement)
                    {
                        DocumentDefinitionBase complexType = (DocumentDefinitionBase)dataMember.Source;
                        ProcessEntityRecursively(complexType);
                    }


                }
            }

        }

    }

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39588
Joined: 17-Aug-2003
# Posted on: 15-Dec-2016 10:28:59   

Thanks for sharing and glad it got your problem solved! simple_smile

Frans Bouma | Lead developer LLBLGen Pro