Adapter, version 2.5

Something strange is happening when I serialize/deserialize a PrefetchPath. In the prefetchpath, I have a FilterRelation that has a CustomFilter. If I fetch using this prefetchpath, I get the data filtered the way I want. When I serialize/deserialize the prefetchpath, and fetch using this, the data comes back unfiltered.

I have no idea why this is happening. Here is the code I use to construct the PrefetchPath:

 'Filter on list of given benefits
            Dim prefetchPath As New PrefetchPath2(CInt(EntityType.EmployeeEntity))
            'Payment
            Dim employeeBenefitPaymentPath As PrefetchPathElement2 = prefetchPath.Add(EmployeeEntity.PrefetchPathEmployeeBenefitPayment)
            employeeBenefitPaymentPath.SubPath.Add(EmployeeBenefitPaymentEntity.PrefetchPathBenefit)
            employeeBenefitPaymentPath.SubPath.Add(EmployeeBenefitPaymentEntity.PrefetchPathBenefitOption)
            employeeBenefitPaymentPath.SubPath.Add(EmployeeBenefitPaymentEntity.PrefetchPathPosting)

            'Loan Payment
            Dim employeeLoanPath As PrefetchPathElement2 = prefetchPath.Add(EmployeeEntity.PrefetchPathEmployeeLoan)
            Dim employeeLoanPaymentPath As PrefetchPathElement2 = employeeLoanPath.SubPath.Add(EmployeeLoanEntity.PrefetchPathEmployeeLoanPayment)
            employeeLoanPath.SubPath.Add(EmployeeLoanEntity.PrefetchPathFirstPlannedPaymentPayPeriod)
            employeeLoanPath.SubPath.Add(EmployeeLoanEntity.PrefetchPathLastPlannedPaymentPayPeriod)

            'EmployeeEmploymentActions
            Dim employeeEmploymentActionPath As PrefetchPathElement2 = prefetchPath.Add(EmployeeEntity.PrefetchPathEmployeeEmploymentAction)
            employeeEmploymentActionPath.Filter.Add(EmployeeEmploymentActionFields.IsActive <> System.DBNull.Value)

            'Filter the payments by the postingType, postingBenefitName and paymentDate
            Dim postingRelation As EntityRelation = EmployeeBenefitPaymentEntity.Relations.PostingEntityUsingPosting_Id
            postingRelation.CustomFilter = New PredicateExpression(PostingFields.PostingType = postingType)
            postingRelation.CustomFilter.AddWithAnd(PostingFields.BenefitName = postingBenefitName)
            postingRelation.CustomFilter.AddWithAnd(PostingFields.PeriodStartDate = paymentDate)
            employeeBenefitPaymentPath.FilterRelations.Add(postingRelation)

            'Filter for Loan Payments
            Dim postingLoanRelation As EntityRelation = EmployeeLoanPaymentEntity.Relations.PostingEntityUsingPosting_Id
            postingLoanRelation.CustomFilter = New PredicateExpression(PostingFields.PostingType = postingType)
            postingLoanRelation.CustomFilter.AddWithAnd(PostingFields.PeriodStartDate = paymentDate)
            employeeLoanPaymentPath.FilterRelations.Add(postingLoanRelation)

            Dim stream As New System.IO.MemoryStream
            Dim formatter As New System.Runtime.Serialization.Formatters.Binary.BinaryFormatter
            formatter.Serialize(stream, prefetchPath)
            stream.Position = 0
            Dim pp As PrefetchPath2 = formatter.Deserialize(stream)

            Dim employeeCollection As ISecureEntityCollection = GetEntityCollection(prefetchPath, Nothing)
            'Dim employeeCollection As ISecureEntityCollection = GetEntityCollection(pp, Nothing)

When I fetch using the "prefetchPath" instance, I get the correct data. When I fetch using the "pp" instance (which is commented out) I get unfiltered data...

prefetchpath(0).FilterRelations(0).CustomFilter.Count

returns "5"

pp(0).FilterRelations(0).CustomFilter.Count

returns "1"

So, it appears some of the elements of the customfilter are being wiped out in the serialization process.

Sounds fishy, hang on we will be checking it out.

(EDIT)

Dim postingRelation As EntityRelation = EmployeeBenefitPaymentEntity.Relations.PostingEntityUsingPosting_Id postingRelation.CustomFilter = New PredicateExpression(PostingFields.PostingType = postingType) postingRelation.CustomFilter.AddWithAnd(PostingFields.BenefitName = postingBenefitName) postingRelation.CustomFilter.AddWithAnd(PostingFields.PeriodStartDate = paymentDate) employeeBenefitPaymentPath.FilterRelations.Add(postingRelation)

Code: prefetchpath(0).FilterRelations(0).CustomFilter.Count

returns "5"

Code: pp(0).FilterRelations(0).CustomFilter.Count

returns "1"

Shouldn't it return 3? Or am I missing something here.

Walaa: the 'And' operators are also an object inside the collection

The issue is very strange. The objects in question are all serialized by .NET, so there's no ISerializable implementation on them, and all members are not excluded. What I find particularly odd is that it still contains a value after deserialization...

(edit) I can reproduce it with this:

IEntityRelation relation = CustomerEntity.Relations.OrderEntityUsingCustomerId;
relation.CustomFilter = new PredicateExpression(OrderFields.EmployeeId == 1);
relation.CustomFilter.AddWithAnd(OrderFields.ShipCountry == "Germany");

MemoryStream stream = new MemoryStream();
BinaryFormatter formatter = new BinaryFormatter();
formatter.Serialize(stream, relation);
stream.Seek(0, SeekOrigin.Begin);
IEntityRelation relationD = (IEntityRelation)formatter.Deserialize(stream);
stream.Close();
Assert.AreEqual(relation.CustomFilter.Count, relationD.CustomFilter.Count);

There is NO ISerializable code in there, all data is serializable and the members aren't marked with [Notserialized]. When I serialize just a PredicateExpression it works. If I put it in the relation, just 1 element is deserialized. (the first predicate). The rest isn't deserialized as it seems.

However, this works! ->

            formatter.Serialize(stream, relation.CustomFilter);
            stream.Seek(0, SeekOrigin.Begin);
            PredicateExpression filterD = (PredicateExpression)formatter.Deserialize(stream);
            Assert.AreEqual(relation.CustomFilter.Count, filterD.Count);

so serializing the relation instead of the customfilter, then the custom filter isn't properly deserialized. However if I serialize the filter on its own, it is. EntityRelation doesn't implement ISerializable and has _customFilter defined as a normal member, not marked with NotSerialized.

So I thought, what's going on in EntityRelation, so I checked CustomFilter. It's not a normal property. It uses a cached build custom filter. THe cached version has type filters added automatically so the filters match the correct inheritance types.

So when I do this instead:

IEntityRelation relation = CustomerEntity.Relations.OrderEntityUsingCustomerId;

IPredicateExpression filter = new PredicateExpression(OrderFields.EmployeeId == 1);
filter.AddWithAnd(OrderFields.ShipCountry == "Germany");
relation.CustomFilter = filter;

MemoryStream stream = new MemoryStream();
BinaryFormatter formatter = new BinaryFormatter();
formatter.Serialize(stream, relation);
stream.Seek(0, SeekOrigin.Begin);
IEntityRelation relationD = (IEntityRelation)formatter.Deserialize(stream);
stream.Close();
Assert.AreEqual(relation.CustomFilter.Count, relationD.CustomFilter.Count);

It works. The thing is: the CustomFilter property now doesn't return a cached predicate expression. Your code will add the second etc. predicates to the cached filter. The cached filter isn't serialized.

(edit)CustomFilter is used by the framework to inject type filters so filtering using relations from derived entities will automatically also only focus on these derived entities. The thing is that two DQEs use this in their non-ansi join routine. So writing a different method to obtain that filter is difficult as it will break: the DQE needs a given version of the runtime lib to work, which is unacceptable.

I can solve it with internal code though, so I'll fix it that way. You in the meantime should use build collection then set property-style code for this. Hmm, it turns out to be quite difficult to fix this properly without adding code all over the place. I'd like to suggest to you to use the workaround I stated above, unless you have to alter a lot of code to make it work. It's not that common that people read the CustomFilter property of an EntityRelation.

We use the FilterRelation/CustomFilter in a few places... I suppose we can fix up the places we do use it though. Is it your plan to eventually fix this?

Is there much difference in using the CustomFilter of a relation than from just using the Filter property of the PrefetchPathElement?

mikeg22 wrote:

Hmmm...we use the FilterRelation/CustomFilter in a few places... I suppose we can fix up the places we do use it though. Is it your plan to eventually fix this?

I spend 3 hours on this this morning, using a couple of different fixes, but it always broke on some place or another.

the thing is this: An entity relation could be: FamilyCarEntity - BoardMember, where FamilyCar is a subtype of CompanyCar which has the relation with BoardMember. In v2.0, it would simply add the relation of companycar and no filter on familycar, so matches with other types of companycar were also in the query. In v2.5, an entity relation adds a custom filter on the subtype in either pk side or fk side or both. So in v2.5, familycar-boardmember will only match familycar entities, you don't have to add any additional filter yourself. This is done inside the entityrelation.CustomFilter: when you READ the filter, it will append the type filters for you.

Now, this sounds ok to some regard, but there's a catch: these type filters don't have persistence info, so they need to get that persistence info set inside of them before they're used in a DQE. In Adapter, the persistence info set routine calls CustomFilter on the relation, and sets its persistence info if something is returned.

The problem is that your code calls the getter on CustomFilter a couple of times. so each time these type filters would be added. To avoid this, it uses internally a cached version of the customfilter. The thing is though: you set the CustomFilter to filter F and your getter gets the cached one returned, the second filter added is to that cached version. Because that one isn't serialized, it's never send over the wire.

So to set the persistenceinfo in these additional filters, the filter with these additional filters has to be read, though using the property for that will result in the issues you ran into as well...

Normally not a problem, though if you use serialization AND a predicate filter with multiple predicates it is a problem.

So the fixes I had were mostly calculating the set later on, when it was needed, but this caused crashes in situations where the additional filters were added, as these didn't have persistence info. Also cleverly detecting if the set was already created wasn't sufficient. Always re-creating the set or always adding the extra filters is 'a' somewhat lame solution but that leads to duplicate filters which isn't that great.

So a complex 'has there been something added to the cached filter and if so, copy it over to the real filter' routine might fix it, though that too gives headaches with it, as ISerializable isn't implemented on the class. This means that it doesn't know when serialization occurs and as additions are done to the predicateexpression, it's not detectable if anything was added...

Is there much difference in using the CustomFilter of a relation than from just using the Filter property of the PrefetchPathElement?

No difference unless you use left/right joins with a CustomFilter. A custom filter is appended to the ON clause. If you have a normal filter, it doesn't influence the resultset unless left/right joins are used.

Otis wrote:

The problem is that your code calls the getter on CustomFilter a couple of times. so each time these type filters would be added. To avoid this, it uses internally a cached version of the customfilter. The thing is though: you set the CustomFilter to filter F and your getter gets the cached one returned, the second filter added is to that cached version. Because that one isn't serialized, it's never send over the wire.

Thats what I don't get. Why isn't _cachedBuildCustomFilter serialized? Shouldn't .NET serialization take care of this?

Edit: I missed the [NonSerialized] attribute! Why is that there?!

mikeg22 wrote:

Otis wrote:

The problem is that your code calls the getter on CustomFilter a couple of times. so each time these type filters would be added. To avoid this, it uses internally a cached version of the customfilter. The thing is though: you set the CustomFilter to filter F and your getter gets the cached one returned, the second filter added is to that cached version. Because that one isn't serialized, it's never send over the wire.

Thats what I don't get. Why isn't _cachedBuildCustomFilter serialized? Shouldn't .NET serialization take care of this?

Edit: I missed the [NonSerialized] attribute! Why is that there?!

As that would lead to multiple times the filters being added as well. The main problem is: there are two predicateexpressions used for the same thing, it has to be one to make it work. However using 1 (which is thus simply _customFilter) is leading to multiple times the filters being added to the filter.

One could say: "Add a flag for that!", which was one of my considerations as well. But then you get the following tearjerker: as ISerializable isn't implemented, you have to have the same BUILD on client and server. Otherwise deserialization will break with an exception.

We're currently discussing to change the API abit so the filters are added to the same predicate expression and an extra, serialized, flag is added to signal if the filters are already added.