Use Custom Relation for Prefetch Path
I need to create a prefetch path element that uses a custom relation.
The primary key field used in the relationship differs from the primary key field in the entity.
Why? Long answer. 
The problem is that Persistence.MergeNormal() method does not build the primary key hash from the relation, rather from the entity itself. This results in my prefetch path being queried, but not merged into the destination entity.
Can you add a flag in the prefetch element to build pkHashes from the relation instead of the entity?
Thanks.
Joined: 03-Nov-2007
rdhatch wrote:
Hi Walaa -
Manually Merge?
But then we the lose ability to use the Context. It's a simple feature request.
JR & I have a Database that we've added a GUID field to on some important tables. Actual PrimaryKeys in the database are horrid. All Composite PrimaryKeys. The GUIDs we've created are now being used as a unique identifier for each Entity.
MergeNormal is inconsistent. It uses the PrimaryKeys from the EntityFields. Yet, it uses the ForeignKeys from the Relationship.
It's not inconsistent. Relations don't contain values, they're used to determine which fields to calculate the hashcodes for. PK's are easy: the entity knows these. The fk's are known by the relation so the relation is consulted. That's not inconsistent to me. You just want to use different pk's for the hashes, that's something else. You also don't mention how the guids affected FK's.
IMHO, keeping such a model around is not the answer, but of course it depends on your freedom what you may change and what not, so it's not always that easy, I understand 
We are simply requesting a flag on the prefetchElement that would use the Relationship for both PrimaryKeys & ForeignKeys.
Why not use the Relationship that's being passed? Ryan
See above.
I'm not going to add some flag for this, as I think it's a bit hacky to do so, even though I understand your reasoning why you want this.
Thanks for the reply, Frans.
I'm not going to add some flag for this, as I think it's a bit hacky to do so, even though I understand your reasoning why you want this.
Our framework allows for utility tables (_ie auditing_) that will service any entity type - using a unique identifier value on each table. This Guid should not need to be the primary key for all tables.
We want to prefetch entities that do not have a formal database relationship. This is because we don't want to create 100+ linking tables, and manage the entities.
We've got at least 5 utility tables, and to create linking tables for 20 tables forces us to create 100 tables to perform the linking. 
If you still opposed to the flag option. What do you recommend we do? We want this to work for adapter & self servicing, as our whole framework does. And as we said before, we would rather not put the requirement on of framework to have all entities have a guid pk.
Thanks Frans,
We'll get to work 
Ryan and I worked on this, and got success using a custom prefetch path, an interface and a delegate. As well as modifying PersistanceCore a little
delegate:
public delegate void SetRelatedEntityProperty(IEntityCore destEntity, IEntityRelation relation, string propertyName, IEntityCore entityToAdd);
interface:
public interface ICustomPrefetchPathElementCore
{
SetRelatedEntityProperty SetRelatedEntityPropertyDelegate { get; set; }
}
custom prefetch path element:
[Serializable]
public class CustomPrefetchPathElement2 : PrefetchPathElement2, ICustomPrefetchPathElementCore
{
public CustomPrefetchPathElement2(IEntityCollection2 retrievalCollection, IEntityRelation relation,
int destinationEntityType, int toFetchEntityType, int maxAmountOfItemsToReturn, ISortExpression sorter, IPredicateExpression filter,
IRelationCollection filterRelations, IEntityFactory2 entityFactoryToUse, string propertyName, RelationType typeOfRelation)
: base(retrievalCollection, relation, destinationEntityType, toFetchEntityType, maxAmountOfItemsToReturn,
sorter, filter, filterRelations, entityFactoryToUse, propertyName, typeOfRelation)
{ }
public SetRelatedEntityProperty SetRelatedEntityPropertyDelegate
{
get;
set;
}
}
Then in PersistanceCore.MergeNormal() we do this:
// GENIUS CODE CHANGE ************** Create entity hashes from relation
if (typeof(ICustomPrefetchPathElementCore).IsAssignableFrom(currentElement.GetType()))
PersistenceCore.CreateEntitiyHashesUsingRelation(pkSideHashes, pkSideCollection,currentElement);
else // END GENIUS CODE CHANGE
to make sure the hash comes from our current element, and this:
if(rootEntitiesArePkSide)
{
if(PersistenceCore.DetermineIfMergeShouldTakePlace(maxCountersPerRootEntity, pkObject, currentElement.MaxAmountOfItemsToReturn))
{
// GENIUS CODE UPDATE
if (typeof(ICustomPrefetchPathElementCore).IsAssignableFrom(currentElement.GetType()))
{
((ICustomPrefetchPathElementCore)currentElement).SetRelatedEntityPropertyDelegate.Invoke(
pkObject,currentElement.Relation,
currentElement.PropertyName,
fkSideCollection[j]);
}
// END GENIUS CODE UPDATE
pkObject.SetRelatedEntityProperty(currentElement.PropertyName, fkSideCollection[j]);
}
}
else
{
if (typeof(ICustomPrefetchPathElementCore).IsAssignableFrom(currentElement.GetType()))
{
// GENIUS CODE UPDATE
if (typeof(ICustomPrefetchPathElementCore).IsAssignableFrom(currentElement.GetType()))
{
((ICustomPrefetchPathElementCore)currentElement).SetRelatedEntityPropertyDelegate.Invoke(
pkObject,currentElement.Relation,
currentElement.PropertyName,
pkObject);
}
// END GENIUS CODE UPDATE
}
fkSideCollection[j].SetRelatedEntityProperty(currentElement.PropertyName, pkObject);
}
To allow our custom delegate perform the SetEnttiyProperty stuff.
And it works. We've only tested it on the primary key side (ie - fetching a collection of related entities), and not the other way around.
So for someone else to use this, they just create a custom prefetch path element, specify their relation, and pass a delegate that will perform the SetEntityProperty stuff. And the default behavior should not be changed.
And we tried to change the source as little as possible.
Did you want me to email the modified source?
JRR wrote:
Ryan and I worked on this, and got success using a custom prefetch path, an interface and a delegate. As well as modifying PersistanceCore a little
delegate:
public delegate void SetRelatedEntityProperty(IEntityCore destEntity, IEntityRelation relation, string propertyName, IEntityCore entityToAdd);interface:
public interface ICustomPrefetchPathElementCore { SetRelatedEntityProperty SetRelatedEntityPropertyDelegate { get; set; } }
No offence but I don't like the new interface requirement. See below
custom prefetch path element:
[Serializable] public class CustomPrefetchPathElement2 : PrefetchPathElement2, ICustomPrefetchPathElementCore { public CustomPrefetchPathElement2(IEntityCollection2 retrievalCollection, IEntityRelation relation, int destinationEntityType, int toFetchEntityType, int maxAmountOfItemsToReturn, ISortExpression sorter, IPredicateExpression filter, IRelationCollection filterRelations, IEntityFactory2 entityFactoryToUse, string propertyName, RelationType typeOfRelation) : base(retrievalCollection, relation, destinationEntityType, toFetchEntityType, maxAmountOfItemsToReturn, sorter, filter, filterRelations, entityFactoryToUse, propertyName, typeOfRelation) { } public SetRelatedEntityProperty SetRelatedEntityPropertyDelegate { get; set; } }Then in PersistanceCore.MergeNormal() we do this:
// GENIUS CODE CHANGE ************** Create entity hashes from relation if (typeof(ICustomPrefetchPathElementCore).IsAssignableFrom(currentElement.GetType())) PersistenceCore.CreateEntitiyHashesUsingRelation(pkSideHashes, pkSideCollection,currentElement); else // END GENIUS CODE CHANGEto make sure the hash comes from our current element,
this method isn't given in your post.
and this:
if(rootEntitiesArePkSide) { if(PersistenceCore.DetermineIfMergeShouldTakePlace(maxCountersPerRootEntity, pkObject, currentElement.MaxAmountOfItemsToReturn)) { // GENIUS CODE UPDATE if (typeof(ICustomPrefetchPathElementCore).IsAssignableFrom(currentElement.GetType())) { ((ICustomPrefetchPathElementCore)currentElement).SetRelatedEntityPropertyDelegate.Invoke( pkObject,currentElement.Relation, currentElement.PropertyName, fkSideCollection[j]); } // END GENIUS CODE UPDATE pkObject.SetRelatedEntityProperty(currentElement.PropertyName, fkSideCollection[j]); } } else { if (typeof(ICustomPrefetchPathElementCore).IsAssignableFrom(currentElement.GetType())) { // GENIUS CODE UPDATE if (typeof(ICustomPrefetchPathElementCore).IsAssignableFrom(currentElement.GetType())) { ((ICustomPrefetchPathElementCore)currentElement).SetRelatedEntityPropertyDelegate.Invoke( pkObject,currentElement.Relation, currentElement.PropertyName, pkObject); } // END GENIUS CODE UPDATE } fkSideCollection[j].SetRelatedEntityProperty(currentElement.PropertyName, pkObject); } To allow our custom delegate perform the SetEnttiyProperty stuff.
This code merges the entity twice? (your own delegate call and the normal call, there's no 'else').
I don't like the delegate because it's really slow but indeed a way to solve the lack of support for merging in the generated code. That's a real extensibility flaw in the design: the generated code should in the 'default' branch call into a virtual method which you should be able to override in a partial class to add your own merge code. I'll make sure the next version has that.
So, it's a bit of a problematic situation: the best way to solve this is through that system: a call to the virtual method which does additional merging. Adding the delegate system will give yet another way to do this, which is IMHO a bit of a pain. Adding the template + virtual method route can't be done as it relies on templates + runtime are versioned at the same time, which some people don't do.
Thats also why I don't like the interface, it's useless once the next version comes around, because it has no value: it has to be merged with IPrefetchPathElementCore. However, removing it next time around, will break code as well...
There's another problem: the delegate can't be serialized. Adding the delegate would cause problems in systems where prefetch paths are serialized across the wire, as in: they won't give the feature the delegate should provide.
I'll give you an alternative, using solely generated code below. - To a partial class of CustomEntityBase, you add:
protected virtual void SetCustomRelatedEntityProperty(string propertyName, IEntity2 entity)
{
// nop
}
- Then, you create a copy of entityIncludeAdapter.template and bind it using a custom templatebindings file. You bind it in that custom templatebindings file to the templateID: SD_EntityIncludeAdapterTemplate and in the generated code dialog at tab 2, you place that templatebindings above the shared template bindings so your template binding overrules the original, for just that templateID.
- in your copy of entityIncludeAdapter.template (we have very little changes in templates due to bugfixes so you're actually pretty safe with this), at line (C#) 47, you replace
default:<[If IsSubType]>
base.SetRelatedEntityProperty(propertyName, entity);<[EndIf]>
with
default:<[If IsSubType]>
base.SetRelatedEntityProperty(propertyName, entity);<[Else]> this.SetCustomRelatedEntityProperty(propertyName, entity);<[EndIf]>
- in a partial class of each entity you want custom merging, you probably already have this method (as it's the method you'd call with the delegate!, so in that case, change the signature so it overrides the CommonEntityBase method), in case you don't you've to add it there,
protected override void SetCustomRelatedEntityProperty(string propertyName, IEntity2 entity)
{
switch(propertyName)
{
case "CustomField":
...
break;
}
}
where you add the merging you want to happen. So the runtime stays untouched, you simply reroute the 'default' clause (which will be used for names not known in the generated code, i.e. your custom relations) in the generated SetRelatedEntityProperty to use the custom merge method you write yourself. Add a new custom property/relation? simply add a case to the SetCustomRelatedEntityProperty override of that entity. This is not using a delegate and IMHO a better solution (and also much faster).
Would that work for you?
It's the route I really would prefer to follow (and will follow for v3)
About the PK hashes, are these then still necessary or are these not necessary anymore? The point is that your original problem that things weren't merged together is caused by the fact that SetRelatedEntityProperty didn't know about your custom relationship, but with the system above, it would work. You need the relationship based PK hash retrieval if you have a different set of PK fields defined in the relationship than you have in the entity. I didn't fully grasp that from your original posts, is that the case?
Thanks for the help, Frans. This solution is much better than the first one.
Note: _In reading your post again, I am not sure this is exactly the approach you offered 
(editing EntityBase), but here it is:_:)
Attached is a diff file for the templates, modified as you specified.
default:
<[If IsSubType]> base.SetRelatedEntityProperty(propertyName, entity);<[Else]>
this.SetCustomRelatedEntityProperty(propertyName, entity);<[EndIf]>
Here are the changes to the runtime:
EnitityBase.cs / EntityBase2.cs -Added virtual method to class
/// <summary>
/// Allow Entity to perform custom related entity procedure
/// </summary>
/// <param name="propertyName">Instance to set as the related entity of type entityType</param>
/// <param name="entity">Entity to set as an related entity</param>
/// <remarks>Will be called by Generated Entity if property name does not match generated property names.</remarks>
public virtual void SetCustomRelatedEntityProperty(string propertyName, IEntity2 entity)
{
}
PrefetchPathElement.cs / PrefetchPathElement2.cs -added flag
/// <summary>
/// Use keys in relation when merging contents with destination entity
/// </summary>
public bool UseKeysFromRelationForMerge
{ get; set; }
PersistenceCore.cs -modified MergeNormal()
// check if we should get hash from relation
if (currentElement.UseKeysFromRelationForMerge)
PersistenceCore.CreateEntitiyHashesUsingRelation(pkSideHashes, pkSideCollection,currentElement);
else // otherwise, use default hashing
{
if (((IEntityCollectionCoreInternal)pkSideCollection).GetCachedPkHashes() == null)
{
// calculate the hashes
((IEntityCollectionCoreInternal)pkSideCollection).SetCachedPkHashes(pkSideHashes);
PersistenceCore.CreateEntityHashes(pkSideHashes, pkSideCollection);
}
else
{
pkSideHashes = ((IEntityCollectionCoreInternal)pkSideCollection).GetCachedPkHashes();
}
}
-Added method CreateEntitiyHashesUsingRelation(). _This is a replica of the existing CreateEntitiyHashes() method, with just a minor change as to deciding what to hash. _
/// <summary>
/// Creates hashes for the collection using the fields specified in the prefetch path
/// </summary>
/// <param name="hashesToFill">Hashes to fill</param>
/// <param name="collectionToHash">Collection to hash</param>
/// <param name="currentElement">relation to use for choosing primary keys</param>
/// <remarks>construct hashtable for looking up entities through keys specified in relationship</remarks>
internal static void CreateEntitiyHashesUsingRelation(Dictionary<int, List<IEntityCore>> hashesToFill, IEntityCollectionCore collectionToHash, IPrefetchPathElementCore currentElement)
{
for (int i = 0; i < collectionToHash.Count; i++)
{
int hashValue = -1;
//**BLOCK UPDATED**
foreach (IEntityFieldCore field in currentElement.Relation.GetAllPKEntityFieldCoreObjects())
{
if (hashValue == -1)
hashValue = collectionToHash[i].GetFieldByName(field.Name).GetHashCode();
else
hashValue ^= collectionToHash[i].GetFieldByName(field.Name).GetHashCode();
}
List<IEntityCore> entitiesPerHash = null;
if (!hashesToFill.ContainsKey(hashValue))
{
// not yet there, add it
entitiesPerHash = new List<IEntityCore>();
hashesToFill.Add(hashValue, entitiesPerHash);
}
else
{
entitiesPerHash = hashesToFill[hashValue];
}
// add entity to arraylist for this hashvalue
entitiesPerHash.Add(collectionToHash[i]);
}
}
The the generated entity looks like this:
[EditorBrowsable(EditorBrowsableState.Never)]
public override void SetRelatedEntityProperty(string propertyName, IEntity2 entity)
{
switch(propertyName)
{
case "Countries":
this.Countries.Add((CountriesEntity)entity);
break;
default:
this.SetCustomRelatedEntityProperty(propertyName, entity);
break;
}
}
And commonEntityBase looks like this:
// __LLBLGENPRO_USER_CODE_REGION_START CustomEntityCode
public override void SetCustomRelatedEntityProperty(string propertyName, IEntity2 entity)
{
switch (propertyName)
{
case "Attributes":
((IEntityCollection2)enhancements.Attributes).Add(entity);
break;
}
}
And it works
| Filename | File size | Added on | Approval |
|---|---|---|---|
| templates2.patch | 2,781 | 01-Sep-2009 19:12.37 | Approved |
You shouldn't edit any class in ormsupportclasses. You can just add the virtual method to CommonEntityBase's partial class (so in the generated code) and you add the overrides of that method to the partial classes of the entities which need custom merging. You then alter the template (a copy) entityIncludeAdapter.template, so no entityBase* editing: you move all methods upwards 1 inheritance level
I'll look into the hash calculation. If you could describe in detail how your PK-FK fields look like in the entity vs. in the DB, it would help also (so why you need the pk fields of the relation instead of the pk fields in the entity on the pk side 
)
Thanks again for your assistance!
You shouldn't edit any class in ormsupportclasses. You can just add the virtual method to CommonEntityBase's partial class (so in the generated code) and you add the overrides of that method to the partial classes of the entities which need custom merging.
Can I make a request? Your idea sounds good, but there is still one problem. EntityBase's method:
public abstract void SetRelatedEntityProperty(string propertyName, IEntity2 entity);
could be virtual, and that would help us out tremendously.
Why?
I've got 30 entities, and they all share the same utility tables. I want to handle SetRelatedEntityProperty() in the CommonEntityBase class. I don't want to override this method in each and every entity in my catalog. 
So we've got another idea - if the SetRelatedEntityProperty method can be switched to virtual on EntityBase, it allows our generated entity to have this :
public override void SetRelatedEntityProperty(string propertyName, IEntity2 entity)
{
switch(propertyName)
{
case "Countries":
this.Countries.Add((CountriesEntity)entity);
break;
default:
base.SetRelatedEntityProperty(propertyName, entity);
break;
}
}
Then in our CommonEntityBase, we can put this code in a custom code region:
public override void SetRelatedEntityProperty(string propertyName, IEntity2 entity)
{
switch (propertyName)
{
case "Attributes":
//Add entity to custom entity collection here
break;
}
}
But the real benefit here is that the templates compile out-of-the-box. And, you wouldn't have to add any new methods to your EntityBase / EntityBase2 classes.
As for the keys, here is a screenshot of some tables:
The Fruit table and the gc_EntityAttribute table both have an integer pk. However, we're using neither field for the relationship. This is because the gc_EnttiyAttribute table stores attributes for any entity type. To build the relationship, then, we are using Guid values.
In our system, each (actual, not type) entity has its own guid, and the gc_EntityAttribute table can store multiple attributes.
So then, the new method CreateEntitiyHashesUsingRelation() is important because then we can generate this sql output for our prefetch path using fields that are not keys:
Query: SELECT [WebAppMoniter].[dbo].[gc_EntityAttribute].[gcEntityGuid] AS [GcEntityGuid], [WebAppMoniter].[dbo].[gc_EntityAttribute].[AttributeName], [WebAppMoniter].[dbo].[gc_EntityAttribute].[AttributeValue], [WebAppMoniter].[dbo].[gc_EntityAttribute].[AttributeId] FROM [WebAppMoniter].[dbo].[gc_EntityAttribute] WHERE ( [WebAppMoniter].[dbo].[gc_EntityAttribute].[gcEntityGuid] IN (@GcEntityGuid1, @GcEntityGuid2, @GcEntityGuid3, @GcEntityGuid4, @GcEntityGuid5))
Parameter: @GcEntityGuid1 : Guid. Length: 0. Precision: 0. Scale: 0. Direction: Input. Value: 7bb01995-d71c-4170-9cd8-8a0271a3c9de.
Parameter: @GcEntityGuid2 : Guid. Length: 0. Precision: 0. Scale: 0. Direction: Input. Value: c45f355b-01d0-4c83-b097-6c4b9ac3104b.
Parameter: @GcEntityGuid3 : Guid. Length: 0. Precision: 0. Scale: 0. Direction: Input. Value: 60155371-b69c-4d56-88b0-0023ad603e57.
Parameter: @GcEntityGuid4 : Guid. Length: 0. Precision: 0. Scale: 0. Direction: Input. Value: b8f1b0ac-c917-444c-ab96-102d9fd4fa7c.
Parameter: @GcEntityGuid5 : Guid. Length: 0. Precision: 0. Scale: 0. Direction: Input. Value: 548fbd37-470b-4645-83bf-dbfa508bcaad.
This requires a new method (shown in previous post) that can calculate hashes on the fields specified by the relation. Otherwise, the code will hash the primary key fro the Fruit entity (FruidId), and no hashes in the prefetch will match.
So in the end, this approach does require a few in the ORMSupportClasses project:
- PrefetchPathElement / PrefetchPathElement2 - added flag: UseKeysFromRelationForMerge
- PersistenceCore - allow MergeNormal to handle PK hashes using Relation
- EntityBase / EntityBase 2 - set SetRelatedEntityProperty() to virtual
Does this seem plausible to you? Or do you see other options still that we haven't explored?
Joined: 03-Nov-2007
Thanks JRR for the details I'll reply to Ryan's post so I don't have to reply to you both
rdhatch wrote:
Our primary goal is to allow any LLBLGen database to instantly add functionality for Auditing, special Attributes, along with any other utility tables. Without facing a maintenance nightmare. We believe this would not only be a benefit to us, but to many new & existing LLBLGen users as well.
Auditing as in: other than what's built into llblgen now?
We need 5 utility tables on 20+ entities, without resulting in 100+ linking tables. Thus, our request to you for a prefetch on a truly custom relation. So, in summary - we are requesting 2 (and only 2) simple things:
- 1.) Allow PKHashes to be built using Relation
- 2.) Allow EntityBase.SetRelatedEntityProperty as virtual
Point 1 can be investigated, I'll make the change and run the tests. Shouldn't be a difference as the pk field in a relation are the pk ones in an entity for generated code.
Point 2 is a strange one, as ... EntityBase.SetRelatedEntityProperty is abstract:
/// <summary>
/// Sets the related entity property to the entity specified. If the property is a collection, it will add the entity specified to that collection.
/// </summary>
/// <param name="propertyName">Name of the property.</param>
/// <param name="entity">Entity to set as an related entity</param>
/// <remarks>Used by prefetch path logic.</remarks>
public abstract void SetRelatedEntityProperty(string propertyName, IEntity entity);
i.o.w., it's already virtual. ? as in: you can override it and as you have a method in CommonEntityBase, the base call always lands there. (as it doesn't call EntityBase(2)'s version anyway).
So I'm not really sure what you mean with making that method 'virtual'
Joined: 03-Nov-2007
Joined: 03-Nov-2007
rdhatch wrote:
We would feel very comfortable knowing this would be included in v3.0. We are not asking you to release new templates for v2.x.
Our goal is to allow all new & existing LLBLGen projects to be able to use dynamic prefetch paths. We humbly assert that dynamic prefetch paths is of great value to the LLBLGen Community & should be a standard feature of LLBLGen Pro.
We want dynamic prefetch paths to be available to everyone, but also optional. If someone wants to use them, simply override CommonEntityBase.SetRelatedEntityProperty.
Right now, base.SetRelatedEntityProperty will fail for someone not using dynamic prefetch paths because its abstract on the EntityBase. If it was virtual, everything would work excellent & smoothly.
Thank you for everything, Frans. We look forward to v3!
Ryan
Ok, then we understand each other .
I'll make a note with the change request that the base method should be virtual (empty) instead of abstract so overriding it in templates can be optional
Thanks for all your help on this, Frans. We are very grateful!
In testing the dll, there seems there is one flaw, however.
When prefetching multiple elements, the pk-hash from the first element will be cached. Therefore, the rest of the elements will be merged using that cached pk-hash.
or
"First element wins"
This code will only populate countries
FruitEntity result = new FruitEntity(fruitId);
PrefetchPath2 prefetch = new PrefetchPath2(result.LLBLGenProEntityTypeValue);
// generated prefetch
prefetch.Add(FruitEntity.PrefetchPathCountries);
// custom prefetch
prefetch.Add((IPrefetchPathElement2)result.enhancements.PrefetchPathIncludeAttributes);
But this will only populate attributes:
FruitEntity result = new FruitEntity(fruitId);
PrefetchPath2 prefetch = new PrefetchPath2(result.LLBLGenProEntityTypeValue);
// custom prefetch
prefetch.Add((IPrefetchPathElement2)result.enhancements.PrefetchPathIncludeAttributes);
// generated prefetch
prefetch.Add(FruitEntity.PrefetchPathCountries);