Frans,

Is there a way to test if an entity field (collection based on a m:n relation) is fetched?

(Adapter)

Marcus

entity.Collection.Count > 0

There is no other way. If the data isn't fetched, the count is 0 (you'll get a new collection)

Otis wrote:

entity.Collection.Count > 0

I'm doing that at the moment... but my method receives an Entity which may or may not have its relations prefetched... They could very well have Count = 0 even after the fetch which would cause me to do the fetch again...

Would be nice to extend the collection class to indicate state in the same way the Entity.Fields.State works.

Marcus

Marcus wrote:

Otis wrote:

entity.Collection.Count > 0

I'm doing that at the moment... but my method receives an Entity which may or may not have its relations prefetched... They could very well have Count = 0 even after the fetch which would cause me to do the fetch again...

Ah the lazy loading problem. In there I stored a flag which was set once the collection was fetched.

However with adapter, you know when you fetch something, so isn't this a bit of a red herring?

Would be nice to extend the collection class to indicate state in the same way the Entity.Fields.State works. Marcus

That's not always something which would work. Problem is: is the collection part of the entity or not? And what if you fetch a filtered set of data into the collection and that gives 0 results, is the collection then fetched or not? (I'd say: no, because there are probably results for that relation but not with the filter used the last time).

I'm not totally decided what to do with containing collections as in: are they part of the containing entity or not.

Otis wrote:

However with adapter, you know when you fetch something, so isn't this a bit of a red herring?

Yes I see what you're getting at... I could re-factor the code to ensure that this method always expects a fully fetched entity... I'm using the strategy pattern to perform prefetches but the full strategy is not always known up front. In a sense I am doing a custom lazy load further down the pipeline and this particular method is shared by many strategies. Will need to think it through some more.

Otis wrote:

That's not always something which would work. Problem is: is the collection part of the entity or not? And what if you fetch a filtered set of data into the collection and that gives 0 results, is the collection then fetched or not? (I'd say: no, because there are probably results for that relation but not with the filter used the last time).

I'm not totally decided what to do with containing collections as in: are they part of the containing entity or not.

How about putting a flag in the Entity itself:

IEntityCollection2 Entity.SomeCollection{ get; set; }
bool Entity.SomeCollectionIsFetched { get; }

Marcus wrote:

Otis wrote:

That's not always something which would work. Problem is: is the collection part of the entity or not? And what if you fetch a filtered set of data into the collection and that gives 0 results, is the collection then fetched or not? (I'd say: no, because there are probably results for that relation but not with the filter used the last time).

I'm not totally decided what to do with containing collections as in: are they part of the containing entity or not.

How about putting a flag in the Entity itself:

IEntityCollection2 Entity.SomeCollection{ get; set; }
bool Entity.SomeCollectionIsFetched { get; }

Isn't that a bit weird? I mean: the adapter classes are considered disconnected from whatever persistence store, so if there is no data in the collection, either you didn't fetch it or it wasn't there. Which one it is, depends on your own code. If you request the data for some entity and some related collections, the data is thus fetched and apparently there was no data.

the thing is that you should simply think in these steps: - I want to process something - I need data - I get the data - I got the data and will now process it. - I'm finished processing - Next!

You're either in this cycle or outside this cycle.

It's not that weird.

In the very simplest case, I would want my code to test that the correct data exists (was pre-fetched) before it performed some calculation, and throw an exception otherwise.

So, for example, say I extend a PersonEntity to create a shortcut to some deeper Entity's property.

partial class PersonEntity {

public int NumberOfToes { get { if (!this.Toes.HasBeenFullyFetchedOrWhatever) throw new Exception("Not all data has been prefetched."); return this.Toes.Count; } }

}

This is the most logical place to do this kind of check. However, it requires that the entity know a little more about itself, namely what data it has available.

That's a very old thread, next time, please open a new thread as described here: http://www.llblgen.com/TinyForum/Messages.aspx?ThreadID=7725

Also you can test if an entity is fetched by checking the myEntity.Fields.Status property. Please check the Reference manual for possible values.

I did open a new thread and I was directed to this thread.

Anyway, Fields.Status does not help in this case since I'm talking about related collections and prefetched paths, and any entity-wide status is not specific enough.

So my question still stands. I'll repost here to make it easier to find:

It would be nice to know if a property on an entity has been prefetched, and if it has, which arguments where used to prefetch it. With this information, business logic could do something like the following:

1) Check if a property is prefetched. If it is not, either do so or throw an exception. 2) If it was, make sure that the filters used to prefetched it provide enough data for this use case. If not, prefetch additional data, or throw an exception.

Right now, as far as I know, the best you can do in the case of an x-to-1 property is to fetch it if it is currently null, and if you don't get any results then you know that it truly is null. For m-to-n properties, you always have to assume that it may not be the full collection. It could have not been fetched at all or it could have been fetched with a filter.

Is there already something in place for this that I don't know about. If not, do you have some code we could use to modify our code templates that would add this functionality.

As is, you just don't really know what the state of an entities' properties.

This is context bound information. Say I have a customer collection, and I want to check if they have any orders. If that customer collection already lives in my application, I can't rely on checks on these entities, as perhaps there are orders now for these customers. So I have to fetch the data. AFTER that fetch I can decide if there are no orders that the customer indeed has no orders because I looked for orders first.

You can't rely on any in-memory state if you don't fetch first. So if you check the in-memory state at location B, you can assume in B that the data was fetched first at location A.

What if we add such a flag, so if I look at myCustomer.Orders.IsPrefetched and see that it is true and the collection is of size 0, does that tell me if the customer has indeed no orders? No it tells me that of time of fetching customer + orders, there weren't any orders. If that fetch took place at an unknown time ago, I can't rely on the size of the in-memory collection for my calculation.

Without wishing to take sides on the subject of data staleness etc, here is a possible partial solution (untested) in that it will tell you whether the collection was retrieved or not (but won't tell you how it was retrieved)

You should be able to add a property as a partial class or via modifying the templates to just check whether the backing field is null or not.

        public bool IsXXXLoaded
        {
            get { return _XXX != null; }
        }

Of course this will only work if you didn't try to access the collection via its normal property in which case this field will be lazy-instantiated.

Cheers Simon

What if we add such a flag, so if I look at myCustomer.Orders.IsPrefetched and see that it is true and the collection is of size 0, does that tell me if the customer has indeed no orders? No it tells me that of time of fetching customer + orders, there weren't any orders. If that fetch took place at an unknown time ago, I can't rely on the size of the in-memory collection for my calculation.

My 2 cents.

This feature has been asked for many times, usually with a reply from "the team" similar to this. To me, the reason for not providing this is a read herring. Based on this logic, if I have an entity with a field customer name. I can't use customer name at any time after the fetch since we don't know that it hasn't changed in the database.

As entities and their related collections get passed, moved or used in an application far from the fetching point, to me it seems perfectly reasonable to want to ask the entity "Is that collection empty because it hasn'r tried to be filled or is it empty because it was tried to be filled but there was no data to put in it". I am probably missing something, but, to me, it seems the kind of thing for the framework to know and provide to the developer.

myCustomer.Orders.IsPrefetched

Of-course this would be needed when we have an empty or null Orders collection. Would something like Customer.NumberOfOrders do the trick? As described in the following blog: http://weblogs.asp.net/fbouma/archive/2006/06/09/LLBLGen-Pro-v2.0-with-ASP.NET-2.0.aspx (look for Extending the CustomerEntityFactory)

So if the Customer.Orders was empty, checking Customer.NumberOfOrders would let us know if actually there are no Orders on the database, or there are some orders and I might need to re-fetch the Customer.Orders

arschr wrote:

What if we add such a flag, so if I look at myCustomer.Orders.IsPrefetched and see that it is true and the collection is of size 0, does that tell me if the customer has indeed no orders? No it tells me that of time of fetching customer + orders, there weren't any orders. If that fetch took place at an unknown time ago, I can't rely on the size of the in-memory collection for my calculation.

My 2 cents.

This feature has been asked for many times, usually with a reply from "the team" similar to this. To me, the reason for not providing this is a read herring. Based on this logic, if I have an entity with a field customer name. I can't use customer name at any time after the fetch since we don't know that it hasn't changed in the database.

No, that's not the same thing. I provide a reason why it's not there simply because it leads to unfounded decisions, and the flipside of that is that if you simply fetch first, do action, etc. you KNOW what you've fetched (as you want to perform an action).

As entities and their related collections get passed, moved or used in an application far from the fetching point, to me it seems perfectly reasonable to want to ask the entity "Is that collection empty because it hasn'r tried to be filled or is it empty because it was tried to be filled but there was no data to put in it". I am probably missing something, but, to me, it seems the kind of thing for the framework to know and provide to the developer.

Just because the pro's of the feature look good, doesn't mean the con's of the feature are as well good. When I add something to the framework, I can't get it out, it will be there forever.

Think for a second WHY you want to have this feature. Number 1 reason: so I don't have to go to the database to fetch them, I can SAVE a roundtrip.

No you can't. Or better: you shouldn't. Relying on that flag is by definition unreliable. That's why the flag isn't there.

Think for a second WHY you want to have this feature. Number 1 reason: so I don't have to go to the database to fetch them

No, that's the number 2 reason I want that feature.

The number 1 reason is, (as I've tried to say before), so I can tell why the collection is empty. Is it because when it was fetched there were none in the database or if it is empty because it wasn't in the list of things to prefetch. Two wildly different situations, that I don't know how to tell apart from looking at the collection.

If it wasn't asked for, I can make a decision (such as tough luck, you didn't ask for this when you asked for the information, or maybe even (gasp) go and get them now [but, if I can avoid a connection and query, I would like too]). If it was asked for I can say "nope at the time this was gotten from the database, there were none". While there may be now, I in the disconnected application, don't care about now. If I did I wouldn't be writing a disconnected application.

If you understand me, and decide this isn't a good feature, then fine, I'll try to keep quite when others ask for it. But every time you answer this request, It seems to me, you are misunderstanding the idea of disconnected apps and why I think it would be good to have.

arschr wrote:

Think for a second WHY you want to have this feature. Number 1 reason: so I don't have to go to the database to fetch them

No, that's the number 2 reason I want that feature.

The number 1 reason is, (as I've tried to say before), so I can tell why the collection is empty. Is it because when it was fetched there were none in the database or if it is empty because it wasn't in the list of things to prefetch. Two wildly different situations, that I don't know how to tell apart from looking at the collection.

I understand that when following that, the problem is there, but think about this: when you start the processing of a set of entities: don't you agree that the only reliable result of that processing is the one where you KNEW UP FRONT that the data you put into process was the data you HAD TO put into the process? (No, this isn't a Duh! moment, it's the root of the problem ) Ok, now: that's pretty obvious, but: how can you reliably say that the data you hold in memory is the data you should put into the process? IMHO only if the data is pulled from the application state (!) right before the process. - process starts (so the process 'scope' starts) - data fetched from application state. - data is after that in theory 'stale', but local to the process scope, so not stale there - data is processed - processed results are persisted to application state. - 'scope' ends, process ends.

That second step is essential, as you can't reliably say: "the data I have in memory is the data I have to use".

So, now we come to your empty collection. If the collection is empty, it has just one conclusion: there was no data. It's that simple.

There's no room for 'what should I conclude: was it fetched or is there no data?' because that question isn't necessary: you fetched the data, so if there's no data, there is really no data.

If it wasn't asked for, I can make a decision (such as tough luck, you didn't ask for this when you asked for the information, or maybe even (gasp) go and get them now [but, if I can avoid a connection and query, I would like too]). If it was asked for I can say "nope at the time this was gotten from the database, there were none". While there may be now, I in the disconnected application, don't care about now. If I did I wouldn't be writing a disconnected application.

So when does data become stale in your opinion? After 10 minutes, after a day? Every application in the business world is disconnected, they all have UI screens which aren't served by the application state governor (RDBMS) so the application state is by definition out of sync with the user state.

However, when you start a processing action, you first need to obtain the data. If you solely do that from user-state, you run the risk of working with seriously outdated data, as you should obtain data from the application state, as the user state is just a store-forward storage of the VIEW of the user on the application state.

So, the data obtain step is part of the process action: after the obtain action took place, the data is inside the user state and up to date for the duration of the process. You dont have to ask questions like you put forward anymore, as you can conclude just one thing.

You also can't avoid the step of obtaining the data. You simply can't: if you do, what's the use of the data resulting of the process action? IMHO nothing, as it's based on an outdated view on reality. What if you're processing a set of customers and these customers are cached for over an hour already in the application, and you now start an action to process all customers with orders. You now seriously want to rely on a flag in customer.Orders for that process? that data is over an hour old It could be seriously outdated, and it's leading to false positives because the results of the process might end up in the application state (!) as a modification of that application state but the INPUT for that modification wasn't pulled from the application state, but from the user state.

If you understand me, and decide this isn't a good feature, then fine, I'll try to keep quite when others ask for it. But every time you answer this request, It seems to me, you are misunderstanding the idea of disconnected apps and why I think it would be good to have.

I do know how disconnected apps work and what staleness is of data, and it is a problem which plagues all the designs of applications which communicate with a service, database or other data source.

What I don't want to add is a feature which leads to software which isn't reliable. That's also why there's a discussion about user state and application state in the manual: it's very important to understand what the difference is and how to work around application-wide staleness and local correctness.

I said I wouldn't take sides but...

What Frans says applies to, and works well, with a single use-case. In that scenario, you KNOW exactly what data was requested and you KNOW it isn't stale data. Therefore you know that a zero means zero.

But suppose the entity containing the child collection is then passed to another routine which doesn't know the history/staleness. In that scenario it would be useful for that routine to be able to detect whether certain information is present or not. (I'm trying to be as abstract as possible here but I can provider a real scenario if required)

The bottom line is that IF you know the history then you KNOW what that zero means. If you don't know the history then you don't KNOW what that zero means. Frans is of the opinion that you should always KNOW the history; others appear to have developed scenarios where they don't KNOW the history and they can't get the history.

I am tending towards the latter camp but the obvious solution, putting a flag on the collection, has potential performance implications and so I tend to agree with Frans that it shouldn't be added without all the implications/other solutions considered first.

One partial solution is listed above, adding a method to check whether the backing field is null or not (this may only apply to v2.5 though). This is simple and requires no changes to the runtime or templates but can fail if the collection's public property is accessed first. A further solution might be to intercept the exact state of the entity at the point it was fetched and store the null/not-null state of the backing field. The new property then returns that information rather than the current state.

Cheers Simon

I felt the need to have some kind of boolean property to know if a related collection was already here or not when I started using adapter. Some of my entites are passed to many methods which need to access their related collection. Depending on the places from where these methods are called, the related collection might be there... or not, and if it's there, it is a good point to avoid refetching everything. Of course I agree that the data you're working on needs to be fresh, in my case there is no user interaction between method calls, so it's a matter of milliseconds.

I solved this issue by adding a "shouldFetch" boolean parameter to my methods so they know if data can be used as it or needs to be refetched. It's a minor annoyance and not a bad thing because it forces me to do the thinking.

I think basically you are making two mistakes in your reasoning: 1) That it is not possible to fetch stale data from the database and that an application would have no desire to do so. 2) That every object that ever touches the data knows the complete history of it, or if it does not, that it does not need to know.

*** First, I'll address #1. ***

You argued the following:

...how can you reliably say that the data you hold in memory is the data you should put into the process? IMHO only if the data is pulled from the application state (!) right before the process.

• process starts (so the process 'scope' starts)
• data fetched from application state.
• data is after that in theory 'stale', but local to the process scope, so not stale there
• data is processed
• processed results are persisted to application state.
• 'scope' ends, process ends.

Application state is not the only source of reliable data. You're making a lot of assumptions here that are probably not true for most people. 1) You assume that the database does not have any concept of time. As soon as you add a few columns to a table to let you know when a row was created or tombstoned, you are now able to get the exact state of the data at any point in time. In other words, though the application data is, in theory, stale, the new data you can fetch becomes stale at the same rate. 2) You assume that the whole process is not done within a single transaction that allows every data fetch to be done in such a way that it is not equally stale. In essence, this puts the application and the data in the same "process scope", so they both become stale at the same rate. 3) You assume that even if a transaction or some sort of time-traveling is not used, that the application even cares that the newly fetched data is stale by 1/100 of a second.

As you can see, it is perfectly easy to fetch additional data that a caller may not have known a callee needed and still have all of the data be valid. True, it may not be the most efficient way to do it, but that is not the argument.

There are lots of reasons an unplanned lazy loading may occur: 1) Influences outside of your control 2) Cases you may not have planned for. These could be optimized and handled later, but time constraints could require that if lazy loading is needed, that it is done until it is optimized out. 3) It may be more efficient to lazy-load. In some common data-fetch method, you may just fetched enough data to be used by 99% of the cases, but in the rare 1% of the places the data is used may require additional data, and the overhead of fetching the extra data for the extra 99% may be too much overhead.

So you can't just discount it and say: "No, no one will ever want to lazy load at any time if they are using a pre-fetch model," because that's just not reality.

*** Second, I'll address #2. ***

For this argument, it's simpler to ignore lazy loading all together, since there is more than enough reasoning without even considering cases it is needed.

The primary premise to my argument is this: In LLBLGen's default generated code, an entity's current state is never fully known!!!!!!! The programmer may or may not know the state, but the code does not.

Please re-read that premise, as it is important.

First, before you respond by saying that the entity's state is know because you are the one who did the data fetch, let me say that it is the programmer that might know this state, and not the code itself. This may be the case if you are writing very simple methods that fetch the data, use it only within that method's scope (no help from any outside methods or any reusable code), and then disposes of it. But once you start creating reusable objects, and performing proper encapsulation, the code itself needs to know the state (history) of the entities that are passed to it, and not just the programmer. You can't just assume that the code using an entity knows the entity's complete history or even that it knows who its caller is.

For proper OO encapsulation, you cannot make assumptions about the data that's being given to your object. At very least, you need to put prerequisites (i.e. a contract) on that data and throw exceptions if your prerequisites are not met. In order to know if youre prerequisites have been me, the "state" of the data needs come along with the data.

The point is that the object is saying: "I DIDN'T fetch the data. Somebody else fetched the data and gave it to me. Now I need to figure out what to do with it."

The way I see it, there are only two ways to work with the LLBLGen code as is: 1) Rely on the programmer to handle every possible case, which can only easily be done using simple fetch/do-something/dispose patterns, and very little code reuse. 2) Remember entity state through custom code. The only way to know is to pass around the entire graph of all of the prefetch path objects, filters, relations, sorting, etc. that were ever used on an entity and its children. You would then have to parse that graph somehow , and determine the state of the entity/entitycollection with which you are concerned. But that is basically what we are asking LLBLGen to do for us, as it seems like a very important missing piece.

You mentioned over and over that you should always just know the history of the data that has been fetched:

So, now we come to your empty collection. If the collection is empty, it has just one conclusion: there was no data. It's that simple. … There's no room for 'what should I conclude: was it fetched or is there no data?' because that question isn't necessary: you fetched the data, so if there's no data, there is really no data.

The first quote only addresses one scenario. Let me rephrase that quote to something more generic that better exemplifies the issue:

So, now we come to your [related entity property or entity collection]. ...it has just one conclusion: [the data that was there is an exact copy of the persisted data]. It's that simple.

Sorry, but that's not true at all. The programmer may be able the conclude this, but the code certainly can not. Here are some possible cases of unknown state that I can think of: 1) An m-to-1 related entity property is null. This could mean three things: a) The data does not exist (in the repository). b) The data was not prefetched at all. c) The data was prefetched but filtered out by predicate expressions. 2) A 1-to-n related entity collection property is empty or contains any number of items. This could mean many, many things. Here are four of them: a) The non-existent entities do not exist (in the repository). b) The data was not prefetched at all. c) The data was prefetched but filtered out by predicate expressions. d) The data may be fetched or not fetched, but is in the wrong order due to a sorting expression that was applied.

In other words, if you don't know the state (history) of the data then the code doesn't know what ANY related entity properties or entity collection property mean. So, for a very common example, consider the following: Entity A could have a related entity collection of B's with 69,834 items in it, but A could have been prefetched with a filter that excluded 146 items of entity collection B. Therefore, it is incomplete. Without the history, you just don't know that.

*** Software Reliability ***

What I don't want to add is a feature which leads to software which isn't reliable.

It seems to me that applying the principles of OO design, including property object encapsulation would only make software more reliable, not less. As I've show above, encapsulation can only be done by providing data state.

*** Example *** Let me conclude with a concrete example of psuedo code (easier to read) to show you one very common scenario for me.

Public Property SomePropertyThatExtendsAGeneratedEntity
    // Check prerequisites on FirstEntityCollection
    If (Not this.FirstEntityCollectionToWorkOn.SyncState.ContainsEntitiesWhere(FirstEntityFields.IsEnabled == True))
    
        #if ThrowOnMissingData
           Throw new EntityStateException("Not all required data has been prefetched.  Please tell your programmers to be more careful.")
        #endif
        
        #if FetchMissingData
            this.FetchMissingFirstEntityDataWhere(FirstEntityFields.IsEnabled == True)
        #endif
    
    End If
    
    // Check prerequisites on SecondEntity
    If (not this.SecondEntitySyncState.IsFetchedUsing(SecondEntityFields.IsEnabled == True)
        
        #if ThrowOnMissingData
           Throw new EntityStateException("Not all required data has been prefetched.  Please tell your programmers to be more careful.")
        #endif
        
        #if FetchMissingData
            this.FetchMissingSecondEntityDataWhere(FirstEntityFields.IsEnabled == True)
        #endif
    
    End If
    
    // return results
    Return This.DoSomethingThatNeedsAllEnabledFirstEntitiesAndAnEnabledSecondEntity(this.FirstEntities, this.SecondEntity)
End Property

And then if you want to eliminate the possibility of invalid data, you either dispose of this object before then end of the transaction, or else you change the prerequisite-check predicate to the following:

FirstEntityFields.IsEnabled == True
 && FirsetEntityFields.TombstoneStartTime <= this.FetchTime
 && (FirstEntityFields.TombstoneEndTime = Null  || FirstEntityFields.TombstoneEndTime > this.FetchTime)

joshmouch wrote:

I think basically you are making two mistakes in your reasoning: 1) That it is not possible to fetch stale data from the database and that an application would have no desire to do so. 2) That every object that ever touches the data knows the complete history of it, or if it does not, that it does not need to know.

ad 1: the moment you fetch the data, it's by definition 'stale'. However for the PROCESS which fetches the data, it's not. However if you obtain data from somewhere else, it IS stale for the process as well, as the PROCESS hasn't fetched it. So the PROCESS doesn't know the history. That's all I am arguing.

ad 2: I'm not claiming 2. The object itself doesn't know any history. That's the whole point here. The people who want to have the flag want to make the object know its history, however only the last action but when that took place, in which context etc. that's unclear.

The process itself knows what it did, so it doesn't have to wonder IF something was fetched, as it did all the fetching itself. That's what I am arguing.

If the process obtains data from somewhere else, it doesn't know what happened. So it has to guess. That's unreliable, as a flag doesn't make it true that there are or aren't entities available in the db when the process STARTS.

*** First, I'll address #1. ***

You argued the following:

...how can you reliably say that the data you hold in memory is the data you should put into the process? IMHO only if the data is pulled from the application state (!) right before the process.

• process starts (so the process 'scope' starts)
• data fetched from application state.
• data is after that in theory 'stale', but local to the process scope, so not stale there
• data is processed
• processed results are persisted to application state.
• 'scope' ends, process ends.

Application state is not the only source of reliable data. You're making a lot of assumptions here that are probably not true for most people.

I hardly doubt that, but let's address that below

1) You assume that the database does not have any concept of time. As soon as you add a few columns to a table to let you know when a row was created or tombstoned, you are now able to get the exact state of the data at any point in time. In other words, though the application data is, in theory, stale, the new data you can fetch becomes stale at the same rate.

I'm talking about memory vs. database. Not database vs. database. You have a process, you need customers and their orders. You have customers cached somewhere, you think "I'll re-use those". However, you need to examine the Orders collection to see if there are orders and if not you MIGHT need to fetch them. You can't reliably check that NOW as there's no other information than "there are no orders". So that can mean a) there are no orders, or b) they weren't fetched. With the FLAG, you might say: "the flag says they were fetched, so as I don't have any orders, there aren't any orders". And that's wrong. Because IF you would instead fetched the customers + orders from the database at the time when the process started, you would a) be able to determine if there aren't any orders, namely the collection is empty, and b) you have the latest data.

2) You assume that the whole process is not done within a single transaction that allows every data fetch to be done in such a way that it is not equally stale. In essence, this puts the application and the data in the same "process scope", so they both become stale at the same rate.

I think we have a confusion what 'stale' means in my reasoning. Let me re-phrase it as 'out of sync'. So when I mentioned 'stale' I meant that the in-memory entities' data was state, or better: the entities were likely to be out-of-sync and should be refetched.

3) You assume that even if a transaction or some sort of time-traveling is not used, that the application even cares that the newly fetched data is stale by 1/100 of a second.

No, that's not what I meant. I don't mean the difference between 'staleness' of data right after it's fetched compared to the data in the db, because that's irrelevant for this issue: as the issue is about a flag if you can reliably determine IN-MEMORY if the data IN-MEMORY is usable as-is or that you have to fetch it. That's not possible, as the scope of the process sets the borders of the staleness of data in-memory.

As you can see, it is perfectly easy to fetch additional data that a caller may not have known a callee needed and still have all of the data be valid. True, it may not be the most efficient way to do it, but that is not the argument.

That's also not the issue. The issue is if you can IGNORE the fact that you have to fetch data, because you can ASSUME there is no data because a flag says "this collection has already been fetched". But perhaps that fetch took place with a filter that doesn't apply to this situation or it took place 2 hours ago and the collection was cached, could you tell? No that's not possible from a flag.

So the only way to know IF there is any data is to fetch it from the db at the start of the process.

*** Second, I'll address #2. *** For this argument, it's simpler to ignore lazy loading all together, since there is more than enough reasoning without even considering cases it is needed.

The primary premise to my argument is this: In LLBLGen's default generated code, an entity's current state is never fully known!!!!!!! The programmer may or may not know the state, but the code does not.

Please re-read that premise, as it is important.

First, before you respond by saying that the entity's state is know because you are the one who did the data fetch, let me say that it is the programmer that might know this state, and not the code itself. This may be the case if you are writing very simple methods that fetch the data, use it only within that method's scope (no help from any outside methods or any reusable code), and then disposes of it. But once you start creating reusable objects, and performing proper encapsulation, the code itself needs to know the state (history) of the entities that are passed to it, and not just the programmer. You can't just assume that the code using an entity knows the entity's complete history or even that it knows who its caller is.

Then there is no way to tell what happened. int count = myCustomer.Orders.Count;

count appears to be 0.

What does this tell you? It can only tell you one thing: there is NO DATA. That's it. There is NO OTHER thing you can conclude. Now lets see if I add a flag: if(!myCustomer.Orders.HasBeenPrefetched && count==0) { // fetch the data }

This is silly code. The thing is this: it assumes that HasBeenPrefetched will return the orders needed for the code making the call to the flag. That is wrong. Perhaps the fetch used a filter on the prefetch node. Perhaps it only returned the last 10 orders and you have code which checks for customers with an order count > 20. As the flag says the data has been fetched (apparently) the count of the collection is the truth, right? No! What if I indeed fetched only the last 10 orders, then I can't use any customer, as they all have 10 or less order entities. Still, the flag says the data has been fetched.

So you also need to know the whole context in which the data was fetched: which filters were used etc. etc. You then have to examine them to make a good judgement. Right?

No you don't need to. The reason is simple: if I call a method to do some processing and the routine itself accepts the data to process, it simply assumes the data to process is there. If there's no data, the only conclusion is that there's no data to process. PERIOD. Otherwise the callee should have fetched it properly and passed it in.

If there's a complex chain of callers which pass on data and together they form a whole process, do you really think you can add any code which fetches data on the fly based on a flag? No you can't, because there's no scope information in these separate methods other than that the data they get passed in is the reality they'll work with. If you THINK you can do fetching based on a flag inside a collection INSIDE such a method, you at that spot assume the scope when the entity data of the entities containing the collections (e.g. customers) was fetched is always the same. But that's not the case. That can be very different. For you to re-use code you therefore can't assume that, you need to work with the data you get passed in and not fetch additional data inside the method based on history embedded in a single flag.

For proper OO encapsulation, you cannot make assumptions about the data that's being given to your object. At very least, you need to put prerequisites (i.e. a contract) on that data and throw exceptions if your prerequisites are not met. In order to know if youre prerequisites have been me, the "state" of the data needs come along with the data.

Erm... isn't that what I was saying all along? You get the data, there's just one conclusion possible, there's NO flag which can help you as that's unreliable. If you want to base decisions on that flag, you can't do that.

The point is that the object is saying: "I DIDN'T fetch the data. Somebody else fetched the data and gave it to me. Now I need to figure out what to do with it."

But it can just draw one conclusion when there's no data, e.g. a customer.Orders.Count is 0: there's no data. You can't at THAT spot make decisions what to do otherwise. If you think you need to/should, please explain to me how you would do that, as you need to examine the complete history of fetch statements, i.e. : filters, prefetch paths etc. etc. and then decide if it is a valid decision to assume there is indeed no data or there 'might be' data and you STILL have to fetch from the database.

If the collection is empty and the flag is set you can't decide there is no data IF you make that flag the decision maker what to do: the in-memory data can be seriously out of sync with the data in the db + the fetch could have used different parameters than you would/will apply now.

The way I see it, there are only two ways to work with the LLBLGen code as is: 1) Rely on the programmer to handle every possible case, which can only easily be done using simple fetch/do-something/dispose patterns, and very little code reuse. 2) Remember entity state through custom code. The only way to know is to pass around the entire graph of all of the prefetch path objects, filters, relations, sorting, etc. that were ever used on an entity and its children. You would then have to parse that graph somehow , and determine the state of the entity/entitycollection with which you are concerned. But that is basically what we are asking LLBLGen to do for us, as it seems like a very important missing piece.

Are you seriously asking for history tracking ? Why is that part of the framework if it is part of YOUR application? How do you see this in action then? It's very easy to write 2 bulletpoints saying that there are only 2 ways to work with llblgen pro code: 1 is too simple for most apps and the other one is such an incredible pain that it is impossible to do, as if it's impossible to write normal software today. Huh?

What's wrong with normal code which gets data to process and processes it as-is. IF the caller wanted it work on more data, it should have fetched that data.

How else would you be able to determine in ANY form if the data you HAVE is indeed fetched with the exact parameters you would otherwise use to fetch it? Because that's point 2) in your list of 2 ways how to work with our code, right?

You mentioned over and over that you should always just know the history of the data that has been fetched:

So, now we come to your empty collection. If the collection is empty, it has just one conclusion: there was no data. It's that simple. … There's no room for 'what should I conclude: was it fetched or is there no data?' because that question isn't necessary: you fetched the data, so if there's no data, there is really no data.

The first quote only addresses one scenario. Let me rephrase that quote to something more generic that better exemplifies the issue:

So, now we come to your [related entity property or entity collection]. ...it has just one conclusion: [the data that was there is an exact copy of the persisted data]. It's that simple.

Sorry, but that's not true at all.

Strawman argument. My motivation doesn't need rephrasing, and if it does, please do it properly. You forgot the essential piece: at the time of the process start. Now you have forgotten that essential piece by rephrasing my argument you now say I'm wrong.

This kind of discussions is precisely why I added a page to the concepts about application state vs. user state. These aren't the same, and can't be the same as the user application works outside the db, the central store with the application state. When the application starts a process, it fetches data, works on that, persists the results. this might take 20 minutes, who knows. During that time the data OF COURSE gets stale, as it doesn't get updates from the db that there are new rows and what not, it works in isolation. After the process has ended, it is done and it can give back the results to the application state.

Nowhere did I claim that in the middle of the process the data in-memory is perfectly the same as the data in the application state, as that's totally not what I meant.

The programmer may be able the conclude this, but the code certainly can not. Here are some possible cases of unknown state that I can think of: 1) An m-to-1 related entity property is null. This could mean three things: a) The data does not exist (in the repository). b) The data was not prefetched at all. c) The data was prefetched but filtered out by predicate expressions. 2) A 1-to-n related entity collection property is empty or contains any number of items. This could mean many, many things. Here are four of them: a) The non-existent entities do not exist (in the repository). b) The data was not prefetched at all. c) The data was prefetched but filtered out by predicate expressions. d) The data may be fetched or not fetched, but is in the wrong order due to a sorting expression that was applied.

In other words, if you don't know the state (history) of the data then the code doesn't know what ANY related entity properties or entity collection property mean. So, for a very common example, consider the following: Entity A could have a related entity collection of B's with 69,834 items in it, but A could have been prefetched with a filter that excluded 146 items of entity collection B. Therefore, it is incomplete. Without the history, you just don't know that.

And what about fetching the right data for the process, so A has indeed its 69,834 B's and the process is completed OK? Why do you need the history?!

There's just 1 reason: to avoid db calls. If you can examine the history, you can conclude if the state / data you have is valid, so you don't have to fetch the data when the process starts

How would you check history in an entity? Do you WANT to ? If yes, is the history meant for the OBJECT or the DATA ? Because the OBJECT is just a bucket. The data is the entity, so is the history for the entity?

And then what? Then you have the history, all the prefetch paths, filters and what not, and how would that work then? Is THAT even reliable? If you say: "yes", in what context? The process? The complete application across all threads? All servers in the farm?

Impossible to say. And IMHO also unnecessary complexity. Fetch the data you need, pass the data to the routine to process it (or ask the fetched data to process itself) and thus if there's no data, there IS NO DATA, at least not for the process, otherwise it would have been there.

You, in a lot of words, more or less say that without a big, overly complex history tracker and querying system it is impossible to properly write software using llblgen pro code, am I correct in that?

*** Software Reliability ***

What I don't want to add is a feature which leads to software which isn't reliable.

It seems to me that applying the principles of OO design, including property object encapsulation would only make software more reliable, not less. As I've show above, encapsulation can only be done by providing data state.

What have some general OO principles to do with the specific problem at hand? What was suggested was a flag to see if a collection was prefetched. That flag has a meaning. If I add it I can't remove it anymore, and everyone seeing that flag will conclude things from it, and all of them are probably not correct as they run a big risk of concluding the wrong thing. I then say: no, this flag isn't going to be added, because it gives people the ability to conclude the wrong thing while they THINK they conclude the RIGHT thing.

*** Example *** Let me conclude with a concrete example of psuedo code (easier to read) to show you one very common scenario for me.

Public Property SomePropertyThatExtendsAGeneratedEntity
    // Check prerequisites on FirstEntityCollection
    If (Not this.FirstEntityCollectionToWorkOn.SyncState.ContainsEntitiesWhere(FirstEntityFields.IsEnabled == True))
    
        #if ThrowOnMissingData
           Throw new EntityStateException("Not all required data has been prefetched.  Please tell your programmers to be more careful.")
        #endif
        
        #if FetchMissingData
            this.FetchMissingFirstEntityDataWhere(FirstEntityFields.IsEnabled == True)
        #endif
    
    End If
    
    // Check prerequisites on SecondEntity
    If (not this.SecondEntitySyncState.IsFetchedUsing(SecondEntityFields.IsEnabled == True)
        
        #if ThrowOnMissingData
           Throw new EntityStateException("Not all required data has been prefetched.  Please tell your programmers to be more careful.")
        #endif
        
        #if FetchMissingData
            this.FetchMissingSecondEntityDataWhere(FirstEntityFields.IsEnabled == True)
        #endif
    
    End If
    
    // return results
    Return This.DoSomethingThatNeedsAllEnabledFirstEntitiesAndAnEnabledSecondEntity(this.FirstEntities, this.SecondEntity)
End Property

And then if you want to eliminate the possibility of invalid data, you either dispose of this object before then end of the transaction, or else you change the prerequisite-check predicate to the following:

FirstEntityFields.IsEnabled == True
 && FirsetEntityFields.TombstoneStartTime <= this.FetchTime
 && (FirstEntityFields.TombstoneEndTime = Null  || FirstEntityFields.TombstoneEndTime > this.FetchTime)

That's all cool, but you also know it's impossible to build this. The simple fact alone to match prefetch paths + node filters + predicates is showing this isn't going to work. Furthermore, it breaks a simple rule: the entity itself shouldn't know how it is fetched, as it doesn't fetch itself, the repository, manager or whatever you call it, does that. By adding that code to the entity, you leak an abstraction into the entities. This makes the code completely unmaintainable: what if the process changes and some fetch rules are re-ordered? Your code inside the entities, scattered all over the place has to be updated as well. 10 to 1 you'll have a hard time doing that and what's worse it will break at runtime if you forget one or two.

Your logic which checks a related entity's fetch rule is typical: you run into the problem which plagues OO software in general: inside an object, you only see yourself and your references. if you want information about the bigger scope you're acting in, you can't (or better: you shouldn't) know that or it has to be told to you. You shouldn't ask, it should be told to you or you should tell another object what to do. As soon as you ask another object about a state, you're tying things together.

So your code needs info about that related entity's fetch rule to do something. But is that the right place to do that? If you say "yes", you indeed need a history as you don't know anything about the caller. However, the CALLER is a different matter: the caller knows what's going on, it called you. So the code you posted can also simply check if the data is there. If it's not, it can conclude: the data isn't there, conclude it's indeed not there. Because: otherwise it creates a dependency back to the caller's info, which isn't in the callee's object (and for a reason!) So instead, it should rely on the caller that the data is there as the caller uses the object in a process and the method in a process.

I know it's a difficult matter, but there's no reliable way other than "start the process, fetch the data, process the data, persist results", and if you do that in 1 routine or scattered over 1000 objects, it doesn't matter: if you want to fragment 'fetch the data' during the phase 'process the data', you then run into the situation where you need to make decisions you can't make as there's no info at that level where you likely will make the decision, namely inside callee's.

Ok, here is an example of why a flag could be useful....

• WorkItemA is a treeview containing a hierachy 3 levels deep - Ships, Voyages, PortCalls.
• Each node's textual description is based on some information from that entity.
• Taking Voyage as an example - its textual description is based partly on data in the Voyage but also some date information on it gleaned from a "Fixture" (Voyages: Fixtures is a 1:m relationship)

• The treeview is created when the app starts (using TypedLists to get the minimum amount of information) and then it is then updated on a timer.

• WorkItemB is the full Edit screen for a Voyage and the user may Add, Remove Fixtures, or Modify the dates on existing Fixtures.

• It may be started from any number of places in the app: from the treeview, from a search screen etc.

• When changes are saved in WorkItemB, it fires a loosely-coupled event (to any subscriber in the app who registers for it) to say "FYI: Here is the latest information for this Voyage"

• WorkItemC is a voyage search screen containing an editable grid which shows a small subset of information about Voyages.

• It contains nothing about associated Fixtures.
• When changes are saved in WorkItemC, it also fires the loosely-coupled event.

None of the WorkItems are aware of any of any of others.

The loosely-coupled event handler for the treeview can look to see if the Voyage contained in the event data matches any currently shown within the treeview and also **whether there is any Fixture information within it. ** The Voyage node text can then be updated using the information directly from the Voyage but also from the Fixture dates, if any are present.

Without a flag, the handler can't know whether there are 0 Fixtures because a) none were prefetched originally (WorkItemC) or b) Fixtures were deleted in WorkItemB. Hence, to the user who just saved the Voyage in WorkItemB, the treeview is showing old data (at least until the next auto-refresh event fires when it is magically updated)

Cheers Simon

simmotech wrote:

Ok, here is an example of why a flag could be useful....

• WorkItemA is a treeview containing a hierachy 3 levels deep - Ships, Voyages, PortCalls.
• Each node's textual description is based on some information from that entity.
• Taking Voyage as an example - its textual description is based partly on data in the Voyage but also some date information on it gleaned from a "Fixture" (Voyages: Fixtures is a 1:m relationship)

• The treeview is created when the app starts (using TypedLists to get the minimum amount of information) and then it is then updated on a timer.

• WorkItemB is the full Edit screen for a Voyage and the user may Add, Remove Fixtures, or Modify the dates on existing Fixtures.

• It may be started from any number of places in the app: from the treeview, from a search screen etc.

• When changes are saved in WorkItemB, it fires a loosely-coupled event (to any subscriber in the app who registers for it) to say "FYI: Here is the latest information for this Voyage"

• WorkItemC is a voyage search screen containing an editable grid which shows a small subset of information about Voyages.

• It contains nothing about associated Fixtures.
• When changes are saved in WorkItemC, it also fires the loosely-coupled event.

None of the WorkItems are aware of any of any of others.

The loosely-coupled event handler for the treeview can look to see if the Voyage contained in the event data matches any currently shown within the treeview and also **whether there is any Fixture information within it. ** The Voyage node text can then be updated using the information directly from the Voyage but also from the Fixture dates, if any are present.

Without a flag, the handler can't know whether there are 0 Fixtures because a) none were prefetched originally (WorkItemC) or b) Fixtures were deleted in WorkItemB. Hence, to the user who just saved the Voyage in WorkItemB, the treeview is showing old data (at least until the next auto-refresh event fires when it is magically updated)

Cheers Simon

Though, that makes the flag only useful in THAT context, but the flag is a general flag, it should be usable in ANY context, and that's not always the case.

Say I fetch 10 ships and their last 2 voyages (assuming it's a 1:n relation). I do something with the data in-memory, persist it. At that moment, I have the 10 ships in memory, and the flags are set on Voyages: prefetched.

Does that tell you anything useful? IMHO No. In fact: using that flag's value for any decision is unreliable, as I had a limit set on the prefetch path: 2, not 'all'. So in another process (== sequence of actions), I could use that 10 ships again, and they're already in memory. BUt I can't rely on the flag set if there is data fetched or not. This is what I described with for this process the data is 'stale', and should be refetched. So for this other process, you can't rely on the 10 ship entities, you have to fetch them all again.

OR

you can assume that in-memory data is the reality. Then you can re-use the entities, as every change made to an entity is first done on the in-memory graph.

I don't see your code in front of me so I have to guess how everything is wired up, but it sounds similar to what I do in the designer: you have a model and a couple of viewers which subscribe to events, model elements themselves can also subscribe to events, though with one exception: you don't raise events from the changed entities, you raise events from the process dialogs (as I understand it correctly). This is a fundamental difference.

When such an event is raised ('I've been changed!') the viewer's job is to reflect the change. That's it, there's no other meaning to it. So the viewer then can only do 1 thing: reflect the data it sees, and if that can be out-dated because another routine causing the change has saved data into the db and these aren't loaded yet, you have to re-load the data.

I don't see how that flag can help in any way. What does it tell you? Does it tell you which process changed what and how? No, it only tells you that somewhere back in time, a prefetch path fetched the collection. That's it.

However, and this is very important, the prefetch took place in another sequence of actions (== what I call 'process'), which had a start and an end. You can't use information that something was fetched in another process and base your decision IF you have to refetch data again on that particular previous process, simply because that previous process might have used totally different filters and what not, valid for THAT PROCESS, but not for your current process. (process A fetches all last 10 orders for customer, process B needs ALL orders)

Your viewer problem is IMHO caused by this: you have a viewer which should reflect the state of the model at any given time T. This is typically the same as my project explorer: whatever change is made: the tree should reflect the state.

So when something changes, THAT OBJECT raises a change event. If you rename a field, the field raises a changed event. The subscriber to that is the entity. It raises a changed event that the entity was changed and as info it passes on 'field information'. This is received by related entities and by the entity container, which raises the event that some entity was changed, passing along the same info, the project receives this info and raises the event that the project was changed, and its node changes icon. The tree subscribes to all entity's changed event handlers. So when an entity raises its changed event, the tree gets an event that entity X was changed. It looks at the info what exactly was changed and based on that it refreshes info in the tree.

Back to your tree. Your model doesn't signal subcribers that an element has changed. You signal parts of the UI via a different route. That's OK, however in your tree controller, you then have to make the decision: do I rely on in-memory data, or do I refetch the data?

Seriously: do you think you can rely on a flag on in-memory data, if two or more processes modify that in-memory data using totally different fetch strategies? (One fetches no related entities, the other does for example). No, you can't as the flag doesn't tell you what strategy was used. So you either take the stand and say: "the in-memory data is the central object graph and every process works on that graph and persists parts of it, so reflecting the state of the in-memory object graph reflects reality"

OR

you refetch the data from the db for that node.

With the flag, you can't make a solid decision: What if it is 'true'. Do you then not fetch the data but rely on in-memory data? Why? Because the set in-memory is the FULL set? How do you know that, based on a bool?

(edit). Also: what if there's a flag and you rely on it. It's set to true, so you assume there was a prefetch and you can rely on the state of the collection. Then the user alters the set again. As the flag is set, you don't get new data, and the tree will reflect outdated info.

Or do you want to reset the flag somewhere? But when? That's also context bound!

I agree with you about the 'generality' of the flag: In my sample scenario it would be useful but generally, although it would prove whether something was not prefetched, the other side of the coin is that it can't give further information about what was prefetched.

It is for this reason that I am not taking sides and saying it must/must not be included.

I also think there is a similarity here between null fields (in the situation where Nullable types are not used). If you have an Int32 property on an entity and query its value and it returns 0 - is that a real 0 or a default value returned by the framework? To be absolutely sure, you need to call TestOriginalFieldValueForNull(xxx) - an extra step.

There is a difference of course - in this case, once you take that extra step, you know once and for all whether it is a real or default number whereas for our hypothetical member collection flag scenario, you would only know whether something was prefetched or not. It is then context dependent, as you say, as to whether that incomplete knowledge is enough or not.

But, as things stand, the facility is not available at all. It was added, it might be used when it really shouldn't be (even if the reasons were documented); if it isn't added, then some users will feel aggrieved that information is seemingly being 'hidden' from them and have to work out alternate ways of including that functionality.

At the end of the day, a compromise might be to have a partial class template which comes with the framework (or is a 3rd-party add-on if you really, really object ) to provide the facility but is not switched on by default. That way, it is not part of the framework per se but where a developer really feels they need that facility and are happy with the limitations, they can get it without having to develop their own solutions such as adding bools to methods etc. Off the top of my head, I see no reason why it can't all be contained in the template - a BitArray32 with one bit for each child collection (so that subsequent lazy-instantiations do not affect the result), take a snapshot at the point of creation (not sure of the best point off the top of my head) and then a method (or maybe one per member collection) which just returns the bit for the collection in question.

Cheers Simon

it would prove whether something was not prefetched, the other side of the coin is that it can't give further information about what was prefetched.

And that is exactly what I want it for. To tell me if the prefetch request was or was not made, nothing more, nothing less.

arschr wrote:

it would prove whether something was not prefetched, the other side of the coin is that it can't give further information about what was prefetched.

And that is exactly what I want it for. To tell me if the prefetch request was or was not made, nothing more, nothing less.

So say it says 'true', so the call was made. When was that call made? You can't say. Also, what if the call was made but the filter used didn't match any entities, however you now want to see all related entities without a filter... you can't determine which filter was used based on the flag.

To answer these questions, you therefore have to KNOW what kind of prefetch path would be used otherwise and when. This alone is enough info to fill in if the data was fetched or not.