We use DevExpress as GUI library and we noticed that after an entity update (refetch after writing data to database) it takes to long, more then 3 seconds, the GridView is accepting the changes.

My question is what happens after an entity, in a collection, is changed? Are there events fired?

It seems that a prefetchpath to a table with 250.000 items makes the grid looks like very slow! After removing the prefetchpath the grid is buildup fast. But I need the items! Is there a way to make the entity collection faster after one entity is changed?

The situation is as following: - Table ControlModels has a relation with table ControlModulesAC; - Table ControlModules has 15.000 entities and ControlModulesAC up to 50 x 15.000; - Values in table ControlModulesAC are needed for entities in ControlModules.

What happens according ORM profiler with ControlModelsEntity.PrefetchpathControlModulesAC: - EntityCollection of ControlModelsEntity is fetched; - EntityCollection of ControlModulesAC: is fetched; - In memory both collections are processed ?!

Problem: - A ControlModelsEntity is changed and written to the database and values are refetched; - ORM framework is busy for 3-4 seconds (post processing); - ORM profiler doesn't show the time that is needed for client side processing.

Question: - is there a possibility to re-fetch entity data without the post processing? - what does recursive save in this case, is it enough to put recursive on False?

Do you have 15K entities in memory and bound to the grid? Or do you have 250K entities bound to the grid? Also what version are you using? Fetching that many rows is fast (as fast as it can be) but still can take some time. You might try to page through entities instead... (no user is going to look at 250K rows )

In any case, you'll get an IListChanged event at one point when an entity is changed and persisted.

We are using version 5.1.3

I have 15.000 entries bound to the grid and max 50 (per entity) "sub" items that are used for unbound column values (dynamic column extension of the grid)

We need all 15.000 items in the grid and that seems not to be a problem!

The problem is that after re-fetch of one of the 15k entities LLBLGen needs 3-4 seconds to do something ?!

Do you want the refetch in the first place? i.e are there any fields updated at the database level?

Btw, your Graph has 765K Entities, and that's huge to keep in memory. One way to optimize this if you don't want to use paging nor a read only list of fields, is to exclude fields you don't need form the entities you fetch and the prefetchPaths as well.

Use the overloads of the fetch method and the PrefetchPath.Add method that accepts excludedIncludedFields

Meindert wrote:

We are using version 5.1.3

I have 15.000 entries bound to the grid and max 50 (per entity) "sub" items that are used for unbound column values (dynamic column extension of the grid)

We need all 15.000 items in the grid and that seems not to be a problem!

It is, you just don't see it. The grid pages through the data in memory, but if something changes, it e.g. has to resort things in-memory.

The problem is that after re-fetch of one of the 15k entities LLBLGen needs 3-4 seconds to do something ?!

Please use a profiler (a .net profiler) and check what's going on. If you save a graph with 15K entities and just 1 is dirty, it will traverse all of them, as it doesn't know which one is dirty. Then it saves that one, raises the ListChanged event and the collection's EntityView object will re-index based on that, if you have set the view to update when changes occur.

Like I said, no user is going to look at 15K entities in memory, you have to page through them, it's a massive amount.

You can switch off refetching btw: https://www.llblgen.com/Documentation/5.2/ReferenceManuals/LLBLGenProRTF/html/18E1127C.htm

Thanks Walaa and Otis for answering. To be correct/complete I have made a diagram of the situation. I like to know how LLBLGen works so I have some questions based on your answers. At the moment I have not the capability to profile but please try to explain me what happens in the ORM layer.

Situation (see also diagram): - ControlModules are bound to the datagrid/treelist (DevExpress); - ControlModuleAC are prefetched on ControlModule; - One particular ControlModule is saved and re-fetched NOT the whole collection is saved; - Wait 3-4 seconds before Grid/Tree becomes active again.

Why I think LLBGen is my concern and not the grid/tree: - When I remove the prefetchpath to ControlModuleAC the grid reacts instantly. This means that the IListChanged is not a concern regarding the datagrid/treelist update!

Questions: - If I change one entity and write/re-fetch this entity then the ORM layer is processing 3-4 seconds (No profiler outcome but test scenario made that clear to me); - What happens if one entity is re-fetched and recursive is true; - What happens not if one entity value is written to the database and re-fetch is false (besides out of sync).

Please attach a picture as a picture, not as a pdf. We don't open pdfs.

With performance problems, there's actually nothing else to do but first to profile what's going on. We can debate all day what is likely happening, but it might not be the bottleneck. What I can tell you is that loading a massive amount of entities into a UI and binding these to controls could be slowing down by any number of reasons, which are unknown until it's profiled (e.g. is it in the entity view's indexing, or is it in the grid which tries to resort all rows?).

Meindert wrote:

At the moment I have not the capability to profile but please try to explain me what happens in the ORM layer.

Situation (see also diagram): - ControlModules are bound to the datagrid/treelist (DevExpress); - ControlModuleAC are prefetched on ControlModule;

So we're talking 750.000+ entities live in memory ? that's an insane amount, sorry.

• One particular ControlModule is saved and re-fetched NOT the whole collection is saved;
• Wait 3-4 seconds before Grid/Tree becomes active again.

...which can be anything. To give you an idea: when you change in the designer a field mapping of a field, if we wouldn't do anything, it would become a slow experience where you'd have to wait for 2-3 seconds, even though the property change is instantly. Weird, right?

The main issue was that because of a change, containing objects notified observers that they were changed, and containing objects of those containing objects did the same, which ended up a LOT of objects notifying observers something was changed and they better repaint/update themselves. 99.9% of this work is redundant: even if a node in the tree would need to be updated, it would so just once, not a lot of times. So after profiling everything we saw the vast majority of the performance was in event handlers doing the same thing over and over again. So we added a simple event throttler object between event and control, which only re-raised the events further every x ms, and filtering out duplicates. Everything is then fast and smooth. There were many of these 'event floods', which start simple and logical, and due to multiple paths through the object graph, can bubble upwards to a lot of events which would all result in 'hey, repaint the grid!', which of course will slow down things tremendously.

That's why profiling is essential for all these kind of discussions, full stop. Guesswork just wastes anyone's time, trust me

Why I think LLBGen is my concern and not the grid/tree: - When I remove the prefetchpath to ControlModuleAC the grid reacts instantly. This means that the IListChanged is not a concern regarding the datagrid/treelist update!

How do you know? You cut the # of entities by a massive number too doing that, that might be the problem too, who knows?

Questions: - If I change one entity and write/re-fetch this entity then the ORM layer is processing 3-4 seconds (No profiler outcome but test scenario made that clear to me);

what does 'the orm layer is processing' mean, if I may ask, and how did you measure this? with a stopwatch call around SaveEntity() or unit of work commit?

• What happens if one entity is re-fetched and recursive is true;

If recursive is true, all entities reachable through that entity are examined whether they're 'dirty' or have dirty related entities which can sync FK values with them (so they become dirty). This is done using a fast graph traversal algorithm, but as you can imagine, if every entity is reachable in your 750.000 entities in-memory it will still take a while. After that's done, all entities found to be persisted are saved with an insert/update query. After each entity has been saved, and refetch isn't switched off (see my post earlier), it's refetched and set as Fetched. I checked the codebase whether it signals any events, and it just raises AfterSave, which isn't bound by our own code afaik.

• What happens not if one entity value is written to the database and re-fetch is false (besides out of sync).

then the entity graph is still traversed, if recurse is true (default in adapter), though after the insert/update query is ran, nothing is done further.

Seriously, I'm not advising you to do weird things, I'm dealing with performance oriented code for a long time, there are really no other ways to look at things than this, so please:

1) seriously cut down the # of entities loaded into a UI. Use paging instead. No human is able to look at 100's of rows, let alone 1000s. Load data on-demand, not all at once. If you need all rows in your database in-memory for aggregates, calculate them in the DB. The runtime isn't slow, but massive amounts of data in a UI will make UIs in general simply slow to work with unless you take serious measures. E.g. adding a new entity to a collection with 10000 entities will take more time than adding it to an empty collection, simply because it will check if the entity is already there. Profiling will show you that. This is caused by a feature which in general is 'great to have' but in large sets can be a burden, so it can be switched off. But until you know that's the cause, switching it off is just trying to switch off a light by trying every switch in the building. 2) profile, profile, profile. Only then you know where the bottleneck is and you can do something about it, if that's required. E.g. loading 750000 rows in a grid hierarchy will take more time than loading 30 rows in a grid. Profiling that will show you a 'bottleneck' which isn't really there, it's caused by a massive amount of data: it might even be 3-4 seconds is very fast compared to what you'd expect (I have no idea, just an example) with the amount of data at hand.

At Monday I wil start to use the profiler, and hereby the picture of the database structure!

Profile outcome

From the looks of it, it's 1200ms or thereabout? So that's not it. I also don't see any grid related calls.

But again, it traverses likely the complete graph, which spans likely all 750000 entities in memory as via whatever entity you pick, it will see the rest as they're reachable indirectly. With finding performance bottlenecks you have to first understand what's going on, then measure how long that takes, then reason about whether that's a reasonable time or not and if not take action. Calculating the total spanning graph and traverse 750000 or more entities in memory might take that long, it's a massive amount of entities. It's the same as traveling 200km by car, you won't get there in an hour in a normal road car, it takes time.

This should be easy to test btw: specify 'false' for recursive save, it then shouldn't take time to find the entire graph.

Because of time pressure, no hours left for this particular project, i made a own implementation of the prefetch of additional column values. This works well. If there is time/money in the future we will investigate this again. Thanks for your time.

Meindert wrote:

Because of time pressure, no hours left for this particular project, i made a own implementation of the prefetch of additional column values. This works well. If there is time/money in the future we will investigate this again. Thanks for your time.

what do you mean with ' i made a own implementation of the prefetch of additional column values' ? The profile you showed has no time spent in the prefetch... But you shouldn't fetch that much data, if the project is used later on with even more data (e.g. millions of rows), it will bog down again, simply because of that.

First of all: We need that much data and those additional column items in the grid !

DevExpress provides functionality with columns, not bounded to entity fields, and the data can be get/set via events!

The moment an event fires I have to find the correct value for the particular main entity and AD column. Initial plan was to find that value in the Main entity prefetched AC list, but that seems to be slow (customer feeling)!

Now I fetch the Additional column entities separate and create a fast collection to search for the value when I need it!