I have been reading about LINQ2LLBL. When will it be available?

Thanks Craig

Very soon. I hope to finish today or tomorrow the hierarchical fetches, and then some small things are left which won't take a lot of time, so I hope this week. Though I haven't ran any selfservicing test, (although everything for selfservicing is build in already), so if those break it might be a bit later, but not a lot later. (98% of the code is generic, so don't worry if selfservicing took a backseat, it didn't/doesn't)

We'll release a CTP, so v2.6 in full will be released in a month or so. The CTP contains the linq assembly, updated ormsupportclasses, updated DQE's and a set of templates.

Hierarchical fetches are things like: var q = from c in customers select new { c.CompanyName, Orders = c.Orders };

so a nested query inside the projection. If everything goes to plan this will be very very fast and efficient (1 query per nested query ) Prefetch paths already work.

Thanks for the quick reply

Hi Otis

Do you have any sample projects that we can download yet (or just code snippets on a web page somewhere)?

Thanks Craig

The linq code itself isn't available so no examples for Linq either. We'll ship our unittests for linq with the CTP so you have a lot of queries to work with

You can check out the articles on my blog (url in my sig) to see examples of queries

Hi Frans,

Question....so v2.6 will support a generic LINQ query provider ? DevExpress quote for their new 2008v1 suite :

XtraGrid Suite and XtraEditors Library LINQ Server Mode When the grid control works in server mode, it delegates all data processing to the server and downloads only records to be dislayed on screen. This allows you to dramatically increase performance against large datasets. Previously, this data operation mode could be enabled only with XPO data sources. Now, any LINQ query provider is also supported.

grtz, Danny

DvK wrote:

Hi Frans,

Question....so v2.6 will support a generic LINQ query provider ?

It comes with a Linq provider for LLBLGen Pro, so you can write linq queries to fetch data through llblgen pro

DevExpress quote for their new 2008v1 suite :

XtraGrid Suite and XtraEditors Library LINQ Server Mode When the grid control works in server mode, it delegates all data processing to the server and downloads only records to be dislayed on screen. This allows you to dramatically increase performance against large datasets. Previously, this data operation mode could be enabled only with XPO data sources. Now, any LINQ query provider is also supported.

grtz, Danny

I'm not sure how they're going to do that, but if they're spitting out an expression tree, then it should work, but I have my doubts.

If they've implemented the same way as they've done with their 'XPO Linq provider', forget it. Their XPO linq provider is not really useful in any way beyond very basic queries. Our compiled provider is 170KB of code (just the provider), theirs just 26KB.

But we'll see. I also dont know if it works with the LinqDataSource of ASP.NET, I haven't done any testing with that as well.

Currently profiling the code. My hierarchical fetches seem to be slower than expected, so I have to do some refactoring there.

It's weird though... the first linq expression tree evaluation is slow (40-100ms) as it has to init a lot of types apparently, however any subsequential expression tree evaluation is fast (1ms).

I tried something different with the hierarchical fetches of random sets with nested sets in the projection

(example:

var q = from c in metaData.Customer
        select new
        {
            c.CustomerId,
            Orders = c.Orders,
            c.CompanyName
        };

i.e.: use a compiled lambda function to compare parent with child, but that turned out to be slow: hashes are the way (which are used in prefetch paths).

Fetching the above query takes 500ms, using prefetch paths with linq takes 71ms. (debug code). So I've to rework the parent-find-child routine with hashes instead of my lambda func. Oh well.

The prefetch path code in Linq to LLBLGen Pro has Linq to Sql for breakfast btw Our query:

var q3 = (from c in metaData.Customer select c).WithPath(
                new PathEdge<OrderEntity>(CustomerEntity.PrefetchPathOrders,
                        new PathEdge<OrderDetailEntity>(OrderEntity.PrefetchPathOrderDetails)));

(fetch customers, orders and order details)

takes 191 ms to complete (debug code).

Linq to sql with loadoptions:

DataLoadOptions loadOptions = new DataLoadOptions();
loadOptions.LoadWith<Customer>(c => c.Orders);
loadOptions.LoadWith<Order>(o => o.Order_Details);
nw.LoadOptions = loadOptions;
var q = from c in nw.Customers select c;

takes 1081ms.

And this difference only gets bigger with bigger sets, as LLBLGen Pro executes 3 queries, and linq to sql a truckload of queries .

OK, so you did some background checking on other providers.... I asked the DevEx guys about their provider, how this works and how it is implemented, i.o.w. how to put to work a 3rd party provider.

I'll let you know....

grtz, Danny

DvK wrote:

OK, so you did some background checking on other providers....

Not as in "how did they do this?", but more as in: is this a full provider or not. For example the devexpress provider is far from complete, similar to the nhibernate linq provider which also lacks a lot of features at the moment. I haven't checked others though.

I asked the DevEx guys about their provider, how this works and how it is implemented, i.o.w. how to put to work a 3rd party provider. I'll let you know.... grtz, Danny

Thanks, would be good info

Heheh

Linq to sql:

public static void FetchNestedSetTest()
{
    NorthwindDataContext nw = new NorthwindDataContext();
    var q = from c in nw.Customers
            select new
            {
                c.CustomerID,
                Orders = c.Orders,
                c.CompanyName
            };

    foreach(var v in q)
    {
    }
}

called 100 times: total time: 11545ms. On average this took per call: 115ms

LLBLGen Pro:

public static void FetchNestedSetTest()
{
    using(DataAccessAdapter adapter = new DataAccessAdapter())
    {
        LinqMetaData metaData = new LinqMetaData(adapter);
        var q = from c in metaData.Customer
                select new
                {
                    c.CustomerId,
                    Orders = c.Orders,
                    c.CompanyName
                };

        foreach(var v in q)
        {
        }
    }
}

called 100 times: total time: 2770ms. On average this took per call: 27ms (so 27 ms for this whole routine, adapter creation, metadata creation (which is nothing ) query parsing and fetching.

(on core 2 quad with networked sql2000 db). I can't believe how big the difference is. LLBLGen Pro executes 2 queries per call, linq to sql executes 92 queries per call (1 for the customers, and per customer 1 query for the orders).

Looking VERY good

And, how many programmers have been working on this Linq to sql thing ?

DvK wrote:

And, how many programmers have been working on this Linq to sql thing ?

the whole team was 25 people or so.

And then this is their result of lots of hours of brain-usage, programming and sweat and tears.....?

DvK wrote:

And then this is their result of lots of hours of brain-usage, programming and sweat and tears.....?

Yup And not for 1 year, but 4. Take into account that most of them are managers

Anyway, to give them a bit credit, their object materializer is very good. This linq to sql query runs in average 6ms:

var q = from o in nw.Orders
        select new
        {
            o.OrderID,
            CompanyName = (from c in nw.Customers where c.CustomerID == o.CustomerID select c.CompanyName ).Single()
        };

(LLBLGen Pro takes 16ms there. )

however, when I change it to use an anonymous type:

var q = from o in nw.Orders
        select new
        {
            o.OrderID,
            CompanyName = (from c in nw.Customers where c.CustomerID == o.CustomerID select new { c.CompanyName } ).Single()
        };

it can't fold the nested query into the projection as a scalar query anymore, so it will execute for every order row (818 ) a query, so this one takes 656ms.

(LLBLGen Pro takes 19ms there. So the nested set merging just takes 3ms).

I can gain some speed with a custom type projector which uses a lambda to instantiate instances instead of Activator.CreateInstance, but that will be a couple of ms.

I now also can pass a nested set to an in-memory method call, or whatever complex piece of code should be executed on the client ->

[Test]
public void InMemoryMethodCallWithNestedSetResultAsParameter()
{
    using(DataAccessAdapter adapter = new DataAccessAdapter())
    {
        LinqMetaData metaData = new LinqMetaData(adapter);
        var ordersFromGermany = from o in metaData.Order
                                where o.Customer.Country=="Germany"
                                select o.OrderId;

        List<int> orderIds = ordersFromGermany.ToList();
        Assert.AreEqual(121, orderIds.Count);

        var q = from o in metaData.Order
                select new  {
                        o.OrderId,
                        Value = InMemoryMethodCallsForTests.IsUsefulEntity(o.Customer)
                    };

        int count = 0;
        foreach(var v in q)
        {
            count++;
            if(v.Value)
            {
                Assert.IsTrue(orderIds.Contains(v.OrderId));
            }
            else
            {
                Assert.IsFalse(orderIds.Contains(v.OrderId));
            }
        }
    }
}

o.Customer is a nested query and every result is passed to the method. This is transparent to the fetcher / set merger I use a 2-pass projector for this: first the rows are fetched using a normal projector which produces object[] rows. At the spot of the nested set, a dummy value is placed. Then I fetch every nested set, and per set I materialize them (so if they're parents, they first fetch their children etc. using recursion), and then I find the parent rows for the children. I place the groups of child results in each object[] row of the parent belonging to the children and after that I project the object[] rows as if they're coming from the db! So everything fancy done in-memory etc. works as if everything was fetched from the db

Btw, entity framework team I've heard is over 100 people. Also already for several years.

I have been reading about LINQ2SQL and contrasting it with some of the glimpses FRANS is leaking from LLBL2LINQ. One area of interest is LINQ2SQL's DataContext object versus LINQ2LLBL's MetaData object.

Of the things the DataContext takes care of is full change tracking and concurrency management. It makes handling concurrency exceptions easy by throwing the (ChangeConflictException) in which we can use code like this

catch (ChangeConflictException)
{
    var exceptionDetail =
        from conflict in context.ChangeConflicts 
        from member in conflict.MemberConflicts
        select new
        {
                   TableName = context.GetTableName(conflict.Object), 
                   MemberName = member.Member.Name,
                   CurrentValue = member.CurrentValue.ToString(),  
                   DatabaseValue = member.DatabaseValue.ToString(),
                   OriginalValue = member.OriginalValue.ToString() 
        };
    ObjectDumper.Write(exceptionDetail);
}

Another nicety is the (Query Visualizer) in VS2008 that can be used to show the full SQL statement that will be issued. Can LINQ2LLBL have a similar visualizer (I know this will make alot of people on the forum happy as a lot of threads asked for an easy way to inspect the issued SQL statement)?

omar wrote:

I have been reading about LINQ2SQL and contrasting it with some of the glimpses FRANS is leaking from LLBL2LINQ. One area of interest is LINQ2SQL's DataContext object versus LINQ2LLBL's MetaData object. Of the things the DataContext takes care of is full change tracking and concurrency management.

LinqMetaData is there to be able to create a query, so it produces DataSource<T> objects, which are more or less the container for the provider, but that's it.

What's a big disadvantage of the datacontext is that it has central change tracking. LLBLGen Pro entities do the change tracking themselves. The advantage of doing it inside the entities is that wherever you pass the entity to, you're able to do change tracking. So if I have a tier with a couple of routines which do data-access for me, I can fetch entities there, receive them from those methods, use them in whatever code I have, pass them over a wire etc. change them, pass them back and without any effort I can persist the changes.

With the datacontext: if you don't keep it around (and in webscenario's that's often the case) you have to babysit the change tracking: you have to re-attach an entity to a datacontext when you want to persist it. This is a big pain in practise as you have to keep track of which entities are new, what their original data was etc. otherwise the context doesn't know which fields changed, if the entity is new or has a couple of changed fields etc.

This has lead to requests for a serializable datacontext or lightweight datacontext for linq to sql (entity framework has the same problem, Danny Simmons has now a petproject for this problem) to carry around so people don't have to babysit the changetracking, however MS has promissed but not yet delivered such an object.

So be careful what you wish for, this aspect of the datacontext is really really bad in practise.

It makes handling concurrency exceptions easy by throwing the (ChangeConflictException) in which we can use code like this

catch (ChangeConflictException)
{
    var exceptionDetail =
        from conflict in context.ChangeConflicts 
        from member in conflict.MemberConflicts
        select new
        {
                   TableName = context.GetTableName(conflict.Object), 
                   MemberName = member.Member.Name,
                   CurrentValue = member.CurrentValue.ToString(),  
                   DatabaseValue = member.DatabaseValue.ToString(),
                   OriginalValue = member.OriginalValue.ToString() 
        };
    ObjectDumper.Write(exceptionDetail);
}

You can control concurrency as you wish in LLBLGen Pro as well with the same granuality and without the mess of re-attaching a souped up entity to a context and telling it that it is REALLY a new / existing entity etc. (Linq to sql only offers a small set of concurrency options) You can generate concurrency filters on the fly by concurrencyfactories you inject automatically into entities. As the changetracking is INSIDE the entity, everything you need is available to you.

Linq to Sql can only throw these exceptions due to failure of update queries or failure of delete queries, which is what llblgen pro does too: when these queries fail, you get an ormconcurrencyexception. Enclosed in that exception, the entity persisted / deleted is enclosed.

But what exactly triggered the failure? Now that's a question linq to sql can't answer for you either. Say I update 4 fields in my entity and save it. 3 of those fields trigger a 'original value has changed in the db' kind of failure. WHich 3? The update is 1 statement with 1 where clause which tests all 4 fields. So are you sure you get only the 3 fields in the conflict exception? Or all 4?

Anyway, you have the same info available to you in the entity in llblgen pro as well: which fields were changed and what was their original db value. This without unnecessary re-attaching overhead and other babysit code: it's transparent for you.

Concurrency is something which is often done in a 'I'll check when I have to save' kind of way, but in most cases that leads to loss of work, so data merging or locking of features is the way to go.

Another nicety is the (Query Visualizer) in VS2008 that can be used to show the full SQL statement that will be issued. Can LINQ2LLBL have a similar visualizer (I know this will make alot of people on the forum happy as a lot of threads asked for an easy way to inspect the issued SQL statement)?

You mean, in pseudo sql like in the other visualizers? It's possible we'll add such a visualizer. The main problem is with these queries: var q = (from c in metaData.Customer where c.CustomerID=="ALFKI" select c.CompanyName).Single();

this isn't a deferred executed query, this query is executed immediately, due to a big design flaw in how Linq works (it's a combination of results AND query specification, not just a query specification). So 'q' isn't a query here, it's a string. So how to visualize this?

I use a special method into the query (Which I typed with ILLBLGenProQuery) which allows me to evaluate the query, not execute it. It stores every expression tree along the way, so I can visualize them in a nice tree. THe problem is that with these queries like the one above, this isn't possible, as q isn't a query, it's a string value. So nowhere to tap into the system... I then have to break into the provider itself to visualize the expression tree. It's a pain.

It's thus better if someone has a way to write the linq query in a tool and execute it. LinqPad comes to mind but that's tied to linq to sql.

Btw I just tried a Linq to sql query to show up in a visualizer, but all it did was produce a string as the 'ToString' for the provider, there wasn't any visualizer. Do you mean a separate debugger visualizer?

Btw2: there's another thing: with nested queries, like the ones I posted, you only see the top query, as there are more than 1 query to execute during the execution. So in the end, tracing is what you need to see what really went to the db.

So what's the real thing people want: 1) fiddle with linq queries till they're the ones required, so messing with statements without hurting the application or requiring to setup a test environment. If so, this requires a separate tool like linqpad though linqpad doesn't work with llblgen pro's Linq (it's hardwired to linq to sql). or 2) see what queries really went to the db? If so, this is only possible with tracing.

About the MemberConflicts: I checked the docs and it seems they indeed fetch the original data back (I haven't tested) as it has 'IsResolved' which suggests it can determine which field is in conflict. This is abit weird. The thing is: if an UPDATE statement fails because its where clause fails, you've to fetch the row to determine which fields are actually having a conflict. Now, this is OK till the lock is lifted. THe lock is lifted when the transaction ends. And here is the problem: the transaction has to be kept open if the information which fields have a conflict is presented to a user for example. This breaks the ACID principle. So the transactio is rolled back, information is shown, however by that time the conflict information can be outdated.

It's speculation, because I haven't checked if they fetch the row for field conflict checking or not.

A couple of linq related questions...

Is there any reason why you're not using lambdas for prefetch paths? IMO it would be quite nice to be able to write something like:

query.WithPath(
    new PathEdge<CustomerEntity>(c => c.Orders, new PathEdge<OrderEntity>(o => o.OrderDetail))
); 

I was also interested in this statement from your latest blog post:

The sort expression is a different matter: sorting is specified in Linq using 'orderby', but that's only accepted at a specific place, namely inside a query, but not at every spot. So we've to fall back onto normal LLBLGen Pro sort expressions for that...

Would you be able to elaborate on this? Would it not be possible to do something like:

query.WithPath(
    new PathEdge<CustomerEntity>(
        c => c.Orders.OrderBy(o => o.OrderDate)
    ) 
);

Thanks for the excellent series of blog posts btw...I've really enjoyed them and learned a lot in the process!

Jeremy

Jez wrote:

A couple of linq related questions...

Is there any reason why you're not using lambdas for prefetch paths? IMO it would be quite nice to be able to write something like:

query.WithPath(
    new PathEdge<CustomerEntity>(c => c.Orders, new PathEdge<OrderEntity>(o => o.OrderDetail))
); 

That was indeed the first design, but it had a couple of problems: 1) you need 2 types in the PathEdge ctor: new PathEdge<CustomerEntity, OrderEntity>, as the first is used for the element to fetch, and the second is used for the filter on the element: new PathEdge<...>(..., o=>o.EmployeeId==3, ...)

I found it too verbose. the second reason which was actually the showstopper was that the 'c.Orders' makes it actually pretty hard to obtain the actual prefetch path information from the entity itself. It required dirty reflection and as there's already code available, namely the prefetch path properties themselves which people are used to, I opted for that instead. it was a bit of a surprise that it didn't work to be honest. Everything compiled fine, I ran the test and it broke down at an awkward point and it then became clear it was problematic.

I was also interested in this statement from your latest blog post:

The sort expression is a different matter: sorting is specified in Linq using 'orderby', but that's only accepted at a specific place, namely inside a query, but not at every spot. So we've to fall back onto normal LLBLGen Pro sort expressions for that...

Would you be able to elaborate on this? Would it not be possible to do something like:

query.WithPath(
    new PathEdge<CustomerEntity>(
        c => c.Orders.OrderBy(o => o.OrderDate)
    ) 
);

Hmmm, possibly. It has the same problem with the info gathering for 'Orders', but that aside, what you suggest is simply that if you want to specify a sorting you should do it by adding an extension method call. though what if you also want to filter? Would it then become c=>c.Orders.Where(o=>o.EmployeeId>4).OrderBy(o=>o.OrderDate) ?

I'm not sure I like this over what's in the framework now.

Thanks for the excellent series of blog posts btw...I've really enjoyed them and learned a lot in the process!

Jeremy

I quickly fixed a flaw in my post though I forgot to filter on country in the 1st linq to sql query so it wasnt THAT slow after all (still 10 times slower compared to my code, but not 900ms )

Otis wrote:

Would it then become c=>c.Orders.Where(o=>o.EmployeeId>4).OrderBy(o=>o.OrderDate) ?

Yes.

Otis wrote:

I'm not sure I like this over what's in the framework now.

Its a matter of preference I guess. Personally, I like the lambda based approach.

Otis wrote:

I found it too verbose.

I actually think this is less verbose:

new PathEdge<CustomerEntity, OrderEntity>(
    c => c.Orders.Where(o => o.EmployeeId == 4).OrderByDescending(o => o.OrderDate),
    new PathEdge<OrderEntity, OrderDetailEntity>(
        // etc
    )
);

than this:

new PathEdge<OrderEntity>(
    CustomerEntity.PrefetchPathOrders,
    o => o. EmployeeId == 4,
    new SortExpression(OrderFields.OrderDate | SortOperator.Descending),
    new PathEdge<OrderDetailEntity>(
        //etc
    )
);

In the first approach, everything seems consistent. Mixing and matching lambdas with LLBLGen SortExpression/PrefetchPaths seems...odd.

Taking it further, this would be nice too (this is just off the top of my head...haven't had the opportunity to think through any implications)

query.WithPath<CustomerEntity>(c => c.Orders.Where(o => o.EmployeeId == 4).OrderBy(o => OrderDate).WithPath<OrderEntity>(o => o.OrderDetail);

I think method chaining in conjunction with lambdas is really quite elegant.

(continued)

Otis wrote:

...the second reason which was actually the showstopper was that the 'c.Orders' makes it actually pretty hard to obtain the actual prefetch path information from the entity itself. It required dirty reflection...

Could you elaborate on this? I'd actually be quite interested to see how you were planning on using reflection to get the prefetchpath information.

Jez wrote:

(continued)

Otis wrote:

...the second reason which was actually the showstopper was that the 'c.Orders' makes it actually pretty hard to obtain the actual prefetch path information from the entity itself. It required dirty reflection...

Could you elaborate on this? I'd actually be quite interested to see how you were planning on using reflection to get the prefetchpath information.

I looked it up after posting and it was more different. The c.Orders is changed into an EntityExpression object of type 'Order', with the correlation relation Customer - Order. The actual member access is lost, as it's not important.

I could add that again and then pull it out of it when PathEdge is constructed, using the name to do a lookup for the prefetch path on teh entity.

There's another issue though. Prefetch paths really have to be on 1 node. So from order, I can't have o.Employee.Customers, that's 2 hops away, the prefetch path wouldn't work that way.

About the chaining of methods: It's elegant, though it's hard to push the developer into one direction, so the developer could go overboard the problem then is: how to pull apart the query the developer has specified and re-build it as a prefetch path node?

That's a bit of a problem. I can call any extension method on c.Orders. What to do with those at that spot? It might even make sense, so the developer will expect them to work...

I do agree though that mixing them with llblgen pro elements isn't the most elegant choice. I didn't want to use that approach, but the downsides are a bit big for the other approach.

I think method chaining in conjunction with lambdas is really quite elegant.

Also please consider us poor VB'ers as VB.NET2008 does not support Lambda statements (it only supports Lambda expressions)

omar wrote:

I think method chaining in conjunction with lambdas is really quite elegant.

Also please consider us poor VB'ers as VB.NET2008 does not support Lambda statements (it only supports Lambda expressions)

Hmm, that could be a problem indeed (but I keep forgetting what VB.NET doesn't have, so bare with me here ).

Say I want to define a prefetch path: Customer - Order, and I want to filter the orders on EmployeeId==3. Can I then specify a filter like: o=>o.EmployeeId==3 in VB? I think that's possible in VB, something with Function...

So, I'm a bit puzzled about which construct with a lambda you refer to which isn't supported, a quick google search didn't give any clear info. Could you give an example?

During the CTP period, people can give their feedback, about things which work and don't work, what they'd like to see changed etc. So if some construct requires a lot of work in VB and with a different construct it's much better, we'll obviously look into changing that. One thing I can say we won't do is to support 'Skip': we use our own 'TakePage' extension method to page through data.