I know the linq support is a work in progress and I wouldn't want to pin you down about your specific implementation but I'm curious what high level overall architecture you would imagine for linq in the context of using the adapter model. I'm mainly concerned from an architectural standpoint in an attempt to avoid coding myself into a corner so that my stuff will play well.

Current architecture: -- Winforms client binds that binds directly to entities and entity collections -- Data loading is done through separate manager classes which wrap up common data load queries behind a method facade to avoid rewriting predicates over and over -- Manager classes in turn call off to a factory which determines if it should utilize a local or remote data access adapter and passes calls onto an interface which represents either a local daa or a proxy to a daa across a remoting boundary -- Another factory exists whereever the daa actually lives (local or remote) which determines if the daa type should be sql or oracle and actually performs the operation required.

What I'm not sure of is at which level the linq support is going to be implemented. In the common scenario now a specific entity or group of entities are loaded via a prefetch/set of filters which then figure out the tables required and the subset of data to retrieve. As linq selects can involve multiple joined tables and criteria on any given table in the set it seems like that would replace the normal fetch logic. Will this occur as a layer on top of the normal data access adapter, go around it, or perhaps generate the prefetch/filter logic for you so that you can in turn pass that to the daa?

Also, will the linq information be serializable such that the contents of the query/results can be passed across a remoting boundary?

Just trying to sorta get a grasp of what to expect, I know from your standpoint you don't want to nail yourself down to anything too specific but a vague outline of the future architecture would be very helpful in planning things out.

Thanks alot, many thanks to you in all the hard work you've put in on the product. It's really is a work of art in it's brilliance and it very clean in design.

GizmoTN76 wrote:

I know the linq support is a work in progress and I wouldn't want to pin you down about your specific implementation but I'm curious what high level overall architecture you would imagine for linq in the context of using the adapter model.

First of all: the Linq addition is just that: an add-on. This means that you can use Linq to work with LLBLGen Pro, but it's not a requirement. We keep on supporting our own API, because it's more flexible and has more features. Linq support has forced us to add more features to our own API as well (like derived tables and flexible creation of relations in code), but that's beneficial for the users of our own API as well, and the code is added to .NET 2.0/1.x versions of the runtime as well.

So in any case: if Linq forces you in a system which doesn't fit how you want to work, don't worry, you can always keep on using our own API and for example use Linq where you do see it as an advantage and leave it for other situations.

I'm mainly concerned from an architectural standpoint in an attempt to avoid coding myself into a corner so that my stuff will play well.

Current architecture: -- Winforms client binds that binds directly to entities and entity collections -- Data loading is done through separate manager classes which wrap up common data load queries behind a method facade to avoid rewriting predicates over and over -- Manager classes in turn call off to a factory which determines if it should utilize a local or remote data access adapter and passes calls onto an interface which represents either a local daa or a proxy to a daa across a remoting boundary -- Another factory exists whereever the daa actually lives (local or remote) which determines if the daa type should be sql or oracle and actually performs the operation required.

What I'm not sure of is at which level the linq support is going to be implemented. In the common scenario now a specific entity or group of entities are loaded via a prefetch/set of filters which then figure out the tables required and the subset of data to retrieve. As linq selects can involve multiple joined tables and criteria on any given table in the set it seems like that would replace the normal fetch logic. Will this occur as a layer on top of the normal data access adapter, go around it, or perhaps generate the prefetch/filter logic for you so that you can in turn pass that to the daa?

There are a couple of things about Linq which will be a mandatory issue for people if they decide to use Linq or not. We have and will try to work around these and offer options to our users to make using Linq less of a pain.

1) Linq queries embed the provider that will execute the query 2) Linq queries have no notion of hierarchical fetches 3) Linq queries are executed when they're enumerated.

To start with 1). With LLBLGen Pro, you specify the filters, relations etc. somewhere and execute it elsewhere using a data-access adapter of choice, setup for that particular fetch, or if you're using selfservicing, you want to fetch data inside an already running transaction for example.

With Linq this isn't the case. You create a new instance of the meta-data container, create a Linq query with it and with that, the provider which executes the query is EMBEDDED in the linq query.

With our system, this isn't necessary. You can set the Transaction or DataAccessAdapter inside the Linq query object later on. This is essential, as it gives the freedom to write the query in class X and execute the query in class Y using a given adapter or transaction defined in class Y so you don't need a reference to the adapter in class X.

Ad. 2). Linq to Sql has an attempt to do hierarchical fetches with LoadOptions. This is severily flawed. The main issue with it is that the hierarchy isn't defined with the Linq query, but with the object which executes the query. The main downside of that is that if you execute two queries in a row, you can't specify different hierarchies with it. Let's not debate the lameness of the hierarchy fetches of Linq to Sql anyway, it is a flawed implementation.

We'll offer a way to specify the prefetch path on a query using an extension method. This extension method is usable on the query definition. So this way you can specify the hierarchy with the query defined in class X and use it in class Y.

Ad. 3) This is a serious problem with Linq. It's not always the case that you want to enumerate. So we added our own interface to the query object that's created, and you can cast the created Linq query to that interface. that interface lets you also set the transaction/adapter as described above, and it also has an Execute() method. When called, the query is simply executed as if you requested the enumerator.

One advantage is that you get the data directly in an entitycollection for example. If you want to fetch the data into a list, the Linq query has a ToList() method. But that's really a copy. Our execute method doesn't copy.

Also, will the linq information be serializable such that the contents of the query/results can be passed across a remoting boundary?

No, as an expression tree isn't serializable. If you want to send filters etc. over the wire, use our own API, which is continued to be supported.

There are ways to serialize an expression tree, but that's with 3rd party software, and I don't know if it will serialize other data in the query. So in short: it's doubtful you'll be able to send the query across a wire. The Linq design also doesn't really offer this, because the provider which executes the query is IN the query. So serializing the query would suggest to serialize the provider as well.

Just trying to sorta get a grasp of what to expect, I know from your standpoint you don't want to nail yourself down to anything too specific but a vague outline of the future architecture would be very helpful in planning things out.

The progress of the Linq support is going well the last couple of weeks, finally. So we'll have a better look at what it will end up looking and what we can and can't support. It looks like we'll support a great deal of the Linq features after all. The things we won't support are for example Skip (but we have TakePage to page) and Union, because LLBLGen Pro doesn't support these features.

The main issue is still to find out if the code is able to handle all possible expression trees that will be thrown at it. It's a hard thing to do, as it's in general unclear what these trees look like, what's possible...

Thanks alot, many thanks to you in all the hard work you've put in on the product. It's really is a work of art in it's brilliance and it very clean in design.

Thanks!

Thanks for the heads up. I guess it wasn't exactly what I wanted to hear, but I understand the limitations you are working within. Just to clarify a couple items for my own sake it sounds like:

-- In order to utilize the linq capabilities all linq queries will need to be performed in the same appdomain as the data access adapters. IE if I want to deal with them in a remoting environment they'll all need to live on the remote server.

-- That there will be some mechanism for changing out the dataaccessadapter used such that I could utilize the same linq query to perform queries against multiple db types.

I guess my hope was that linq would be able to simplify/drop the verbosity involved in doing compond predicate expressions and prefetches as well as make it such that someone familiar with linq wouldn't need to be as familiar with the intricacies of llblgen predicate logic (not that anyone is an expert on linq right now either.) It sounds like some of that will be possible if I alter my architecture some to move all query manager calls on the remoting server instead of building up the query client side and passing that across. That's probably a better architecture anyway but it was easier for me to deal with the remoting at the lower level of a wrapper around the dataaccess adapter than to deal with remoting all the managers.

Just when I thought I was done with the templates they pull me back in .

Will the typedlists and typedviews also support linq or will it just be entities?

I wouldn't suppose you had considered doing an alternate non linq dsl to generate the predicate expression/prefetch logic in a way that would be serializable (and wouldn't cause jr programmers to run away screaming ). Right now the process for a complex query seems to be:

1. Write it out in sql first to figure out what you really want.
2. Try to figure out a way to reproduce that within llblgen.
3. Put that logic in a routine so you never have to figure it out again.

I'm definitely not criticizing, the methodology in place beats the alternatives hands down in code reuse, type safety, and multi database support. There are just times when I've found myself beating my head against the wall trying to figure out a complex query thinking "there's got to be an easier way." Something that is as intuitive as sql, with all the benefits your framework provides. Clearly that's something that's far easier said than done and all your work is greatly appreciated. Your tool is saving me a considerable amount of work on my current project and I am impressed with the amount of improvement I've seen in every release. (I've been watching llblgen for years but didn't get to utilize it for a production app until recently)

In any case I really appreciate the heads up, it should give me what I need to alter my path a little to best utilize what's coming down the road. Your company's support is top notch (far more responsive than any other company I've dealt with) and information like this will save me a lot of effort when you do ship 2.6.

Thanks again,

Scott

Sorry for the late reply.

GizmoTN76 wrote:

Thanks for the heads up. I guess it wasn't exactly what I wanted to hear, but I understand the limitations you are working within.

What did you want to hear?

Just to clarify a couple items for my own sake it sounds like:

-- In order to utilize the linq capabilities all linq queries will need to be performed in the same appdomain as the data access adapters. IE if I want to deal with them in a remoting environment they'll all need to live on the remote server.

correct, expression trees are created from the linq query. So: var q = from ... , then 'q' is an object which contains an expression tree. That object isn't serializable as expression trees aren't serializable. The main reason for this is that they could contain references to objects for variables. E.g.:

var q = from c in metaData.Customer where c.Country==myCountry select c;

here, 'myCountry' is a string variable for example. This can't be serialized as 'myCountry' is added to the tree as a reference.

As we also support our own API (and will continue to do so), you aren't left in the cold: simply use our own API in remoting scenario's.

-- That there will be some mechanism for changing out the dataaccessadapter used such that I could utilize the same linq query to perform queries against multiple db types.

Correct

I guess my hope was that linq would be able to simplify/drop the verbosity involved in doing compond predicate expressions and prefetches as well as make it such that someone familiar with linq wouldn't need to be as familiar with the intricacies of llblgen predicate logic (not that anyone is an expert on linq right now either.) It sounds like some of that will be possible if I alter my architecture some to move all query manager calls on the remoting server instead of building up the query client side and passing that across. That's probably a better architecture anyway but it was easier for me to deal with the remoting at the lower level of a wrapper around the dataaccess adapter than to deal with remoting all the managers.

Trust me, a LOT of stuff is very very complicated in Linq. The silly simple stuff is indeed straight forward. However, when it becomes more complex, it's easier to write LLBLGen Pro query elements than a linq query.

Just when I thought I was done with the templates they pull me back in . Will the typedlists and typedviews also support linq or will it just be entities?

For now, just entities. They can be added at a later date though.

I wouldn't suppose you had considered doing an alternate non linq dsl to generate the predicate expression/prefetch logic in a way that would be serializable (and wouldn't cause jr programmers to run away screaming ). Right now the process for a complex query seems to be:

1. Write it out in sql first to figure out what you really want.
2. Try to figure out a way to reproduce that within llblgen.
3. Put that logic in a routine so you never have to figure it out again.

You shouldn't take step 1. You should think in sets and what you want to obtain from that set. So if you want to obtain a set of orders, based on a filter on 2 related entities and fetch their related customers as well, you should use that information and directly put every element into llblgen pro objects, not first in SQL.

what you want to fetch -> the entity collection type/entity what you want to fetch with it -> prefetch path filter on related entities -> add relations to related entities. filter -> add predicates to predicate expression.

In Linq it's harder, as linq isn't set based at all. So converting from SQL to Linq is even harder, as you don't have elements which are close to SQL equivalents like in LLBLGen Pro, you have to make a true conversion. I often scratch my head when I want to write a complex query in linq to test something out. I assume it won't be different for others as well.

I'm definitely not criticizing, the methodology in place beats the alternatives hands down in code reuse, type safety, and multi database support. There are just times when I've found myself beating my head against the wall trying to figure out a complex query thinking "there's got to be an easier way." Something that is as intuitive as sql, with all the benefits your framework provides. Clearly that's something that's far easier said than done and all your work is greatly appreciated. Your tool is saving me a considerable amount of work on my current project and I am impressed with the amount of improvement I've seen in every release. (I've been watching llblgen for years but didn't get to utilize it for a production app until recently)

It does have a learning curve, but it's not THAT steep SQL is also complex when you start with it: how many times have you seen cursor based queries which were written by people who clearly have no understanding of the concept 'set oriented language'? .