LINQ Provider Exploration

Just a few more thoughts on LINQ providers and how they work.

If you look at the IQueryable and IQueryProvider interface, you'll see that it presents a very generic specification while the implementation expects much more specific output.

For example, the property IQueryable.Expression can't just be any expression, it has to be of a specific type - in particular, the output of resolving the expression has to be of type IQueryable (I think). If you are implementing a new class for this you might set the default value to Expression.Constant(this), with this in this context being the IQueryable object. This expression basically says, "return all of the items associated with me".

The expression itself is the internal representation of a query and it is built out of the LINQ code for you. For example, by default this (silly sample) code:

var query = from x in lq

    where x == "4"

    select x.Clone();

resolves into an expression that looks like this:

{value(TestLinqProvider.LinqQuery`1[System.String]).Where(x => (x = "4")).Select(x => x.Clone())}

Which is basically a function that looks like this (the variable lq above is of the type TestLinqProvider.LinqQuery):

lq.Where(x => (x = "4")).Select(x => x.Clone())

That final expression isn't built all at once, however. In my tests it appears that it is built one piece at a time, using IQueryProvider.CreateQuery to chain the results. In the above example, I have a public constructor for my LinqQuery object (lq above) that sets its default Expression value to Expression.Constant(this). Subsequent query statements will each in turn filter or transform this default object. The call to IQueryProvider.CreateQuery is done twice; the first expression value is just the default constant:

lq

Which is then added to the "where" clause of the LINQ statement by the first call to CreateQuery:

lq.Where(x => (x = "4"))

Which then is finished by the last call to CreateQuery:

lq.Where(x => (x = "4")).Select(x => x.Clone())

And that is the actual expression for the LinqQuery object that will be executed. If you don't take the expression value passed in to CreateQuery and use it in your returned query object, the expression tree won't be built. Once again, testing has also indicated that the returned expression in query objects associated with CreateQuery has to be resolvable into the appropriate IQueryable type.

Most of the query providers out there don't actually do anything in the CreateQuery call but take the new appended query statement and attach it to a freshly created query object. The query provider is then left with the task of developing a completely new query from scratch in the IQueryProvider.Execute method. The query object becomes just a holder for an expression in this scenario.

However, it is also possible to modify the expression on each CreateQuery statement to incorporate more data. For example, you may want to pre-parse statements to separate functions you might resolve client-side from those you might want to resolve server-side in a SQL statement.

The basic result is that most expression trees created legitimately via LINQ would wind up having a fairly specific format; however, to conform to the interface, you are pretty much stuck resolving any combination of tree elements that come up. Some query providers might deal with unexpected structures by executing any questionable expressions client-side; for example, instead of placing a "where" clause at the SQL server where it would be optimal, one could download the entire table and let the default IQueryable.Where method do the work client-side. New query providers can probably start with implementing the obvious methods as they are built by the default LINQ implementation and then figure out how to deal with variety later.