XAML Syntax Discovery

I know I've been promising to finish some LINQ provider stuff, but I have ADD...

If you're like me, you have had a bit of trouble debugging data binding in XAML. Typical syntax looks like this:

<TextBlock Text="{Binding ElementName=mySourceElement,Path=Text}">

... which, to me, looks like some kind of crazy nutjob specialized syntax specific to binding that has no real (Intelli)sense. For example, where can you use "{Binding...}"? What classes can you use this with?

Turns out I should read a syntax guide. The code block above is essentially equivalent to:

<TextBlock>
<TextBlock.Text>
<Binding Path="Text" ElementName="mySourceElement">
</TextBlock.Text>
</TextBlock>

I didn't understand that the above syntax was a fairly simple, standardized shortcut because I have never (until just now) seen an example of data binding that used the long form. Perhaps I should have done more to try to understand the underlying syntax of XAML, but this sample really opened my eyes and helped me diagnose problems. In particular, by avoiding the braces shortcut in the top syntax, you get much more effective Intellisense which has helped me a lot with my WPF data binding.

I could have known this had I done more boring reading. The full details are available in the MSDN library page called XAML Syntax Terminology. In particular, the Markup Extensions section explains why some classes, such as StaticResource and Binding, can use this syntax. For more information on how markup extensions work, check out Markup Extensions and XAML on MSDN.

More LINQ Challenges

One of the craziest things to manipulate in a custom LINQ provider is the expression tree.

As I discussed in a prior post, this sample LINQ statement:

var query = from x in lq
where x == "4"
select x.Clone();

becomes an expression that basically is this function:

lq.Where(x => (x = "4")).Select(x => x.Clone());

(Before you say it, yes, that query is a bit weird; it's for illustrative purposes only and probably doesn't make a lot of sense...)

The function itself isn't rocket science but it's constructed using MethodCallExpression objects, which actually kind of read from right to left. That is, in the above example, the main expression you receive when you execute the query will start with the Select method - the second parameter will be a LambdaExpression containing this:

x => x.Clone()

and the first parameter will be another MethodCallExpression containing this:

lq.Where(x => (x = "4"))

... and that MethodCallExpression will have the second parameter set to this:

x => (x = "4")

and the first parameter set to a constant value representing this:

lq

... in other words, right-to-left.

One of the first questions I had was how the compiler and expression tree deals with embedded functions, for example the use of the Clone() function above, and how are those functions included in the tree alongside the IQueryable functions like Select() and Where().

If you also recall from last time, .NET calls CreateQuery() repeatedly for every line of the LINQ syntax query to build the expression (at least in my example). It also expects that each expression returned will have a return type of IQueryable, where T is the appropriate query type (in the example above, the return type is a single string). If this behavior is a key part of normal LINQ behavior you can then assume your query will be a string of MethodCallExceptions, with all embedded functions in the query implemented as LambdaExpressions in the second column of the MethodCallException. You could therefore create an expression tree visitor that makes a few assumptions:

1. The node you receive to execute will always be a MethodCallException (probably one that references IQueryable.Select()).
2. The first parameter will always chain to another MethodCallException to the root node, which should be (if you follow the advice of this and other postings on the topic) will be a ConstantExpression representing the original IQueryable object.
3. Any other functions that need to be resolved will likely be in a LambdaExpression on the second parameter of a MethodCallException.

Now, you might get a tree that violates these assumptions, and how you might navigate to that root node may be tricky. However, if you can find the root node and at least partially resolve the tree you can take unresolved nodes and hopefully execute them client-side (MSDN has a weird howto on this under "How to: Execute an Expression Tree" - I use the offline MSDN so if you want a link to that search the title) and that should work in some cases.

Lining up query parameters to input parameters and dealing with Select() transformations is another story I hope to build up to as well - as a more complex query builds and builds the datatype can evolve (for example, a Select() that transforms one record type to another).

I'm getting closer to being able to build a generic SQL LINQ layer that will be as simple as possible to use for other purposes, such as to front-end an old database engine that I worked on years ago under the super-secret code name "Juggernaut" (it's similar in nature to Microsoft Velocity and a few other products out there, although it's slightly different in focus).

Crazy Canadian Constitutional Crisis

It's amazing that even the current state of Canadian politics fails to captivate Canadian citizens...

Check out this link, one of many that outlines the current pending constitutional crisis:

http://ca.news.yahoo.com/s/capress/081202/national/parliament_crisis

This is actually a bigger deal than people might be thinking. Our government actually operates on a number of assumed rules inherited from hundreds of years of monarchy. For example, did you know that the reason you have to pay municipal tax is based on tenement law, which basically says you owe the king a certain amount of product from what you produce on your land? (That's a bit of an oversimplification but I leave it to you to read the details...)

The truth is that many foreign governments operate with coalition minority parties forming their own government. Israel, for example, has numerous (i.e. dozens of) fringe parties that always seem to meld together to make a government. Even the last Conservative government struck an alliance with whoever it could to make things work. The idea of all the minority governments teaming up to make a new prime minister is not actually that problematic constitutionally (although Stephen Harper would have you believe the communists had stormed Parliament Hill - that's not a statement against Stephen Harper, it's just a comment on the tone of the current rhetoric).

The two real problems are simply this. First, the Bloc has no business in control of anything Canadian. They are separatists and should likely be charged with treason. The only reason they still exist is because they aren't real separatists; rather, they are more akin to the western Reform party a few years back who are just a regional party seeking a better deal for their local constituents. (Perhaps this is the only true form of regional representation left?) However, even though their soft separatist attitudes are probably closer to the American Republican party's allegiance to the no-abortion-no-queers-6000-year-old-universe fundamentalist Christians than any real allegiance to the formation of a new country. (If I were PM: I'd create a document identifying the terms of separation and leave it in Quebec's hands to sign at their leisure. I say this as a Canadian and a francophone. Yes, when I was a kid we spoke conversational French in the house as well as English. Je me souviens.)

Where was I? Ah, yes: an anti-Canadian party shouldn't exist let alone be part of a leading coalition.

Second point: the problem will eventually be solved by Michaelle Jean. Not that she's not perfectly smart and a good human being, but let's all remember that she is the figurehead representative of Queen Elizabeth II. Constitutionally speaking, we'd be just as well off sending the issue to the British House of Lords for a fucking vote. Bash it all you want, at least the American electoral college is a local construct.

Also, we don't pick governor generals because of constitutional law insight. We might as well ask Ricky Gervais how to resolve the issue, and who knows, maybe in a few years he'll be appointed as governor general and he'll be able to weigh in. Will we truly allow the prime minister to be essentially appointed by someone who wasn't even a Canadian citizen until she was asked to be governor general? (Don't get me wrong: she's as competent as any other appointee to the position; I'm arguing that the relative irrelevancy of the position is sufficient to possibly result in a constitutional crisis given the current turmoil. By the way, until she became governor general she was French - not Quebec French, citizen of France French.)

Just as I was writing this my main man Jeff directed me to some Albertan jackass op-ed piece which will probably represent most writing on this topic. Let me summarize for you: "this is a Liberal power grab, they shouldn't be allowed to do this". Thanks, buddy, for your wondrous insight on constitutional and electoral processes. I won't even pretend that this isn't a power grab, of course it is - all political parties seek to gain and hold power. This is no different from the conservatives calling an early election in spite of their promise not to do so during the prior election. The truth is that there are significant questions about this process outside of the perceived or real "power grab" that call to question the Canadian parliamentary system and how it's structured and organically grown.

Ode to Dane Cook

(The following is an excerpt from an email conversation between myself and my old runnin' buddy from Handsome Boy Modelling School, Tall Mike.)

I saw Dane Cook on his ridiculous HBO special, and last night he had a 5 minute bit on some entertainment show where he basically says how he's not nervous about his HBO special even though he's doing something "no one else has ever done before". What is that, you might ask? He was using a round stage with people all around him, instead of a typical stage with everyone on one side and he's in front of them, 'cause standard stage layout is for amateurs, not pros like Dane Cook. Like, what? You fucking invented that? Like, if no one had done that before, I guess they built a custom auditorium just for his stage for the first time ever, right? I mean, what's $800 million fucking dollars for one night with Dane, am I right? No one has ever watched something in a circle before, right? Like, the Coliseum in Rome looks like a fucking ring from 2000 years ago but it's not, it's just some shitty pretend ring because Dane Fucking Cook apparently *invented* it.

And even if he did invent it, it's not like a cure for cancer.

Dane Cook: "Man, I'm so nervous about the show tonight, I changed the auditorium layout slightly, it's going to revolutionize live performance."

Dane Cook Fan #1: "Wow! It's in a circle! I have never seen such magnificence before!"

Dane Cook Fan #2: "I mean, like, if he was just in front of us, I'd be, like, 'okay, that's sorta funny' but all around us it's like, 'damn! That's some funny shit!'"

Like, are you fucking kidding me?

It's not just that he's not that funny. It's not just that everyone in his audience thinks he's the funniest man alive. It's that *he* thinks he's funny. After every joke he looks at the camera with this look on his face like he just said the funniest thing known to man, and just realized it, and was suddenly proud of unleashing such a funny, funny joke.

But the joke he just told and is so proud of himself over was the one about when you just need to have a good cry. Which is fucking ridiculous, because the word on the street is that he's a man. Man up, buddy. Like, don't spend 15 minutes on "needing a good cry". I'm not saying I've never cried. I'm saying I've never needed a good cry, because I am a man. Even a homosexual man would be like, "Dude, man up there buddy."

Yeah, I saw the commercial for that Dane Cook movie and I think I'd rather watch Sisterhood of the Travelling Pants III: Kelly's Revenge. (I assume there's at least one Kelly in a movie like that.) Of course I'm sure he thought he invented movies too.

Will someone explain to me what the attraction is?

LINQ Provider Exploration

Just a few more thoughts on LINQ providers and how they work.

If you look at the IQueryable and IQueryProvider interface, you'll see that it presents a very generic specification while the implementation expects much more specific output.

For example, the property IQueryable.Expression can't just be any expression, it has to be of a specific type - in particular, the output of resolving the expression has to be of type IQueryable (I think). If you are implementing a new class for this you might set the default value to Expression.Constant(this), with this in this context being the IQueryable object. This expression basically says, "return all of the items associated with me".

The expression itself is the internal representation of a query and it is built out of the LINQ code for you. For example, by default this (silly sample) code:

var query = from x in lq

    where x == "4"

    select x.Clone();

resolves into an expression that looks like this:

{value(TestLinqProvider.LinqQuery`1[System.String]).Where(x => (x = "4")).Select(x => x.Clone())}

Which is basically a function that looks like this (the variable lq above is of the type TestLinqProvider.LinqQuery):

lq.Where(x => (x = "4")).Select(x => x.Clone())

That final expression isn't built all at once, however. In my tests it appears that it is built one piece at a time, using IQueryProvider.CreateQuery to chain the results. In the above example, I have a public constructor for my LinqQuery object (lq above) that sets its default Expression value to Expression.Constant(this). Subsequent query statements will each in turn filter or transform this default object. The call to IQueryProvider.CreateQuery is done twice; the first expression value is just the default constant:

lq

Which is then added to the "where" clause of the LINQ statement by the first call to CreateQuery:

lq.Where(x => (x = "4"))

Which then is finished by the last call to CreateQuery:

lq.Where(x => (x = "4")).Select(x => x.Clone())

And that is the actual expression for the LinqQuery object that will be executed. If you don't take the expression value passed in to CreateQuery and use it in your returned query object, the expression tree won't be built. Once again, testing has also indicated that the returned expression in query objects associated with CreateQuery has to be resolvable into the appropriate IQueryable type.

Most of the query providers out there don't actually do anything in the CreateQuery call but take the new appended query statement and attach it to a freshly created query object. The query provider is then left with the task of developing a completely new query from scratch in the IQueryProvider.Execute method. The query object becomes just a holder for an expression in this scenario.

However, it is also possible to modify the expression on each CreateQuery statement to incorporate more data. For example, you may want to pre-parse statements to separate functions you might resolve client-side from those you might want to resolve server-side in a SQL statement.

The basic result is that most expression trees created legitimately via LINQ would wind up having a fairly specific format; however, to conform to the interface, you are pretty much stuck resolving any combination of tree elements that come up. Some query providers might deal with unexpected structures by executing any questionable expressions client-side; for example, instead of placing a "where" clause at the SQL server where it would be optimal, one could download the entire table and let the default IQueryable.Where method do the work client-side. New query providers can probably start with implementing the obvious methods as they are built by the default LINQ implementation and then figure out how to deal with variety later.

LINQ Providers

Well, in my last post I talked about not using stoopid kung fu to solve problems that are easily solved elsewhere. It is in direct opposition to that wisdom I have been exploring writing my own LINQ provider.

More on this later, but for now, here's a great set of articles on the topic. I read the blog, then the reference docs, then did some basic watches/breakpoints in a VS2008 test app. With those three I'm starting to get a handle on it.

Here's the blog posts.

Software Complexity as an Objective Measurement

I had a conversation last Thursday that really drove home the need to make software complexity more concrete. I had just spent 2 hours talking to one of the senior guys around the office, a person with a known propensity for making software complex. For the sake of conversation, let's call him "Mr. F". Of course, Mr. F. disagrees that his software is complex at all and most of my attempts to suggest the solution he had come up with could be done simpler were met with, "Why? I didn't even write the code. I hired contractor X to do it."

So, after two long hours, Mr. F. makes the statement that finally puts the right words to my complexity concerns. It went something like this: "So, yeah, in the last week or so I spent a lot of time at my computer coding the solution together with contractor X, he thought it was a ton of change but he learned a lot, and we ended up getting into production." So, there it is: the only reason they hit their deadline was that the one person who really understood what was going on sat down and did the work (contractor X might have been sitting beside him, but I doubt it was equal contribution if the person writing the software actually needed to be taught how the code he supposedly wrote was working). In the absence of Mr. F., the software would become unmaintainable due to complexity. Of course, as far as he's concerned, the software is "just right", and I really don't have anything but a sense that it's not.

There's lots of measurements for complexity. For example, cyclomatic complexity. One of our projects recently used this as a major benchmark. Unfortunately, an application with 14,000 classes and 150,000 functions can still have a cyclomatic complexity of 3. You can also count other items; Robert Cecil Martin has what I consider as comprehensive a list as anyone if you're talking about actual calculable metrics.

Unfortunately, these numbers aren't what I think really make a piece of code easy or hard. They certainly contribute heavily but they are usually a metric of quality as opposed to complexity. Worse scores in these concrete areas almost always stem from inexperience. In other words, a junior developer is much more likely to score higher on the complexity side than even a senior person like Mr. F. would.

The problems I have tend to be in areas where a technique is used that can be done well or poorly, but is almost impossible to do well because of its inherent difficulty. For example, if I were to take a sample of 100 developers, I could probably get them to do the following things pretty well:

1. Write a simple SQL query with a couple of joins. Everyone learns this at NAIT and it's hard to screw up. Most queries are pretty simple in practice from what I've seen, and those that are not can be abstracted away by correct use of views. Query optimization plans work well with views so this isn't a performance hit.
2. Create a screen with simple validations and bind it to a data object. Every developer knows one technique for doing this, and most can be trained to do whatever flavor you feel like documenting.
3. Write a function to do a validation or calculation that is at least somewhat reusable.
4. Create a small database.

However, most developers don't seem to be able to do the following other things:

1. Write a thread-safe application that allows multiple threads to participate on a single activity. Synchronizing global state across multiple threads, especially when there's a higher chance of two threads wanting the same thing, is officially rocket science. I'd say maybe 5% of developers can do this, and the rest may or may not be trainable if they needed to pick thread safety up. If you have implemented a business process in a way that requires multithreaded knowledge, best of luck to you.
2. Create and use a comprehensive domain model. See a recent post from me for some thoughts and counter-thoughts about domain models. You can usually get a person to correctly use a simple data-only model but if it's too complex antipatterns start to emerge, like creating giant "god objects" with everything and their dog bolted to them or "lazy load" scenarios where every object access round-trips to the database like 37 times. And creating one from scratch? Forget it. At least in this case I can say maybe one in five can do it properly.
3. Properly implement a transaction other than a default, pessimistic locked, fully isolated, single database transaction. It's virtually impossible to find people who by default know the uses and consequences of different transaction models.

You might say, "I can do all of that, and I know a ton of people who also can." Well, fine, but I know a ton more who can't. I'm talking about teams with a dozen members not having a single person who can properly deal with at least some of the three items above.

What's my point? If you design an answer to a problem, and you need knowledge of the three areas above (there's definitely more than just these three), your solution is too complex. Either you have to be solving a really difficult problem or you have just overdesigned your solution.

I'm not saying you shouldn't use these more advanced techniques. I'm saying that if you can do what you need without them, you should. Cool code kung fu is not needed to write a data entry screen. I'd think it would be valuable to put together a list of design patterns or decisions such as the three above and maybe use them as a complexity checklist, where more checks equals more complexity.

Thank you, Martin Fowler, for creating a new blog

You might wonder why I am attributing this blog to Mr. Fowler. Well, here's why.

I got a note in my email the other day that looked like this:

From: Justice Gray
Sent: Wednesday, October 08, 2008 10:40 AM
To: Joe Friesenhan; Christians Izquierdo; Andrew Mackenzie
Subject: Only sending this as food for thought

Or fodder for discussion. Honestly!! I’m not even too sure I agree with this entirely, but it’s always good to be challenged…

http://www.martinfowler.com/bliki/AnemicDomainModel.html

-J

So, I read the article. What the article taught me is that, in the computing industry, once you say one or two smart things, everybody starts thinking you are smart, and then you start to get cocky and say things that make no sense. So, I sent back this note:

From: Joe Friesenhan
Sent: Monday, October 20, 2008 1:31 PM
To: Justice Gray; Christians Izquierdo; Andrew Mackenzie
Subject: RE: Only sending this as food for thought

I know it’s been a long time since you sent this note, but I finally got to it and now I just have to say that I now officially think Martin Fowler is an idiot.

The notion that a data-only domain model that allows one to provide logic services (whether heavier impact WS-* based or just internally distinct from the domain model) is actually an antipattern is so utterly purist that I can’t believe he’d record this thought for others to see. I would, in fact, make the opposite argument: is there even a realistic technology model where a pure “data+logic” business domain model makes sense?

Let’s also keep in mind that even if you think I’m a douchebag for being so opposed to this post, I wouldn’t be alone: the core tenet of the Zachman framework is that the what and the how (data and logic) *must* be separate. I remember hearing that and thinking it was too purist as well, and it is, but apparently the entire Zachman framework is an “antipattern”.

Seriously, I don’t follow blogs enough to know if calling Martin Fowler a genuine idiot is some kind of thing the cool kids do these days, but I don’t see how such a myopic point of view that has no grounding in technical implementation savvy can come from someone with such a good reputation. I’d have to say the reputation is unfounded..

I can’t wait for Mr. Fowler’s post on how ignorant the world is for not realizing it is only 6000 years old.

And then, I received this response in return:

From: Justice Gray
Sent: Wednesday, October 08, 2008 10:40 AM
To: Joe Friesenhan; Christians Izquierdo; Andrew Mackenzie
Subject: RE: Only sending this as food for thought

I wish you had a blog.
-J

So, long story short, here I am. I'd like to thank Martin Fowler for giving me the extra push needed to get off my ass and publish my own blog so that hopefully I can undo the small bit of damage his post above has done to the computing industry. And, with Sarah Palin running for the vice presidency of the USA, there's a *lot* more to cover!