Reality in EVE Online: A Rebuttal

I've been thinking about the EVE Online scenarios in a previous post and how a game can be written that offers a similar freedom and fun but with more realism and less of a hardcore gamer element. As I was thinking about this, a lot of "flaws" started to show up in the two scenarios that could be solved by applying more in-game automation and less of a "wild wild west" approach.



For example, take the Ubiqua Seraph assassination. Although many of that corporation's members likely resigned outright in its aftermath, EVE Online supposedly attracted numerous new players because of the "raw realism" of the events that transpired. But how realistic is it that a group of people could infiltrate a relatively small number of high level positions and then essentially liquidate the entire company? This is hardly real-world, and the flaws are largely to do with in-game corporation management.



Look also at the Nightfreeze scam. This is reminiscent of the Bernie Madoff scandal in recent American news. Essentially, a person claims to invest a large amount of money for a group of people but just shuffles it around and takes a big cut for himself, leaving all of the investors with little to no resources remaining.



It has been said that the EVE Online manifestation was realistic, but is that totally true? Not really. For one thing, once the scam was revealed, Bernie Madoff didn't get to keep his money. In the EVE scam Nightfreeze immediately quit the game after handing $300 million to some new player, but that would never happen. If Bernie Madoff killed himself that might be equivalent to "quitting the game" but if he walked up to a McDonald's employee and handed him a cheque for billions it's not likely that the American government would have allowed him to keep it.



So, is it possible to have a more realistic in-game situation that allows full player freedom without being so arbitrarily non-interventionist?

Signs You Need A Hobby, Part 26: Data Access Stimulates You

Here's a manifesto about Microsoft's Entity Framework. I need you to understand something about the tone of this message: there's actually enough people who feel passionately about data access to write a manifesto about Microsoft's Entity Framework, and I frankly find that shocking. You know what? If you care about this so much, go look at a mountain, have sexual intercourse, do some old-school breakdancing, do some tequila shots. Seriously, time to prioritize.

I gotta pick this blog post apart. I realize that, in doing so, I am lowering myself to that level, but if it's any colsolation it's already 3:00 in the afternoon so I'm probably already drunk while I write this.

Let's start with a few choice quotes:

The signatories of this letter are unanimous in expressing concern for the
welfare of software projects undertaken in the Microsoft customer community that
will make use of the forthcoming ADO .NET Entity Framework.

And here's another:

We remain willing to collaborate with Microsoft and the ADO .NET Entity
Framework team to forge a positive action plan to help the Microsoft customer
community achieve success with entity architecture applications.

Now if the first quote was written like this:

The signatories of this treaty are unanimous in expressing concern for the
welfare of Palestinians living in the West Bank community that
will need to make use of the forthcoming UN emergency aid package.

And the second like this:

We remain willing to collaborate with the UN Security Council and the Israeli government and millitary to forge a positive action plan to help the Palestinian Authority achieve success with its efforts to support innocent civilians.

I could understand the need for enthusiasm. But let me reiterate: THIS IS A FUCKING DATA ACCESS LAYER. I don't normally like to swear, particularly in all caps, but seriously.

There's also a ton of technical statements that are basically phrased like "most people think you can access data in more than one way, but we members of the Entity Zealot community believe there can be only one, and only morons don't recognize that ours is the one true path." For example:

The Entity Framework encourages the Anemic Domain Model anti-pattern by
discouraging the inclusion of business logic in the entity classes.

I'm sorry, "Anemic Domain Model"? Ever heard of messaging oriented applications? Also, how do you propose taking your non-anemic domain model and spreading it over a WSDL boundary? Is there no value in a data-only approach? Normally one might suggest that the article in question merely suggests one possible way of doing things; however, by referring to "data-only objects" as the "Anemic Domain Model anti-pattern", I think it's basically saying only fuck-ups use it.

I'll restrict myself to two more rebuttals, even though I could really get stoopid and just comment on every sentence in this document.

Here's something on lazy loading:

Lazy loading is an essential capability of a data access framework for
entity-based applications. Without lazy loading, an entity-based application’s
codebase will need to include unnecessary prescriptive data loading procedures
for every possible business scenario in which the business entities will be
used.

I'd probably add the following sentence to this quote: "Of course, you will have to augment the built-in lazy loading capabilities of most ORMs with a large amount of optimization hints and restrictions on use to avoid making 47 round-trips to the database every time you ask for a simple set of information, but hey, that's definitely easier than writing a SQL statement. I mean, 'Select Name from People', what the hell does that mean?"

Last one: Here's something on the value of "persistence ignorance":

A team’s ability to do evolutionary design and incremental delivery is
damaged by the Entity Framework’s inattention to fundamental software design
principles like Separation of Concerns.

I always enjoy it when a person writes "Separation of Concerns" but they really mean "Separation of Concerns exactly as I would implement it". Has it occurred to no one that lazy loading, by hiding the fact that property accesses may or may not return to a database depending on current cache state, forces a business application to be heavily dependent on side effects or non-functional behaviors that are generally difficult to understand fully enough to predict with any certainty the final behavior of the system? Isn't it of benefit to be explicit when you say, "Now I'd like you to go to a database, which is historically the bottleneck in an enterprise application, and perform your multi-millisecond activity?"

This is exactly why I stopped working on my custom LINQ provider stuff. I've come to realize that the data access question is simply a problem we've already solved a thousand times, and each solution has its own advantages and disadvantages; the intelligentsia is better served by focusing on technology choices that actually have a direct impact to clients.

One last dig: whether you like it or not, if you are a big ORM advocate you should probably just go back to .NET 1.1 and stick with typed datasets. I've heard all the arguments about what makes ORMs great, and I could argue that typed datasets do each of those features better. (FYI: I HATE typed datasets.)

Copy from Byte Array To Structure

I've seen a few bad (or at least suboptimal) implementations of this out there, so I thought I'd throw my own implementation into the mix.

There are three reasons why I have some issues with many other implementations I've been seeing:

1. The implementation uses managed pointers to access the byte array as though they are unmanaged pointers. Unfortunately, this doesn't really work.
2. The implementation locks the managed pointer on the byte array, which affects garbage collection and can seriously affect throughput. Locking should be avoided for copying small structs around.
3. The implementation copies the data properly into an unmanaged pointer first, but it uses a heap allocation method to get the memory, which for small structures is somewhat suboptimal - stack memory is probably better for this sort of thing.

The following code snippet avoids all three of the issues above. The only problem is there is an extra memory copy which could only be avoided with managed C++ and its managed pointer construct (I may post that another time). Enjoy.

public sealed abstract class BinaryStreamSupport
{
private BinaryStreamSupport() { }
public static T ReadStruct(Stream s)
{
int size;
byte[] data;

size = Marshal.SizeOf(typeof(T));
data = new byte[size];
s.Read(data, 0, size);
return _ReadStruct(data, 0, size);
}

public static T ReadStruct(byte[] data, int start)
{
return _ReadStruct(data, start, Marshal.SizeOf(typeof(T)));
}

private static T _ReadStruct(byte[] data, int start, int size)
{
T returnValue;
unsafe
{
byte* stackDataPtr = stackalloc byte[size];
IntPtr stackData = new IntPtr(stackDataPtr);
Marshal.Copy(data, start, stackData, size);
returnValue = (T)Marshal.PtrToStructure(stackData, typeof(T));
}

return returnValue;
}
}

More LINQ Challenges

One of the craziest things to manipulate in a custom LINQ provider is the expression tree.

As I discussed in a prior post, this sample LINQ statement:

var query = from x in lq
where x == "4"
select x.Clone();

becomes an expression that basically is this function:

lq.Where(x => (x = "4")).Select(x => x.Clone());

(Before you say it, yes, that query is a bit weird; it's for illustrative purposes only and probably doesn't make a lot of sense...)

The function itself isn't rocket science but it's constructed using MethodCallExpression objects, which actually kind of read from right to left. That is, in the above example, the main expression you receive when you execute the query will start with the Select method - the second parameter will be a LambdaExpression containing this:

x => x.Clone()

and the first parameter will be another MethodCallExpression containing this:

lq.Where(x => (x = "4"))

... and that MethodCallExpression will have the second parameter set to this:

x => (x = "4")

and the first parameter set to a constant value representing this:

lq

... in other words, right-to-left.

One of the first questions I had was how the compiler and expression tree deals with embedded functions, for example the use of the Clone() function above, and how are those functions included in the tree alongside the IQueryable functions like Select() and Where().

If you also recall from last time, .NET calls CreateQuery() repeatedly for every line of the LINQ syntax query to build the expression (at least in my example). It also expects that each expression returned will have a return type of IQueryable, where T is the appropriate query type (in the example above, the return type is a single string). If this behavior is a key part of normal LINQ behavior you can then assume your query will be a string of MethodCallExceptions, with all embedded functions in the query implemented as LambdaExpressions in the second column of the MethodCallException. You could therefore create an expression tree visitor that makes a few assumptions:

1. The node you receive to execute will always be a MethodCallException (probably one that references IQueryable.Select()).
2. The first parameter will always chain to another MethodCallException to the root node, which should be (if you follow the advice of this and other postings on the topic) will be a ConstantExpression representing the original IQueryable object.
3. Any other functions that need to be resolved will likely be in a LambdaExpression on the second parameter of a MethodCallException.

Now, you might get a tree that violates these assumptions, and how you might navigate to that root node may be tricky. However, if you can find the root node and at least partially resolve the tree you can take unresolved nodes and hopefully execute them client-side (MSDN has a weird howto on this under "How to: Execute an Expression Tree" - I use the offline MSDN so if you want a link to that search the title) and that should work in some cases.

Lining up query parameters to input parameters and dealing with Select() transformations is another story I hope to build up to as well - as a more complex query builds and builds the datatype can evolve (for example, a Select() that transforms one record type to another).

I'm getting closer to being able to build a generic SQL LINQ layer that will be as simple as possible to use for other purposes, such as to front-end an old database engine that I worked on years ago under the super-secret code name "Juggernaut" (it's similar in nature to Microsoft Velocity and a few other products out there, although it's slightly different in focus).

LINQ Provider Exploration

Just a few more thoughts on LINQ providers and how they work.

If you look at the IQueryable and IQueryProvider interface, you'll see that it presents a very generic specification while the implementation expects much more specific output.

For example, the property IQueryable.Expression can't just be any expression, it has to be of a specific type - in particular, the output of resolving the expression has to be of type IQueryable (I think). If you are implementing a new class for this you might set the default value to Expression.Constant(this), with this in this context being the IQueryable object. This expression basically says, "return all of the items associated with me".

The expression itself is the internal representation of a query and it is built out of the LINQ code for you. For example, by default this (silly sample) code:

var query = from x in lq

    where x == "4"

    select x.Clone();

resolves into an expression that looks like this:

{value(TestLinqProvider.LinqQuery`1[System.String]).Where(x => (x = "4")).Select(x => x.Clone())}

Which is basically a function that looks like this (the variable lq above is of the type TestLinqProvider.LinqQuery):

lq.Where(x => (x = "4")).Select(x => x.Clone())

That final expression isn't built all at once, however. In my tests it appears that it is built one piece at a time, using IQueryProvider.CreateQuery to chain the results. In the above example, I have a public constructor for my LinqQuery object (lq above) that sets its default Expression value to Expression.Constant(this). Subsequent query statements will each in turn filter or transform this default object. The call to IQueryProvider.CreateQuery is done twice; the first expression value is just the default constant:

lq

Which is then added to the "where" clause of the LINQ statement by the first call to CreateQuery:

lq.Where(x => (x = "4"))

Which then is finished by the last call to CreateQuery:

lq.Where(x => (x = "4")).Select(x => x.Clone())

And that is the actual expression for the LinqQuery object that will be executed. If you don't take the expression value passed in to CreateQuery and use it in your returned query object, the expression tree won't be built. Once again, testing has also indicated that the returned expression in query objects associated with CreateQuery has to be resolvable into the appropriate IQueryable type.

Most of the query providers out there don't actually do anything in the CreateQuery call but take the new appended query statement and attach it to a freshly created query object. The query provider is then left with the task of developing a completely new query from scratch in the IQueryProvider.Execute method. The query object becomes just a holder for an expression in this scenario.

However, it is also possible to modify the expression on each CreateQuery statement to incorporate more data. For example, you may want to pre-parse statements to separate functions you might resolve client-side from those you might want to resolve server-side in a SQL statement.

The basic result is that most expression trees created legitimately via LINQ would wind up having a fairly specific format; however, to conform to the interface, you are pretty much stuck resolving any combination of tree elements that come up. Some query providers might deal with unexpected structures by executing any questionable expressions client-side; for example, instead of placing a "where" clause at the SQL server where it would be optimal, one could download the entire table and let the default IQueryable.Where method do the work client-side. New query providers can probably start with implementing the obvious methods as they are built by the default LINQ implementation and then figure out how to deal with variety later.

LINQ Providers

Well, in my last post I talked about not using stoopid kung fu to solve problems that are easily solved elsewhere. It is in direct opposition to that wisdom I have been exploring writing my own LINQ provider.

More on this later, but for now, here's a great set of articles on the topic. I read the blog, then the reference docs, then did some basic watches/breakpoints in a VS2008 test app. With those three I'm starting to get a handle on it.

Here's the blog posts.

Software Complexity as an Objective Measurement

I had a conversation last Thursday that really drove home the need to make software complexity more concrete. I had just spent 2 hours talking to one of the senior guys around the office, a person with a known propensity for making software complex. For the sake of conversation, let's call him "Mr. F". Of course, Mr. F. disagrees that his software is complex at all and most of my attempts to suggest the solution he had come up with could be done simpler were met with, "Why? I didn't even write the code. I hired contractor X to do it."

So, after two long hours, Mr. F. makes the statement that finally puts the right words to my complexity concerns. It went something like this: "So, yeah, in the last week or so I spent a lot of time at my computer coding the solution together with contractor X, he thought it was a ton of change but he learned a lot, and we ended up getting into production." So, there it is: the only reason they hit their deadline was that the one person who really understood what was going on sat down and did the work (contractor X might have been sitting beside him, but I doubt it was equal contribution if the person writing the software actually needed to be taught how the code he supposedly wrote was working). In the absence of Mr. F., the software would become unmaintainable due to complexity. Of course, as far as he's concerned, the software is "just right", and I really don't have anything but a sense that it's not.

There's lots of measurements for complexity. For example, cyclomatic complexity. One of our projects recently used this as a major benchmark. Unfortunately, an application with 14,000 classes and 150,000 functions can still have a cyclomatic complexity of 3. You can also count other items; Robert Cecil Martin has what I consider as comprehensive a list as anyone if you're talking about actual calculable metrics.

Unfortunately, these numbers aren't what I think really make a piece of code easy or hard. They certainly contribute heavily but they are usually a metric of quality as opposed to complexity. Worse scores in these concrete areas almost always stem from inexperience. In other words, a junior developer is much more likely to score higher on the complexity side than even a senior person like Mr. F. would.

The problems I have tend to be in areas where a technique is used that can be done well or poorly, but is almost impossible to do well because of its inherent difficulty. For example, if I were to take a sample of 100 developers, I could probably get them to do the following things pretty well:

1. Write a simple SQL query with a couple of joins. Everyone learns this at NAIT and it's hard to screw up. Most queries are pretty simple in practice from what I've seen, and those that are not can be abstracted away by correct use of views. Query optimization plans work well with views so this isn't a performance hit.
2. Create a screen with simple validations and bind it to a data object. Every developer knows one technique for doing this, and most can be trained to do whatever flavor you feel like documenting.
3. Write a function to do a validation or calculation that is at least somewhat reusable.
4. Create a small database.

However, most developers don't seem to be able to do the following other things:

1. Write a thread-safe application that allows multiple threads to participate on a single activity. Synchronizing global state across multiple threads, especially when there's a higher chance of two threads wanting the same thing, is officially rocket science. I'd say maybe 5% of developers can do this, and the rest may or may not be trainable if they needed to pick thread safety up. If you have implemented a business process in a way that requires multithreaded knowledge, best of luck to you.
2. Create and use a comprehensive domain model. See a recent post from me for some thoughts and counter-thoughts about domain models. You can usually get a person to correctly use a simple data-only model but if it's too complex antipatterns start to emerge, like creating giant "god objects" with everything and their dog bolted to them or "lazy load" scenarios where every object access round-trips to the database like 37 times. And creating one from scratch? Forget it. At least in this case I can say maybe one in five can do it properly.
3. Properly implement a transaction other than a default, pessimistic locked, fully isolated, single database transaction. It's virtually impossible to find people who by default know the uses and consequences of different transaction models.

You might say, "I can do all of that, and I know a ton of people who also can." Well, fine, but I know a ton more who can't. I'm talking about teams with a dozen members not having a single person who can properly deal with at least some of the three items above.

What's my point? If you design an answer to a problem, and you need knowledge of the three areas above (there's definitely more than just these three), your solution is too complex. Either you have to be solving a really difficult problem or you have just overdesigned your solution.

I'm not saying you shouldn't use these more advanced techniques. I'm saying that if you can do what you need without them, you should. Cool code kung fu is not needed to write a data entry screen. I'd think it would be valuable to put together a list of design patterns or decisions such as the three above and maybe use them as a complexity checklist, where more checks equals more complexity.