Economy Size Geek

Happy families are all alike; every unhappy family is unhappy in its own way. - Leo Tolstoy

It is funny - If I look back a year ago I had two very big software projects (since I can’t say what they were we’ll call one Y and the X). Y went really well, and by all accounts is a big success! What about X - not so much. It is slowly turning out that the second project is going to have a much more lasting effect on my design and thought processes than the first. I mean I haven’t worked directly on X in months - it is still teaching me lessons. Proving not only that I have a lot to learn - but also that your mistakes can teach you a lot more than your successes.

I recently gave a presentation on a number of lessons I learned from the project ( A couple of those ideas I’m hoping to sit down and actually write out for posterity - and hopefully as a reminder so I don’t repeat them). But just yesterday, I got another lesson out of X! It really is the gift that keeps on giving.

My World

First, let me say that my development world is fairly constrained. Although I have worked on some seriously scaling infrastructure - by and large my focus isn’t on those kind of problems. I always end up focusing on really complex domains. I largely credit one book with really having an impact - Domain Driven Design. I first read it back in 2004. It does what a great book of this should do - it introduced me to great new ideas and also confirmed things I was already doing (plus giving me a language to finally describe it).

I make this point - because it means I spend a lot more thinking about what if represented in a given system (and how it is represented) more than I do about about many transactions the database can handle. Because of that - the details of the domain are seriously important.

Actual vs Logical

Let’s talk about something that is totally outside of the domain I work in. How about cars?

I have one - it has the best feature of all the cars on the road - it is paid off! (I highly recommend that as a feature).

I have VW Beetle.

For most circumstances - that is plenty of information. Meaning - I have a car. If you want to know if I can go some place - that is probably enough information.

When it comes time to go to lunch - we have to figure out who is going to drive. My car can fit 4 people. That detail is important depending on how many people want to go.

If I want to find out how much my car is worth on Kelley Blue Book I have to add the details that it is a 2000 GLT Turbo (plus lot of other details - 5 speed, seat warmers, etc).

The point of this is - if you asked me about my car - what you want to know about it is driven by the problem you are trying to solve by asking the question. If you ask:

Can you get to work?

Yes, I have a car. (Totally sufficient answer)

Yes, I have a 2000 VW New Beetle GLS Turbo Hatchback 2D with a 5 speed manual and seat warmers. (Is a bit nutty)

I’ll say that this is a scale from the logical to the actual. Logical being a sort of encapsulation of details that provides very limited information to the actual which has a ton of details.

Some examples:

I realized one of the problems with Project X was exactly this problem. Namely that it collected enough information to drown an actuarial. (Amazingly there this an entire site devoted to jokes about actuarials - www.actuarialjokes.com).

Why did it collect so much detail?

Data Pack Rat

I’m a 5th generation pack rat in the real world (I’ve been through some counseling and my wife has helped a lot), but I’m ten times worse in the digital realm. I collect data - of all sorts. This inevitably bleeds over to the things I work on.

Solving The Hard Problems

That was not the real reason - even if it is a convenient answer. The real reason is that the really tough problems in the domain for X required the data. Sure the day to day stuff only needed to know the basics - but the hard problems are hard precisely because they depend on knowing the details involved. This is the same reason that Kelley Blue Book needs so much detail about my car - it is hard to give an accurate estimate if you don’t know what you are estimating.

CSI & The Infinite Zoom

This is the other part of the reason why getting the actual can be so important. On CSI, they strive for what seems like scientific accuracy (One can only assume as a layperson). That being said - there is one situation where all the rules are thrown out. They love to zoom into pictures infinitely - even when they are digital. The worst case was the time they used footage from a security camera to get a picture of someone’s eye - and then look at the reflection to catch the killer. Going to have to get one of these futuristic security cameras - since everyone I’ve ever seen is seriously blurry.

What is the point of that little rant? Simple, if you don’t have the resolution you can’t just make it up. The same rules apply. If you only collect the make and model of cars (like they did to get my parking tag) you can’t really say how much it would be to replace that vehicle. If that is your goal - you’re screwed. My solution is typically to think about all the data we are going to need and work backward.
This doesn’t always work out - Project X being an example.

The Thin Vertical Slice

There is another complication in this whole process. We do a lot of Thin Vertical Slicing. It is incredibly useful since it means we get better feed back from the users and it keeps us from building a lot of stuff we don’t end up needing. The down side is that when you are dealing with a lot of data that has to be collected (i.e. a world wide audit of something) - you have to decide - do you collect what you need now - or while you are there - do you grab as much as you can while you are there. Once you grab it - what do you do with it. Since if you track it but don’t include it in the tool you roll out - how do you keep the data up to date? If you ignore it - you’re going to be back to audit the rest eventually (or maybe you won’t? YAGAI (You Aren’t Gonna Audit It) a subset of YAGNI)

Matching The User

This is the real crux of the issue. Both Project X & Y obsessed over the actual. Project Y was a success - X was not. Why not? At the end of the day, a simple answer - matching the domain data to the user.

In the case of Project Y, the users of the system were also obsessed with the actual data. The tool let them manage the data down to the smallest detail. Which made them more accurate and more effective. They felt no sense of overload because they were steeped in the details as part of the job. The tool just made it easier to switch to a new task. Awesome!

In the case of Project X, the users of the system didn’t care as much about the actual data. They just needed a couple of logical collections to make their decisions. The tool forced them to wade through so much data to get something done. Which meant when you were doing something very very complicated it was great - but most of the time things weren’t complicated - which means it sucked.

The lesson here is that we could have had both. We could have had the detail underneath (because we actually needed it to solve problems), but exposed less of it unless you wanted to dive in. We actually moved this way at the end of the project (some what unconsciously) - but by that time - other issues with the system were in the way of success.

Final Thoughts: So basically - keep in mind the problem you are trying to solve and the data required to do it, but overwhelm your users with that data at your own risk!

p.s. There is probably a corollary to this idea relating to integrating applications via services. I haven’t finished this idea - but its probably along these lines (Almost stolen verbatim from a friend). Namely - that when you are consuming services - focus on what you need - and nothing more. Meaning :

• Don’t validate data from the service you don’t care about. Since you don’t want to throw out a response because it includes stuff that doesn’t effect you
• Don’t ask for more data than you need - since you’re wasting resources and confusing the issue on what data you actually care about

Both things are part of the process of keeping the services from becoming to intertwined, but still manage to loop back to this idea of getting the detail level right.

Currently I’m grinding through the process of translating my old unit tests on a project into much nicer Rspec tests. Over all it has been a good experience. Especially since I caught some things in the process that the old tests didn’t cover.

So I started on new controller and I wrote something like this:

describe "My Cool Test" do
before(:each) do
@obj = String.new
puts "Hello"
@obj.should_receive(:stupid_method).once
end
it "should do something"
it "should do something else"
it "should not do this"
end

The object never calls stupid_method so it should fail the expectation right?

Well it turns out that although the begin is executed - because all of the examples are pending! If you run the above code you’ll see “Hello” several times - but no error.

Now add in the following example:

it "should be empty" do
@obj.should == ""
end

You’ll get:

Spec::Mocks::MockExpectationError in ' My Cool Test should be empty'
Mock 'String' expected :stupid_method with (any args) once, but received it 0 times


So there you go - (And don’t worry I submitted a bug)

You may be wondering why I said triggers a lot of work - well I’m currently using the Rspec mock library. I was very close to switching to mocha - which would have meant modifying a heck of a lot of tests.

Ok I’ve been busy lately (by lately I mean since 2.2 came out). I got a note a while ago from a friend about a problem with my RSS feed - didn’t think much about it. Then today I got an email from another friend who is trying our firefox 3 - it has a plugin that warns you when you visit a site that has malware. Apparently, my site is listed as a malware site…..

It turns out there was a hole in WordPress at some point that let them inject javascript and whatnot into the posts. They were hiding a bunch of it - but it showed up in the RSS.

I think I’ve cleaned out the bad posts (If you see one let me know), I’ve cleared out the back log of comments (all 25K of them), and updated to 2.5.1 - which is the latest and greatest - oh and it has security fixes for 2.5 (just released recently).

The only annoying thing is that every time I do this I have to deal with putting back in my patches to WordPress. I’m going to to give git a whril on this problem to see if it helps.

Also I noticed that some people are having problems with Postie on 2.5.x - I didn’t notice because I wasn’t running 2.5 - but since I am now - I’ll be fixing whatever the main problem is since I still need Postie to work.

I signed up for GitHub (More for the tracking than anything) - I still had to sort out a git repo for the code I don’t want out there in the wild.

Turned out to be pretty straight forward ( Good Instructions or Even better ones if you like packages)

Looks like gitosis was really the key. The only problem I ran into was my main server is still a Debian sarge box - ( Yes I know that’s old. Yes it annoys the crap out of me since every other box I deal with is actually an Ubuntu box. Yes I’m waiting for Hardy Haron to make a clean start of it.) I had to pull in a lot of packages from either etch or testing (1.5.4 is in testing). Not ideal but it works for now.

I actually spent the last three days in the UK and while I was there I had very limited internet connectivity. No big deal - I did a bunch of work in a git repo locally. Now that I’m back home I’d prefer to make sure that the code isn’t just on my laptop (Backups are nice - so is my home workstation). I was able to easily add the new repo to my new git server. No problem.

Example from the above articles:
mkdir free_monkey
git init
git remote add origin git@YOUR_SERVER_HOSTNAME:free_monkey.git
Add in some files - blank repos are no longer pushable
git push origin master:refs/heads/master


On my workstation I just did
git clone git@YOUR_SERVER_HOSTNAME:free_monkey.git
cd free_monkey

If I need to add a new repo - or a new user - I can just

git clone git@YOUR_SERVER_HOSTNAME:gitosis-admin.git
cd gitosis-admin

Very nice!

Then I hit a small wall - basically I was in the middle of some feature work. I had two different branches off of trunk - one related to a new feature that is still a day away from being finished - and a strain of development related to migrating the app to Rails 2.0.x. It wasn’t obvious what the hell to do. Basically you have local branches and remote branches. I’ve obviously master local branches - now how the hell do I get them to be remote.

Then I found this overall guide Git Guide - SourceMage Wiki

on my laptop (with the branches - and after I had commited master)

git co rails-2.0.x
git push origin rails-2.0.x:refs/heads/rails-2.0.x


on my workstation (update/get list/create local branch)
git pull
git branch -r
git branch --track rails-2.0.x origin/rails-2.0.x

(Apparently if I ever need to delete the remote branch I just do this)

git branch -r -d origin/rails-2.0.x

This is all a little bit confusing - but I’m sure once I get the hang of it I’ll end up adding some global aliases to make like easier. All this power and no ability to replicate the functionality of svn:externals - the mind boggles.

I saw the announcement this morning that Rails is moving their development to Git from Subversion. I’ve been toying with Git for about 6 months but haven’t made the switch because there isn’t a tool that supports externals the way I use it at work.

In the last few days, I’ve been poking François Beausoleil, the author of Piston, to get more of my external goodness sorted out. This would allow me to spend my days working in Git while the rest of the team I work with stays in SVN - which is just about the perfect solution.

So far it still has its bumps - but I’m optimistic.

That being said I wasn’t switching to Git because DHH said to. I was more moved by the video of Linus at Google.

The funny thing about that is a long long time ago - I used bit keeper at work. Linus used bit keeper to maintain the kernel. I hated bitkeeper! At the time, the argument was constantly - it must be good - they use it to maintain the kernel - which was a crap argument considering there were only 6 of us and we all sat in the same room - we weren’t a distributed team of thousands. I eventually got us to switch to Subversion (Which had the nice benefit of saving my dept $1500/seat a year for scm and got us shiny new laptops) and things were good.

When I started seeing grumblings about Git a year ago - I was prepared to ignore it. It seemed like it was going to be bitkeeper all over again - meaning a great solution for Linus - and a bad one for me. I’ve been watching for signs that I could keep my externals workflow. François even posted on my blog to give me a heads up about the new - not quite released version of his tool.

Last night I got one of my project migrated so I can use either SVN or Git on it. I figure after a few weeks of working I’ll know if Git is what it is cracked up to be - and if it isn’t I can go back to the land of SVN. I remember what it was like back in the day when I went from CVS to SVN. I loved the improvements in SVN and hated the fact that there was zero tool support. That eventually settled down - here’s hoping that the path is much shorter for Git.

I’m back in the JavaScript trenches. I continue to be amazed at how flexible the language is - once you really dig into it. That’s the good news - but then comes the bad news.

I’ve grown very used to a number of the great features of Ruby. To be completely honest in a number of cases I didn’t really even think about them being there - because Rails uses them for incredibly great programmer outcomes. Now that I’m in JavaScript and they are missing - the sadness is ensuing.

An easy example of what I’m talking about:

person.name = “Bob Roberts”
person.name #Bob Roberts

On the Ruby side that can actually be implemented as a set of methods on the object.

class Person
  def name()
     return(@name)
  end
  def name=(name)
    @name = name
  end
end

Ruby hides the method part from you - which is nice. If you wanted to get a little crazy you could do it with a method missing call (This is not the way to do it but I’m making a point here - stick with me).

class Person
  def method_mission(m, *args)
    if  m == "name"
     if args[0]
       @name = args[0]
       return true
     else
      return(@name)
    end
  end
end

Not pretty - but very very powerful. I’ve run into some really cool articles on meta programming in JavaScript - so I was hoping there was a way to emulate this kind of functionality in Javascript. Turns out - there isn’t. Then I ran into a presentation from Jon Resig (Author of jQuery and JavaScript Bad Ass). If you check out his recent presentation on ECMAScript 4 (You can skip ahead to slide #13 & 14) You’ll see that they are specially adding the features I’m talking about to the new ECMAScript 4 (AKA JavaScript 2.0). There are some other neat things in there (there are also some other crazy things that make it feel much like the time I watch PHP4 convert to PHP5 and it felt like they were adding things based purely on emulating other languages).

That’s the good news - the bad news… FireFox 3 (which is currently in beta) will only support JavaScript 1.8. The only reference to a date in the presentation is 2008-2009. I found this blog post that says they are hoping to firm it all up by October of 2008.

Guess that means I’m not going to be waiting for it - since I actually have stuff to get done right now (almost literally - I have a code demo due on Monday for a library I’m working on). At least I know that the renaissance I’ve seen in the JavaScript world is really just getting started.

*Side Note* - To give you an idea of how not a baked cake all of this thinking is - here is a PDF that shows some concerns over some of the features that are possibly going to be included.

I saw this in the RSpec code while I was writing tests today:


[self.errors.on(attribute)].flatten.compact


It struck me as a very concise way to handle the case where it is either a scalar or an array (assuming you don’t care if the returned array is nested). It might get you out of some gymnastics trying to account for both cases.

I’ve been watching some Peepcode stuff this weekend and I’m officially jealous of the Growl support. It turns out I’m not the only one. After surfing around for a while I found this :

ikhono.net: Gnome autotest Notifications

I had to update the regex to handle a change in the output of Rspec - but otherwise it works like a charm. (I also modified it to change the tray icon to the same icon it displays on the status update).

Here are my changes:

Autotest.add_hook :initialize do |at|
@notify = Notification.new
end

Paul Graham:

To the extent there is any discomfort about the language you used, you can often use the pointy-haired bosses’ mistaken ideas against them. If they take as an axiom that “all languages are equivalent,” (if this is true, btw, we’re all wasting our time talking about language design on this list) they’ll easily believe that it will be a trivial matter to port your code to some more boss-friendly language, and that the language your software is written in is thus no serious barrier to acquisition.

Re: Y Store now C++

I been spending a lot of time lately in the land of plugins for Rails. I’m in the process of overhauling a bunch of GUI helpers I wrote a long time ago. One thing that was slowly driving me crazy is that every time I make a change I had to restart the web server to get it to reload everything in development. My old tricks for reloadability just didn’t seem to work.

I had looked for a solution before but came up empty handed. This time I hit pay dirt. Turns out you have to add some small bit of code to you environment.rb and you get reloadability back. Now I can get down to some business.

spotstory » Upgrading to Rails 1.2

Ok. I need to reload my plugins. Bleh. Further, this has to happen in environment.rb, not in environments/development.rb. Bleh.

# Array of plugins with Application model dependencies.
reloadable_plugins = ["acts_as_commentable"]
# Force these plugins to reload, avoiding stale object references.
reloadable_plugins.each do |plugin_name|
reloadable_path = RAILS_ROOT “/vendor/plugins/#{plugin_name}/lib”
Dependencies.load_once_paths.delete(reloadable_path)
end