If the Shoe Fits

Иконописиконографиямека мебелИкониI was driving in my car and catching up on some podcasts. I switched on the Change Log. The episode was all about DevOps. I love it when I find out that there is a term for something I’ve been doing for a while.

In the broadest sense, DevOps is about bringing development practices into the system administration universe. Most of the time I end up playing the system administrator on the team (until we get large enough that we can hire an actual system administrator). I always felt like the practices would be improved by adding in a more developer way of looking at things.

For a while I used something called cfengine. It was a HUGE improvement over no configuration control whatsoever. It is the grandfather of this whole movement in a lot of ways. Just as a word of warning, unless things have significantly changed since I used it – please don’t use cfengine. It is very exciting at the beginning when you start to get stuff to work, but as change occurs it gets harder and harder to keep up.

After seeing the potential of cfengine, I started looking for an alternative. I found Puppet It is written in Ruby (which is great since I’m ruby fluent). It also seemed to be able to address a lot of the problems and frustrations I had with cfengine. There weren’t a lot of alternatives. I spent some time trying to get everything to work. My dream was to be able to build a complete virtual environment in kvm that would let me model the entire deployment of our applications. Not too long after I started down this path, someone announced that Puppet sucked and that they were starting a new tool – Chef. Chef had a more bare metal approach to configuration management. I ended up avoiding it because at the time there was no way to tell if they were going to build a community and survive.

In the end, I ended up dumping all of them. Not because they weren’t good, but because instead I started launching apps at Heroku. Since it takes care of so much of the deployment stuff, I just quick worrying about DevOps all together. It takes me about 5 minutes to get a Rails/Javascript app working on my workstation and it seems a little silly to bother with Chef or Puppet for that.

From the sounds of the discussion on the podcast, Chef has come a long way since there. There is both a community and a company behind it. Which is good to hear. About ten minutes in they started to discuss something that is having a profound impact on my relationship with my brother – Cinderella

It only takes me a few minutes to get my development environment up and running. My brother is my designer. As you might expect he runs a mac. Actually, he runs a lot of macs. He has a mini at home, a macbook in his bag, and a mac pro at the office. It is not bad enough that getting all the open source stuff onto his box is a pain, but keeping all three boxes working is a special form of hell for me. It turns out I’m not the only dev out there who hates having to support a designer on a mac.

That’s where Cinderella comes it. It is built up on homebrew and chef. Officially, to get running you just do the following on a mac:

curl https://github.com/atmos/cinderella/raw/master/bootstrap.sh \
-o - | sh

You wait for it to complete (about an hour) and then you are off to the races.

We ended up doing a lot of experimentation with this so let me help avoid some serious pitfalls:

  1. Update X-Code!. The first time we installed Cinderella was on a box we had just installed Snow Leopard on. It turns out that there is an X-Code update you need before Cinderella will work.
  2. Update Rubygems. Apple is trying to be helpful by shipping macs with a bunch of gems pre-installed. They are all out of date. The most important one to fix is RubyGems itself.
    sudo gem update --system before you start the Cinderalla process or you will run into problems installing bundler later.
  3. You can do this on a mac that isn’t freshly installed. We did it once on a freshly installed macs and then switched over to installing in on his macs that already had other software installed. The key is to get rid of fink/macport, /usr/local, rvm, and any other open source software that you installed in the mac in some other way. You could probably keep them around, but it will end up being a lot more confusing so clean out before you start installing
  4. I put in a bug about this – but if you are using .rvmrc files in your project – add .rvmrc file to the person’s home directory with:

    export rvm_install_on_use_flag=1
    export rvm_gemset_create_on_use_flag=1

    That will allow rvm to automatically install stuff and will be a lot less confusing for the mac user.

It takes a while for everything to come down, but there is something truly amazing about the fact that you get all of this stuff just installed and working for you. As a bonus, the script can be re-run again to update things. Also, everything it put into ~/Developer so getting rid of it is a lot easier if you decide to bail out.

The only thing I need to figure out now is how to choose which version the software installs (since it chose Postgres 9.0 but to mirror Heroku it needs to be 8.4). That’s a pretty small problem compared to how much effort it saved me.

So if you have a designer and you’re stuck supporting things like ruby,git, mysql, postgres, or node.js – give this a try.

Hibernation for Winter

I have run Ubuntu on a laptop for a long long time. Through that time, I’ve had hit and miss support for hibernate and suspend. Sometimes it never works on a given laptop. Sometimes it starts out working and eventually it just stops working.

Most of the time I have just chalked this up to a rough edge that will eventually be smoothed out for me by the fine people who work on open source. It doesn’t help that about the time things get good, I end up upgrading to a new laptop that has a new list of unsupported not-yet-supported hardware.

My most recent upgrade was to a Toshiba Protege r705. I wanted something more powerful than a netbook, but since I’ve switched back to workstations for development, it didn’t have to be a 10 pounder.

It is largely supported, but hibernation has been a disaster. It claims to hibernate (which takes a while),and restoring from hibernation takes so long I have a hard time imagining something hasn’t crashed. It has gotten so bad, that I did something I hate doing. I waded into the Google on the matter. After some searching I ended up at the Ubuntu forums. I’ll admit I’ve found more than my share of solutions in forums, but only after a thousand dead ends. All too often I find the a forum post with my exact problem and 10 pages of people who never seem to discuss the solution.

In this case this thread proved very useful.

To save you the trouble – here are the steps I followed:

  1. sudo apt-get install hibernate laptop-mode-tools
  2. edit /etc/default/acpi-support – add “hibernate” to the list of supported methods
  3. edit the /etc/hibernate/suspend2.conf
    • Change Compressor lzf to Compressor lzo
    • Added in some services to shutdown/startup RestartServices mysql postgresql-8.4
  4. Manually replaced a script to force hibernation
    • cd /usr/sbin
    • sudo mv pm-hibernate pm-hibernate.orig
    • sudo ln -s `which hibernate` pm-hibernate

After that, I can actually hibernate in a reasonable amount of time (with a nice little screen report of how close to being done). The best part is booting back from a suspend happens fast. Fast enough to make it worth it to me to hibernate again.

Nothing in here feels very Toshiba specific, so if your hibernation sucks – you can try this as well.

Getting the Most out of Github Issues

For the longest time, I ran my own git server. I got a github account when I decided I wanted to start sharing code publicly. Then when I got serious about a startup, it just seemed like the obvious choice to host the code for our product since our team is spread out (San Antonio, Chicago, and Warsaw).

GitHub has a lot of features beyond just simple repository hosting. The social aspect of being able to not only fork, but use that as a means of a communication is really powerful. I’ll admit that much like I think better of an application that ships a deb, a project on github makes me more comfortable mixing it in to my tool kit.

The tool that took me the longest to the hang of is the Issue tracking system built into github. It is a little slow (given how much work they’ve done on the Wiki feature I’m hoping this is going to get some love), but other than that it is really pretty awesome.


If you are going to use the Issues tracker, you need to make sure that everything you do fits with its purpose.

The purpose of Issues is to provide a single place to store possible development work, and to manage what work is going to be done.

That gets broken down into two parts. The first is as a single place to store possible development work. When I originally started working with github, I used the wiki a lot more than Issues. The Wiki just seemed friendlier to the format and creation of stories. We would pick work from there and then use it to track everything. There is a distinct possibility that you could make that work, but what inevitably happened was that as soon as we started working on a story it got transfered to a issue. That just made it a heck of a lot easier to track the workflow on the story. Over time, I stopped putting stories in the wiki at all, and just went straight to Issues.

Even though it should be the single place for storing possible work, it does not mean it is the only place you should take feedback. At TruckingOffice, we use Zendesk to deal with customer issues. One each customer ticket that gets submitted, it gets triaged. If the problem they are having needs to be solved by code changes, then a new issue is opened in github.

I’ve experimented in the past with allowing tickets to flow directly from customers to developers. Unless your customers are developers, it just ends up in a really messy communication flow since through no fault of their own – customers rarely capture everything you need to know if their first submission, and for really bad issues you get a lot of duplicate submissions. The goal is to put issues into github in a form that allows a developer to take action on the problem. In the case of a story, you need to give a full statement. In the case of a bug, that means some guidance on how you detect it and if possible how you replicate it.

There are also a couple of practical issues with having customers put tickets into github. They need a github account, which if they already have one says something about what kind of person they are. If your repository is private, then I believe you would have to add them to the project which is going to be expensive and unwieldy unless you only have a couple of customers.

The point of all of this is, you can use lots of different tools to gather what ideas about what needs to be done. The most important thing is to make sure it all gets into the issue system. Maybe it was because I love lists, or because I spent so much time at Rackspace, but my first question once an issue gets brought up is – “did you start a ticket for this?”. Get in the habit of opening issues. It is much much easier to close an issue if it exists than if it is just something you talked about at a meeting.


Now that you have all of you work in one place, you get to the second key of using Issues – manging what you are working on. If you are like me, you start with just dumping issues into the system. You start by just working on the first in first out (or more likely last in first out). Once you get enough issues, you start messing with priority and trying to keep the stuff you want to look at at the top. After you cannot keep up with that, you might finally glance over at the right hand side and notice a thing called “Labels” which is just a way of tagging your issues.

Now I’ll admit that the label system is not perfect. For example you can filter the open list down by only one tag at a time, but there are some easy ways to work around that problem.

Here is how I setup my labels. I’ve included a list of my standards as well as my color code. You can always add or subtract to the list. You can also choose different colors. I recommend you use colors because it makes it much easier to get a sense about the issue you are looking at. Github allows you to tag an issue with multiple tags. Make use of that. Because of the issues with sorting, it is really really essential that you tag an issue with all the tags that apply to make sure it will show up in the list as you sort it different ways.

  • Bug: Color: red. This should be the most obvious category. This issue is about something that is broken.
  • Deploy: Color: bright blue. In small teams, one or more of the devs usually gets stuck with the deployment job (or more accurately when I’m on a small team I get stuck with it as well as development). Keeping track of deployment issues in the same place as everything else makes release planning a lot easier. It also smooths things out if a deployment issue needs dev work.
  • Feature/Story: Color:black . You can use either term. On official projects I tend to use the term story because I tend to be more disciplined about actually writing stories (As an X I want to Y so that I can Z). On my personal projects, tends to get a little more free form. Any time you see something in the issue that means you are making an improvement add this tag. It will help you write your change log later for your customers.
  • I-1 or I-n: Color: white,grey, bright green. Color depends on the stage. They are default grey when created. If they are completed I make them white. If one of them is active I make it bright green. The I stands for Iteration. I use a number once it is actually in use. I use an n as a place holder. I have bastardized the term iteration in this usage. Basically, I use these tags to collect up issues that follow a theme. For example – user authentication or integrating with google. When it comes time to schedule work, I can actually choose a bundle of issues that are thematically related. Or in some cases, ignore a bundle of issues because their overarching theme is not that valuable yet. I try to have only one be bright green. That way I can see what is currently in play.
  • Question:Color: pastel blue. This is for any issue that needs further discussion. You could have questions for the person who submitted it or you need to talk to other devs about how to solve the problem. This tag makes it easier when you are looking at the list to see issues that need some kind of external input separate from code.
  • R-n: Color: white,grey, bright green. This follows the same pattern as the I tags. White for complete, grey for created, and green for active. They get a number depending on the release you are on. You can tag any issues for iterations that are part of the release with the release as well. I also use it as a tag to see where things are at in terms of bugs and other issues that are coming up before and after a release. So for example, you can tag all of the deploy/bug/features that are part of the next release with this tag. That helps track the punch list building up to your next big push out. Afterward, you can use it to collect bugs that come in after things have gone out or any issues that don’t have a clear theme. Once things have calmed down, you can move on to the next release tag and continue planning.
  • Ready To Test:Color: yellow. I use this when an issue has been submitted and I think it is done, but I need some kind of confirmation from the person who submitted the issue. This one became more important when I was working with remote developers. I can go in an look for this tag and then look at the tickets that I opened to see which ones I need to verify. If things look good I can close the ticket. If they don’t I remove the tag and add a comment, and it will end up back in the queue for work.
  • Technical Debt:Color: Orange. I did say keep all possible work here right? Technical debt is one of those things that is often discussed, but not often integrated into the work plan. Instead of burying it in a wiki page, or worse just letting it float in developers heads, put it in into the system. I find this really liberating. When I commit some kind of technical sin, I open a ticket and flag it. It allows me to stay focused on the feature without a ton of guilt. If I find a way to work on it after my feature work is done, fine, but this way it is captured and can be added to the schedule. You would be surprised at how much more of this stuff you would clear out if the list of what needed to be done is explicit.
  • Unsorted:Color: Dark Grey. I’m still playing with this one. When I add a feature and I do not see an iteration or release tag that is appropriate, I mark it unsorted. To me it means either this is a one off issue that needs to be put into a release, or it is the beginning of a new theme. It basically a reminder that things with this tag are probably never going to be worked on until some one organizes them.
  • Ux:Color: Bright Purple. This stands for User Experience. It was originally UI (User Interface), but it got broadened. You could conceivably have both. Mark it UX if it is related to either how something looks or its work flow. So if buttons are drawn in the wrong color, or there should be a confirmation on a delete. I break this out because a lot of the time, I partner with my brother who handles these kinds of issues. That’s also the reason for the color since he loves purple. Sometimes this serves as a warning to me that the feature or bug I’m taking is going to be more than just a quick codefest. It may require some styling or more thought about how to make something less confusing for the user.
  • Userid:Color:grey. Github Issues doesn’t let you say who is working/responsible for a ticket. The easy solution to this is to create a label with the userid of each team member. When you take a ticket, label it with your name. That makes it easy to see what things are unclaimed and who to talk to about the work.

Here is a sample from one of my projects. I’m just finishing up the first release – which included I-1,2,and 3.:



You may have to tweak some of my suggestions to fit the size of your team and your workflow, but at least now you have a starting point to take one of the features of github that is all to often squandered and turn it into something that enables you to keep focus on what you need to get done.

One Side Note

For those of you who spend as much time at the console as you do the web browser, github actually has a complete REST API for Issues. I ended up making heavy use of it when I migrated from another wiki in to github. I basically just created a csv in open office. Then I used their API to push all the information into Issues for me. That saved me a ton of time.

If you want something a little more flexible you can check out ghi. This is a command line tool that wraps all of that API. It give you another path to interacting with Issues. For example, I use ghi to make it easier to create technical debt tickets from the command line so I don’t get distracted by going to the website or leaving my work. A little bit helps.

Hanging on to Lucidity

I ended up reinstalling my home workstation over the weekend. I did it for all the wrong reasons it turned out. Mostly because the problem I needed to solved ended up being in my gnome configs instead of the system as a whole. But in the process, I did something I don’t think I ever done. Maverick (10.10) has been out for a few months. I went and looked at all of the articles about what new things it brings to the table. After reading them, I decided to stick with Lucid (10.04). Partially because as an LTS it is stable and partially because by staying here I won’t be forced to upgrade every other box I use. (Yes I know I don’t have to but I’ve learned it ends up being annoying when you have different software on different boxes – best to keep them in sync).

I’m super excited about Unity. I won’t miss X when it goes away (assuming they can get Wayland to work). While I’m waiting, I did learn a couple of tricks to improve my Lucid install while still avoiding Meerkat.

Since I’m using an SSD as my boot drive, you need a newer kernel than the one that ships with Lucid to get TRIM support.

They are currently offering the Natty Narwhal as a backport for Lucid.

sudo add-apt-repository ppa:kernel-ppa/ppa
sudo apt-get update
sudo apt-get install linux-image-generic-lts-backport-natty linux-headers-generic-lts-backport-natty

Once you switch over you can add discard option to your fstab to get TRIM to turn on.

The only pain I experienced was the Nvidia drivers for my card stopped worked because of the newer kernel. I downloaded the Natty packages for that


and they installed like a charm. I’ve even survived some updates to the kernel without losing X (Which is a nice change of pace really). Now they just need to release that 200 line patch to the kernel that is suppose to make the desktop experience even more awesome.

Vim vs Rubymine?

I’ve been in the VI/Vim camp for a long long time. When I got started with Unix (I started out on HP-UX – but don’t worry Richard showed me Linux and I’ve never looked back.) you had two choices – VI and pico. Pico was a simple text editor. If memory serves it was developed to be the text editor with the pine email system. As a consequence, it had one feature that could make it a nightmare – it automatically wrapped lines. (as pointed out by Richard you can avoid this fate by remembering to add a -W. It didn’t help that for licensing reasons – there was/is no pico for Linux. They have nano now – but by the time that was standard I’d paid my dues) That’s pretty desirable for email, but terrible if you are writing code or a config file. I ended up learning the basics of VI.

I have a friend or two who swear by emacs. It was has always been on my list of things to spend a week really learning, but I’ve just never gotten there. The reason is simple. Every Linux box ships with Vim. It is right there waiting for you. (Ubuntu’s default is a little crippled but I’ll get to that). So if you end up modifying files on remote machines (which was once a regular thing for me) it is an essential skill.

If you are going to use Vim – there are a couple of things that I couldn’t survive without. Namely splitting and search & replace. Splitting is easy to explain – you are able to split the terminal in half (or less depending on how many things you keep open at once). You can even split the file you are in so you can look at two different parts of the same file. That ends up being pretty useful when you are dealing with code, or you are looking at a config file that should be similar to another block in the same file. Keyboard command is

ESC :sp

If you need to switch between “windows” there are a bunch of ways (like everything else in Vim) but you can stick to Ctrl-W to rotate through them.

Several of the people I’ve worked with complained most about my constant opening files, splitting, making changes, and then closing files before they knew what files I was opening let alone what I was changing. Once you get the hang of it, it is awesome. Unless of course you are on an Ubuntu box that has vim-tiny. Vim-tiny doesn’t support splitting. It drives me insane and is the first thing I upgrade when I touch a box.

Which leads to other essential skill – search and replace. There are probably books written on this subject. I won’t repeat, but I will show you some things that I end up doing a lot.

%s/\(.*\)/mv \1 \1/g

That says for all lines (%s – s means a single line) take the contents of the line and replace it with mv (the contents of the line) (the contents of the line). I end up using this to build a shell script to rename files en mass. You can do more than this – but it gives you a taste. And if editing a single file isn’t enough you can do the old

vi `grep -rl pattern *'
:argdo %s/pattern/new_pattern/ge |update

That builds up a list of files recursively and throws it into Vim. Then the argdo lets you do a search and replace across all the files. You can do something similar with :args but I can never remember how to make it recursive.

Over the years I’ve programmed in a lot of different languages. I always seem to move on to the next one just as the tools for the one I’m in start to get good. So I’ve never really used an IDE (Integrated Developer Environment) as part of my work. I couldn’t afford them when I did C/C++. They didn’t exist for PHP or Python when I wrote in them. The Ruby on Rails community was completely dominated by Textmate (and the mac fanboyism that went with it). But since then things have improved a lot. There are even choices now. I originally got exposed to RubyMine when the developers on the team were starting to move away from Textmate. They seemed to like it. I didn’t think much of it since I had Vim. Then later when I was working as a developer on TruckingOffice in a pair, I got an ultimatum. I had to use a tool in common with the other developers or I wasn’t going to get to stay in the pit. I guess I split files one too many times. I reluctantly switched over to RubyMine. As time has passed, I’ve come to really like it. I have even been playing with the community edition of IntelliJ since I’m experimenting with Android development.

It has more features than I could shake a stick at, but there are two that have become essential to my sanity. The first is focus tests. You can point at a single example in a spec file and have it run. This saves me a bunch of time when I’m working on adding a feature or fixing a bug. At one point, I spent a week learning the internals of Vim enough that I could add a plugin to do the same thing. I got something to work, but it is no where near as elegant. Especially because of the other feature that I’m hooked on – debugging.

In the past, debugging to me meant adding printf/echo/puts to the file that was causing the problem. Rubymine has an excellent debugger integrated in. And if a test fails, you can easily add a break point and switch from running the spec to debugging it. Between RubyMine’s debugger and Chrome’s debugger, I’m left wondering how I programmed so long blindly.

I would love to say that now I do everything in RubyMine. I guess old habits die hard. I do all my coding, and testing there. I still end up with terminals running tasks (which I know RubyMine can do) and opening files from time to time in Vim because in some ways it is just how I think about it (even though RUbymine has enough keyboard shortcuts to satisfy a six armed man).

So if you are a system administrator – get learning Vim. Add a new command a day and in ten years you’ll have mastered it. If you are Ruby dev, I highly recommend RubyMine. I’ve resisted a lot of changes to how I develop, but this one is worth the learning curve.

    Stuff I want to read

    Shelfari: Book reviews on your book blog

    Stuff I've Read

    Shelfari: Book reviews on your book blog
    You are currently browsing the Economy Size Geek weblog archives for December, 2010.