Ok so I’ve been playing with VM Server lately (Who isn’t?) - I’ve installed in on a test box - and was going to admin it from my primary workstation.

In theory - this is dead simple on Feisty

add this to your /etc/apt/sources:

deb http://archive.canoncial.com/ubuntu feisty-commercial main


Then go get a SN from their web site and do


apt-get install vmware-server vmware-tools-kernel-modules


That’s when I hit the wall -you see the deb expects eth0 in the install scripts. For some strange reason I don’t have an eth0 on my workstation. It was driving me crazy - then I learned about /etc/iftab

This little file allows you to put in a label and a mac address and you can then assign that label any time that card is up. It looks like I don’t have an eth0 because it was being reserved for my old network card (which is gone and never coming back). A quick edit to the file and a reboot (because I’m lazy) and everything is fixed.

Kind of useful - but a shame it is in such a strange place…

If you’re in the US - or your servers are - you need to make sure that you are ready for the new Daylight Savings Time (DST) changes. It turns out this was already handled by an update to Ubuntu.

You can easily confirm it just by running the following command:

sudo zdump -v /etc/localtime |grep 2007

Assuming you see March 11th and Nov 4th you should be good to go.

Original Thread

I’m about to be without a sys admin on my team. That totally sucks for me :(

That means that I’m stuck doing a lot more sys admin work than I would like. Normally this wouldn’t bother me, but right now I have a lot of development projects in play. They all have different deployments and it means there are a lot of details to manage. In the long run, that translates to making it harder to get someone to take it over when I do get a new sys admin.

So in the interim - I figured I’d take some time and get things a little more ship shape. Enter Cfengine. Basically, this is a tool you can use to audit, manage, and maintain servers under your control. In my case, I’m going to be focused on a set of machines that will operate as a Ruby On Rails cluster.

Although my final deployment will be focused on doing multiple web heads and replicating database servers, this series of posts is going to focus on the simple two server configuration.

web1.ubuntu.com - The web server
db1.ubuntu.com - The MySQL database server.

Please note I don’t own ubuntu.com. But if you following the examples you’ll be setting up a hosts file to make resolution work - so it doesn’t really matter. I have no idea how to do that on Windows or Mac OSX - but I’m sure someone will figure it out.

This is part one. It will deal with the pre-setup. Part two will focus on some of the most basic cfengine stuff. Part three will show it all working together so you can actually “rake deploy” to your little virtual cluster.

Read on for more details…
Read the rest of this entry »

Ok - don’t get me wrong I still like Debian - I just do almost everything in Ubuntu. This statement is so classic I had to blog about it.

Linux.com | New installer gives Debian Etch an edge
I

f it’s true that Ubuntu is a Swahili word meaning “user too dumb to install Debian,” then I have to wonder if I’ll still be running Ubuntu when Etch gets released, because even I can install Debian now. If installing Debian has intimidated you in the past, keep an eye on Etch and its installer as they get nearer to launch time. They won’t intimidate you.

Apparently - the install is what is holding Debian back. That’s it. I mean now that they have a GUI installer - you should throw away Ubuntu and go back to Debian. I thought that wrong headed view of the world went out with a single RedHat distribution.

The reality is that it is not the installer that makes or breaks Linux. Seriously! I spend a half an hour yesterday helping a co-worker get his wifi card (Man I hate broadcom) working under Dapper. He had tried and tried and made no headway.

This is a classic Linux problem for new users.

You have some hardware and it doesn’t work. You don’t know why it doesn’t work. You don’t know how to make it work - just that it doesn’t work. Guess what - that sucks! You add on another couple of problems where you expect it to do something that is easy under Windows and eventually they give up.

So I’m excited that Ubuntu has lit enough of a fire under Debian to make a release in the sub two year release cycle. That’s good for everyone because it means that the foundation that Ubuntu is built on is still healthy and thriving. But - you are missing the entire point of Ubuntu if you think that the installer is the only difference between Debian and Ubuntu.

Every now and then I end up adding in a repository for an apt and I get the error

W: GPG error: http://kubuntu.org dapper Release: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY A506E6D4DD4D5088

Basically it means that you don’t have the gpg signing setup for this source.

Don’t worry it’s easy to fix.

Basically you take the last 8 digits of the code provided - so for this error the key is - DD4D5088

Now you just run the following command
gpg --keyserver subkeys.pgp.net --recv-keys DD4D5088| gpg --export DD4D5088 | apt-key add -

Once you do that the signing relationship is setup and the error should go away.

I know I’ve used in a couple of recent posts - but especially since I’m on Ubuntu (and they tend not to be with the installing dev packages) - the command

apt-get build-dep [package-name]

Is seriously awesome! I’m not sure how I got through my early Debian life without using it (probably because I always said - install everything).

Basically what it does is install all other packages you need if you are going to rebuild a package from source. This ends up saving a lot of time since you then know that you have everything when it comes time to compile.

You probably already knew about it - but just in case…

Ok call me crazy - but I’ve had bad luck with hardware based RAID (especially SCSI based). This problem is probably made worse because I end up sitting on a server for a long time. Which means when there is a problem the same equipment can be hard to find. All of that goes away when you use software RAID under Linux.

To be honest, it probably has a lot less to do with my bad experiences so far as much as it does with my good experiences with software RAID. I’ve been running software raid for a long time and it has always served me well - even when things looked really really bad. At some point I got the bright idea to go for the real deal and having just survived a close call with data death, I’m going back to my roots.

So now I have a new server up in a datacenter somewhere. It’s running Debian Sarge. It is currently booting off of the primary IDE drive. Now the test will be getting it to book off of a software RAID setup without touching the server physically. Once that is working I’m going to migrate over my old machine to this new server and call it a day. Last time I did this I think I had to go to the dc at some point, so here goes.

Full disclosure - I started down this path and it turned out badly. Eventually I had to have the server completely re-installed to try again. The directions below try to capture how I actually got it working.

Gettting Started

First things first - make sure that you have your deb box up to date. There is no point to not having the latest everything before you start working thru this. You will also need to add the package mdadm (which is the tool to handle all the raid stuff). In the process of updating, a new kernel got installed. So I’m rebooting before I go on or create any RAID partitions.

After a quick reboot…

Kernel

It turns out my server is a dual proc box, but the installer didn’t install an SMP kernel. I updated this. Then I used the instructions here to download the current version of the kernel (2.6.16-14 at the time of this article). (Also he has a nice article on how to use distcc to compile the kerne;. Using it cut my compile time in half - which is pretty impressive considering the fact that the server I’m compiling on is pretty fast to begin with.)

I ended up having to install

apt-get install kernel-package fakeroot libncurses5-dev

Then I just followed the instructions in the article. In the end, I got a custom kernel which has RAID and ext3 statically compiled in. The first time I tried to do all of this I didn’t do that - so I’m hoping this is the key to solving my problem.

Great - that was easy - now to get some work done. As per normal recomendation - I have 2 250GB IDE drives on my server. Each one is on a different IDE channel (To maximize performance) so hda/hdc in Linux terms.

Another quick reboot to make sure lilo works

Drive Prep

Before you do this use fdisk to make sure that the drives are actually the same size in terms of bytes. I’ve had drives from the same manufacturer that weren’t and it messes things up later. In this case both my drives are 8225280.

/dev/hda is the main drive
/dev/hdc is the seconday drive (where the RAID will be intially installed)

I’m going to be making 4 partitions. Approximatly:

hdc1 50M /boot
hdc2 1GB swap
hdc3 48GB /
hdc4 185GB /data (basically storage for /var /home)

I could have gone with a single / but on my home file server I’ve had odd problems when the partitions got large.

Also the file system type for all other patitions partitions it is fd (RAID - autodetect).

So the actual output looks like this

/dev/hdc1 1 7 56196 fd Linux raid autodetect
/dev/hdc2 8 138 1052257+ fd Linux raid autodetect
/dev/hdc3 * 139 6364 50010345 fd Linux raid autodetect
/dev/hdc4 6365 30401 193077202+ fd Linux raid autodetect

RAID it!

Now that you have the first drive partitioned we need to set up the actual RAID array. Since we are currently using hda to boot off of we will be seting up hdc RAID using a missing drive config and then adding in drives later.

In the event that the drives you are using have ever been used in a RAID array - use the following commands to wipe out the old info

mdadm --zero-superblock /dev/hda
mdadm --zero-superblock /dev/hdc

Here are the commands to setup this array

mdadm --create /dev/md0 --verbose --level 1 --raid-devices=2 /dev/hdc1 missing
mdadm --create /dev/md1 --verbose --level 1 --raid-devices=2 /dev/hdc2 missing
mdadm --create /dev/md2 --verbose --level 1 --raid-devices=2 /dev/hdc3 missing
mdadm --create /dev/md3 --verbose --level 1 --raid-devices=2 /dev/hdc4 missing


Putting missing second is very important!. There is a bug in Lilo that prevents it from installing if the first drive in the array is broken. This causes other complaints from lilo because it doesn’t like that hdc is not the first drive but at least it works.

You can confirm that everything is working by


cat /proc/mdstat



Personalities : [raid1]
md3 : active raid1 hdc4[1]
193077120 blocks [2/1] [U_]

unused devices: <none>


Now format the drives (from here on out you just deal with the RAID devices)


mkfs.ext3 /dev/md0
mkswap /dev/md1
mkfs.ext3 /dev/md2
mkfs.ext3 /dev/md3


It is not a bad idea to generate a config file to store all the info about the arrays that have been created. Later you can update the file when you add new devices.

Create /etc/mdadm/mdadm.conf - make sure it has the following line at the top

DEVICE partitions
Then update it with the following
mdadm --detail --scan >> /etc/mdadm/mdadm.conf

Mount & Copy

Now you need to mount the arrays to copy over all the data.

cd /mnt
mkdir md
mount /dev/md2 md
cd md
mkdir boot data
mount /dev/md0 boot
mount /dev/md3 data
mkdir data/var data/home
ln -s data/var var
ln -s data/home home


Now you have the RAID devices mounted. Time to copy. Normally I use rsync but I saw this and couldn’t resist (Found it here in another article about BOOT RAID):


cd /mnt/md
tar -C / -clspf - . | tar -xlspvf -
cd /mnt/md/boot
tar -C /boot -clspf - . | tar -xlspvf -


Update the /mnt/md/etc/fstab


/dev/md2 / ext3 rw 0 1
/dev/md3 /data ext3 rw 0 1
/dev/md1 none swap sw 0 0
none /proc proc defaults 0 0
/dev/md0 /boot ext3 rw 0 2
none /proc/bus/usb usbdevfs defaults
#/dev/fd0 /floppy auto users,noauto 0 0


Now modify the /mnt/md/etc/lilo.conf (The other two entries are actually fall backs in the event that raid doesn’t work)


lba32
prompt
boot=/dev/md0
install=/boot/boot.b
raid-extra-boot=mbr-only
map=/boot/map
default=linux-raid
append="console=tty0 console=ttyS0,9600"
serial="0,9600n8"
timeout=50
delay=20

image=/vmlinuz.old
label=linux.old
initrd=/initrd.img.old
read-only
optional
root=/dev/hda3



mount -t proc /proc /mnt/md/proc
lilo -r /mnt/md


Now make sure it gets put on hda

Back up the lilo

cp /etc/lilo.conf /etc/lilo.conf.original
lilo


cp /mnt/md/etc/lilo.conf /etc/
lilo


Now in a perfect world You would reboot and partition up hda and use


mdadm --add /dev/md0 /dev/hda1
mdadm --add /dev/md1 /dev/hda2
mdadm --add /dev/md2 /dev/hda3
mdadm --add /dev/md3 /dev/hda4

There are two more parts that need to happen. This part is a little fuzzy.

Basically use fdisk to mark hda3/hdc3 as bootable.

Based on some instructions from here

You build a special initrd.img and use that too boot off of.

The reason I say fuzzy - is basically I didn’t do these last two steps the first time around. The system ended up not being able to boot and things got very complicated. I got special access and was able to fix the machine. Normally I would have started all over again and made sure everything worked, but after spending a weekend of working thru this stuff - I figured it would be a bad idea to trash a server that was finally working. So if you go thru this and figure out a better way let me know for next time.

Fine I admit it - I’m one of the application developers that true Sys-Admins hate. Yes - I admit it - I don’t care about security. Ok that’s probably too strong a statement - I do care but I will admit that it’s not often the first thing on my mind.

I’ve been working on making sure that Cross side scripting and SQL injection don’t effect my rails apps. But my server is another matter.

I mean I try to keep it up to date. Thanks to sarge I’ve been able to stay on the straight and narrow of released (All the other servers I have end up running Debian unstable).

That being said I ran across this today:

Preventing SSH Dictionary Attacks With DenyHosts

I’m a big fan of HowTo Forge already - since they have a number of articles that helped me get things sorted out that I’m sure I could have figured out eventually but it’s so much nicer to just be able to sit down and get something to work and then start tweaking as oppsoed to my normal method - which is to spend a lot of time just getting it to work.

The basic idea of the tool is that it tracks bad login attempts and just shuts out the ip address. Seems straight forward enough. Just the kind of security I like - make sense on it’s face and isn’t terribly difficult to install or remove. Maybe I’ll get the hang of this Sys-Admin thing some day…

Ok so I have already converted my workstation (and my fiancee’s) over to Ubuntu. That leaves my laptop and my home file server.

You may be asking yourself why I’m converting all my boxes. At the moment - it’s really just for sanity. You see I run SID on everything because it’s the only way I can keep up to date. Unfortunately that ends up breaking from time to time (especially if I don’t update all the time). Ubuntu is hopefully going to get me off that crazy train. Dapper Drake is going to be out April 10th. Breezy just came out and they already have the release date set. I guess I just kind of like that.

Some people seem to be nervous about Ubuntu on a “server”. I end up running Gnome on this “server” anyway so I don’t know that it is much differnt from a normal desktop other than the fact that it is running software RAID. Underneath Ubuntu is Debian - so I would think they would have to do something terribly stupid in order to dimish the server component.

I originally tried to do some of the apt-get update tricks to migrate over (Which if you are on sarge you may be able to do that - see the bottom of the article) , but since all my computers were on Sid it turned out to be a lot more complicated than I thought. It’s ok because I made backups before I started so I can always go back.

I tried to do a very complicted solution to my probelm involving debootstrap. Which is a pretty cool tool, and FYI it natively supports ubuntu. Well about step 10 of the process I hosed the box. Basically I moved a directory I shouldn’t have and lost even the most basic commands - like ls/mv etc.

So I busted out a cdrom drive and hooked it up to the file server. (Actually it turned out to be a dvd drive which I think I’m leaving in there now that I found it.) Anyway - I fired up the install cd for Breezy. I was really scared about doing the install this way because my files server runs software RAID1. It even boots of of a raided partition. It was a pain to setup in the first place. I didn’t look forward to having to go through that again. But I was prepared.

The screens booted up and I got the most wonderful suprise. The installer could see all of my RAID partitions. It let me work with them just like normal partitions. I was even able to switch to the console, move all the old data on the root partition to a directory called /old and continuing the install.

I can’t tell you how awesome this is - I mean it is so nice to beable to just put in a cd and not have to figure out what to do with 400 GB of backups and data, but at the same time be able to install a new linux distro.

Now I know this feature is probably available now for Debian too - but for me it is just another nice thing about Ubunutu - getting it up and running is dead simple.

Actually when I rebooted I got a kernel panic. I was thinking - damn it - I spoke too soon again! But it turns out that the problem was that I forgot to move all the old kernels out of the way on /boot and the installer defaults to the first kernel it finds when it reboots - which in this case was 2.4.1 which was none too happy about my setup :)

Also in the process of getting every up I found a this site which has some nice grub splash screens for startup. - One piece of advice - download whatever pictures you want - the copy one of them to /boot/grub/splash.xpm.gz - then run update-grub. This will automatically link in the image and things will work nicely.

If you need more you can go here

Now I just have to get everything configured the way I like it -but that should be the easy part!

Alternate Solution

If you are using a released version of debian you might be able to do this: (from : Here) It didn’t work for me - but it did for other people.

/etc/apt/sources.list :
----------------------------------------------------------------------------------
deb http://archive.ubuntu.com/ubuntu hoary main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu hoary-updates main restricted universe multiverse

# If you want fresh backports:
#deb http://backports.ubuntuforums.org/backports hoary-backports main universe multiverse restricted
#deb http://backports.ubuntuforums.org/backports hoary-extras main universe multiverse restricted
#deb http://security.ubuntu.com/ubuntu hoary-security main restricted universe multiverse
----------------------------------------------------------------------------------

/etc/apt/preferences :
----------------------------------------------------------------------------------
Package: *
Pin: release hoary
Pin-Priority: 1001

Package: *
Pin: release o=Debian
Pin-Priority: 200
----------------------------------------------------------------------------------
This downgrade all to "hoary" release ;-)

apt-get update
apt-get dist-upgrade

Ok I admit it - I’m hooked on NPR. It ends up being the only thing I listen to in my car. They stream my local station on-line here. (it’s actually handled by a company I know pretty well) One small problem - it is only available in Window Media Format. That wouldn’t be a big deal expect for the fact that I spend most of my time booted into Debian on my computer.

Enter Princess Leia. This small how-to got me up and running with Mplayer in about 5 minutes. Once it was all installed I can actually stream the radio under Linux and I’m back in business.

It may be a small victory - but I’ll take what I can get :)