As unix and internet specialist, I'm maintaining servers for ISP's and other companies. Most of the machines I see when installing them and hardly ever thereafter. They usually are placed at the premises of my customers and I do remote administration via the internet. Some machines are located thousands of kilometres away.
Recently a severe hardware failure on one server caused filesystems to be badly damaged. The machine needed to have the harddisks replaced and be reinstalled from scratch. Not a big deal, but a big distance: I'm located in Switzerland and the machine is standing in Japan. While the people at the customer's site do have experience with PC hardware repairs, they don't have the necessary knowledge to setup a unix system. And anyway, I prefer to do the installation myself for machines, which I will be administrating.
While I sure would have loved to get on a plane and spend a week in Japan, this would have meant sending quite a hefty bill to the customer. So the magic word for once was not 'remote administration', but 'remote installation'.
I usually examine the machines carefully when I have the chance to get to them physically and keep a good logbook for each server. So I know quite well what hardware is being used, what software is installed on it and how it is configured. When doing remote administration, it is also good practice to have access to at least a second system at the site. This can be an old, low powered system. The further away the customer's site, the more important is this alternative system.
In this case, the machine that had crashed was a DNS and mailserver, running FreeBSD. This site also has a backup system which I'm responsible for, so this backup machine is at the same time the all important second system. With this system up and a little bit of help from the customer, I was able to completely reinstall the crashed machine. (no paid flight to Japan, sniff, sniff)
What is needed for such an installation from a faraway location? Exactly the same as for the installation locally: a target machine without hardware problems and the software to install. Plus some way to access the machine.
The damaged harddisks were replaced by the customer. No big deal: formatting or anything the like was not necessary, neither any changes to BIOS settings; a FreeBSD installation can be made ignoring what the BIOS thinks about the harddisk settings.
FreeBSD and other unix variants usually offer more than one way to install the OS. FreeBSD, for example, can be loaded from CD, disk, tape, via NFS mounts or FTP, and more. I was going to use FTP and, having sufficient diskspace, transferred the necessary files to the second machine at the customer's site. This way the files could be transferred in advance during the night. So for the customer the additionally generated traffic on the internet link would not cause any noticeable degradation of internet access speed. And also the installation would be much faster, as the data could be fetched from the local network at full ethernet speed.
The two boot floppies I prepared on the backup system. I asked the customer to insert a floppy into the floppy drive of the backup machine, created the first boot disk, asked them to insert a second disk and created the second floppy. It sure is not as quick as doing it all by yourself; e-mail must be sent to say 'the floppy is in the drive' or 'I created the first disk, please take it out and insert the second one', but hey, it does work.
Now, how to get the screen output from Japan to Switzerland and the keyboard input from my notebook to the server? Fortunately unix knows what is called the console. This is the main control of the machine. When nothing else goes, you can still control the machine via the console. When available, the console is usually accessed via keyboard and monitor but it can also be configured to be accessible via a serial port. So a null modem cable from the backup system to the target server did the trick. And access to the backup system could be made via the internet.
With the new harddisks installed, the null modem cable in place and the boot
disks created, the fun could begin. By default, the console is accessed via
keyboard and monitor. But when booting into FreeBSD 3.x, also from the installation
disks, you get, at the very beginning, a bar which is going round in circles.
It is made up of the signs /-\ and |. At the very
beginning, when this appears on the screen, the space key can be pressed to
stop the booting process and get a prompt:
>>FreeBSD/i386 BOOT
Default: 0:fd(0,a)/boot/loader
boot:
At this prompt, typing -h followed by the ENTER key,
will switch the console to the first serial port. From now on all output and
input is being done via this serial port. (Except for one more line of text
printed to the screen before the last part of the first floppy is being read.)
Instead of interrupting the boot process and type the command, it can also be
entered in a file called /boot.config: just a single line with
the two letters -h. There is enough space left on the boot disk to
create this file. If -P is used instead, FreeBSD will check the presence
of a keyboard and set the console to keyboard and monitor if a keyboard is available
and to the first serial port if no keyboard could be detected.
To prevent further unnecessary delays and e-mail messages, I instructed the customer to watch the floppy access light and once it went out, meaning the first floppy was fully read, insert the second floppy disk into the drive. From this moment I could take over with the installation, accessing the console on the serial port via the internet, the second machine and the null modem cable.
Installing FreeBSD is not very difficult. But for beginners The
Complete FreeBSD has quite good instructions on how to go about the
task. First you are given the chance to make changes
to the parameters used to autodetect your hardware, then you specify how you
would like your disk layout, third the components to install are being selected
and finally, 
after
the software has been installed on the harddisk, a couple of configuration question
have to be answered.
I have done this many times, usually sitting directly at the target machine. This time, doing it all using a vt100 terminal emulation over the internet and a null modem cable it was slower, but all the menus were the same and after a short time I said yes when being asked as to whether I wanted to boot the machine. Of course, the machine would not boot, as the second installation floppy was still in the drive. So I sent off another e-mail, asking the customer to remove the floppy from the drive and press the reset button. This having been done, the machine came up just fine, ready for further configuration.
There are several ways to backup a machine. Full backups are one way, incremental another and there are more. They all have their use. With many machines like the one being reinstalled I take another approach. 95 % of the files are placed on the harddrive during installation and never change. Another 4 % are additional software compiled from source and don't change either. Only about 1 % of all the files change and need to be backed up. These are on one hand logfiles, which are copied to another machine regularly and a couple of configuration files on the other hand. Even though the configuration files all could be created again from scratch, in some of them several hours of work can be invested. This investment is indeed worth protecting.
If the need arises, a machine can be reinstalled in a relatively short time, also the software needing compilation from source code does not take so much time. So I don't care too much about backup copies of these (except for the source code packages of the additional utilities), especially since I have several machines on the internet from which I easily could copy over any files deleted by mistake. For the configuration files, which usually are unique for each machine, I use RCS, which stands for Revision Control System. With RCS, all changes over the whole lifetime of a file are stored in a single repository. It is enough to backup all these repositories, which are identified easily by their name. This way the amount of data to backup is conveniently small and the logbook for the machine is not required to detail every little change to each file. So only a couple of MB are sufficient to backup a whole machine: the logfiles, the RCS repositories, the additional source packages and the logbook. With this it is possible to restore a whole machine even from scratch and it usually does not even take much longer than restoring from a full backup.
Here again the main points which saved my customer quite some money, saved me quite some time (and because of which I could not enjoy a paid holiday in Japan):
unix (FreeBSD)
consoles can be directed to the serial port access to
at least two systems when doing remote administrations
RCS to manage and backup configuration files a good and
detailed logbook for each machine