Beware of rsync -C option.

To my surprise, rsync reads .cvsignore file and follows the rule in the files! That's not a good thing to me since I wanted to backup necessary files, not less files. (If you know what I am saying.) So I should have known better and have read the manual!

-C, --cvs-exclude           auto-ignore files in the same way CVS does

Backing-up OSCAR 5 Data

According to the start_over script, the OSCAR 5 data could possibly be backed-up with the tar command:

tar cf /backup/oscar.tar /opt/kernel_picker /etc/systemimager /etc/dhcpd.conf /etc/profile.d/00-modules.* /etc/profile.d/c3.* /etc/profile.d/oscar_home.* /etc/profile.d/ssh-oscar.* /etc/systeminstaller /opt/env-switcher* /opt/lam* /opt/maui /opt/modules /opt/mpich* /opt/pbs /opt/perl-Qt /opt/sync_files /tftpboot/initrd.img /tftpboot/kernel /tftpboot/pxelinux.* /usr/lib/systemimager /usr/lib/systeminstaller /var/lib/ganglia /var/lib/mysql /var/lib/oscar /var/lib/systemimager /var/log/systemimager /var/spool/pbs /opt/oscar /etc/httpd/conf.d/ganglia.conf /etc/ssh

This can be useful when you want to change your head node or simply backing it up. (This entry is an imaginary solution to backing up the essential part of the head node/cluster server.)

Backing-up OSCAR 3 Nodes by Hand

Just do this on the computing node:
rsync -avxHS --numeric-ids --delete --exclude 'lost+found' --exclude '/tmp/*' --exclude '/var/spool/clientmqueue/*' --exclude '/var/spool/pbs/aux/*' --exclude '/var/spool/pbs/mom_logs/*' --exclude '/var/spool/pbs/mom_priv/jobs/*' --exclude '/var/spool/pbs/spool/*' --exclude '/var/spool/pbs/undelivered/*' --exclude '*.pid' --exclude '*.lock' --exclude 'swapfile0' / 10.0.0.254:/usr/cluster/nodes/oscar3/root/
and also this:
rsync -avxHS --numeric-ids --delete --exclude 'lost+found' --delete-excluded /boot/ 10.0.0.254:/usr/cluster/nodes/oscar3/boot/
Done. (Update: please be careful about the directories that has been ignored... I don't trust the exclude expression.)

Reimaging OSCAR 3 Nodes by Hand

Hard drive failure and irreparable filesystem are pretty common during the operation the cluster, the first thought that comes to us always is to re-imaging the pre-stored image from the server. However in OSCAR (at least in v3 as well as v4) you need to flush the OpenPBS setting or restart the PBS server. That causes a lot of problem if you have a lot of jobs running on the rest of the computing nodes. Here is my notes to reimage OSCAR 3 nodes manually:

1. pxe boot into fc2 installation untill load into gui(stage2)
2. alt-F2 enter the shell
3. To erase the previous partition, do:

   parted -s -- /dev/sda mklabel msdos
   parted -s -- /dev/sda print

4. Then build the partition table:

   parted -s -- /dev/sda mkpart primary ext2 0 24
   parted -s -- /dev/sda mkpart extended 24 76319
   parted -s -- /dev/sda mkpart logical 24 2072
   parted -s -- /dev/sda mkpart logical ext2 2072 76319

5. Format the partition:

   mke2fs /dev/sda6
   mkswap /dev/sda5
   mke2fs /dev/sda1
   mkdir -p /a/boot
   mount /dev/sda6 /a/
   mount /dev/sda1 /a/boot/

6. FC2 boot disk doesn't have rsync, retrieving from the server:

   scp 10.0.0.254:/usr1/cluster/nodes/rayl3/root/usr/bin/rsync /usr/bin/

7. Copy:

   rsync -avxHS --numeric-ids --exclude 'lost+found' \ 
           10.0.0.254:/usr/cluster/nodes/oscar3/root/ /a/
   rsync -avxHS --numeric-ids --exclude 'lost+found' \ 
           10.0.0.254:/usr/cluster/nodes/oscar3/boot/ /a/boot/

8. Create swapfile since we have extra swap requirement for gaussian and didn't enlarge the swap size at the first place:

   dd if=/dev/zero of=/a/var/vm/swapfile0 bs=1M count=1024
   mkswap /a/var/vm/swapfile0

9. Modify the files /etc/sysconfig/network, /etc/sysconfig/network-scripts/ifcfg-eth0, /etc/pfilter.cmds, /etc/pfilter.src for ip address
10. Restoring GRUB

    chroot /a/
    grub
    root (hd0,0)
    setup (hd0)
    quit
    exit

11. Reboot, Change the BIOS boot sequence if necessary.

Rsnapshot on Fedora Linux

The dependencies of rsnapshot also shows how it works.

I've been evaluating possible deployment of rsnapshot since the RAID0 catastrophe we had on our lab portal. I need an automatic backing-up scheme with historical copies on different time frames. This can prevent failing filesystem from destroying the master copy. Of course, we cannot guarantee anything without diligent monitoring the logs.

According to the website,

"rsnapshot is a filesystem snapshot utility for making backups of local and remote systems. Using rsync and hard links, it is possible to keep multiple, full backups instantly available."

This sounds perfect to me, but I still need to dig more on the strategies we might want when backing-up our data. My concerns were mainly on availability and the efficiency because we have several hundreds gigabyte data in the lab and it's very likely to disrupt the file server availability when doing the backup. Rsync seems to be a no-brainer, and automatic rsync (rsnapshot) is even better.

There are two way to backup the data on the workstations from the backup server, you can actively push the backup to the server or suck the backup on the workstations from the server. Due to the security concerns on the workstations, opening a read-only, exclusive NFS export for the server or opening exclusive rsync access seem to be more secure.

I am not familiar securing rsync:// server at this point, and I am a little bit lazy designing new procedure in the meantime. Without hesitation, I chose to use NFS on the workstations.

In Redhat/Fedora flavored Linux, there is a default autofs rule on /net directory that can mount remote NFS volumes without writing the autofs configurations for mount points and locations. So the rsnapshot configuration is very easy under this setup, I only need to specify interval and backup rules. These are my part of configuration...

--- /etc/rsnapshot.conf.default 2005-04-09 15:59:25.000000000 -0700
+++ /etc/rsnapshot.conf 2005-12-28 22:44:36.000000000 -0800
@@ -23,7 +23,8 @@
 ###########################
 
 # All snapshots will be stored under this root directory.
-snapshot_root /.snapshots/
+#snapshot_root /.snapshots/
+snapshot_root /backup/
 
 # If no_create_root is enabled, rsnapshot will not automatically create the
 # snapshot_root directory. This is particularly useful if you are backing
@@ -49,7 +50,7 @@
 cmd_rsync /usr/bin/rsync
 
 # Uncomment this to enable remote ssh backups over rsync.
-#cmd_ssh /usr/bin/ssh
+#cmd_ssh  /usr/bin/ssh
 
 # Comment this out to disable syslog support.
 cmd_logger /usr/bin/logger
@@ -63,7 +64,7 @@
 # i.e. hourly, daily, weekly, etc.      #
 #########################################
 
-interval hourly 6
+interval hourly 3
 interval daily 7
 interval weekly 4
 #interval monthly 3
@@ -105,6 +106,8 @@
 #include ???
 #exclude ???
 #exclude ???
+exclude  'swapfile*'
+exclude  'lost+found'
 
 # The include_file and exclude_file parameters, if enabled, simply get
 # passed directly to rsync. Please look up the --include-from and
@@ -115,8 +118,9 @@
 
 # Default rsync args. All rsync commands have at least these options set.
 #
-#rsync_short_args -a
+rsync_short_args -a
 #rsync_long_args --delete --numeric-ids --relative --delete-excluded
+rsync_long_args --bwlimit=1024 --delete --numeric-ids --relative --delete-excluded
 
 # ssh has no args passed by default, but you can specify some here.
 #
@@ -145,14 +149,27 @@
 ###############################
 
 # LOCALHOST
-backup /home/  localhost/
-backup /etc/  localhost/
-backup /usr/local/ localhost/
+#backup /home/  localhost/
+#backup /etc/  localhost/
+#backup /usr/local/ localhost/
 #backup /etc/passwd localhost/
 #backup /home/foo/My Documents/  localhost/
 #backup /foo/bar/ localhost/ one_fs=1, rsync_short_args=-urltvpog
 #backup_script  /usr/local/bin/backup_pgsql.sh localhost/postgres/
 
+# NFS
+#backup /net/KOON/    QUEEN/
+#backup /net/PARTHENON/home/ PARTHENON/
+backup /net/ATROPOS/           ATROPOS/
+backup /net/LACHESIS/          LACHESIS/
+backup /net/CLOTHO/home/       KLOTHO/ 
+backup /net/MARJAR/            MARJAR/ 
+backup /net/TOKI/              TOKI/   
+backup /net/SSIZZ/home/        SSIZZ/  
+backup /net/MACHI/homedisk/    MACHI/  
+backup /net/KYO/usr/home/      KYO/    
+backup /net/AUXO/home/         AUXO/   
+
 # EXAMPLE.COM
 #backup root@example.com:/home/ example.com/
 #backup root@example.com:/etc/ example.com/

also don't forget the crontab:

0  0,8,16     * * *     rsnapshot hourly
50 23         * * *     rsnapshot daily
40 23 1,8,15,22 * *     rsnapshot weekly
30 23         1 * *     rsnapshot monthly

The foreseeable problem is the bandwidth and intensity of local I/O, that's why I use "--bwlimit=1024" to use only 1Mbps for single rsync session. Also only 3 hourly backup will be performed for one day to avoid overlapping slow backing up progress.

Software RAID on FC4

This is my configuration: 1GB*3 for /dev/md0 as RAID1 mirror on /boot while 36GB for /dev/md1 as RAID5 on the root (/). So in the end I have 1G for /boot (actually too much) and 71G for the root. I tested two scenarios as followed:

1. One disk removed then reconnected.
2. First disck failed.

To reconnect a raid member back to RAID1/RAID5, use this example: mdadm /dev/md0 -a /dev/hdb2, before doing this, a mdadm -D /dev/md0 can give you more idea. Every time you disconnect a disk and boot it up, you have to re-add the missing disk next time. Damn you software raid....

One the problem we can see is that if the first disk fails, the machine cannot continue its service because the grub is still on the first disk (e.g., /dev/hda). To solve this, we can install grub in any disk other than first one. Why? Because if two disks fail, it dies. Then we can run grub to setup.

grub>   device (hd1) /dev/hdb
grub>   root (hd1,0)
grub>   setup (hd1)

Use exit or quit to leave grub. Now edit the /boot/grub/grub.conf, add this line before the first title line:

fallback=1

(supposed that you have only one boot menu, if the first menu fails, it goes to the second menu.) and copy the first menu and append after first menu, but replace hd0 to hd1. Now it's done. This is my grub.conf:

default=0
faillback=1
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Fedora Core (2.6.11-1.1369_FC4)
        root (hd0,0)
        kernel /vmlinuz-2.6.11-1.1369_FC4 ro root=/dev/md0 rhgb quiet
        initrd /initrd-2.6.11-1.1369_FC4.img
title Fallback Fedora Core (2.6.11-1.1369_FC4)
        root (hd1,0)
        kernel /vmlinuz-2.6.11-1.1369_FC4 ro root=/dev/md0 rhgb quiet
        initrd /initrd-2.6.11-1.1369_FC4.img

Now we can allow the first disk failing and still boot ....

Proposal: Lab computing environment

Here are some antidotes before you read those proposals.

1. Remember, we are in labs for computing, not computer labs.
2. Your lab might fall apart after you leave (, so why bother? @_@).

(Originally posted on 09282003)

1. Avoid sharing desktops.
2. Never try to improve boss' system unless required.
3. Everyone should be responsible for backing up his or her data.
4. Never mount each other between two hosts!!!!
5. Always use automount(8)/autofs(8) for users' machines
6. Never give others root nor even sudo permission unless he or she is capable to do things.
7. There gonna be a central file server with RAID (level 1 or 5).
8. Never activate auto-update features if the users can not figure out the problems when an update fails.
9. NIS is not a good idea unless there are some other administrators available. (Systems always crash after the adms are in vacation.)