amber11 pmemd + LAM-7,1,4/torque

The default optimization flag for pmemd.MPI is -fast, which causes some trouble in our cluster since the torque library doesn't like -static at all. You might already know that -fast is equivalent to "-xHOST -O3 -ipo -no-prec-div -static". My suggestion is to use "-axSTP -O3 -ipo -no-prec-div" instead. The reason for that is compatibility, -xHost also isn't a good optimization flag either. All processors in our cluster are not exactly the same, -xHost is just adding a possibility to mess with.

Ubuntu 10.04 LTS and ifort 11 (x86_64)

Basically you need libstdc++.so.5, thus you can follow the steps in http://www.hackourlives.com/ubuntu-10-04-lucid-lynx-libstdc-so-5/ . My experience on this for my x86_64 virtualbox was that you might not need the 32bit library at all, if you don't have /usr/lib32 .

$ wget http://mirrors.kernel.org/ubuntu/pool/universe/g/gcc-3.3/libstdc++5_3.3.6-17ubuntu1_amd64.deb
$ sudo dpkg --install libstdc++5_3.3.6-17ubuntu1_amd64.deb

VMD requirement on a newly installed fc13

compat-libstdc++-33

xterm

tcsh

xorg-x11-fonts-misc

E1618 on PowerEdge 2950

From the manual:

E1618

PS # Predictive

Power supply voltage is out of acceptable range; specified power supply is improperly installed or faulty.

trajectory movie generation sucks

I just wanted to generate a movie for my box simulation, in the end I tried vmd and pymol for numerous times and still can get a satisfying result. (yeah and flickr video sucks.)

Please remember to click HQ icon, or you see some blurred crap.

g03 oops

Duh! I need to add this into /etc/sysctl.conf and run the sysctl command on every cluster node.

# for g03
kernel.randomize_va_space = 0

(skipped, the commands are easy part, the hard part is to recover this from my own rusty brain.

The profile file, beware of that if /usr/tmp is not working and gives you shits like "Erroneous write during file extend. write -1 instead of 4096", check the writability permission on $GAUSS_SCRDIR. Or just set GAUSS_SCRDIR to $HOME.

export PATH="/home/software/g03/g03:${PATH}"
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/home/software/g03/g03:/home/software/g03/gv/lib"
export g03root="/home/software/g03/g03"
export GAUSS_EXEDIR="/home/software/g03/g03"
export GAUSS_SCRDIR="/usr/tmp"
export G03BASIS="/home/software/g03/g03/basis/"
export GAUSS_ARCHDIR="/home/software/g03/g03/arch"
export GV_DIR="/home/software/g03/gv"
alias gv=$GV_DIR'/gview'
alias gview=$GV_DIR'/gview'

My installation on the cluster:

$ find /home/software/g03 -type d
/home/software/g03
/home/software/g03/g03
/home/software/g03/g03/arch
/home/software/g03/g03/basis
/home/software/g03/g03/bsd
/home/software/g03/g03/tests
/home/software/g03/g03/tests/com
/home/software/g03/g03/tests/i386
/home/software/g03/g03/tests/rs6k
/home/software/g03/gv
/home/software/g03/gv/bin
/home/software/g03/gv/data
/home/software/g03/gv/data/biofrags
/home/software/g03/gv/data/biofrags/fragments
/home/software/g03/gv/data/elements
/home/software/g03/gv/data/elements/bitmaps
/home/software/g03/gv/data/elements/fragments
/home/software/g03/gv/data/fonts
/home/software/g03/gv/data/rgroups
/home/software/g03/gv/data/rgroups/bitmaps
/home/software/g03/gv/data/rgroups/fragments
/home/software/g03/gv/data/rings
/home/software/g03/gv/data/rings/bitmaps
/home/software/g03/gv/data/rings/fragments
/home/software/g03/gv/help
/home/software/g03/gv/help/icons
/home/software/g03/gv/help/pix
/home/software/g03/gv/help/refs
/home/software/g03/gv/lib

Beware of rsync -C option.

To my surprise, rsync reads .cvsignore file and follows the rule in the files! That's not a good thing to me since I wanted to backup necessary files, not less files. (If you know what I am saying.) So I should have known better and have read the manual!

-C, --cvs-exclude           auto-ignore files in the same way CVS does

MPI communication notes, continued.

1. MPI_Gather(array)+SUM(array) is faster than MPI_Reduce(array,Sum)
2. (This is not MPI related.)FORTRAN's module variables do not have sequential address space.

Interesting MPI Communication

1. time(MPI_ALLREDUCE) > time(MPI_REDUCE+MPI_BCAST)
2. time(MPI_REDUCE) > time(MPI_BARRIER+MPI_REDUCE)

Duuuuuuh

(update: no, barrier doesn't help, the problem was due to faulty nodes)

Things Sucked.

Memo to myself: always remember to qalter the job resources list after changing the default settings of the server.

% sudo qalter 1234.server.local -l cput=360000000

Bonus memo:

% qstat -f | grep -e Job -e cput
Job Id: 1236.raylsrvr.local
    Job_Name = heat_dip
    Job_Owner = mjhsieh@raylsrvr.local
    resources_used.cput = 01:31:38
    Resource_List.cput = 100000:00:00
%

Always check the cput from time to time, it runs faster than you thought. Don't let the system kill your long, un-restart-able running jobs.

amber tool (amber9) requires flex

nice to know.

To summarize, it requires flex, libXt-devel, libX11-devel, libICE-devel and libSM-devel, possible also byacc.

Beware of tmpwatch

Default tmpwatch script at /etc/cron.daily/ (at least for Fedora 3 ~ 8) only honors the atime (access time) so if you are running a very long calculation and storing the working files in the /tmp, you need to be careful. In the case of AMBER's sander program, the output files are opened all the time, that means the atime might not change at all and your precious data will be gone after 10 days.
Possible solution is to increase the time of deleting old files or add an option '-m' into tmpwatch script.

perl -pi -e 's/tmpwatch -x \/tmp/tmpwatch -m -x \/tmp/' /etc/cron.daily/tmpwatch /var/lib/systemimager/images/*/etc/cron.daily/tmpwatch

Also remember to do that on the nodes.

Dell. PowerEdge RAID Controller (PERC) nodes on OSCAR5

Again the PERC might not load itself in the boel-kernel environment, so you can add "modprobe megaraid_sas" into your /var/lib/systemimager/scripts/pre-install/99all.harmless_example_script file. That took me several an hour to figure it out how to get around of it.

Ethernet Ports Numbering of PE1950

In BIOS: the numbering for the onboard ethernet port is the same as the labeling on the machine. In Linux, the numbering is reversed. After you activated the PXE booting on the secondary port in the BIOS, you need to reboot to set the boot priority in the BIOS again.

xleap(AMBER9) Xext linking problem in Fedora Core 6

Hint:

ln -s  libXext.so.6.4.0 libXext.so

Extra point, xleap also requires byacc, libX11-devel, libXt-devel, libICE-devel and libSM-devel.