amber11 pmemd + LAM-7,1,4/torque

The default optimization flag for pmemd.MPI is -fast, which causes some trouble in our cluster since the torque library doesn't like -static at all. You might already know that -fast is equivalent to "-xHOST -O3 -ipo -no-prec-div -static". My suggestion is to use "-axSTP -O3 -ipo -no-prec-div" instead. The reason for that is compatibility, -xHost also isn't a good optimization flag either. All processors in our cluster are not exactly the same, -xHost is just adding a possibility to mess with.

MPI communication notes, continued.

1. MPI_Gather(array)+SUM(array) is faster than MPI_Reduce(array,Sum)
2. (This is not MPI related.)FORTRAN's module variables do not have sequential address space.

Interesting MPI Communication

1. time(MPI_ALLREDUCE) > time(MPI_REDUCE+MPI_BCAST)
2. time(MPI_REDUCE) > time(MPI_BARRIER+MPI_REDUCE)

Duuuuuuh

(update: no, barrier doesn't help, the problem was due to faulty nodes)

Double Precision Accuracy

via random google search result:

"Single Precision occupies 32 bits (4 bytes) and has a significand precision of 24 bits. This gives it an accuracy of about 7 decimal digits.
Double precision occupies 64 bits (8 bytes) and has a significand precision of 53 bits. This gives it an accuracy of about 16 decimal digits."

Todo: review the floating point part of the computor architecture textbook

Intel Compiler Settings on Linux

After checking my .cshrc resource profile, I found this reveals the history of our lab being using Intel compilers.

#Compilers: however I am not sure whether 5.0/6.0/7.0 work or not.
unsetenv IC50
unsetenv IC60
unsetenv IC70
unsetenv IC80
unsetenv IC90
setenv IC81
if ($?IC50) then
    source /opt/intel/compiler50/ia32/bin/ifcvars.csh
    source /opt/intel/compiler50/ia32/bin/iccvars.csh
else if ($?IC60) then
    source /opt/intel/compiler60/ia32/bin/ifcvars.csh
    source /opt/intel/compiler60/ia32/bin/iccvars.csh
else if ($?IC70) then
    source /opt/intel/compiler70/ia32/bin/ifcvars.csh
    source /opt/intel/compiler70/ia32/bin/iccvars.csh
    setenv MKL_HOME /opt/intel/mkl721
    if ($?INCLUDE) then
       setenv INCLUDE ${MKL_HOME}/include:${INCLUDE}
    else
       setenv INCLUDE ${MKL_HOME}/include
    endif
    setenv LD_LIBRARY_PATH ${MKL_HOME}/lib/32:${LD_LIBRARY_PATH}
else if ($?IC80) then
    source /opt/intel/compiler80/bin/ifortvars.csh
    setenv MKL_HOME /opt/intel/mkl721
    if ($?INCLUDE) then
       setenv INCLUDE ${MKL_HOME}/include:${INCLUDE}
    else
       setenv INCLUDE ${MKL_HOME}/include
    endif
    setenv LD_LIBRARY_PATH ${MKL_HOME}/lib/32:${LD_LIBRARY_PATH}
else if ($?IC90) then
    source /opt/intel/cc/9.0/bin/iccvars.csh
    source /opt/intel/fc/9.0/bin/ifortvars.csh
    setenv MKL_HOME /opt/intel/mkl/8.0.1
    source ${MKL_HOME}/tools/environment/mklvars32.csh
else if ($?IC81) then
    source /opt/intel/fc/8.1/bin/ifortvars.csh
    setenv MKL_HOME /opt/intel/mkl/8.0.1
    source ${MKL_HOME}/tools/environment/mklvars32.csh
endif

gettim and secnds in FORTRAN

If you are porting fortran program from old Microsoft Visual Fortran, you might find this useful. I believe these two function calls (gettim and secnds) are from PDP-11 (perhaps). Let me write down my bridge of these two to gfortran.

      subroutine secnds(T)
      real*8 :: t, tnow
      real*8 :: myclock

      tnow = myclock
      t = tnow - t

      return
      end subroutine secnds

!------------------------------------------------------------------------------
      subroutine gettim(ih,imin,isec,iif)
      character(len=13) cd, ct, cz
      integer :: ih, imin, isec, iif
      integer, dimension(8) ::  iv(8)
      cd='             '
      ct=cd
      cz=cd
      call date_and_time(cd,ct,cz,iv)
      ih=iv(5)
      imin=iv(6)
      isec=iv(7)
      iif=iv(8)
      
      return
      end subroutine gettim

!------------------------------------------------------------------------------
      real*8 function myclock()
      character(len=13) cd, ct, cz
      integer, dimension(8) ::  iv(8)
      integer :: iclock
      
      cd='             '
      ct=cd
      cz=cd
      
      CALL DATE_AND_TIME(cd,ct,cz,iv)
      iclock = 86400*iv(3)+3600*iv(5)+60*iv(6)+iv(7)
      myclock = dble(iclock)+1.0d-3*dble(iv(8))
      end function myclock

OSCAR 4 lam-mpi and ifort

This is a note for ifort with lam-mpi under OSCAR cluster. You can either compile lam by yourself:

% env CC=icc CXX=icpc FC=ifort F77=ifort F90=ifort ./configure \ 
   --prefix=/home/lammpi

Or, use "ifort -assume 2underscores" as your FC for your applications.

By the way, the run-time ssi option SYSV would be good for general cases since the communication goes through shared memory on the same machine and through tcp on different machines.

Also check this out:

#!/bin/sh
#PBS -N "LAMMPIjob"
#PBS -q workq
#PBS -l nodes=1:ppn=2
#PBS -S /bin/sh

# To check if this is forked by qsub
# Not necessary
if [ -z "$PBS_ENVIRONMENT" -a "$SSH_TTY" ]
then
    # go to some directory
    cd /home/demo/LAM
else
    # go to some directory
    cd $PBS_O_WORKDIR
fi
# using the lazy default tm module (pbs)
/opt/lam-7.0.6/bin/lamboot
/opt/lam-7.0.6/bin/mpirun -ssi rpi sysv -np 2 \ 
         ./sander.LAM -O -i mdin -c inpcrd.equil -p prmtop \ 
        -o /tmp/output.txt -x /tmp/trajectory.crd -r /tmp/restart.rst
/opt/lam-7.0.6/bin/lamhalt
# Data Retreival
mv /tmp/output.txt /tmp/trajectory.crd /tmp/restart.rst .

Arguments in FORTRAN

This is my example to use arguments in the fortran programs:

       program main
       implicit none
       integer   :: IArgC,ArgC,i
       character :: ArgV(4)*20

       ArgC = IArgC()
       if (ArgC==4) then
          do i=1,ArgC
             call GetArg(i,ArgV(i))
             write(6,*) ArgV(i)
          enddo
       else
          write(6,*) "Usage: mytest arg1 arg2 arg3 arg4"
          stop
       endif
       end program

This should be easy to integrate into most of fortran programs. If ifc (ver 7) does not compile, use -lPEPCF90 in the compiler command.