Things Sucked.

Memo to myself: always remember to qalter the job resources list after changing the default settings of the server.

% sudo qalter 1234.server.local -l cput=360000000

Bonus memo:

% qstat -f | grep -e Job -e cput
Job Id: 1236.raylsrvr.local
    Job_Name = heat_dip
    Job_Owner = mjhsieh@raylsrvr.local
    resources_used.cput = 01:31:38
    Resource_List.cput = 100000:00:00
%

Always check the cput from time to time, it runs faster than you thought. Don't let the system kill your long, un-restart-able running jobs.