Job 14549723.pbspl1.nas.nasa.gov started on Fri Sep 16 08:48:45 PDT 2022 The job requested the following resources: ncpus=252 place=scatter:excl walltime=05:00:00 PBS set the following environment variables: FORT_BUFFERED = 1 TZ = PST8PDT On r1i2n8: Warning: no access to tty (Bad file descriptor). Thus no job control in this shell. Current directory is /nobackup/melalaou/CCMC/2022/APL/GAMERA_Storm_Jan13to16_2022_MEA_t02 Job 14549723.pbspl1.nas.nasa.gov started at Fri Sep 16 08:48:58 PDT 2022 on r1i2n8. The following modules are loaded: Currently Loaded Modulefiles: 1) pkgsrc/2021Q2 4) szip/2.1.1 2) comp-intel/2020.4.304 5) hdf5/1.8.18_mpt 3) mpi-hpe/mpt.2.23 The active environment variables are: MPI_WATCHDOG_TIMER=720 MKLROOT=/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/mkl MANPATH=/nasa/hdf5/1.8.18_mpt/share/man:/nasa/hpe/mpt/2.23_cent7/man:/nasa/hpe/mpt/2.23_cent7/usr/man:/nasa/intel/Compiler/2020.4.304/documentation_2020/en/man/common:/nasa/pkgsrc/toss3/2021Q2/man:/opt/c3/man:/opt/sgi/share/man:/usr/share/catman:/usr/share/man:/usr/catman:/usr/man KAIJU_QUICKSTART_DIR=/nobackup/melalaou/CCMC/2022/APL/kaiju/quickstart HOSTNAME=r1i2n8 MPI_SYSLOG_COPY=2 IPPROOT=/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/ipp HOST=r1i2n8 SHELL=/bin/csh MPI_IB_FAILOVER=127 MPIO_LUSTRE_GCYC_MIN_ITER=1 TMPDIR=/var/tmp/pbs.14549723.pbspl1.nas.nasa.gov PBS_JOBNAME=geo_mpi LIBRARY_PATH=/nasa/hdf5/1.8.18_mpt/lib:/nasa/hpe/mpt/2.23_cent7/lib:/nasa/hpe/mpt/2.23_cent7/usr/lib64:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/ipp/lib/intel64:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/compiler/lib/intel64_lin:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/tbb/lib/intel64/gcc4.8:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/tbb/lib/intel64/gcc4.8:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/daal/lib/intel64_lin OMP_STACKSIZE=1000M FPATH=/nasa/hdf5/1.8.18_mpt/include:/nasa/hpe/mpt/2.23_cent7/include:/nasa/hpe/mpt/2.23_cent7/usr/include PBS_ENVIRONMENT=PBS_BATCH PBS_O_TZ=PST8PDT PBS_O_WORKDIR=/nobackup/melalaou/CCMC/2022/APL/GAMERA_Storm_Jan13to16_2022_MEA_t02 NCPUS=28 MPI_IB_RAILS=1+ RUNID=geo_mpi GROUP=s8873 PBS_TASKNUM=1 USER=melalaou LD_LIBRARY_PATH=/nasa/hdf5/1.8.18_mpt/lib:/nasa/szip/2.1.1/lib:/nasa/hpe/mpt/2.23_cent7/lib:/nasa/hpe/mpt/2.23_cent7/usr/lib64:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/compiler/lib/intel64_lin:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/ipp/lib/intel64:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/tbb/lib/intel64/gcc4.8:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/daal/lib/intel64_lin PSTLROOT=/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/pstl PBS_O_HOME=/home1/melalaou CPATH=/nasa/hdf5/1.8.18_mpt/include:/nasa/szip/2.1.1/include:/nasa/hpe/mpt/2.23_cent7/include:/nasa/hpe/mpt/2.23_cent7/usr/include:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/ipp/include:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/mkl/include:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/pstl/include:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/pstl/stdlib:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/tbb/include:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/tbb/include:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/daal/include OSCAR_HOME=/opt/oscar HOSTTYPE=x86_64-linux FORT_BUFFERED=1 MPT_VERSION=2.23_cent7 PBS_MOMPORT=15003 MPI_IB_TIMEOUT=20 PBS_O_QUEUE=normal NLSPATH=/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/compiler/lib/intel64/locale/%l_%t/%N:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin/locale/%l_%t/%N PBS_O_LOGNAME=melalaou PATH=/nobackup/melalaou/CCMC/2022/APL/kaiju/quickstart/geo_serial:/nobackup/melalaou/CCMC/2022/APL/kaiju/build/bin:/nasa/hdf5/1.8.18_mpt/bin:/nasa/hpe/mpt/2.23_cent7/bin:/nasa/hpe/mpt/2.23_cent7/usr/bin:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/bin/intel64:/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/bin:/nasa/pkgsrc/toss3/2021Q2/bin:/nasa/pkgsrc/toss3/2021Q2/sbin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/PBS/bin:/usr/sbin:/opt/c3/bin:/opt/sgi/sbin:/opt/sgi/bin:/nobackup/melalaou/CCMC/2022/APL/kaiju/scripts:/nobackup/melalaou/CCMC/2022/APL/kaiju/scripts/datamodel:/nobackup/melalaou/CCMC/2022/APL/kaiju/scripts/helio:/nobackup/melalaou/CCMC/2022/APL/kaiju/scripts/legacy:/nobackup/melalaou/CCMC/2022/APL/kaiju/scripts/preproc:/nobackup/melalaou/CCMC/2022/APL/kaiju/scripts/postproc:/nobackup/melalaou/CCMC/2022/APL/kaiju/scripts/quicklook PBS_O_LANG=C PBS_JOBCOOKIE=292A599617F4E2497DDA6BB56F2B71C0 MPI_UD_RECV_MSGS=32768 TBBROOT=/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/tbb PWD=/nobackup/melalaou/CCMC/2022/APL/GAMERA_Storm_Jan13to16_2022_MEA_t02 _LMFILES_=/nasa/modulefiles/pkgsrc/toss3/pkgsrc/2021Q2:/nasa/modulefiles/toss3/comp-intel/2020.4.304:/nasa/modulefiles/toss3/mpi-hpe/mpt.2.23:/nasa/modulefiles/toss3/szip/2.1.1:/nasa/modulefiles/toss3/hdf5/1.8.18_mpt PBS_NODENUM=0 LANG=C MODULEPATH=/usr/share/modules/modulefiles:/nasa/modulefiles/toss3:/nasa/modulefiles/spack/gcc-4.8:/nasa/modulefiles/pkgsrc/toss3 KAIJU_INSTALL_DIR=/nobackupp17/melalaou/CCMC/2022/APL/kaiju LOADEDMODULES=pkgsrc/2021Q2:comp-intel/2020.4.304:mpi-hpe/mpt.2.23:szip/2.1.1:hdf5/1.8.18_mpt PBS_JOBDIR=/home1/melalaou TZ=PST8PDT MPI_UD_TIMEOUT=40 C3_RSH=ssh -oConnectTimeout=10 -oForwardX11=no PBS_O_SHELL=/bin/csh LM_LICENSE_FILE=28518@license2 DAALROOT=/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/daal MPI_SHEPHERD=true PBS_JOBID=14549723.pbspl1.nas.nasa.gov MPI_IB_RNR_TIMER=26 ENVIRONMENT=BATCH SHLVL=2 HOME=/home1/melalaou OSTYPE=linux KAIJUHOME=/nobackup/melalaou/CCMC/2022/APL/kaiju PKGSRC_BASE=/nasa/pkgsrc/toss3/2021Q2 PBS_O_HOST=pfe26.nas.nasa.gov MPI_IB_RECV_MSGS=1024 VENDOR=unknown PYTHONPATH=:/nobackup/melalaou/CCMC/2022/APL/kaiju MPI_ROOT=/nasa/hpe/mpt/2.23_cent7 MACHTYPE=x86_64 LOGNAME=melalaou CVS_RSH=ssh CLASSPATH=/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/daal/lib/daal.jar PBS_QUEUE=normal MODULESHOME=/usr/share/Modules PKG_CONFIG_PATH=/nasa/intel/Compiler/2020.4.304/compilers_and_libraries_2020.4.304/linux/mkl/bin/pkgconfig OMP_NUM_THREADS=28 PBS_O_MAIL=/var/mail/melalaou arch=intel64 INFOPATH=/nasa/pkgsrc/toss3/2021Q2/info MPI_TYPE_DEPTH=100 INCLUDE=/nasa/hdf5/1.8.18_mpt/include PBS_O_SYSTEM=Linux PBS_NODEFILE=/var/spool/pbs/aux/14549723.pbspl1.nas.nasa.gov PBS_O_PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/X11R6/bin:/PBS/bin:/usr/sbin:/sbin:/opt/c3/bin:/opt/sgi/sbin:/opt/sgi/bin _=/usr/bin/printenv The current directory is /nobackup/melalaou/CCMC/2022/APL/GAMERA_Storm_Jan13to16_2022_MEA_t02. Running /nobackup/melalaou/CCMC/2022/APL/build/bin/voltron.x on model geo_mpi. =>> PBS: job killed: walltime 18048 exceeded limit 18000 Terminated ____________________________________________________________________ Job Resource Usage Summary for 14549723.pbspl1.nas.nasa.gov CPU Time Used : 127:11:29 Real Memory Used : 2975644kb Walltime Used : 05:00:59 Exit Status : 143 Number of CPUs Requested : 252 Walltime Requested : 05:00:00 Execution Queue : normal Charged To : s8873 Job Stopped : Fri Sep 16 13:50:02 2022 ____________________________________________________________________