The new Juno cluster is now available to the test users only. Please, create a request if you want to participate. It is currently composed of 216 CPUs. The login node name is juno. It's a CentOS 7.3 host. The latest CentOS 7.3 Linux operating system is on nodes jx01-jx10 and the Centos 6.9 Linux operating system is on node ju14 . All cluster nodes have access to Isilon solisi file system. The CentOS 7.3 nodes have access to the GPFS storage named juno.
Queues
The Juno cluster uses LSF (Load Sharing Facility) 10.1 SP6 from IBM to schedule jobs. The default LSF queue, ‘general’, includes all Juno compute nodes.
Job resource control enforcement in LSF with cgroups
LSF 10.1 makes use of Linux control groups (cgroups) to limit the CPU cores and memory that a job can use. The goal is to isolate the jobs from each other and prevent them from consuming all the resources on a machine. All LSF job processes are controlled by the Linux cgroup system. If the job processes on a host use more memory than it requested, the job will be terminated by the Linux cgroup memory sub-system.
New LSF cluster level resource configurations
Memory is consumable resource in GB per slot/task (-n).
All jobs should specify Walltime (-W), otherwise the default Walltime 6 hours will be used.
LSF will terminate the job if it exceeds the Walltime.
The maximum Walltime for general queue is 744 hours (31 days).
Job Default parameters
Queue name: default
Operating System: CentOS 7.3
Number of slots (-n): 1
Waltime (job running time): 6 hours
Memory (RAM) : 2GB
Job Submission
The default operating system for Juno jobs is CentOS 7.3. Users can request that jobs will run on one specific type of operating system or any operating system.
To submit the job to default (CentOS 7.3) operating system:
bsub -n 1 -W 1:00 -R "rusage[mem=2]"
To submit the job to CentOS 6.9 operating system:
bsub -n 1 -app anyOS -W 1:00 -R "rusage[mem=2]" -R "select[type==CentOS6]"
To submit the job to any operating system:
bsub -n 1 -app anyOS -W 1:00 -R "rusage[mem=2]"
To submit the job on node with available NVMe scratch /fscratch :
bsub -n 1 -R "fscratch" ....
Difference in LSF configuration on Juno vs Luna
- There is no iounits on Juno
- Memory (GB) is per slot/task on Juno. Memory(GB) is per job on Luna.
- All jobs should have -W (Walltime) on Juno. LSF enforced it. Please, don't use -We on Juno.
- All nodes on Juno will have internet access.
- GPFS and OS reserve 10GB of RAM per host on Juno CentOS7 hosts. Usable memory for LSF jobs: Total RAM - 10 GB