General Documentation
- Welcome FAQ
- Secure Shell SSH
- Available Software
- Installing Software
- Guidelines and Policies
- Glossary
- Grant Support
- Sharing Data
- Containers & Singularity
- UserGroup Presentations
- Jupyter Notebook Usage
LSF Primer
Lilac Cluster Guide
Juno Cluster Guide
Cloud Resources
Backup Policy on server/node local drives
File lists
Page History
The new juno Juno cluster is now available only for testing. Please request "Access another cluster" from http://hpc.mskcc.org/compute-accounts/account-change-request/ if you want to participate in the test. juno currently has 216 . Juno resources are specifically for investigators associated with the Center for Molecular Oncology. If you have an account on luna, you have access to juno.
Juno currently has 2556 CPUs. The login node is juno'Juno', and runs CentOS 7 with GPFS . Nodes jx01-jx10 as the main file system.
All nodes are running CentOS 7 with GPFS, while node ju14 is running CentOS 6 without GPFS. All juno nodes access to the new GPFS /juno storage.
All Juno nodes also have access to the solisi Solisi Isilon file systems. CentOS 7 nodes also have access to /juno GPFS storage.
Differences in LSF Configuration between juno and luna
- We reserve ~12GB of RAM per host for the operating system and GPFS on Juno CentOS 7 hosts.
- On each jx## node (CentOS 7, GPFS installed), 240GB of RAM is available for LSF jobs.
- On each ju## node (CentOS 6, no GPFS), 250GB of RAM is available for LSF jobs.
- When specifying RAM for LSF jobs, specify GB of RAM per task (slot) on juno (unlike luna, where RAM is specified per job).
- All jobs must have -W (maximum execution Walltime) specified on juno. Please do not use -We on juno.
- There is no /swap on CentOS 7 nodes. Memory usage is enforced by cgroups so jobs never swap. A job will be terminated if memory usage exceeds its LSF specification.
- To check jobs which are DONE or status EXIT, use "bhist -l JobID" or "bhist -n 0 -l JobID". bacct is also available. "bjobs -l JobID" only shows RUNNING and PEND jobs.
- There is no iounits resource on juno.
Queues
The juno
Table of Contents |
---|
Configuration change log
New nodes jx01-34 added to the cluster.
New nodes jy01-03 added to the cluster. These nodes don't have NVMe /fscratch. To request node with /fscratch partition: bsub -n 1 -R fscratch ...
Slides from November 2019 User Group: HPC-User-Group-2019-10.pdf
Slides from March 21 2019 User Group Juno_UG_032019_final.pdf
Slides from Sep 27 2018 User Group meeting (updated Nov 28) Juno_UG_092018_final.pdf
December 17, 2018: The new queue "control" has been added. Please, check "Queues"
November 28, 2018: The default OS type is CentOS07. Please, check "Job Submission"
Differences in LSF Configuration between Juno and luna
Please see here: http://mskcchpc.org/display/CLUS/Juno+vs.+Luna
Queues
The Juno cluster uses LSF (Load Sharing Facility) 10.1FP6 1FP8 from IBM to schedule jobs. The Juno cluster has two queues: general and control. The default queue, ‘general’, includes all Juno compute nodes.
The control queue doesn't have wall-time limitation and has one node with 144 oversubscribed slots. The control queue should be used only for monitoring or control jobs (the jobs which doesn't use real CPU and memory resources).
To submit the job to the control queue:
Code Block | ||
---|---|---|
| ||
bsub -n 1 -q control -M 1 |
Job Resource Control Enforcement in LSF with cgroups
...
Queue name: general
Operating System: CentOS 67
Number of slots (-n): 1
Maximum Waltime (job runtime): 6 hours
Memory (RAM) : 2GB
Short vs. Long Jobs and Node Availability
Juno has CMOPI and DEVEL SLAs. When CMOPI/DEVEL jobs are not filling their assigned nodes, 100% of those job slots are available to non-CMOPI jobs with a duration under 2 hours, 75% of slots are available to jobs under 4 hours, and 50% of slots are available to jobs under 31 days.
Nodes assigned to other SLAs are available to non-SLA jobs up to 6 hours.
Job Submission
By default jobs submitted on juno only run on CentOS 6 nodes. Users can specify CentOS 7 nodes (with GPFS), CentOS 6 nodes (Isilon only), or either type.
To submit a job to CentOS 6 7 nodes use either of these formats:
Code Block | ||
---|---|---|
| ||
bsub -n 1 -W 1:00 -R "rusage[mem=2]"
bsub -n 1 -W 1:00 -app anyOS -R "select[type==CentOS6]" -R "rusage[mem=2]" |
To submit a job to CentOS 7 nodes:
Code Block | ||
---|---|---|
| ||
bsub -n 1 -W 1:00 -app anyOS -R "select[type==CentOS7]" -R "rusage[mem=2]" |
To submit a job to any nodes , running either CentOS 6 or 7:
Code Block | ||
---|---|---|
| ||
bsub -n 1 -W 1:00 -app anyOS -R "rusage[mem=2]" |
To submit a job to nodes with NVMe /fscratch:
Code Block | ||
---|---|---|
| ||
bsub -n 1 -R "fscratch" bsub -n 1 -R "pic" |