This page documents the various questions related to storage on the Juno cluster.
Juno home storage :
- Description : Isilon shared parallel filesystem.
- Purpose: To store software related code and scripts, default quota size is small and fixed.
- Mount: /home/<user>
- Access: All Juno nodes, including compute storage and login nodes.
- Default quota: 100GB
- Snapshots: 7 days of snapshots. ( not backed up ). Can be accessed in /home/.snapshots/<user>
- Replicated: yes
Juno compute storage :
- Description : GPFS shared parallel filesystem, not replicated, and not backed up.
- Purpose: For jobs to read and write compute data from login and compute nodes, default quota size is larger with flexibility to request larger quota.
- Mount: /juno/res/<lab group>, /juno/work/<lab group>.
- Access: All Juno nodes, including compute storage and login nodes.
- Default quota: 1TB ( Increased/Decreased on request )
- Snapshots: 7 days of snapshots. ( not backed up ). Can be accessed in /data/.snapshots/<date>/<lab group>
- Replicated: no
Juno warm storage :
- Description : GPFS shared parallel filesystem, not replicated but can be backed up upon request.
- Purpose: To store long term data. Only accessible from login nodes and cannot be accessed from compute nodes.
- Mount: /warm/<lab group>
- Access: Only Juno login nodes.
- Default quota: 1TB ( Increased/Decreased on request )
- Snapshots: 7 days of snapshots. ( not backed up ). Can be accessed in /warm/.snapshots/<date>/<lab group>
- Replicated: no
Local scratch storage :
- Description : Local SATA drive, not replicated, and not backed up. Local to each compute node, not a shared filesystem
- Purpose: To store local temporary data related to compute jobs. Since this is not a shared filesystem, the temporary data needs to be cleaned up and copied back to shared filesystem after job completion. Files older than 31 days are automatically deleted.
- Mount: /scratch/
- Access: Local to the compute node.
- Default quota: No quota and limited to free disk space in /scratch.
- Snapshots: No snapshots.
- Replicated: no
Local fast scratch storage :
- Description : Local NVMe drive, not replicated, and not backed up. Local to each compute node, not a shared filesystem
- Purpose: To store local temporary data related to compute jobs. Since this is not a shared filesystem, the temporary data needs to be cleaned up and copied back to shared filesystem after job completion. Files older than 31 days are automatically deleted.
- Mount: /fscratch/
- Access: Local to the compute node. Not available on all nodes. Must be requested on bsub with -R fscratch
- Default quota: No quota and limited to free disk space in /fscratch.
- Snapshots: No snapshots.
- Replicated: no
How to get information about your GPFS storage usage:
You can get real time information about storage quotas and usage from grafana at https://hpc-grafana.mskcc.org/d/000000027/storage-quota
You can also get information from the command line
For the home directory /home/$USERNAME, use:
mmlsquota juno:home
For the working directory /data/$LAB, use:
mmlsquota -j work_lab --block-size auto juno
or
df -h /data/$LAB
GPFS also has quotas on number of files -- not just data size.
mmlsquota gives information about quota on number of files too, along with information about block quota.
Filesystem | Fileset | type | blocks | quota | limit | in_doubt | grace | | | files | quota | limit | in_doubt | grace | Remarks |
Filesystem name | Fileset name | fileset/usr/grp | Blocks currently occupied | Your block quota | Your limit for "7 days" beyond quota. | Blocks in doubt that will be counted towards your quota. Happens when many files are added/deleted recently. | Countdown of 7 days is set once you occupy more blocks than mentioned in quota. |
| Number of files currently present | Your quota on number of files | Your limit on number of files for "7 days" beyond quota. | Number of files in doubt that will be counted towards your quota. | Countdown of 7 days is set once you occupy more files than mentioned in quota. |
|
Once the number of blocks or number of files reach the value mentioned in "quota" - Storage system will give 7 days as a grace period, to fill up until the max value mentioned in "limit" is reached. Storage system will not allow any more data to be written when:
- The block limit/file limit is reached.
- 7 days have passed since the blocks/files have occupied more than "quota". The grace field will show you the number of days left, before which the number of blocks/files need to go less than the value mentioned in "quota".