General Documentation

LSF Primer

Lilac Cluster Guide

Juno Cluster Guide

Cloud Resources

Backup Policy on server/node local drives

File lists

Skip to end of metadata
Go to start of metadata

This page documents the various questions related to storage on the Juno cluster. 

Juno home storage : 

    • Description : GPFS shared parallel filesystem.
    • Purpose: To store software related code and scripts, default quota size is small and fixed.
    • Mount: /home/<user>
    • Access: All Juno nodes, including compute storage and login nodes.
    • Default quota: 100GB
    • Snapshots: 7 days of snapshots. ( not backed up ). Can be accessed in /home/.snapshots/<user>
    • Replicated: yes

Juno compute storage : 

    • Description : GPFS shared parallel filesystem, not replicated, and not backed up.
    • Purpose: For jobs to read and write compute data from login and compute nodes, default quota size is larger with flexibility to request larger quota.
    • Mount: /juno/res/<lab group>, /juno/work/<lab group>.
    • Access: All Juno nodes, including compute storage and login nodes.
    • Default quota: 1TB ( Increased/Decreased on request )
    • Snapshots: 7 days of snapshots. ( not backed up ). Can be accessed in /data/.snapshots/<date>/<lab group>
    • Replicated: no

Juno warm storage : 

    • Description : GPFS shared parallel filesystem, not replicated but can be backed up upon request. 
    • Purpose: To store long term data. Only accessible from login nodes and cannot be accessed from compute nodes.
    • Mount: /warm/<lab group>
    • Access: Only Juno  login nodes. 
    • Default quota: 1TB ( Increased/Decreased on request )
    • Snapshots: 7 days of snapshots. ( not backed up ). Can be accessed in /warm/.snapshots/<date>/<lab group>
    • Replicated: no 

Local scratch storage : 

  • Description : Local SATA drive, not replicated, and not backed up. Local to each compute node, not a shared filesystem
  • Purpose: To store local temporary data related to compute jobs. Since this is not a shared filesystem, the temporary data needs to be cleaned up and copied back to shared filesystem after job completion.  Files older than 31 days are automatically deleted.
  • Mount: /scratch/
  • Access: Local to the compute node.
  • Default quota: No quota and limited to free disk space in /scratch.
  • Snapshots: No snapshots.
  • Replicated: no

Local fast scratch storage : 

  • Description : Local NVMe drive, not replicated, and not backed up. Local to each compute node, not a shared filesystem
  • Purpose: To store local temporary data related to compute jobs. Since this is not a shared filesystem, the temporary data needs to be cleaned up and copied back to shared filesystem after job completion.  Files older than 31 days are automatically deleted.
  • Mount: /fscratch/
  • Access: Local to the compute node. Not available on all nodes. Must be requested on bsub with -R fscratch
  • Default quota: No quota and limited to free disk space in /fscratch.
  • Snapshots: No snapshots.
  • Replicated: no

How to get information about your GPFS storage usage:

You can get real time information about storage quotas and usage from grafana at https://hpc-grafana.mskcc.org/d/000000027/storage-quota

You can also get information from the command line

For the home directory /home/$USERNAME, use:

              mmlsquota juno:home 

For the working directory /data/$LAB, use:

       mmlsquota -j work_lab  --block-size auto juno

or

       df -h  /data/$LAB

GPFS also has quotas on number of files -- not just data size.

mmlsquota gives information about quota on number of files too, along with information about block quota.

Filesystem

Fileset

type

blocks

quota

limit

in_doubt

grace

|

files

quota

limit

in_doubt

grace

Remarks

Filesystem name

Fileset name

fileset/usr/grp

Blocks currently occupied

Your block quota

Your limit for "7 days" beyond quota.

Blocks in doubt that will be counted towards your quota.

Happens when many files are added/deleted recently.

Countdown of 7 days is set once you occupy more blocks than mentioned in quota.


Number of files currently present

Your quota on number of files

Your limit on number of files for "7 days" beyond quota.

Number of files in doubt that will be counted towards your quota.

Countdown of 7 days is set once you occupy more files than mentioned in quota.


Once the number of blocks or number of files reach the value mentioned in "quota" - Storage system will give 7 days as a grace period, to fill up until the max value mentioned in "limit" is reached. Storage system will not allow any more data to be written when:

  1. The block limit/file limit is reached.
  2. 7 days have passed since the blocks/files have occupied more than "quota". The grace field will show you the number of days left, before which the number of blocks/files need to go less than the value mentioned in "quota".