General Documentation

LSF Primer

Lilac Cluster Guide

Juno Cluster Guide

Cloud Resources

Backup Policy on server/node local drives

File lists

Versions Compared

Old Version 1

changes.mady.by.user Joanne Edington

Saved on

compared with

New Version Current

changes.mady.by.user Joanne Edington

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

This page documents the various questions related to storage on the Juno storagecluster. 

Juno home storage : 

    • Description : Isilon GPFS shared parallel filesystem.
    • Purpose: To store software related code and scripts, default quota size is small and fixed.
    • Mount: /home/<user>
    • Access: All Juno nodes, including compute storage and login nodes.
    • Default quota: 100GB
    • Snapshots: 7 days of snapshots. ( not backed up ). Can be accessed in /home/.snapshots/<user>
    • Replicated: yes

Juno compute storage : 

    • Description : GPFS shared parallel filesystem, not replicated, and not backed up.
    • Purpose: For jobs to read and write compute data from login and compute nodes, default quota size is larger with flexibility to request larger quota.
    • Mount: /juno/res/<lab group>, /juno/work/<lab group>.
    • Access: All Juno nodes, including compute storage and login nodes.
    • Default quota: 1TB ( Increased/Decreased on request )
    • Snapshots: 7 days of snapshots. ( not backed up ). Can be accessed in /data/.snapshots/<date>/<lab group>
    • Replicated: no

Juno warm storage : 

    • Description : GPFS shared parallel filesystem, not replicated but can be backed up upon request. 
    • Purpose: To store long term data. Only accessible from login nodes and cannot be accessed from compute nodes.
    • Mount: /warm/<lab group>
    • Access: Only Juno  login nodes. 
    • Default quota: 1TB ( Increased/Decreased on request )
    • Snapshots: 7 days of snapshots. ( not backed up ). Can be accessed in /warm/.snapshots/<date>/<lab group>
    • Replicated: no 

...

Local scratch storage : 

  • Description : local Local SATA drive, not replicated, and not backed up. Local and to each compute node, not a shared filesystem
  • Purpose: To store local temporary data related to compute jobs. Since this is not a shared filesystem, the temporary data needs to be cleaned up and copied back to shared filesystem after job completion.  Files older than 31 days are automatically deleted.
  • Mount: /scratch/
  • Access: Local to the compute node.
  • Default quota: No quota and limited to free disk space in /scratch.
  • Snapshots: No snapshots.
  • Replicated: no

...

Local fast scratch storage : 

  • Description : local Local NVMe drive, not replicated, and not backed up. Local and to each compute node, not a shared filesystem
  • Purpose: To store local temporary data related to compute jobs. Since this is not a shared filesystem, the temporary data needs to be cleaned up and copied back to shared filesystem after job completion.  Files older than 31 days are automatically deleted.
  • Mount: /fscratch/
  • Access: Local to the compute node. Not available on all nodes. Must be requested on bsub with -R fscratch
  • Default quota: No quota and limited to free disk space in /fscratch.
  • Snapshots: No snapshots.
  • Replicated: no

How to get information about your GPFS storage usage:

You can get real time information about storage quotas and usage from grafana at https://hpc-grafana.mskcc.org/d/000000027/storage-quota

...

       mmlsquota -j work_hpcadmin  lab  --block-size auto juno

or

df        df -h  /data/$LAB

lilac also GPFS also has quotas on number of files -- not just data size.

 Copy files from other clusters:

Depending on the size and number of files to copy, you may run multiple rsync commands simultaneously to copy different directories.

...

.

mmlsquota gives information about quota on number of files too, along with information about block quota.

Filesystem

Fileset

type

blocks

quota

limit

in_doubt

grace

|

files

quota

limit

in_doubt

grace

Remarks

Filesystem name

Fileset name

fileset/usr/grp

Blocks currently occupied

Your block quota

Your limit for "7 days" beyond quota.

Blocks in doubt that will be counted towards your quota.

Happens when many files are added/deleted recently.

Countdown of 7 days is set once you occupy more blocks than mentioned in quota.


Number of files currently present

Your quota on number of files

Your limit on number of files for "7 days" beyond quota.

Number of files in doubt that will be counted towards your quota.

Countdown of 7 days is set once you occupy more files than mentioned in quota.


Once the number of blocks or number of files reach the value mentioned in "quota" - Storage system will give 7 days as a grace period, to fill up until the max value mentioned in "limit" is reached. Storage system will not allow any more data to be written when:

  1. The block limit/file limit is reached.
  2. 7 days have passed since the blocks/files have occupied more than "quota". The grace field will show you the number of days left, before which the number of blocks/files need to go less than the value mentioned in "quota".