Page History

...

1. Check Quota for GPFS filesystem:

   Info
   Since blocks on Lilac GPFS home/compute storage are replicated, quota is double the apparent size of data.

   
   

   • • • Lilac home storage :

         Code Block
         language bash theme Midnight firstline 1 title Command line linenumbers true
         mmlsquota lila:home --block-size auto

         
         

       • Lilac compute storage : 

         Code Block
         language bash theme Midnight firstline 1 title Command line linenumbers true
         mmlsquota lila:data_<lab group name> --block-size auto

         
         

         Code Block
         language bash theme Midnight firstline 1 title Command line linenumbers true
         df -h /data/<lab group name> df -ih /data/<lab group name>

         
         

       • Lilac warm storage (oscar) :

         Code Block
         language bash theme Midnight firstline 1 title Command line linenumbers true
         mmlsquota oscar:warm_<lab group name> --block-size auto

         
         

         Code Block
         language bash theme Midnight firstline 1 title Command line linenumbers true
         df -h /warm/<lab group name> df -ih /warm/<lab group name>

         
         

     • 
       

       Tip
       mmlsquota gives information about quota on number of files too, along with information about block quota

       
       

       
       

2. Copy files from other clusters:

• • • • SABA/LUNA/LUX: 
        To copy files from other clusters, first ssh -A into the other cluster to forward your keys.

        Code Block
        language bash theme Midnight firstline 1 title Command line linenumbers true
        ssh -A $USERNAME@$CLUSTER

        We recommend rsync -va to copy files and directories.

        Make note of the source directory/source files and destination directory/files on Lilac and copy them as below:

        Code Block
        language bash theme Midnight firstline 1 title Command line linenumbers true
        rsync -av --progress $SOURCEPATH lilac:$DESTPATH

        
        

      • HAL:
        Remember that the hal cluster is outside the MSKCC network, and does not have access to lilac. 

        First - Make note of the source directory/source files on HAL and destination directory/files on Lilac:
        

        To transfer data, ssh into lilac as below :

        Code Block
        language bash theme Midnight firstline 1 title Command line linenumbers true
        ssh -A $USERNAME@lilac.mskcc.org

        Then pull files from HAL:

        Code Block
        language bash theme Midnight firstline 1 title Command line linenumbers true
        rsync -av --progress hal:$SOURCEPATH $DESTPATH

        
        

        
        

        Tip

        Make sure you calculate the size of data you will copy to lilac, and that you have enough space on lilac to avoid hitting your hard quota. lilac uses data replication to for safety, so a file containing 1G of data consumes 2G of quota on lilac. You can see the size of files and directories with du, which will show 2G for 1G of file data due to replication. To see file size without replication overhead use du --apparent-size instead: Depending on the size and number of files to copy, you may run multiple rsync commands simultaneously to copy different directories. The HPC private network is faster than the MSKCC campus network, so using short names (lilac, saba, luna, selene, etc.) will often make transfers faster than using fully qualified domain names such as luna.mskcc.org. This does not apply to hal, though