Jupyter Notebook should only be run on compute nodes through LSF as an interactive job, not on the login servers juno or lilac. You will be able to connect to it through SSH tunnels.
Information about submitting LSF jobs is at http://mskcchpc.org/display/CLUS/LSF+Primer. Your interactive LSF job should request the resources your Jupyter notebook will need.
Once you are comfortable to submitting lsf jobs
1. download miniconda and install Jupyter notebook with miniconda
2. start an interactive session with bsub and start Jupyter notebook
3. Pay attention to the message showing on the screen once you start Jupyter notebook. It will tell you the actual node you are on and the port number ,etc.
Here is an example of a simple LSF interactive job submission:
>bsub -n 2 -W 4:00 -R 'span[hosts=1] rusage[mem=8]' -Is /bin/bash
This example requests a GPU for NLP(natural language processing on Lilac:
username@lilac:~> bsub -n 2 -W 2:00 -q gpuqueue -gpu "num=1" -R "span[hosts=1]" -Is /bin/bash
After you submit your interactive bsub you will get a prompt on a compute node when one is available. The compute node name will be in the prompt.
start jupyter-notebook on the "node" you land on, not on lilac!!!
username@lt14:~> ./miniconda3/bin/jupyter-notebook --no-browser --port 9898
Pay attention to the url of the jupyter-notebook from the output and the compute node name in the prompt. You will need these to connect to it.
You can connect to jupyter-notebook using a SSH tunnel. When you create a SSH tunnel, it associates a specific port on the remote system to one on the local system. The basic command to create a SSH tunnel that passes through the login node to the compute node on your laptop is:
"ssh -L<Port-A>:localhost:<Port-B> user@Host-B -t ssh -L<Port-B>:localhost:<Port-C> user@Host-C"
To set up tunnels through lilac to a compute node lt14 running jupyter–notebook on port 9898:
>ssh -L 9898:localhost:9898 lilac ssh -L 9898:localhost:9898 -N lt14
Then open a Browser on your laptop:(You will be able to find the exact url from the output of the jupyter-notebook)
http://127.0.0.1:9898/?token=40756158b4856f474dcc07912a7027a6c14903df684f6bf8
Here are some useful links you can read:
https://www.digitalocean.com/community/tutorials/how-to-install-run-connect-to-jupyter-notebook-on-remote-server
https://medium.com/@sankarshan7/how-to-run-jupyter-notebook-in-server-which-is-at-multi-hop-distance-a02bc8e78314
http://joehamman.com/2013/10/04/remote-hosting-of-ipython-notebooks/