Jupyter Notebook should only be run on compute nodes through LSF as an interactive job, not on the login servers juno or lilac. You will be able to connect to it by setting up SSH tunnels. You will need to make sure that you have enabled SSH authentication forwarding as per Secure Shell SSH.
Information about submitting LSF jobs is at http://mskcchpc.org/display/CLUS/LSF+Primer. Your interactive LSF job should request the resources your Jupyter notebook will need. These are the steps for starting and connecting to your jupiter-notebook on the cluster:
- Download miniconda and install Jupyter notebook with miniconda.
- Start an interactive session with bsub and start jupyter-notebook.
- Pay attention to the message showing on the screen once you start jupyter-notebook. It will tell you the actual node you are on, the port number and the URL.
- Set up SSH tunnels to the node running jupiter-notebook.
- Connect using the browser on your local machine.
Here is an example of a simple LSF interactive job submission:
username@lilac:~>bsub -n 2 -W 4:00 -R 'span[hosts=1] rusage[mem=8]' -Is /bin/bash
This example requests a GPU for NLP(natural language processing) on Lilac:
username@lilac:~> bsub -n 2 -W 2:00 -q gpuqueue -gpu "num=1" -R "span[hosts=1]" -Is /bin/bash
After you submit your interactive bsub, you will get a prompt on a compute node when one is available. The compute node name will be in the prompt.
start jupyter-notebook on the "node" you land on, not on lilac!!!
username@lt14:~> ./miniconda3/bin/jupyter-notebook --no-browser --port 9898
Pay attention to the url of the jupyter-notebook from the output and the compute node name in the prompt. You will need these to connect to it.
You can connect to jupyter-notebook by setting up a SSH tunnel. When you create a SSH tunnel, it associates a specific port on the remote system to one on the local system. SSH authentication forwarding must be configured for this to work. The basic command to create a SSH tunnel that passes through the login node to the compute node on your laptop or local server is:
"ssh -L<Port-A>:localhost:<Port-B> user@Host-B -t ssh -L<Port-B>:localhost:<Port-C> user@Host-C"
This example sets up SSH tunnels from your laptop through lilac to a compute node lt14 running jupyter–notebook on port 9898:
yourlaptop>ssh -L 9898:localhost:9898 lilac ssh -L 9898:localhost:9898 -N lt14
Then open a Browser on your laptop. The exact url is in the output of the jupyter-notebook.
http://127.0.0.1:9898/?token=40756158b4856f474dcc07912a7027a6c14903df684f6bf8
Here are some useful links you can read:
https://www.digitalocean.com/community/tutorials/how-to-install-run-connect-to-jupyter-notebook-on-remote-server
https://medium.com/@sankarshan7/how-to-run-jupyter-notebook-in-server-which-is-at-multi-hop-distance-a02bc8e78314
http://joehamman.com/2013/10/04/remote-hosting-of-ipython-notebooks/