Running a Jupyter notebook remotely using port forwarding

After you run through this tutorial once, the most important part to keep referring back to is the summary at the bottom of this page. For your first time through, let's first set up a convenience function. On Mac we just open a terminal on our laptop and do

                    
                        nano ~/.bashrc
                    
                

At the bottom (or top if it's a new file), add

                    
                        sshfw () {
                            ssh -fNL "$1":localhost:"$1" "$2"
                        }
                    
                

then quit with Ctrl+X and say yes when it asks if you want to save. On Windows, we first need to open a powershell and change our settings to allow custom scripts (only ones you write--it will still block potential malicious ones downloaded from the internet so should be safe) with:

                    
                        Set-ExecutionPolicy -ExecutionPolicy RemoteSigned
                    
                

Then try to open your profile using notepad:

                    
                        notepad $PROFILE
                    
                

Unless you've used a powershell profile before, this should bring up a dialog saying that there's no file at that path, in which case you need to close notepad and create one with

                    
                        New-Item -ItemType File -Path $PROFILE -Force
                    
                

Then open it with notepad as above. Then add this function in notepad and save:

                    
                        function sshfw {
                            param([String] $1, [String] $2)
                            ssh -fNL ${1}:localhost:${1} ${2}
                        }
                    
                

In both cases (Windows and Mac) you should now close your terminal / powershell and open a new one so your modified startup script runs in the background. The remaining tutorial assumes you have already gone through all the tutorials above it under Resources. In particular, make sure you have followed Keeping a terminal session running with the screen command to start a jupyter notebook running on one of the workstations (say hr8799). The most common pitfall in all of this is losing track of which computer you're running these different commands. First we need to check things on the remote workstation where we are actually running the notebook. So ssh into hr8799 (assuming that's where you ran the screen command), and do

                    
                        jupyter notebook list
                    
                

The output will look something like this:

If you don't see your jupyter notebook running there, double check that your terminal is running on the workstation (you could open a new terminal and ssh into it again). If you still see no output from jupyter notebook list, then you probably didn't go through the steps in the previous 'screen' tutorial to start it (or you stopped it at some point). Follow that tutorial again.

To connect remotely, we need to know what port to connect to. By default jupyter notebook will try to use port 8888, but it might already be in use by someone else. In that case, it keeps going up until it finds an open port. We can see the port being used boxed in yellow (8890). Highlight the string after 'token=' (red box) and copy it so we can use it later.

To see the output on your own laptop we now have to run an additional command on your laptop (i.e., we don't want to run it on the workstation!). So start by closing the terminal/powershell window and opening a new one (so we are now on your laptop instead of the remote workstation). Then type

                    
                        sshfw 8890 hr8799 
                    
                

'sshfw' (ssh forward) is our custom command we set up at the beginning of this tutorial, where next we put in the appropriate port (here 8890 from the yellow box, but will vary in your own case), and finally the remote workstation (this is the name we chose for hr8799 in our ssh config file in the Simplifying ssh logins tutorial).

If you now open any web browser on your own laptop, and type in the url 'localhost:8890' (replace 8890 with your particular port), you should get the jupyter notebook running on the workstation. You may be prompted for a password or a token. If so, paste the token we copied earlier (from the red box in the picture above), or run through the steps above to copy it again. Now everything with look like a regular jupyter notebook, except it is running remotely on a powerful workstation instead of on your laptop!


Summary

Key: You need to keep track of which computer you're running commands on.

On the remote workstation, e.g., trappist1: ssh into the remote workstation, start a screen session, start a Jupyter notebook, and detach (Keeping a terminal session running with the screen command). If we did that correctly, then if we run the command 'jupyter notebook list' on trappist1, we should see our session, along with its port and token. Once we've copied that token, we can quit out of that terminal since we don't need the workstation anymore.

On your laptop: We now need to start a new terminal or Powershell on our laptop and run the sshfw command to get the notebook output remotely. Assuming our jupyter notebook was running on trappist1 on port 8989, we'd do that with

                    
                        sshfw 8989 trappist1
                    
                

We need to keep that terminal window open (could minimize), then open any browser on your laptop and put localhost:8989 in the URL. It may prompt you for a token, which we copied above and can now paste.

Ater you do start a jupyter notebook screen session and do this once, it gets much easier. My typical workflow is that I'll start a jupyter notebook on the workstation, and leave it running there for a long time (weeks!). After you log in once with a token, it shouldn't ask you again. So each day I work on research, I just have to open a terminal on my own laptop, run e.g., sshfw 8989 trappist1, and then open my browser to localhost:8989 and pick up where I left off (maybe I left something running overnight). Easy!--after several tutorials :)