The Jupyter Notebooks allow you to have a very nice web UI to develop and run your applications.
To access the Jupyter web UI first you have to start the VPN.
To start Jupyter you can run:
and it will provide you the address you need to point your browser.
Under the hood the Jupyter notebooks runs a Spark session to run your Spark commands in the cluster.
If you want to customize the version of Python used you can do it by loading first the Anaconda module you are interested in, and after that running Jupyter. For example:
module load anaconda3/2018.12 start_jupyter
You can also try the new Jupyter Lab interface:
module load anaconda2/2018.12 start_jupyter-lab