Welcome to the ML Hackathon 2022 Resources Page

1. Overview

All participants have been added the hackathon class account on UCI’s HPC3 cluster. You are not required to use this resource, but it is there for you. Any accounts created for just the hackathon will be closed approximately two weeks after the conclusion of the event.

Access is via a JupyterHub instance located at https://hpc3.rcic.uci.edu/physicsjhub/. You can login via your UCNetID and password.

The cluster is not being used exclusively by the hackathon. There are others who are sharing this hub, and you should only request the resources that you will really use to accomplish your processing. The defaults are 1 cores and 4GB of memory. You can request up to 4 cores of memory. If you would like to use GPUs, you can request access.
Please do not change browsers (e.g., if you use Chrome to access your lab, always use Chrome). Jupyter writes session cookies and using multiple browsers can lead to very odd error conditions.

This guide will help you get started to launch your own JupyterLab on one of HPC3’s existing computing nodes. The specific JupyterLab instance has access to a subset of software that is available on HPC3 and includes ones that we believe will be quite useful for the hackathon including:

  • R 4.1.2

  • Python 3.8.0 with SciKit Learn, Hugging Face, Pandas and SciPy

  • Pytorch 1.5.1

  • Tensorflow 2.0.0

  • Sqlite3

2. Log in and start a Server

2.1. Log in

You must be on the campus VPN and then point your browser to the hackathon server URL: https://hpc3.rcic.uci.edu/physicsjhub/

When you see the following screen use your UCNetID and password for access:

Hackathon Login
Figure 1. Login page for the hackathon

2.2. Select Jupyter Server Options

When you have logged in you should see the JupyterHub launch screen as on the figure below.

Hackathon Image Select
Figure 2. JupyterHub launch screen page
  • Select the Hackathon 2022 image in the in the Select a Containerized Notebook Image menu.

  • You may have multiple accounts to charge the run, select hackathon in account to charge

  • The partition is set to standard

hackathon account
Figure 3. JupyterHub launch screen page - Account selection

2.3. Start the Server

Once you made your selections press Start to launch your Lab server. If the cluster is not busy and has free resources, your lab should start within 1 minute. You may see a screen that looks similar to:

hub starting
Figure 4. Lab Server starting up

3. The Lab Environment

3.1. Access Python

Your JupyterLab server is a Singularity container that is similar to a Docker container. Singularity requires fewer system-level privileges to run and is generally considered more suitable on a shared cluster. The Hackathon container is a subset of the full HPC3 software stack. A standard Jupyter Notebook can be started from within your lab server. You also have access to a Linux command-line terminal, should you need it.

Here’s what your Lab interface should look like the first time you start. The highlighted area is where you can upload/download files from/to your laptop. The circled areas open empty Python or R notebooks.

hackathon launcher
Figure 5. The initial JupyterLab Startup

The is also a Unix terminal application, you can use it for command-line access. This example shows the available software modules and the modules loaded by default.

hackathon terminal
Figure 6. Command-line terminal

3.2. Add Python packages with pip

You can use Python from .Console, .Notebook or Terminal simply via clicking on the App image in the Launcher window.

While the lab environment is reasonably complete, you may want additional Python modules installed. This example uses pip to install additional packages. In this case, we’ll install the Python package ttictoc. In your lab, click on the Terminal Icon, and then run the command

$ pip install --user ttictoc

Your output should look something like:

lab pip install
Figure 7. Pip --user install

3.3. Access R

The R environment has about 200 R modules installed. R can be run from a jupyter notebook or from the terminal.

3.4. Add R modules

You may need to install additional R modules.

you must be in a terminal to add new R modules as a user. Once you have added the module, you should be able to utilize it in an R notebook.

Once you are in the R environment depending on your desired package you may need to use different options for installing:

  1. Most packages can be installed with the regular install command giving it a desired package name, for example to install cicerone package:

    > install.packages("cicerone")
  2. Some packages can be installed using source in github. Here is an example of installing SCopeLoomR:

    > library('devtools')
    > devtools::install_github("aertslab/SCopeLoomR")

3.5. Access Pytorch and Tensorflow

If you want to use pytorch or tensorflow, simply use them in a python notebook

4. Upload/Download Files

You have a few choices to upload/download files. Among these are

  • Use the upload/download capabilities of the built-in file browser

  • use scp from the Terminal widget in the lab (you must initiate scp from within your lab)

  • use curl to download files from the web.

5. Shutdown your Lab

Please shutdown your lab environment when you don’t need it. This releases backend computing resources for other active computational needs. Your notebooks and other files should be saved prior to shutting down your lab. Jupyter writes a "state" file for your lab, so that when you relaunch, you will be back where you were prior to shutting down.

Your labs have a default 24 hour run time. If you don’t manually shutdown your lab, it will be automatically shutdown by the system.

In the menu, go to File → Hub Control Panel

lab control
Figure 8. Lab control panel

You will be presented with the window where you can press on Stop My Server

lab stop
Figure 9. Stop your Server