diff --git a/docs/Software/Available_Applications/ollama.md b/docs/Software/Available_Applications/ollama.md new file mode 100644 index 000000000..91603ea67 --- /dev/null +++ b/docs/Software/Available_Applications/ollama.md @@ -0,0 +1,53 @@ +--- +created_at: 2026-05-04 +description: How to run ollama on the REANNZ GPUs +tags: + - llm +--- + +## Starting ollama in a Slurm job + +!!! warn + We don't reccomend running ollama like this except for small test jobs. + It is a very inefficient use of GPUs. + + +```sl +#!/bin/bash -e + +#SBATCH --account nesi99991 +#SBATCH --job-name ollama test +#SBATCH --time 01:00:00 +#SBATCH --mem 10G +#SBATCH --gpus-per-node l4:1 + +PORT=16000 # please choose your own port number between 1024 and 49151 + +module purge +module load ollama +export OLLAMA_HOST=${HOSTNAME}:${PORT} +ssh -NfR ${PORT}:${HOSTNAME}:${PORT} ${SLURM_SUBMIT_HOST} + +ollama serve +``` + +Then on the login node run, + +```sh +module purge +module load ollama +export OLLAMA_HOST=: +ollama +``` + +Where `` is the host name of the node running your job (you can find this with `sacct` or `squeue --me`), +and `` is your selected port. + +!!! tip + For debugging set + + ```sh + GIN_MODE=debug + ``` + + before starting `ollama`.