Storage Quotas on the UEB HPC¶

To keep the shared storage responsive and reliable for everyone, I enforce quotas on user home directories. Quotas prevent a few large jobs from filling the filesystem and degrading service for the whole cluster.

Why quotas are necessary¶

Our RAID-backed shared storage is limited to 46 TB. With ever-growing datasets in today’s omics and protein modeling, this capacity can be consumed quickly. When the shared filesystem approaches full capacity:

Jobs fail unpredictably (no space for temp files or logs).
I/O latency spikes for all users; NFS can become unstable.
Metadata exhaustion (inodes) breaks tools that create many small files.
Backups and snapshots become slow or impossible.

Goal: keep usage well below the danger zone so the cluster remains fast and stable.

What the quota covers¶

Your home directory (e.g., /home/<user>) on the HPC storage.
Both space (blocks/GB) and file count (inodes) may be limited.

Rule of thumb: keep home for code, configs, small results, and documentation. Place large data, raw reads, and bulky intermediates outside home (e.g., project shares, scratch, or your own NAS, see Offloading to Synology NAS below).

Soft vs. hard quota¶

Soft quota: a limit you’re allowed to exceed temporarily (grace period). You’ll see warnings; please clean up or move data.
Hard quota: absolute limit. Writes will fail once you hit this number (no grace).

Grace period¶

I set a grace window (e.g., 14 days). If you remain over soft quota after the grace expires, the soft limit behaves like a hard limit until you drop below it.

How to check your quota and usage¶

On any cluster node:

# Human-readable space and inode usage per filesystem
quota -s

# Or verbose
quota -v

# Your home usage summary
du -sh $HOME

# Biggest directories inside your home
du -h -d 1 $HOME | sort -h
# or deeper:
du -h -d 2 $HOME | sort -h | tail -n 30

# Filesystems and free space view
df -h $HOME

Example quota -s output (illustrative for user pavel):

Disk quotas for user pavel (uid 1234):
Filesystem  space   quota   limit   grace   files   quota   limit   grace
/home       85G     100G    120G             520k   700k    800k

- quota = soft limit, limit = hard limit. Keep well under soft.

If you are over quota¶

Identify large items (see du commands above).
Remove intermediates and temporary files (tmp, cache, .conda/pkgs, etc.).
Compress lots of small text files into an archive (e.g., tar zcf results.tgz results/).
Move large datasets off home (see Offloading to Synology NAS below).
If the data is both large and needed on the cluster, ask admins about a project share or a temporary quota extension (justify briefly).

Offloading data to a Synology NAS (NFS)¶

For users who need to keep large data secure and conveniently accessible but outside HPC storage, the NAS is the best option. I personaly use Synology NAS, a very affordable solution that enables me to manage, secure, and protect my data. I set Synology NAS as a shared folder via NFS. Mounting it on the cluster keeps my workflows simple while protecting the HPC home from growth.

Good housekeeping on shared storage¶

Keep one canonical copy of raw data; avoid duplicating fastq/bam across projects.
Clean up large temporary directories after successful runs.
Log software versions and parameters so results are reproducible without keeping huge intermediates.
For pipelines, write intermediates to a fast scratch/project path and publish only final outputs to home or NAS.

FAQ¶

Q: My write failed with “Disk quota exceeded”. What now?
A: You hit the hard limit. Free space under your home (or move data to NAS) and retry. Writes will succeed again after you’re below soft quota.

Q: Does quota count inodes (file count)?
A: Yes, it may. If you create millions of small files, you can hit the inode limit before space. Archive/bundle small files.

Q: Can I get more quota?
A: Send me a request with a short justification (size, duration, project). Temporary bumps are possible depending on capacity.

Q: Is scratch exempt?
A: No. Scratch/project areas and /tmp are under user space so quota applies.

How to mount external NAS¶

Mounts require admin action. Send me your request an NFS mount on the cluster.¶

Provide:

NAS hostname/IP
Export path (e.g., nas:/volume1/ngs-pavel)
NFS version (recommend NFSv4.1), and any required options (e.g., rw,noatime,nconnect=8)

Upon receiving the request I will create the etc/fstab line and mount your NAS directory under /mnt with an exclusive access and symlink in your home dir.

Verify the mount (as user, NAS mountead as /mnt/NAS_pavel + symlink in home dir)¶

# After the NAS is mountedcheck it system-wide:
df -h $HOME/NAS_pavel
touch $HOME/NAS_pavel/.write_test && rm $HOME/NAS_pavel/.write_test

Moving data off HPC home to NAS¶

Don’t run cleanup on the login node! The login node has only one CPU with 4 cores, it will be heavy; use a batch slurm script!:

#!/usr/bin/env bash
#SBATCH -J mv_seqdata_to_synology
#SBATCH -p short
#SBATCH -c 2
#SBATCH --mem=4G
#SBATCH -t 00:30:00
#SBATCH -o %x.%j.out
#SBATCH -e %x.%j.err

set -euo pipefail

SRC="$HOME/Seq_Data"
DST_BASE="$HOME/NAS_pavel"                 # NFS mount point
DST="$DST_BASE/Seq_Data"                   # final destination subdir
LOG="$PWD/rsync_seqdata_$(date +%F_%H%M%S).log"

# 0) Sanity checks
[[ -d "$SRC" ]] || { echo "Source not found: $SRC"; exit 1; }
[[ -d "$DST_BASE" ]] || { echo "Destination base not found: $DST_BASE"; exit 1; }
mountpoint -q "$DST_BASE" || { echo "Destination is not a mountpoint: $DST_BASE (is Synology NFS mounted?)"; exit 1; }

1) Ensure destination exists¶

mkdir -p "$DST"

2) Space check¶

need_bytes=$(du -sb "$SRC" | cut -f1)
avail_bytes=$(df -B1 --output=avail "$DST_BASE" | tail -1)
if (( avail_bytes < need_bytes )); then
  echo "Not enough space on $DST_BASE: need ${need_bytes}B, have ${avail_bytes}B"
  exit 1
fi

3) TEST-RUN first (uncomment to preview)¶

rsync -aHAX --info=stats2,progress2 --dry-run "$SRC/" "$DST/" | tee "$LOG.dryrun"

4) Move: copy then delete sources that copied OK¶

rsync -aHAX --info=stats2,progress2 --partial  --remove-source-files   "$SRC/" "$DST/" | tee "$LOG"

5) Clean up empty source directories after a successful move¶

find "$SRC" -type d -empty -delete
echo "Done. Moved content from $SRC -> $DST"

Tips for performance over NFS:

Use fewer, larger files (archive tiny files).

Prefer sequential I/O (rsync, tar) over millions of random small writes.

6) Space check¶

need_bytes=$(du -sb "$SRC" | cut -f1)
avail_bytes=$(df -B1 --output=avail "$DST_BASE" | tail -1)
if (( avail_bytes < need_bytes )); then
  echo "Not enough space on $DST_BASE: need ${need_bytes}B, have ${avail_bytes}B"
  exit 1
fi

7) DRY-RUN first (uncomment to preview)¶

rsync -aHAX --info=stats2,progress2 --dry-run "$SRC/" "$DST/" | tee "$LOG.dryrun"

8) Move: copy then delete sources that copied OK¶

rsync -aHAX --info=stats2,progress2 --partial  --remove-source-files   "$SRC/" "$DST/" | tee "$LOG"

9) Clean up empty source directories after a successful move¶

find "$SRC" -type d -empty -delete
echo "Done. Moved content from $SRC -> $DST"