Server Deployment Guide

Purpose

This document describes the server-side deployment pattern used for hosting April Robots Online, April Hub, and Keycloak on the april server.

This document intentionally does not cover:

Framework-specific configuration (FastAPI, SvelteKit, etc.)
Application-level environment variables and business logic
Database schemas

Core principles

The server hosts multiple applications simultaneously. Each application:

runs under a dedicated system user
is isolated at the filesystem and process level
listens only on localhost — never on a public interface

Apache2 is the sole entry point for all external HTTP/HTTPS traffic. Direct external access to applications is not permitted.

System user per application

A dedicated system user is created for each application:

adduser --system \
  --home /srv/apps/<app-name> \
  --shell /usr/sbin/nologin \
  --group <app-name>

Add your current user to the group to allow file access during setup:

sudo usermod -a -G <app-name> $USER

Directory structure

All applications live under /srv/apps/:

/srv/apps/<app-name>/
├── app/        # application code or k8s manifests
├── env/        # environment files (never committed)
├── logs/       # application logs
├── run/        # pid / socket files
└── mount/      # persistent data mounted into containers (if applicable)

Set ownership and permissions:

chown -R <app-name>:<app-name> /srv/apps/<app-name>
chmod 750 /srv/apps/<app-name>

For the April Robots Online and April Hub applications, the relevant directories on this server are /srv/apps/april_robots_online/ and /srv/apps/april_hub/. The mount/ subdirectory is used to persist files that need to survive pod restarts — for example, static assets downloaded during cloud sync.

Process management — k3s

April Robots Online, April Hub, and Keycloak run as containers inside a k3s Kubernetes cluster. k3s itself is managed by systemd and starts automatically on boot. Applications inside k3s are managed as StatefulSets — Kubernetes ensures they restart on failure and survive node reboots. See Infrastructure for the full picture of how the cluster is organised and which namespaces exist.

To check the status of k3s:

sudo systemctl status k3s

To inspect running pods across all namespaces:

kubectl get pods -A

Applications that are not containerized (e.g., tools running directly on the host) should use systemd as the process manager for auto-start and auto-restart on failure.

Apache2 — reverse proxy

Apache2 serves exclusively as:

the HTTPS entry point (ports 80 and 443)
TLS termination (Let’s Encrypt certificates via Certbot)
reverse proxy to internal services

Apache2 does not run application code or connect to databases.

General proxying pattern

<VirtualHost *:443>
    ServerName <domain>

    SSLEngine on
    Include /etc/letsencrypt/options-ssl-apache.conf
    SSLCertificateFile /etc/letsencrypt/live/<domain>/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/<domain>/privkey.pem

    ProxyPreserveHost On
    RequestHeader set X-Forwarded-Proto "https"
    RequestHeader set X-Forwarded-Port "443"

    ProxyPass / http://127.0.0.1:<internal-port>/
    ProxyPassReverse / http://127.0.0.1:<internal-port>/
</VirtualHost>

HTTP virtual hosts redirect permanently to HTTPS:

<VirtualHost *:80>
    ServerName <domain>
    RewriteEngine On
    RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

k3s routing

For k3s-hosted applications, Apache proxies to the k3s nginx ingress controller at 127.0.0.1:8080. The ingress controller then routes to the correct service based on the Host header. Apache sets X-Forwarded-Proto: https to prevent redirect loops with TLS-enforcing ingresses.

See Infrastructure for the full routing diagram.

TLS certificates

Certificates are managed with Certbot (Let’s Encrypt) on the host level. To issue a certificate for a new domain:

sudo certbot --apache -d <domain>

Certbot auto-renews certificates via a systemd timer. Verify renewal is active:

sudo systemctl status certbot.timer

k3s also manages its own TLS certificates internally via cert-manager for ingress resources that require it. These are separate from the host-level Certbot certificates. See April Robots Infrastructure — TLS certificate chain for a concrete example.

MongoDB

MongoDB for April Robots Online runs as a StatefulSet inside k3s with a PersistentVolumeClaim for storage. It is not installed directly on the host.

Key points:

MongoDB is only accessible within the april-robots-online namespace via ClusterIP service — never exposed externally.
A dedicated application user with readWrite access on the fastapi database is provisioned at startup via an init script (see Infrastructure — April Robots).
Root credentials are stored in a k8s Secret (mongodb-secret).
Automated backups run twice daily via CronJob and are stored on the host at /mnt/backups/mongodb/. See April Robots Infrastructure — MongoDB backups for the full backup and cleanup schedule.

Secrets and environment variables

Secrets are never stored in any repository. Depending on the deployment type:

Context	Secret storage
k3s applications	Kubernetes Secrets (applied manually from a gitignored `secrets.yml` — see April Robots Infrastructure)
Host processes	`.env` files with `chmod 600`, owned by the app user
CI/CD pipelines	GitHub Secrets and Variables (see Electron Wrapper — Environment variables for an example)

Environment files must not be readable by other users:

chmod 600 /srv/apps/<app-name>/env/.env
chown <app-name>:<app-name> /srv/apps/<app-name>/env/.env

Access to the april server is restricted to SSH key authentication. Password login is disabled. All credentials and SSH keys are stored in PSONO.

Logs

Applications should stream logs to:

stdout / stderr (captured by k3s / journald)
/srv/apps/<app-name>/logs/ for host-level processes

For k3s pods, view logs with:

kubectl logs -n <namespace> <pod-name>
kubectl logs -n <namespace> <pod-name> --follow

Log rotation, format, and retention are left to the individual project team.

Updating and restarting applications

k3s applications

The standard update flow via GitHub Actions:

Build and push a new Docker image to GHCR.
SSH into the server.
Run kubectl rollout restart statefulset/<name> -n <namespace>.
Kubernetes pulls the new image and replaces the pod.
Verify with kubectl get pods -n <namespace>.

Host-level processes

# pull new code
cd /srv/apps/<app-name>/app && git pull

# restart via systemd
sudo systemctl restart <app-name>

# verify
sudo systemctl status <app-name>

What is left to each project team

Framework and runtime configuration
Internal port / socket assignments
Database schema and migrations
Log format and rotation policy
Scaling strategy
CI/CD pipeline implementation

Further read

Infrastructure overview — full server architecture, Apache virtual hosts, k3s cluster, and all hosted projects.
April Robots Online Infrastructure — namespace breakdown, StatefulSets, ingress, backups, monitoring, and CI/CD for AR Online.
April Robots Backends — backend structure for both online and offline applications.
Synchronization flow — how offline apps sync data and robot files with the online backend.
Center authentication — how offline centers authenticate sync requests.
Keycloak — SSO integration across April Robots Online and April Hub.
Electron Wrapper — how the offline app is packaged and distributed as a Windows desktop application.

Keycloak Integration