April Robots Online Infrastructure

Overview

April Robots Online (AR Online) runs in the april-robots-online namespace inside the k3s cluster on the april server. The namespace contains three stateful workloads (frontend, backend, MongoDB), two ingresses, monitoring tools, scheduled backup jobs, and RBAC resources for CI/CD automation.

The infrastructure is defined as Kubernetes manifests in the april-robots-online repository under the k8s/ directory, managed with Kustomize.

Repository structure

k8s/
├── kustomization.yml          # root — includes base/app only
├── secrets.yml                # actual secret values (gitignored, applied manually)
├── base/
│   ├── app/                   # main application resources
│   │   ├── kustomization.yml
│   │   ├── backend/
│   │   │   ├── deployment.yml
│   │   │   └── service.yml
│   │   ├── frontend/
│   │   │   ├── deployment.yml
│   │   │   └── service.yml
│   │   ├── mongodb/
│   │   │   ├── deployment.yml
│   │   │   ├── service.yml
│   │   │   ├── pvc.yml
│   │   │   ├── backup.yml
│   │   │   ├── cleanup.yml
│   │   │   └── mongodb-ensure-users-configmap.yml
│   │   ├── ingress-backend.yml
│   │   ├── ingress-frontend.yml
│   │   └── issuer.yml
│   ├── core/                  # one-time bootstrap resources (namespace, secrets, GHCR pull secret)
│   │   ├── namespace.yml
│   │   ├── secrets.yml
│   │   └── gh_secret.yml
│   ├── metrics/               # monitoring stack (Prometheus + Grafana)
│   │   ├── grafana/
│   │   └── prometheus/
│   └── utils/                 # RBAC for CI/CD
│       ├── role.yml
│       └── rolebinding.yml
└── overlays/
    ├── prod/
    │   └── kustomization.yml  # injects secrets, variables, etc.
    └── staging/
        └── kustomization.yml

Architecture

    ---
config:
  flowchart:
    nodeSpacing: 25
    rankSpacing: 55
  layout: elk
title: Namespace — april-robots-online
---
flowchart TB

subgraph External["External"]
  Users["Specialists, Parents, Admins"]
  OfflineApp["Offline App (BackCore)\nsync requests"]
  GHCR["GHCR (Novators-kz)\ndocker images"]
end

subgraph NS["Namespace: april-robots-online"]
  subgraph Ingress["Ingress (nginx)"]
    FEIng["robots.aprilhub.kz\nTLS via cert-manager"]
    BEIng["api.robots.aprilhub.kz\nHTTP only (TLS at Apache)"]
  end

  subgraph Workloads["StatefulSets"]
    FE["april-robots-online-fe\nSvelteKit NodeJS :3000"]
    BE["april-robots-online-general-api\nFastAPI :8000"]
    MDB[("mongodb\n:27017")]
  end

  subgraph Mon["Monitoring"]
    Prom["Prometheus :9090"]
    Graf["Grafana\ngrafana.staging.robots.aprilhub.kz"]
  end

  subgraph Jobs["CronJobs"]
    BJob["mongodb-backup\n0 2,12 * * * Asia/Almaty"]
    CJob["mongodb-backup-cleanup\n0 3 * * * Asia/Almaty"]
  end
end

subgraph HostFS["Host Filesystem"]
  Static[("/srv/apps/april_robots_online/mount/static/")]
  Backups[("/mnt/backups/mongodb/")]
end

Users -->|"robots.aprilhub.kz"| FEIng
OfflineApp -->|"api.robots.aprilhub.kz"| BEIng
FEIng --> FE
BEIng --> BE
FE <-->|"API calls"| BE
BE --> MDB
BE --- Static

Prom -->|"GET /prometheus (Bearer)"| BE
Graf --> Prom

BJob -->|"mongodump"| Backups
CJob -->|"rm dirs older than 14d"| Backups

GHA -->|"ssh → kubectl rollout restart"| FE & BE
GHA -->|"push image"| GHCR
GHA["GitHub Actions"]
  

Application workloads

All three main workloads run as StatefulSets with replicas: 1. StatefulSets are used rather than Deployments to ensure stable pod identity, which is important for the persistent volume and hostPath mounts.

Frontend

• Image — ghcr.io/novators-kz/april-robots-online-fe:<IMAGE_VERSION>
• Container port — 3000 (SvelteKit NodeJS full-stack app)
• Service port — 80 → 3000
• Config — environment variables from env-config ConfigMap (injected by Kustomize overlay)
• Body size limit — BODY_SIZE_LIMIT=104857600 (100 MB) set directly on the container

Backend

• Image — ghcr.io/novators-kz/april-robots-online-general-api:<IMAGE_VERSION>
• Container port — 8000 (FastAPI)
• Service port — 8000 → 8000
• Config — environment variables from env-config ConfigMap; secrets (JWT_SECRET, MongoDB credentials) injected from k8s Secrets
• Init container — wait-for-mongodb (busybox) polls mongodb:27017 with nc -z every 5 seconds before the backend container starts
• Static files — /app/static is mounted from the host path /srv/apps/april_robots_online/mount/static/ (type DirectoryOrCreate). This directory holds static assets (images) downloaded during cloud sync

Resource limits for both frontend and backend:

MongoDB

• Image — mongo:4.4.6-bionic
• Port — 27017
• Storage — PersistentVolumeClaim mongodb-pvc (2 Gi, ReadWriteOnce)
• Root credentials — MONGO_INITDB_ROOT_USERNAME / MONGO_INITDB_ROOT_PASSWORD from mongodb-secret
• App user — created at startup by ensure-users.js (mounted from ConfigMap mongodb-ensure-users-script into /docker-entrypoint-initdb.d/). The script authenticates as root, then creates an app user with readWrite on the fastapi database. Duplicate user creation is silently skipped

The app user credentials (MONGOUSER, MONGOPASSWORD) are injected into the backend container from mongodb-secret.

Networking

Services

All services are ClusterIP — accessible only within the cluster.

Ingress

Two ingress resources route external traffic into the namespace. Both use the nginx ingress class, which is the k3s nginx ingress controller running at port 8080 on the host (proxied from Apache).

Frontend ingress (robots.aprilhub.kz) — TLS is managed by cert-manager using the letsencrypt-prod ClusterIssuer. The TLS certificate is stored in secret fe-tls-secret (name injected by the Kustomize overlay via FE_TLS_SECRET_NAME). ssl-redirect: true is set; Apache’s X-Forwarded-Proto: https header prevents a redirect loop.

Backend ingress (api.robots.aprilhub.kz) — no TLS at the k3s level; Apache terminates HTTPS externally using a host-level Let’s Encrypt certificate. The ingress rewrites the path to /.

TLS certificate chain

Let's Encrypt CA
  ├── Host cert (Certbot)     — used by Apache2 for robots.aprilhub.kz and api.robots.aprilhub.kz
  └── k3s cert (cert-manager) — used by nginx ingress for robots.aprilhub.kz (direct cluster access)

The letsencrypt-prod ClusterIssuer (issuer.yml) uses the ACME HTTP-01 challenge solver via nginx.

MongoDB backups

Two CronJobs in the namespace manage automated backups. Both run in the Asia/Almaty timezone and use mongo:4.4.6-bionic with hostPath volume /mnt/backups/mongodb/.

mongodb-backup — schedule 0 2,12 * * * (daily at 02:00 and 12:00).

Runs mongodump using root credentials from mongodb-secret. Each run creates a timestamped directory under /mnt/backups/mongodb/<YYYY-MM-DD-HH-MM>/.

mongodb-backup-cleanup — schedule 0 3 * * * (daily at 03:00, one hour after the morning backup).

Runs find /backup/* -maxdepth 0 -type d -mtime +14 -exec rm -rf {} \; to delete backup directories older than 14 days. This keeps the backup volume from growing unboundedly.

Monitoring

Prometheus and Grafana run in the april-robots-online namespace alongside the application, applied manually from base/metrics/ (not part of the regular Kustomize deploy flow).

Prometheus scrapes the backend at april-robots-online-general-api:8000/prometheus every 15 seconds using a Bearer token for authorization. Data retention is set to 15 days (--storage.tsdb.retention.time=15d). Prometheus storage uses an emptyDir volume — metrics are not persisted across pod restarts.

Grafana is exposed via ingress at grafana.staging.robots.aprilhub.kz and is configured with Prometheus as a data source. It uses emptyDir for data storage. Default admin password is admin with sign-up disabled.

What still needs to be done:

• Persistent volumes for both Prometheus and Grafana
• Grafana dashboards
• Alerting rules in Prometheus
• Proper credentials for Grafana (replace default admin password)
• Grafana ingress for the prod environment

Kustomize overlays

Two overlays exist — overlays/prod/ and overlays/staging/ — and their kustomization.yml files are structurally identical. Both reference ../../base/app and apply the same configMapGenerator + replacements pattern. The only difference between environments is the .env file that lives alongside each kustomization.yml on the server (not committed to the repository).

The configMapGenerator reads the .env file and produces a ConfigMap named env-config (disableNameSuffixHash: true keeps the name stable so StatefulSets can reference it without knowing the content hash). Kustomize replacements then patch the generated manifests, injecting the env values into the correct resource fields before kubectl apply.

Variables in .env and where they are injected:

The server described in this document runs the prod overlay. Prod and staging differ only in .env values (different hostnames, TLS secret names, and potentially different image versions).

Deployment and CI/CD

When a new version is ready, the GitHub Actions workflow:

1. Builds and pushes the new Docker image to GHCR with the updated tag.
2. SSHes into the april server.
3. Runs kubectl rollout restart on the frontend and backend StatefulSets.

Kubernetes then pulls the new image from GHCR (using ghcr-secret) and replaces the running pod.

RBAC for GitHub Actions

base/utils/ defines a minimal RBAC setup for the deployment step:

• Role restart-only — allows get and patch on StatefulSets in the namespace, which is the minimum needed to trigger a rollout restart.
• RoleBinding gh-deploy-restart — binds the role to ServiceAccount gh-deploy in the april-robots-online namespace.

Secrets

Three secrets are maintained in the namespace.

backend-secret — contains JWT_SECRET used by the FastAPI backend for signing authentication tokens.

mongodb-secret — contains:

• MONGO_INITDB_ROOT_USERNAME / MONGO_INITDB_ROOT_PASSWORD — MongoDB root credentials used by the init container and backup jobs
• MONGO_APP_USER / MONGO_APP_PASS — application-level credentials with readWrite on the fastapi database
• MONGO_USERS_LIST — semicolon-delimited list parsed by ensure-users.js to provision users at startup

ghcr-secret — a kubernetes.io/dockerconfigjson secret used as imagePullSecret on both frontend and backend pods to authenticate with the private GHCR registry (ghcr.io/novators-kz/...).