Disimula – A Party Game of Secret Missions

Disimula – Monitoring and observability

Structured logs with request ids end to end, business and technical metrics, provisioned Grafana dashboards, alerts to Telegram and an external heartbeat for when the server itself is down.

Observability stack

Logs: one request, one id, everywhere

  • The backend logs one line per API call (RequestLog: method, path, status, duration, caller) and one line per thing that happens in a service (a game created, started, finished; a payment; a generation approved...).
  • In production every line is a JSON object (Spring Boot's structured logging, logstash format), which Loki reads field by field. Locally it stays plain text.
  • Every line carries the request id: Cloudflare's Cf-Ray header when behind Cloudflare, a random one otherwise. It is sent back to the browser as X-Request-Id. From an error a user reports to its log lines, and to Cloudflare's own logs, with one string.
  • 5xx answers are logged as WARN; unexpected errors with their stack trace and a generic 500 internal_error to the client. Health checks are not logged; the AI worker's polling only when it fails.
  • Missions are never logged. Lines carry ids, emails and counts.
  • The online Worker logs on Cloudflare (Workers Logs): every request, plus room_opened, room_deleted, room_expired and room_error.
{service="backend"} | request_id="<the request id>"           everything one request did
{service="backend"} |= "RequestLog" | json | status >= 400     failed API calls
{level="ERROR"}                                               every error on the server

Metrics

SourceWhat
HostCPU, memory, disks
Backend (/actuator/prometheus)requests and latencies, JVM, Hikari connection pool
Backend eventsdisimula_<what>_total: games created, started, finished, deleted; accounts; generations queued and how they ended; checkouts, payments, refunds, webhooks by outcome, payments needing a hand, codes redeemed
Outbound callsdisimula_client_requests_seconds{client, outcome} for Paddle and the online rooms
PostgreSQLsize, connections, transactions, plus usage and sales queries (counts and sums only)
Probesevery 30 s: the frontend, the tunnel's readiness, Loki

Event counters restart at 0 when the backend restarts, which increase() handles; for money and totals, the database queries are the record. Domain code emits events through an Activity port in the shared kernel, implemented with Micrometer, so the domain stays framework-free.

The stack

Grafana, Prometheus, Loki and Grafana Alloy as a separate compose project on the server:

  • Alloy reads the containers' logs through the Docker socket, labels them by project, service, container and level, attaches the backend's request_id as structured metadata, and scrapes metrics.
  • Everything is kept 30 days; every service is memory-capped, around 600 MB in total.
  • Grafana is not on the internet: no tunnel hostname, LAN only, optionally bound to localhost and reached through SSH.

Everything is provisioned

Nothing in Grafana is configured by hand:

WhatWhere
What is collectedconfig.alloy, postgres-queries.yaml
Data sourcesgrafana/provisioning/datasources/
Dashboardsgrafana/dashboards/*.json
Alert rulesgrafana/provisioning/alerting/rules.yaml
Contact points, message templatescontact-points.yaml, templates.yaml
Secrets (bot token, heartbeat URL)the server's env file, never the repository

Saving from Grafana's UI is disabled, so no change can live only there: you edit, Grafana refuses to save and shows the JSON, which goes back into the repository. Before every production deploy, check.sh loads all of it in a throwaway Grafana and stops the deploy if anything is broken.

Dashboards

  • Home server (opens first): the machine, every service up or down, the backend, warnings and errors of every container.
  • Disimula: how the app is used: games, players, people, AI generations, recent events.
  • Disimula billing: what sells, refunds, Paddle's webhooks and API latency.
  • Disimula logs: every line, searchable by request id, text and level.
  • Disimula database: size, connections, transactions and backups.

Linked to each other in a bar at the top (by tag).

Alerts

To a Telegram chat, one short message each: what is wrong and what command to run. A problem that stays is repeated every 12 hours, and its resolution is announced.

AlertWhen
Backend down / a service not answering / database downno answer for a few minutes
Answering with errorsany recent 5xx
Logged an erroran ERROR line from any of its services
Slowp95 latency too high for a while
Waiting for database connectionsrequests queueing for the pool
Running out of memoryheap close to full for a while
Nobody is writing the queued AI generationsqueued, none running, for a while
An AI generation failedas it happens
Cannot reach Paddle or the online roomsan outbound call with no answer or a 5xx
Paddle's webhooks are being rejectedinvalid signatures arriving
No recent database backupnone in the last day
Disk, memory, CPUrunning low or saturated
The monitoring itselfAlloy, Prometheus or Loki stopped for 5 min

Who watches the watcher

All those alerts are sent from the server, so a power cut or a dead connection would silence them. Grafana therefore pings an external heartbeat (healthchecks.io, free) every 5 minutes; if the pings stop for 10 minutes, healthchecks.io alerts the same Telegram chat. The heartbeat proves Grafana runs and reaches the internet; the other alerts cover the rest.

In the app

Besides Grafana, admins have the in-app panels (usage, AI usage, user explorer, billing), built on SQL read models in the stats feature, for questions about the product rather than the machine.