A server ran out of memory and hung. Nothing told us. We found out when a site was down.
We had tools. Horizon watched the queues, Telescope the requests, Sentry the errors. Each one watched one app. Telescope was too heavy to leave on in production. Reading a container’s logs meant opening a shell on the server. Mail delivery lived in the Resend dashboard, away from the app that sent the mail.
The brief was one place that hears every app on every server, keeps enough history to show a trend, and speaks first. One limit shaped every decision after it: Carabdheer itself stays under 512 MB of memory, not counting the database.
Three ways in, one door
Every event takes the same road, and the database is the last stop. Nothing touches it while the sender waits, so a burst of errors from one app never slows another app’s requests.
- Three ways in
A thin Laravel package sends exceptions, sampled requests, slow queries, jobs, mail and backup events in batches. A host agent reports memory, CPU, disk and containers every ten seconds. Webhooks bring deploys, backups, mail events and GitHub issue state.
- One door
A single ingest endpoint finds the workspace from the key and answers 202 at once. Over its rate limit, it answers 429.
- A queue
The batch goes onto Redis. The sender is already gone.
- The worker switches
A worker picks the batch up, switches to that workspace’s database and writes.
- Today’s partition
Rows land in the day they belong to, ready to be dropped whole later.
The agent is a small Go binary that runs as a container. It mounts the Docker socket and /proc read-only and uses 10 to 15 MB. It exists because of the socket: whoever holds the Docker socket is root on the host, and the web app should never have that.
| Turned down | Why not |
|---|---|
| Laravel reads the socket | Root for the web app, and a PHP loop polling every ten seconds. |
| A systemd binary | Lives outside Coolify. Every upgrade by hand, on every server. |
| cAdvisor or Vector | Heavier, and their output still has to be mapped into ours. |
As a container it deploys like everything else on the server. A second server is the same container with a different key.
Drop a day, not a row
Logs, error events, requests and host metrics are most of what Carabdheer stores, and most of it matters only for a while. Each of those tables is split by day in Postgres 17. Pruning never runs a DELETE. At 03:30 every day that has left its keep window is dropped whole. Dropping a partition is instant and leaves nothing behind for vacuum.
Before raw detail goes, it has already been rolled up by hour and by day. The rollups stay forever, so a year’s trend survives after last month’s log lines are gone.
| Data | Kept |
|---|---|
| Logs | 14 days |
| Host metrics, raw every 10 s | 7 days |
| Sampled requests, jobs | 30 days |
| Error events, mail events | 90 days |
| Error issues | Forever |
| Hourly and daily rollups | Forever |
Search runs on pg_trgm and full-text indexes. That is enough for level:error app:web-1 across fourteen days, which is the question people actually ask.
| Turned down | Why not |
|---|---|
| ClickHouse | Needs 1 to 2 GB of memory before it does any work. That alone breaks the limit. |
| Loki | A second system to run, back up and upgrade beside the database. |
| MySQL | Weaker partition drop and weaker text search. |
| SQLite | One writer at a time. Ingest is many writers. |
A database for each workspace
One central database knows who everyone is and where each workspace lives. Everything a workspace hears goes into its own database, with its own partitions and its own nightly prune.
| Where | What lives there |
|---|---|
| Central | People, workspaces, invites, ingest keys, roles, report links, and the map from each workspace to its database. |
| crb_ws_<id> | Apps, hosts, events, logs, metrics, issues, alert rules, settings, rollups. |
| Web request | Switches by the workspace in its session. |
| Ingest request | Switches by its key. |
| Queued job | Switches by the workspace id it carries. |
A heavy workspace never shares a table, an index or a prune run with a light one. Exporting a workspace is one pg_dump. Deleting one drops its database, and nothing is left to hunt down.
A schema per workspace was on the table: the same isolation per table, one connection pool. We chose full databases and paid for it. Each database has its own pool, so PgBouncer sits in front of Postgres in transaction mode, and every migration runs once per workspace.
The switching is our own code. Two connections, central and workspace. One call sets the database name, purges and reconnects, and it happens in the three places in the table above, nowhere else.
| Turned down | Why not |
|---|---|
| stancl/tenancy | Large, built around subdomains we don’t use, with its own conventions. Three switch points don’t need a framework. |
One detail shows why the map is central. A shared health report opens from a link, before anyone signs in. The token has to find its workspace before any workspace database is open, so report links live in the central database, stored only as a SHA-256 hash.
An error is a log line with a story
The first version had a logs page and an errors page. To answer one question, what happened at 03:04, people moved between the two. So they became one system. An exception writes one row into the log stream at level error, with its issue in the context. The error line opens its issue. There is no errors list.
- 03:02hostdb-2 memory 86% for 4 min
- 03:04web-1QueryException: deadlock found → issue #214
- 03:04agentworker restarted · exit 137 · out of memory
- 03:10cronbackup nightly ok · 2.1 GB
- 07:30summarymorning summary sent · 1 issue, 1 restart
It needed no new column on the log table, which matters: changing a table split by day is a change to every day. The package also skips the line Laravel logs for an exception it has just sent, so nothing is counted twice. Lines a developer writes by hand stay.
Every event of the same error groups into one issue. A new issue becomes ongoing while it keeps happening, is resolved by hand or by its linked GitHub issue, and if it comes back it is marked regressed rather than opened again, so its history stays in one place.
| State | Means |
|---|---|
| New | First heard in this period. |
| Ongoing | Still happening. |
| Resolved | Closed by hand, or by the linked GitHub issue. |
| Regressed | Was resolved, came back. |
| Ignored | Still counted, never alerts. |
We gave something up for it. The errors index had a thirty-day chart with deploy markers and a period switch. It is gone. Error counts now live on the overview, the app page, the reports and the log chart.
Told at the right hour
There are nine alert rules, fixed in code. Nobody writes a new rule. Thresholds and channels can change, but what counts as a problem is decided once, with care, and tested.
| Rule | Fires when | 23:00–07:00 |
|---|---|---|
| Site down | health check fails twice | Gets through |
| Backup missed | no backup event in 26 hours | Gets through |
| Memory high | over 85% for 5 minutes | Waits |
| Host offline | agent silent 3 minutes, never under 3 intervals | Waits |
| Error spike | more than 10 errors in 10 minutes | Waits |
| New error type | an exception heard for the first time | Waits |
| Failed jobs | failed after the last retry | Waits |
| Bounce rate | over 2% across 24 hours | Waits |
| Certificate expiring | fewer than 14 days left | Waits |
From 23:00 to 07:00 only a site that is down and a backup that was missed get through. Everything else waits for the morning summary. During planned maintenance a site going down opens no incident and sends no alert, and the status page says why.
Host offline took the most care. The agent has to be silent for three minutes, and never less than three of its own report intervals, so a slow interval cannot make a healthy host look dead.
Each channel has one job. WhatsApp is for urgent. Telegram is for look at it today. Email carries the summary. WhatsApp goes through Meta’s Cloud API from one business number with approved templates. It costs per message and every template waits for approval. That is the price of not running a browser for every customer.
| Turned down | Why not |
|---|---|
| A WhatsApp Web sidecar | About 300 MB of Chromium per workspace, and against Meta’s terms. |
A summary that never guesses
Every hour for today, and at 07:00 for yesterday, Carabdheer writes a short summary. It compares the period with the average of the last seven days across errors, failed jobs, p95 response time, memory, bounces and downtime, then ranks the biggest moves. Two or three sentences come from fixed templates. Every claim links the page that proves it.
Errors on web-1 are three times the 7-day average, almost all from one deadlock issue.
Memory on db-2 stayed over 85% for four minutes at 03:02 and a worker restarted once. Backups ran on time.
Deterministic, free, testable, never guesses.
There is no language model in it. When the summary says errors tripled, the link shows the errors. A summary written by a model costs per call, cannot be tested line by line, and can name a cause nobody saw.
Behaviour lives in the database
Retention days, prune time, quiet hours, thresholds, channels and limits are stored in the database and changed on screen. The environment file holds only what the app needs to boot. Changing how Carabdheer behaves never needs a deploy. Settings read on hot paths are cached, and the cache clears on save. Secrets stored this way are encrypted.



