Long corridor of server light under a stone vault

Carabdheer

Watching every app on every server, in one place. Somali for “long tongue”: the one who tells.

StatusWorking
What it isMonitoring
Read the essay

A server ran out of memory and hung. Nothing told us. We found out when a site was down.

We had tools. Horizon watched the queues, Telescope the requests, Sentry the errors. Each one watched one app. Telescope was too heavy to leave on in production. Reading a container’s logs meant opening a shell on the server. Mail delivery lived in the Resend dashboard, away from the app that sent the mail.

The brief was one place that hears every app on every server, keeps enough history to show a trend, and speaks first. One limit shaped every decision after it: Carabdheer itself stays under 512 MB of memory, not counting the database.

Three ways in, one door

Every event takes the same road, and the database is the last stop. Nothing touches it while the sender waits, so a burst of errors from one app never slows another app’s requests.

  1. Three ways in

    A thin Laravel package sends exceptions, sampled requests, slow queries, jobs, mail and backup events in batches. A host agent reports memory, CPU, disk and containers every ten seconds. Webhooks bring deploys, backups, mail events and GitHub issue state.

  2. One door

    A single ingest endpoint finds the workspace from the key and answers 202 at once. Over its rate limit, it answers 429.

  3. A queue

    The batch goes onto Redis. The sender is already gone.

  4. The worker switches

    A worker picks the batch up, switches to that workspace’s database and writes.

  5. Today’s partition

    Rows land in the day they belong to, ready to be dropped whole later.

The agent is a small Go binary that runs as a container. It mounts the Docker socket and /proc read-only and uses 10 to 15 MB. It exists because of the socket: whoever holds the Docker socket is root on the host, and the web app should never have that.

Turned downWhy not
Laravel reads the socketRoot for the web app, and a PHP loop polling every ten seconds.
A systemd binaryLives outside Coolify. Every upgrade by hand, on every server.
cAdvisor or VectorHeavier, and their output still has to be mapped into ours.

As a container it deploys like everything else on the server. A second server is the same container with a different key.

Drop a day, not a row

Logs, error events, requests and host metrics are most of what Carabdheer stores, and most of it matters only for a while. Each of those tables is split by day in Postgres 17. Pruning never runs a DELETE. At 03:30 every day that has left its keep window is dropped whole. Dropping a partition is instant and leaves nothing behind for vacuum.

logs · 2 Novlogs · 3 Novlogs · 4 Novlogs · 15 Novlogs · 16 Nov
Example: logs kept 14 days. At 03:30 the day that left the window is dropped whole. No DELETE, nothing left for vacuum.

Before raw detail goes, it has already been rolled up by hour and by day. The rollups stay forever, so a year’s trend survives after last month’s log lines are gone.

DataKept
Logs14 days
Host metrics, raw every 10 s7 days
Sampled requests, jobs30 days
Error events, mail events90 days
Error issuesForever
Hourly and daily rollupsForever
Defaults. Each workspace can change its own, within its limit.

Search runs on pg_trgm and full-text indexes. That is enough for level:error app:web-1 across fourteen days, which is the question people actually ask.

Turned downWhy not
ClickHouseNeeds 1 to 2 GB of memory before it does any work. That alone breaks the limit.
LokiA second system to run, back up and upgrade beside the database.
MySQLWeaker partition drop and weaker text search.
SQLiteOne writer at a time. Ingest is many writers.

A database for each workspace

One central database knows who everyone is and where each workspace lives. Everything a workspace hears goes into its own database, with its own partitions and its own nightly prune.

WhereWhat lives there
CentralPeople, workspaces, invites, ingest keys, roles, report links, and the map from each workspace to its database.
crb_ws_<id>Apps, hosts, events, logs, metrics, issues, alert rules, settings, rollups.
Web requestSwitches by the workspace in its session.
Ingest requestSwitches by its key.
Queued jobSwitches by the workspace id it carries.

A heavy workspace never shares a table, an index or a prune run with a light one. Exporting a workspace is one pg_dump. Deleting one drops its database, and nothing is left to hunt down.

A schema per workspace was on the table: the same isolation per table, one connection pool. We chose full databases and paid for it. Each database has its own pool, so PgBouncer sits in front of Postgres in transaction mode, and every migration runs once per workspace.

The switching is our own code. Two connections, central and workspace. One call sets the database name, purges and reconnects, and it happens in the three places in the table above, nowhere else.

Turned downWhy not
stancl/tenancyLarge, built around subdomains we don’t use, with its own conventions. Three switch points don’t need a framework.

One detail shows why the map is central. A shared health report opens from a link, before anyone signs in. The token has to find its workspace before any workspace database is open, so report links live in the central database, stored only as a SHA-256 hash.

An error is a log line with a story

The first version had a logs page and an errors page. To answer one question, what happened at 03:04, people moved between the two. So they became one system. An exception writes one row into the log stream at level error, with its issue in the context. The error line opens its issue. There is no errors list.

  1. 03:02hostdb-2 memory 86% for 4 min
  2. 03:04web-1QueryException: deadlock found → issue #214
  3. 03:04agentworker restarted · exit 137 · out of memory
  4. 03:10cronbackup nightly ok · 2.1 GB
  5. 07:30summarymorning summary sent · 1 issue, 1 restart
Example log lines from one night.

It needed no new column on the log table, which matters: changing a table split by day is a change to every day. The package also skips the line Laravel logs for an exception it has just sent, so nothing is counted twice. Lines a developer writes by hand stay.

Every event of the same error groups into one issue. A new issue becomes ongoing while it keeps happening, is resolved by hand or by its linked GitHub issue, and if it comes back it is marked regressed rather than opened again, so its history stays in one place.

StateMeans
NewFirst heard in this period.
OngoingStill happening.
ResolvedClosed by hand, or by the linked GitHub issue.
RegressedWas resolved, came back.
IgnoredStill counted, never alerts.

We gave something up for it. The errors index had a thirty-day chart with deploy markers and a period switch. It is gone. Error counts now live on the overview, the app page, the reports and the log chart.

Told at the right hour

There are nine alert rules, fixed in code. Nobody writes a new rule. Thresholds and channels can change, but what counts as a problem is decided once, with care, and tested.

RuleFires when23:00–07:00
Site downhealth check fails twiceGets through
Backup missedno backup event in 26 hoursGets through
Memory highover 85% for 5 minutesWaits
Host offlineagent silent 3 minutes, never under 3 intervalsWaits
Error spikemore than 10 errors in 10 minutesWaits
New error typean exception heard for the first timeWaits
Failed jobsfailed after the last retryWaits
Bounce rateover 2% across 24 hoursWaits
Certificate expiringfewer than 14 days leftWaits
The nine rules, fixed in code. Thresholds and channels are editable.

From 23:00 to 07:00 only a site that is down and a backup that was missed get through. Everything else waits for the morning summary. During planned maintenance a site going down opens no incident and sends no alert, and the status page says why.

Host offline took the most care. The agent has to be silent for three minutes, and never less than three of its own report intervals, so a slow interval cannot make a healthy host look dead.

Each channel has one job. WhatsApp is for urgent. Telegram is for look at it today. Email carries the summary. WhatsApp goes through Meta’s Cloud API from one business number with approved templates. It costs per message and every template waits for approval. That is the price of not running a browser for every customer.

Turned downWhy not
A WhatsApp Web sidecarAbout 300 MB of Chromium per workspace, and against Meta’s terms.

A summary that never guesses

Every hour for today, and at 07:00 for yesterday, Carabdheer writes a short summary. It compares the period with the average of the last seven days across errors, failed jobs, p95 response time, memory, bounces and downtime, then ranks the biggest moves. Two or three sentences come from fixed templates. Every claim links the page that proves it.

Errors on web-1 are three times the 7-day average, almost all from one deadlock issue.

Memory on db-2 stayed over 85% for four minutes at 03:02 and a worker restarted once. Backups ran on time.

Example morning summary. Every underlined claim links the page that proves it.

Deterministic, free, testable, never guesses.

There is no language model in it. When the summary says errors tripled, the link shows the errors. A summary written by a model costs per call, cannot be tested line by line, and can name a cause nobody saw.

Behaviour lives in the database

Retention days, prune time, quiet hours, thresholds, channels and limits are stored in the database and changed on screen. The environment file holds only what the app needs to boot. Changing how Carabdheer behaves never needs a deploy. Settings read on hot paths are cached, and the cache clears on save. Secrets stored this way are encrypted.

If you build, dream, or make, you belong here.

Find your way in