I have one instinct about any process with a metric: once you put eyes on it, it improves. Over the last two weeks I got to test it with the team, on a production system. It worked, but only after we got three things right: one metric, a report that puts it in front of everyone every few hours, and alerts that make people look at what moves it.

One number

A dashboard shows many numbers, and the one that matters gets lost among them. So we picked one that people were already talking about: finished output a day. Everything else came second. We did not chase retries or other side numbers. We picked work by whether it moved that number.

The naive way

The naive way to put eyes on a metric is a dashboard, plus alerts in a side channel. That is what we had at first. The dashboard was broken. The alerts came into a separate channel, and they were noise. They watched a queue that the work no longer went through, so they called busy workers idle and missed real stops. Most of the alerts had a separate cause: a worker that did not answer for a moment and then came back. Nobody replied to them.

The number, every three hours

The most important piece is not an alert. It is a report that posts the number itself to the main team channel every three hours: output in the last three hours against a target, how busy the workers were, and the idle time split by cause. Nobody has to open anything to see it. If output falls behind, the next lines say why.

A sketch of the three-hour report, with no real numbers: the one metric against its target first, then why it is where it is.

The first design on the table had close to fifty items. I cut it down to four checks and this one report.

The report keeps eyes on the number. The alerts are for what happens in between: the queue running low, a limit being hit, a dependency that stops working, a worker that should be taking work and is not.

My take is that an alert has to go out to people. A dashboard alone does not do that.

Left: the metric as one dashboard tile among many, with no person attached. Right: a sketch of an alert on a cause that moves that metric, with a state, an owner and a reason.

How one alert works

Here is the loop we built in its place.

1. Crons collect what happened. Every few minutes, a job reads each worker: what it is running, what it finished, and how much work is left in the queue. Every 3 hours it also writes the report.
2. Rules decide what is wrong. Each problem is checked against the rules in a fixed order and lands in one of them, so one event posts one alert.
3. Rules first, then a decision layer. Rules are cheap, so they rule out the noise first: most events never need a model. What gets through, where the cause is not obvious, goes to a decision layer, a decision model like Jev or an LLM, which earns its cost there. Then a template writes the message. It is a pattern I use in other places too. Every alert names who acts. Where the rules can tell, it also gives the likely cause and the fix.
4. The alert has a state. It is ACT NOW when it opens, STILL while it stays open, and RESOLVED when it clears. My first version of the new alerts was noise too: dozens of posts in two days. The fix had three parts. I turned off the STILL reminders. An alert that came back within a few hours kept its number and stayed quiet. And the check for idle workers stopped counting slots that could not take work, which had been making busy workers look free. Open alerts are listed in the 3-hour report instead.
5. The alert lands where people look. Every alert goes to an alerts channel. The urgent types are also posted in the main team channel.

Crons read the workers, rules pick one alert per event, a template adds the owner and, where it can, the cause and fix, and the alert type decides whether it also goes to the main channel.

What changed in the message

The difference shows best side by side. The old alert gave a worker name and a raw error, and nothing about why. Every new alert names who acts and, where the rules can tell, why. One type goes further and gives a reason for each worker: at a capacity limit, not picked for this job, busy, or a free slot that did not take work. Then it tags the owner.

Some alerts look ahead. One fires when the queue is close to empty while workers still have free slots, so the person who fills the queue knows before everything goes idle. When a capacity limit is hit, the alert gives the time it resets. The 3-hour report lists what is close to its limit.

Top: the old alert, a worker name and an error. Bottom: the new alert, with a reason per worker, a state and an owner.

Where it lands

I put the urgent alerts in the main team channel because I wanted people to see them. In the alerts channel, nobody had replied to the old alerts in weeks.

Seeing is not enough when many people share one system. Each kind of problem needs its own owner. A queue running low goes to the person who fills the queue. A worker that stops taking work goes to the engineer who runs the workers. An alert that tags everyone moves no one. An alert with one name on it gets an answer.

One case shows the loop working. One night a new alert fired for a dependency that kept failing. It named who owned it, and the owner found the cause that night. When a failure slipped through without an alert, we added an alert for it.

What it added up to

In under two weeks, output went up about 7.5x. The factors multiply: about 3.5x more capacity, each unit of it about 1.5x faster from engineering, and about 1.4x from longer outputs and a second kind of work. I do not claim the alerts created any of it. They showed where output was being lost, so the capacity and the engineering went to the right places.

The alerts also have a weak spot. A capacity-limit alert can stay open until the limit resets. It tells people, but most of the time nobody can act on it.

The rule

If you run a process, for example usage or engagement, pick one clear metric and put the team’s effort where it moves that number. Then put eyes on it: send an alert, not just a dashboard. Demand that people look at the alerts, write the metric down, understand why it moved, and know why each alert exists.

This system is one case. Next I am trying the same process on the other things I own.