AUTOMATA CRM Update: Site Monitoring and an On-Call Agent That Triages Incidents

A site goes down, and you learn about it from a popup on your desktop instead of a client phone call. Ninety seconds later the CRM holds a ticket with the error, and fifteen minutes later there is an agent report: what it checked, what it found, what it needs from you. That is what on-call duty looks like inside AUTOMATA CRM now.

We had been carrying manual work in monitoring for too long. Before this update it was a set of scripts that pinged sites every five minutes and wrote statuses into a file, occasionally reaching Telegram. Three problems came with it: the scripts themselves failed now and then, data diverged between sources, and alerts depended on a channel that might be unavailable exactly when an outage happens. The update flips the logic: instead of scripts checking sites, a process carries an incident from detection to closure.

What changed

The update touches three layers, and each one works on its own.

Monitoring. All 17 sites, our own projects and client ones, moved to Uptime Kuma running on our server. Checks run every 60 seconds with two retries for network errors, and certificate and domain expiry are tracked separately. Check history lives with us, not in someone else’s cloud: you can see what happened at 3 a.m. two days ago, not just that things are fine right now.

Dashboard. The Today tab in the CRM now has a Monitoring block: status, uptime over the last hundred checks, response time, days until SSL and domain expiry, for every site including client ones. No more juggling a separate interface: you look where you already work.

Incidents. Every failure becomes a ticket with a client, a priority and an SLA. But the ticket is only half of it. The other half is automated diagnostics and an on-call agent.

The path of one outage

The chain is simple enough to describe in one sentence: a monitor noticed, a daemon recorded, the CRM opened a ticket, the agent triaged it.

1. Detection. Uptime Kuma checks the site every 60 seconds. Network hiccups do not count as downtime: two retries filter random timeouts and junk responses.
2. Transition. A daemon on the workstation keeps a persistent connection to the monitor and, at the moment the state changes from up to down, writes an event into a local feed. No polling, only transitions, so there are no duplicate alerts.
3. Ticket. The CRM handles the event: creates a critical ticket for the domain, attaches the error text (timeout, ENOTFOUND, 502), sets a one hour SLA, sends the event to the feed and shows a desktop notification. If an incident for that domain is already open, a second one is not created.
4. Diagnostics. A few seconds after the ticket, a table of facts is appended to it: DNS from the laptop and from the server, HTTP from both points, certificate days left, free disk space, web server state, the last error lines for that domain. Five to nine checks produce a hypothesis: DNS, server or app, unreachable host, expiring certificate, disk, local network.
5. Agent. Every fifteen minutes during working hours the on-call agent looks for new incidents. If there is one, it confirms the cause from logs, appends its analysis to the ticket, writes a journal entry and raises a popup if a human decision is needed. If there is none, it exits with a single line.
6. Closure. The site recovers, the CRM closes the ticket and removes it from the triage queue. No manual steps.

From failure to ticket is about ninety seconds. From ticket to a human readable cause takes a couple more minutes. This is not monitoring for a checkbox: by the time you are in the elevator, you already have a hypothesis and the facts behind it.

What automated diagnostics actually checks

The point is that when a human opens the ticket, they do not start from zero. The checks cover the failure modes we have seen on our own and client sites:

Check	What it reveals
DNS locally and from the server	expired domain, changed nameservers, parking, broken zone
HTTP from laptop and from server	site down for everyone, or only for one network path
SSL and days remaining	failed renewal, broken chain
Free disk space	logs or backups filled the disk, a classic night outage
Web server and failed units	crashed service, stuck certbot, broken tunnel
Error log excerpt for the domain	connect() failed, 502 from upstream, PHP timeouts

Then one rule does a lot of work: if HTTP works from the server but not from the laptop, the problem is the local network, not the site. In that case the agent is not started at all, because there is no reason to wake anyone over a home connection. The opposite case, down from both points, immediately points at the server or the app and narrows the search to logs.

An agent with boundaries

The main question about any AI on-call is what it will do without asking. Our answer is strict. The agent may read logs, diagnose, repeat checks and take only safe actions. Restarting services, editing web server configuration, spending money and emailing clients are forbidden without the owner’s approval. If a restart is required, the agent does not do it: it writes “owner action required” into the ticket and raises a critical notification.

This is not caution for its own sake. An automatic restart before the cause is found often hides the problem: the service comes back, the incident closes, and a day later it fails again with the evidence gone. The agent is responsible for the speed of diagnosis, not for pretending everything is fine.

Why a desktop popup, not Telegram

We deliberately do not build the critical path on Telegram. In Russia it depends on blocks and workarounds: at the exact moment you need a VPN, an outage alert may not arrive. Critical events go through two channels that do not depend on outside infrastructure: a local desktop notification and the CRM event feed, where an incident stays until acknowledged. On top of that, an always visible widget shows status, uptime and ping for every site, refreshed every fifteen seconds.

This does not mean no messengers. It means a messenger is a convenient extra channel, not the only one. An outage alert should not have a single point of failure.

What it gives the business

First, time. An incident that used to surface an hour later from a client call now surfaces in a minute with facts attached. The first thirty minutes are the expensive ones: that is when you either find the cause or start panicking. Diagnostics give those thirty minutes back.

Second, predictability. Certificate and domain expiries are no longer somewhere in a calendar. They are in one block, and they do not slip: when a client site goes down because of an expired certificate, the contractor is to blame, and that is us.

Third, scale without routine. Eighteen sites do not get checked by eye. Adding a nineteenth is one line in monitoring, not a new process. That is why client sites sit in the same loop: monitoring our clients is an obligation, not a nice to have.

Fourth, an honest SLA. A ticket has a deadline, an incident has a cause, a closure has a timestamp. When someone asks next month what happened on Wednesday night, the answer takes ten seconds instead of a message thread.

What it does not replace

Monitoring checks availability, not meaning. A site can return 200 while showing zero prices, an empty catalog or failing to accept an order. That is business logic, and it is handled by a different tool: push checks. A stock sync script reports success to monitoring; if the signal is missing for a day, that is an incident. Backups, certificate renewals and content publishing are tracked the same way. The next step for this update is exactly that class of checks.

How to reproduce it

The scheme needs no boxed product and no subscription:

1. Run Uptime Kuma on your own server (Docker, one container) and add every site with a 60 second interval.
2. Turn on domain and certificate expiry notifications, free checks that most often save your reputation.
3. Write a small daemon that keeps a websocket to monitoring and records only up to down transitions; on transitions, open tasks in your own system.
4. Add the diagnostics table above. Even without any AI it removes half of the routine.
5. If you add an agent, define the prohibitions first: what it does not do without a human. Speed comes from the allowed actions, trust from the forbidden ones.

The agent does not replace an on-call engineer. It removes the part of duty where a human acts as a script: notice, open, check three obvious things, write it down. The rest, decisions, priorities, conversations with clients, stays with people. It just starts from a prepared ticket that already says what happened.

If you want the same monitoring and incident loop on your own sites, start with an audit of what you have: which sites, where they are hosted, who hears about outages today and how many minutes later.