Skip to content
Lightning VMBeta

Articles

How the smart helpers work without an outside AI service

What data the usage watch, the month-end forecast, the schedule suggestion, the quietest hours, payment risk, support help and the periodic digest look at, where their thresholds lie, what they never do, and where they say "not enough data".

About 9 min read · Last updated

None of the features the panel calls "smart" asks an AI service anything. They are rules you can explain: counting, taking a median, comparing with a threshold. They run inside the panel on the data in the panel's own database. This article goes through what each one looks at, at which point it speaks, and where it stays silent. For where the screens are, see the Smart helpers page.

Where the computing happens

  • The usage watch runs in the worker, once an hour.
  • The budget forecast runs in the API every 6 hours, only for accounts that have a budget and have not been warned yet that month; it is also checked the moment a budget is saved.
  • The schedule suggestion, the quietest hours, payment risk and support help have no background job; they are computed when the page is opened or while you type.
  • The periodic digest is prepared in the API every five minutes for the people whose digest is due; each period is worked out exactly once.

No outside AI and no other outside service is used, and no data leaves the installation. The text you type for support help goes to no outside service and is not stored.

Which one looks at what

HelperData it looks atMinimum amount of data
Usage watch, CPUThe server's hourly CPU averages, the 14 days before12 hours from 4 different earlier days
Usage watch, networkThe server's daily traffic totals, the 14 days before7 days
Month-end forecastThe prices billing keeps and the hourly usage it has recorded this monthNone; a budget, for the comparison with a budget
Schedule suggestionThe hourly CPU averages of the last 4 weeks21 days
Quietest hoursThe servers' hourly CPU averages and vCPU counts, the last 4 weeks21 days
Periodic digestWhat the panel already works out: insights, payment risk, the month-end forecastNone; something new to say
Payment riskThe next renewal, the expiry date of the card that will be charged, the balance; for staff, invoices and charge attemptsNone
Support helpThe ticket's subject and the customer's first messageAt least 2 shared words

Hourly CPU rows are kept for 63 days and raw samples for 14. Closed usage findings are deleted after 90 days.

The thresholds in plain words

Usage watch

For every server the panel works out two things: the usual level (the median of use in the same hours) and the usual variation (how far samples typically sit from that median). A median was chosen because one odd day does not move it.

For CPU, a finding opens when the last 3 hours are at least 20 points above the usual level and the difference is about 4 times the usual variation. The variation is never taken as less than 5 points, so a small wobble on a flat-running server is not a finding. A finding closes when the difference falls below 10 points or below twice the variation. Having separate thresholds to open and to close keeps a server that hovers at the edge from opening and closing the finding over and over.

A weekly rhythm is not a finding either: a departure must also hold against the same hours of the same weekday one and two weeks earlier. That is why a server that works on weekdays and idles at the weekend does not raise a warning every weekend. This check only removes findings; it does not notice a weekly job that failed to run.

The comparison is with the same hours of the day, which is why the nightly load of a server that is busy at the same time every night counts as usual. Hours are in UTC and one hour either side is included; a daylight-saving change fits inside that band.

For network, a finding opens when the day's traffic is at least 5 GiB, at least 3 times a usual day, and clearly outside the usual variation as well. If use stays at its new level for days, the panel takes it as the new normal and the finding closes by itself in about a week.

Month-end forecast and budget

The forecast is not a prediction model; it is an addition. The price of monthly servers is taken in proportion to the time they existed this month; for hourly servers, the price of the hours left is added to the usage billing has recorded. The budget comparison has two thresholds: at 80% of the budget the forecast is "close to" it, above 100% it is "heading over". The notification goes out once a month.

Power schedule suggestion

For an hour to count as "idle" it must have been seen in at least 3 weeks, and in each of them its average must be below 3% and its highest sample below 10%. An hour with no measurement counts as in use; the panel does not take an hour it knows nothing about as idle. The suggested schedule turns the server on an hour before the hours it is used and off an hour after them. No suggestion is made for a server that would be off for less than 20 hours a week.

Payment risk

The customer is told about two situations: the card billing will charge (the default card) expires before the renewal and the balance is not enough, or no card lasts long enough and the balance is smaller than the renewal. Billing charges the default card only, so a spare card that lasts long enough does not remove the warning. This is looked at only where automatic collection is on and the renewal is at most 30 days ahead. No conclusion is drawn about a card whose expiry date is not known. The staff list holds accounts with an overdue invoice or with at least 2 failed charges in 30 days.

Quietest hours for maintenance

Load is measured in "busy vCPUs": each server's hourly CPU average is multiplied by its vCPUs and the results are added up. The hours of the last 4 weeks are laid out over the 168 hours of the week, and an hour is compared only if it was seen in at least 3 weeks. The panel gives the quietest window of 2 consecutive hours in two forms: the one that starts at the same hour every day, and the quietest of the whole week. Beside it stand the window's load, the week's average and the load of the busiest hour; with less than 21 days of history it says "not enough data".

Periodic digest

The digest computes nothing new; it gathers the results of the above into one message. It goes out weekly or daily, at the hour the person chose in their own time zone. With nothing new to say (an open finding, something opened or closed since the last digest, an account at risk, a failed operation, a capacity finding; for a customer an amount or a renewal) no message goes out for that period.

Support help

The text is split into words, common words are dropped, and what is left is compared for shared words. A word found in many tickets counts for little, a rare one for more. A result needs at least 2 shared words; at most 3 results are shown, each with the words that caused it to be shown. There is no understanding of meaning: two tickets that describe the same problem in different words may not match.

Why am I seeing this

Every finding and suggestion comes with the numbers behind it: the usual level and the present one, the budget and the forecast, the idle hours, the card's expiry date and the renewal amount, the shared words. Each can be hidden. Hiding applies to that one finding: a new finding, a new month or a new renewal period shows up again.

What they will not do

  • They do not touch a server without a click. A schedule is created, the way any schedule is, when someone with permission to power the server on and off presses the button. The usage watch only tells you.
  • They do not show one customer another customer's ticket. For a customer, only their own answered tickets and the public status page are read; an integration test checks this. Results shown to staff never include internal notes either.
  • They charge nothing, suspend nothing and write to no one. Payment risk only lists; reminders and suspension stay in billing's own flow.
  • They do not create an abuse finding. "Resembles mining" is a note; abuse monitoring's own checks and approval flow are separate.
  • They promise no saving in money. A server that is off is billed the same; the schedule suggestion gives only the energy that would be saved, or the hours off.
  • They do not schedule maintenance by themselves. The quietest hours are only shown. The update window changes when someone with the permission presses the button; the button that empties a host runs at once as before, and the panel does not postpone the migration to that hour.
  • They send no digest unless you turn it on. The periodic digest is off unless turned on; customers can turn theirs on only after an administrator allows it, and a digest holds no other customer's data.
  • They send no reply by themselves. Copying the answer of a similar ticket, and what to write to the customer, is the staff member's decision.

Limits

  • Platforms without CPU measurement. XCP-ng, oVirt, KVM (libvirt) and Incus do not report per-server CPU. On those servers the usage watch reaches no conclusion about CPU, no schedule is suggested, and they are left out of the quietest hours; the server page says "not enough data".
  • A server with no traffic records. Its network is not judged. Traffic is counted on VMware and Proxmox only.
  • A new server. Nothing is said until history has built up; the schedule suggestion says how many days there are.
  • Comparison with its own history. A server that has run hot since its first day does not look unusual; that case belongs to right-sizing and to abuse monitoring.
  • What was tested. Each of the rules has unit tests, and the separation between customers in support help is checked by an integration test. The helpers were also run end to end on a seeded history (200 servers of 20 customers, five patterns of use, 14 days of samples): with ordinary use the usage watch opened no finding, and it found the three real departures that were added. They were not tried with real customer workloads, so how often they would raise a false alarm there is not known. Delivery of the periodic digest by e-mail and chat apps was not tried with a real mail server or app.

Source: docs/smart-features.md (in Turkish).