← AI Foundry lab dashboard › Worklog
The lab dashboard froze for 21 hours: why, and what watches it now
From Wednesday 17:50 to Thursday 15:32 (PDT) the dashboard showed the same data. aifoundry2, the only machine that publishes it, was off for 20 hours after the lab's power cut. When it came back, its check that the page is still public could not find the page among the 50 newest spaces that spacesheep lists, so it skipped every publish, and only a publish could have put the page back on that list. Now that check never blocks a publish, a keeper on the publishing machines repairs the collector, aifoundry3 takes over when aifoundry2 goes quiet, and a watchdog outside the lab alerts the maintainer.
1. What happened
| When (PDT) | What |
|---|---|
| Wed 7 Oct 17:50 | The last data. aifoundry2's collector reads the three machines every 10 minutes and publishes on any change, or at least every hour. |
| about 18:16 | A power cut across the lab. aifoundry1 and aifoundry3 came back by themselves; aifoundry2 stopped at its BIOS screen. |
| about 19:07 | The dashboard drops out of the 50 most recently updated spaces of its owner's account, as other pages keep being published. |
| Thu 8 Oct 14:28 | aifoundry2 is back (it restarted once more at 14:50). |
| 14:32 to 15:22 | Six collector runs read all three machines without trouble, then skip the publish: visibility unverified: list missing. |
| 15:29 | The unchanged page is published once from outside the lab, which makes it the newest space again. |
| 15:32 | aifoundry2 publishes fresh data. The dashboard is current again. |
| 17:24 to 17:28 | The keepers start on aifoundry2 and aifoundry3, and the standby is tested: aifoundry3 publishes once (17:27, all three machines answering), then hands back to aifoundry2 (17:27). |
2. Why it could not recover by itself
Every run first makes sure the page is still public, because a publish can change a page's visibility. It did that
by finding the page in spacesheep list, which returns only the 50 most recently updated spaces of the
account, and it treated a page it could not find as "cannot verify: do not publish".
- publish, at least hourly
- the page is among the newest spaces
- the check finds it
↻ and the next run publishes again
- about 60 other spaces updated
- the page is off the 50-space list
- the check skips the publish
↻ and skipping keeps it off the list
The machine was fine, and so were cron, the credential and the network: the history page, which aifoundry2 publishes the same way without that check, kept publishing all afternoon. The loop needs a long silence to start, which is why it never showed before: on a normal day the dashboard is published every hour and stays near the top.
3. What watches the dashboard now
The check no longer blocks a publish
A page the list does not show is checked directly, with a signed-out request. Whatever the check cannot verify, the run publishes anyway: the page is meant to be public, so publishing exposes nothing, and the next run, with the page now on the list, checks again. The private mode, for a page that must stay private, is as strict as before.
tools/lab/dashboard/update.sh; tested by tests/guard_test.sh
A keeper beside each collector
A small user service that comes back by itself after a reboot. Every minute it checks its machine's collector
(its cron line, the last run and publish, the page's age, a halt) and sends one status line to the live stream
aifoundry-dash/<machine>. It puts back a missing cron line and starts the collector when it has not
run for 25 minutes or the page is 90 minutes old. It never clears a halt, which is a person's step, and never
touches a card.
tools/lab/dashboard/keeper.py
A standby collector
The same collector runs on aifoundry3 every 10 minutes and stands by while aifoundry2 publishes. When the page's data is 90 minutes old, aifoundry3 publishes instead, and it hands back by itself once aifoundry2's first run after its return has published. "This version", at the bottom of the dashboard, names the machine that collected.
"standby_after_min": 90 in its configuration
A watchdog that reads the page
On the maintainer's own machine, every 15 minutes, it reads the dashboard signed out, the way a visitor does, and the machines' live streams. When the data is more than 90 minutes old it works out why (a machine off, or on but not publishing), tries the repairs that need nobody, and alerts the maintainer, who can approve the rest from his phone.
The history page now publishes every 15 minutes instead of every 5. The owner's account can publish 400 times a day in all, and the history page alone took about 288 of them; on 7 October the limit ran out in the afternoon, which stops the dashboard too. The history page still draws each machine's live readings while it is open.
4. When the dashboard says its data is old
- A red banner at the top says how old the data is. The keeper on the collecting machine restarts its collector within minutes, aifoundry3 takes over after 90 minutes, and the maintainer is alerted.
- If aifoundry2 and aifoundry3 are both off, nothing can publish: the page keeps its last data, greyed, and the
banner says so. Tell the maintainer in
#community-lab. - For the maintainer, on the collecting machine:
tools/lab/dashboard/update.sh status(the last runs, the last publish, a halt) andtools/lab/dashboard/keeper.py status(what the keeper sees). The manual has the rest.
5. The changes
| Commit | What |
|---|---|
| 718b246e | The visibility check no longer blocks a publish in the public mode; the standby setting, and the dashboard's own key folder for a box whose key cannot publish; the keeper and its tests (80 and 61 checks, run on aifoundry3); the Worklog's Thursday row and its way back up to the dashboard; the back-link rule (AGENT.md §8.7); the history page every 15 minutes |
| the next commit | This page, its source and its row in docs/reports/MIRROR.md |
← Back to the Worklog of the AI Foundry lab dashboard