← AI Foundry lab dashboard › Worklog

The lab dashboard froze for 21 hours: why, and what watches it now

Thursday 8 October 2026 · found and fixed from the maintainer's own machine and on aifoundry2 and aifoundry3 · every change is a commit in the repository

From Wednesday 17:50 to Thursday 15:32 (PDT) the dashboard showed the same data. aifoundry2, the only machine that publishes it, was off for 20 hours after the lab's power cut. When it came back, its check that the page is still public could not find the page among the 50 newest spaces that spacesheep lists, so it skipped every publish, and only a publish could have put the page back on that list. Now that check never blocks a publish, a keeper on the publishing machines repairs the collector, aifoundry3 takes over when aifoundry2 goes quiet, and a watchdog outside the lab alerts the maintainer.

1. What happened

When (PDT)What
Wed 7 Oct 17:50The last data. aifoundry2's collector reads the three machines every 10 minutes and publishes on any change, or at least every hour.
about 18:16A power cut across the lab. aifoundry1 and aifoundry3 came back by themselves; aifoundry2 stopped at its BIOS screen.
about 19:07The dashboard drops out of the 50 most recently updated spaces of its owner's account, as other pages keep being published.
Thu 8 Oct 14:28aifoundry2 is back (it restarted once more at 14:50).
14:32 to 15:22Six collector runs read all three machines without trouble, then skip the publish: visibility unverified: list missing.
15:29The unchanged page is published once from outside the lab, which makes it the newest space again.
15:32aifoundry2 publishes fresh data. The dashboard is current again.
17:24 to 17:28The keepers start on aifoundry2 and aifoundry3, and the standby is tested: aifoundry3 publishes once (17:27, all three machines answering), then hands back to aifoundry2 (17:27).

2. Why it could not recover by itself

Every run first makes sure the page is still public, because a publish can change a page's visibility. It did that by finding the page in spacesheep list, which returns only the 50 most recently updated spaces of the account, and it treated a page it could not find as "cannot verify: do not publish".

The machine was fine, and so were cron, the credential and the network: the history page, which aifoundry2 publishes the same way without that check, kept publishing all afternoon. The loop needs a long silence to start, which is why it never showed before: on a normal day the dashboard is published every hour and stays near the top.

3. What watches the dashboard now

update.sh · both machines

The check no longer blocks a publish

A page the list does not show is checked directly, with a signed-out request. Whatever the check cannot verify, the run publishes anyway: the page is meant to be public, so publishing exposes nothing, and the next run, with the page now on the list, checks again. The private mode, for a page that must stay private, is as strict as before.

tools/lab/dashboard/update.sh; tested by tests/guard_test.sh

aifoundry2 · aifoundry3

A keeper beside each collector

A small user service that comes back by itself after a reboot. Every minute it checks its machine's collector (its cron line, the last run and publish, the page's age, a halt) and sends one status line to the live stream aifoundry-dash/<machine>. It puts back a missing cron line and starts the collector when it has not run for 25 minutes or the page is 90 minutes old. It never clears a halt, which is a person's step, and never touches a card.

tools/lab/dashboard/keeper.py

aifoundry3

A standby collector

The same collector runs on aifoundry3 every 10 minutes and stands by while aifoundry2 publishes. When the page's data is 90 minutes old, aifoundry3 publishes instead, and it hands back by itself once aifoundry2's first run after its return has published. "This version", at the bottom of the dashboard, names the machine that collected.

"standby_after_min": 90 in its configuration

outside the lab

A watchdog that reads the page

On the maintainer's own machine, every 15 minutes, it reads the dashboard signed out, the way a visitor does, and the machines' live streams. When the data is more than 90 minutes old it works out why (a machine off, or on but not publishing), tries the repairs that need nobody, and alerts the maintainer, who can approve the rest from his phone.

The history page now publishes every 15 minutes instead of every 5. The owner's account can publish 400 times a day in all, and the history page alone took about 288 of them; on 7 October the limit ran out in the afternoon, which stops the dashboard too. The history page still draws each machine's live readings while it is open.

4. When the dashboard says its data is old

5. The changes

CommitWhat
718b246eThe visibility check no longer blocks a publish in the public mode; the standby setting, and the dashboard's own key folder for a box whose key cannot publish; the keeper and its tests (80 and 61 checks, run on aifoundry3); the Worklog's Thursday row and its way back up to the dashboard; the back-link rule (AGENT.md §8.7); the history page every 15 minutes
the next commitThis page, its source and its row in docs/reports/MIRROR.md

← Back to the Worklog of the AI Foundry lab dashboard