Skip to main content

Module watchdog

Module watchdog 

Source
Expand description

Runs a command that is expected to keep talking, and diagnoses it if it stops.

A coverage run hung for over an hour: two web integration tests stopped producing output while the jobโ€™s heartbeat kept pulsing, so nothing looked wrong until the deadline killed it โ€” with no indication of which tests were responsible. Diagnosing it meant catching the pod live and hand-running pg_stat_activity and /proc inspection before it disappeared. The next occurrence should arrive already diagnosed.

Only for commands whose silence is itself a symptom. The musl link is quiet for twelve minutes and is perfectly healthy; wrapping that would be a way to kill good builds.

Constantsยง

STALL
How long a watched command may produce nothing before it is declared stuck.

Functionsยง

diagnose ๐Ÿ”’
Everything worth knowing about a stuck run, captured before it is killed.
pump ๐Ÿ”’
Forwards a stream line by line, recording when each line arrived.
report ๐Ÿ”’
Best effort: a diagnostic that cannot run must not mask the stall itself.
run_watched
Runs a command, forwarding its output, and fails if it goes quiet.
run_watched_for
As run_watched, with the silence budget supplied.