OPS-04: make the deploy assertion catch a crash-looping bot

Codex review: with `restart: unless-stopped` a bot that panics at boot is
restarted, so a single `docker inspect` still reads Running while it crash-loops
— the deploy went green while broken. The assertion now waits for the bot to
reach READY (it logs "Logged in as ..." from the Ready handler) within a
stability window AND requires RestartCount == 0, breaking early and failing the
moment a restart is observed, dumping container logs on any failure.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XrE7F9ZuBWdQnS8CZvYDE
This commit is contained in:
Clupai8o0 2026-08-30 16:00:58 +10:00
parent e8b611ede4
commit 4bcfb8668f

View file

@ -45,21 +45,47 @@ jobs:
echo "Container created." echo "Container created."
# `docker compose up -d` exits 0 once the container is created, not once the # `docker compose up -d` exits 0 once the container is created, not once the
# program inside is running. The bot can panic at boot (a bad .env value, a # program inside is working. And with `restart: unless-stopped` a container
# Supabase hiccup) and exit within a second while the deploy still goes green. # that panics at boot is restarted, so a crash loop still reads Running at any
# Assert the container is actually alive, and surface its logs if not (OPS-04). # single instant — "Running" alone is not proof of a healthy deploy. Assert
- name: Verify the bot is still running # the bot actually reached READY (it logs "Logged in as ..." from the Ready
# handler) within a stability window AND has not restarted, and dump its logs
# and fail otherwise (OPS-04).
- name: Verify the bot came up cleanly
working-directory: ${{ github.workspace }} working-directory: ${{ github.workspace }}
run: | run: |
sleep 30
id=$(sudo docker compose ps -q dsec_bot) id=$(sudo docker compose ps -q dsec_bot)
running=$(sudo docker inspect -f '{{.State.Running}}' "$id") if [ -z "$id" ]; then
if [ "$running" != "true" ]; then echo "::error::dsec_bot container was not created"
echo "::error::dsec_bot is not running 30s after deploy" sudo docker compose logs --tail=200 dsec_bot || true
sudo docker compose logs --tail=100 dsec_bot
exit 1 exit 1
fi fi
echo "Deployment complete."
ready=""
for _ in $(seq 1 30); do
sleep 2
running=$(sudo docker inspect -f '{{.State.Running}}' "$id" 2>/dev/null || echo "false")
restarts=$(sudo docker inspect -f '{{.RestartCount}}' "$id" 2>/dev/null || echo "0")
# A non-zero restart count means it has already crashed at least once:
# stop waiting and fail rather than let a later restart look healthy.
if [ "$restarts" != "0" ]; then
break
fi
if [ "$running" = "true" ] \
&& sudo docker compose logs dsec_bot 2>&1 | grep -q "Logged in as"; then
ready="yes"
break
fi
done
running=$(sudo docker inspect -f '{{.State.Running}}' "$id" 2>/dev/null || echo "false")
restarts=$(sudo docker inspect -f '{{.RestartCount}}' "$id" 2>/dev/null || echo "0")
if [ "$ready" != "yes" ] || [ "$running" != "true" ] || [ "$restarts" != "0" ]; then
echo "::error::dsec_bot did not come up cleanly (running=$running restarts=$restarts ready=${ready:-no})"
sudo docker compose logs --tail=200 dsec_bot
exit 1
fi
echo "Deployment complete: dsec_bot reached READY with no restarts."
- name: Remove environment file - name: Remove environment file
if: always() if: always()