mirror of
https://github.com/dsec-hub/dsec-discord-bot.git
synced 2026-09-22 07:44:26 +00:00
OPS-04: make the deploy assertion catch a crash-looping bot
Codex review: with `restart: unless-stopped` a bot that panics at boot is restarted, so a single `docker inspect` still reads Running while it crash-loops — the deploy went green while broken. The assertion now waits for the bot to reach READY (it logs "Logged in as ..." from the Ready handler) within a stability window AND requires RestartCount == 0, breaking early and failing the moment a restart is observed, dumping container logs on any failure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017XrE7F9ZuBWdQnS8CZvYDE
This commit is contained in:
parent
e8b611ede4
commit
4bcfb8668f
1 changed files with 36 additions and 10 deletions
46
.github/workflows/deploy.yml
vendored
46
.github/workflows/deploy.yml
vendored
|
|
@ -45,21 +45,47 @@ jobs:
|
|||
echo "Container created."
|
||||
|
||||
# `docker compose up -d` exits 0 once the container is created, not once the
|
||||
# program inside is running. The bot can panic at boot (a bad .env value, a
|
||||
# Supabase hiccup) and exit within a second while the deploy still goes green.
|
||||
# Assert the container is actually alive, and surface its logs if not (OPS-04).
|
||||
- name: Verify the bot is still running
|
||||
# program inside is working. And with `restart: unless-stopped` a container
|
||||
# that panics at boot is restarted, so a crash loop still reads Running at any
|
||||
# single instant — "Running" alone is not proof of a healthy deploy. Assert
|
||||
# the bot actually reached READY (it logs "Logged in as ..." from the Ready
|
||||
# handler) within a stability window AND has not restarted, and dump its logs
|
||||
# and fail otherwise (OPS-04).
|
||||
- name: Verify the bot came up cleanly
|
||||
working-directory: ${{ github.workspace }}
|
||||
run: |
|
||||
sleep 30
|
||||
id=$(sudo docker compose ps -q dsec_bot)
|
||||
running=$(sudo docker inspect -f '{{.State.Running}}' "$id")
|
||||
if [ "$running" != "true" ]; then
|
||||
echo "::error::dsec_bot is not running 30s after deploy"
|
||||
sudo docker compose logs --tail=100 dsec_bot
|
||||
if [ -z "$id" ]; then
|
||||
echo "::error::dsec_bot container was not created"
|
||||
sudo docker compose logs --tail=200 dsec_bot || true
|
||||
exit 1
|
||||
fi
|
||||
echo "Deployment complete."
|
||||
|
||||
ready=""
|
||||
for _ in $(seq 1 30); do
|
||||
sleep 2
|
||||
running=$(sudo docker inspect -f '{{.State.Running}}' "$id" 2>/dev/null || echo "false")
|
||||
restarts=$(sudo docker inspect -f '{{.RestartCount}}' "$id" 2>/dev/null || echo "0")
|
||||
# A non-zero restart count means it has already crashed at least once:
|
||||
# stop waiting and fail rather than let a later restart look healthy.
|
||||
if [ "$restarts" != "0" ]; then
|
||||
break
|
||||
fi
|
||||
if [ "$running" = "true" ] \
|
||||
&& sudo docker compose logs dsec_bot 2>&1 | grep -q "Logged in as"; then
|
||||
ready="yes"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
running=$(sudo docker inspect -f '{{.State.Running}}' "$id" 2>/dev/null || echo "false")
|
||||
restarts=$(sudo docker inspect -f '{{.RestartCount}}' "$id" 2>/dev/null || echo "0")
|
||||
if [ "$ready" != "yes" ] || [ "$running" != "true" ] || [ "$restarts" != "0" ]; then
|
||||
echo "::error::dsec_bot did not come up cleanly (running=$running restarts=$restarts ready=${ready:-no})"
|
||||
sudo docker compose logs --tail=200 dsec_bot
|
||||
exit 1
|
||||
fi
|
||||
echo "Deployment complete: dsec_bot reached READY with no restarts."
|
||||
|
||||
- name: Remove environment file
|
||||
if: always()
|
||||
|
|
|
|||
Loading…
Reference in a new issue