Observability and Alerting for Trading Bots: What to Log, What to Page On, What to Ignore
The observability component is where most bots cheat: a print statement and a prayer. Then the one night something breaks, the logs say “ERROR” forty times with no context, and you debug by candlelight. Observability is a designed system: structured logs for forensics, metrics for trends, alerts for emergencies — each with rules about what earns your attention.
What should a trading bot log?
Structured, per event, one line each (JSON lines — greppable, parsable): orders (sent/accepted/rejected — with the quoted price at send and the rejection reason verbatim), fills (price, size, fees, signal ID), rejections and errors (full context: endpoint, request ID, clock offset — see clock skew signatures), risk events (limit touches, kill-switch triggers — see kill switch design), and lifecycle (connects, disconnects, resyncs — see the websocket pattern). Two fields most bots skip and every forensic session wants: state snapshot on error (positions and orders at the moment of failure) and correlation IDs (link a signal → orders → fills → exits across lines — the attribution schema depends on this).
Metrics: the five you trend weekly
- Fill quality — your fills versus quoted-at-send: slippage per trade, trended. Rising slippage is infrastructure decay speaking.
- Reject rate — rejections per 100 orders by reason code: clustering signatures (rate limits, margin, auth) each mean different fixes (see rate-limit patterns).
- Latency — signal-to-order, order-to-ack: slow execution quietly converts edge to noise.
- PnL per attribution cut — the weekly four-cut review (attribution guide).
- Risk distance — how close your positions run to limits and liquidation buffers (perps: see liquidation mechanics): a bot that’s always at 95% of limits is one wiggle from the news.
Alerting: what pages you, what waits for morning
Pages immediately (money at risk, clock running): kill-switch fired, risk limit breached, positions that failed to flatten (the drill’s real test — see kill switch design), delegation found on a wallet (the monitoring stack covers it), repeated auth failures (a bot that can’t authenticate with open positions is an unmanaged position).
Waits for morning (trends, not incidents): elevated reject rates, slippage drift, stale-data episodes the stale-gate caught cleanly, attribution shifts. If you page yourself for trends, you’ll silence the channel — and then miss the pages that matter.
Ignored on purpose (log-only): individual rate-limit 429s handled by backoff, single reconnects handled by resync, informational lifecycle events. The three-file log discipline applies: logs are for reading on schedule; alerts are for reading now.
The weekly observability review (20 minutes)
Read the logs’ error sections first (they’re boring when things are fine — that’s the point), trend the five metrics, reconcile one random day’s fills against the exchange records (external verification — a successful bot is not a verified bot until the records agree), and check that alerts would have fired for anything weird this week. The deployment checklist runs this review as a gate for scaling size.
FAQ
What logging does a trading bot need? Structured one-line JSON events with correlation IDs: orders, fills, rejections with reasons, risk events, lifecycle — plus state snapshots on error. If you can’t reconstruct a trade’s full lifecycle from logs, the logging isn’t done.
When should a trading bot alert me? Money-at-risk incidents only: kill-switch fired, limits breached, flatten failures, delegation found, auth dead with open positions. Trends (slippage drift, reject rates) wait for the weekly review — over-paging trains you to ignore the channel.
How do I measure my bot’s slippage? Compare each fill to the quoted price at order-send, logged per trade and trended weekly — the gap is your execution cost. Rising slippage means infrastructure decay before it means bad luck.
What metrics predict trading bot failure? Rising reject rates, growing latency, shrinking risk-distance headroom, and attribution decay (edge thinning per the weekly cuts) — each is a leading indicator with its own fix-path in the guides above.
How often should I review bot logs? Weekly, structured, 20 minutes: error sections, metric trends, one day’s reconciliation against exchange records. Daily reading is for alerts; the weekly pass is where trends get caught — see the review structure above.
Educational content, not financial advice. Not affiliated with Polymarket. Last updated October 2026.