Do you need an LLM monitoring tool, or just 20 lines of try/except?

The checks themselves are not the hard part

"Was the response valid JSON? Did it refuse? Did it contain the words it should? Was it too slow?" — that's genuinely about twenty lines, and ChirpPal's are open source precisely because there's nothing secret in them. If you can write those checks and wire an alert to Slack, do it. You don't need us for the checking.

Where the 20 lines quietly stop being enough

The parts people underestimate aren't the checks — they're everything around them:

• Alert noise. The naive version (alert on every failure) spams you until you mute it; then a real outage arrives muted. Doing it right means a rolling window and a cooldown so interleaved pass/fail streams don't flood you or hide a failure. That's the part you'll rewrite three times.
• Aggregation across deploys and services. One failing check in one process is noise; the same check failing 40% of the time across every instance is a signal. Seeing that needs somewhere to collect and count events over time — which is the thing you were trying not to build.
• Model-deprecation tracking. Your own code can't tell you that a model you call is scheduled to shut down next month. That's an external dataset someone has to maintain and cross-check against three providers' pages.
• Not becoming a data-liability. The easy logging approach dumps prompts and responses somewhere. Keeping checks in-process and sending only pass/fail metadata is more work to get right — and it's the difference between "a log line" and "a pile of user content you now have to secure."

So: which are you?

Write your own if you have one call site, you're already watching it, and a mute-able Slack ping is fine. Genuinely — that's a reasonable place to be, and you don't need to pay anyone.

Reach for a tool when the noise, the cross-deploy counting, the deprecation tracking, or the "I don't want to store user content" constraint start eating your afternoons. That's the line ChirpPal is built to sit on: the checks are the easy part you could write; the alert de-noising, the aggregation, and the deprecation radar are the parts you'd rather not.

More: ChirpPal vs Langfuse · monitoring without sending your prompts · all LLM monitoring tools compared.