AI Integration Checklist for Founders

Before you ship AI to paying customers, you need answers to these questions. If your engineer or vendor can't answer most of them, the integration isn't production-ready.

Quality

• [ ] Is there an eval harness? With how many examples?
• [ ] What's the measured accuracy on your golden set?
• [ ] What's the regression detection on prompt or model changes?
• [ ] How do users report quality issues? Where does feedback go?
• [ ] What's the human-review escalation path for low-confidence outputs?

Cost

• [ ] What's the cost per request? Per user per month?
• [ ] Is there a per-user rate limit? Budget? Both?
• [ ] Is prompt caching enabled where applicable?
• [ ] Is tier routing implemented (cheap model for easy tasks)?
• [ ] What's the cost dashboard? Who sees it?

Latency

• [ ] What's P50, P95, P99 latency?
• [ ] Is the response streamed?
• [ ] Is there a "thinking..." UX while waiting?
• [ ] Are timeouts configured? With what fallback?

Reliability

• [ ] What happens when the AI provider has an outage?
• [ ] Is there a fallback model from a different provider?
• [ ] Are retries idempotent? With exponential backoff?
• [ ] Is there a circuit breaker for repeated failures?
• [ ] What's the SLA you can promise users?

Security & data

• [ ] What user data goes to the AI provider?
• [ ] Is PII redacted before sending?
• [ ] Is there a data processing agreement (DPA) with the provider?
• [ ] Is the AI provider in a compliant region (GDPR, etc.)?
• [ ] Are AI responses sanitized before showing to users?
• [ ] Is there prompt injection protection?

Multi-tenancy (if B2B SaaS)

• [ ] Is each tenant's RAG corpus isolated?
• [ ] Can tenants configure their own AI prompts/personas?
• [ ] Are AI usage costs attributable per tenant?
• [ ] Is there per-tenant rate limiting?

Observability

• [ ] Are AI calls traced (Langfuse, OpenTelemetry, etc.)?
• [ ] Are errors logged with full context?
• [ ] Are cost spikes alerted?
• [ ] Are quality regressions alerted?

Compliance & UX

• [ ] Are users informed they're interacting with AI?
• [ ] Is there a feedback mechanism (thumbs up/down)?
• [ ] Are AI hallucinations contained (citations, source links)?
• [ ] Is there a "this is wrong, fix it" flow?

Production readiness

• [ ] Has the feature been tested with adversarial inputs?
• [ ] Has it been load-tested?
• [ ] What happens at 10x current traffic?
• [ ] What happens if the AI provider raises prices 50%?

My self-test

When I ship an AI feature, I won't go live until:

• Eval harness is green
• Cost per user is modeled and < 30% of plan price
• Latency P95 < 3s for non-streaming, first token < 800ms streaming
• Fallback chain handles single-provider outage gracefully
• Cost dashboard is wired and accessible to the founder

Anything less is a demo, not production.

Email [email protected] to scope an AI integration with these patterns built in.