10 best practices for deploying AI chatbots in production

Deploying an AI chatbot is not “turn on the widget and add a load balancer.” The failure modes are wrong answers, silent policy contradictions, actions that fire without permission, and handoffs that drop context. Infrastructure still matters. It is not the first problem.

Guzli publishes this page. We sell the chatbot and voice agent you can put through this sequence. The practices apply to Fin, Lyro, Ada, or any retrieval-grounded agent. Vendor-specific test names (Fin Simulations, Batch tests, Previews) are cited so you can map them—not because you must buy Intercom.

If you are still choosing a platform: buyer’s guide, Guzli vs Intercom vs Chatbase, ROI guide.

1. Start with a scoped pilot, not the whole catalog

Pick one audience and one intent cluster that already has written answers: shipping cutoffs, password reset, plan differences, WISMO (“where is my order?”). Intercom’s own deploy guidance says to train, test, deploy with audience controls, then analyze—not to open every channel on day one (How to deploy an AI agent).

Write success metrics before go-live:

• Correct-answer rate on a held-out question set (questions you did not use while editing articles — not “resolution rate” alone)
• Escalation rate and why (customer asked / low confidence / policy)
• Time to first human reply after handoff
• For lead capture: meetings held, not chats started

Internal staff or a beta segment first. A homepage widget for every visitor is not a pilot.

2. Audit knowledge before you “train”

The bot cannot outperform contradictory articles. A practical readiness pass looks for coverage against real tickets, duplicates, contradictions, freshness with a named owner, and answers that include the resolution steps—not a feature tour. Fini, a vendor, publishes one version of that checklist (knowledge readiness); treat it as a vendor list, not third-party proof.

Do this on your top 20–30 intents:

• One topic per article; titles in the customer’s words
• Kill or label outdated policies
• Separate public answers from internal macros
• Decide whether solved tickets are in the corpus (Guzli can ingest them when connected; that is a permission decision, not a default)

Fin’s knowledge guide is explicit that outdated policies are a top cause of bad answers and that high-involvement, low-resolution articles should be rewritten first (Fin knowledge management).

3. Build an eval set from real questions

Unit-testing “intents” in isolation is not enough. Pull 30–50 questions from tickets or chat. Tag the expected behavior: answer, refuse, escalate, or call a tool.

Intercom documents three layers that map to any serious vendor:

• Preview — you act as the customer (can hit live APIs; treat as dangerous)
• Batch tests — many real questions, content coverage
• Simulations — multi-step procedures, sandboxed (Simulations vs Batch tests vs Previews)

On Guzli, run the same idea: a spreadsheet of questions, a human score (correct / incomplete / wrong / should-have-escalated), plus a staging widget. Do not connect production Stripe cancels to preview chats.

Hypothetical scoring rule (not a measured industry law): do not expand audience until the held-out set is at least ~90% acceptable and zero wrong-but-confident policy answers. Fin’s deploy article uses 90%+ as a reasonable target before go-live; treat that as their guidance, not a guarantee.

4. Design fallbacks as product behavior, not a snippet

Low confidence should not become a hallucinated paragraph. Define the branches:

A confidence < 0.7 JavaScript snippet is not a strategy. Thresholds differ by vendor and are often not exposed as a single float. Write the behavior, then ask the vendor where it is configured.

5. Write the handoff contract

Before launch, the receiving teammate must see:

• Customer identifier and page URL
• Questions asked and answers attempted
• Tools already called (and whether they succeeded)
• Why the bot handed off

Guzli can open a ticket in Salesforce, Zendesk, or Guzli’s help desk with that thread. Fin documents the difference between a procedure handoff (you configured it; billable) and a default escalation (customer asked for a human; not billed)—Fin outcomes. Your playbook should name the equivalent.

Also decide after-hours: queue, callback, or keep the bot on informational intents only. Phone is a separate channel; see can AI answer phone calls if voice is in scope.

6. Monitor correctness, not just uptime

Dashboards that only show “messages sent” hide harm.

Watch weekly:

• Wrong-answer samples (human review, 20 conversations)
• Content-gap clusters (same question, no article)
• Action failures (Shopify timeout, calendar full)
• Handoff abandon (ticket created, no agent)
• Cost units (Guzli message credits; Fin outcomes; Tidio Lyro conversations)

Alert on: spike in escalations, spike in “assumed resolution” if you pay for that, or any action error rate above your agreed floor.

Infrastructure alerts (5xx, widget fail) still belong here. They are necessary and insufficient.

7. Version the corpus and the rules like code

Conversation design is configuration. Treat it as such:

• Staging widget with the next article pack
• Diff help-center changes that the bot can see
• Named owner and rollback (revert the article or disable the action)
• Do not “hot-edit” a refund policy in production on a Friday

Git for prompts and allowlists is useful. Git for the entire help center is optional if the help center already has drafts and publish control.

8. Lock down data and actions

Production chat sees whatever the visitor types. Minimum bar:

• TLS in transit; vendor encryption at rest (ask for the report, do not assume)
• PII redaction or “do not store” for payment-card and government IDs
• Tool permissions: read-only order lookup first; writes (refund, cancel, address change) behind policy checks and audit logs
• Rate limits on the public widget
• GDPR/CCPA: what is stored, where, and how a deletion request flows

Guzli public copy names Shopify, Stripe, Calendly, Zendesk, HubSpot, and Salesforce—not “every MCP.” Do not enable a tool you have not permissioned.

9. Scale traffic after the answers are stable

Auto-scaling and caches help when the widget is already correct. They do not fix a bad corpus.

When volume grows:

• Cache retrieval documents, not generated answers (stale answers are worse than a slower first token)
• Add a second intent cluster only after the first cluster’s error budget is green (wrong-confident answers stay at the floor you wrote down)
• Add a channel (WhatsApp, email, voice) only when the same knowledge and handoff rules apply

A generic @cache.memoize around get_product_info is not a chatbot architecture. If the product catalog is the source of truth, call Shopify live or cache with an explicit TTL (how long a cached document may be reused) and a stampede plan so many widgets do not refetch at once.

10. Close the loop every week

Deployment is the start of the content job.

• Weekly: sample transcripts, ship article fixes, retune refusals
• Monthly: refresh the eval set from new tickets
• Quarterly: decide whether to add an action or a channel

Fin’s content-gap workflow (Recommendations + Topics Explorer) is one vendor’s version of “read the conversations you failed” (close content gaps). On Guzli, use the same habit with your analytics and ticket notes.

Go-live checklist

Use this as a gate, not decoration:

• Intent cluster and audience written down
• Top articles reviewed for contradictions and freshness
• Eval set scored; wrong-confident policy answers at zero
• Human request always works
• Handoff destination and required fields tested
• Actions in staging only, or none
• Monitoring owner and weekly review booked
• Rollback: disable widget or disable the dangerous action
• Support team briefed on what the bot will refuse
• Cost alerts on the billing unit you actually have

Pitfalls that still show up

1. Promising “we handle everything” — publish the intent list
2. Skipping edge cases — refunds, angry users, multi-account users
3. Never updating articles — the bot will keep the old policy
4. No user research — you automated the wrong FAQ
5. Dropping context on handoff — the customer repeats themselves and blames AI

Conclusion

Ship a narrow, grounded, measurable chatbot. Expand when the transcripts say you can. The rest is maintenance.

If you want to run this sequence on Guzli: book a demo, see current plans, or start a trial if checkout still shows one. There is no start-free promise unless checkout shows a trial.