An AI chatbot for business use is no longer a scripted decision tree. Large language models can hold a natural conversation, understand loosely worded questions and draft useful answers. They can also state things that are wrong with complete confidence.
The gap between a helpful assistant and an embarrassing one is almost entirely planning. This guide covers the decisions to make before anyone writes a prompt.
Start with a narrow, valuable use case
The most common planning error is scope. A chatbot that is supposed to do everything usually does nothing well.
Pick one job where questions are frequent, answers already exist in writing, and a wrong answer is recoverable. Typical starting points:
- Customer support: answering product, shipping, billing and policy questions from your help content
- Sales assistance: explaining services, qualifying inquiries and booking calls
- Internal knowledge: helping staff find procedures, HR policies or technical documentation
- Order and account lookups: checking status through a connection to your existing systems
Write down what the chatbot should handle and, just as important, what it should not. Legal advice, medical guidance, refund approvals and anything involving an upset customer are common items for the exclusion list.
Ground answers in your own content with RAG
A language model on its own knows nothing reliable about your business. It was trained on general text, and it will fill gaps with plausible guesses. This is the main source of fabricated answers, often called hallucinations.
Retrieval-augmented generation, or RAG, is the standard way to address this.
How RAG works
- Your content, such as help articles, product pages, manuals and policies, is split into passages and indexed.
- When a user asks a question, the system searches that index for the most relevant passages.
- Those passages are passed to the model along with the question.
- The model writes an answer based on the supplied text, ideally with a reference to the source.
The model is no longer answering from memory. It is answering from your documents.
Why content quality decides the outcome
RAG retrieves what you give it. If your documentation is outdated, contradictory or missing, the chatbot will reflect that faithfully. Before building, audit the source material:
- Remove or archive obsolete pages
- Resolve conflicts between documents, for example two different return policies
- Fill the gaps that support staff currently answer from memory
- Assign an owner who keeps the content current
Also instruct the system to say it does not know when the retrieved content does not contain an answer. A clear “I am not sure, let me connect you with someone” is far better than an invented reply.
Design the human handoff
No chatbot resolves every conversation. Plan the exit path as carefully as the main path.
A good handoff has three properties:
- Clear triggers. The user asks for a person, the bot fails to answer more than once, the topic is on the exclusion list, or the message suggests frustration or urgency.
- Full context. The human agent receives the conversation transcript and any details collected, so the customer does not repeat themselves.
- Honest expectations. If nobody is available, say so and offer a callback or ticket instead of leaving the user waiting.
Make it easy to reach a person. Hiding the option behind several failed attempts damages trust more than any wrong answer.
Also tell users they are talking to an AI assistant. It sets expectations correctly, and some jurisdictions and platforms require the disclosure.
Measure quality, not just volume
Conversation counts tell you the chatbot is used. They do not tell you it is right.
Before launch
Build a test set of real questions, taken from support tickets, emails and sales calls. Include easy ones, hard ones and questions the bot should refuse. Run the set, review every answer by hand, and rerun it after each change to prompts, content or models. This is the only reliable way to know whether a change made things better or worse.
After launch
Track a small number of meaningful measures:
- Resolution rate: conversations that ended without needing a human
- Escalation rate and reasons: what the bot could not handle, which shows you content gaps
- Answer accuracy: a regular human review of a sample of transcripts
- User feedback: a simple thumbs up or down after an answer
- Unanswered questions: queries where nothing relevant was retrieved
Review transcripts weekly in the first months. They are the most direct view you will ever get of what customers actually ask.
Handle privacy and data deliberately
A chatbot collects whatever users type into it, and users type more than you expect. Decide the data rules up front.
Questions to answer:
- Where is conversation data processed? With a hosted model API, messages leave your infrastructure. Check the provider’s terms on retention and on whether inputs are used for model training.
- What do you store, and for how long? Keep transcripts only as long as they are useful, and restrict who can read them.
- What sensitive data might appear? Payment details, health information and identity numbers should be masked or blocked where possible.
- Who can the bot retrieve content for? An internal assistant must respect document permissions. A staff member should not get answers from files they are not allowed to open.
- What regulations apply? Depending on your industry and the states you operate in, privacy and sector rules may affect the design.
If your data cannot leave your control, the model can run on infrastructure you manage. See private AI hosting for that approach.
Protect against misuse
Users will try to make the bot ignore its instructions, a technique known as prompt injection. Treat everything the model reads, including user messages and retrieved documents, as untrusted input. Limit what the bot is permitted to do, require confirmation for any action with consequences, and never place secrets in the prompt.
Common mistakes to avoid
- Launching without a test set. You cannot improve what you have not measured.
- Indexing everything. More content is not better content. Irrelevant documents produce irrelevant answers.
- No fallback. A bot that never admits uncertainty will eventually state something false.
- Giving it authority it should not have. Do not let a chatbot make binding commitments on pricing, refunds or contracts without controls.
- Treating launch as the finish line. Content changes, products change and models change. Budget for ongoing review.
- Hiding the human. If people cannot escalate, they leave.
- Choosing the model first. The model matters less than the content, the retrieval quality and the handoff design.
A practical rollout sequence
- Define one use case, its boundaries and its success measures.
- Clean and organize the source content.
- Build a prototype with retrieval and a test set.
- Test internally with staff who know the right answers.
- Release to a limited audience with handoff in place.
- Review transcripts, fix content gaps and expand gradually.
Each stage produces evidence for the next. Skipping one usually means returning to it later, with customers watching.
Key takeaways
- Start with one well-defined use case where answers already exist in writing.
- Ground the chatbot in your own content with RAG, and clean that content first.
- Instruct the bot to admit when it does not know, and make human handoff easy and context-rich.
- Measure accuracy with a real test set and regular transcript reviews, not only conversation volume.
- Decide where data is processed, what is stored and who can access it before launch.
- Treat the chatbot as an ongoing product with an owner, not a one-time project.
iSky Creative designs and builds AI chatbots grounded in your own content, and you can get in touch to discuss your use case.