What an AI assistant can and can't do inside your product

Adding an AI assistant to a product has become one of the most common requests we receive. Large language models such as Anthropic's Claude are capable enough that a well-built assistant can answer most routine customer questions accurately. A poorly built one can confidently make things up, frustrate customers and create liability. The difference is rarely the model. It is the design around it.

This article covers what an assistant inside your product can reasonably do, where its limits are, and the five design decisions that determine whether it helps: grounding, citations, handoff, evaluation and privacy.

What an assistant does well

• Answering questions from your documentation, help center, policies and product guides, in the customer's own words rather than by keyword matching.
• Explaining how to complete a task in your product, step by step, at the moment the customer is stuck.
• Summarizing long content, such as a contract, a support history or a report.
• Drafting text for a person to review: a reply to a customer, a product description, a first pass at a form.
• Taking simple, well-defined actions through tools you expose, such as looking up an order status, when you control exactly what it can call.

These tasks share a property: the assistant works from information you provide, and a person or a clear rule can check the result.

What it should not do

A language model generates plausible text. Without constraints, plausible and correct are not the same thing. An assistant should not be the final authority on any of the following.

• Prices, refunds, contract terms or eligibility decisions, unless it is quoting an approved source and linking to it.
• Medical, legal or financial advice specific to the user's situation.
• Anything it cannot find in the material you gave it. "I couldn't find that; here is how to reach a person" is a correct answer.
• Irreversible actions, such as cancelling an account or issuing a refund, without explicit confirmation from the user and, often, a human review.

Most problems with AI assistants come from asking them to do things on this list without guardrails. The rest of this article is about those guardrails.

Grounding: answer from your content, not the internet

Grounding means the assistant answers from a defined set of documents rather than from its general knowledge. The usual technique is retrieval: your content is split into sections and indexed, the most relevant sections are retrieved for each question, and the model is instructed to answer using only those sections and to say so when they don't contain the answer.

Grounding quality depends on your content more than on the model. If your help center contradicts itself or is out of date, the assistant will repeat those problems fluently. Before launch, we review the source content and fix gaps and contradictions, which tends to improve the help center for human readers too.

Citations: show where each answer came from

Every factual answer should link to the document it came from. Citations let customers verify an answer and read further, they let your team trace a wrong answer back to its source, and they discourage the model from answering beyond its material. Claude can return the specific passages it relied on, which makes this practical to build rather than a best-effort add-on.

If an answer has no source to cite, treat that as a signal. The assistant should say it couldn't find the answer rather than produce one anyway, and that conversation is a good candidate for handoff.

Handoff: know when to bring in a person

An assistant should make reaching a human easier, not harder. Customers who feel trapped in a chatbot loop remember it. Define clear triggers for handoff before launch.

• The customer asks for a person, in any wording.
• The assistant cannot find a grounded answer to the question.
• The topic is on a sensitive list you define, such as billing disputes, account security or legal complaints.
• The customer repeats the same question or shows clear frustration.

When handoff happens, pass the conversation along so the customer doesn't have to repeat themselves. Integrations with help desks such as Zendesk, Intercom or a shared support inbox let your agents pick up with full context, including which documents the assistant already showed.

Evaluation: test it like any other feature

You wouldn't ship a checkout flow without testing it, and an assistant deserves the same treatment. Before launch, build an evaluation set: real questions your customers ask, each with the correct answer and the document that supports it. Include questions the assistant should decline or hand off, and questions phrased badly, because real customers phrase things badly.

• Run the evaluation set every time you change the prompt, the model version, the retrieval settings or the source content.
• Check three things for each answer: is it correct, does the citation actually support it, and did handoff fire when it should have.
• Review a sample of real conversations each week after launch, and add every failure you find to the evaluation set.
• Track how often conversations end in handoff and on which topics. Those topics usually point to missing or unclear documentation.

Privacy: decide what the assistant can see

An assistant that can see a customer's account is far more useful than one that cannot, and also far more sensitive. Decide deliberately what data reaches the model, and write the decision down.

• Send only the fields the assistant needs for the question, not the whole customer record.
• Enforce permissions in your own backend, so the assistant can only retrieve data the signed-in user is already allowed to see.
• Avoid sending sensitive identifiers such as full payment card numbers or government ID numbers at all.
• Set retention rules for conversation logs, and tell users in your privacy policy that an AI assistant is in use and how their data is handled.
• Read your model provider's commercial terms on data retention and training use, and make sure they match what you tell your own customers.

Where to start

The best first assistant is narrow: one product area, one well-maintained set of documents, a clear handoff path and an evaluation set of a few dozen real questions. Launch it to a portion of your users, review the conversations, and widen its scope only once it answers that narrow area reliably.

Papel Copilot, our early-access product built on Anthropic's Claude, is designed around these principles: grounded answers with citations, configurable handoff and conversation review. When a product needs something more specific, we build custom assistants as part of our AI integration service.