Support Agent › guide
The worry with an AI chat on a support page is not that it fails to answer: it is that it answers wrongly and sounds sure. It tells someone they can have a refund you don't offer, or makes up a delivery date. These five steps keep the useful part (questions answered at 2am) and drop the risky part. The numbers come from running the open-source Support Agent on BANKING77, a public set of 13,083 real online-banking questions.
An agent can only be as right as the text it answers from. Write one short article per question customers ask, and write out the facts: "Standard delivery takes 3 to 5 working days", not "Delivery is quick". Put each topic under its own heading, because the agent copies the passage under the heading that matches. Your current saved replies are a good first draft.
Some messages should never get an automatic answer, however good the match. Support Agent hands these off before it searches at all:
Every answer gets a confidence between 0 and 1. Where to draw the line is a business decision: a wrong answer costs more than a handoff, which costs someone a few minutes. So write 30 or more questions the way customers write them (typos, "my parcel never came", "it's been a week"), each with the article that answers it or handoff. Include some the articles don't cover. The tool then tries every bar and recommends the one that answers the most correctly while keeping wrong answers at or under, say, 5% of answers.
Between the answer bar and a lower suggest bar, the agent does not state an answer: it offers its two best articles with "talk to a person" one click away. A wrong suggestion costs the customer a glance.
Questions customers really asked, each labelled with the article that answered it or handoff, teach the agent how your customers word things and which questions looked answerable but weren't. On BANKING77 (22 articles; test questions held out from everything we tuned):
| Past questions per topic | Answered right | Answered wrong (share of answers) | Offered articles |
|---|---|---|---|
| none | 65 (3.0%) | 0 (0%) | 1,172 |
| 10 | 786 (36.4%) | 39 (4.7%) | 431 |
| 40 | 1,007 (46.6%) | 66 (6.2%) | 211 |
2,160 test questions, 1,360 of them answerable from an article; the rest should go to a person. The bar was set on training questions to keep wrong answers at or under 5%; on unseen questions it came out at 4.7% and 6.2%, which is why you re-run your own test questions after every change.
BANKING77 is hard on purpose: "my card payment was declined" (needs a person) reads almost the same as "contactless isn't working" (an article covers it). A shop with fewer look-alike questions will do better. The only way to know is to run your own.
Without a model, Support Agent's answers are sentences copied from your article, so nothing can be invented. A model (your own Anthropic, OpenAI or Gemini key, or Ollama on your own machine) can make the wording friendlier, but it is never the one deciding whether to answer. It rewrites an answer already chosen, from that article only, and its text is rejected if it contains a number, a link, an email address or a promise ("refund", "free of charge", "guarantee") that the article does not, or if it cites no passage. Then the copied answer is used instead.
The browser page runs steps 1 to 4 on your own articles and questions without uploading anything. To put the chat on your site:
ADMIN_KEY=$(openssl rand -hex 16) node server.mjs <script src="https://your-agent.example.com/widget.js" defer></script>
Handoffs go to your inbox through a webhook (for example a Support Desk mailbox) with the chat so far, and the admin page lists what customers asked that no article answered.