Skip to content
Prathamesh Pathak logo, home Prathamesh

Case Study

OK support: a bot that resolves first

Mining 50 real chat scenarios, I redesigned the bot around an FAQ plus Yes/No loop, removing roughly 40 percent of the agent queue.

· · 6 min read
OK support: a bot that resolves first

At OK, the 10-minute grocery delivery brand inside SuperAGI, the AI company I work at, shoppers use native Android and iOS apps while support lives on WhatsApp through Wati, a WhatsApp business API. Volume grows with orders. One evening a shopper typed that she ordered amla and got lemon. A rider was demanding cash on a prepaid order somewhere else in the city. Both chats landed in the same queue, behind someone asking what time the store closes. That night is why the bot got redesigned.

Problem and context

The v1 bot offered three doors: track order, report issue, talk to support. All five issue paths (missing, damaged, incorrect, cancel, safety) collapsed into a single assign-to-team node. Every branch ended with a support agent arriving shortly. The bot was a receptionist with one line: please hold.

That bot existed for cost, not quality. At low volumes a couple of humans had to sit in the loop anyway, since refunds and judgment calls needed people. So the bot’s job was a cheap front door: greet, collect the order ID, name the issue, hand over. Routing everything to agents was not a flaw at that stage. It was the design, sized for a queue two people could drain.

The v1 graph, simplified. Every road ends at the same node:

graph TD
    Start([Customer opens<br/>support]) --> Track[Track order]
    Start --> Report[Report issue]
    Start --> Talk[Talk to<br/>support]
    Track --> Status[Order status<br/>shown]
    Report --> Issue{Issue type}
    Issue --> Missing[Missing]
    Issue --> Damaged[Damaged]
    Issue --> Incorrect[Incorrect]
    Issue --> Cancel[Cancel]
    Issue --> Safety[Safety<br/>incident]
    Missing --> Team[[Assign to<br/>support team]]
    Damaged --> Team
    Incorrect --> Team
    Cancel --> Team
    Safety --> Team
    Talk --> Team
    Team --> Wait([An agent will<br/>be here shortly])

That shape made sense for week one. By month three it meant agents triaged questions the bot already knew how to answer while edge cases waited behind them. And the speed contract kept creating support chats on its own. Orders pack within 60 seconds, so address changes, cancellations and item edits after confirm are all impossible by design. A shopper who taps checkout and instantly regrets one item has exactly one move: open a chat and plead. Every one of those becomes a conversation.

Discovery and validation

The evidence sat in our support inbox. I read the Wati chat logs the way an agent would, shift after shift. Then I mined them into 50 granular scenarios in customer voice. The amla-for-lemon swap. Short brinjal instead of long, with a photo. A double charge of Rs 732 on one order. Someone asking to cancel idli batter, typed half in Hinglish, the Hindi-English mix our shoppers write in. A rider asking for extra money at the door.

Real patterns emerged that no dashboard showed. Most contacts were informational or simple: timings, areas, catalog, refunds already in motion. The emotional ones were rare and unmistakable: extortion, spoiled meat, swapped bags. Those needed a human fast, not eventually. The v1 diagram confirmed the structural gap behind both patterns. No path resolved without an agent, and no success state existed anywhere in the graph. The system could not say yes to anything.

Trade-offs and prioritization

The v2 redesign makes one visible cut: the generic Talk to Support button becomes a Help and FAQs menu, and every answer ends in a DidThisHelp Yes/No loop. Yes ends the chat warmly. No connects to an agent. The agent path stays one tap away for everything, including a Something Else escape hatch for the shopper whose problem matches no menu. In one sentence: the FAQ menu answers, the Yes/No loop confirms, and the agent path stays one tap away throughout.

The v2 flow, simplified:

graph TD
    Start([Customer opens<br/>support]) --> Menu[Help and<br/>FAQs menu]
    Menu --> Answer[Answer shown]
    Answer --> Loop{Did this help}
    Loop -->|Yes| End([Chat ends<br/>warmly])
    Loop -->|No| Agent[[Agent<br/>takes over]]
    Menu --> Else[Something Else<br/>escape hatch]
    Else --> Agent

What I did not automate mattered more. Issue paths still escalate to humans, and refunds still need agent judgment under a Rs 500 autonomy cap, with anything above going to the shift manager over the escalations group. Agents hold no payment portal access at all, so money can never move on one person’s impulse. Automating apologies without the power to fix would have felt like a trick. The bot resolves information. Humans resolve money and safety.

The larger bet behind this sits in the CX master plan: trust-based refunds for high lifetime value customers with low refund rates. Measured runs removed roughly 40 percent of the agent queue against an estimated 1 to 2 percent fraud cost. The wrong-item policy already points that way. Keep the item, take the refund, no exchanges. At 10-minute speeds reverse logistics is absurd, so the plan stops pretending otherwise.

Execution and alignment

I wrote the flow, then worked with developers implementing the WhatsApp bot and the CX leads who owned tone. Fifty canned dispositions standardized replies across 7 categories, so agent answers matched bot answers word for word. An agent’s shift runs on rails now: type a slash to pull the right template, check the dashboard, process the refund inside the cap, tag the conversation, clear the ticket flag. Anything above the cap or outside the script goes to the shift manager with the chat link pasted in the group.

Hiring moved with the system. Candidates face three scenarios, including a four-chat triage and a rider no-show, and answers like asking for five bag photos or telling the customer to wait 24 hours fail on the spot. New hires pass a 90 percent SOP test plus sandbox hours before touching a live chat. And the 10-minute SLA is not a poster on a wall. An n8n watchdog watches every customer message, waits ten minutes, rechecks the Wati thread, and pings Slack with the customer name, phone and last four messages when no agent replied. A pleasantry scanner keeps it honest: short thanks, an ok or achha, a folded-hands emoji. None of those page anyone at midnight. The bar is explicit because the queue is unforgiving: 10-minute first response, refunds averaging 13 minutes, CSAT above 4.5.

Measurable outcomes

The program numbers at OK scale: 80% of support operations automated through agents and chatbots, average refund resolution held to 13 minutes, with standardized workflows running the operation end to end. The v2 flow specifically moved answerable questions (areas, timings, catalog) into the Yes/No loop, with the Yes rate as the standing metric and Other free-text mining as the feed for new FAQ entries. Acquisition was never the question on this project. Users arrive through the app stores, and support meets them where they already are, inside WhatsApp.

Retrospective and learnings

Three open questions I am tracking on the bot itself. Do repeat callers mash No without reading, turning the loop into a speed bump on the way to a human. Does Something Else become the default door, which would mean the menu fails its one job. Do agents quietly rewrite the canned dispositions, which would signal the tone system does not survive contact with real chats. The Yes rate and the escape-hatch share answer all three, and both get reviewed with the SOP every month.

The assumption that held: customers accept bot answers when the escape to a human is obvious. Nobody resents a menu that answers in seconds. They resent a menu that traps. The loop works because No is always one tap away.

Prathamesh Pathak

Written by

Prathamesh Pathak

Building AI-native products. Writing about PM craft, career growth, and shipping.

Visit LinkedIn →

More case studies

OK analytics: the funnel before the fix
analytics

OK analytics: the funnel before the fix

OK leadership flew blind on pricing, carts and catalog. I scoped a 5-week Mixpanel integration across client and server with identity stitching and nine answered business questions.

Prathamesh Pathak Prathamesh Pathak · ·5 min read