Slackbot: Using LLMs in a human-in-the-loop system

I’ve built human-in-the-loop agents in production, and I’ve hit all kinds of failures along the way. Some common ones are service outages, token overruns, and LLM output in the wrong format.

The goal with these systems is to let a model take over the low-level judgment calls so that people only have to approve what’s risky. The hard part is making that safe when any piece of the system can fail. To show how I approach it, I built a reference app you can read: maysco-demo-agent. There’s also a video walkthrough.

The app is a marketplace that pays its sellers through Stripe Connect in test mode. Before each payout, it screens the seller’s name against the US Treasury’s OFAC sanctions list. Every payout lands in one of 3 tiers:

1. Code decides. If the seller’s name isn’t close to anything on the list, plain code clears the payout.
2. The agent decides, and people see a log. For a close call on a small payout, a Claude model can clear it by itself.
3. People decide. Everything else waits for 1 or 2 reviewers to approve or reject it in Slack.

This is a demo, not legal or compliance advice. The seller names are invented, and every near miss below matches one of 3 fictional entries that I added to a copy of the real sanctions list.

The architecture

The app runs on AWS Lambda, SQS, and Postgres. It talks to 3 outside services: Stripe, Slack, and the Claude API. All of the infrastructure is defined in one file, sst.config.ts.

All state lives in Postgres. Only the worker Lambda reads the queue, calls Claude, or moves money in Stripe. The Slack Lambda only verifies a click and queues it, because Slack allows 3 seconds for a response.

Payout saga

Here is one payout that needs a review, from the buyer’s payment to the seller’s transfer.

1. Intake. A buyer pays $125.00, and Stripe sends a webhook. One transaction records the event, creates a saga, and writes an outbox message.
2. Screening. The worker fetches the seller from Stripe and scores the legal name against the sanctions list.
3. Decision. The score picks a tier. In the gray zone, the worker asks Claude, and a policy function in code makes the call.
4. Card. The worker posts the evidence to Slack. A payout that needs a review gets Approve and Reject buttons.
5. Review. A reviewer clicks. If nobody answers, the review expires and the payout stays held.
6. Recheck. The worker confirms the seller’s details haven’t changed since the screening.
7. Transfer. The worker creates the Stripe transfer for $112.50, and the saga moves to paid.

Every step follows the same pattern: read a message, do the work, then save the new status and the next step’s message in one transaction.

Whatever fails, 4 things have to stay true. A seller is never paid twice. A payout never gets stuck where nobody can see it. Money never moves without the required approvals. And the model can never do more than the code allows.

The 3 tiers

The app compares the seller’s legal name against every entry on the sanctions list and keeps the highest similarity score, a number from 0 to 1. That score decides who makes the call.

Code decides. Brindlecove Ceramics LLC scored 0.77, so code cleared it without calling the model. It was marked paid 4.6 seconds after Stripe’s webhook arrived.

The agent decides. Copperline Marine Supply scored 0.85 against one of the fictional entries, Vesserot Marine Supply. The model found no match with high confidence, because the two names share only the generic words “Marine Supply”. The payout was under $500, so the agent cleared it, 9.6 seconds after the webhook arrived. In Slack it shows up as a log entry:

People decide. Quillmark Traders LLC scored 0.84 against the fictional Quillmark Trading House. This time the model called it a possible match, because the distinctive word “Quillmark” appears in both names. Anything other than a confident “no match” goes to a person, so a reviewer got this card:

The rules are ordinary code. One function, decide, maps a score, a model result, and an amount to a tier:

export function decide(topScore: number, result: AssessmentResult, amount: number): Decision {
  // ...
  if (topScore >= THRESHOLDS.strong) {
    return {
      tier: "strong",
      requiredApprovals: 2,
      reason: `Name score ${score} is a strong match`,
    };
  }
  if (result.kind === "failed") return review(`The agent couldn't decide: ${result.reason}`);
  if (topScore >= THRESHOLDS.review) return review(`Name score ${score} needs a reviewer`);
  if (result.kind === "skipped") {
    return {
      tier: "clear",
      requiredApprovals: 0,
      reason: `Top name score ${score} is below review`,
    };
  }

  const { verdict, confidence } = result.assessment;
  if (verdict !== "no_match") return review(`The agent flagged a ${verdict.replace("_", " ")}`);
  if (confidence !== "high")
    return review(`The agent saw no match, but only with ${confidence} confidence`);
  if (amount > AGENT_CLEAR_LIMIT) {
    return review(
      `The agent saw no match, but the amount is over its $${AGENT_CLEAR_LIMIT / 100} limit`,
    );
  }
  return {
    tier: "agent_clear",
    requiredApprovals: 0,
    reason: `The agent cleared it: no match, high confidence, within its $${AGENT_CLEAR_LIMIT / 100} limit`,
  };
}

The order of the checks is what makes the tiers trustworthy. A score of 0.95 or higher returns before the function looks at the model’s result, and a failed model call always goes to a reviewer.

None of these limits depend on the model following its system prompt. That matters because the model reads text I don’t control: the seller’s legal name and LEI record, and the names, aliases, and remarks from the sanctions list.

The limits decide where the model can be wrong. They don’t make a wrong answer safe. If the model is wrong, or someone plants a prompt injection in that text, it can clear a payout of $500 or less for a seller who scored between 0.82 and 0.90. That payout goes out with no reviewer, and the only record is a log entry in Slack.

The cap is per payout. Nothing counts a seller’s earlier clears, so the same wrong answer can repeat on the next one. The score band is also a weaker limit against an attacker, who writes the name that produces the score.

Failure 1: invalid model output

When ordinary code hits a bug, it throws. SQS delivers the message again, and after 5 receives the message moves to a dead-letter queue where someone can find it.

A model call can return HTTP 200 and still be unusable. The output may be cut off at the token limit, refused, off-schema, or it may cite a sanctions entry the model was never shown. None of these is a bug in my code, so the dead-letter queue is the wrong place for them. What I want is for a person to look at the payout.

So assess.ts checks the reply before any code trusts it, starting with why the model stopped:

      // Check why the model stopped before reading what it said.
      if (response.stop_reason === "max_tokens") {
        return {
          kind: "failed",
          reason: "the model ran out of output tokens before it finished",
          retryable: false,
          call: call({ ...usage, outcome: "max_tokens" }),
        };
      }
      // ...

Each case returns a typed failure with a reason written for the reviewer, and decide sends it to 1 reviewer. This is also why the code calls messages.create and not the SDK’s messages.parse helper, which throws on a reply that was cut off.

To test it, I capped the model’s output at 64 tokens and ran another Quillmark payment. The call stopped with a stop_reason of max_tokens, and the reviewer got this card:

Only an unavailable API is retried. Any other failure goes straight to a person, because the same prompt would likely fail the same way.

Failure 2: the dual-write problem

Every step saves its new status to Postgres, and then a message has to reach SQS to trigger the next step. That’s 2 writes to 2 systems with no shared transaction. If the Lambda dies between them, the database says the step is done and nothing ever triggers the next one. No error is raised anywhere. The payout just stops.

The fix is the transactional outbox pattern. A step writes its outgoing messages to an outbox table in the same transaction as the status change, and commit sends them to SQS afterward.

If the send fails or never happens, the message is still sitting in the outbox. The sweeper runs every minute and sends any outbox row that is more than 60 seconds old and still unsent.

To test it, I committed a Brindlecove payment and skipped the send. The sweeper found the stranded messages 118 seconds later:

An outbox sends each message at least once, which means sometimes twice. So every receiver has to be safe to run twice:

• Webhooks. An inbox table stores each event ID under a unique key. A webhook I resent returned 200 with the body "duplicate".
• Step messages. Each message carries the status it expects. If the saga has moved on, the worker ignores the message.
• The transfer. completeTransfer looks for an existing Stripe transfer before it creates one. The idempotency key isn’t enough by itself, because Stripe can remove a key once it’s 24 hours old.

The Exactly Once lab on this site shows why this matters at volume.

What live testing uncovered

I designed for both failures from the start. Two more problems only showed up against the real Stripe API, because a fake behaves the way you assumed the real service would.

Fees. My original plan transferred the full $125.00 payment to the seller. The first live transfer failed with balance_insufficient: Stripe had deducted its processing fee, which left $121.07 available. The fix is a 10% commission, one constant in intake.ts. It’s why every payout in this post is $112.50.

Bank account changes. A reviewer’s approval is tied to exactly what they saw. payloadHash hashes the seller’s account, legal name, and bank account, plus the amount and currency, and any change voids the approval.

For the bank account, I first used the fingerprint that Stripe provides. On the first live test, I changed a seller’s bank account after an approval, and the approval survived: Stripe’s test bank token produces the same fingerprint every time. The field now holds the bank account record’s ID.

Here is the fix on a strong match, which needs 2 different reviewers. I approved Orlenka Freight LLC once, changed its bank account about 2 minutes later, and the card reset to 0 of 2:

What I’d change for production

• The wait. It’s a status column plus a sweeper that polls every minute. I’d use Step Functions task tokens or Lambda durable functions.
• Held funds. The demo holds funds but never releases them. A real system needs an audited path for that.
• The numbers. The thresholds and the $500 limit haven’t been tuned against real seller names. A wrong or manipulated model can release up to $500 per payout with no person involved, and nothing caps the total per seller.

The split itself is what I’d carry into any human-in-the-loop system. Let code decide what code can decide. Let the model decide only inside limits that code enforces. And when the model fails, send the work to a person, because a retry loop or a dead-letter queue won’t make the decision for you.

Resources and references

• maysco-demo-agent: the repo, with the commands for every demo scene. The code in this post is pinned to commit 123d4ce.
• Hector Garcia-Molina and Kenneth Salem, “Sagas” (ACM SIGMOD, 1987): the original paper on long-lived transactions with compensating steps.
• Chris Richardson, Transactional outbox: the pattern behind the fix for failure 2.
• Stripe, Idempotent requests: the source for the 24-hour key lifetime.
• Stripe, Separate charges and transfers: a transfer fails when it exceeds the platform’s available balance.
• Anthropic, Stop reasons and fallback: what each stop_reason means.