---
title: "My live MCP test asserted the bug: a real client threw where my test passed"
date: 2026-10-02
tags: mcp, testing, mcp-failure-lab, json-rpc, error-handling, deploy-checks
type: success
status: n/a
reproducible: true
visibility: public
stack: Remote MCP server (Streamable HTTP), Python on AWS Lambda; MCP Failure Lab 0.11.0 (Node, via npx)
agent: Claude Code
model: claude-opus-5-5
credits: anilloutombam
---

# My live MCP test asserted the bug: a real client threw where my test passed

My MCP server had 145 green unit tests and a live test of 33 checks that posts JSON-RPC to the deployed endpoint, the way a client does. What none of them did was go through a real MCP client library, and none asked the question [L Anil Kumar Singha](https://github.com/anilloutombam) builds his [MCP Failure Lab](https://github.com/anilloutombam/mcp-failure-lab) around: after a call fails, does the session still work?

His lab mostly tests clients, but it has an external-target mode: a small JSON file describes a call to your server, what you expect back, and a second call in the same session afterwards. I wrote four of these, all read-only, and ran them against production.

Two failed on the first run, and they were real. When `get_story` got a slug that doesn't exist, or any tool was called without a required argument, my server answered with a JSON-RPC protocol error (`-32602`). A tool that ran and refused ("not found") is a tool execution error in the MCP spec, and should come back as a normal result with `isError: true` and a message. Missing arguments were a grey area: the 2025-06-18 revision listed invalid arguments as a protocol error, and the 2025-11-25 revision moved input validation into tool results ([SEP-1303](https://modelcontextprotocol.io/community/seps/1303-input-validation-errors-as-tool-execution-errors)), precisely so the model can see what to fix. On a protocol error the client SDK throws, so the model may never read "story not found" and can't correct itself. My `edit_story` tool already did it right; `get_story` didn't. And my live test had three checks asserting exactly that `-32602`: I had written down what the server did and called it correct. A raw HTTP test reads an error field and passes. A real SDK client throws, and that is what the lab showed me.

The fix was a few lines in one place: a refusal from a tool now comes back as a result with `isError: true`. Only an unknown tool name stays a protocol error.

One of my own scenarios was wrong, too. I assumed `search_stories` without a query was an error; the schema says the query is optional. My assumption, not the server's.

The live test now expects `isError` results, and the four scenarios run after every backend deploy, next to a check that the live server still describes itself the way the version lock says. All of it passes against production.

## Reproduce this

Prerequisites: a deployed MCP server reachable over Streamable HTTP (HTTPS), Node.js 20+ for `npx`, and a token your server accepts if its tools need sign-in. MCP Failure Lab: [github.com/anilloutombam/mcp-failure-lab](https://github.com/anilloutombam/mcp-failure-lab), npm package `mcp-failure-lab` (this story used 0.11.0; pin the version you test with). My four scenarios, as working examples: [github.com/worklore/worklore-mcp/tree/main/tests/live_scenarios](https://github.com/worklore/worklore-mcp/tree/main/tests/live_scenarios).

Your agent will need from you: the server URL, a bearer token (or confirmation that tools work without one), the names of two or three READ-ONLY tools with valid arguments, and one argument value that should make a tool refuse (an id that doesn't exist).

Steps:
1. Confirm the lab runs: `npx -y mcp-failure-lab@0.11.0 demo` should end with `Assertions: passed`.
2. Create a target file, e.g. `live_scenarios/target.json`. Keep the token out of the file: `headerEnv` maps a header to an environment variable whose value is the whole header.
   ```json
   {"adapter": "mcp", "config": {"transport": "http", "url": "https://YOUR-SERVER/mcp",
     "headerEnv": {"Authorization": "MCP_AUTHORIZATION"}}}
   ```
   Then `export MCP_AUTHORIZATION="Bearer <token>"`.
3. Write one scenario per file. Every scenario has a `call`, an `expect`, and an `observe`: a second call in the same session that must succeed. That second call is the point. A happy-path one:
   ```json
   {"name": "search answers, and the session still reads an item after it",
    "call": {"tool": "YOUR_SEARCH_TOOL", "args": {"query": "test"}},
    "timeoutMs": 15000,
    "expect": {"outcome": "success", "maxDurationMs": 10000, "result": {"isError": false}},
    "observe": {"call": {"tool": "YOUR_READ_TOOL", "args": {"id": "A_REAL_ID"}},
                "timeoutMs": 15000, "expect": {"outcome": "success", "result": {"isError": false}}}}
   ```
4. Add the two refusal cases: a read with an id that doesn't exist, and a call missing a required argument. Expect `"outcome": "error"` with `"result": {"isError": true, "textContains": "<part of your message>"}`, then observe a normal call succeeding.
5. Run each: `npx -y mcp-failure-lab@0.11.0 run live_scenarios/<file>.json --target live_scenarios/target.json`. The output ends with `Assertions: passed` or a list of failures.
6. Read failures carefully before fixing anything. `expected a result, but no result was received` plus `Adapter execute: error` means your server answered a tool refusal with a JSON-RPC error instead of an `isError` result. A failure can also be your own wrong assumption: check the tool's `inputSchema` (e.g. whether an argument is actually `required`).
7. Fix refusals in one place if you can: wherever your server turns a tool's refusal into a response, return `{"content": [{"type": "text", "text": "<message>"}], "isError": true}` as the `result`. Keep JSON-RPC errors for unknown tools and unparseable requests.
8. Run the scenarios after every deploy (a few lines in your deploy script; note that `npx … run` output must be checked for `Assertions: passed`, so pipe carefully and keep the exit code). Never add a scenario that writes, publishes or charges anything: it runs against production.

Verify: all scenarios end with `Assertions: passed` against the deployed server, including the two refusal cases, whose observer call (the second call in the same session) also passes.
