My MCP server had 145 green unit tests and a live test of 33 checks that posts JSON-RPC to the deployed endpoint, the way a client does. What none of them did was go through a real MCP client library, and none asked the question L Anil Kumar Singha builds his MCP Failure Lab around: after a call fails, does the session still work?
His lab mostly tests clients, but it has an external-target mode: a small JSON file describes a call to your server, what you expect back, and a second call in the same session afterwards. I wrote four of these, all read-only, and ran them against production.
Two failed on the first run, and they were real. When get_story got a slug that doesn't exist, or any tool was called without a required argument, my server answered with a JSON-RPC protocol error (-32602). A tool that ran and refused ("not found") is a tool execution error in the MCP spec, and should come back as a normal result with isError: true and a message. Missing arguments were a grey area: the 2025-06-18 revision listed invalid arguments as a protocol error, and the 2025-11-25 revision moved input validation into tool results (SEP-1303), precisely so the model can see what to fix. On a protocol error the client SDK throws, so the model may never read "story not found" and can't correct itself. My edit_story tool already did it right; get_story didn't. And my live test had three checks asserting exactly that -32602: I had written down what the server did and called it correct. A raw HTTP test reads an error field and passes. A real SDK client throws, and that is what the lab showed me.
The fix was a few lines in one place: a refusal from a tool now comes back as a result with isError: true. Only an unknown tool name stays a protocol error.
One of my own scenarios was wrong, too. I assumed search_stories without a query was an error; the schema says the query is optional. My assumption, not the server's.
The live test now expects isError results, and the four scenarios run after every backend deploy, next to a check that the live server still describes itself the way the version lock says. All of it passes against production.
Reproduce this
Prerequisites: a deployed MCP server reachable over Streamable HTTP (HTTPS), Node.js 20+ for npx, and a token your server accepts if its tools need sign-in. MCP Failure Lab: github.com/anilloutombam/mcp-failure-lab, npm package mcp-failure-lab (this story used 0.11.0; pin the version you test with). My four scenarios, as working examples: github.com/worklore/worklore-mcp/tree/main/tests/live_scenarios.
Your agent will need from you: the server URL, a bearer token (or confirmation that tools work without one), the names of two or three READ-ONLY tools with valid arguments, and one argument value that should make a tool refuse (an id that doesn't exist).
Steps:
- Confirm the lab runs:
npx -y mcp-failure-lab@0.11.0 demoshould end withAssertions: passed. - Create a target file, e.g.
live_scenarios/target.json. Keep the token out of the file:headerEnvmaps a header to an environment variable whose value is the whole header.
{"adapter": "mcp", "config": {"transport": "http", "url": "https://YOUR-SERVER/mcp",
"headerEnv": {"Authorization": "MCP_AUTHORIZATION"}}}
Then export MCP_AUTHORIZATION="Bearer <token>".
- Write one scenario per file. Every scenario has a
call, anexpect, and anobserve: a second call in the same session that must succeed. That second call is the point. A happy-path one:
{"name": "search answers, and the session still reads an item after it",
"call": {"tool": "YOUR_SEARCH_TOOL", "args": {"query": "test"}},
"timeoutMs": 15000,
"expect": {"outcome": "success", "maxDurationMs": 10000, "result": {"isError": false}},
"observe": {"call": {"tool": "YOUR_READ_TOOL", "args": {"id": "A_REAL_ID"}},
"timeoutMs": 15000, "expect": {"outcome": "success", "result": {"isError": false}}}}
- Add the two refusal cases: a read with an id that doesn't exist, and a call missing a required argument. Expect
"outcome": "error"with"result": {"isError": true, "textContains": "<part of your message>"}, then observe a normal call succeeding. - Run each:
npx -y mcp-failure-lab@0.11.0 run live_scenarios/<file>.json --target live_scenarios/target.json. The output ends withAssertions: passedor a list of failures. - Read failures carefully before fixing anything.
expected a result, but no result was receivedplusAdapter execute: errormeans your server answered a tool refusal with a JSON-RPC error instead of anisErrorresult. A failure can also be your own wrong assumption: check the tool'sinputSchema(e.g. whether an argument is actuallyrequired). - Fix refusals in one place if you can: wherever your server turns a tool's refusal into a response, return
{"content": [{"type": "text", "text": "<message>"}], "isError": true}as theresult. Keep JSON-RPC errors for unknown tools and unparseable requests. - Run the scenarios after every deploy (a few lines in your deploy script; note that
npx … runoutput must be checked forAssertions: passed, so pipe carefully and keep the exit code). Never add a scenario that writes, publishes or charges anything: it runs against production.
Verify: all scenarios end with Assertions: passed against the deployed server, including the two refusal cases, whose observer call (the second call in the same session) also passes.