Show HN: Keep large tool output out of LLM context: 3x accuracy 95% fewer tokens

LLM agents often place raw JSON tool outputs directly in the prompt. After a few tool calls, earlier results get compacted or truncated and answers become incorrect or inconsistent.

I built Sift, a drop-in MCP gateway that stores tool outputs as local artifacts (filesystem blobs indexed in SQLite) and returns an `artifact_id` plus compact schema hints when responses are large or paginated.

Instead of reasoning over full JSON in the prompt, the model runs a small Python query:

    def run(data, schema, params):
        return max(data, key=lambda x: x["magnitude"])["place"]

Query code runs in a constrained subprocess (AST/import guards + timeout/memory caps). Only the computed result is returned to the model.

Benchmark (Claude Sonnet 4.6, 103 questions across 12 datasets):

- Baseline (raw JSON in prompt): 34/103 (33%), 10.7M input tokens

- Sift (artifact + code query): 102/103 (99%), 489K input tokens

Open benchmark + MIT code: https://github.com/lourencomaciel/sift-gateway

Install:

    pipx install sift-gateway
    sift-gateway init --from claude

Works with Claude Code, Cursor, Windsurf, Zed, and VS Code. Existing MCP servers and tools require no changes.

Summary

The article discusses the Sift Gateway, an open-source API gateway designed to simplify the management and integration of microservices. It highlights the gateway's features, including load balancing, authentication, and monitoring, which aim to enhance the reliability and performance of microservice-based architectures.

Story

Show HN: Keep large tool output out of LLM context: 3x accuracy 95% fewer tokens