MCP Integration

Model Context Protocol (MCP) lets AI assistants like Claude, Cursor, Windsurf, and VS Code (with GitHub Copilot) use Truesight as a tool provider. Instead of writing API calls yourself, your AI assistant can manage datasets, build and deploy evaluations, run live scoring, review results, and perform error analysis, all through natural language.

Overview

Truesight exposes a curated set of tools through MCP. When connected, your AI assistant can:

• List, inspect, create, and upload datasets
• Configure dataset inputs and judgment criteria
• Create and deploy live evaluation endpoints
• Run live evaluations and score inputs on demand
• View evaluation results
• Browse, flag, and judge review queue items; promote reviewed data back to datasets
• Perform error analysis: suggest error notes, consolidate categories, and apply mappings

MCP connections use the same Platform API Keys as the REST API. The scopes you assign to the key determine which tools the assistant can use.

Quick setup

Step 1: Create a platform API key

1. Go to Settings in Truesight
2. Click Create Key in the Truesight API Keys section
3. Enter a name (e.g., "Claude", "Cursor", or "VS Code")
4. Select the scopes you want the assistant to have access to (or select all)
5. Click Create

Step 2: Copy the MCP config

After creating the key, the key reveal dialog shows a collapsible Use with MCP section. Expand it and click Copy to copy the MCP configuration JSON to your clipboard.

The config looks like this:

{
  "mcpServers": {
    "truesight": {
      "url": "https://api.truesight.goodeyelabs.com/mcp/",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY_HERE"
      }
    }
  }
}

Step 3: Add to your MCP client

Paste the configuration into your MCP client's config file. See the client-specific instructions below.

Client setup

VS Code (GitHub Copilot)

Click a button to install, then replace YOUR_API_KEY_HERE with your platform API key. Requires VS Code 1.99+ with GitHub Copilot enabled.

Alternatively, add it manually:

1. Open VS Code and press Cmd/Ctrl + Shift + P to open the Command Palette
2. Run MCP: Add Server
3. Select HTTP as the server type
4. Enter the URL: https://api.truesight.goodeyelabs.com/mcp/
5. Give it the name truesight

Or add it directly to your .vscode/settings.json:

{
  "mcp": {
    "servers": {
      "truesight": {
        "url": "https://api.truesight.goodeyelabs.com/mcp/",
        "headers": {
          "Authorization": "Bearer YOUR_API_KEY_HERE"
        }
      }
    }
  }
}

Replace YOUR_API_KEY_HERE with your actual platform API key.

Cursor

Click the button to install with one click, then replace YOUR_API_KEY_HERE with your platform API key in Cursor's MCP settings (~/.cursor/mcp.json).

Alternatively, add it manually to your project's .cursor/mcp.json:

{
  "mcpServers": {
    "truesight": {
      "url": "https://api.truesight.goodeyelabs.com/mcp/",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY_HERE"
      }
    }
  }
}

Replace YOUR_API_KEY_HERE with your actual platform API key and restart Cursor.

Claude Code

Add the server with one command, replacing YOUR_API_KEY_HERE with your platform API key:

claude mcp add --transport http truesight \
  https://api.truesight.goodeyelabs.com/mcp/ \
  --header "Authorization: Bearer YOUR_API_KEY_HERE"

This adds the server to your user-level config (available in all projects). To scope it to a specific project instead, add --scope project before the server name — this writes the config to .mcp.json in your project root.

Claude Desktop

1. Open Claude
2. Go to Settings (gear icon) > Developer > Edit Config
3. This opens your claude_desktop_config.json file. Add the Truesight server to the mcpServers object:

{
  "mcpServers": {
    "truesight": {
      "url": "https://api.truesight.goodeyelabs.com/mcp/",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY_HERE"
      }
    }
  }
}

4. Replace YOUR_API_KEY_HERE with your actual platform API key
5. Save the file and restart Claude
6. You should see Truesight listed as an available tool provider

Windsurf

1. Open Windsurf and press Cmd/Ctrl + , to open Settings
2. Search for MCP or navigate to Cascade > MCP Servers > View raw config
3. Add the Truesight server to the mcpServers object:

{
  "mcpServers": {
    "truesight": {
      "serverUrl": "https://api.truesight.goodeyelabs.com/mcp/",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY_HERE"
      },
      "disabled": false
    }
  }
}

4. Replace YOUR_API_KEY_HERE with your actual platform API key
5. Save and click Refresh (or restart Windsurf)

Workflow prompts

The Truesight MCP exposes two built-in prompts for clients that support the MCP Prompts primitive (such as Claude Desktop). These provide step-by-step workflow guidance without requiring any additional setup.

For clients that don't support MCP Prompts, use the companion agent skills instead. See Agent Skills.

Available tools

Each tool requires a specific scope on your API key. If the key doesn't have the required scope, the tool will return an error.

Pagination

All list tools accept optional page and page_size parameters. Responses include total, page, page_size, and total_pages fields alongside the items array. Defaults vary by tool:

Idempotency

The following mutating tools accept an optional idempotency_key parameter for safe retries: create_dataset, upload_dataset, deploy_live_evaluation, create_and_deploy_evaluation, and add_reviewed_items_to_dataset. If you provide the same idempotency key within a 10-minute window, the server returns the cached response instead of repeating the operation.

Agent skills

For clients that support agent skills (Claude Code, Cursor, and others), companion skills are available that give your AI assistant step-by-step workflow guidance for all Truesight MCP workflows.

Install both skills with:

# truesight-mcp skill: covers scoring inputs, error analysis, and the review loop
curl -fsSL https://raw.githubusercontent.com/Goodeye-Labs/truesight-mcp-skills/main/skills/truesight-mcp/SKILL.md \
  -o .claude/skills/truesight-mcp/SKILL.md --create-dirs

# create-evaluation skill: covers scoping and deploying new live evaluations
curl -fsSL https://raw.githubusercontent.com/Goodeye-Labs/truesight-mcp-skills/main/skills/create-evaluation/SKILL.md \
  -o .claude/skills/create-evaluation/SKILL.md --create-dirs

Or install globally (available in all projects):

curl -fsSL https://raw.githubusercontent.com/Goodeye-Labs/truesight-mcp-skills/main/skills/truesight-mcp/SKILL.md \
  -o ~/.claude/skills/truesight-mcp/SKILL.md --create-dirs

curl -fsSL https://raw.githubusercontent.com/Goodeye-Labs/truesight-mcp-skills/main/skills/create-evaluation/SKILL.md \
  -o ~/.claude/skills/create-evaluation/SKILL.md --create-dirs

The skills are hosted at github.com/Goodeye-Labs/truesight-mcp-skills.

Example interactions

Once connected, you can ask your AI assistant things like:

• "List my Truesight datasets"
• "Show me the rows in dataset ds_abc123"
• "Create a dataset from this CSV data and set up an evaluation for response quality"
• "Run an evaluation on this text using my customer_support live evaluation"
• "What are the latest evaluation results?"
• "Flag evaluation run run_abc123 for review"
• "Analyze the errors in my dataset and suggest categories"
• "Consolidate the similar error categories in my dataset"

The assistant will use the appropriate Truesight MCP tools to fulfill your request.

Troubleshooting

"Authentication required" or 401 errors

• Verify your API key is correct and hasn't been revoked
• Check that the key hasn't expired
• Make sure the Authorization header is formatted correctly: Bearer ts_pat_...

"Missing required scope" errors

• The tool requires a scope your key doesn't have
• Create a new key with the needed scopes, or revoke and recreate with additional scopes

Tools not appearing

• Restart your MCP client after adding the configuration
• Verify the MCP endpoint URL is correct: https://api.truesight.goodeyelabs.com/mcp/
• In Windsurf, make sure the server is not set to "disabled": true
• In VS Code, verify GitHub Copilot agent mode is enabled (requires VS Code 1.99+)
• Check your MCP client's logs for connection errors

"Subscription inactive" or 402 errors

• Your organization's subscription may need attention
• Check your billing status in the Truesight web app

Security notes

• MCP connections use the same authentication and access control as the REST API
• Your AI assistant can only access resources you have permission to view or modify
• Scope restrictions apply: the assistant can only use tools covered by the key's scopes
• Revoke the key from Settings to immediately disconnect all MCP clients using it