AI Agent Misidentifies Diagram Elements When Given Raw SVG: The Rendering vs. Semantics Problem

TLDR

Returning raw SVG markup in response to AI tool calls causes LLMs to misclassify diagram components and infer connections based on visual proximity rather than actual wire topology. SVG is designed for display rendering, not machine reasoning. Returning a structured JSON payload with explicit component types and { from, to } port connection pairs eliminates AI ambiguity completely.

Returned Payload Format	Component Classification	Net Connectivity Resolution	Reliability
Raw SVG String	Guesswork based on CSS class strings	Inferred from visual spatial proximity	Unreliable (Defective)
Structured JSON Payload	Explicit (type: 'resistor')	Explicit (from: {id, port})	100% Deterministic

---

Problem statement: the AI component counting glitch

When testing an AI agent connected to a schematic editor via MCP tool calls, we asked the agent: "How many resistors are in this diagram, and what are they connected to?"

The MCP tool returned the canvas's raw svgElement.innerHTML.

The bug

1. The AI misidentified small circular wire junction dots as resistor components.
2. It completely missed resistors grouped inside nested <g> elements.
3. It hallucinated connections between components that were visually adjacent on canvas but had no connecting wires.

---

Technical failure mode: mixing display markup with structural semantics

SVG encodes display instructions rather than domain semantics:

• A resistor is represented as a collection of <path> elements.
• A connection is represented as a polyline string (d="M 10 20 L 30 40"), requiring spatial intersection math to resolve endpoint anchors.

When an LLM parses raw SVG markup, it attempts to infer structural domain semantics from display attributes. Varied CSS class names and unstandardized SVG group trees cause the inference model to fail.

---

The fix: replacing SVG dumps with semantic JSON

Replace SVG responses with a structured JSON payload describing exact domain entities:

// REFACTORED: Structured domain response for MCP tool calls
function handleReadDiagramPayload() {
  const payload = {
    schema: 'diagram-v1',
    components: getComponents().map(c => ({
      id: c.id,
      type: c.symbolType,       // Explicit: 'resistor', 'capacitor', etc.
      label: c.label,
      position: { x: c.cx, y: c.cy },
      properties: c.properties  // Explicit: { value: '10k' }
    })),
    connections: getWires().map(w => ({
      id: w.id,
      from: { id: w.startComponent, port: w.startPort },
      to:   { id: w.endComponent,   port: w.endPort }
    }))
  };
return { content: [{ type: 'text', text: JSON.stringify(payload, null, 2) }] }; }

With this structured payload:

• Component types are explicit (type: 'resistor'), eliminating class-name guesswork.
• Connection pairs are explicit (from/to), eliminating visual proximity inference.

Rule of thumb: Format AI tool responses using structured data models optimized for LLM reasoning rather than display markup (SVG, HTML) optimized for browser rendering.