tldraw and AI Agents: A Visual Approach to Coding

Agents on the canvas describes a way of using an infinite whiteboard as a shared workspace where humans and coding agents can see, interpret, and modify the same visual context. Instead of keeping AI inside a chat box, tools such as agent-first development platforms and canvas systems like tldraw tu

MMichael SwindellJune 15, 202615 min read

Agents on the canvas describes a way of using an infinite whiteboard as a shared workspace where humans and coding agents can see, interpret, and modify the same visual context. Instead of keeping AI inside a chat box, tools such as agent-first development platforms and canvas systems like tldraw turn diagrams, annotations, wireframes, and UI sketches into actionable inputs. That matters because many software tasks are spatial, iterative, and collaborative. A canvas can hold structure that text alone often loses, including layout, grouping, revision marks, and parallel work. For product and engineering teams, this creates a more interactive loop between intent, implementation, and review.

At a Glance

How the infinite canvas is becoming an AI workspace

The canvas works best when AI needs visual context, iterative markup, and shared collaboration.

● Core idea: A canvas lets agents read drawings, annotations, layout, and relationships that are awkward to express in plain text.

● Early win: Visual-to-code workflows can turn sketches and screenshots into working prototypes, then refine them through markup.

● Big shift: Agents become more useful when they can both understand the canvas and act on it directly.

● Team angle: Multi-user, multi-agent work benefits from a shared visual workspace rather than isolated chat threads.

● Practical takeaway: Canvas-native AI is most compelling for prototyping, wireframing, workflows, and code-linked diagrams.

The most interesting part is not that AI can draw, but that visual context can become an editable interface for planning and execution.

What does it mean to put agents on a canvas?

Putting agents on a canvas means giving them access to a shared visual workspace where they can read shapes, annotations, layouts, and relationships, then make changes inside that same space. The canvas stops being just a picture and becomes a live interface for planning, prototyping, and collaboration.

Most AI tooling still assumes the main interface is chat. That works for linear tasks, but software work is rarely linear. Product flows, UI concepts, architecture, and revision notes are often easier to understand spatially.

In a canvas-based setup, humans can sketch, circle, label, move, and group things. Agents can interpret that context, generate outputs from it, and sometimes manipulate the same objects directly. This creates a tighter loop than copying screenshots into a prompt and explaining what changed.

That distinction matters. A visual workspace preserves intent in ways text often strips away.

Why is a canvas better than chat for some AI workflows?

A canvas is better than chat when the task depends on layout, grouping, iteration, or multiple parallel ideas. Chat is one-dimensional, while a canvas lets people and agents work across space, compare alternatives, and use markup as part of the instruction itself.

There are several cases where this matters:

UI design and prototyping: position and hierarchy are easier to show than describe.
Annotation-driven edits: you can draw directly on top of an output to request changes.
Branching workflows: multiple prompts or subflows can exist side by side.
Collaboration: many people can inspect and adjust the same context at once.

This maps to what engineering teams already know. Whiteboards remain useful because they externalize structure. The same logic applies to coding agents.

The broader trend also supports this. The Stack Overflow Developer Survey shows AI tools are widely used by developers, but adoption alone does not solve workflow design. Interface matters. A canvas is one answer when text-only interaction becomes a bottleneck.

How can sketches and annotations become working software?

Sketches and annotations can become working software when a model treats the visual input as specification material. A rough drawing, screenshot, or annotated mockup can be interpreted as a UI plan, then converted into code or markup that behaves like the sketched design.

One of the clearest examples is visual prototyping. A user can describe a simple interface with text, but they can also draw a layout and ask for that sketch to become a functioning prototype. The model reads the image, infers the intended UI, and generates an interactive result.

The more interesting step comes after generation. Instead of rewriting the prompt, you can mark up the output itself. Draw over a button, label a color change, or indicate where a control should move. That annotation becomes the next instruction.

This creates a feedback loop that is often faster than text refinement alone:

Create a rough visual idea.
Generate a working prototype.
Annotate the prototype directly.
Regenerate based on those edits.

For non-engineers, this can lower the barrier to creating technical artifacts. For engineers, it speeds up the transition from concept to something testable.

What kinds of tasks work especially well on a canvas?

Canvas-based AI works especially well for tasks that are spatial, compositional, or iterative. That includes wireframes, workflow graphs, concept exploration, diagramming, and collaborative planning where relationships matter as much as the raw content.

Based on the examples shown, the strongest use cases include:

UI wireframes that agents can extend with more screens or states.
Workflow graphs where each node represents a reusable prompt step.
Visual brainstorming where text, sketches, and references mix in one space.
Diagram-to-code loops where visuals explain how software works, then feed changes back into implementation.
Collaborative editing where multiple people and agents interact in the same document.

A good rule is simple: if describing the task in pure text feels tedious, the canvas may be the better interface.

How do agents actually act on the canvas instead of just reading it?

Agents act on the canvas by using the same underlying objects and operations that a human user can manipulate. Instead of returning a paragraph about what should change, they create, move, edit, and review shapes directly inside the workspace.

This is a meaningful jump from image understanding. Reading a sketch is useful, but acting on a live document is more powerful.

In practice, this means an agent can:

draw new elements,
modify existing items,
rearrange layouts,
add screens to wireframes,
review its own work before the user accepts it.

That review step matters because the interaction resembles an agent loop rather than a one-shot completion. The system can inspect the result, judge whether it matches the request, and present the change for approval.

For engineering teams, this starts to look less like “AI assistant” and more like a visual contributor.

Give agents shared project context, not isolated prompts

If your team is experimenting with canvas-based agents, a control plane like Levr helps connect those agents to issues, tests, review gates, and shared memory.

See the agent workflow →

What changes when multiple agents share the same workspace?

When multiple agents share a canvas, the interface becomes a coordination surface rather than a single assistant panel. Agents can be visible, stateful, and assigned work in parallel, which makes teamwork more inspectable than hidden background orchestration.

That is the idea behind spatialized agents on the canvas. Instead of one sidebar assistant, multiple agent instances appear in the document itself. They can be configured, prompted, and observed as they work.

The useful part is not the character design. It is the orchestration model:

agents can exist simultaneously.
one can coordinate while others execute.
their states can be visible.
users can inspect how tasks are delegated.

That becomes interesting for multi-step generation, group editing, or tasks where parallel work helps. It also aligns with a growing shift toward multi-agent workflows in software teams.

How does collaboration work when humans and agents are both present?

Human and agent collaboration works best when both sides operate in the same visible context. People can sketch, annotate, and redirect work, while agents generate, refine, or transform those artifacts without requiring everything to be re-explained in chat.

The canvas is naturally collaborative. Multiple users can join the same document, see each other’s cursors, and modify the same content. Extending that to agents creates a mixed workspace where human-human and human-agent collaboration happen together.

This is especially useful when a team needs to:

Co-design a flow.
Turn that flow into wireframes.
Clean up the resulting output.
Review revisions in place.

In a normal AI chat tool, that context can fragment into separate threads. In a shared canvas, the state of the work remains visible.

For engineering management, this visibility is not just convenient. It is operational. The GitHub Octoverse reporting consistently highlights how collaboration patterns shape delivery quality. As coding agents become part of the workflow, teams need collaboration surfaces that preserve shared context instead of scattering it.

How do canvas workflows connect back to code?

Canvas workflows connect back to code when the visual document becomes a bidirectional interface. You can ask an agent to generate a diagram from code, then modify the diagram and request code changes that reflect the new visual intent.

This is one of the more promising directions because it moves beyond mockups. The canvas becomes a model of the system, not just a design artifact.

Examples include:

Creating a quick architecture diagram from an existing codebase.
Editing the diagram to reflect a desired change.
Asking the agent to update implementation accordingly.

That only works if the system can interpret the document structurally and has some access to code or runtime behavior. This is where Model Context Protocol, or MCP, enters the picture. It provides a way for tools and agents to access external context and capabilities more systematically.

One practical way to handle this in Levr is to pair visual planning with a shared control plane. Teams can define intent, let connected agents pick up work, run automated verification, and keep human review as the final gate through Levr’s workflow.

Why are runtime APIs and local execution such a big deal here?

Runtime APIs and local execution matter because they let agents do more than call predefined tools. When an agent can access the editor programmatically, it can script behavior, generate interactions, and create dynamic canvas experiences that were not hardcoded in advance.

This is a subtle but important architectural shift. Traditional tool use says, “Here are the functions you may call.” A runtime API says, “Here is a programmable environment.”

In the examples discussed, that made it possible for an agent to:

Program interactions on top of the canvas,
Make drawn controls behave dynamically,
Inject code that changes how objects respond.

The result can feel surprising or messy, but it expands what the system can produce. Instead of just generating content, the agent starts generating behavior.

That said, this flexibility comes with tradeoffs. Local execution and code generation can be powerful, but they also raise questions about speed, safety, observability, and reproducibility.

What are the main limitations of AI on the canvas today?

AI on the canvas is compelling, but it is still limited by model reliability, latency, and inconsistent spatial reasoning. The interface can be excellent while the underlying agent still struggles with precise edits, coordination, or complex visual changes.

Several limitations were clear:

Spatial mistakes: agents may misplace or misinterpret elements.
Variable consistency: similar prompts can produce uneven results.
Latency: richer loops and local scripting can be slow.
Overreach: generated code or behavior may work in surprising but fragile ways.
Coordination edge cases: multi-agent flows are still easy to derail.

That does not make the approach weak. It just means the best implementations narrow the use case, tighten the feedback loop, and add approval gates.

How should engineering teams apply this without turning it into a novelty?

Engineering teams should apply canvas-based AI to narrow, high-friction workflows where visual context already matters. Start with one repeated task, keep the review loop explicit, and connect outputs to a real delivery system instead of treating the canvas as a toy demo surface.

A practical rollout looks like this:

Pick a constrained use case, such as wireframe expansion, architecture diagrams, or annotated UI iteration.
Use the canvas as source context, not just decoration.
Require acceptance review before changes become final.
Tie work to delivery artifacts, such as issues, tests, and implementation tasks.
Measure where it actually saves time.

That last point matters. Visual AI is easiest to demo in broad, playful examples. Real value usually appears when the workflow is narrower and quality expectations are clearer.

Teams using Levr’s project and testing features typically handle this by turning emerging agent output into tracked work with shared context, automated verification, and human approval rather than leaving it as an isolated experiment.

Technical Deep Dive FAQ

What is an infinite canvas in AI tooling?

An infinite canvas is a zoomable visual workspace where users can place drawings, text, embeds, diagrams, and interface elements without the fixed boundaries of a page. In AI tooling, it becomes more than a whiteboard. It serves as structured context that agents can inspect and sometimes modify. The benefit is that layout, proximity, grouping, and annotation become part of the instruction. That is hard to preserve in linear chat, especially for design, planning, and system modeling work.

What does “agents on the canvas” actually mean?

It means agents are treated as active participants in the visual workspace rather than remote assistants hidden behind a prompt box. They can read the canvas, generate new objects on it, revise existing elements, and sometimes coordinate with other agents in the same document. The key idea is shared state. Humans and agents operate on the same substrate, which makes instructions, edits, and review more transparent than switching between a whiteboard and a separate AI chat window.

How do annotations improve prompt quality for visual generation?

Annotations reduce ambiguity by turning feedback into visible markup rather than descriptive prose. Instead of writing a long explanation about which button should move or which panel should change color, you draw on the artifact itself. The model sees both the original output and the visual correction in one input. This often makes iterative design faster, especially for spatial changes. It also helps teams communicate intent without translating every design adjustment into a carefully worded text instruction.

Can a canvas-based agent generate real UI, or only mockups?

A canvas-based agent can generate real UI when the system converts visual input into code or markup rather than a flat image. The examples discussed included working prototypes created from sketches and screenshots, then refined through additional visual edits. Whether the output is production-ready depends on the implementation and model quality. In most cases, the result is best treated as a fast prototype or starting point. Engineering teams still need testing, review, and integration steps before relying on it in shipped software.

How do multiple agents coordinate inside one canvas document?

Multi-agent coordination usually requires some combination of shared state, role assignment, and visible task execution. One agent may act as a coordinator while others handle specific subtasks. In a canvas environment, that coordination can be spatialized so users can see where agents are working and what state they are in. This improves inspectability, but it does not automatically solve routing or memory. For more durable multi-agent workflows, teams typically need an infrastructure layer that preserves shared context across runs and tools.

What role does MCP play in canvas-based agent workflows?

MCP, or Model Context Protocol, provides a structured way for agents to access tools, data, and external context. In a canvas workflow, that can help an agent move between the visual document and other environments such as codebases, local files, or application state. It is useful when you want a diagram to reflect implementation details, or when edits on the canvas should drive changes elsewhere. MCP does not replace the canvas. It helps connect the canvas to the broader system the agent needs to understand.

Why do runtime APIs matter more than simple tool calls?

Simple tool calls limit an agent to predefined actions. A runtime API exposes a programmable environment, which gives the agent much more freedom. In the canvas context, that means it can select objects, create behavior, manipulate relationships, and sometimes script interactivity directly on top of the document. This expands what the agent can build, but it also increases complexity and risk. More freedom means more surprising outputs, so teams need stronger observability and approval boundaries when agents can effectively program the interface.

What are the biggest failure modes for agents on the canvas?

The main failure modes are weak spatial reasoning, inconsistent edits, slow iteration loops, and brittle generated behavior. An agent may understand a rough concept but still miss precise placement or misread visual cues. Multi-agent setups add coordination failures, especially when roles or boundaries are not clear. Generated code can also work in unexpected ways that are difficult to maintain. The most reliable implementations narrow the problem, keep humans in the approval loop, and connect agent work to tests and observable delivery steps.

How do you operationalize canvas-based AI work in an engineering team?

The safest approach is to treat the canvas as an upstream planning and iteration layer, then route accepted work into your normal delivery system. A strong pattern is: define intent visually, let agents generate or revise artifacts, convert accepted changes into tracked issues, run automated verification, and keep human review before merge or release. Agent-first platforms such as Levr are useful here because they provide a shared control plane for issues, test suites, and agent observability instead of leaving the work stranded in disconnected tools.

Key Takeaways

Where canvas-native AI is genuinely useful

The value shows up when visual context is part of the work, not just decoration around it.

● Use the right interface: Chat is fine for linear tasks, but spatial work often needs a canvas.

● Annotations matter: Drawing on top of outputs can be a faster and clearer revision method than rewriting prompts.

● Direct manipulation is the leap: Agents become more useful when they can edit live canvas objects, not only interpret screenshots.

● Multi-agent setups need structure: Visibility helps, but coordination still depends on shared memory, routing, and review.

● Narrow beats broad: The strongest production use cases are constrained workflows like wireframing, diagramming, and code-linked visual planning.

The future of agentic development may involve more visual workspaces, but the winners will be the teams that connect those spaces to real delivery systems.