CanvasAgent: Enabling Complex Image Creation and Editing via Visual Tool Orchestration

The Core Finding

Complex image creation—combining generation, segmentation, editing, and compositing—requires an agent that can chain multiple visual tools together intelligently. Existing multimodal agents excel at perception tasks but struggle with generative workflows where tools actively transform images rather than analyze them. The authors introduce CanvasAgent, a multimodal agent trained to orchestrate heterogeneous visual tools, paired with CanvasCraft, a dataset of 140K annotated execution trajectories showing how to use these tools in sequence to complete real image editing tasks.

The Dataset and Training Approach

The key bottleneck was supervision. Existing multimodal datasets focus on perception (recognizing objects, answering questions) rather than teaching agents how to edit. CanvasCraft addresses this by collecting 140K fully annotated trajectories—step-by-step walkthroughs of agents using tools like image generation, object detection, segmentation masks, editing operations, and enhancement filters to fulfill complex user requests. The dataset also includes 10K reward specifications for reinforcement learning, enabling the agent to optimize beyond simple imitation.

The training pipeline has two stages:

1. Supervised fine-tuning (SFT): CanvasAgent learns to produce executable reasoning-action sequences by imitating the demonstration trajectories. This gives the agent a foundation in which tools to invoke and when.

2. Reinforcement learning (GRPO): The agent is then optimized using a hybrid reward combining outcome-level signals (Did the final image meet the user's goal?) and process-level signals (Were intermediate reasoning steps sound?). This teaches the agent not just to produce images, but to do so through sensible reasoning about visual state.

Why This Matters

Visual creation is fundamentally different from visual understanding. When an agent reads text in an image, it can reason passively. When an agent must remove an object from a photo, synthesize a new one, and blend it seamlessly, it must act on the world and react to the results. Existing agents lack the supervision and training paradigm for this. CanvasAgent closes that gap by:

• Teaching tool orchestration: The agent learns that some tasks require a sequence—first detect an object, then generate a replacement, then composite it—and adapts as intermediate results change.
• Grounding in realistic workflows: The 140K trajectories come from actual complex editing tasks, not synthetic or simplified scenarios.
• Hybrid reward optimization: Outcome rewards alone can succeed by luck; process rewards teach the agent why sequences work, making behavior more generalizable.

The evaluation spans both final image quality and trajectory behavior, indicating that the authors measure not just whether the output looks good, but whether the agent's decision-making process was sound. This two-axis evaluation is crucial for safety and interpretability in generative systems.

CanvasAgent represents a shift in multimodal agent design: from reasoning engines that inspect images to manipulation engines that transform them. The scale of CanvasCraft and the dual-training approach make it a significant step toward agents that can handle real-world creative workflows.