DocsUseful Pluginsimage_result_formatter

image_result_formatter

Enable vision models to see images from tool results

Download

What it does

When a tool returns a base64-encoded image (screenshot, generated image, etc.), this plugin:

Detects base64 images

Recognizes data URLs (data:image/png;base64,...) and plain base64 strings.

Converts to vision format

Transforms the tool result into OpenAI vision API format so the LLM can see the image visually.

Quick Start

main.py

from connectonion import Agent
from connectonion.useful_plugins import image_result_formatter

def take_screenshot(url: str) -> str:
    """Take a screenshot of a webpage."""
    # Returns base64-encoded PNG
    return capture_screenshot(url)

agent = Agent(
    "vision_assistant",
    tools=[take_screenshot],
    plugins=[image_result_formatter],
    model="gpt-4o"  # Use a vision model
)

agent.input("Take a screenshot of example.com and describe what you see")

Python REPL

Interactive

[Tool: take_screenshot("example.com")]

🖼️ Formatted 'take_screenshot' result as image

The screenshot shows a simple webpage with the heading "Example Domain"...

Without vs With Plugin

Without Plugin

The LLM receives raw base64 text:

Tool result: "iVBORw0KGgoAAAANSUhEUgAAA..."

LLM cannot interpret this as an image!

With Plugin

The LLM receives proper image format:

{type: "image_url", url: "data:image/png;..."}

LLM can see and analyze the image!

How it works

main.py

@after_tools
def _format_image_result(agent):
    trace = agent.current_session['trace'][-1]

    if trace['type'] != 'tool_execution' or trace['status'] != 'success':
        return

    result = trace['result']
    is_image, mime_type, base64_data = _is_base64_image(result)

    if not is_image:
        return

    # Find and modify the tool result message
    messages = agent.current_session['messages']
    for i, msg in enumerate(reversed(messages)):
        if msg['role'] == 'tool' and msg.get('tool_call_id') == trace['call_id']:
            # Shorten tool message (save tokens)
            msg['content'] = "Screenshot captured (image provided below)"

            # Insert user message with actual image
            messages.insert(len(messages) - i, {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Tool returned an image. See below."},
                    {"type": "image_url", "image_url": {"url": f"data:{mime_type};base64,{base64_data}"}}
                ]
            })
            break

Supported Formats

Format	Example
PNG	`data:image/png;base64,...`
JPEG	`data:image/jpeg;base64,...`
WebP	`data:image/webp;base64,...`
GIF	`data:image/gif;base64,...`
Plain base64	Long base64 strings (defaults to PNG)

Use Cases

Screenshots: Browser automation tools that capture pages
Image generation: Tools that create images (charts, diagrams)
Visual analysis: Any tool returning visual data for LLM interpretation

Events Used

Event	Handler	Purpose
`after_tools`	_format_image_result	Convert base64 to vision format

Uses after_tools (not after_each_tool) because message modification must happen after all tools complete.

Source

connectonion/useful_plugins/image_result_formatter.py

main.py

# The plugin is just a list with one event handler
image_result_formatter = [after_tools(_format_image_result)]

eval

shell_approval