Image generation

In Ask Mode, the Query Agent can generate images as part of its response. Each image is grounded in the data the agent retrieves.

Client versions

Image generation requires weaviate-agents 1.9.0 or later in Python, and 1.8.0 or later in JavaScript/TypeScript. See the installation page.

To generate an image, pass GeneratedImage as the output format:

Python (Agents)JavaScript/TypeScript (Agents)

from weaviate.agents.query import QueryAgent
from weaviate.agents.classes import GeneratedImage

qa = QueryAgent(
    client=client, # your Weaviate cloud client
)
response = qa.ask(
    "Chart the temperature for the first four weeks of 2023",
    collections=["Weather"],
    output_format=GeneratedImage
)

response.final_answer_parsed.base64 # base64 of the image
response.final_answer_parsed.image_prompt # prompt used to generate image based on data

import { QueryAgent, GeneratedImage } from 'weaviate-agents';

const qa = new QueryAgent(client); // your Weaviate Cloud client
const response = await qa.ask(
    "Chart the temperature for the first four weeks of 2023",
    { collections: ["Weather"], outputFormat: GeneratedImage }
);

response.finalAnswerParsed.base64; // base64 of the image
response.finalAnswerParsed.image_prompt; // prompt used to generate image based on data

GeneratedImage is an object with two fields, base64 and image_prompt. base64 holds the generated PNG, and image_prompt holds the prompt the agent wrote for the image model.

Saving and displaying the image

To decode the image from its base64:

Python (Agents)JavaScript/TypeScript (Agents)

from base64 import b64decode
image_bytes = b64decode(response.final_answer_parsed.base64) # PNG

Optionally, you can display the image with Pillow:

from PIL import Image
from io import BytesIO
image = Image.open(BytesIO(image_bytes))
image.show()

const imageBytes = Buffer.from(response.finalAnswerParsed.base64, "base64"); // PNG

You can save it as a PNG file:

Python (Agents)JavaScript/TypeScript (Agents)

with open("image.png", "wb") as f:
    f.write(image_bytes)

import { writeFileSync } from 'node:fs';

writeFileSync("image.png", imageBytes);

Uploading and searching the image in Weaviate

Provided you have a valid multimodal embedding vectorizer set up on your Weaviate collection, for example, from the JinaAI multi2vec module, you can upload the base64 of the image directly to your collection.

Python (Agents)JavaScript/TypeScript (Agents)

from weaviate.classes.config import Configure, Property, DataType, Multi2VecField

collection = client.collections.create(
    "QueryAgentImageUpload",
    vector_config=Configure.Vectors.multi2vec_jinaai(
        name="image_vector",
        image_fields=[Multi2VecField(name="image")],
    ),
    properties=[
        Property(name="image", data_type=DataType.BLOB),
    ]
)

collection.data.insert(
    properties={
        "image": response.final_answer_parsed.base64,
    }
)

import weaviate from 'weaviate-client';

const collection = await client.collections.create({
    name: "QueryAgentImageUpload",
    vectorizers: weaviate.configure.vectors.multi2VecJinaAI({
        name: "image_vector",
        imageFields: [{ name: "image" }],
    }),
    properties: [
        { name: "image", dataType: weaviate.configure.dataType.BLOB },
    ],
});

await collection.data.insert({
    image: response.finalAnswerParsed.base64,
});

You can also search using the base64 directly on a collection that is set up for image searching.

Python (Agents)JavaScript/TypeScript (Agents)

search_response = collection.query.near_image(
    near_image=response.final_answer_parsed.base64,
    return_properties=["image"]
)

const searchResponse = await collection.query.nearImage(
    response.finalAnswerParsed.base64,
    { returnProperties: ["image"] }
);

See more about building an image search application in Weaviate.

Structured outputs with images

You can also use the GeneratedImage type within a structured output specification, allowing any field to return a generated image as part of a structured response. For example:

Python (Agents)JavaScript/TypeScript (Agents)

from pydantic import BaseModel, Field
from weaviate.agents.classes import QueryAgentCollectionConfig

class AdvertsResponse(BaseModel):
    speech: str = Field(description="What to say during the presentation")
    adverts: list[GeneratedImage] = Field(
        description="A list of advertisements for each of the best selling products",
        min_length = 2,
        max_length = 4 # max_length must be specified for lists of images
    )

response = qa.ask(
    "Find the best selling items in the store and then create a series of adverts for them.",
    collections=[
        QueryAgentCollectionConfig(
            name="ECommerce",
            target_vector=["name_description_brand_vector"],
        )
    ],
    output_format=AdvertsResponse,
)

import { z } from 'zod';

const AdvertsResponse = z.object({
    speech: z.string().describe("What to say during the presentation"),
    adverts: z.array(GeneratedImage)
        .min(2)
        .max(4) // max must be specified for lists of images
        .describe("A list of advertisements for each of the best selling products"),
});

const advertsResult = await qa.ask(
    "Find the best selling items in the store and then create a series of adverts for them.",
    {
        collections: [{
            name: "ECommerce",
            targetVector: ["name_description_brand_vector"],
        }],
        outputFormat: AdvertsResponse,
    }
);

See Structured outputs for more on typed responses.

You can place images anywhere in your output format, within some constraints. For example, you can nest types and add custom descriptions:

Python (Agents)JavaScript/TypeScript (Agents)

from weaviate.agents.classes import QueryAgentCollectionConfig

class Slide(BaseModel):
    speech: str = Field(description = "What to say during this particular slide")
    slide: GeneratedImage = Field(
        description = "A presentation slide detailing a single product"
    )

class PresentationResponse(BaseModel):
    highlights: str = Field(description="Overall things to highlight across the full presentation")
    slides: list[Slide] = Field(
        min_length=2, max_length=4
    )

response = qa.ask(
    "Find the best selling items in the store and then create a presentation for them.",
    collections=[
        QueryAgentCollectionConfig(
            name="ECommerce",
            target_vector=["name_description_brand_vector"],
        )
    ],
    output_format=PresentationResponse,
)

const Slide = z.object({
    speech: z.string().describe("What to say during this particular slide"),
    slide: GeneratedImage.describe("A presentation slide detailing a single product"),
});

const PresentationResponse = z.object({
    highlights: z.string().describe("Overall things to highlight across the full presentation"),
    slides: z.array(Slide).min(2).max(4),
});

const presentationResult = await qa.ask(
    "Find the best selling items in the store and then create a presentation for them.",
    {
        collections: [{
            name: "ECommerce",
            targetVector: ["name_description_brand_vector"],
        }],
        outputFormat: PresentationResponse,
    }
);

Set a description with Field(description=...) in Pydantic or .describe() in Zod. The image model receives the description of its image field, so use it for design instructions.

note

If you describe a list of images, the description applies to the list, not to each image in it.

Python (Agents)JavaScript/TypeScript (Agents)

image_field: list[GeneratedImage] = Field(description="...") describes the list as a whole. The image model does not receive that description for each image. To describe each image, use the following:

from typing import Annotated

class AnnotatedImagesResponse(BaseModel):
    image_field: list[Annotated[GeneratedImage, Field(description="<image guidance/style description here>")]] = Field(
        max_length = 4 # max_length must be specified for lists of images
    )

image_field: z.array(GeneratedImage).max(4).describe("...") describes the list as a whole. The image model does not receive that description for each image. To describe each image, use the following:

const AnnotatedImagesResponse = z.object({
    image_field: z.array(GeneratedImage.describe("<image guidance/style description here>")).max(4),
});

Union types with images

GeneratedImage is a type, so you can use it in a union. For example, the agent can return null when it finds no relevant data:

Python (Agents)JavaScript/TypeScript (Agents)

class UnionResponse(BaseModel):
    chart: GeneratedImage | None # Union[GeneratedImage, None] for Python < 3.10

response = qa.ask(
    "Chart the temperature for the first four weeks of 2024. If no data exists, return null",
    collections=["Weather"],
    output_format=UnionResponse
)

print(response.final_answer_parsed.chart is None)

const UnionResponse = z.object({
    chart: GeneratedImage.nullable(), // same as z.union([GeneratedImage, z.null()])
});

const unionResult = await qa.ask(
    "Chart the temperature for the first four weeks of 2024. If no data exists, return null",
    { collections: ["Weather"], outputFormat: UnionResponse }
);

console.log(unionResult.finalAnswerParsed.chart === null);

Customizing shape

You can set the shape of a generated image. In both clients, you set the shape on an image field in your structured output model.

Python (Agents)JavaScript/TypeScript (Agents)

Import GeneratedImageOptions and add it as an annotation to a GeneratedImage field in your structured output model.

from typing import Annotated
from weaviate.agents.classes import GeneratedImageOptions
from weaviate.agents.classes import QueryAgentCollectionConfig

class HatResponse(BaseModel):
    picture: Annotated[GeneratedImage, GeneratedImageOptions(shape="portrait")]

response = qa.ask(
    "Generate a picture of someone wearing the most expensive hat",
    collections=[
        QueryAgentCollectionConfig(
            name="ECommerce",
            target_vector=["name_description_brand_vector"],
        )
    ],
    output_format=HatResponse,
)

Import generatedImageWithOptions and call it in place of GeneratedImage on a field in your Zod schema.

import { generatedImageWithOptions } from 'weaviate-agents';

const HatResponse = z.object({
    picture: generatedImageWithOptions({ shape: "portrait" }),
});

const hatResult = await qa.ask(
    "Generate a picture of someone wearing the most expensive hat",
    {
        collections: [{
            name: "ECommerce",
            targetVector: ["name_description_brand_vector"],
        }],
        outputFormat: HatResponse,
    }
);

Shape is the only image option. The shapes and their dimensions are:

• "square": 1024×1024
• "landscape" (default): 1536×1024
• "portrait": 1024×1536

Raw JSON Schema

If you pass the output format as a raw JSON Schema (instead of a Pydantic model or Zod schema), add "X-query-agent-image": true to any object you want generated as an image. This is a custom keyword that the Query Agent recognizes. Other JSON Schema tools ignore it, so your schema stays valid. The object needs an image_prompt string property, where the agent writes the prompt for the image model. Do not declare a base64 property yourself: the server adds it to the response, holding the generated PNG as a base64 string.

You can also set "X-image-shape" on the object to "square", "landscape" (the default) or "portrait". Any description you give the field is passed to the image model as extra instructions. Image quality cannot be configured. You can add other properties next to image_prompt, such as alt_text, and the agent fills them in like any other field.

For example, your output format can be:

Python (Agents)JavaScript/TypeScript (Agents)

from weaviate.agents.classes import QueryAgentCollectionConfig

response = qa.ask(
    "Find the most expensive item in the store",
    collections=[
        QueryAgentCollectionConfig(
            name="ECommerce",
            target_vector=["name_description_brand_vector"],
        )
    ],
    output_format={
        "type": "object",
        "properties": {
            "answer": { "type": "string" },
            "image": {
                "X-query-agent-image": True,
                "X-image-shape": "square",
                "description": "A product photo on a white background",
                "type": "object",
                "properties": {
                    "image_prompt": { "type": "string" }
                }
            }
        }
    }
)

const res = await qa.ask(
    "Find the most expensive item in the store",
    {
        collections: [{
            name: "ECommerce",
            targetVector: ["name_description_brand_vector"],
        }],
        outputFormat: {
            type: "object",
            properties: {
                answer: { type: "string" },
                image: {
                    "X-query-agent-image": true,
                    "X-image-shape": "square",
                    description: "A product photo on a white background",
                    type: "object",
                    properties: {
                        image_prompt: { "type": "string" }
                    }
                },
            }
        }
    }
);

This declares two fields: answer and image, where the image field is an object with only the declared property image_prompt. The server adds base64 to the image object in the response.

Cost

Each generated image consumes 1 request, on top of the 4 requests an Ask Mode query consumes. A query that generates 2 images consumes 6 requests. See usage limits.

If an image is optional, only the images that get generated are counted. This applies to a list with a variable number of images or to a nullable field.

Constraints

• Images are generated independently. For a consistent theme among your images, add a consistent description to your image field. Try specific layout instructions, hex color codes and stylistic choices.
• Image generation adds latency. A request for several images can exceed the Python QueryAgent timeout, which defaults to 60 seconds. To raise it, pass timeout to the constructor, for example QueryAgent(client=client, timeout=120). The Weaviate client timeout does not affect this limit. The JavaScript/TypeScript client has no timeout of its own.
• A request can include at most 10 images. Requests over this limit are rejected before any image is generated.
• A list of images must set a maximum number of items. Recursive schemas that contain images are rejected.

Streaming

Image generation works with streaming in Ask Mode. The streamed tokens include the text of image_prompt, because the agent writes the image prompt as part of its answer. Images are generated after the answer is complete, so the base64 of each image arrives only in the final response.

Questions and feedback

Have a question or feedback? Here's how to reach us.

Community Forum

Ask questions and connect with other developers on our Community forum.

Support

Weaviate Cloud user or customer? Find the right channel on the Support page.