Multimedia search
Multimedia search uses images, video, audio and other multimedia formats as a search input to perform vector similarity search via the near_media operator.
For image-only search using near_image, see the dedicated Image search page. The near_media operator shown here provides a unified interface for multiple media types.
How to configure Weaviate to use multimedia search?
Configure multimedia search
To use images, video, or audio as search inputs, configure a multi-modal vectorizer integration that supports these media types for your collection.
For example, Google's multi2vec-google with the gemini-embedding-2-preview model supports image, video, and audio inputs. See the model provider integrations page for available options.
Collection configuration
The collection must be configured with the appropriate media fields. For example:
from weaviate.classes.config import Configure, Property, DataType
client.collections.delete("MediaExample")
client.collections.create(
name="MediaExample",
properties=[
Property(name="title", data_type=DataType.TEXT),
Property(name="mediaType", data_type=DataType.TEXT),
],
vector_config=Configure.Vectors.multi2vec_google_gemini(
model="gemini-embedding-2-preview",
text_fields=["title"],
# video_fields=["video"],
# image_fields=["image"],
),
)
By local file path
Use the near_media operator to search by providing a file path to a video.
API docsIf a snippet doesn't work or you have feedback, please open a GitHub issue.
from pathlib import Path
from weaviate.classes.query import NearMediaType
collection = client.collections.use("MediaExample")
response = collection.query.near_media(
media=Path("./videos/dog.mp4"),
media_type=NearMediaType.VIDEO,
return_properties=["title", "mediaType"],
limit=5,
)
for obj in response.objects:
print(obj.properties)
By base64 representation
You can also provide a base64-encoded video string:
API docsIf a snippet doesn't work or you have feedback, please open a GitHub issue.
import base64
from weaviate.classes.query import NearMediaType
with open("./videos/butterfly.mp4", "rb") as f:
video_base64 = base64.b64encode(f.read()).decode("utf-8")
collection = client.collections.use("MediaExample")
response = collection.query.near_media(
media=video_base64,
media_type=NearMediaType.VIDEO,
return_properties=["title", "mediaType"],
limit=5,
)
for obj in response.objects:
print(obj.properties)
Set a maximum distance
Set a maximum distance to filter results and return the distance metadata:
API docsIf a snippet doesn't work or you have feedback, please open a GitHub issue.
import base64
from weaviate.classes.query import MetadataQuery, NearMediaType
with open("./videos/cat.mp4", "rb") as f:
video_base64 = base64.b64encode(f.read()).decode("utf-8")
collection = client.collections.use("MediaExample")
response = collection.query.near_media(
media=video_base64,
media_type=NearMediaType.VIDEO,
distance=0.8, # Maximum accepted distance
return_metadata=MetadataQuery(distance=True),
return_properties=["title", "mediaType"],
limit=5,
)
for obj in response.objects:
print(f"{obj.properties} - Distance: {obj.metadata.distance}")
With a filter
Combine near media searches with filters to narrow results:
API docsIf a snippet doesn't work or you have feedback, please open a GitHub issue.
import base64
from weaviate.classes.query import NearMediaType
from weaviate.classes.query import Filter
with open("./videos/dog.mp4", "rb") as f:
video_base64 = base64.b64encode(f.read()).decode("utf-8")
collection = client.collections.use("MediaExample")
response = collection.query.near_media(
media=video_base64,
media_type=NearMediaType.VIDEO,
filters=Filter.by_property("mediaType").equal("animals"),
return_properties=["title", "mediaType"],
limit=5,
)
for obj in response.objects:
print(obj.properties)
Other media types
The near_media operator supports other media types such as audio. Set the media_type parameter to the appropriate NearMediaType value:
API docsIf a snippet doesn't work or you have feedback, please open a GitHub issue.
import base64
from weaviate.classes.query import NearMediaType
collection = client.collections.use("MediaExample")
response = collection.query.near_media(
media=Path("./audio/dog.wav"),
media_type=NearMediaType.AUDIO,
return_properties=["title", "mediaType"],
limit=5,
)
for obj in response.objects:
print(obj.properties)
Supported media types
The NearMediaType enum supports the following media types. Available types depend on the vectorizer module and model used.
| Media type | Enum value | Description |
|---|---|---|
| Audio | NearMediaType.AUDIO | Audio files (e.g., .wav, .mp3) |
| Image | NearMediaType.IMAGE | Image files (e.g., .jpg, .png). See also Image search. |
| Video | NearMediaType.VIDEO | Video files (e.g., .mp4, .avi) |
| Depth | NearMediaType.DEPTH | Depth map data |
| Thermal | NearMediaType.THERMAL | Thermal image data |
| IMU | NearMediaType.IMU | Inertial measurement unit data |
Not all models support all media types. For example, gemini-embedding-2-preview supports image, video, and audio. Check your model provider's documentation for supported modalities.
Further resources
- How-to: Query & Search - Image search
- How-to: Query & Search - Vector similarity search
- Model provider integrations - Google multimodal embeddings
Questions and feedback
If you have any questions or feedback, let us know in the user forum.