Multimedia search
Multimedia search uses images, video, audio and other multimedia formats as a search input to perform vector similarity search via the near_media operator.
For image-only search using near_image, see the dedicated Image search page. The near_media operator shown here provides a unified interface for multiple media types.
How to configure Weaviate to use multimedia search?
Configure multimedia search
To use images, video, or audio as search inputs, configure a multi-modal vectorizer integration that supports these media types for your collection.
For example, Google's multi2vec-google with the gemini-embedding-2 model supports image, video, and audio inputs. See the model provider integrations page for available options.
Collection configuration
The collection must be configured with the appropriate media fields. For example:
from weaviate.classes.config import Configure, Property, DataType
client.collections.delete("MediaExample")
client.collections.create(
name="MediaExample",
properties=[
Property(name="title", data_type=DataType.TEXT),
Property(name="mediaType", data_type=DataType.TEXT),
],
vector_config=Configure.Vectors.multi2vec_google_gemini(
model="gemini-embedding-2",
text_fields=["title"],
# video_fields=["video"],
# image_fields=["image"],
),
)
By local file path
Use the near_media operator to search by providing a file path to a video.
API docsIf a snippet doesn't work or you have feedback, please open a GitHub issue.
from pathlib import Path
from weaviate.classes.query import NearMediaType
collection = client.collections.use("MediaExample")
response = collection.query.near_media(
media=Path("./videos/dog.mp4"),
media_type=NearMediaType.VIDEO,
return_properties=["title", "mediaType"],
limit=5,
)
for obj in response.objects:
print(obj.properties)
By base64 representation
You can also provide a base64-encoded video string:
API docsIf a snippet doesn't work or you have feedback, please open a GitHub issue.
import base64
from weaviate.classes.query import NearMediaType
with open("./videos/butterfly.mp4", "rb") as f:
video_base64 = base64.b64encode(f.read()).decode("utf-8")
collection = client.collections.use("MediaExample")
response = collection.query.near_media(
media=video_base64,
media_type=NearMediaType.VIDEO,
return_properties=["title", "mediaType"],
limit=5,
)
for obj in response.objects:
print(obj.properties)
Set a maximum distance
Set a maximum distance to filter results and return the distance metadata:
API docsIf a snippet doesn't work or you have feedback, please open a GitHub issue.
import base64
from weaviate.classes.query import MetadataQuery, NearMediaType
with open("./videos/cat.mp4", "rb") as f:
video_base64 = base64.b64encode(f.read()).decode("utf-8")
collection = client.collections.use("MediaExample")
response = collection.query.near_media(
media=video_base64,
media_type=NearMediaType.VIDEO,
distance=0.8, # Maximum accepted distance
return_metadata=MetadataQuery(distance=True),
return_properties=["title", "mediaType"],
limit=5,
)
for obj in response.objects:
print(f"{obj.properties} - Distance: {obj.metadata.distance}")
With a filter
Combine near media searches with filters to narrow results:
API docsIf a snippet doesn't work or you have feedback, please open a GitHub issue.
import base64
from weaviate.classes.query import NearMediaType
from weaviate.classes.query import Filter
with open("./videos/dog.mp4", "rb") as f:
video_base64 = base64.b64encode(f.read()).decode("utf-8")
collection = client.collections.use("MediaExample")
response = collection.query.near_media(
media=video_base64,
media_type=NearMediaType.VIDEO,
filters=Filter.by_property("mediaType").equal("animals"),
return_properties=["title", "mediaType"],
limit=5,
)
for obj in response.objects:
print(obj.properties)
Other media types
The near_media operator supports other media types such as audio. Set the media_type parameter to the appropriate NearMediaType value:
API docsIf a snippet doesn't work or you have feedback, please open a GitHub issue.
import base64
from weaviate.classes.query import NearMediaType
collection = client.collections.use("MediaExample")
response = collection.query.near_media(
media=Path("./audio/dog.wav"),
media_type=NearMediaType.AUDIO,
return_properties=["title", "mediaType"],
limit=5,
)
for obj in response.objects:
print(obj.properties)
Supported media types
The NearMediaType enum supports the following media types. Available types depend on the vectorizer module and model used.
| Media type | Enum value | Description |
|---|---|---|
| Audio | NearMediaType.AUDIO | Audio files (e.g., .wav, .mp3) |
| Image | NearMediaType.IMAGE | Image files (e.g., .jpg, .png). See also Image search. |
| Video | NearMediaType.VIDEO | Video files (e.g., .mp4, .avi) |
| Depth | NearMediaType.DEPTH | Depth map data |
| Thermal | NearMediaType.THERMAL | Thermal image data |
| IMU | NearMediaType.IMU | Inertial measurement unit data |
Not all models support all media types. For example, gemini-embedding-2 supports image, video, and audio. Check your model provider's documentation for supported modalities.
Further resources
- How-to: Query & Search - Image search
- How-to: Query & Search - Vector similarity search
- Model provider integrations - Google multimodal embeddings
Questions and feedback
Have a question or feedback? Here's how to reach us.