Cluster resource messages

Messages on this page mean that a Weaviate node asked the operating system for a resource and was refused. The data is intact, but the node cannot open the files it needs until the limit is raised or the load is reduced. If your message is not here, the message index lists the other groups.

Not enough memory mappings

Message id

core-mem001

Raised by

Weaviate Database

Kind

Error

What it means

A shard refused to open, so its collection or tenant is unavailable for reads and writes. Nothing is corrupted.

The fix

Raise vm.max_map_count on the host and restart the node.

What you see

In the node's logs, an error ending in not enough memory mappings:

memory pressure: cannot init shard: not enough memory mappings

In the response your client gets when the failure happened inside a request you made (REST shown; gRPC and GraphQL carry the same suffix):

{"error":[{"message":"updating db: TYPE_UPDATE_TENANT: memory pressure: cannot init shard: not enough memory mappings (see https://docs.weaviate.io/e/core-mem001)"}]}

Which log entry carries it depends on the operation that hit the limit:

Log message	action field	Level	Where the phrase is
failed to load shard: memory pressure: cannot load shard: not enough memory mappings	load_shard	error	In the message
loading shard "MyTenant" failed	tenant_activation_lazy_load_shard	error	Appended to the message, or in a separate error field before v1.40
failed to reload local index	(none)	error	In a separate error field
failure while loading shard: memory pressure: cannot init shard: not enough memory mappings	replication_engine	error	In the message, while a replica is moved or copied
drop-vector: load lazy shard: memory pressure: cannot load shard: not enough memory mappings	(none)	warning	In the message, while a vector index is dropped

Search for the phrase, not the line

not enough memory mappings is the one part that appears somewhere in the entry on every version, sometimes in a sibling error field rather than the message. On versions that carry it, every one of these entries also has a docs_url field pointing here.

The failure often shows up next to replication errors such as broadcast: cannot reach enough replicas, because the shard those requests needed is the one that did not load.

Why it happens

Every shard memory-maps several files, and the Linux kernel caps how many mappings one process may hold: vm.max_map_count. Weaviate reads that cap at startup, budgets 70% of it to leave room for other processes, and refuses to open a shard that would take it past the budget. Refusing early is deliberate: exhausting the kernel limit outright fails unpredictably somewhere else in the process.

The usual trigger is scale, not a bug. Distribution defaults were chosen for ordinary workloads, and a node holding thousands of active tenants or collections passes them.

How to fix it

The limit belongs to the host kernel, so where you set it depends on how you run Weaviate.

Linux host

1. Read the current limit on the node that logged the error. Most distributions ship 65530:

   sysctl vm.max_map_count

2. Raise it. The value is a count, not an amount of memory, so a high value costs nothing on its own. 262144 is the floor other memory-mapped databases ask for; a node holding thousands of active tenants needs a value in the millions, and 4194304 leaves room to grow:

   sysctl -w vm.max_map_count=4194304

3. Make it survive a reboot:

   echo "vm.max_map_count=4194304" >> /etc/sysctl.conf

4. Restart the affected node so Weaviate reads the new limit.

Kubernetes

The setting belongs to the host kernel, not to the container, so setting it inside an unprivileged pod has no effect. Apply it on every node that can schedule a Weaviate pod, through one of:

• your node configuration or image, with vm.max_map_count=4194304 in /etc/sysctl.conf
• a DaemonSet that runs sysctl -w vm.max_map_count=4194304 on each node
• a privileged init container on the Weaviate pod that sets it before Weaviate starts

Then restart the affected pod so Weaviate reads the new limit.

Weaviate Cloud

The host configuration is not yours to change. Open a support ticket with the message text and your cluster URL.

If the limit is already high and the error keeps coming back, the node holds more shards than it was sized for: deactivate or offload tenants that are not in use, or add nodes and rebalance.

Learn more

Known issue: shard init failureResource planningTenant states and offloadingDeployment troubleshootingImprove your cluster

Questions and feedback

Have a question or feedback? Here's how to reach us.

Community Forum

Ask questions and connect with other developers on our Community forum.

Support

Weaviate Cloud user or customer? Find the right channel on the Support page.