Cluster resource messages

Messages on this page mean that a Weaviate node asked the operating system for a resource and was refused. The data is intact, but the node cannot open the files it needs until the limit is raised or the load is reduced. If your message is not here, the message index lists the other error groups.

Not enough memory mappings

Message id

core-mem001

Raised by

Weaviate Database

Kind

Error

Since

v1.39.5

What it means

A shard refused to open, so its collection or tenant is unavailable for reads and writes. Nothing is corrupted.

The fix

Raise vm.max_map_count on the host and restart the node.

What you see

In the node's logs, an error ending in not enough memory mappings:

memory pressure: cannot init shard: not enough memory mappings

In the response your client gets when the failure happened inside a request you made (here, from a tenant-activation request — REST shown, gRPC and GraphQL carry the same suffix):

{"error":[{"message":"updating db: TYPE_UPDATE_TENANT: memory pressure: cannot init shard: not enough memory mappings (see https://docs.weaviate.io/e/core-mem001)"}]}

The exact log line varies by operation and version, so search the logs for not enough memory mappings, it appears in the message or in a sibling error field, depending on the entry. Where the entry has an action field, it is one of load_shard, tenant_activation_lazy_load_shard, update_tenants, update_tenants_process, or replication_engine.

The exact log lines (as printed by v1.39.5 and v1.40.0)

<tenant> and <error> stand for the tenant name and the appended error text; a row without <error> carries the error text in a sibling error field instead.

Log message	action field	Level
failed to load shard, loading the rest anyway: <error>	load_shard	error
loading shard "<tenant>" failed: <error>	tenant_activation_lazy_load_shard	error
failed to reload local index	(none)	error
error updating tenants	update_tenants	error
error updating tenants: <error>	update_tenants_process	error
apply command	(none)	error
failure while loading shard: <error> (while a replica is moved or copied)	replication_engine	error
drop-vector: load lazy shard: <error> (while a vector index is dropped)	(none)	warning

The drop-vector entry logs at warning even though the condition it reports is the same error.

The failure often shows up next to replication errors such as broadcast: cannot reach enough replicas, because the shard those requests needed is the one that did not load.

Why it happens

Every shard memory-maps several files, and the Linux kernel caps how many mappings one process may hold: vm.max_map_count. Weaviate reads that cap at startup, budgets 70% of it to leave room for other processes, and refuses to open a shard that would take it past the budget. Refusing early is deliberate: exhausting the kernel limit outright fails unpredictably somewhere else in the process.

The usual trigger is scale, not a bug. The kernel default — 65530 on most distributions — was chosen for ordinary workloads, and a node holding thousands of active tenants or collections passes it.

How to fix it

The limit belongs to the host kernel, so where you set it depends on how you run Weaviate.

Linux host

1. Read the current limit on the node that logged the error. Most distributions ship 65530:

   sysctl vm.max_map_count

2. Raise it. The value is a count, not an amount of memory, so a high value costs nothing on its own. 262144 is the minimum Elasticsearch requires; a node holding thousands of active tenants needs a value in the millions, and 4194304 leaves room to grow. Weaviate budgets 70% of the limit for itself, so 4194304 gives a node roughly 2.9 million mappings — opening a shard reserves 3 up front, and a loaded shard's real consumption grows with its memory-mapped segment files:

   sudo sysctl -w vm.max_map_count=4194304

3. Make it survive a reboot. A drop-in file is idempotent — rerunning it never stacks duplicate lines — and is what current distributions expect:

   echo "vm.max_map_count=4194304" | sudo tee /etc/sysctl.d/99-weaviate.conf
   sudo sysctl --system

4. Restart the affected node so Weaviate reads the new limit.

Docker

On Linux, containers share the host kernel, so the limit is the host's to raise: follow the Linux host steps on the machine that runs Docker, then restart the container. Setting it per container does not work — docker run --sysctl vm.max_map_count=... is refused, because Docker permits only namespaced sysctls and this one belongs to the whole host.

Docker Desktop on Mac and Windows runs containers inside a Linux VM, so the Linux steps on your own machine change the wrong kernel. Set the limit inside the VM instead.

On Windows, with the WSL 2 backend:

wsl -d docker-desktop -u root sysctl -w vm.max_map_count=4194304

To make it survive a restart, add this to %USERPROFILE%\.wslconfig:

[wsl2]
kernelCommandLine = "sysctl.vm.max_map_count=4194304"

On macOS, apply it from a privileged container. The setting lasts until Docker Desktop restarts:

docker run --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i sysctl -w vm.max_map_count=4194304

Then restart the Weaviate container so Weaviate reads the new limit.

Kubernetes

The setting belongs to the host kernel, not to the container, so setting it inside an unprivileged pod has no effect. Apply it on every node that can schedule a Weaviate pod, through one of:

• the official Helm chart, which already ships a privileged init container, enabled by default, that runs sysctl -w vm.max_map_count on the pod's node before Weaviate starts. Its default of 524288 (initContainers.sysctlInitContainer.sysctlVmMaxMapCount) is well below the value this page recommends, so set initContainers.sysctlInitContainer.sysctlVmMaxMapCount: 4194304 in your values rather than adding infrastructure. The same init container also sets vm.overcommit_memory=1, so raise its value rather than disabling it in favor of your own — it sets more than the map count
• your node configuration or image, with vm.max_map_count=4194304 in /etc/sysctl.conf
• a DaemonSet whose privileged container runs sysctl -w vm.max_map_count=4194304 on each node
• a privileged init container on the Weaviate pod that sets it before Weaviate starts

Then restart the affected pod so Weaviate reads the new limit.

Weaviate Cloud

The host configuration is not yours to change. Open a support ticket with the message text and your cluster URL.

If the limit is already high and the error keeps coming back, the node holds more shards than it was sized for: deactivate or offload tenants that are not in use, or add nodes and rebalance.

Learn more

Known issue: shard init failureResource planningTenant states and offloadingDeployment troubleshootingImprove your cluster

Questions and feedback

Have a question or feedback? Here's how to reach us.

Community Forum

Ask questions and connect with other developers on our Community forum.

Support

Weaviate Cloud user or customer? Find the right channel on the Support page.