Memory-Aware Auto Load/Release Mechanism for Collections

[michaelsheka]

Hi Milvus team,

I'd like to propose a new feature that could enhance how Milvus handles memory and collection management.

Background
Currently, Milvus requires users to manually load and release collections, which can become cumbersome, especially in multi-tenant or memory-constrained deployments. If a collection is not loaded, queries or inserts will fail unless the user explicitly loads it.

Proposal
Introduce a memory-bound collection management mechanism where Milvus automatically handles loading and releasing collections based on system memory constraints.

The idea is:
When a user queries or inserts documents into an unloaded collection, Milvus checks:
If there is enough memory to load the collection.
If not, it automatically releases other collections (e.g., using an LRU as strategy to free memory).
Then loads the requested collection automatically.

Would love to hear your thoughts. I'm happy to elaborate further or help contribute if there's interest in this feature.

Thanks!

[yhmo]

So far as I know, the target of the next major version Milvus v2.6.x includes these points for loading:

• Segment lazy load: lazy loading allows for a mode where data can be stored in object storage, cached locally, and fetched from object storage when the local cache is missed.
• Tiered Storage: Lazy/Partial Load for raw data, scalar index and vector index.

The goal is to make the loading process more controllable and precise. In my opinion, if all these features are done, it is possible to achieve "automatically load/release collection".

Wait @xiaofan-luan to comment.

[michaelsheka]

Wow very quick reply. Thank you @yhmo.
I think that this is a big issue, think about classic RDBMS, you don't need to load or release tables - the RDBMS do it for you.

[yhmo]

Understand.

[xiaofan-luan]

Tiered storage might be something you need.
you can load everything but they are actually lazy loaded so don't take any resources.
When search happened, milvus will just load data from remote storage and cache it in memory or disk.
The search could be dramatically slow(like hundred milliseconds or even seconds) but it should be still faster than load/release and manage collection by yourself

[nairan-deshaw]

@yhmo @xiaofan-luan

Is the tiered storage system coming with 2.6.0 release? If yes, can we expect client applications to not make any load / release calls in their application and simply search a particular collection? Also, would a one time load be required for any collection or would that also be handled automatically?

We're a bit unclear from the contents of the blog - https://milvus.io/blog/introduce-milvus-2-6-built-for-scale-designed-to-reduce-costs.md and wanted more clarity on this feature.

[yhmo]

The source code of tiered storage is ready in v2.6.0, but it is not enabled. So far, the v2.6.0 is still using the old storage machinery, users need to explicitly call load/release in their applications. QA team is still concerned about the new storage machinery. Maybe it is enabled by default in the next official release minor version, the final decision has not yet been made.

[xiaofan-luan]

2.6 GA will support the new storage V2 and the LRU load feature.

We will update the document about how to config . Milvus will manage the memory and disk usage and evict to S3 when disk/
memory is under pressure

[subhamhimself]

Yay! Thank you!