What is the problem you are trying to solve?
Mimir excels at large scale petabytes inverted index search for series in the object storage, but performs suboptimal at high cardinality (despite endless tuning). I had a thought of a feature like dynamic allocation or migration of series, above certain threshold, into another storage engine or indexing method, in a way that when specific metrics label count goes over let's say 10k 50k labels, whatever configured, it'll move these into a different indexing and storage mode like in Victoria Metrics or something, that will cater to that specific use case of those series and will do this migration transparent for the user (like compactor takes care of blocks behind the scenes, same way the 'heavy' metrics will be migrated to a different mode of operation). That would make mimir universal for any use case, and it's already such a good system, so with this feature it'll become so much better.
Which solution do you envision (roughly)?
Additional indexing method and storage format, plus live migration under the hood.
Have you considered any alternatives?
Migrating from Mimir to other types of storage
Any additional context to share?
At high cardinality in a single tenant, Mimir mechanism of inverted index search is no longer optimal past some threshold and some other approach might work better, but there's no other approach from what I understand, it's the primary method of locating the series in large blobs of data in object storage, and indexes grow very large because of many labels.
How long do you think this would take to be developed?
Large (~3 months dev)
What are the documentation dependencies?
No response
Proposer?
No response
What is the problem you are trying to solve?
Mimir excels at large scale petabytes inverted index search for series in the object storage, but performs suboptimal at high cardinality (despite endless tuning). I had a thought of a feature like dynamic allocation or migration of series, above certain threshold, into another storage engine or indexing method, in a way that when specific metrics label count goes over let's say 10k 50k labels, whatever configured, it'll move these into a different indexing and storage mode like in Victoria Metrics or something, that will cater to that specific use case of those series and will do this migration transparent for the user (like compactor takes care of blocks behind the scenes, same way the 'heavy' metrics will be migrated to a different mode of operation). That would make mimir universal for any use case, and it's already such a good system, so with this feature it'll become so much better.
Which solution do you envision (roughly)?
Additional indexing method and storage format, plus live migration under the hood.
Have you considered any alternatives?
Migrating from Mimir to other types of storage
Any additional context to share?
At high cardinality in a single tenant, Mimir mechanism of inverted index search is no longer optimal past some threshold and some other approach might work better, but there's no other approach from what I understand, it's the primary method of locating the series in large blobs of data in object storage, and indexes grow very large because of many labels.
How long do you think this would take to be developed?
Large (~3 months dev)
What are the documentation dependencies?
No response
Proposer?
No response