Skip to content

Feature Request: CreateLookupVindex for multiple lookup tables #14017

@maxenglander

Description

@maxenglander

Feature Description

Currently CreateLookupVindex only lets you backfill a single lookup vindex at a time. Enhance it to backfill multiple lookup vindexes at a time.

Use Case(s)

CreateLookupVindex will create a stream from each shard to every other shard in a keyspace. This is pretty expensive to begin with, and if you try to run multiple CreateLookupVindex workflows at the same time, it is very easy to OOM the tablets.

If CreateLookupVindex behaved more like MoveTables, each shard-shard connection would move through Copying phase for each requested lookup Vindex in sequence, periodically switch to catch-up mode, and use a single BLS to keep all tables up-to-date in Running phase.

Metadata

Metadata

Assignees

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions