Bug Report: Source table not found when create LookupVindex on Kubernetes cluster

### Overview of the Issue

I'm trying to do region-based sharding on Kubernetes using the Vitess operator. I'm creating a very simple cluster using only 2 vttablet (primary/replica - no shards) in a keyspace called `main`. In this keyspace, I create the `customer` table provided in the examples from the Git repository and populate it. Then I apply my sharding VSchema. However, when I try to create my lookup Vindex, I get an error on my stream: **`vttablet: rpc error: code = Unknown desc = table customer not found`**.
I don't understand — the `customer` table definitely exists in the `main` keyspace 🤔 🤷‍♂ 

### Reproduction Steps

I am using the files contained in the `examples/operator` and `examples/region_sharding` directories from version 21 of Vitess : https://github.com/vitessio/vitess/tree/release-21.0/examples

1. I am deploying the Vitess operator :

```
❯ kubectl apply -f operator.yaml
customresourcedefinition.apiextensions.k8s.io/etcdlockservers.planetscale.com created
customresourcedefinition.apiextensions.k8s.io/vitessbackups.planetscale.com created
customresourcedefinition.apiextensions.k8s.io/vitessbackupschedules.planetscale.com created
customresourcedefinition.apiextensions.k8s.io/vitessbackupstorages.planetscale.com created
customresourcedefinition.apiextensions.k8s.io/vitesscells.planetscale.com created
customresourcedefinition.apiextensions.k8s.io/vitessclusters.planetscale.com created
customresourcedefinition.apiextensions.k8s.io/vitesskeyspaces.planetscale.com created
customresourcedefinition.apiextensions.k8s.io/vitessshards.planetscale.com created
serviceaccount/vitess-operator created
role.rbac.authorization.k8s.io/vitess-operator created
rolebinding.rbac.authorization.k8s.io/vitess-operator created
priorityclass.scheduling.k8s.io/vitess created
priorityclass.scheduling.k8s.io/vitess-operator-control-plane created
deployment.apps/vitess-operator created
```

2. I am deploying my Vitess cluster with 2 vttablets :

```
❯ kubectl apply -f 101_initial_cluster.yaml
vitesscluster.planetscale.com/example created
secret/example-cluster-config created
```

_101_initial_cluster.yaml_:
```
# The following example is minimalist. The security policies
# and resource specifications are not meant to be used in production.
# Please refer to the operator documentation for recommendations on
# production settings.
apiVersion: planetscale.com/v2
kind: VitessCluster
metadata:
  name: example
spec:
  images:
    vtctld: vitess/lite:v21.0.4
    vtadmin: vitess/vtadmin:v21.0.4
    vtgate: vitess/lite:v21.0.4
    vttablet: vitess/lite:v21.0.4
    vtbackup: vitess/lite:v21.0.4
    vtorc: vitess/lite:v21.0.4
    mysqld:
      mysql80Compatible: vitess/lite:v21.0.4
    mysqldExporter: prom/mysqld-exporter:v0.11.0
  cells:
  - name: zone1
    gateway:
      authentication:
        static:
          secret:
            name: example-cluster-config
            key: users.json
      replicas: 1
      resources:
        requests:
          cpu: 100m
          memory: 256Mi
        limits:
          memory: 256Mi
  vitessDashboard:
    cells:
    - zone1
    #extraFlags:
    #  security_policy: read-only
    replicas: 1
    resources:
      limits:
        memory: 128Mi
      requests:
        cpu: 100m
        memory: 128Mi
    extraVolumes:
    - name: countries-volume
      secret:
        secretName: example-cluster-config
    extraVolumeMounts:
    - name: countries-volume
      mountPath: /mnt/lolo_volume
  vtadmin:
    rbac:
      name: example-cluster-config
      key: rbac.yaml
    cells:
      - zone1
    apiAddresses:
      - http://localhost:14001
    replicas: 1
    readOnly: false
    apiResources:
      limits:
        memory: 128Mi
      requests:
        cpu: 100m
        memory: 128Mi
    webResources:
      limits:
        memory: 128Mi
      requests:
        cpu: 100m
        memory: 128Mi
  keyspaces:
  - name: main
    durabilityPolicy: none
    turndownPolicy: Immediate
    vitessOrchestrator:
      resources:
        limits:
          memory: 128Mi
        requests:
          cpu: 100m
          memory: 128Mi
      extraFlags:
        recovery-period-block-duration: 5s
    partitionings:
    - equal:
        parts: 1
        shardTemplate:
          databaseInitScriptSecret:
            name: example-cluster-config
            key: init_db.sql
          tabletPools:
          - cell: zone1
            type: replica
            replicas: 2
            vttablet:
              extraFlags:
                db_charset: utf8mb4
              resources:
                limits:
                  memory: 256Mi
                requests:
                  cpu: 100m
                  memory: 256Mi
            mysqld:
              resources:
                limits:
                  memory: 1024Mi
                requests:
                  cpu: 100m
                  memory: 512Mi
            dataVolumeClaimTemplate:
              accessModes: ["ReadWriteOnce"]
              resources:
                requests:
                  storage: 10Gi
  updateStrategy:
    type: Immediate
---
apiVersion: v1
kind: Secret
metadata:
  name: example-cluster-config
type: Opaque
stringData:
  countries.json: |
    {
      "United States": 1,
      "Canada": 2,
      "France": 64,
      "Germany": 65,
      "China": 128,
      "Japan": 129,
      "India": 192,
      "Indonesia": 193
    }
  users.json: |
    {
      "user": [{
        "UserData": "user",
        "Password": ""
      }]
    }
  init_db.sql: |
    # This file is executed immediately after mysql_install_db,
    # to initialize a fresh data directory.

    ###############################################################################
    # Equivalent of mysql_secure_installation
    ###############################################################################
    # We need to ensure that super_read_only is disabled so that we can execute
    # these commands. Note that disabling it does NOT disable read_only.
    # We save the current value so that we only re-enable it at the end if it was
    # enabled before.
    SET @original_super_read_only=IF(@@global.super_read_only=1, 'ON', 'OFF');
    SET GLOBAL super_read_only='OFF';

    # Changes during the init db should not make it to the binlog.
    # They could potentially create errant transactions on replicas.
    SET sql_log_bin = 0;
    # Remove anonymous users & disable remote root access (only allow UNIX socket).
    DROP USER IF EXISTS ''@'%', ''@'localhost', 'root'@'%';

    # Remove test database.
    DROP DATABASE IF EXISTS test;

    ###############################################################################
    # Vitess defaults
    ###############################################################################

    # Vitess-internal database.
    CREATE DATABASE IF NOT EXISTS _vt;
    # Note that definitions of local_metadata and shard_metadata should be the same
    # as in production which is defined in go/vt/mysqlctl/metadata_tables.go.
    CREATE TABLE IF NOT EXISTS _vt.local_metadata (
      name VARCHAR(255) NOT NULL,
      value VARCHAR(255) NOT NULL,
      db_name VARBINARY(255) NOT NULL,
      PRIMARY KEY (db_name, name)
      ) ENGINE=InnoDB;
    CREATE TABLE IF NOT EXISTS _vt.shard_metadata (
      name VARCHAR(255) NOT NULL,
      value MEDIUMBLOB NOT NULL,
      db_name VARBINARY(255) NOT NULL,
      PRIMARY KEY (db_name, name)
      ) ENGINE=InnoDB;

    # Admin user with all privileges.
    CREATE USER 'vt_dba'@'localhost';
    GRANT ALL ON *.* TO 'vt_dba'@'localhost';
    GRANT GRANT OPTION ON *.* TO 'vt_dba'@'localhost';
    GRANT PROXY ON ''@'' TO 'vt_dba'@'localhost' WITH GRANT OPTION;

    # User for app traffic, with global read-write access.
    CREATE USER 'vt_app'@'localhost';
    GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, PROCESS, FILE,
      REFERENCES, INDEX, ALTER, SHOW DATABASES, CREATE TEMPORARY TABLES,
      LOCK TABLES, EXECUTE, REPLICATION CLIENT, CREATE VIEW,
      SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER
      ON *.* TO 'vt_app'@'localhost';

    # User for app debug traffic, with global read access.
    CREATE USER 'vt_appdebug'@'localhost';
    GRANT SELECT, SHOW DATABASES, PROCESS ON *.* TO 'vt_appdebug'@'localhost';

    # User for administrative operations that need to be executed as non-SUPER.
    # Same permissions as vt_app here.
    CREATE USER 'vt_allprivs'@'localhost';
    GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, PROCESS, FILE,
      REFERENCES, INDEX, ALTER, SHOW DATABASES, CREATE TEMPORARY TABLES,
      LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW,
      SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER
      ON *.* TO 'vt_allprivs'@'localhost';

    # User for slave replication connections.
    # TODO: Should we set a password on this since it allows remote connections?
    CREATE USER 'vt_repl'@'%';
    GRANT REPLICATION SLAVE ON *.* TO 'vt_repl'@'%';

    # User for Vitess filtered replication (binlog player).
    # Same permissions as vt_app.
    CREATE USER 'vt_filtered'@'localhost';
    GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, PROCESS, FILE,
      REFERENCES, INDEX, ALTER, SHOW DATABASES, CREATE TEMPORARY TABLES,
      LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW,
      SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER
      ON *.* TO 'vt_filtered'@'localhost';

    # custom sql is used to add custom scripts like creating users/passwords. We use it in our tests
    # {{custom_sql}}

    # We need to set super_read_only back to what it was before
    SET GLOBAL super_read_only=IFNULL(@original_super_read_only, 'ON');
  rbac.yaml: |
    rules:
    - resource: "*"
      actions:
        - "get"
        - "create"
        - "put"
        - "ping"
      subjects: ["*"]
      clusters: ["*"]
    - resource: "Shard"
      actions:
        - "emergency_failover_shard"
        - "planned_failover_shard"
      subjects: ["*"]
      clusters: ["*"]
```
 
3. I am deploying the following schema :

`vtctldclient ApplySchema --sql="$(cat create_main_schema.sql)" main`

_create_main_schema.sql_:
```
CREATE TABLE IF NOT EXISTS customer (
  id int NOT NULL,
  fullname varchar(256),
  nationalid varchar(256),
  country varchar(256),
  primary key(id)
  );
```

4.  I am deploying the following Vschema :

```
❯ vtctldclient ApplyVSchema --vschema="$(cat main_vschema_initial.json)" main
New VSchema object:
{
  "sharded": false,
  "vindexes": {},
  "tables": {
    "customer": {
      "type": "",
      "column_vindexes": [],
      "auto_increment": null,
      "columns": [],
      "pinned": "",
      "column_list_authoritative": false,
      "source": ""
    }
  },
  "require_explicit_routing": false,
  "foreign_key_mode": "unspecified",
  "multi_tenant_spec": null
}
If this is not what you expected, check the input data (as JSON parsing will skip unexpected fields).
```

_main_vschema_initial.json_:
```
{
    "tables": {
      "customer": {}
    }
}
```
5. I am inserting sample data into the `customer` table:

`mysql < insert_customers.sql`

_insert_customers.sql_:
```
insert into customer(id, fullname, nationalid, country) values (1, 'Philip Roth', '123-456-789', 'United States');
insert into customer(id, fullname, nationalid, country) values (2, 'Gary Shteyngart', '234-567-891', 'United States');
insert into customer(id, fullname, nationalid, country) values (3, 'Margaret Atwood', '345-678-912', 'Canada');
insert into customer(id, fullname, nationalid, country) values (4, 'Alice Munro', '456-789-123', 'Canada');
insert into customer(id, fullname, nationalid, country) values (5, 'Albert Camus', '912-345-678', 'France');
insert into customer(id, fullname, nationalid, country) values (6, 'Colette', '102-345-678', 'France');
insert into customer(id, fullname, nationalid, country) values (7, 'Hermann Hesse', '304-567-891', 'Germany');
insert into customer(id, fullname, nationalid, country) values (8, 'Cornelia Funke', '203-456-789', 'Germany');
insert into customer(id, fullname, nationalid, country) values (9, 'Cixin Liu', '789-123-456', 'China');
insert into customer(id, fullname, nationalid, country) values (10, 'Jian Ma', '891-234-567', 'China');
insert into customer(id, fullname, nationalid, country) values (11, 'Haruki Murakami', '405-678-912', 'Japan');
insert into customer(id, fullname, nationalid, country) values (12, 'Banana Yoshimoto', '506-789-123', 'Japan');
insert into customer(id, fullname, nationalid, country) values (13, 'Arundhati Roy', '567-891-234', 'India');
insert into customer(id, fullname, nationalid, country) values (14, 'Shashi Tharoor', '678-912-345', 'India');
insert into customer(id, fullname, nationalid, country) values (15, 'Andrea Hirata', '607-891-234', 'Indonesia');
insert into customer(id, fullname, nationalid, country) values (16, 'Ayu Utami', '708-912-345', 'Indonesia');
```

6. I am checking the data:

```
❯ mysql -e 'select * from main.customer'
+----+------------------+-------------+---------------+
| id | fullname         | nationalid  | country       |
+----+------------------+-------------+---------------+
|  1 | Philip Roth      | 123-456-789 | United States |
|  2 | Gary Shteyngart  | 234-567-891 | United States |
|  3 | Margaret Atwood  | 345-678-912 | Canada        |
|  4 | Alice Munro      | 456-789-123 | Canada        |
|  5 | Albert Camus     | 912-345-678 | France        |
|  6 | Colette          | 102-345-678 | France        |
|  7 | Hermann Hesse    | 304-567-891 | Germany       |
|  8 | Cornelia Funke   | 203-456-789 | Germany       |
|  9 | Cixin Liu        | 789-123-456 | China         |
| 10 | Jian Ma          | 891-234-567 | China         |
| 11 | Haruki Murakami  | 405-678-912 | Japan         |
| 12 | Banana Yoshimoto | 506-789-123 | Japan         |
| 13 | Arundhati Roy    | 567-891-234 | India         |
| 14 | Shashi Tharoor   | 678-912-345 | India         |
| 15 | Andrea Hirata    | 607-891-234 | Indonesia     |
| 16 | Ayu Utami        | 708-912-345 | Indonesia     |
+----+------------------+-------------+---------------+
```

7. I am applying my region-based sharding VSchema:

```
❯ vtctldclient ApplyVSchema --vschema="$(cat main_vschema_sharded.json)" main
New VSchema object:
{
  "sharded": true,
  "vindexes": {
    "region_vdx": {
      "type": "region_json",
      "params": {
        "region_bytes": "1",
        "region_map": "/mnt/lolo_volume/countries.json"
      },
      "owner": ""
    }
  },
  "tables": {
    "customer": {
      "type": "",
      "column_vindexes": [
        {
          "column": "",
          "name": "region_vdx",
          "columns": [
            "id",
            "country"
          ]
        }
      ],
      "auto_increment": null,
      "columns": [],
      "pinned": "",
      "column_list_authoritative": false,
      "source": ""
    }
  },
  "require_explicit_routing": false,
  "foreign_key_mode": "unspecified",
  "multi_tenant_spec": null
}
If this is not what you expected, check the input data (as JSON parsing will skip unexpected fields).
```

_main_vschema_sharded.json_:
```
{
    "sharded": true,
    "vindexes": {
      "region_vdx": {
        "type": "region_json",
        "params": {
          "region_map": "/mnt/lolo_volume/countries.json",
          "region_bytes": "1"
        }
      }
    },
    "tables": {
      "customer": {
        "column_vindexes": [
          {
            "columns": ["id", "country"],
            "name": "region_vdx"
          }
        ]
      }
    }
  }
```

8. I am creating my Lookup Vindex :

```
❯ vtctldclient LookupVindex --name customer_region_lookup --table-keyspace main create --keyspace main --type consistent_lookup_unique --table-owner customer --table-owner-columns=id --tablet-types=PRIMARY
LookupVindex customer_region_lookup created in the main keyspace and the customer_region_lookup VReplication wokflow scheduled on the main shards, use show to view progress
```

9. I am seeing an error in my stream indicating that the `customer` table cannot be found:

```
❯ vtctldclient LookupVindex show --name customer_region_lookup --table-keyspace main
{
  "workflows": [
    {
      "name": "customer_region_lookup",
      "source": {
        "keyspace": "main",
        "shards": [
          "-"
        ]
      },
      "target": {
        "keyspace": "main",
        "shards": [
          "-"
        ]
      },
      "max_v_replication_lag": "-9223372036",
      "shard_streams": {
        "-/zone1-3221216713": {
          "streams": [
            {
              "id": "1",
              "shard": "-",
              "tablet": {
                "cell": "zone1",
                "uid": 3221216713
              },
              "binlog_source": {
                "keyspace": "main",
                "shard": "-",
                "tablet_type": "UNKNOWN",
                "key_range": null,
                "tables": [],
                "filter": {
                  "rules": [
                    {
                      "match": "customer_region_lookup",
                      "filter": "select id as id, keyspace_id() as keyspace_id from customer group by id, keyspace_id",
                      "convert_enum_to_text": {},
                      "convert_charset": {},
                      "source_unique_key_columns": "",
                      "target_unique_key_columns": "",
                      "source_unique_key_target_columns": "",
                      "convert_int_to_enum": {},
                      "force_unique_key": ""
                    }
                  ],
                  "field_event_mode": "ERR_ON_MISMATCH",
                  "workflow_type": "0",
                  "workflow_name": ""
                },
                "on_ddl": "IGNORE",
                "external_mysql": "",
                "stop_after_copy": false,
                "external_cluster": "",
                "source_time_zone": "",
                "target_time_zone": ""
              },
              "position": "",
              "stop_position": "",
              "state": "Error",
              "db_name": "vt_main",
              "transaction_timestamp": {
                "seconds": "0",
                "nanoseconds": 0
              },
              "time_updated": {
                "seconds": "20250507131555",
                "nanoseconds": 0
              },
              "message": "vttablet: rpc error: code = Unknown desc = table customer not found",
              "copy_states": [
                {
                  "table": "customer_region_lookup",
                  "last_pk": "",
                  "stream_id": "1"
                }
              ],
              "logs": [],
              "log_fetch_error": "",
              "tags": [
                ""
              ],
              "rows_copied": "0",
              "throttler_status": {
                "component_throttled": "",
                "time_throttled": {
                  "seconds": "0",
                  "nanoseconds": 0
                }
              },
              "tablet_types": [
                "REPLICA",
                "PRIMARY"
              ],
              "tablet_selection_preference": "ANY",
              "cells": [
                "zone1"
              ]
            }
          ],
          "tablet_controls": [],
          "is_primary_serving": true
        }
      },
      "workflow_type": "CreateLookupIndex",
      "workflow_sub_type": "None",
      "max_v_replication_transaction_lag": "-9223372036854775808",
      "defer_secondary_keys": false,
      "options": {
        "tenant_id": "",
        "sharded_auto_increment_handling": "LEAVE",
        "shards": [],
        "config": {},
        "global_keyspace": ""
      }
    }
  ]
}
```

10. I am checking that the `vt_filtered` user has access to the `customer` table. I am connecting to a vttablet pod :

```
vitess@example-vttablet-zone1-3221216713-83b90656:/$ mysql --socket /vt/socket/mysql.sock -u vt_filtered
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 524868
Server version: 8.0.30 MySQL Community Server - GPL

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> use main:-@primary;
Database changed
mysql> select * from customer;
+----+------------------+-------------+---------------+
| id | fullname         | nationalid  | country       |
+----+------------------+-------------+---------------+
|  1 | Philip Roth      | 123-456-789 | United States |
|  2 | Gary Shteyngart  | 234-567-891 | United States |
|  3 | Margaret Atwood  | 345-678-912 | Canada        |
|  4 | Alice Munro      | 456-789-123 | Canada        |
|  5 | Albert Camus     | 912-345-678 | France        |
|  6 | Colette          | 102-345-678 | France        |
|  7 | Hermann Hesse    | 304-567-891 | Germany       |
|  8 | Cornelia Funke   | 203-456-789 | Germany       |
|  9 | Cixin Liu        | 789-123-456 | China         |
| 10 | Jian Ma          | 891-234-567 | China         |
| 11 | Haruki Murakami  | 405-678-912 | Japan         |
| 12 | Banana Yoshimoto | 506-789-123 | Japan         |
| 13 | Arundhati Roy    | 567-891-234 | India         |
| 14 | Shashi Tharoor   | 678-912-345 | India         |
| 15 | Andrea Hirata    | 607-891-234 | Indonesia     |
| 16 | Ayu Utami        | 708-912-345 | Indonesia     |
+----+------------------+-------------+---------------+
16 rows in set (0.035 sec)
```


### Binary Version

```sh
❯ vtgate --version
vtgate version Version: 21.0.4 (Git revision  branch '') built on Wed Apr  9 14:51:08 UTC 2025 by brew@Sequoia-arm64.local using go1.24.2 darwin/arm64
```

### Operating System and Environment details

```sh
Managed Kubernetes on Digital Ocean : 1.32.2-do.0
```

### Log Fragments

```sh
I0507 12:52:59.600651       1 shard_sync.go:79] Change to tablet state
I0507 12:53:04.374950       1 state_manager.go:740] Replication is healthy
I0507 13:01:04.385013       1 rpc_schema.go:51] ReloadSchema: waiting for replication position: MySQL56/2b034416-2b42-11f0-8e25-ea1a7b14f87d:1-4
I0507 13:01:04.397130       1 rpc_schema.go:57] ReloadSchema requested via RPC
I0507 13:01:04.412591       1 engine.go:559] schema engine created [customer], altered [], dropped []
I0507 13:16:00.523962       1 engine.go:291] Streaming rows for query select id, keyspace_id() from customer, lastpk: []
E0507 13:16:00.533355       1 planbuilder.go:444] table customer not found
E0507 13:16:00.533455       1 rowstreamer.go:176] table customer not found
I0507 13:16:05.629692       1 engine.go:291] Streaming rows for query select id, keyspace_id() from customer, lastpk: []
E0507 13:16:05.637105       1 planbuilder.go:444] table customer not found
E0507 13:16:05.637395       1 rowstreamer.go:176] table customer not found
I0507 13:16:10.703113       1 engine.go:291] Streaming rows for query select id, keyspace_id() from customer, lastpk: []
E0507 13:16:10.710182       1 planbuilder.go:444] table customer not found
E0507 13:16:10.710214       1 rowstreamer.go:176] table customer not found
```

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Bug Report: Source table not found when create LookupVindex on Kubernetes cluster #18247

Overview of the Issue

Reproduction Steps

Binary Version

Operating System and Environment details

Log Fragments

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Bug Report: Source table not found when create LookupVindex on Kubernetes cluster #18247

Description

Overview of the Issue

Reproduction Steps

Binary Version

Operating System and Environment details

Log Fragments

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions