Skip to content

Conversation

@b41sh
Copy link
Member

@b41sh b41sh commented Nov 20, 2025

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

This PR introduces enhancements to the REFRESH VIRTUAL COLUMN command, providing user three new optional clauses to control over the refresh process and crucial mechanisms to prevent Out-Of-Memory (OOM) errors when dealing with large datasets.

  1. WHERE segment_location = '...' | block_location = '...' Clause: Allow user to specify a precise segment_location or block_location to refresh.
  2. LIMIT <number> Clause: Controls the maximum number of blocks to be processed in a single REFRESH command, ensuring that memory consumption remains within acceptable limits.
  3. OVERWRITE Keyword: Forces the re-computation and re-storage of virtual column data for all targeted blocks, regardless of their current state.

for example

-- Refresh up to 10 blocks for the virtual column on table 'test'.
-- This will process blocks that need refreshing (e.g., new data, or data where virtual column is null).
REFRESH VIRTUAL COLUMN ON test LIMIT 10;

-- Refresh up to 10 blocks for the virtual column on table 'test',
-- forcing re-computation and overwriting existing virtual column data.
REFRESH VIRTUAL COLUMN ON test LIMIT 10 OVERWRITE;

-- Refresh only the specified segment for the virtual column on table 'test',
-- processing only 1 block within that segment.
REFRESH VIRTUAL COLUMN ON test WHERE segment_location='1/72857/_sg/h019aa99aba907c81a5af5b8242c2c955_v4.mpk' LIMIT 1;

-- Refresh a specific block, forcing overwrite.
REFRESH VIRTUAL COLUMN ON my_table WHERE block_location='1/72857/_b/h019aa9976b4e7027bac9c37c5cd407cf_v2.parquet' OVERWRITE;
  • fixes: #[Link the issue here]

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions bot added the pr-chore this PR only has small changes that no need to record, like coding styles. label Nov 20, 2025
@BohuTANG
Copy link
Member

@codex review

@chatgpt-codex-connector
Copy link

Codex Review: Didn't find any major issues. Nice work!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@b41sh b41sh marked this pull request as ready for review November 21, 2025 03:37
@b41sh b41sh requested review from BohuTANG and sundy-li November 21, 2025 03:37
@BohuTANG BohuTANG merged commit a3cb89a into databendlabs:main Nov 21, 2025
174 of 176 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-chore this PR only has small changes that no need to record, like coding styles.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants