Skip to content

Conversation

@Hisoka-X
Copy link
Member

@Hisoka-X Hisoka-X commented Jan 7, 2025

Purpose of this pull request

MaxComputeSink support create partition in savemode.

Does this PR introduce any user-facing change?

no

How was this patch tested?

tested in local

Check list

@Hisoka-X Hisoka-X force-pushed the maxcompute-create-partition branch from e7bf956 to af56153 Compare January 9, 2025 08:45
@Hisoka-X Hisoka-X marked this pull request as ready for review January 9, 2025 08:45
@Hisoka-X Hisoka-X changed the title [Improve] MaxComputeSink support create partition in savemode [Improve][Connector-V2] MaxComputeSink support create partition in savemode Jan 9, 2025
@Hisoka-X
Copy link
Member Author

image
image
image
image

Copy link
Contributor

@davidzollo davidzollo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@davidzollo davidzollo merged commit 0b8f9de into apache:dev Jan 12, 2025
5 checks passed
super.recreateSchema();
if (StringUtils.isNotEmpty(readonlyConfig.get(PARTITION_SPEC))) {
((MaxComputeCatalog) catalog)
.createPartition(

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the code here may be copied and pasted from the previous method, so it should be written like this
((MaxComputeCatalog) catalog)
.truncatePartition(
tablePath, new PartitionSpec(readonlyConfig.get(PARTITION_SPEC)));

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @W-dragan , after recreate schema, the old partition will be deleted too, so why we need truncate it not create it?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding of 'recreate schema' is to delete a table or partition, and then recreate the table/partition. In the context of maxcompute, it is equivalent to overwrite=true.

Otherwise, the method you wrote in MaxCompute Catalog will not be called anywhere. Or is this method written in advance for future expansion?

Copy link
Member Author

@Hisoka-X Hisoka-X Jan 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it will be invoked in

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood, there is a deviation in my understanding. Thank you for your guidance

@Hisoka-X Hisoka-X deleted the maxcompute-create-partition branch January 13, 2025 10:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants