Skip to content

[Feature][Connector-V2] Support iceberg sink #6265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Hisoka-X merged 1 commit into from
Feb 19, 2024
Merged

[Feature][Connector-V2] Support iceberg sink #6265

Hisoka-X merged 1 commit into from
Feb 19, 2024

Conversation

@sunxiaojian
Copy link
Contributor

@sunxiaojian sunxiaojian commented Jan 22, 2024 •
edited
Loading

Related issues : #6198

Check list

@sunxiaojian sunxiaojian force-pushed the iceberg-sink branch 2 times, most recently from 7338bc5 to b2569b4 Compare January 22, 2024 12:10
@sunxiaojian
Copy link
Contributor Author

sunxiaojian commented Jan 22, 2024

Related PR : #5490 / #5072

@sunxiaojian
Copy link
Contributor Author

sunxiaojian commented Jan 22, 2024

@hailin0 PTAL

@hailin0
Copy link
Member

hailin0 commented Jan 22, 2024

image

@sunxiaojian sunxiaojian force-pushed the iceberg-sink branch 8 times, most recently from ca300b8 to 09a85ee Compare January 25, 2024 04:23
@sunxiaojian
Copy link
Contributor Author

sunxiaojian commented Jan 25, 2024

image

@hailin0 done

@sunxiaojian sunxiaojian force-pushed the iceberg-sink branch 5 times, most recently from d50148a to c4a7a22 Compare January 25, 2024 09:06
@sunxiaojian
Copy link
Contributor Author

sunxiaojian commented Jan 26, 2024

@hailin0 Can you help me retry CI？

@sunxiaojian sunxiaojian force-pushed the iceberg-sink branch 2 times, most recently from 516b52c to 10be48f Compare January 27, 2024 14:15
@hailin0
Copy link
Member

hailin0 commented Jan 29, 2024

https://github.com/sunxiaojian/seatunnel/runs/20929811031

Requires you to trigger actions

@sunxiaojian

@sunxiaojian
Copy link
Contributor Author

sunxiaojian commented Jan 29, 2024

https://github.com/sunxiaojian/seatunnel/runs/20929811031

Requires you to trigger actions

@sunxiaojian

@hailin0 done

@sunxiaojian
Copy link
Contributor Author

sunxiaojian commented Jan 31, 2024

@hailin0 @ic4y PTAL

@sunxiaojian sunxiaojian mentioned this pull request Feb 6, 2024
Closed
3 tasks
hailin0
hailin0 reviewed Feb 18, 2024
View reviewed changes
...-connector-v2-e2e/connector-iceberg-e2e/src/test/resources/iceberg/mysql_cdc_to_iceberg.conf
}

source {
MySQL-CDC {
Copy link
Member

@hailin0 hailin0 Feb 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sunxiaojian
Is it possible to use fake for testing cdc

https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-clickhouse-e2e/src/test/resources/clickhouse_sink_cdc_changelog_case2.conf#L35
https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-local-e2e/src/test/resources/text/fake_to_local_file_with_multiple_table.conf

Copy link
Contributor Author

@sunxiaojian sunxiaojian Feb 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sunxiaojian Is it possible to use fake for testing cdc

https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-clickhouse-e2e/src/test/resources/clickhouse_sink_cdc_changelog_case2.conf#L35 https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-local-e2e/src/test/resources/text/fake_to_local_file_with_multiple_table.conf

@hailin0 The main purpose of using MySQL CDC is to test the upsert mode, which is closer to the actual scenario

@hailin0 hailin0 added feature New feature iceberg labels Feb 18, 2024
hailin0
hailin0 approved these changes Feb 19, 2024
View reviewed changes
Copy link
Member

@hailin0 hailin0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Hisoka-X
Hisoka-X approved these changes Feb 19, 2024
View reviewed changes
Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Hisoka-X Hisoka-X merged commit 18d3e86 into apache:dev Feb 19, 2024
@hailin0 hailin0 mentioned this pull request Feb 19, 2024
Closed
5 tasks
@hailin0 hailin0 linked an issue Mar 20, 2024 that may be closed by this pull request
[Feature][Connector-v2] Support iceberg sink #2645
Closed
3 tasks
@LitteCandy0511
Copy link

LitteCandy0511 commented Mar 26, 2024

@sunxiaojian this kind of iceberg sink writing to upsert table will generated a lot of small data files? Has any solution to sovle it ?

4chicat pushed a commit to nexr/incubator-seatunnel that referenced this pull request Apr 9, 2024
@sunxiaojian @4chicat
[Feature] Supports iceberg sink apache#6198 (apache#6265)
3a616db
chaorongzhi pushed a commit to chaorongzhi/seatunnel that referenced this pull request Aug 21, 2024
@sunxiaojian
[Feature] Supports iceberg sink apache#6198 (apache#6265)
8cf5476
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hailin0 hailin0 hailin0 approved these changes

@Hisoka-X Hisoka-X Hisoka-X approved these changes

Assignees

No one assigned

Labels

approved feature New feature iceberg reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature][Connector-v2] Support iceberg sink

4 participants

@sunxiaojian @hailin0 @LitteCandy0511 @Hisoka-X