[fix][connectors-v2] repeated commit cause task exceptions #9665

hawk9821 · 2025-08-05T12:07:50Z

Purpose of this pull request

#8730 has been merged into this PR.
repeated commit cause task exceptions when restore from checkpoint in paimon task.

savepoint process :

step1. serialize the state to local
step2. commit the state

restore process:

step1. deserialize the state from local file
step2. commit the state

When step2 of the savepoint process succeeds, the repeated submission of step2 in the restore job throws an exception and the task exits.

[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: java.lang.RuntimeException: File deletion conflicts detected! Give up committing.
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: Don't panic!
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: Conflicts during commits are normal and this failure is intended to resolve the conflicts.
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: Conflicts are mainly caused by the following scenarios:
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 1. Multiple jobs are writing into the same partition at the same time, or you use STATEMENT SET to execute multiple INSERT statements into the same Paimon table.
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR:    You'll probably see different base commit user and current commit user below.
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR:    You can use https://paimon.apache.org/docs/master/maintenance/dedicated-compaction#dedicated-compaction-job to support multiple writing.
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 2. You're recovering from an old savepoint, or you're creating multiple jobs from a savepoint.
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR:    The job will fail continuously in this scenario to protect metadata from corruption.
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDOUT: [8949219017735335991] 2025-08-05 11:12:18,986 INFO  org.apache.seatunnel.api.sink.multitablesink.MultiTableSinkWriter - init multi table sink writer, queue size: 1
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR:    You can either recover from the latest savepoint, or you can revert the table to the snapshot corresponding to the old savepoint.
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: Base commit user is: dd52504a-9c42-451a-a035-02e091307c3a; Current commit user is: d6f5e340-c66d-49d7-ba8d-0a6af5fe33b5
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: Base entries are:
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=ADD, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=4, level=0, fileName=data-2593dd91-d35a-4013-904d-0a0742f4191e-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@dfacb5f1, maxKey=org.apache.paimon.data.BinaryRow@6f3ed396, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=ADD, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=4, level=0, fileName=data-2593dd91-d35a-4013-904d-0a0742f4191e-1.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@ec9064de, maxKey=org.apache.paimon.data.BinaryRow@2d99e697, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=ADD, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=4, level=0, fileName=data-e6510d9e-9af5-41f3-971c-f2623b8c925a-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@2d99e697, maxKey=org.apache.paimon.data.BinaryRow@1dd4c70e, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=ADD, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=4, level=0, fileName=data-46c9959e-786c-4395-8f8e-12340381d2e9-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@58b0535c, maxKey=org.apache.paimon.data.BinaryRow@58b0535c, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=ADD, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=0, level=5, fileName=data-52776f55-19a4-4596-a1ce-9f8685394576-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@89fba125, maxKey=org.apache.paimon.data.BinaryRow@a3c9fbe7, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=ADD, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=1, level=5, fileName=data-21c2cf99-4600-410f-8074-2376842e6a85-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@4b1a2337, maxKey=org.apache.paimon.data.BinaryRow@c8b1eb4, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=ADD, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=2, level=5, fileName=data-4d4f3189-0548-4a7c-826e-8e5273a932f2-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@af534502, maxKey=org.apache.paimon.data.BinaryRow@1853dc50, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=ADD, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=3, level=5, fileName=data-09e88d85-43da-44ca-a813-00bb8e3d525f-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@b0db094a, maxKey=org.apache.paimon.data.BinaryRow@a1a8ed70, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: Changes are:
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=DELETE, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=0, level=0, fileName=data-5ae74afe-e93e-4db2-a22b-ec32724e85f4-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@89fba125, maxKey=org.apache.paimon.data.BinaryRow@89fba125, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=DELETE, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=0, level=0, fileName=data-5ae74afe-e93e-4db2-a22b-ec32724e85f4-1.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@26985dc0, maxKey=org.apache.paimon.data.BinaryRow@912ecf17, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=DELETE, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=0, level=0, fileName=data-f2c71d58-f3e5-438a-85a9-a4e007ca229a-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@c730df2e, maxKey=org.apache.paimon.data.BinaryRow@90c9c338, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=DELETE, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=0, level=0, fileName=data-e21604b9-ee26-4066-b1aa-d17be4a26f54-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@26985dc0, maxKey=org.apache.paimon.data.BinaryRow@4956887d, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=DELETE, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=0, level=0, fileName=data-2becabb0-abd5-4510-be71-fd3cd0538345-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@6163a3bf, maxKey=org.apache.paimon.data.BinaryRow@a3c9fbe7, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=ADD, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=0, level=5, fileName=data-52776f55-19a4-4596-a1ce-9f8685394576-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@89fba125, maxKey=org.apache.paimon.data.BinaryRow@a3c9fbe7, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=DELETE, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=1, level=0, fileName=data-1727e4df-7eed-4fae-b6ee-0943d96d7231-1.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@7e025d0a, maxKey=org.apache.paimon.data.BinaryRow@6e990cbf, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=DELETE, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=1, level=0, fileName=data-f93ac8f3-5ab8-4457-a5f8-10cccbbc98e3-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@4b1a2337, maxKey=org.apache.paimon.data.BinaryRow@4b1a2337, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=DELETE, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=1, level=0, fileName=data-0d92da45-2a92-40aa-ae45-8a99e9bab6ce-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@a0c0c097, maxKey=org.apache.paimon.data.BinaryRow@a0c0c097, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=DELETE, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=1, level=0, fileName=data-1727e4df-7eed-4fae-b6ee-0943d96d7231-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@7e025d0a, maxKey=org.apache.paimon.data.BinaryRow@7e025d0a, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=DELETE, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=1, level=0, fileName=data-d4b16677-8cb5-4d57-b321-dbeb8512de67-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@6e990cbf, maxKey=org.apache.paimon.data.BinaryRow@c8b1eb4, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=ADD, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=1, level=5, fileName=data-21c2cf99-4600-410f-8074-2376842e6a85-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@4b1a2337, maxKey=org.apache.paimon.data.BinaryRow@c8b1eb4, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=DELETE, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=2, level=0, fileName=data-53d2dba3-62e6-4afb-83d7-52edbb1c8deb-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@af534502, maxKey=org.apache.paimon.data.BinaryRow@47791d15, externalPath=null}
[] 2025-08-05 19:12:18,998 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=DELETE, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=2, level=0, fileName=data-53d2dba3-62e6-4afb-83d7-52edbb1c8deb-1.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@af534502, maxKey=org.apache.paimon.data.BinaryRow@e1e383e, externalPath=null}
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=DELETE, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=2, level=0, fileName=data-d749957c-8d65-43f5-8883-4d00e14f83ae-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@6a2ddf4e, maxKey=org.apache.paimon.data.BinaryRow@6a2ddf4e, externalPath=null}
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=DELETE, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=2, level=0, fileName=data-1886c467-2463-435d-a0ff-242d2a475c6c-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@b6e6e602, maxKey=org.apache.paimon.data.BinaryRow@2ea9cff, externalPath=null}
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=DELETE, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=2, level=0, fileName=data-f85e0e3a-0b6f-4919-9ca1-52b03c14d7b6-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@1853dc50, maxKey=org.apache.paimon.data.BinaryRow@1853dc50, externalPath=null}
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=ADD, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=2, level=5, fileName=data-4d4f3189-0548-4a7c-826e-8e5273a932f2-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@af534502, maxKey=org.apache.paimon.data.BinaryRow@1853dc50, externalPath=null}
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=DELETE, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=3, level=0, fileName=data-6bd42ab0-b3e8-40b4-8fef-0af4cfb9ea68-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@b0db094a, maxKey=org.apache.paimon.data.BinaryRow@b0db094a, externalPath=null}
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=DELETE, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=3, level=0, fileName=data-6bd42ab0-b3e8-40b4-8fef-0af4cfb9ea68-1.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@1909a1e4, maxKey=org.apache.paimon.data.BinaryRow@9614fcf1, externalPath=null}
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=DELETE, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=3, level=0, fileName=data-add6291e-afc9-40e9-a37d-ab9e382a87f2-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@e03ea962, maxKey=org.apache.paimon.data.BinaryRow@28a2140b, externalPath=null}
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=DELETE, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=3, level=0, fileName=data-b75e709d-3e2d-4fa8-ad74-0a5d2f1c39e4-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@fbc59803, maxKey=org.apache.paimon.data.BinaryRow@fbc59803, externalPath=null}
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=DELETE, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=3, level=0, fileName=data-0aa86945-1552-477c-808a-3b9c8eefb646-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@a1a8ed70, maxKey=org.apache.paimon.data.BinaryRow@a1a8ed70, externalPath=null}
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: {kind=ADD, partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=3, level=5, fileName=data-09e88d85-43da-44ca-a813-00bb8e3d525f-0.orc, extraFiles=[], minKey=org.apache.paimon.data.BinaryRow@b0db094a, maxKey=org.apache.paimon.data.BinaryRow@a1a8ed70, externalPath=null}
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at org.apache.paimon.operation.FileStoreCommitImpl.createConflictException(FileStoreCommitImpl.java:1370) ~[?:?]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at org.apache.paimon.operation.FileStoreCommitImpl.lambda$noConflictsOrFail$16(FileStoreCommitImpl.java:1226) ~[?:?]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at org.apache.paimon.operation.FileStoreCommitImpl.noConflictsOrFail(FileStoreCommitImpl.java:1242) ~[?:?]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at org.apache.paimon.operation.FileStoreCommitImpl.commit(FileStoreCommitImpl.java:348) ~[?:?]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at org.apache.paimon.table.sink.TableCommitImpl.commitMultiple(TableCommitImpl.java:218) ~[?:?]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at org.apache.paimon.table.sink.TableCommitImpl.filterAndCommitMultiple(TableCommitImpl.java:257) ~[?:?]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at org.apache.paimon.table.sink.TableCommitImpl.filterAndCommitMultiple(TableCommitImpl.java:243) ~[?:?]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at org.apache.paimon.table.sink.TableCommitImpl.filterAndCommit(TableCommitImpl.java:196) ~[?:?]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at org.apache.seatunnel.connectors.seatunnel.paimon.sink.PaimonSinkWriter.<init>(PaimonSinkWriter.java:217) ~[?:?]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at org.apache.seatunnel.connectors.seatunnel.paimon.sink.PaimonSink.restoreWriter(PaimonSink.java:129) ~[?:?]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at org.apache.seatunnel.api.sink.multitablesink.MultiTableSink.restoreWriter(MultiTableSink.java:115) ~[seatunnel-starter.jar:2.3.12-SNAPSHOT]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.restoreState(SinkFlowLifeCycle.java:345) ~[seatunnel-starter.jar:2.3.12-SNAPSHOT]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at org.apache.seatunnel.engine.server.task.SeaTunnelTask.lambda$restoreState$16(SeaTunnelTask.java:401) ~[seatunnel-starter.jar:2.3.12-SNAPSHOT]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) ~[?:1.8.0_342]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_342]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_342]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) ~[?:1.8.0_342]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_342]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_342]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:1.8.0_342]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:1.8.0_342]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_342]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) ~[?:1.8.0_342]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at org.apache.seatunnel.engine.server.task.SeaTunnelTask.restoreState(SeaTunnelTask.java:398) ~[seatunnel-starter.jar:2.3.12-SNAPSHOT]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.lambda$null$0(NotifyTaskRestoreOperation.java:107) ~[seatunnel-starter.jar:2.3.12-SNAPSHOT]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640) ~[?:1.8.0_342]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43) ~[seatunnel-starter.jar:2.3.12-SNAPSHOT]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_342]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_342]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_342]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: Caused by: java.lang.IllegalStateException: Trying to add file {partition=org.apache.paimon.data.BinaryRow@9c67b85d, bucket=0, level=5, fileName=data-52776f55-19a4-4596-a1ce-9f8685394576-0.orc, extraFiles=[], embeddedIndex=null, externalPath=null} which is already added.
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at org.apache.paimon.utils.Preconditions.checkState(Preconditions.java:204) ~[?:?]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at org.apache.paimon.manifest.FileEntry.mergeEntries(FileEntry.java:191) ~[?:?]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at org.apache.paimon.manifest.FileEntry.mergeEntries(FileEntry.java:173) ~[?:?]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	at org.apache.paimon.operation.FileStoreCommitImpl.noConflictsOrFail(FileStoreCommitImpl.java:1240) ~[?:?]
[] 2025-08-05 19:12:18,999 INFO  tc.seatunnel-engine:openjdk:8 - STDERR: 	... 27 more

Does this PR introduce any user-facing change?

no

How was this patch tested?

PaimonSinkWithSchemaEvolutionIT#testMysqlCdcSinkPaimonWithSchemaChangeAndRestore

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config

...nel-connector-v2-e2e/connector-paimon-e2e/src/test/resources/changelog_paimon_to_paimon.conf

...on/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/sink/PaimonSinkWriter.java

.../org/apache/seatunnel/connectors/seatunnel/paimon/sink/commit/PaimonAggregatedCommitter.java

1 1 1 1 1 update doc [Improve][Csv] support configurable CSV delimiter in file connector (apache#9660) [Fix][Doc] Fix rest api finished-jobs doc miss `SAVEPOINT_DONE` (apache#9676) [Fix] [connector-jdbc] prevent precision loss in Float to BigDecimal conversion (apache#9670) [Feature][Connector-File-Hadoop]Support multi table sink feature for HdfsFile (apache#9651) [HotFix][CI] The Some CI subtasks are not executed (apache#9684) [Docs][Connector-Iceberg] Update Iceberg doc with support S3 Tables rest catalogs (apache#9686) [Docs] update jdbc related document (apache#9679) [Fix][Connector-V2] Fix misleading parameter name in DorisStreamLoad (apache#9685) [Feature] [connector-file] Add configurable sheet_max_rows support for Excel sink connector (apache#9668) [fix][connectors-v2] repeated commit cause task exceptions (apache#9665) [Feature][connector-v2] Add Sensorsdata Connector Support apache#9323 (apache#9432) [Improve][API] Optimize the enumerator API semantics and reduce lock calls at the connector level (apache#9671) [Hotfix][Zeta] Fix custom https port configuration not working (apache#9705) [Improve][Doc] Add SeaTunnel tools into Readme (apache#9707) [Improve][Core] Unify the aws-sdk-v2 version to 2.31.30 (apache#9698) [Feature][Transform-V2] Add `TRIM_SCALE` function for sql transform (apache#9700) [Improve][Core] Update apache common to apache common lang3 (apache#9694) Co-authored-by: litiliu <[email protected]>

[fix][connectors-v2] repeated commit cause task exceptions

c5986f2

github-actions bot added connectors-v2 e2e paimon labels Aug 5, 2025

Hisoka-X reviewed Aug 5, 2025

View reviewed changes

...nel-connector-v2-e2e/connector-paimon-e2e/src/test/resources/changelog_paimon_to_paimon.conf Show resolved Hide resolved

Hisoka-X reviewed Aug 6, 2025

View reviewed changes

...on/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/sink/PaimonSinkWriter.java Outdated Show resolved Hide resolved

hawk9821 added 2 commits August 7, 2025 20:56

[fix][connectors-v2] repeated commit cause task exceptions

ce09dc5

[fix][connectors-v2] repeated commit cause task exceptions

b503c8e

Hisoka-X reviewed Aug 9, 2025

View reviewed changes

.../org/apache/seatunnel/connectors/seatunnel/paimon/sink/commit/PaimonAggregatedCommitter.java Outdated Show resolved Hide resolved

hawk9821 added 2 commits August 9, 2025 20:14

[fix][connectors-v2] repeated commit cause task exceptions

8508619

[fix][connectors-v2] repeated commit cause task exceptions

fcdef1f

Hisoka-X approved these changes Aug 11, 2025

View reviewed changes

github-actions bot added approved reviewed labels Aug 11, 2025

corgy-w approved these changes Aug 11, 2025

View reviewed changes

corgy-w merged commit 085023a into apache:dev Aug 11, 2025
5 checks passed

hawk9821 deleted the paimon_states branch August 11, 2025 06:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[fix][connectors-v2] repeated commit cause task exceptions #9665

[fix][connectors-v2] repeated commit cause task exceptions #9665

Uh oh!

hawk9821 commented Aug 5, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[fix][connectors-v2] repeated commit cause task exceptions #9665

[fix][connectors-v2] repeated commit cause task exceptions #9665

Uh oh!

Conversation

hawk9821 commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hawk9821 commented Aug 5, 2025 •

edited

Loading