[Bug] [Zeta] CDC tasks fail to be CheckPoint after a restore operation is performed

[wu-a-ge]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

I ran a CDC task and I ran seatunnel.sh-s 726264186136428545 and then seatunnel.sh-r 726264186136428545-C config/mysql_cdc_to_mysql.conf restore the task, and then check the output of the checkpoint directory.But I found that the engine had a warning log saying that checkpoint could not be performed.

2023-06-28 12:25:33,359 WARN  org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - can't trigger checkpoint with type: CHECKPOINT_TYPE, because checkpoint coordinator already have last completed checkpoint: (SAVEPOINT_TYPE) or shutdown (false).

SeaTunnel Version

2.3.2

SeaTunnel Config

seatunnel:
  engine:
    backup-count: 1
    queue-type: blockingqueue
    print-execution-info-interval: 60
    print-job-metrics-info-interval: 60
    slot-service:
      dynamic-slot: true
    checkpoint:
      interval: 10000
      timeout: 60000
      max-concurrent: 1
      tolerable-failure: 2
      storage:
        type: hdfs
        max-retained: 3
        plugin-config:
          namespace: /data/seatunnel/checkpoint_snapshot
          storage.type: hdfs
          fs.defaultFS: file:/// # Ensure that the directory has written permission

Running Command

seatunnel.sh  -r 726264186136428545 -c config/mysql_cdc_to_mysql.conf

Error Exception

2023-06-28 12:25:22,773 INFO  org.apache.seatunnel.engine.server.checkpoint.CheckpointManager - pipeline(1) start with savePoint on checkPointId(483)
2023-06-28 12:25:22,778 INFO  org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - Turn checkpoint_state_726264186136428545_1 state from SUSPEND to RUNNING
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex
信息: The task Job SeaTunnel_Job (726264186136428545), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-departments]-SourceTask (1/1)] is in state CREATED when init state future
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex
信息: The task Job SeaTunnel_Job (726264186136428545), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-departments]-SplitEnumerator (1/1)] is in state CREATED when init state future
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.master.JobMaster
信息: Job Job SeaTunnel_Job (726264186136428545) waiting for scheduler finished
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.dag.physical.PhysicalPlan
信息: Job SeaTunnel_Job (726264186136428545) turn from state CREATED to SCHEDULED.
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.dag.physical.SubPlan
信息: Job SeaTunnel_Job (726264186136428545), Pipeline: [(1/1)] turn from state CREATED to SCHEDULED.
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex
信息: Job SeaTunnel_Job (726264186136428545), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-departments]-SplitEnumerator (1/1)] turn from state CREATED to SCHEDULED.
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex
信息: Job SeaTunnel_Job (726264186136428545), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-departments]-SourceTask (1/1)] turn from state CREATED to SCHEDULED.
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.dag.physical.SubPlan
信息: Job SeaTunnel_Job (726264186136428545), Pipeline: [(1/1)] turn from state SCHEDULED to DEPLOYING.
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex
信息: Job SeaTunnel_Job (726264186136428545), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-departments]-SplitEnumerator (1/1)] turn from state SCHEDULED to DEPLOYING.
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex
信息: Job SeaTunnel_Job (726264186136428545), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-departments]-SourceTask (1/1)] turn from state SCHEDULED to DEPLOYING.
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.TaskExecutionService
信息: [localhost]:5801 [seatunnel] [5.1] received deploying task executionId [726292067654828033]
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.TaskExecutionService
信息: [localhost]:5801 [seatunnel] [5.1] received deploying task executionId [726292067654893569]
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.TaskExecutionService
信息: [localhost]:5801 [seatunnel] [5.1] deploying task TaskGroupLocation{jobId=726264186136428545, pipelineId=1, taskGroupId=1}, executionId [726292067654828033]
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.TaskExecutionService
信息: [localhost]:5801 [seatunnel] [5.1] deploying task TaskGroupLocation{jobId=726264186136428545, pipelineId=1, taskGroupId=30000}, executionId [726292067654893569]
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.TaskExecutionService
信息: [localhost]:5801 [seatunnel] [5.1] deploying TaskGroup TaskGroupLocation{jobId=726264186136428545, pipelineId=1, taskGroupId=1} init success
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.TaskExecutionService
信息: [localhost]:5801 [seatunnel] [5.1] deploying TaskGroup TaskGroupLocation{jobId=726264186136428545, pipelineId=1, taskGroupId=1} success
2023-06-28 12:25:22,791 INFO  org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - starting seatunnel source split enumerator task, source name: pipeline-1 [Source[0]-MySQL-CDC-departments]
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.TaskExecutionService
信息: [localhost]:5801 [seatunnel] [5.1] deploying TaskGroup TaskGroupLocation{jobId=726264186136428545, pipelineId=1, taskGroupId=30000} init success
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex
信息: Job SeaTunnel_Job (726264186136428545), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-departments]-SplitEnumerator (1/1)] turn from state DEPLOYING to RUNNING.
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.TaskExecutionService
信息: [localhost]:5801 [seatunnel] [5.1] deploying TaskGroup TaskGroupLocation{jobId=726264186136428545, pipelineId=1, taskGroupId=30000} success
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex
信息: Job SeaTunnel_Job (726264186136428545), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-departments]-SourceTask (1/1)] turn from state DEPLOYING to RUNNING.
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.dag.physical.SubPlan
信息: Job SeaTunnel_Job (726264186136428545), Pipeline: [(1/1)] turn from state DEPLOYING to RUNNING.
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask
信息: starting seatunnel transform task, index 0
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.dag.physical.PhysicalPlan
信息: Job SeaTunnel_Job (726264186136428545) turn from state SCHEDULED to RUNNING.
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.master.JobMaster
信息: Job SeaTunnel_Job (726264186136428545) scheduler finished
六月 28, 2023 12:25:22 下午 org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask
信息: starting seatunnel source task, index 0
2023-06-28 12:25:22,817 INFO  org.apache.seatunnel.connectors.seatunnel.cdc.mysql.utils.TableDiscoveryUtils - Read list of available databases
2023-06-28 12:25:22,830 INFO  org.apache.seatunnel.connectors.seatunnel.cdc.mysql.utils.TableDiscoveryUtils -    list of available databases is: [CSV, DEVDB, DEVDB1, DEVDB2, DEVDB20801, DEVDB20803, DEVDB_001, DEVDB_002, DEVDB_003, Sheet1, TXT, am3, base, bkiot, bsc-ems, camel, data, data_sources, database_t_1, database_th_001, database_th_27_01, database_th_27_02, db_bd, db_bd3, dbo, dbo1228, dbo2, dbo230106, dev_114, dolphinscheduler, ds, flh_test1, geega_esb, geega_esb_local, gg, gg2, gmes_print, icc_test_t1, information_schema, jiaData, jiaData1202, jiaData1204, jiaData1205, jiahua_0309, jiahua_0309_2, kafkadata, kafkadata1024, kafkadata816, kp, kp0726, kp220728, linkis, mk_test, mysql, mysql_data_test, mysql_test, performance_schema, queue1, seatunnel, sink0725, sink1116, sink1129, skeleton_demo, sqlserver1031, string_test, sys, t00, test, test2, test22, test_0713_sink, test_0713_sink726, test_0713_sink_table, test_0713_source, test_0713_source1, test_0713_source66, test_0713_source_21, test_0713_source_221, test_1111, test_1111_0713, test_1201, test_1202, test_cyn, test_cyn_two, test_pma_w, test_point, test_sink_0822, test_test_test, test_th_001, testdb, testout0406_2, th_27_001, th_27_002, th_db, tttt_ssss, xuefei_test, y1, y2]
2023-06-28 12:25:22,830 INFO  org.apache.seatunnel.connectors.seatunnel.cdc.mysql.utils.TableDiscoveryUtils - Read list of available tables in each database
2023-06-28 12:25:22,907 INFO  org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase - Open Source Reader.
2023-06-28 12:25:22,907 INFO  org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - received reader register, readerID: TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=726264186136428545, pipelineId=1, taskGroupId=30000}, taskID=40000, index=0}
2023-06-28 12:25:23,314 INFO  io.debezium.jdbc.JdbcConnection - Connection gracefully closed
2023-06-28 12:25:23,457 INFO  org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - received enough reader, starting enumerator...
2023-06-28 12:25:23,462 INFO  org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.SplitFetcher - Starting split fetcher 0
2023-06-28 12:25:23,464 WARN  io.debezium.connector.mysql.MySqlConnection - Database configuration option 'serverTimezone' is set but is obsolete, please use 'connectionTimeZone' instead
2023-06-28 12:25:23,465 INFO  org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceSplitReader - Stream fetcher is created.
2023-06-28 12:25:23,560 INFO  io.debezium.jdbc.JdbcConnection - Connection gracefully closed
2023-06-28 12:25:23,574 WARN  io.debezium.relational.history.DatabaseHistoryMetrics - Unable to register the MBean 'debezium.mysql:type=connector-metrics,context=schema-history,server=mysql_binlog_source': debezium.mysql:type=connector-metrics,context=schema-history,server=mysql_binlog_source
2023-06-28 12:25:23,658 WARN  io.debezium.relational.history.DatabaseHistoryMetrics - Unable to register the MBean 'debezium.mysql:type=connector-metrics,context=schema-history,server=mysql_binlog_source': debezium.mysql:type=connector-metrics,context=schema-history,server=mysql_binlog_source
2023-06-28 12:25:23,658 INFO  io.debezium.connector.mysql.MySqlConnection - Get all known binlogs from MySQL
2023-06-28 12:25:23,662 INFO  org.apache.seatunnel.connectors.seatunnel.cdc.mysql.source.reader.fetch.MySqlSourceFetchTaskContext - MySQL has the binlog file 'binlog.000032' required by the connector
2023-06-28 12:25:23,663 INFO  io.debezium.relational.history.DatabaseHistoryMetrics - Started database history recovery
2023-06-28 12:25:23,663 INFO  io.debezium.relational.history.DatabaseHistoryMetrics - Finished database history recovery of 0 change(s) in 0 ms
2023-06-28 12:25:23,664 WARN  io.debezium.connector.mysql.MySqlValueConverters - Column uses MySQL character set 'utf8mb3', which has no mapping to a Java character set
2023-06-28 12:25:23,664 WARN  io.debezium.connector.mysql.MySqlValueConverters - Using UTF-8 charset by default for column without charset: dept_name VARCHAR(512) CHARSET utf8mb3 DEFAULT VALUE NULL
2023-06-28 12:25:23,664 WARN  io.debezium.connector.mysql.MySqlValueConverters - Column uses MySQL character set 'utf8mb3', which has no mapping to a Java character set
2023-06-28 12:25:23,664 WARN  io.debezium.connector.mysql.MySqlValueConverters - Using UTF-8 charset by default for column without charset: def VARCHAR(100) CHARSET utf8mb3 DEFAULT VALUE NULL
2023-06-28 12:25:23,664 WARN  io.debezium.connector.mysql.MySqlValueConverters - Column uses MySQL character set 'utf8mb3', which has no mapping to a Java character set
2023-06-28 12:25:23,664 WARN  io.debezium.connector.mysql.MySqlValueConverters - Using UTF-8 charset by default for column without charset: abc VARCHAR(100) CHARSET utf8mb3 DEFAULT VALUE NULL
2023-06-28 12:25:23,665 INFO  io.debezium.util.Threads - Requested thread factory for connector MySqlConnector, id = mysql_binlog_source named = binlog-client
2023-06-28 12:25:23,678 INFO  io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Skip 0 events on streaming start
2023-06-28 12:25:23,678 INFO  io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Skip 0 rows on streaming start
2023-06-28 12:25:23,679 INFO  io.debezium.util.Threads - Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
2023-06-28 12:25:23,684 INFO  io.debezium.util.Threads - Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
六月 28, 2023 12:25:23 下午 com.github.shyiko.mysql.binlog.BinaryLogClient connect
信息: Connected to 10.113.75.114:3307 at binlog.000032/41559681 (sid:5428, cid:257142)
2023-06-28 12:25:23,717 INFO  io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Connected to MySQL binlog at localhost:3307, starting at MySqlOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.mysql.Source:STRUCT}, sourceInfo=SourceInfo [currentGtid=null, currentBinlogFilename=binlog.000032, currentBinlogPosition=41559681, currentRowNumber=0, serverId=0, sourceTime=null, threadId=-1, currentQuery=null, tableIds=[], databaseName=null], partition={server=mysql_binlog_source}, snapshotCompleted=false, transactionContext=TransactionContext [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], restartGtidSet=null, currentGtidSet=null, restartBinlogFilename=binlog.000032, restartBinlogPosition=41559681, restartRowsToSkip=0, restartEventsToSkip=0, currentEventLengthInBytes=0, inTransaction=false, transactionId=null, incrementalSnapshotContext =IncrementalSnapshotContext [windowOpened=false, chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null, maximumKey=null]]
2023-06-28 12:25:23,717 INFO  io.debezium.util.Threads - Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
2023-06-28 12:25:23,718 INFO  io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Waiting for keepalive thread to start
2023-06-28 12:25:23,718 INFO  io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Keepalive thread is running
2023-06-28 12:25:33,359 WARN  org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - can't trigger checkpoint with type: CHECKPOINT_TYPE, because checkpoint coordinator already have last completed checkpoint: (SAVEPOINT_TYPE) or shutdown (false).

Flink or Spark Version

No response

Java or Scala Version

1.8

Screenshots

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-a-ge]

This is my config:

env {
    execution.parallelism = 10
    job.mode = "STREAMING"
}
source {
  MySQL-CDC {
    username = "root"
    password = "11111111"
    table-names = ["seatunnel.departments"]
    base-url = "jdbc:mysql://localhost:3307/seatunnel?useSSL=false&serverTimezone=GMT%2B8"
	   result_table_name="departments"
  }
}
transform {
   fieldmapper {
    source_table_name = "departments"
    result_table_name = "departments_1"
    field_mapper = {
        dept_no = dept_no
        dept_name = dept_name
        def = def
    }
   }

}
sink {
  jdbc {
        source_table_name="departments_1"
        url = "jdbc:mysql://localhost:3307/seatunnel?useSSL=false&serverTimezone=GMT%2B8"
        driver = "com.mysql.cj.jdbc.Driver"
        user = "root"
        password = "1111111"
        database = "seatunnel"
        table = "departments_1"
        primary_keys = [dept_no]
        generate_sink_sql=true
    }
}