[Bug] [Kafka source STREAMING] Kafka source connector doesn't capture new data from topic #8107
Description
Search before asking
- I had searched in the issues and found no similar issues.
What happened
SeaTunnel 2.3.8 + Apache Flink Flink engine + Streaming mode.
Kafka source connector doesn't capture new data from topic, but there are no exception while execution.
New data appears in topic, but doesn't transfer to Redis.
In Batch mode everything works, but we need streaming mode with flink.
SeaTunnel Version
SeaTunnel 2.3.8
SeaTunnel Config
# Defining the runtime environment
env {
parallelism = 1
job.mode = "STREAMING"
job.name=SeaTunnel_Job_Kafka_Redis_Streaming
shade.identifier = "base64"
execution.checkpoint.interval = 60000
}
source {
Kafka {
schema = {
fields {
age = "int"
}
}
format = text
field_delimiter = "#"
topic = "registrations"
bootstrap.servers = "_"
commit_on_checkpoint=false
poll.timeout = 1000
start_mode = "earliest"
kafka.config = {
auto.offset.reset = "earliest"
enable.auto.commit = "false"
client.id = client_1
}
}
}
sink {
Redis {
host = "_"
port = 6379
key = age
data_type = hash
auth = "_"
batch_size = 1
}
}
Running Command
slatunnel flink 15 commandError Exception
No errors, task work without error, but there no new data
Zeta or Flink or Spark Version
Apache Flink
Java or Scala Version
No response
Screenshots
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct