Skip to content

Conversation

@chovy-3012
Copy link
Contributor

Purpose of this pull request

Check list

bug as below:
I want to sink data from Kafka in text format, partitioned by date, into different HDFS directories, and the data does not contain timestamps.
test.conf :

env {
  job.name = "test"
  job.mode = "STREAMING"
  checkpoint.interval=20000
}

source {
  Kafka {
    result_table_name="test_table"
    bootstrap.servers = "xxxx:9092"
    topic = "test_topic"
    field_name = "data"
    format.type = "text"
    consumer.group = "test_group"
  }
}

transform {
  Sql {
      source_table_name = "test_table"
      result_table_name = "test_table_result"
      query="SELECT  data,formatdatetime(now(),'yyyy-MM-dd') new_ts   from  test_table  "
  }
}

sink {
   HdfsFile {
      fs.defaultFS = "hdfs://hadoop01:9000"
      is_enable_transaction = true
      have_partition = true
      partition_by = ["new_ts"]
      partition_dir_expression = "${v0}"
      is_partition_field_write_in_file =  false
      path = "/data/test"
      file_format_type = "json"
   }
}

cmd:

./bin/start-seatunnel-flink-13-connector-v2.sh test.conf

exception:

java.lang.UnsupportedOperationException
	at java.util.AbstractList.remove(AbstractList.java:161)
	at java.util.AbstractList$Itr.remove(AbstractList.java:374)
	at java.util.AbstractCollection.removeAll(AbstractCollection.java:376)
	at org.apache.seatunnel.connectors.seatunnel.file.sink.config.FileSinkConfig.<init>(FileSinkConfig.java:142)
	at org.apache.seatunnel.connectors.seatunnel.file.writer.FileSinkConfigTest.testInit(FileSinkConfigTest.java:48)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)

bux fix

fix file sink isPartitionFieldWriteInFile occurred exception when no columns are given for the sink

Hisoka-X
Hisoka-X previously approved these changes Oct 12, 2023
Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM expect one minor comment. cc @TyrantLucifer

if (CollectionUtils.isEmpty(this.sinkColumnList)) {
this.sinkColumnList = Arrays.asList(seaTunnelRowTypeInfo.getFieldNames());
this.sinkColumnList =
new ArrayList<>(Arrays.asList(seaTunnelRowTypeInfo.getFieldNames()));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add some comment in here. Only read the code will confused developers.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added comment in bug fix code

@Hisoka-X Hisoka-X added bug and removed reviewed labels Oct 12, 2023
@Hisoka-X Hisoka-X added this to the 2.3.4 milestone Oct 12, 2023
@Hisoka-X
Copy link
Member

@chovy-3012
Copy link
Contributor Author

@chovy-3012 Could you retry the failed CI? https://github.com/chovy-3012/seatunnel/actions/runs/6492030625/job/17630262359

Retried more than 10 times , still failed.
Error messages show no space left on device , any suggestions?

error message as below:

[INFO] Running org.apache.seatunnel.connectors.seatunnel.jdbc.JdbcAutoGenerateSQLIT
Error:  Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 8.34 s <<< FAILURE! - in org.apache.seatunnel.connectors.seatunnel.jdbc.JdbcAutoGenerateSQLIT
Error:  org.apache.seatunnel.connectors.seatunnel.jdbc.JdbcAutoGenerateSQLIT  Time elapsed: 8.34 s  <<< ERROR!
java.util.concurrent.CompletionException: org.testcontainers.containers.ContainerLaunchException: Container startup failed
Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed
Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=postgres:14-alpine, imagePullPolicy=DefaultPullPolicy(), imageNameSubstitutor=org.testcontainers.utility.ImageNameSubstitutor$LogWrappedImageNameSubstitutor@489496a9)
Caused by: org.testcontainers.containers.ContainerFetchException: Failed to get Docker client for postgres:14-alpine
Caused by: com.github.dockerjava.api.exception.DockerClientException: Could not pull image: failed to register layer: write /usr/lib/libLLVM-15.so: no space left on device

@hailin0 hailin0 merged commit 9fb5499 into apache:dev Oct 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants