[Improve][Connector-file] Add configurable binary chunk size support to BinaryReadStrategy

[CosmosNi]

Add Configurable Binary Chunk Size Support to BinaryReadStrategy

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

• If any new Jar binary package adding in your PR, please add License Notice according
  New License Guide
• If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
• If you are contributing the connector code, please check that the following files are updated:
  1. Update plugin-mapping.properties and add new connector information in it
  2. Update the pom file of seatunnel-dist
  3. Add ci label in label-scope-conf
  4. Add e2e testcase in seatunnel-e2e
  5. Update connector plugin_config

[Copilot]

Pull Request Overview

This PR adds support for configuring how binary files are chunked or read in complete-file mode by introducing two new options and updating the read strategy and tests accordingly.

• Introduce binary_chunk_size and binary_complete_file_mode options in FileBaseSourceOptions
• Update BinaryReadStrategy to load, validate, and apply these new options
• Add BinaryReadStrategyTest to cover default chunking, custom chunk sizes, and complete-file mode

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
seatunnel-connectors-v2/connector-file/.../FileBaseSourceOptions.java	Added BINARY_CHUNK_SIZE and BINARY_COMPLETE_FILE_MODE options with defaults
seatunnel-connectors-v2/connector-file/.../BinaryReadStrategy.java	Loaded and validated new options; refactored read() into readCompleteFile and readFileInChunks
seatunnel-connectors-v2/connector-file/.../BinaryReadStrategyTest.java	Added unit tests for default chunking, custom chunk size, and complete-file mode

Comments suppressed due to low confidence (1)

seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/BinaryReadStrategy.java:61

• Add unit tests to cover invalid chunk size configurations (e.g., zero or negative values, values above the 100MB limit) to verify that IllegalArgumentException is thrown as expected.

if (binaryChunkSize <= 0) {

[Copilot]

[nitpick] Extract the 100 * 1024 * 1024 magic number into a descriptive static final constant (e.g., MAX_BINARY_CHUNK_SIZE) to improve readability and maintainability.

Suggested change

	if (binaryChunkSize > 100 * 1024 * 1024) { // 100MB limit
	if (binaryChunkSize > MAX_BINARY_CHUNK_SIZE) { // 100MB limit

[Copilot]

[nitpick] Avoid hardcoding the config key string "path"; use FileBaseSourceOptions.FILE_PATH.key() to populate the configuration map so it stays in sync with the option definition.

Suggested change

	configMap.put("path", filePath); // Fixed: use "path" instead of "file_path"
	configMap.put(FileBaseSourceOptions.FILE_PATH.key(), filePath);

[Copilot]

[nitpick] Consider using a connector-specific exception (e.g., FileConnectorException) instead of IllegalArgumentException to provide consistent error reporting within the framework.

Suggested change

	throw new IllegalArgumentException(
	"Binary chunk size must be positive, got: " + binaryChunkSize);
	}
	if (binaryChunkSize > 100 * 1024 * 1024) { // 100MB limit
	throw new IllegalArgumentException(
	throw new FileConnectorException(
	"FILE_CONNECTOR_ERROR",
	"Binary chunk size must be positive, got: " + binaryChunkSize);
	}
	if (binaryChunkSize > 100 * 1024 * 1024) { // 100MB limit
	throw new FileConnectorException(
	"FILE_CONNECTOR_ERROR",

[Copilot]

Reading the entire file into memory via IOUtils.toByteArray can lead to high memory usage for large files; consider imposing a size check or using a streaming approach to avoid potential OOM issues.

Suggested change

	byte[] fileContent = IOUtils.toByteArray(inputStream);
	SeaTunnelRow row = new SeaTunnelRow(new Object[] {fileContent, relativePath, 0L});
	row.setTableId(tableId);
	output.collect(row);
	try (ByteArrayOutputStream outputStream = new ByteArrayOutputStream()) {
	byte[] buffer = new byte[binaryChunkSize];
	int bytesRead;
	while ((bytesRead = inputStream.read(buffer)) != -1) {
	outputStream.write(buffer, 0, bytesRead);
	}
	byte[] fileContent = outputStream.toByteArray();
	SeaTunnelRow row = new SeaTunnelRow(new Object[] {fileContent, relativePath, 0L});
	row.setTableId(tableId);
	output.collect(row);
	}

[Hisoka-X]

Thanks @CosmosNi ! Please update the docs.

[Hisoka-X]

Suggested change

	+ "Larger values may improve performance for large files but use more memory.");
	+ "Larger values may improve performance for large files but use more memory. Only valid when file_format_type is binary");

[Hisoka-X]

Suggested change

	+ "When enabled, the entire file content will be read into memory at once.");
	+ "When enabled, the entire file content will be read into memory at once. Only valid when file_format_type is binary");

[CosmosNi]

GET

[Hisoka-X]

Add description too?

[Hisoka-X]

why delete this?

[CosmosNi]

I accidentally deleted it.

[Hisoka-X]

LGTM if ci passes.