[Feature][Connectors-v2] Support auto-increment id for FakeSource #9505

hawk9821 · 2025-06-26T07:42:54Z

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config

corgy-w

Good contribution

...main/java/org/apache/seatunnel/connectors/seatunnel/fake/utils/AutoIncrementIdGenerator.java

...ake/src/main/java/org/apache/seatunnel/connectors/seatunnel/fake/utils/IdGeneratorUtils.java

…e/seatunnel/connectors/seatunnel/fake/utils/AutoIncrementIdGenerator.java Co-authored-by: Jia Fan <[email protected]>

…e/seatunnel/connectors/seatunnel/fake/utils/IdGeneratorUtils.java Co-authored-by: Jia Fan <[email protected]>

Hisoka-X

What if the source runs on multiple workers? Should it be allocated in advance in the enumerator? Or should it be limited to a single degree of parallelism?

Hisoka-X · 2025-06-27T04:43:17Z

...ake/src/main/java/org/apache/seatunnel/connectors/seatunnel/fake/utils/IdGeneratorUtils.java

+
+public class IdGeneratorUtils {
+
+    private static final Map<String, AutoIncrementIdGenerator> idGenerators =


Static variables will cause problems when the same job is run multiple times

Copilot

Pull Request Overview

This PR adds support for auto-incrementing primary key IDs in the FakeSource connector and updates related tests and documentation.

Introduces new auto.increment.enabled and auto.increment.start options in FakeSourceOptions and FakeConfig
Implements ID generators (AutoIncrementIdGenerator, IdGeneratorUtils) and integrates them into FakeDataRandomUtils, FakeDataGenerator, and FakeSourceReader
Adds an end-to-end E2E test (testAutoIncrementId) and updates documentation with examples

Reviewed Changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
connector-fake/src/main/java/.../FakeSourceOptions.java	Defined new connector options for auto-increment
connector-fake/src/main/java/.../FakeConfig.java	Mapped new options into `FakeConfig`
connector-fake/src/main/java/.../AutoIncrementIdGenerator.java	Implemented the ID generator
connector-fake/src/main/java/.../IdGeneratorUtils.java	Cached and retrieved per-job ID generators
connector-fake/src/main/java/.../FakeDataRandomUtils.java	Applied auto-increment logic for ints and bigints
connector-fake/src/main/java/.../FakeDataGenerator.java	Passed `jobId` to random utils
connector-fake/src/main/java/.../FakeSourceReader.java	Thread-safe generator lookup via `jobId`
connector-fake/src/main/java/.../FakeSource.java	Propagated `jobId` into reader creation
connector-fake/src/test/java/.../FakeDataGeneratorTest.java	Updated test constructors and added auto-increment test
connector-pulsar-e2e/.../PulsarBatchIT.java	Adjusted `FakeDataGenerator` constructor calls
connector-fake/src/test/resources/fake-auto-increment-id.conf	Added a config file for auto-increment testing
docs/zh/connector-v2/source/FakeSource.md	Documented new options and example
docs/en/connector-v2/source/FakeSource.md	Documented new options and example

Comments suppressed due to low confidence (5)

docs/zh/connector-v2/source/FakeSource.md:33

The default value for auto.increment.start is missing in the table; it should reflect the code default (1).

| auto.increment.start      | int      | 否   |                        | 自动递增ID的起始值                                                                                                                                                                          |

docs/en/connector-v2/source/FakeSource.md:34

The default value for auto.increment.start is not shown; update the table to indicate the default of 1.

| auto.increment.start    | int      | no       |                         | Starting value for auto increment ID                                                                                                                                                                          |

seatunnel-connectors-v2/connector-fake/src/test/java/org/apache/seatunnel/connectors/seatunnel/fake/source/FakeDataGeneratorTest.java:238

The same config file is provided twice in the @ValueSource; remove the duplicate to avoid redundant test runs.

    @ValueSource(strings = {"fake-auto-increment-id.conf", "fake-auto-increment-id.conf"})

seatunnel-e2e/seatunnel-connector-v2-e2e/connector-pulsar-e2e/src/test/java/org/apache/seatunnel/e2e/connector/pulsar/PulsarBatchIT.java:154

[nitpick] Passing null for jobId may mask issues; consider adding an overload that defaults jobId internally or retrieving a real job ID from the test context.

            FakeDataGenerator fakeDataGenerator = new FakeDataGenerator(fakeConfig, null);

seatunnel-connectors-v2/connector-fake/src/test/java/org/apache/seatunnel/connectors/seatunnel/fake/source/FakeDataGeneratorTest.java:246

[nitpick] The variable name uCompletableFuture is unclear; consider renaming it to simply future or rowFuture for readability.

            CompletableFuture<List<SeaTunnelRow>> uCompletableFuture =

Copilot · 2025-07-02T02:18:38Z

...tor-fake/src/main/java/org/apache/seatunnel/connectors/seatunnel/fake/source/FakeSource.java

 import java.util.List;
 import java.util.stream.Collectors;

+@Slf4j


[nitpick] The @Slf4j annotation is added but no logging calls are used in this class; consider removing it or adding relevant log statements.

Copilot · 2025-07-02T02:18:38Z

.../src/main/java/org/apache/seatunnel/connectors/seatunnel/fake/utils/FakeDataRandomUtils.java


    public Long randomBigint(Column column) {
+        if (fakeConfig.getAutoIncrementEnabled()
+                && IdGeneratorUtils.isPrimaryColumn(fakeConfig, column.getName())) {


There's no overflow check for auto.increment.start + parallelism * rowNum when generating bigints; this could exceed Long.MAX_VALUE.

Suggested change

&& IdGeneratorUtils.isPrimaryColumn(fakeConfig, column.getName())) {

&& IdGeneratorUtils.isPrimaryColumn(fakeConfig, column.getName())) {

long autoIncrementStart = fakeConfig.getAutoIncrementStart();

long parallelism = fakeConfig.getParallelism();

long rowNum = fakeConfig.getRowNum();

if (autoIncrementStart > 0 && parallelism > 0 && rowNum > 0) {

if (autoIncrementStart > Long.MAX_VALUE - (parallelism * rowNum)) {

throw new IllegalArgumentException(

"The auto increment start value is too large, or the combination of parallelism and rowNum causes an overflow. Please check your configuration.");

}

}

Copilot · 2025-07-02T02:18:38Z

...ake/src/main/java/org/apache/seatunnel/connectors/seatunnel/fake/utils/IdGeneratorUtils.java

+                    .expireAfterWrite(30, TimeUnit.MINUTES)
+                    .build();
+
+    public static synchronized Optional<AutoIncrementIdGenerator> getIdGenerator(


The method is declared synchronized, which may become a bottleneck under high concurrency; consider relying on Guava's thread-safe cache instead of external synchronization.

Suggested change

public static synchronized Optional<AutoIncrementIdGenerator> getIdGenerator(

public static Optional<AutoIncrementIdGenerator> getIdGenerator(

…ache#9505)

[feature][connectors-v2]Support auto-increment id for FakeSource

766b07e

github-actions bot added document connectors-v2 labels Jun 26, 2025

hawk9821 requested a review from corgy-w June 26, 2025 07:44

corgy-w previously approved these changes Jun 26, 2025

View reviewed changes

github-actions bot added approved reviewed labels Jun 26, 2025

Hisoka-X reviewed Jun 26, 2025

View reviewed changes

...main/java/org/apache/seatunnel/connectors/seatunnel/fake/utils/AutoIncrementIdGenerator.java Outdated Show resolved Hide resolved

...ake/src/main/java/org/apache/seatunnel/connectors/seatunnel/fake/utils/IdGeneratorUtils.java Outdated Show resolved Hide resolved

Update seatunnel-connectors-v2/connector-fake/src/main/java/org/apach…

e66ae40

…e/seatunnel/connectors/seatunnel/fake/utils/AutoIncrementIdGenerator.java Co-authored-by: Jia Fan <[email protected]>

hawk9821 dismissed corgy-w’s stale review via e66ae40 June 26, 2025 13:53

github-actions bot removed approved reviewed labels Jun 26, 2025

Update seatunnel-connectors-v2/connector-fake/src/main/java/org/apach…

26a3c55

…e/seatunnel/connectors/seatunnel/fake/utils/IdGeneratorUtils.java Co-authored-by: Jia Fan <[email protected]>

nielifeng requested a review from Copilot June 27, 2025 02:08

This comment was marked as outdated.

Sign in to view

Hisoka-X reviewed Jun 27, 2025

View reviewed changes

hawk9821 added 2 commits July 1, 2025 17:45

[feature][connectors-v2]Support auto-increment id for FakeSource

ac9f9ac

[feature][connectors-v2]Support auto-increment id for FakeSource

3e1b358

github-actions bot added the e2e label Jul 2, 2025

[feature][connectors-v2]Support auto-increment id for FakeSource

fc05086

nielifeng requested a review from Copilot July 2, 2025 02:16

Copilot AI reviewed Jul 2, 2025

View reviewed changes

Hisoka-X approved these changes Jul 2, 2025

View reviewed changes

Hisoka-X changed the title ~~[feature][connectors-v2]Support auto-increment id for FakeSource~~ [Feature][Connectors-v2] Support auto-increment id for FakeSource Jul 2, 2025

github-actions bot added approved reviewed labels Jul 2, 2025

corgy-w approved these changes Jul 3, 2025

View reviewed changes

corgy-w merged commit 3a16b4a into apache:dev Jul 3, 2025
7 checks passed

hawk9821 deleted the inc_id branch July 3, 2025 06:18

dybyte pushed a commit to dybyte/seatunnel that referenced this pull request Jul 23, 2025

[Feature][Connectors-v2] Support auto-increment id for FakeSource (ap…

decd3f6

…ache#9505)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Feature][Connectors-v2] Support auto-increment id for FakeSource #9505

[Feature][Connectors-v2] Support auto-increment id for FakeSource #9505

Uh oh!

hawk9821 commented Jun 26, 2025

Uh oh!

corgy-w left a comment

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Hisoka-X left a comment

Uh oh!

Hisoka-X Jun 27, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jul 2, 2025

Uh oh!

Copilot AI Jul 2, 2025

Uh oh!

Copilot AI Jul 2, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		public class IdGeneratorUtils {

		private static final Map<String, AutoIncrementIdGenerator> idGenerators =

-                && IdGeneratorUtils.isPrimaryColumn(fakeConfig, column.getName())) {
+                && IdGeneratorUtils.isPrimaryColumn(fakeConfig, column.getName())) {
+            long autoIncrementStart = fakeConfig.getAutoIncrementStart();
+            long parallelism = fakeConfig.getParallelism();
+            long rowNum = fakeConfig.getRowNum();
+            if (autoIncrementStart > 0 && parallelism > 0 && rowNum > 0) {
+                if (autoIncrementStart > Long.MAX_VALUE - (parallelism * rowNum)) {
+                    throw new IllegalArgumentException(
+                            "The auto increment start value is too large, or the combination of parallelism and rowNum causes an overflow. Please check your configuration.");
+                }
+            }

	public static synchronized Optional<AutoIncrementIdGenerator> getIdGenerator(
	public static Optional<AutoIncrementIdGenerator> getIdGenerator(

[Feature][Connectors-v2] Support auto-increment id for FakeSource #9505

[Feature][Connectors-v2] Support auto-increment id for FakeSource #9505

Uh oh!

Conversation

hawk9821 commented Jun 26, 2025

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Uh oh!

corgy-w left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Hisoka-X left a comment

Choose a reason for hiding this comment

Uh oh!

Hisoka-X Jun 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants