Skip to content

Conversation

@hawk9821
Copy link
Contributor

@hawk9821 hawk9821 commented Jul 8, 2025

Purpose of this pull request

Extract the common connector-cdc-base module to reduce the size of the cdc JAR .

before
image
after
image

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

@github-actions github-actions bot added core SeaTunnel core module connectors-v2 e2e labels Jul 8, 2025
@nielifeng nielifeng requested a review from Copilot July 8, 2025 03:23
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR extracts a common connector-cdc-base module to reduce individual CDC connector JAR sizes and updates discovery logic to load multiple URLs.

  • Renames several connector mappings (e.g., connector-httpconnector-http-base) and adjusts related tests for dynamic paths.
  • Refactors AbstractPluginDiscovery and subclasses to handle lists of URLs when adding to classloaders and implements CDC-specific filtering.
  • Adds connector-cdc-base to E2E tests and distribution assemblies, and scopes various dependencies as provided.

Reviewed Changes

Copilot reviewed 31 out of 31 changed files in this pull request and generated no comments.

Show a summary per file
File Description
seatunnel-plugin-discovery/.../plugin-mapping.properties Updated connector keys to *-base versions
seatunnel-plugin-discovery/src/test/.../*DiscoveryTest.java Refactored test setup to compute seatunnelHome at runtime
seatunnel-plugin-discovery/src/main/java/.../AbstractPluginDiscovery.java Changed URL consumer to accept List<URL> and multi-URL logic
seatunnel-plugin-discovery/src/main/java/.../SeaTunnel*PluginDiscovery.java Updated constructors to accept List<URL> consumer
seatunnel-e2e/.../ContainerUtil.java Conditional copy of transform jars and added getProjectVersion()
seatunnel-e2e/.../connector-cdc-*/pom.xml Added connector-cdc-base test dependency
seatunnel-dist/.../assembly-bin*.xml Included connector-cdc-base in assemblies
seatunnel-dist/pom.xml Added connector-cdc-base as provided dependency
seatunnel-core/.../FlinkExecution.java Simplified classloader URL addition to batch accept
seatunnel-core/.../FlinkAbstractPluginExecuteProcessor.java Updated URL consumer to batch add via reflection
seatunnel-connectors-v2/.../AbstractJdbcCatalog.java Loads Oracle driver explicitly and handles ClassNotFoundException
Comments suppressed due to low confidence (2)

seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/AbstractPluginDiscovery.java:139

  • Missing import for java.util.Collection, which is required for Collection::stream reference. Please add import java.util.Collection; at the top of the file.
                .flatMap(Collection::stream)

seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/AbstractPluginDiscovery.java:208

  • The variable pluginJars is undefined in this scope, causing a compile error. It should likely reference the defined pluginJarPaths or another valid list; please correct the variable name.
                addURLToClassLoaderConsumer.accept(classLoader, (List<URL>) pluginJars);

Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @hawk9821

@Hisoka-X Hisoka-X changed the title [feature][connectors-v2]Optimize the size of CDC JAR Files [Feature][Connectors-v2] Optimize the size of CDC JAR Files Jul 11, 2025
@corgy-w corgy-w merged commit 1dd19c6 into apache:dev Jul 14, 2025
7 checks passed
@hawk9821 hawk9821 deleted the dist branch July 14, 2025 05:40
chncaesar pushed a commit to chncaesar/seatunnel that referenced this pull request Jul 19, 2025
dybyte pushed a commit to dybyte/seatunnel that referenced this pull request Jul 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants