-
Notifications
You must be signed in to change notification settings - Fork 2.1k
[Feature][Connectors-v2] Optimize the size of CDC JAR Files #9546
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR extracts a common connector-cdc-base module to reduce individual CDC connector JAR sizes and updates discovery logic to load multiple URLs.
- Renames several connector mappings (e.g.,
connector-http→connector-http-base) and adjusts related tests for dynamic paths. - Refactors
AbstractPluginDiscoveryand subclasses to handle lists of URLs when adding to classloaders and implements CDC-specific filtering. - Adds
connector-cdc-baseto E2E tests and distribution assemblies, and scopes various dependencies asprovided.
Reviewed Changes
Copilot reviewed 31 out of 31 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| seatunnel-plugin-discovery/.../plugin-mapping.properties | Updated connector keys to *-base versions |
| seatunnel-plugin-discovery/src/test/.../*DiscoveryTest.java | Refactored test setup to compute seatunnelHome at runtime |
| seatunnel-plugin-discovery/src/main/java/.../AbstractPluginDiscovery.java | Changed URL consumer to accept List<URL> and multi-URL logic |
| seatunnel-plugin-discovery/src/main/java/.../SeaTunnel*PluginDiscovery.java | Updated constructors to accept List<URL> consumer |
| seatunnel-e2e/.../ContainerUtil.java | Conditional copy of transform jars and added getProjectVersion() |
| seatunnel-e2e/.../connector-cdc-*/pom.xml | Added connector-cdc-base test dependency |
| seatunnel-dist/.../assembly-bin*.xml | Included connector-cdc-base in assemblies |
| seatunnel-dist/pom.xml | Added connector-cdc-base as provided dependency |
| seatunnel-core/.../FlinkExecution.java | Simplified classloader URL addition to batch accept |
| seatunnel-core/.../FlinkAbstractPluginExecuteProcessor.java | Updated URL consumer to batch add via reflection |
| seatunnel-connectors-v2/.../AbstractJdbcCatalog.java | Loads Oracle driver explicitly and handles ClassNotFoundException |
Comments suppressed due to low confidence (2)
seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/AbstractPluginDiscovery.java:139
- Missing import for java.util.Collection, which is required for Collection::stream reference. Please add
import java.util.Collection;at the top of the file.
.flatMap(Collection::stream)
seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/AbstractPluginDiscovery.java:208
- The variable
pluginJarsis undefined in this scope, causing a compile error. It should likely reference the definedpluginJarPathsor another valid list; please correct the variable name.
addURLToClassLoaderConsumer.accept(classLoader, (List<URL>) pluginJars);
...rc/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/AbstractJdbcCatalog.java
Outdated
Show resolved
Hide resolved
...a/org/apache/seatunnel/core/starter/flink/execution/FlinkAbstractPluginExecuteProcessor.java
Outdated
Show resolved
Hide resolved
seatunnel-plugin-discovery/src/test/resources/duplicate/connectors/plugin-mapping.properties
Outdated
Show resolved
Hide resolved
Hisoka-X
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @hawk9821
Purpose of this pull request
Extract the common connector-cdc-base module to reduce the size of the cdc JAR .
before


after
Does this PR introduce any user-facing change?
How was this patch tested?
Check list
New License Guide