Skip to content

Commit 8e8c534

Browse files
authored
[Improve][Doc] Improve the connector-v2 develop doc (#8190)
1 parent 4d3448d commit 8e8c534

File tree

2 files changed

+36
-30
lines changed

2 files changed

+36
-30
lines changed

seatunnel-connectors-v2/README.md

Lines changed: 21 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -49,14 +49,19 @@ own connectors, you need to follow the steps below.
4949

5050
3.Create two packages corresponding to source and sink
5151

52-
​ package org.apache.seatunnel.connectors.seatunnel.{connector name}}.source
53-
54-
​ package org.apache.seatunnel.connectors.seatunnel.{connector name}}.sink
52+
package org.apache.seatunnel.connectors.seatunnel.{connector name}}.source
53+
package org.apache.seatunnel.connectors.seatunnel.{connector name}}.sink
5554

5655
4.add connector info to plugin-mapping.properties file in seatunnel root path.
5756

5857
5.add connector dependency to seatunnel-dist/pom.xml, so the connector jar can be find in binary package.
5958

59+
6.There are several classes that must be implemented on the source side, namely {ConnectorName}Source, {ConnectorName}SourceFactory, {ConnectorName}SourceReader; There are several classes that must be implemented on the sink side, namely {ConnectorName}Sink, {ConnectorName}SinkFactory, {ConnectorName}SinkWriter Please refer to other connectors for details
60+
61+
7.{ConnectorName}SourceFactory and {ConnectorName}SinkFactory needs to be annotated with the **@AutoService (Factory.class)** annotation on the class name, and in addition to the required methods, source side an additional **creatSource** method needs to be rewritten and sink side an additional **creatSink** method needs to be rewritten
62+
63+
8.{ConnectorName}Source needs to override the **getProducedCatalogTables** method; {ConnectorName}Sink needs to override the **getWriteCatalogTable** method
64+
6065
### **Startup Class**
6166

6267
Aside from the old startup class, we have created two new startup modules,
@@ -205,26 +210,25 @@ In order to automatically create the Source Connector and Sink Connector and Tra
205210
supported by the current connector and the required parameters. We define TableSourceFactory and TableSinkFactory,
206211
It is recommended to put it in the same directory as the implementation class of SeaTunnelSource or SeaTunnelSink for easy searching.
207212

208-
- `factoryIdentifier` is used to indicate the name of the current Factory. This value should be the same as the
209-
value returned by `getPluginName`, so that if Factory is used to create Source/Sink in the future,
210-
A seamless switch can be achieved.
211-
- `createSink` and `createSource` are the methods for creating Source and Sink respectively,
212-
and do not need to be implemented at present.
213+
- `factoryIdentifier` is used to indicate the name of the current Factory. This value should be the same as the
214+
value returned by `getPluginName`, so that if Factory is used to create Source/Sink in the future,
215+
A seamless switch can be achieved.
216+
- `createSink` and `createSource` are the methods for creating Source and Sink respectively.
213217
- `optionRule` returns the parameter logic, which is used to indicate which parameters of our connector are supported,
214-
which parameters are required, which parameters are optional, and which parameters are exclusive, which parameters are bundledRequired.
215-
This method will be used when we visually create the connector logic, and it will also be used to generate a complete parameter
216-
object according to the parameters configured by the user, and then the connector developer does not need to judge whether the parameters
217-
exist one by one in the Config, and use it directly That's it.
218-
You can refer to existing implementations, such as `org.apache.seatunnel.connectors.seatunnel.elasticsearch.source.ElasticsearchSourceFactory`.
219-
There is support for configuring Schema for many Sources, so a common Option is used.
220-
If you need a schema, you can refer to `org.apache.seatunnel.api.table.catalog.CatalogTableUtil.SCHEMA`.
218+
which parameters are required, which parameters are optional, and which parameters are exclusive, which parameters are bundledRequired.
219+
This method will be used when we visually create the connector logic, and it will also be used to generate a complete parameter
220+
object according to the parameters configured by the user, and then the connector developer does not need to judge whether the parameters
221+
exist one by one in the Config, and use it directly That's it.
222+
You can refer to existing implementations, such as `org.apache.seatunnel.connectors.seatunnel.elasticsearch.source.ElasticsearchSourceFactory`.
223+
There is support for configuring Schema for many Sources, so a common Option is used.
224+
If you need a schema, you can refer to `org.apache.seatunnel.api.table.catalog.CatalogTableUtil.SCHEMA`.
221225

222226
Don't forget to add `@AutoService(Factory.class)` to the class. This Factory is the parent class of TableSourceFactory and TableSinkFactory.
223227

224228
### **Options**
225229

226230
When we implement TableSourceFactory and TableSinkFactory, the corresponding Option will be created.
227-
Each Option corresponds to a configuration, but different configurations will have different types.
231+
Each Option corresponds to a configuration, but different configurations will have different types.
228232
Common types can be created by directly calling the corresponding method.
229233
But if our parameter type is an object, we can use POJO to represent parameters of object type,
230234
and need to use `org.apache.seatunnel.api.configuration.util.OptionMark` on each parameter to indicate that this is A child Option.
@@ -237,6 +241,4 @@ please refer to `org.apache.seatunnel.connectors.seatunnel.assertion.sink.Assert
237241
## **Result**
238242

239243
All Connector implementations should be under the ``seatunnel-connectors-v2``, and the examples that can be referred to
240-
at this stage are under this module.
241-
242-
244+
at this stage are under this module.

seatunnel-connectors-v2/README.zh.md

Lines changed: 15 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ SeaTunnel为与计算引擎进行解耦,设计了新的连接器API,通过
1010
### **工程结构**
1111

1212
- ../`seatunnel-connectors-v2` connector-v2代码实现
13-
- ../`seatunnel-translation` connector-v2的翻译层
13+
- ../`seatunnel-translation` connector-v2的翻译层
1414
- ../`seatunnel-transform-v2` transform-v2代码实现
1515
- ../seatunnel-e2e/`seatunnel-connector-v2-e2e` connector-v2端到端测试
1616
- ../seatunnel-examples/`seatunnel-flink-connector-v2-example` seatunnel connector-v2的flink local运行的实例
@@ -39,14 +39,19 @@ SeaTunnel为与计算引擎进行解耦,设计了新的连接器API,通过
3939

4040
3.新建两个package分别对应source和sink
4141

42-
​ package org.apache.seatunnel.connectors.seatunnel.{连接器名}.source
43-
44-
​ package org.apache.seatunnel.connectors.seatunnel.{连接器名}.sink
42+
package org.apache.seatunnel.connectors.seatunnel.{连接器名}.source
43+
package org.apache.seatunnel.connectors.seatunnel.{连接器名}.sink
4544

4645
4.将连接器信息添加到在项目根目录的plugin-mapping.properties文件中.
4746

4847
5.将连接器添加到seatunnel-dist/pom.xml,这样连接器jar就可以在二进制包中找到.
4948

49+
6.source端有几个必须实现的类,分别是{连接器名}Source、{连接器名}SourceFactory、{连接器名}SourceReader;sink端有几个必须实现的类,分别是{连接器名}Sink、{连接器名}SinkFactory、{连接器名}SinkWriter,具体可以参考其他连接器
50+
51+
7.{连接器名}SourceFactory 和 {连接器名}SinkFactory 里面需要在类名上标注 **@AutoService(Factory.class)** 注解,并且除了必须实现的方法外,source端需要额外再重写一个 **createSource** 方法,sink端需要额外再重写一个 **createSink** 方法
52+
53+
8.{连接器名}Source 需要重写 **getProducedCatalogTables** 方法;{连接器名}Sink 需要重写 **getWriteCatalogTable** 方法
54+
5055
### 启动类
5156

5257
和老的启动类分开,我们创建了两个新的启动类工程,分别是`seatunnel-core/seatunnel-flink-starter``seatunnel-core/seatunnel-spark-starter`.
@@ -154,13 +159,12 @@ Sink可以根据组件属性进行选择,到底是只实现`SinkCommitter`或`
154159
为了实现自动化的创建Source或者Sink,我们需要连接器能够声明并返回创建他们所需要的参数列表和每个参数的校验规则。为了实现这个目标,我们定义了TableSourceFactory和TableSinkFactory,
155160
建议将其放在和SeaTunnelSource或SeaTunnelSink实现类同一目录下,方便寻找。
156161

157-
- `factoryIdentifier` 用于表明当前Factory的名称,这个值应该和`getPluginName`返回的值一致,这样后续如果使用Factory来创建Source/Sink,
158-
就能实现无缝切换。
159-
- `createSink``createSource` 分别是创建Source和Sink的方法,目前不用实现。
162+
- `factoryIdentifier` 用于表明当前Factory的名称,这个值应该和`getPluginName`返回的值一致,这样后续如果使用Factory来创建Source/Sink,就能实现无缝切换。
163+
- `createSink``createSource` 分别是创建Source和Sink的方法。
160164
- `optionRule` 返回的是参数逻辑,用于表示我们的连接器参数哪些支持,哪些参数是必须(required)的,哪些参数是可选(optional)的,哪些参数是互斥(exclusive)的,哪些参数是绑定(bundledRequired)的。
161-
这个方法会在我们可视化创建连接器逻辑的时候用到,同时也会用于根据用户配置的参数生成完整的参数对象,然后连接器开发者就不用在Config里面一个个判断参数是否存在,直接使用即可。
162-
可以参考现有的实现,比如`org.apache.seatunnel.connectors.seatunnel.elasticsearch.source.ElasticsearchSourceFactory`。针对很多Source都有支持配置Schema,所以采用了通用的Option,
163-
需要Schema则可以引用`org.apache.seatunnel.api.table.catalog.CatalogTableUtil.SCHEMA`
165+
这个方法会在我们可视化创建连接器逻辑的时候用到,同时也会用于根据用户配置的参数生成完整的参数对象,然后连接器开发者就不用在Config里面一个个判断参数是否存在,直接使用即可。
166+
可以参考现有的实现,比如`org.apache.seatunnel.connectors.seatunnel.elasticsearch.source.ElasticsearchSourceFactory`。针对很多Source都有支持配置Schema,所以采用了通用的Option,
167+
需要Schema则可以引用`org.apache.seatunnel.api.table.catalog.CatalogTableUtil.SCHEMA`
164168

165169
别忘记添加`@AutoService(Factory.class)` 到类上面。这个Factory即TableSourceFactory 和 TableSinkFactory的父类。
166170

@@ -173,4 +177,4 @@ Sink可以根据组件属性进行选择,到底是只实现`SinkCommitter`或`
173177

174178
## 实现
175179

176-
现阶段所有的连接器实现及可参考的示例都在seatunnel-connectors-v2下,用户可自行查阅参考。
180+
现阶段所有的连接器实现及可参考的示例都在seatunnel-connectors-v2下,用户可自行查阅参考。

0 commit comments

Comments
 (0)