Skip to content

Commit 28f2314

Browse files
zhilinli123zhilinli123
authored
[Docs][Connector-V2][Mysql] Refactor connector-v2 docs using unified format Mysql (#4590)
1 parent 85d9058 commit 28f2314

File tree

2 files changed

+331
-0
lines changed

2 files changed

+331
-0
lines changed

‎docs/en/connector-v2/sink/Mysql.md‎

Lines changed: 174 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,174 @@
1+
# MySQL
2+
3+
> JDBC Mysql Sink Connector
4+
5+
## Support Those Engines
6+
7+
> Spark<br/>
8+
> Flink<br/>
9+
> Seatunnel Zeta<br/>
10+
11+
## Key Features
12+
13+
- [x] [exactly-once](../../concept/connector-v2-features.md)
14+
- [x] [cdc](../../concept/connector-v2-features.md)
15+
16+
> Use `Xa transactions` to ensure `exactly-once`. So only support `exactly-once` for the database which is
17+
> support `Xa transactions`. You can set `is_exactly_once=true` to enable it.
18+
19+
## Description
20+
21+
Write data through jdbc. Support Batch mode and Streaming mode, support concurrent writing, support exactly-once
22+
semantics (using XA transaction guarantee).
23+
24+
## Supported DataSource Info
25+
26+
| Datasource | Supported Versions | Driver | Url | Maven |
27+
|------------|----------------------------------------------------------|--------------------------|---------------------------------------|---------------------------------------------------------------------------|
28+
| Mysql | Different dependency version has different driver class. | com.mysql.cj.jdbc.Driver | jdbc:mysql://localhost:3306:3306/test | [Download](https://mvnrepository.com/artifact/mysql/mysql-connector-java) |
29+
30+
## Database Dependency
31+
32+
> Please download the support list corresponding to 'Maven' and copy it to the '$SEATNUNNEL_HOME/plugins/jdbc/lib/' working directory<br/>
33+
> For example Mysql datasource: cp mysql-connector-java-xxx.jar $SEATNUNNEL_HOME/plugins/jdbc/lib/
34+
35+
## Data Type Mapping
36+
37+
| Mysql Data type | Seatunnel Data type |
38+
|-----------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------|
39+
| BIT(1)<br/>INT UNSIGNED | BOOLEAN |
40+
| TINYINT<br/>TINYINT UNSIGNED<br/>SMALLINT<br/>SMALLINT UNSIGNED<br/>MEDIUMINT<br/>MEDIUMINT UNSIGNED<br/>INT<br/>INTEGER<br/>YEAR | INT |
41+
| INT UNSIGNED<br/>INTEGER UNSIGNED<br/>BIGINT | BIGINT |
42+
| BIGINT UNSIGNED | DECIMAL(20,0) |
43+
| DECIMAL(x,y)(Get the designated column's specified column size.<38) | DECIMAL(x,y) |
44+
| DECIMAL(x,y)(Get the designated column's specified column size.>38) | DECIMAL(38,18) |
45+
| DECIMAL UNSIGNED | DECIMAL((Get the designated column's specified column size)+1,<br/>(Gets the designated column's number of digits to right of the decimal point.))) |
46+
| FLOAT<br/>FLOAT UNSIGNED | FLOAT |
47+
| DOUBLE<br/>DOUBLE UNSIGNED | DOUBLE |
48+
| CHAR<br/>VARCHAR<br/>TINYTEXT<br/>MEDIUMTEXT<br/>TEXT<br/>LONGTEXT<br/>JSON | STRING |
49+
| DATE | DATE |
50+
| TIME | TIME |
51+
| DATETIME<br/>TIMESTAMP | TIMESTAMP |
52+
| TINYBLOB<br/>MEDIUMBLOB<br/>BLOB<br/>LONGBLOB<br/>BINARY<br/>VARBINAR<br/>BIT(n) | BYTES |
53+
| GEOMETRY<br/>UNKNOWN | Not supported yet |
54+
55+
## Sink Options
56+
57+
| Name | Type | Required | Default | Description |
58+
|-------------------------------------------|---------|----------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
59+
| url | String | Yes | - | The URL of the JDBC connection. Refer to a case: jdbc:mysql://localhost:3306:3306/test |
60+
| driver | String | Yes | - | The jdbc class name used to connect to the remote data source,<br/> if you use MySQL the value is `com.mysql.cj.jdbc.Driver`. |
61+
| user | String | No | - | Connection instance user name |
62+
| password | String | No | - | Connection instance password |
63+
| query | String | No | - | Use this sql write upstream input datas to database. e.g `INSERT ...`,`query` have the higher priority |
64+
| database | String | No | - | Use this `database` and `table-name` auto-generate sql and receive upstream input datas write to database.<br/>This option is mutually exclusive with `query` and has a higher priority. |
65+
| table | String | No | - | Use database and this table-name auto-generate sql and receive upstream input datas write to database.<br/>This option is mutually exclusive with `query` and has a higher priority. |
66+
| primary_keys | Array | No | - | This option is used to support operations such as `insert`, `delete`, and `update` when automatically generate sql. |
67+
| support_upsert_by_query_primary_key_exist | Boolean | No | false | Choose to use INSERT sql, UPDATE sql to process update events(INSERT, UPDATE_AFTER) based on query primary key exists. This configuration is only used when database unsupport upsert syntax. **Note**: that this method has low performance |
68+
| connection_check_timeout_sec | Int | No | 30 | The time in seconds to wait for the database operation used to validate the connection to complete. |
69+
| max_retries | Int | No | 0 | The number of retries to submit failed (executeBatch) |
70+
| batch_size | Int | No | 1000 | For batch writing, when the number of buffered records reaches the number of `batch_size` or the time reaches `batch_interval_ms`<br/>, the data will be flushed into the database |
71+
| batch_interval_ms | Int | No | 1000 | For batch writing, when the number of buffers reaches the number of `batch_size` or the time reaches `batch_interval_ms`, the data will be flushed into the database |
72+
| is_exactly_once | Boolean | No | false | Whether to enable exactly-once semantics, which will use Xa transactions. If on, you need to<br/>set `xa_data_source_class_name`. |
73+
| xa_data_source_class_name | String | No | - | The xa data source class name of the database Driver, for example, mysql is `com.mysql.cj.jdbc.MysqlXADataSource`, and<br/>please refer to appendix for other data sources |
74+
| max_commit_attempts | Int | No | 3 | The number of retries for transaction commit failures |
75+
| transaction_timeout_sec | Int | No | -1 | The timeout after the transaction is opened, the default is -1 (never timeout). Note that setting the timeout may affect<br/>exactly-once semantics |
76+
| auto_commit | Boolean | No | true | Automatic transaction commit is enabled by default |
77+
| common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |
78+
79+
### Tips
80+
81+
> If partition_column is not set, it will run in single concurrency, and if partition_column is set, it will be executed in parallel according to the concurrency of tasks.
82+
83+
## Task Example
84+
85+
### Simple:
86+
87+
> This example defines a SeaTunnel synchronization task that automatically generates data through FakeSource and sends it to JDBC Sink. FakeSource generates a total of 16 rows of data (row.num=16), with each row having two fields, name (string type) and age (int type). The final target table is test_table will also be 16 rows of data in the table. Before run this job, you need create database test and table test_table in your mysql. And if you have not yet installed and deployed SeaTunnel, you need to follow the instructions in [Install SeaTunnel](../../start-v2/locally/deployment.md) to install and deploy SeaTunnel. And then follow the instructions in [Quick Start With SeaTunnel Engine](../../start-v2/locally/quick-start-seatunnel-engine.md) to run this job.
88+
89+
```
90+
# Defining the runtime environment
91+
env {
92+
# You can set flink configuration here
93+
execution.parallelism = 1
94+
job.mode = "BATCH"
95+
}
96+
97+
source {
98+
# This is a example source plugin **only for test and demonstrate the feature source plugin**
99+
FakeSource {
100+
parallelism = 1
101+
result_table_name = "fake"
102+
row.num = 16
103+
schema = {
104+
fields {
105+
name = "string"
106+
age = "int"
107+
}
108+
}
109+
}
110+
# If you would like to get more information about how to configure seatunnel and see full list of source plugins,
111+
# please go to https://seatunnel.apache.org/docs/category/source-v2
112+
}
113+
114+
transform {
115+
# If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
116+
# please go to https://seatunnel.apache.org/docs/category/transform-v2
117+
}
118+
119+
sink {
120+
jdbc {
121+
url = "jdbc:mysql://localhost:3306/test"
122+
driver = "com.mysql.cj.jdbc.Driver"
123+
user = "root"
124+
password = "123456"
125+
query = "insert into test_table(name,age) values(?,?)"
126+
}
127+
# If you would like to get more information about how to configure seatunnel and see full list of sink plugins,
128+
# please go to https://seatunnel.apache.org/docs/category/sink-v2
129+
}
130+
```
131+
132+
### Exactly-once :
133+
134+
> For accurate write scene we guarantee accurate once
135+
136+
```
137+
sink {
138+
jdbc {
139+
url = "jdbc:mysql://localhost:3306/test"
140+
driver = "com.mysql.cj.jdbc.Driver"
141+
142+
max_retries = 0
143+
user = "root"
144+
password = "123456"
145+
query = "insert into test_table(name,age) values(?,?)"
146+
147+
is_exactly_once = "true"
148+
149+
xa_data_source_class_name = "com.mysql.cj.jdbc.MysqlXADataSource"
150+
}
151+
}
152+
```
153+
154+
### CDC(Change Data Capture) Event
155+
156+
> CDC change data is also supported by us In this case, you need config database, table and primary_keys.
157+
158+
```
159+
sink {
160+
jdbc {
161+
url = "jdbc:mysql://localhost:3306/test"
162+
driver = "com.mysql.cj.jdbc.Driver"
163+
user = "root"
164+
password = "123456"
165+
166+
generate_sink_sql = true
167+
# You need to configure both database and table
168+
database = test
169+
table = sink_table
170+
primary_keys = ["id","name"]
171+
}
172+
}
173+
```
174+

0 commit comments

Comments
 (0)