You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/en/other-engine/flink.md
+7-7Lines changed: 7 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,8 +1,8 @@
1
-
# Seatunnel runs on Flink
1
+
# Seatunnel Runs On Flink
2
2
3
-
Flink is a powerful high-performance distributed stream processing engine,More information about it you can,You can search for `Apache Flink`
3
+
Flink is a powerful high-performance distributed stream processing engine. More information about it you can search for `Apache Flink`
4
4
5
-
### Set Flink configuration information in the job
5
+
### Set Flink Configuration Information In The Job
6
6
7
7
Begin with `flink.`
8
8
@@ -19,9 +19,9 @@ env {
19
19
Enumeration types are not currently supported, you need to specify them in the Flink conf file ,Only these types of Settings are supported for the time being:<br/>
20
20
Integer/Boolean/String/Duration
21
21
22
-
### How to set up a simple Flink job
22
+
### How To Set Up A Simple Flink Job
23
23
24
-
This is a simple job that runs on Flink Randomly generated data is printed to the console
24
+
This is a simple job that runs on Flink. Randomly generated data is printed to the console
25
25
26
26
```
27
27
env {
@@ -79,6 +79,6 @@ sink{
79
79
}
80
80
```
81
81
82
-
### How to run a job in a project
82
+
### How To Run A Job In A Project
83
83
84
-
After you pull the code to the local, go to the `seatunnel-examples/seatunnel-flink-connector-v2-example` module find `org.apache.seatunnel.example.flink.v2.SeaTunnelApiExample`To complete the operation of the job
84
+
After you pull the code to the local, go to the `seatunnel-examples/seatunnel-flink-connector-v2-example` module and find `org.apache.seatunnel.example.flink.v2.SeaTunnelApiExample`to complete the operation of the job.
Copy file name to clipboardExpand all lines: docs/en/seatunnel-engine/about.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,21 +18,21 @@ In the future, SeaTunnel Engine will further optimize its functions to support f
18
18
19
19
### Cluster Management
20
20
21
-
- Support stand-alone operation;
21
+
- Support standalone operation;
22
22
- Support cluster operation;
23
23
- Support autonomous cluster (decentralized), which saves the users from specifying a master node for the SeaTunnel Engine cluster, because it can select a master node by itself during operation, and a new master node will be chosen automatically when the master node fails.
24
24
- Autonomous Cluster nodes-discovery and nodes with the same cluster_name will automatically form a cluster.
25
25
26
26
### Core functions
27
27
28
-
- Supports running jobs in local mode, and the cluster is automatically destroyed after the job once completed;
29
-
- Supports running jobs in Cluster mode (single machine or cluster), submitting jobs to the SeaTunnel Engine service through the SeaTunnel Client, and the service continues to run after the job is completed and waits for the next job submission;
28
+
- Support running jobs in local mode, and the cluster is automatically destroyed after the job once completed;
29
+
- Support running jobs in cluster mode (single machine or cluster), submitting jobs to the SeaTunnel Engine service through the SeaTunnel client, and the service continues to run after the job is completed and waits for the next job submission;
30
30
- Support offline batch synchronization;
31
31
- Support real-time synchronization;
32
32
- Batch-stream integration, all SeaTunnel V2 connectors can run in SeaTunnel Engine;
33
-
- Supports distributed snapshot algorithm, and supports two-stage submission with SeaTunnel V2 connector, ensuring that data is executed only once.
34
-
- Support job invocation at the Pipeline level to ensure that it can be started even when resources are limited;
35
-
- Supports fault tolerance for jobs at the Pipeline level. Task failure only affects the Pipeline where it is located, and only the task under the Pipeline needs to be rolled back;
33
+
- Support distributed snapshot algorithm, and supports two-stage submission with SeaTunnel V2 connector, ensuring that data is executed only once.
34
+
- Support job invocation at the pipeline level to ensure that it can be started even when resources are limited;
35
+
- Support fault tolerance for jobs at the Pipeline level. Task failure only affects the pipeline where it is located, and only the task under the Pipeline needs to be rolled back;
36
36
- Support dynamic thread sharing to synchronize a large number of small data sets in real-time.
Copy file name to clipboardExpand all lines: docs/en/seatunnel-engine/checkpoint-storage.md
+14-14Lines changed: 14 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,11 +18,11 @@ SeaTunnel Engine supports the following checkpoint storage types:
18
18
- HDFS (OSS,S3,HDFS,LocalFile)
19
19
- LocalFile (native), (it's deprecated: use Hdfs(LocalFile) instead.
20
20
21
-
We used the microkernel design pattern to separate the checkpoint storage module from the engine. This allows users to implement their own checkpoint storage modules.
21
+
We use the microkernel design pattern to separate the checkpoint storage module from the engine. This allows users to implement their own checkpoint storage modules.
22
22
23
23
`checkpoint-storage-api`is the checkpoint storage module API, which defines the interface of the checkpoint storage module.
24
24
25
-
if you want to implement your own checkpoint storage module, you need to implement the `CheckpointStorage` and provide the corresponding `CheckpointStorageFactory` implementation.
25
+
If you want to implement your own checkpoint storage module, you need to implement the `CheckpointStorage` and provide the corresponding `CheckpointStorageFactory` implementation.
26
26
27
27
### Checkpoint Storage Configuration
28
28
@@ -46,12 +46,12 @@ Notice: namespace must end with "/".
46
46
47
47
#### OSS
48
48
49
-
Aliyun oss base on hdfs-file, so you can refer [hadoop oss docs](https://hadoop.apache.org/docs/stable/hadoop-aliyun/tools/hadoop-aliyun/index.html) to config oss.
49
+
Aliyun OSS based hdfs-fileyou can refer [Hadoop OSS Docs](https://hadoop.apache.org/docs/stable/hadoop-aliyun/tools/hadoop-aliyun/index.html) to config oss.
50
50
51
51
Except when interacting with oss buckets, the oss client needs the credentials needed to interact with buckets.
52
52
The client supports multiple authentication mechanisms and can be configured as to which mechanisms to use, and their order of use. Custom implementations of org.apache.hadoop.fs.aliyun.oss.AliyunCredentialsProvider may also be used.
53
-
if you used AliyunCredentialsProvider (can be obtained from the Aliyun Access Key Management), these consist of an access key, a secret key.
54
-
you can config like this:
53
+
If you used AliyunCredentialsProvider (can be obtained from the Aliyun Access Key Management), these consist of an access key, a secret key.
For additional reading on the Hadoop Credential Provider API see: [Credential Provider API](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html).
74
+
For additional reading on the Hadoop Credential Provider API, you can see: [Credential Provider API](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html).
For Aliyun OSS Credential Provider implements, you can see: [Auth Credential Providers](https://github.com/aliyun/aliyun-oss-java-sdk/tree/master/src/main/java/com/aliyun/oss/common/auth)
77
77
78
78
#### S3
79
79
80
-
S3 base on hdfs-file, so you can refer [hadoop s3 docs](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html) to config s3.
80
+
S3 based hdfs-file you can refer [hadoop s3 docs](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html) to config s3.
81
81
82
82
Except when interacting with public S3 buckets, the S3A client needs the credentials needed to interact with buckets.
83
83
The client supports multiple authentication mechanisms and can be configured as to which mechanisms to use, and their order of use. Custom implementations of com.amazonaws.auth.AWSCredentialsProvider may also be used.
84
-
if you used SimpleAWSCredentialsProvider (can be obtained from the Amazon Security Token Service), these consist of an access key, a secret key.
85
-
you can config like this:
84
+
If you used SimpleAWSCredentialsProvider (can be obtained from the Amazon Security Token Service), these consist of an access key, a secret key.
85
+
You can config like this:
86
86
87
87
```yaml
88
88
@@ -104,8 +104,8 @@ seatunnel:
104
104
105
105
```
106
106
107
-
if you used `InstanceProfileCredentialsProvider`, this supports use of instance profile credentials if running in an EC2 VM, you could check [iam-roles-for-amazon-ec2](https://docs.aws.amazon.com/zh_cn/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html).
108
-
you can config like this:
107
+
If you used `InstanceProfileCredentialsProvider`, which supports use of instance profile credentials if running in an EC2 VM, you can check [iam-roles-for-amazon-ec2](https://docs.aws.amazon.com/zh_cn/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html).
108
+
You can config like this:
109
109
110
110
```yaml
111
111
@@ -146,11 +146,11 @@ seatunnel:
146
146
# important: The user of this key needs to have write permission for the bucket, otherwise an exception of 403 will be returned
147
147
```
148
148
149
-
For additional reading on the Hadoop Credential Provider API see: [Credential Provider API](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html).
149
+
For additional reading on the Hadoop Credential Provider API, you can see: [Credential Provider API](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html).
Copy file name to clipboardExpand all lines: docs/en/seatunnel-engine/deployment.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,18 +7,18 @@ sidebar_position: 3
7
7
8
8
SeaTunnel Engine(Zeta) supports three different deployment modes: local mode, hybrid cluster mode, and separated cluster mode.
9
9
10
-
Each deployment mode has different usage scenarios, advantages, and disadvantages. When choosing a deployment mode, you should choose according to your needs and environment.
10
+
Each deployment mode has different usage scenarios, advantages, and disadvantages. You should choose a deployment mode according to your needs and environment.
11
11
12
12
**Local mode:** Only used for testing, each task will start an independent process, and the process will exit after the task is completed.
13
13
14
14
**Hybrid cluster mode:** The Master service and Worker service of SeaTunnel Engine are mixed in the same process. All nodes can run jobs and participate in the election to become the master, that is, the master node is also running synchronous tasks simultaneously. In this mode, Imap (saving the state information of the task to provide support for the fault tolerance of the task) data will be distributed among all nodes.
15
15
16
16
**Separated cluster mode(experimental feature):** The Master service and Worker service of SeaTunnel Engine are separated, and each service is a single process. The Master node is only responsible for job scheduling, rest api, task submission, etc., and Imap data is only stored in the Master node. The Worker node is only responsible for the execution of the task, does not participate in the election to become the master, and does not store Imap data.
17
17
18
-
**Usage suggestion:** Although [separated cluster mode](separated-cluster-deployment.md) is an experimental feature, the first recommended usage will be made in the future. In the hybrid cluster mode, the Master node needs to run tasks synchronously. When the task scale is large, it will affect the stability of the Master node. Once the Master node crashes or the heartbeat times out, it will lead to the switch of the Master node, and the switch of the Master node will cause fault tolerance of all running tasks, which will further increase the load of the cluster. Therefore, we recommend using the separated mode more.
18
+
**Usage suggestion:** Although [Separated Cluster Mode](separated-cluster-deployment.md) is an experimental feature, the first recommended usage will be made in the future. In the hybrid cluster mode, the Master node needs to run tasks synchronously. When the task scale is large, it will affect the stability of the Master node. Once the Master node crashes or the heartbeat times out, it will lead to the switch of the Master node, and the switch of the Master node will cause fault tolerance of all running tasks, which will further increase the load of the cluster. Therefore, we recommend using the separated mode more.
Copy file name to clipboardExpand all lines: docs/en/seatunnel-engine/download-seatunnel.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,7 +6,7 @@ sidebar_position: 2
6
6
import Tabs from '@theme/Tabs';
7
7
import TabItem from '@theme/TabItem';
8
8
9
-
# Download and Make Installation Packages
9
+
# Download And Make Installation Packages
10
10
11
11
## Step 1: Preparation
12
12
@@ -16,7 +16,7 @@ Before starting to download SeaTunnel, you need to ensure that you have installe
16
16
17
17
## Step 2: Download SeaTunnel
18
18
19
-
Go to the [seatunnel download page](https://seatunnel.apache.org/download) to download the latest version of the release version installation package `seatunnel-<version>-bin.tar.gz`.
19
+
Go to the [Seatunnel Download Page](https://seatunnel.apache.org/download) to download the latest version of the release version installation package `seatunnel-<version>-bin.tar.gz`.
tar -xzvf "apache-seatunnel-${version}-bin.tar.gz"
27
27
```
28
28
29
-
## Step 3: Download the connector plug-in
29
+
## Step 3: Download The Connector Plugin
30
30
31
31
Starting from the 2.2.0-beta version, the binary package no longer provides the connector dependency by default. Therefore, when using it for the first time, you need to execute the following command to install the connector: (Of course, you can also manually download the connector from the [Apache Maven Repository](https://repo.maven.apache.org/maven2/org/apache/seatunnel/), and then move it to the `connectors/seatunnel` directory).
32
32
33
33
```bash
34
-
sh bin/install-plugin.sh 2.3.6
34
+
sh bin/install-plugin.sh
35
35
```
36
36
37
37
If you need a specific connector version, taking 2.3.6 as an example, you need to execute the following command.
@@ -65,6 +65,6 @@ If you want to install connector plugins by manually downloading connectors, you
65
65
66
66
:::
67
67
68
-
Now you have completed the download of the SeaTunnel installation package and the download of the connector plug-in. Next, you can choose different running modes according to your needs to run or deploy SeaTunnel.
68
+
Now you have completed the download of the SeaTunnel installation package and the download of the connector plugin. Next, you can choose different running modes according to your needs to run or deploy SeaTunnel.
69
69
70
-
If you use the SeaTunnel Engine (Zeta) that comes with SeaTunnel to run tasks, you need to deploy the SeaTunnel Engine service first. Refer to [Deployment of SeaTunnel Engine (Zeta) Service](deployment.md).
70
+
If you use the SeaTunnel Engine (Zeta) that comes with SeaTunnel to run tasks, you need to deploy the SeaTunnel Engine service first. Refer to [Deployment Of SeaTunnel Engine (Zeta) Service](deployment.md).
0 commit comments