Skip to content
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 36 additions & 27 deletions docs/en/concept/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,8 +205,19 @@ sql = """ select * from "table" """

## Config Variable Substitution

In config file we can define some variables and replace it in run time. **This is only support `hocon` format file**.
In a configuration file, we can define variables and replace them at runtime. However, note that only HOCON format files are supported.

### Usage of Variables:
- `${varName}`: If the variable is not provided, an exception will be thrown.
- `${varName:default}`: If the variable is not provided, the default value will be used. If you set a default value, it should be enclosed in double quotes.
- `${varName:}`: If the variable is not provided, an empty string will be used.

If you set a variable in the configuration file but do not pass it during execution, an exception will be thrown. Example:
```shell
Caused by: org.apache.seatunnel.core.starter.exception.CommandExecuteException: Variable substitution error: ${resName}_table
```

### Example:
```hocon
env {
job.mode = "BATCH"
Expand All @@ -216,24 +227,24 @@ env {

source {
FakeSource {
result_table_name = ${resName}
row.num = ${rowNum}
result_table_name = "${resName:fake_test}_table"
row.num = "${rowNum:50}"
string.template = ${strTemplate}
int.template = [20, 21]
schema = {
fields {
name = ${nameType}
age = "int"
name = "${nameType:string}"
age = ${ageType}
}
}
}
}

transform {
sql {
source_table_name = "fake"
source_table_name = "${resName:fake_test}_table"
result_table_name = "sql"
query = "select * from "${resName}" where name = '"${nameVal}"' "
query = "select * from ${resName:fake_test}_table where name = '${nameVal}' "
}

}
Expand All @@ -245,26 +256,24 @@ sink {
password = ${password}
}
}

```

In the above config, we define some variables, like `${rowNum}`, `${resName}`.
We can replace those parameters with this shell command:
In the configuration above, we have defined several variables like `${rowNum}`, `${resName}`. We can replace these parameters using the following shell command:

```shell
./bin/seatunnel.sh -c <this_config_file>
-i jobName='this_is_a_job_name'
-i resName=fake
-i rowNum=10
-i strTemplate=['abc','d~f','hi']
-i nameType=string
-i ageType=int
-i nameVal=abc
-i username=seatunnel=2.3.1
-i password='$a^b%c.d~e0*9('
-m local
```

Then the final submitted config is:
In this case, `resName`, `rowNum`, and `nameType` are not set, so they will take their default values.

The final submitted configuration would be:

```hocon
env {
Expand All @@ -275,13 +284,13 @@ env {

source {
FakeSource {
result_table_name = "fake"
row.num = 10
string.template = ["abc","d~f","h i"]
result_table_name = "fake_test_table"
row.num = 50
string.template = ['abc','d~f','hi']
int.template = [20, 21]
schema = {
fields {
name = string
name = "string"
age = "int"
}
}
Expand All @@ -290,9 +299,9 @@ source {

transform {
sql {
source_table_name = "fake"
source_table_name = "fake_test_table"
result_table_name = "sql"
query = "select * from fake where name = 'abc' "
query = "select * from fake_test_table where name = 'abc' "
}

}
Expand All @@ -302,16 +311,16 @@ sink {
source_table_name = "sql"
username = "seatunnel=2.3.1"
password = "$a^b%c.d~e0*9("
}
}
}
```

Some Notes:
- Quota with `'` if the value has special character such as `(`
- If the replacement variables is in `"` or `'`, like `resName` and `nameVal`, you need add `"`
- The value can't have space `' '`, like `-i jobName='this is a job name' `, this will be replaced to `job.name = "this"`
- If you want to use dynamic parameters, you can use the following format: -i date=$(date +"%Y%m%d").

### Important Notes:
- If a value contains special characters like `(`, enclose it in single quotes (`'`).
- If the substitution variable contains double or single quotes (e.g., `"resName"` or `"nameVal"`), you need to include them with the value.
- The value cannot contain spaces (`' '`). For example, `-i jobName='this is a job name'` will be replaced with `job.name = "this"`.
- For dynamic parameters, you can use the following format: `-i date=$(date +"%Y%m%d")`.
-
## What's More

- Start write your own config file now, choose the [connector](../connector-v2/source) you want to use, and configure the parameters according to the connector's documentation.
Expand Down
44 changes: 28 additions & 16 deletions docs/zh/concept/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,18 @@ sql = """ select * from "table" """

在配置文件中,我们可以定义一些变量并在运行时替换它们。但是注意仅支持 hocon 格式的文件。

变量使用方法:
- `${varName}`,如果变量未传值,则抛出异常。
- `${varName:default}`,如果变量未传值,则使用默认值。如果设置默认值则变量需要写在双引号中。
- `${varName:}`,如果变量未传值,则使用空字符串。

如果您在配置文件中设置变量变量,但在执行中未传递,则会抛出异常。
如:
```shell
Caused by: org.apache.seatunnel.core.starter.exception.CommandExecuteException: Variable substitution error: ${resName}_table
```

具体样例:
```hocon
env {
job.mode = "BATCH"
Expand All @@ -201,24 +213,24 @@ env {

source {
FakeSource {
result_table_name = ${resName}
row.num = ${rowNum}
result_table_name = "${resName:fake_test}_table"
row.num = "${rowNum:50}"
string.template = ${strTemplate}
int.template = [20, 21]
schema = {
fields {
name = ${nameType}
age = "int"
name = "${nameType:string}"
age = ${ageType}
}
}
}
}

transform {
sql {
source_table_name = "fake"
source_table_name = "${resName:fake_test}_table"
result_table_name = "sql"
query = "select * from "${resName}" where name = '"${nameVal}"' "
query = "select * from ${resName:fake_test}_table where name = '${nameVal}' "
}

}
Expand All @@ -230,7 +242,6 @@ sink {
password = ${password}
}
}

```

在上述配置中,我们定义了一些变量,如 ${rowNum}、${resName}。
Expand All @@ -239,16 +250,17 @@ sink {
```shell
./bin/seatunnel.sh -c <this_config_file>
-i jobName='this_is_a_job_name'
-i resName=fake
-i rowNum=10
-i strTemplate=['abc','d~f','hi']
-i nameType=string
-i ageType=int
-i nameVal=abc
-i username=seatunnel=2.3.1
-i password='$a^b%c.d~e0*9('
-e local
-m local
```

其中 `resName`,`rowNum`,`nameType` 我们未设置,他将获取默认值


然后最终提交的配置是:

```hocon
Expand All @@ -260,8 +272,8 @@ env {

source {
FakeSource {
result_table_name = "fake"
row.num = 10
result_table_name = "fake_test_table"
row.num = 50
string.template = ['abc','d~f','hi']
int.template = [20, 21]
schema = {
Expand All @@ -275,9 +287,9 @@ source {

transform {
sql {
source_table_name = "fake"
source_table_name = "fake_test_table"
result_table_name = "sql"
query = "select * from "fake" where name = 'abc' "
query = "select * from fake_test_table where name = 'abc' "
}

}
Expand All @@ -286,7 +298,7 @@ sink {
Console {
source_table_name = "sql"
username = "seatunnel=2.3.1"
password = "$a^b%c.d~e0*9("
password = "$a^b%c.d~e0*9("
}
}

Expand Down
Loading