Skip to content

Commit 0261f86

Browse files
hailin0hailin0
authored
[Feature][Zeta][Core] Support output log file of job (#7712)
1 parent 61d1964 commit 0261f86

File tree

6 files changed

+639
-19
lines changed

6 files changed

+639
-19
lines changed

‎config/log4j2.properties‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,22 @@ appender.consoleStderr.filter.acceptGteWarn.level = WARN
6161
appender.consoleStderr.filter.acceptGteWarn.onMatch = ACCEPT
6262
appender.consoleStderr.filter.acceptGteWarn.onMismatch = DENY
6363

64+
appender.routing.name = routingAppender
65+
appender.routing.type = Routing
66+
appender.routing.purge.type = IdlePurgePolicy
67+
appender.routing.purge.timeToLive = 60
68+
appender.routing.route.type = Routes
69+
appender.routing.route.pattern = $${ctx:ST-JID}
70+
appender.routing.route.system.type = Route
71+
appender.routing.route.system.key = $${ctx:ST-JID}
72+
appender.routing.route.system.ref = fileAppender
73+
appender.routing.route.job.type = Route
74+
appender.routing.route.job.appender.type = File
75+
appender.routing.route.job.appender.name = job-${ctx:ST-JID}
76+
appender.routing.route.job.appender.fileName = ${file_path}/job-${ctx:ST-JID}.log
77+
appender.routing.route.job.appender.layout.type = PatternLayout
78+
appender.routing.route.job.appender.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%-30.30c{1.}] [%t] - %m%n
79+
6480
appender.file.name = fileAppender
6581
appender.file.type = RollingFile
6682
appender.file.fileName = ${file_path}/${file_name}.log

‎docs/en/seatunnel-engine/logging.md‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
---
2+
sidebar_position: 14
3+
---
4+
5+
# Logging
6+
7+
All SeaTunnel Engine processes create a log text file that contains messages for various events happening in that process. These logs provide deep insights into the inner workings of SeaTunnel Engine, and can be used to detect problems (in the form of WARN/ERROR messages) and can help in debugging them.
8+
9+
The logging in SeaTunnel Engine uses the SLF4J logging interface. This allows you to use any logging framework that supports SLF4J, without having to modify the SeaTunnel Engine source code.
10+
11+
By default, Log4j 2 is used as the underlying logging framework.
12+
13+
## Structured logging
14+
15+
SeaTunnel Engine adds the following fields to MDC of most of the relevant log messages (experimental feature):
16+
17+
- Job ID
18+
- key: ST-JID
19+
- format: string
20+
21+
This is most useful in environments with structured logging and allows you to quickly filter the relevant logs.
22+
23+
The MDC is propagated by slf4j to the logging backend which usually adds it to the log records automatically (e.g. in log4j json layout). Alternatively, it can be configured explicitly - log4j pattern layout might look like this:
24+
25+
```properties
26+
[%X{ST-JID}] %c{0} %m%n.
27+
```
28+
29+
## Configuring Log4j2
30+
31+
Log4j 2 is controlled using property files.
32+
33+
The SeaTunnel Engine distribution ships with the following log4j properties files in the `confing` directory, which are used automatically if Log4j 2 is enabled:
34+
35+
- `log4j2_client.properties`: used by the command line client (e.g., `seatunnel.sh`)
36+
- `log4j2.properties`: used for SeaTunnel Engine server processes (e.g., `seatunnel-cluster.sh`)
37+
38+
By default, log files are output to the `logs` directory.
39+
40+
Log4j periodically scans this file for changes and adjusts the logging behavior if necessary. By default this check happens every 60 seconds and is controlled by the monitorInterval setting in the Log4j properties files.
41+
42+
### Configure to output separate log files for jobs
43+
44+
To output separate log files for each job, you can update the following configuration in the `log4j2.properties` file:
45+
46+
```properties
47+
...
48+
rootLogger.appenderRef.file.ref = routingAppender
49+
...
50+
51+
appender.file.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%-30.30c{1.}] [%t] - %m%n
52+
...
53+
```
54+
55+
This configuration generates separate log files for each job, for example:
56+
57+
```
58+
job-xxx1.log
59+
job-xxx2.log
60+
job-xxx3.log
61+
...
62+
```
63+
64+
### Configuring output mixed logs
65+
66+
*This configuration mode by default.*
67+
68+
To all job logs output into SeaTunnel Engine system log file, you can update the following configuration in the `log4j2.properties` file:
69+
70+
```properties
71+
...
72+
rootLogger.appenderRef.file.ref = fileAppender
73+
...
74+
75+
appender.file.layout.pattern = [%X{ST-JID}] %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%-30.30c{1.}] [%t] - %m%n
76+
...
77+
```
78+
79+
### Compatibility with Log4j1/Logback
80+
81+
SeaTunnel Engine automatically integrates Log framework bridge, allowing existing applications that work against Log4j1/Logback classes to continue working.
82+
83+
## Best practices for developers
84+
85+
You can create an SLF4J logger by calling `org.slf4j.LoggerFactory#LoggerFactory.getLogger` with the Class of your class as an argument.
86+
87+
Of course, you can also use `lombok` annotation `@Slf4j` to achieve the same effect.
88+
89+
```java
90+
import org.slf4j.Logger;
91+
import org.slf4j.LoggerFactory;
92+
93+
public class TestConnector {
94+
private static final Logger LOG = LoggerFactory.getLogger(TestConnector.class);
95+
96+
public static void main(String[] args) {
97+
LOG.info("Hello world!");
98+
}
99+
}
100+
```
101+
102+
In order to benefit most from SLF4J, it is recommended to use its placeholder mechanism. Using placeholders allows avoiding unnecessary string constructions in case that the logging level is set so high that the message would not be logged.
103+
104+
The syntax of placeholders is the following:
105+
106+
```java
107+
LOG.info("This message contains {} placeholders. {}", 1, "key1");
108+
```
109+
110+
Placeholders can also be used in conjunction with exceptions which shall be logged.
111+
112+
```java
113+
try {
114+
// some code
115+
} catch (Exception e) {
116+
LOG.error("An {} occurred", "error", e);
117+
}
118+
```

‎docs/zh/seatunnel-engine/logging.md‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
---
2+
sidebar_position: 14
3+
---
4+
5+
# 日志
6+
7+
每个 SeaTunnel Engine 进程都会创建一个日志文件，其中包含该进程中发生的各种事件的消息。这些日志提供了对 SeaTunnel Engine 内部工作原理的深入了解，可用于检测问题（以 WARN/ERROR 消息的形式）并有助于调试问题。
8+
9+
SeaTunnel Engine 中的日志记录使用 SLF4J 日志记录接口。这允许您使用任何支持 SLF4J 的日志记录框架，而无需修改 SeaTunnel Engine 源代码。
10+
11+
默认情况下，Log4j2 用作底层日志记录框架。
12+
13+
## 结构化信息
14+
15+
SeaTunnel Engine 向大多数相关日志消息的 MDC 添加了以下字段（实验性功能）：
16+
17+
- Job ID
18+
- key: ST-JID
19+
- format: string
20+
21+
这在具有结构化日志记录的环境中最为有用，允许您快速过滤相关日志。
22+
23+
MDC 由 slf4j 传播到日志后端，后者通常会自动将其添加到日志记录中（例如，在 log4j json 布局中）。或者，也可以明确配置 - log4j 模式布局可能如下所示：
24+
25+
```properties
26+
[%X{ST-JID}] %c{0} %m%n.
27+
```
28+
29+
## 配置 Log4j2
30+
31+
Log4j2 使用属性文件进行控制。
32+
33+
SeaTunnel Engine 发行版在 `confing` 目录中附带以下 log4j 属性文件，如果启用了 Log4j2，则会自动使用这些文件：
34+
35+
- `log4j2_client.properties`: 由命令行客户端使用 (e.g., `seatunnel.sh`)
36+
- `log4j2.properties`: 由 SeaTunnel 引擎服务使用 (e.g., `seatunnel-cluster.sh`)
37+
38+
默认情况下，日志文件输出到 `logs` 目录。
39+
40+
Log4j 会定期扫描上述文件以查找更改，并根据需要调整日志记录行为。默认情况下，此检查每 60 秒进行一次，由 Log4j 属性文件中的 monitorInterval 设置控制。
41+
42+
### 配置作业生成单独的日志文件
43+
44+
要为每个作业输出单独的日志文件，您可以更新 `log4j2.properties` 文件中的以下配置：
45+
46+
```properties
47+
...
48+
rootLogger.appenderRef.file.ref = routingAppender
49+
...
50+
51+
appender.file.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%-30.30c{1.}] [%t] - %m%n
52+
...
53+
```
54+
55+
此配置为每个作业生成单独的日志文件，例如：
56+
57+
```
58+
job-xxx1.log
59+
job-xxx2.log
60+
job-xxx3.log
61+
...
62+
```
63+
64+
### 配置混合日志文件
65+
66+
*默认已采用此配置模式。*
67+
68+
要将所有作业日志输出到 SeaTunnel Engine 系统日志文件中，您可以在 `log4j2.properties` 文件中更新以下配置：
69+
70+
```properties
71+
...
72+
rootLogger.appenderRef.file.ref = fileAppender
73+
...
74+
75+
appender.file.layout.pattern = [%X{ST-JID}] %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%-30.30c{1.}] [%t] - %m%n
76+
...
77+
```
78+
79+
### 兼容 Log4j1/Logback
80+
81+
SeaTunnel Engine 自动集成了大多数 Log 桥接器，允许针对 Log4j1/Logback 类工作的现有应用程序继续工作。
82+
83+
## 开发人员最佳实践
84+
85+
您可以通过调用 `org.slf4j.LoggerFactory#LoggerFactory.getLogger` 并以您的类的类作为参数来创建 SLF4J 记录器。
86+
87+
当然您也可以使用 lombok 注解 `@Slf4j` 来实现同样的效果
88+
89+
```java
90+
import org.slf4j.Logger;
91+
import org.slf4j.LoggerFactory;
92+
93+
public class TestConnector {
94+
private static final Logger LOG = LoggerFactory.getLogger(TestConnector.class);
95+
96+
public static void main(String[] args) {
97+
LOG.info("Hello world!");
98+
}
99+
}
100+
```
101+
102+
为了最大限度地利用 SLF4J，建议使用其占位符机制。使用占位符可以避免不必要的字符串构造，以防日志级别设置得太高而导致消息无法记录。
103+
104+
占位符的语法如下：
105+
106+
```java
107+
LOG.info("This message contains {} placeholders. {}", 1, "key1");
108+
```
109+
110+
占位符还可以与需要记录的异常结合使用
111+
112+
```java
113+
try {
114+
// some code
115+
} catch (Exception e) {
116+
LOG.error("An {} occurred", "error", e);
117+
}
118+
```

‎seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/java/org/apache/seatunnel/engine/e2e/ClusterSeaTunnelContainer.java‎

Lines changed: 0 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,6 @@
4747
import java.util.Map;
4848
import java.util.concurrent.TimeUnit;
4949
import java.util.concurrent.atomic.AtomicInteger;
50-
import java.util.stream.Stream;
5150

5251
import static io.restassured.RestAssured.given;
5352
import static org.apache.seatunnel.e2e.common.util.ContainerUtil.PROJECT_ROOT_PATH;
@@ -120,24 +119,6 @@ public void testSubmitJobWithCustomJobId() {
120119
i,
121120
paramJobName + "&jobId=" + CUSTOM_JOB_ID,
122121
true));
123-
124-
String serverLogs = server.getLogs();
125-
String secondServerLogs = secondServer.getLogs();
126-
Stream.of(
127-
// [862969647010611201] 2024-08-24 16:01:21,155 INFO
128-
// org.apache.seatunnel.connectors.seatunnel.fake.source.FakeSourceSplitEnumerator - Starting to calculate splits.
129-
"\\[862969647010611201\\].* INFO org\\.apache\\.seatunnel\\.connectors\\.seatunnel\\.fake\\.source\\.FakeSourceSplitEnumerator",
130-
// [862969647010611201] 2024-08-24 16:01:21,278 INFO
131-
// org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter
132-
// - subtaskIndex=0 rowIndex=63: SeaTunnelRow#tableId=fake
133-
// SeaTunnelRow#kind=INSERT : qOWCd, 1033892054, 671516661
134-
"\\[862969647010611201\\].* INFO org\\.apache\\.seatunnel\\.connectors\\.seatunnel\\.console\\.sink\\.ConsoleSinkWriter")
135-
.map(
136-
regex -> {
137-
Assertions.assertTrue(
138-
serverLogs.matches(regex) || secondServerLogs.matches(regex));
139-
return regex;
140-
});
141122
}
142123

143124
@Test

0 commit comments

Comments
 (0)