Commit 1800d15
authored
Always remove unecessary software from github runners for all jobs (fix intermittent out of space on runners) (#19122)
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Closes #19121
## Rationale for this change
- DataFusion requires a lot of diskspace (rust compilation, etc)
- Github often adds significant overhead as well with pre-installed
software
- Some specific jobs remove unecessary pre-installed software, but it is
not applied to all jobs (and so new jobs fail from time to time, such as
#19121)
## What changes are included in this PR?
1. Let's apply the cleanup to all jobs
Note I also looked into any other potential space savings, and I can't
really find any (there is nothing else obvious to delete). Here is some
`du` sleuthing from [this
run](https://github.com/apache/datafusion/actions/runs/19971702815/job/57277837847?pr=19122):
```
info: installing component 'rustfmt'
info: component 'rustfmt' for target 'x86_64-unknown-linux-gnu' is up to date
Prepare all required actions
Run ./.github/actions/setup-rust-runtime
Run echo "RUST_BACKTRACE=1" >> $GITHUB_ENV
Run git config --global --add safe.directory "$GITHUB_WORKSPACE"
Run echo "Disk space before cleanup:"
Disk space before cleanup:
Filesystem Size Used Avail Use% Mounted on
overlay 72G 56G 16G 79% /
tmpfs 64M 0 64M 0% /dev
shm 64M 0 64M 0% /dev/shm
/dev/root 72G 56G 16G 79% /__w
tmpfs 3.2G 1.2M 3.2G 1% /run/docker.sock
tmpfs 7.9G 0 7.9G 0% /proc/acpi
tmpfs 7.9G 0 7.9G 0% /proc/scsi
tmpfs 7.9G 0 7.9G 0% /sys/firmware
605M /__e
5.8G /__t
59M /__w
0 /bin
4.0K /boot
0 /dev
2.1M /etc
4.0K /github
4.0K /home
0 /lib
0 /lib64
4.0K /media
4.0K /mnt
4.0K /opt
du: cannot read directory '/proc/136/task/136/net': Invalid argument
du: cannot read directory '/proc/136/net': Invalid argument
du: cannot read directory '/proc/193/task/193/net': Invalid argument
du: cannot read directory '/proc/193/net': Invalid argument
du: cannot access '/proc/361/task/361/fd/4': No such file or directory
du: cannot access '/proc/361/task/361/fdinfo/4': No such file or directory
du: cannot access '/proc/361/fd/3': No such file or directory
du: cannot access '/proc/361/fdinfo/3': No such file or directory
0 /proc
12K /root
8.0K /run
0 /sbin
4.0K /srv
0 /sys
4.0K /tmp
2.8G /usr
37M /var
114M /usr/bin
4.0K /usr/games
57M /usr/include
530M /usr/lib
4.0K /usr/lib64
101M /usr/libexec
1.8G /usr/local
5.0M /usr/sbin
233M /usr/share
4.0K /usr/src
4.0K /usr/local/bin
20M /usr/local/cargo
4.0K /usr/local/etc
4.0K /usr/local/games
4.0K /usr/local/include
12K /usr/local/lib
4.0K /usr/local/libexec
0 /usr/local/man
1.8G /usr/local/rustup
4.0K /usr/local/sbin
16K /usr/local/share
4.0K /usr/local/src
16K /__w/_PipelineMapping
2.2M /__w/_actions
88K /__w/_temp
57M /__w/datafusion
Disk space after cleanup:
Filesystem Size Used Avail Use% Mounted on
overlay 72G 51G 22G 71% /
tmpfs 64M 0 64M 0% /dev
shm 64M 0 64M 0% /dev/shm
/dev/root 72G 51G 22G 71% /__w
tmpfs 3.2G 1.2M 3.2G 1% /run/docker.sock
tmpfs 7.9G 0 7.9G 0% /proc/acpi
tmpfs 7.9G 0 7.9G 0% /proc/scsi
tmpfs 7.9G 0 7.9G 0% /sys/firmware
```
## Are these changes tested?
by CI
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->1 parent 2b05b09 commit 1800d15
2 files changed
+14
-25
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
272 | 272 | | |
273 | 273 | | |
274 | 274 | | |
275 | | - | |
276 | | - | |
277 | | - | |
278 | | - | |
279 | | - | |
280 | | - | |
281 | | - | |
282 | | - | |
283 | | - | |
284 | | - | |
285 | | - | |
286 | | - | |
287 | 275 | | |
288 | 276 | | |
289 | 277 | | |
| |||
374 | 362 | | |
375 | 363 | | |
376 | 364 | | |
377 | | - | |
378 | | - | |
379 | | - | |
380 | | - | |
381 | | - | |
382 | | - | |
383 | | - | |
384 | | - | |
385 | | - | |
386 | | - | |
387 | | - | |
388 | | - | |
389 | | - | |
390 | 365 | | |
391 | 366 | | |
392 | 367 | | |
| |||
0 commit comments