ci(workflows): harden linux-gcc-cmake-odbc apt/dpkg installs with retries and diagnostics

[Copilot]

The linux-gcc-cmake-odbc job intermittently fails during apt/dpkg steps with no visible errors. Symptoms suggest interactive prompts, unmet dependencies, or transient network/disk issues.

Changes

Made all apt/dpkg operations non-interactive:

• Set DEBIAN_FRONTEND=noninteractive on all package installation steps
• Added dpkg options --force-confdef and --force-confold to auto-resolve config conflicts

Added retry logic for transient failures:

• 3-attempt retry loop for apt-get install with 10s backoff
• Acquire::Retries=3 for apt-get update operations

Fixed dpkg dependency issues:

• Run apt-get install -y -f after dpkg installations to resolve unmet dependencies
• Allow dpkg to partially fail (|| true) before fixing deps

Added failure diagnostics:

• Capture apt/dpkg logs, disk usage, memory, and system errors on failure
• Always upload diagnostics as artifacts (apt-dpkg-logs) for post-mortem analysis

Re-run on failure

If failures persist, check the uploaded apt-dpkg-logs artifact for apt/dpkg logs and system diagnostics.

Original prompt

Problem

The GitHub Actions job linux-gcc-cmake-odbc on ubuntu-22.04 intermittently fails during apt/dpkg steps with no visible error in the logs. Symptoms indicate apt may be prompting interactively, dpkg -i may leave unmet dependencies, or runner disk/network/transient issues cause silent failures. We need to harden the workflow to make installs non-interactive, retry transient apt failures, fix dpkg unmet deps, capture apt/dpkg diagnostics on failure, and always upload those logs as artifacts so failures can be diagnosed.

Goal

Create a pull request that updates the workflow(s) under .github/workflows which define the job named linux-gcc-cmake-odbc. For each workflow file containing that job, replace the existing apt install / cleanup / dpkg steps with robust, non-interactive steps and add diagnostic upload steps. The change must target the main branch.

Required changes

1. Search all YAML files under .github/workflows for a job with the id or name linux-gcc-cmake-odbc. For each matching workflow file, update the job steps as follows:

• Replace the existing steps that run:

  • sudo apt -y update && sudo apt -y install libssl-dev unixodbc-dev alien libaio1 gnupg2 curl odbcinst1debian2 libodbc1 odbcinst
  • Free disk space step that runs apt autopurge, apt clean, removes dotnet, npm cache clean, docker system prune
  • Setup Oracle ODBC connector step that downloads devartodbcoracle_amd64.deb and runs sudo dpkg -i
  • Setup Microsoft packages & msodbcsql18 step (dpkg installing packages-microsoft-prod and apt-get install msodbcsql18)

  with the following new sequence of steps (YAML snippet to be inserted in place of the original steps):

  • name: Pre-check runner disk & cleanup
    run: |
    echo "Disk usage before cleanup:"
    df -h .
    free -h
    sudo rm -f /var/lib/dpkg/lock-frontend /var/cache/apt/archives/lock || true
    sudo apt-get -y autoremove --purge || true
    sudo apt-get -y clean || true
    sudo rm -rf /usr/share/dotnet || true
    sudo npm cache clean --force || true
    sudo docker system prune -a --force || true
    echo "Disk usage after cleanup:"
    df -h .

  • name: Install basic system dependencies (robust)
    env:
    DEBIAN_FRONTEND: noninteractive
    run: |
    set -euxo pipefail
    sudo rm -f /var/lib/dpkg/lock-frontend /var/cache/apt/archives/lock || true
    sudo apt-get update -o Acquire::Retries=3
    for i in 1 2 3; do
    sudo apt-get install -y --no-install-recommends
    -o Dpkg::Options::="--force-confdef"
    -o Dpkg::Options::="--force-confold"
    libssl-dev unixodbc-dev alien libaio1 gnupg2 curl
    odbcinst1debian2 libodbc1 odbcinst || {
    echo "apt-get install attempt $i failed; retrying in 10s..."
    sleep 10
    continue
    }
    break
    done

  • name: Setup Oracle ODBC connector (.deb) and fix deps
    env:
    DEBIAN_FRONTEND: noninteractive
    run: |
    set -euxo pipefail
    wget https://www.devart.com/odbc/oracle/devartodbcoracle_amd64.deb
    sudo dpkg -i devartodbcoracle_amd64.deb || true
    sudo apt-get update -o Acquire::Retries=3
    sudo apt-get install -y -f
    dpkg -l | grep -Ei 'devart|odbc|oracle' || true

  • name: Setup Microsoft packages & msodbcsql18
    env:
    DEBIAN_FRONTEND: noninteractive
    run: |
    set -euxo pipefail
    curl -sSL -O https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packages-microsoft-prod.deb
    sudo dpkg -i packages-microsoft-prod.deb
    rm packages-microsoft-prod.deb
    sudo apt-get update -o Acquire::Retries=3
    sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18

  • name: Capture apt/dpkg/system diagnostics on failure
    if: failure()
    run: |
    set -euxo pipefail
    echo "===== APT / DPKG DIAGNOSTICS ====="
    sudo tail -n +1 /var/log/apt/term.log || true
    sudo tail -n +1 /var/log/apt/history.log || true
    sudo tail -n +1 /var/log/dpkg.log || true
    df -h || true
    free -h || true
    uname -a || true
    dmesg --level=err,warn | tail -n 200 || true
    mkdir -p $GITHUB_WORKSPACE/ci-apt-logs
    sudo cp /var/log/apt/* $GITHUB_WORKSPACE/ci-apt-logs/ 2>/dev/null || true
    sudo cp /var/log/dpkg.log $GITHUB_WORKSPACE/ci-apt-logs/ 2>/dev/null || true
    ls -l $GITHUB_WORKSPACE/ci-apt-logs || true

  • name: Upload apt/dpkg logs
    if: always()
    uses: actions/upload-artifact@v4
    with:
    name: apt-dpkg-logs
    path: ci-apt-logs

2. Preserve the remaining steps in the job (cmake, build, tests) unchanged.

3. Ensure the YAML indentation and syntax match existing workflow style.

4. Add a concise commit message and PR description explaining the changes and rationale (prevents interactive prompts, retries transient errors, captures logs).

Deliverable

Open a pull request on pocoproject/poco targeting the main branch with the changes described above. The PR should include:

• modified workflow fil...

This pull request was created as a result of the following prompt from Copilot chat.

Problem

The GitHub Actions job linux-gcc-cmake-odbc on ubuntu-22.04 intermittently fails during apt/dpkg steps with no visible error in the logs. Symptoms indicate apt may be prompting interactively, dpkg -i may leave unmet dependencies, or runner disk/network/transient issues cause silent failures. We need to harden the workflow to make installs non-interactive, retry transient apt failures, fix dpkg unmet deps, capture apt/dpkg diagnostics on failure, and always upload those logs as artifacts so failures can be diagnosed.

Goal

Create a pull request that updates the workflow(s) under .github/workflows which define the job named linux-gcc-cmake-odbc. For each workflow file containing that job, replace the existing apt install / cleanup / dpkg steps with robust, non-interactive steps and add diagnostic upload steps. The change must target the main branch.

Required changes

1. Search all YAML files under .github/workflows for a job with the id or name linux-gcc-cmake-odbc. For each matching workflow file, update the job steps as follows:

• Replace the existing steps that run:

  • sudo apt -y update && sudo apt -y install libssl-dev unixodbc-dev alien libaio1 gnupg2 curl odbcinst1debian2 libodbc1 odbcinst
  • Free disk space step that runs apt autopurge, apt clean, removes dotnet, npm cache clean, docker system prune
  • Setup Oracle ODBC connector step that downloads devartodbcoracle_amd64.deb and runs sudo dpkg -i
  • Setup Microsoft packages & msodbcsql18 step (dpkg installing packages-microsoft-prod and apt-get install msodbcsql18)

  with the following new sequence of steps (YAML snippet to be inserted in place of the original steps):

  • name: Pre-check runner disk & cleanup
    run: |
    echo "Disk usage before cleanup:"
    df -h .
    free -h
    sudo rm -f /var/lib/dpkg/lock-frontend /var/cache/apt/archives/lock || true
    sudo apt-get -y autoremove --purge || true
    sudo apt-get -y clean || true
    sudo rm -rf /usr/share/dotnet || true
    sudo npm cache clean --force || true
    sudo docker system prune -a --force || true
    echo "Disk usage after cleanup:"
    df -h .

  • name: Install basic system dependencies (robust)
    env:
    DEBIAN_FRONTEND: noninteractive
    run: |
    set -euxo pipefail
    sudo rm -f /var/lib/dpkg/lock-frontend /var/cache/apt/archives/lock || true
    sudo apt-get update -o Acquire::Retries=3
    for i in 1 2 3; do
    sudo apt-get install -y --no-install-recommends
    -o Dpkg::Options::="--force-confdef"
    -o Dpkg::Options::="--force-confold"
    libssl-dev unixodbc-dev alien libaio1 gnupg2 curl
    odbcinst1debian2 libodbc1 odbcinst || {
    echo "apt-get install attempt $i failed; retrying in 10s..."
    sleep 10
    continue
    }
    break
    done

  • name: Setup Oracle ODBC connector (.deb) and fix deps
    env:
    DEBIAN_FRONTEND: noninteractive
    run: |
    set -euxo pipefail
    wget https://www.devart.com/odbc/oracle/devartodbcoracle_amd64.deb
    sudo dpkg -i devartodbcoracle_amd64.deb || true
    sudo apt-get update -o Acquire::Retries=3
    sudo apt-get install -y -f
    dpkg -l | grep -Ei 'devart|odbc|oracle' || true

  • name: Setup Microsoft packages & msodbcsql18
    env:
    DEBIAN_FRONTEND: noninteractive
    run: |
    set -euxo pipefail
    curl -sSL -O https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packages-microsoft-prod.deb
    sudo dpkg -i packages-microsoft-prod.deb
    rm packages-microsoft-prod.deb
    sudo apt-get update -o Acquire::Retries=3
    sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18

  • name: Capture apt/dpkg/system diagnostics on failure
    if: failure()
    run: |
    set -euxo pipefail
    echo "===== APT / DPKG DIAGNOSTICS ====="
    sudo tail -n +1 /var/log/apt/term.log || true
    sudo tail -n +1 /var/log/apt/history.log || true
    sudo tail -n +1 /var/log/dpkg.log || true
    df -h || true
    free -h || true
    uname -a || true
    dmesg --level=err,warn | tail -n 200 || true
    mkdir -p $GITHUB_WORKSPACE/ci-apt-logs
    sudo cp /var/log/apt/* $GITHUB_WORKSPACE/ci-apt-logs/ 2>/dev/null || true
    sudo cp /var/log/dpkg.log $GITHUB_WORKSPACE/ci-apt-logs/ 2>/dev/null || true
    ls -l $GITHUB_WORKSPACE/ci-apt-logs || true

  • name: Upload apt/dpkg logs
    if: always()
    uses: actions/upload-artifact@v4
    with:
    name: apt-dpkg-logs
    path: ci-apt-logs

2. Preserve the remaining steps in the job (cmake, build, tests) unchanged.

3. Ensure the YAML indentation and syntax match existing workflow style.

4. Add a concise commit message and PR description explaining the changes and rationale (prevents interactive prompts, retries transient errors, captures logs).

Deliverable

Open a pull request on pocoproject/poco targeting the main branch with the changes described above. The PR should include:

• modified workflow file(s) under .github/workflows
• commit message: "ci(workflows): make apt installs non-interactive, add retries and diagnostic log upload for linux-gcc-cmake-odbc"
• PR description summarizing the problem and the exact fixes applied, and instructions for the user to re-run the CI to capture logs if failure persists.

Notes

• If multiple workflow files define the job, apply the change to all.
• If the agent cannot find a job named exactly linux-gcc-cmake-odbc, find jobs whose name contains linux-gcc-cmake or contains odbc and ask for clarification before making changes.
• Do not change other unrelated jobs or files.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[aleks-f]

@copilot that change passed in this PR, but is failing again in main and other branches

[Copilot]

Pull request overview

This PR hardens the linux-gcc-cmake-odbc GitHub Actions workflow to address intermittent failures during apt/dpkg package installation steps. The changes make all package operations non-interactive, add retry logic for transient failures, fix dependency resolution issues, and capture diagnostic logs for troubleshooting.

Key Changes:

• Made all apt/dpkg operations non-interactive using DEBIAN_FRONTEND=noninteractive and dpkg configuration options
• Added 3-attempt retry loop for apt-get install with 10-second backoff between attempts
• Implemented comprehensive failure diagnostics with automatic artifact upload for post-mortem analysis

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The Oracle ODBC dpkg command should also have explicit configuration options to avoid interactive prompts, similar to the Microsoft packages step. While || true allows it to fail, adding dpkg options would make the non-interactive intent clearer and match the hardening approach:

sudo DEBIAN_FRONTEND=noninteractive dpkg -i --force-confdef --force-confold devartodbcoracle_amd64.deb || true

This ensures dpkg won't prompt for configuration file conflicts.

Suggested change

	sudo dpkg -i devartodbcoracle_amd64.deb || true
	sudo dpkg -i --force-confdef --force-confold devartodbcoracle_amd64.deb || true

[Copilot]

The wget command lacks retry logic and timeout configuration, which could fail due to transient network issues. Given the PR's goal of hardening against transient failures, consider adding retry and timeout options:

wget --tries=3 --timeout=30 https://www.devart.com/odbc/oracle/devartodbcoracle_amd64.deb

This aligns with the retry philosophy applied to apt-get operations throughout the PR.

Suggested change

	wget https://www.devart.com/odbc/oracle/devartodbcoracle_amd64.deb
	wget --tries=3 --timeout=30 https://www.devart.com/odbc/oracle/devartodbcoracle_amd64.deb

[Copilot]

The curl command lacks retry logic and timeout configuration. Given the PR's goal of hardening against transient failures, consider adding retry options:

curl --retry 3 --retry-delay 5 --max-time 60 -sSL -O https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packages-microsoft-prod.deb

This aligns with the retry philosophy applied to apt-get operations throughout the PR.

Suggested change

	curl -sSL -O https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packages-microsoft-prod.deb
	curl --retry 3 --retry-delay 5 --max-time 60 -sSL -O https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packages-microsoft-prod.deb

[Copilot]

Using set -euxo pipefail with a diagnostics step that should tolerate failures is problematic. The -e flag will cause the script to exit on the first command that returns a non-zero exit status, even though all commands have || true to prevent failures. While the || true will prevent individual command failures from stopping the script, the set -e is unnecessary here since this is a diagnostic step that should never fail.

Consider using set -uxo pipefail instead (without -e) for this diagnostic step, or remove set entirely since all commands already have || true fallbacks.

Suggested change

	set -euxo pipefail
	set -uxo pipefail

[Copilot]

The retry logic has a subtle bug: if the last attempt (attempt 3) fails, the script will exit with a non-zero status due to set -e, but the loop will have already completed all iterations. This means the retry won't actually fail the step - it will silently succeed.

The issue is that after the loop completes all 3 iterations without success, there's no explicit check to verify if the installation actually succeeded. Consider adding a verification after the loop:

for i in 1 2 3; do
  sudo apt-get install -y --no-install-recommends \
    -o Dpkg::Options::="--force-confdef" \
    -o Dpkg::Options::="--force-confold" \
    libssl-dev unixodbc-dev alien libaio1 gnupg2 curl \
    odbcinst1debian2 libodbc1 odbcinst && break || {
    echo "apt-get install attempt $i failed; retrying in 10s..."
    if [ "$i" -eq 3 ]; then
      echo "All 3 attempts failed"
      exit 1
    fi
    sleep 10
  }
done

Suggested change

	for i in 1 2 3; do
	sudo apt-get install -y --no-install-recommends \
	-o Dpkg::Options::="--force-confdef" \
	-o Dpkg::Options::="--force-confold" \
	libssl-dev unixodbc-dev alien libaio1 gnupg2 curl \
	odbcinst1debian2 libodbc1 odbcinst || {
	echo "apt-get install attempt $i failed; retrying in 10s..."
	sleep 10
	continue
	}
	break
	done
	success=0
	for i in 1 2 3; do
	sudo apt-get install -y --no-install-recommends \
	-o Dpkg::Options::="--force-confdef" \
	-o Dpkg::Options::="--force-confold" \
	libssl-dev unixodbc-dev alien libaio1 gnupg2 curl \
	odbcinst1debian2 libodbc1 odbcinst && { success=1; break; } || {
	echo "apt-get install attempt $i failed; retrying in 10s..."
	sleep 10
	continue
	}
	done
	if [ "$success" -ne 1 ]; then
	echo "apt-get install failed after 3 attempts"
	exit 1
	fi

[Copilot]

The artifact upload step uses a hardcoded path ci-apt-logs instead of $GITHUB_WORKSPACE/ci-apt-logs which is used in the diagnostic capture step. While this may work if the working directory is $GITHUB_WORKSPACE, it's inconsistent and could fail if the working directory changes.

For consistency and clarity, use the full path:

path: ${{ github.workspace }}/ci-apt-logs

or ensure the path matches what's created in line 883.

Suggested change

	path: ci-apt-logs
	path: ${{ github.workspace }}/ci-apt-logs

[Copilot]

Missing DEBIAN_FRONTEND=noninteractive environment variable on the dpkg command. While the env is set at the step level, being explicit when calling dpkg can prevent potential interactive prompts if the environment isn't properly inherited.

Consider adding it explicitly to the dpkg command:

sudo DEBIAN_FRONTEND=noninteractive dpkg -i packages-microsoft-prod.deb

This is especially important since this step had the -o Dpkg::Options:: flags for confdef/confold in the system dependencies step but those options are not applicable to dpkg command directly (they're apt-get options).

Suggested change

	sudo dpkg -i packages-microsoft-prod.deb
	sudo DEBIAN_FRONTEND=noninteractive dpkg -i packages-microsoft-prod.deb