Ci dynamic matrix and rename #45

angerman · 2025-09-04T00:38:47Z

Summary

This PR adds support for dynamic builds in the CI matrix and fixes several issues with RTS sublibraries and dynamic linking.

Changes

CI Improvements

Add dynamic build matrix (DYNAMIC=0 and DYNAMIC=1) to GitHub workflows
Rename workflow to "Nix CI" for clarity

RTS Sublibrary Support

Fix AutoApply.cmm inclusion in RTS sublibraries via header includes
Add -no-ghc-internal flag to prevent auto-injection when building RTS sublibraries
Promote boot libraries (ghc-internal) to RTLD_GLOBAL to prevent duplicate loading
Use portable ${MKDIR_P} in configure.ac instead of mkdir -p

Dynamic Linking Fixes

Add -rdynamic/-flat_namespace for GHC API programs and ghc-iserv
Dynamically compute RTS rpath dependencies instead of hardcoded list
Add symlinks for nested shared libraries in bindist

Testsuite Improvements

Hash TEST_HC binary to ensure ghcconfig recomputation on compiler changes
Improve testlib.py error handling for missing paths/directories
Adjust tests for RTS split (disable rts_so size test)

Build System

Use concrete file target for testsuite-timeout instead of .PHONY
Add -optc-Wno-error for libffi-clib to prevent build failures
Add -no-auto-link-packages to hp2ps and unlit
Use find -exec instead of for loops in Makefile

Fixes

Fixes Move ci.yml to nix-ci.yml #51
Fixes Test dynamic builds #52

angerman · 2025-09-04T01:58:11Z

Interestingly, this now seems to fail due to the missing rts sublib issues. That are fixed in #44. Specifically this:

#44 (comment)

angerman · 2025-09-04T02:36:35Z

Fixes #51 and #52

angerman · 2025-09-17T08:30:24Z

testsuite/mk/boilerplate.mk

+sanitized_hc := $(subst $(space),_,$(subst :,_,$(subst /,_,$(subst \,_,$(TEST_HC)))))
+test_hc_hash := $(shell \
+  if command -v openssl >/dev/null 2>&1; then \
+    openssl dgst -sha256 $(TEST_HC) | awk '{print substr($$2, 1, 8)}'; \
+  elif command -v sha256sum >/dev/null 2>&1; then \
+    sha256sum $(TEST_HC) | awk '{print substr($$1, 1, 8)}'; \
+  elif command -v shasum >/dev/null 2>&1; then \
+    shasum -a 256 $(TEST_HC) | awk '{print substr($$1, 1, 8)}'; \
+  else \
+    echo "no_hash"; \
+  fi)
+ghc_config_mk = $(TOP)/mk/$(test_hc_hash)_ghcconfig$(sanitized_hc).mk


I should add a comment here why we do this. It adds a hash to the ghc config file. This ensure we always read the correct ghcconfig, if the ghc changed, the hash changed, and we recompute.

testsuite/driver/testlib.py

rts/rts.cabal

hsyl20 · 2025-09-17T10:04:27Z

compiler/GHC/Driver/Session.hs

        (NoArg (setGeneralFlag Opt_NoHsMain))
  , make_ord_flag defGhcFlag "no-rts"
        (NoArg (setGeneralFlag Opt_NoRts))
+  , make_ord_flag defGhcFlag "no-ghc-internal"


Why is this needed now?

We have the following dependeny as visible to cabal:

ghc-internal + rts

however, ghc will try to insert:

ghc-internal + rts-sublib (based on the -threaded / -debug flag) + rts

If we try to build the rts-sublib with ghc, we can end up trying to load ghc-internal, due to auto-injection of libraries.

Maybe a better solution is to add a flag to outright disable ghc's auto population of libs, instead of having separate ones for each lib 😅

.gitignore

cabal.project.stage1

Makefile

compiler/GHC/Driver/Flags.hs

rts/configure.ac

hasufell · 2025-12-05T07:03:46Z

Makefile

 	$@/bin/ghc-pkg recache
 	# Copy headers
 	@$(call copy_all_stage2_h,$@/bin/ghc-pkg)
+	# Add basename symlinks for nested shared libs (.dylib, .so) in lib/$(HOST_PLATFORM)


What is this? Why is this needed?

This is something we should ultimately fix in cabal; we do not want shared objects to be in various subdirectories,as that requires ballooning the executable with RPATHs for each subdirecty. We want dynamic lirbaries to live in one directory.

Makefile

- Add DYNAMIC=0/1 build/test matrix for static and dynamic builds - Trigger CI on stable-master branch in addition to stable-ghc-9.14 - Restrict push trigger to stable-ghc-9.14 branch

- Add basename symlinks for nested shared libraries (.dylib, .so) in lib/$(HOST_PLATFORM) to make them discoverable by the linker - Create ghc-iserv-dyn symlink pointing to ghc-iserv so GHC can find the dynamic interpreter executable - Apply same symlink logic for cross-compilation targets

Replace .PHONY testsuite-timeout target with a concrete file target (testsuite/timeout/install-inplace/bin/timeout) to avoid unnecessary rebuilds. Make only rebuilds the timeout utility when the output file doesn't exist. Also change test target dependencies from _build/bindist (which would trigger full rebuilds) to specific test tool binaries, further reducing unnecessary work when running the test suite.

- Add -no-ghc-internal flag to prevent auto-injection of ghc-internal when building RTS sublibraries (mirrors existing -no-rts flag). This prevents circular dependency issues during RTS build. - Move CMM sources into common rts-cmm-sources-base stanza shared by all sublibraries (threaded, debug, etc.) - Implement AutoApply.cmm.h workaround: generate .cmm.h files in main library, include them via wrapper .cmm files in sublibraries. This ensures each sublibrary gets properly parameterized CMM code. - Double all build flags (ghc-options, cpp-options, cmm-options, cc-options) in rts.cabal to ensure they're definitely propagated - Remove auto-link from hp2ps and unlit (C-only utilities that don't need Haskell runtime linking)

Add libffi-clib package configuration to suppress C compiler warnings that are promoted to errors by default. The bundled libffi code produces warnings that would otherwise fail the build. This is needed because: - GHC uses bundled libffi for FFI support - The libffi-clib wrapper exposes libffi to the RTS - Upstream libffi code triggers compiler warnings - With -Werror (often set by default), these become fatal errors

This commit fixes several dynamic linking issues that arise when using GHC with RTS sublibraries and the GHC API: - Inject rpath for RTS and libffi-clib in dynamic builds. This ensures the dynamic linker can find these libraries at runtime, especially when cabal passes -dyload deploy. - Promote ghc-internal to RTLD_GLOBAL when loaded via dlopen. This prevents duplicate symbol errors when multiple shared libraries reference the same ghc-internal symbols. - Export RTS symbols from ghc-iserv for dynamic builds. Programs using the GHC API load shared libraries via dlopen() that reference RTS symbols like stg_INTLIKE_closure. - Apply -rdynamic unconditionally for GHC API programs (Linux/FreeBSD) and -flat_namespace for macOS. This makes RTS symbols visible to dynamically loaded libraries even when the main executable wasn't compiled with -dynamic. See Note [Export dynamic symbols for GHC API programs] in Linker/Static.hs and Note [ghc-iserv and dynamic symbol export] in ghc-iserv.cabal.in.

Improve error handling and path resolution in the test driver: - Add proper error handling for missing directories and files when searching for shared objects, raising StatsException with clear error messages instead of silently failing - Fix path resolution in collect_size_func to handle both absolute and relative paths correctly - Improve ghc-pkg output parsing to handle various output formats - Add fallback logic for finding shared objects: try inplace first, fall back to non-inplace if that fails - Convert silent failures to explicit StatsException raises so test failures are properly reported

Adjust the test suite for the RTS sublibrary split: - Prefix ghcconfig filename with hash of TEST_HC binary to ensure we recompute the config when the compiler changes. This prevents stale config values when switching between different GHC versions. - Disable rts test which is invalid since the RTS split (the test assumes monolithic RTS structure) - Mark T2228 as not broken (it was incorrectly marked) - Add testsuite-specific .gitignore entries

Replace hardcoded ["rts", "libffi-clib"] list with a function that dynamically computes which packages need rpath injection by checking: 1. Any package named "rts" (covers all RTS sublibraries) 2. Any direct dependency of an RTS package This is more robust as it will automatically handle any future library dependencies the RTS might gain. Also adds Note [RTS sublibrary rpath injection] explaining why GHC must always inject rpaths for RTS-related libraries regardless of Cabal's -dynload deploy setting - Cabal fundamentally cannot see the RTS sublibrary selection which happens at GHC link time.

- rts/configure.ac: Use portable ${MKDIR_P} instead of mkdir -p (AC_PROG_MKDIR_P macro for better portability across systems) - Makefile: Replace for loops with find -exec for creating symlinks (cleaner and avoids issues with special characters in filenames) - Makefile: Add explanatory comments for symlink creation (explains why RTS sublibraries need top-level symlinks)

angerman requested a review from hasufell September 4, 2025 00:38

angerman self-assigned this Sep 4, 2025

angerman force-pushed the ci-dynamic-matrix-and-rename branch from 4de99d0 to 23dd6fa Compare September 4, 2025 00:50

angerman force-pushed the ci-dynamic-matrix-and-rename branch 3 times, most recently from dabc558 to bbb815a Compare September 4, 2025 12:46

angerman force-pushed the stable-ghc-9.14 branch from a10be51 to d3180ae Compare September 5, 2025 00:06

angerman requested review from doyougnu, hsyl20 and luite as code owners September 5, 2025 00:06

angerman force-pushed the stable-ghc-9.14 branch 4 times, most recently from d9f06c2 to a1edbb5 Compare September 5, 2025 07:32

angerman force-pushed the ci-dynamic-matrix-and-rename branch 2 times, most recently from 7c8212f to e74ec54 Compare September 11, 2025 06:05

angerman force-pushed the ci-dynamic-matrix-and-rename branch from b78b210 to 1967f08 Compare September 17, 2025 03:20

angerman changed the base branch from stable-ghc-9.14 to stable-ghc-9.14-rebased September 17, 2025 08:28

angerman commented Sep 17, 2025

View reviewed changes

testsuite/driver/testlib.py Outdated Show resolved Hide resolved

angerman commented Sep 17, 2025

View reviewed changes

rts/rts.cabal Show resolved Hide resolved

hsyl20 reviewed Sep 17, 2025

View reviewed changes

angerman force-pushed the ci-dynamic-matrix-and-rename branch from 8c7a84c to 3e7c948 Compare November 27, 2025 06:18

angerman changed the base branch from stable-ghc-9.14-rebased to stable-ghc-9.14.2025.11.12 November 27, 2025 06:20

angerman force-pushed the ci-dynamic-matrix-and-rename branch from 2536c91 to d16343c Compare November 28, 2025 01:36

angerman force-pushed the stable-ghc-9.14.2025.11.12 branch 2 times, most recently from e808dde to 537b3a7 Compare November 28, 2025 08:40

angerman deleted the branch stable-ghc-9.14 November 29, 2025 02:16

angerman closed this Nov 29, 2025

angerman reopened this Nov 29, 2025

angerman changed the base branch from stable-ghc-9.14.2025.11.12 to stable-ghc-9.14 November 29, 2025 02:28

angerman force-pushed the ci-dynamic-matrix-and-rename branch 2 times, most recently from 25abcc6 to c9444e4 Compare December 2, 2025 04:05

angerman commented Dec 2, 2025

View reviewed changes

.gitignore Show resolved Hide resolved

angerman commented Dec 2, 2025

View reviewed changes

cabal.project.stage1 Show resolved Hide resolved

angerman commented Dec 2, 2025

View reviewed changes

Makefile Show resolved Hide resolved

angerman commented Dec 2, 2025

View reviewed changes

compiler/GHC/Driver/Flags.hs Show resolved Hide resolved

angerman force-pushed the ci-dynamic-matrix-and-rename branch 2 times, most recently from dfe6246 to 429aa4d Compare December 3, 2025 03:50

hasufell reviewed Dec 5, 2025

View reviewed changes

angerman force-pushed the ci-dynamic-matrix-and-rename branch from f2b01e0 to 27c9326 Compare December 5, 2025 08:24

angerman added 10 commits December 11, 2025 16:05

ci: add dynamic build matrix to GitHub workflows

fea2f26

- Add DYNAMIC=0/1 build/test matrix for static and dynamic builds - Trigger CI on stable-master branch in addition to stable-ghc-9.14 - Restrict push trigger to stable-ghc-9.14 branch

angerman force-pushed the ci-dynamic-matrix-and-rename branch from 27c9326 to 37e2399 Compare December 11, 2025 07:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Ci dynamic matrix and rename #45

Ci dynamic matrix and rename #45

Uh oh!

angerman commented Sep 4, 2025 •

edited

Loading

Uh oh!

angerman commented Sep 4, 2025 •

edited

Loading

Uh oh!

angerman commented Sep 4, 2025 •

edited

Loading

Uh oh!

angerman Sep 17, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

hsyl20 Sep 17, 2025

Uh oh!

angerman Sep 17, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hasufell Dec 5, 2025

Uh oh!

angerman Dec 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Ci dynamic matrix and rename #45

Are you sure you want to change the base?

Ci dynamic matrix and rename #45

Uh oh!

Conversation

angerman commented Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

CI Improvements

RTS Sublibrary Support

Dynamic Linking Fixes

Testsuite Improvements

Build System

Fixes

Uh oh!

angerman commented Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

angerman commented Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

angerman Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hsyl20 Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

angerman Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hasufell Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

angerman Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

angerman commented Sep 4, 2025 •

edited

Loading

angerman commented Sep 4, 2025 •

edited

Loading

angerman commented Sep 4, 2025 •

edited

Loading

angerman Sep 17, 2025 •

edited

Loading