Small release asset for CMake's FetchContent

[nlohmann]

Problem

Using the repository as source for a CMake FetchContent or ExternalProject can be a bit disappointing, because a complete checkout is quite large and takes time: cloning tag v3.10.5 requires downloading 285.41 MB. Alternatively, a different Git URL such as https://github.com/ArthurSonzogni/nlohmann_json_cmake_fetchcontent.git can be used.

Due to https://gitlab.kitware.com/cmake/cmake/-/issues/17770, the situation cannot be improved by using GIT_SHALLOW TRUE, because this does not imply git clone --depth 1, but git clone --depth 1 --no-single-branch.

Alternative solution

All that is needed to actually use the library with CMake are the following files (LICENSE.MIT is not needed technically, but some corporations can get crazy without it):

json
├── CMakeLists.txt
├── LICENSE.MIT
├── cmake
│   ├── config.cmake.in
│   ├── nlohmann_jsonConfigVersion.cmake.in
│   └── pkg-config.pc.in
├── include
└── single_include

These files are 1.6 MB uncompressed and can get as small as 100 KB as xzipped tarball.

mkdir json
cp -r LICENSE.MIT CMakeLists.txt cmake include single_include json
rm json/cmake/ci.cmake json/cmake/download_test_data.cmake
tar --create --directory=json --file - . | xz --compress -9e - > json.tar.xz
rm -fr json

If we would create such a tarball and add it to the GitHub release assets, we could use the following as CMake integration:

# URL is not working as the file has not been uploaded
FetchContent_Declare(json
    URL https://github.com/nlohmann/json/releases/download/v3.10.5/json.tar.xz
)
FetchContent_MakeAvailable(json)

Discussion

Upsides:

• Minimal network footprint.
• Drop-in replacement of the existing Git-based FetchContent approach.
• Creation of the tarball can be integrated into the release process and needs no additional workflow such as external repositories.
• xz compression makes shipping both the single-header and the multi-header version "free": removing either version only reduces the xzipped tarball size by 2 KB.

Downsides:

• The URL is obviously not as "discoverable" as the .git URL taken directly from GitHub.
• Compared to the previous approach using the repository, JSON_BuildTests=ON does not work.

What do you think?

[LeSpocky]

You have my +1 for creating such a stable release tarball!

I'm currently packaging 'nlohmann-json' for ptxdist and had a look at the experimental tar.xz files. I noticed two things:

1. The tarball extracts in the current directory. This is annoying when unpacking it manually and some build systems struggle hard with that. Please put it all in a folder and pack that. +1 for Karma if that folder has the version number in it. So instead of this folder structure:

cmake
include
└── nlohmann
    ├── detail
    │   ├── conversions
    │   ├── input
    │   ├── iterators
    │   ├── meta
    │   │   └── call_std
    │   └── output
    └── thirdparty
        └── hedley
single_include
└── nlohmann

Let it look like this:

nlohmann-json-3.10.5
├── cmake
├── include
│   └── nlohmann
│       ├── detail
│       │   ├── conversions
│       │   ├── input
│       │   ├── iterators
│       │   ├── meta
│       │   │   └── call_std
│       │   └── output
│       └── thirdparty
│           └── hedley
└── single_include
    └── nlohmann

2. I don't know how you did that, but in the current tar.xz tarball I get files listed multiple times with both tar and midnight commander, see output of tar -tf ~/Downloads/src_archive/nlohmann-json-3.10.5.tar.xz | sort | tail -n20 for example:

./include/nlohmann/thirdparty/hedley/hedley.hpp
./include/nlohmann/thirdparty/hedley/hedley_undef.hpp
./include/nlohmann/thirdparty/hedley/hedley_undef.hpp
./include/nlohmann/thirdparty/hedley/hedley_undef.hpp
./include/nlohmann/thirdparty/hedley/hedley_undef.hpp
./include/nlohmann/thirdparty/hedley/hedley_undef.hpp
./include/nlohmann/thirdparty/hedley/hedley_undef.hpp
./LICENSE.MIT
./LICENSE.MIT
./nlohmann_json.natvis
./nlohmann_json.natvis
./single_include/
./single_include/
./single_include/nlohmann/
./single_include/nlohmann/
./single_include/nlohmann/
./single_include/nlohmann/json.hpp
./single_include/nlohmann/json.hpp
./single_include/nlohmann/json.hpp
./single_include/nlohmann/json.hpp

So, some issues in detail, but in general 👍🏼 for working on this. 😃

[nlohmann]

Can you check if the following file works for you: json.tar.xz.zip

(I had to zip the tar.xz file, because GitHub does not allow to attach xz files).

It is built like this:

mkdir json
cp -r $(shell find LICENSE.MIT nlohmann_json.natvis CMakeLists.txt cmake/*.in include single_include -type f) json
gtar --sort=name --mtime="@$(shell git log -1 --pretty=%ct)" --owner=0 --group=0 --numeric-owner --pax-option=exthdr.name=%d/PaxHeaders/%f,delete=atime,delete=ctime --create --file - json | xz --compress -9e --threads=2 - > json.tar.xz

(I am using Homebrew, so gtar is GNU tar). I will not add a version number, because I also do not do this for the other release artifacts, and GitHub already puts the version into the URL)

[LeSpocky]

Can you check if the following file works for you: json.tar.xz.zip

This has all files flat in one directory.

It is built like this:

mkdir json
cp -r $(shell find LICENSE.MIT nlohmann_json.natvis CMakeLists.txt cmake/*.in include single_include -type f) json

This probably needs some more thoughts. I have no spontaneous idea, maybe I can propose something later?

gtar --sort=name --mtime="@$(shell git log -1 --pretty=%ct)" --owner=0 --group=0 --numeric-owner --pax-option=exthdr.name=%d/PaxHeaders/%f,delete=atime,delete=ctime --create --file - json | xz --compress -9e --threads=2 - > json.tar.xz

(I am using Homebrew, so `gtar` is GNU tar). I will not add a version number, because I also do not do this for the other release artifacts, and GitHub already puts the version into the URL)

Fine.

[nlohmann]

This has all files flat in one directory.

Yes, indeed. I'll fix it.

[nlohmann]

Second try: json.tar.xz.zip

$ tar -tf release_files/json.tar.xz
json/
json/CMakeLists.txt
json/LICENSE.MIT
json/cmake/
json/cmake/config.cmake.in
json/cmake/nlohmann_jsonConfigVersion.cmake.in
json/cmake/pkg-config.pc.in
json/include/
json/include/nlohmann/
json/include/nlohmann/adl_serializer.hpp
json/include/nlohmann/byte_container_with_subtype.hpp
json/include/nlohmann/detail/
json/include/nlohmann/detail/conversions/
json/include/nlohmann/detail/conversions/from_json.hpp
json/include/nlohmann/detail/conversions/to_chars.hpp
json/include/nlohmann/detail/conversions/to_json.hpp
json/include/nlohmann/detail/exceptions.hpp
json/include/nlohmann/detail/hash.hpp
json/include/nlohmann/detail/input/
json/include/nlohmann/detail/input/binary_reader.hpp
json/include/nlohmann/detail/input/input_adapters.hpp
json/include/nlohmann/detail/input/json_sax.hpp
json/include/nlohmann/detail/input/lexer.hpp
json/include/nlohmann/detail/input/parser.hpp
json/include/nlohmann/detail/input/position_t.hpp
json/include/nlohmann/detail/iterators/
json/include/nlohmann/detail/iterators/internal_iterator.hpp
json/include/nlohmann/detail/iterators/iter_impl.hpp
json/include/nlohmann/detail/iterators/iteration_proxy.hpp
json/include/nlohmann/detail/iterators/iterator_traits.hpp
json/include/nlohmann/detail/iterators/json_reverse_iterator.hpp
json/include/nlohmann/detail/iterators/primitive_iterator.hpp
json/include/nlohmann/detail/json_pointer.hpp
json/include/nlohmann/detail/json_ref.hpp
json/include/nlohmann/detail/macro_scope.hpp
json/include/nlohmann/detail/macro_unscope.hpp
json/include/nlohmann/detail/meta/
json/include/nlohmann/detail/meta/call_std/
json/include/nlohmann/detail/meta/call_std/begin.hpp
json/include/nlohmann/detail/meta/call_std/end.hpp
json/include/nlohmann/detail/meta/cpp_future.hpp
json/include/nlohmann/detail/meta/detected.hpp
json/include/nlohmann/detail/meta/identity_tag.hpp
json/include/nlohmann/detail/meta/is_sax.hpp
json/include/nlohmann/detail/meta/type_traits.hpp
json/include/nlohmann/detail/meta/void_t.hpp
json/include/nlohmann/detail/output/
json/include/nlohmann/detail/output/binary_writer.hpp
json/include/nlohmann/detail/output/output_adapters.hpp
json/include/nlohmann/detail/output/serializer.hpp
json/include/nlohmann/detail/string_escape.hpp
json/include/nlohmann/detail/value_t.hpp
json/include/nlohmann/json.hpp
json/include/nlohmann/json_fwd.hpp
json/include/nlohmann/ordered_map.hpp
json/include/nlohmann/thirdparty/
json/include/nlohmann/thirdparty/hedley/
json/include/nlohmann/thirdparty/hedley/hedley.hpp
json/include/nlohmann/thirdparty/hedley/hedley_undef.hpp
json/nlohmann_json.natvis
json/single_include/
json/single_include/nlohmann/
json/single_include/nlohmann/json.hpp

[LeSpocky]

Looks good. 👍🏼

[nlohmann]

I just merged #3289. I will update documentation and the releases next.