String parameter macro artifacts are not materialized when cached or built remotely

[cbarrete]

Consider the following:

genrule(
    name = "gen",
    out = "out.txt",
    cmd = "echo $(location :bin) > $OUT",
)

cxx_binary(
    name = "bin",
    srcs = ["main.cpp"],
)

Where the contents of main.cpp are irrelevant.

If RBE is enabled and :bin is built remotely or cached, it will not be materialized locally, so out.txt will point to a file that does not exist.

This can be simply reproduced by building :gen remotely, and it can be proved that this is an RBE-related materialization issue because buck2 build --local-only --no-remote-cache :gen has the expected behavior: bin is materialized locally.

[dtolnay]

This is behaving as intended. In general, actions in a build have specific inputs and outputs. The C++ compilation/linking performed by your :bin target has main.cpp as its input and an executable bin as its output. The action performed by :gen has bin as its input and produces out.txt as output. When something uses the output of a dependency as an input, or when you request to build the output of some target, you are not also going to get the entire graph of inputs that went into building that output.

It sounds like you are expecting to have bin also as an output of :gen. You can implement that behavior in a rule but that is not how genrule is supposed to behave. You would need to use some other rule that either uses Artifact.with_associated_artifacts on the out.txt output artifact to attach bin to it as an associated artifact, or that uses DefaultInfo.other_outputs to produce bin as another output whenever out.txt is produced as a build output.

[cbarrete]

Thank you very much for the explanation, this is very useful.

Thankfully, my actual use case involves a custom rule, not genrule, so I can "fix" this.

When something uses the output of a dependency as an input, or when you request to build the output of some target, you are not also going to get the entire graph of inputs that went into building that output.

This makes a lot of sense when you put it this way, it's trivial in hindsight, when comparing with other types of builds.

Still, it is unintuitive (at least to myself at first, and to several people I know), so I will see if I can some improvements to the docs worth raising a PR for.

[cormacrelf]

Docs can maybe describe this as a difference from Nix. In Nix, every output goes through a dependency scanner that literally greps for "/nix/store/[0-9a-f]+" to find out which store paths to consider runtime dependencies. With Nix, if you build such a text file, you will automatically get a runtime dependency and those deps will be materialized to /nix/store.

Basically buck doesn't have this, but it has better APIs for describing runtime dependencies and recovering them later as David described. (Better than for Nix + Java... where jars are compressed and you have to write an uncompressed text file in nix-support/...). With those APIs, one can lay out an executable and all its other_outputs in a file system tree for distribution. These could be image resources or even dynamic libraries if you get your @rpath right. Many rules support a convention for this, resources = ..., and there are some packages and directory layout conventions for finding resources at runtime relative to the executable path (https://docs.rs/buck-resources). Layout requirements for runtime dependencies differ sometimes from language to language. You sorta need one layout for Python, one for Rust, maybe some others can be encompassed by the Rust convention too.

This is stuff I have learned mainly from reading the prelude and some trial and error, maybe it could go in the docs.

[cbarrete]

Docs can maybe describe this as a difference from Nix. In Nix, every output goes through a dependency scanner that literally greps for "/nix/store/[0-9a-f]+" to find out which store paths to consider runtime dependencies. With Nix, if you build such a text file, you will automatically get a runtime dependency and those deps will be materialized to /nix/store.

Exactly, that's where I got my mental model from.

I'm familiar with the resource system and we use it quite a bit in tests, but the problem is that it is not transitive. E.g. if I have a test that depends on a filegroup of config files, which themselves embed paths to other artifacts (say binaries to run, for example), the resource system won't be useful to propagate those dependencies.
In this specific example, the filegroup will also drop other_outputs because it takes sources instead of deps.

[cbarrete]

@dtolnay @cormacrelf I have opened a PR to add some docs about this.
Please let me know if something is inaccurate, as the details are not totally clear to me at the moment.