I started using Buck2 because I wanted to generate some types from Kubernetes CRDs.
The repository had a pile of CRDs from different projects. I wanted Python models for some of them, KCL schemas for others, and a few generated schemas for unrelated tools. This was not supposed to become an architectural commitment. I mostly wanted code generation to stop being a pile of scripts.
The first thing I really liked was that Starlark let me turn the structure of the source tree directly into the structure of the build graph.
ALL_CRDS = glob(
[
"**/*.yaml",
"**/*.yml",
],
exclude = [
"**/README.*",
"**/*.md",
"**/rbac*.yml",
"**/*example*",
"**/*whoami*",
"**/*gateway*",
"**/docker-labels*",
"**/file.*",
],
)
PYTHON_GENERATORS = crd_codegen_from_tree(
name_prefix = "gen",
srcs = ALL_CRDS,
namespace_prefix = "generated.models",
)
KCL_GENERATORS = (
kcl_codegen_from_tree(srcs = ALL_CRDS)
+ frp_codegen_groups()
)
filegroup(
name = "generated_python",
srcs = PYTHON_GENERATORS,
visibility = ["PUBLIC"],
)
filegroup(
name = "generated_kcl",
srcs = KCL_GENERATORS,
visibility = ["PUBLIC"],
)
The interesting part was crd_codegen_from_tree.
def crd_codegen_from_tree(
*,
name_prefix,
srcs,
namespace_prefix,
):
groups = {}
for src in srcs:
top = src.split("/")[0]
groups.setdefault(top, []).append(src)
targets = []
for group, group_srcs in groups.items():
safe = group.replace("-", "_")
python_codegen(
name = safe,
srcs = group_srcs,
namespace = safe,
)
targets.append(":{}".format(safe))
return targets
The macro runs during analysis, groups the CRDs by their source-tree location, and emits independent codegen targets. Cert-manager, Flux, DigitalOcean, and everything else can be scheduled and cached independently instead of becoming one giant codegen command.
Tantalizing.
The rule underneath it was similarly small:
def _kcl_crd_import_impl(ctx):
out = ctx.actions.declare_output(ctx.label.name, dir=True)
kcl = ctx.attrs._kcl[RunInfo]
args = cmd_args()
args.add("import")
args.add("-m", "crd")
for src in ctx.attrs.srcs:
args.add(src)
args.add("--output", out.as_output())
ctx.actions.run(
cmd_args(kcl, args),
category = "kcl_crd_import",
)
return [DefaultInfo(default_outputs = [out])]
kcl_crd_import = rule(
impl = _kcl_crd_import_impl,
attrs = {
"srcs": attrs.list(attrs.source(), default = []),
"_kcl": attrs.exec_dep(default = "toolchains//:kcl"),
},
)
Instead of thinking in terms of commands, I was starting to think in terms of declared tools, inputs, outputs, and transformations. Using https://github.com/tweag/buck2.nix the tools themselves could come from Nix and become ordinary dependencies in the graph.
flake.package(
name = "kcl",
path = ".",
package = "kcl",
binary = "kcl",
visibility = ["PUBLIC"],
)
flake.package(
name = "go-schema-kcl",
path = ".",
package = "go-schema-kcl",
binary = "go-schema-kcl",
visibility = ["PUBLIC"],
)
flake.package(
name = "python-crd-cloudcoil",
path = ".",
package = "python-crd-cloudcoil",
binary = "python-crd-cloudcoil",
visibility = ["PUBLIC"],
)
Nix still provided the environment. Buck increasingly described what could happen inside it.
Once you start looking at a repository this way, it becomes very easy to ask what else could be a node.
That question has caused me problems.
An iteration of Attune originally used Modal for remote execution. I actually liked Modal, but as more of Attune moved into Buck I ended up with two descriptions of the same experiment cell. Modal knew what should run remotely and with what inputs; Buck knew the executable, its declared artifacts, its dependencies, its invalidation boundary, and whether the result could be reused.
Once I connected Buck to BuildBuddy, the duplication became difficult to justify.
_TERMINAL_EXECUTION = (
["toolchains//:deterministic_execution"]
if read_config("attune_rbe", "enabled", "false") == "true"
else []
)
genrule(
name = "w13_cell_action",
srcs = [
":add_one_component",
":cell_action_input",
":w13_declared_cell_plan",
"toolchains//:wasmtime_c_api",
],
out = "cell-action.json",
cmd = """
env LD_LIBRARY_PATH="$(location toolchains//:wasmtime_c_api)/lib" \
$(exe :cell_action_driver) \
$(location :w13_declared_cell_plan) \
$(location :cell_action_input) \
"$OUT" \
- \
$(location :add_one_component)
""",
exec_compatible_with = _TERMINAL_EXECUTION,
visibility = ["PUBLIC"],
)
The experiment cell was already a build action. Making that action eligible for the remote execution platform was smaller than maintaining a separate remote-compute abstraction beside it.
So Buck2 ate Modal.
Not all of Modal, obviously. I needed a narrow part of it: take an exact computation with exact inputs and run it somewhere else. Unfortunately, that turns out to be an extremely build-system-shaped problem.
Once experiment stages were build actions, the graph itself started becoming useful experimental infrastructure.
def experiment_18_graph():
genrule(
name = "checkpoint_18a_concept",
srcs = [":experiment_17_boundary"],
out = "checkpoint-18a-concept",
cmd = "$(exe :stage) concept "
"$(location :experiment_17_boundary) $OUT",
)
genrule(
name = "checkpoint_18b_capability_census",
srcs = [":checkpoint_18a_concept"],
out = "checkpoint-18b-capability-census",
cmd = "$(exe :stage) census "
"$(location :checkpoint_18a_concept) $OUT",
)
genrule(
name = "checkpoint_18c_realization",
srcs = [
":experiment_17_boundary",
":checkpoint_18a_concept",
":checkpoint_18b_capability_census",
],
out = "checkpoint-18c-realization",
cmd = "$(exe :stage) realization "
"$(location :experiment_17_boundary) "
"$(location :checkpoint_18a_concept) "
"$(location :checkpoint_18b_capability_census) "
"$OUT",
)
A downstream result could depend on exactly the checkpoints used to derive it. Change an upstream artifact and the downstream computation changes. Leave everything alone and the cache can answer whether that exact observation has already been produced.
Attune still had metadata, Parquet, and reports. Buck was not my observability UI. But the basic provenance question, what exact computation produced this exact observation?, was increasingly answered by the graph itself. The experiment had become vaguely Merkle-tree shaped without requiring a database to establish its fundamental lineage.
Buck2 ate another piece.
SearchBench rounds eventually accumulated a lifecycle: validate the experiment, estimate its cost, approve it, run the evaluation matrix, merge the shards, and report the result.
For a while I thought this was becoming a scheduler problem. Instead I started using BXL to inspect the graph and emit the actions that were currently meaningful.
def _preflight_actions(ctx, round_target, matches, estimate_target):
q = ctx.uquery()
manifest_dir = _manifest_dir_for_round_target(round_target)
if manifest_dir == "":
return []
package_label = "//{}".format(manifest_dir)
candidates = [
{
"label": package_label + ":preflight_tool_policy",
"kind": "run_preflight_target",
"reason": "Tool-policy compatibility must pass before live/provider-backed actions.",
"priority": 20,
},
{
"label": package_label + ":preflight_evaluate_n",
"kind": "run_preflight_target",
"reason": "Evaluate-N cost preflight should be fresh before provider-backed retries.",
"priority": 35,
},
{
"label": package_label + ":preflight_matrix_{}".format(matches),
"kind": "run_preflight_target",
"reason": "Matrix preflight should match the planned live matrix width.",
"priority": 30,
},
]
actions = []
for row in candidates:
if _target_exists(q, ctx, row["label"]):
actions.append(_literal_action_doc(
row["kind"],
row["label"],
_command_for_label(row["label"], False),
row["reason"],
False,
row["priority"],
))
if estimate_target != "":
actions.append(_literal_action_doc(
"run_cost_preflight",
estimate_target,
_command_for_label(estimate_target, False),
"Cost estimate must exist before any provider-backed round action is considered safe.",
False,
40,
))
return _dedupe_action_docs(actions)
The durable evidence records what happened, while the graph records the prerequisites. An inspection step can derive the next legal action from those two things instead of requiring a daemon to remain authoritative over the workflow. The BXL interface literally describes the agent loop as inspect, execute, inspect again.
By now Buck was doing code generation, toolchains, builds across several languages, tests, WASI component assembly, experiment checkpoints, provenance, dataset materialization, cost preflight, evaluation matrices, local and remote execution, reuse boundaries, workflow gates, and agent-facing repository operations.
I originally adopted it to generate some types.
Anyway, this is what "just use Buck for the build" eventually turned into:
def searchbench_live_matrix_shards(
name_prefix,
manifest,
artifact_root_base,
dataset_total,
shard_size = 16,
dataset_config = "py",
dataset_split = "dev",
dataset_distinct_repos = True,
parallel_matches = 3,
matrix_material_concurrency = 3,
matrix_concurrent_roles = True,
matrix_force = False,
round_label = "",
cost_plan = "",
repo_root = "."):
"""Generate shard targets + filegroup alias for a full dataset run."""
shard_count = (dataset_total + shard_size - 1) // shard_size
shard_targets = []
for i in range(shard_count):
skip = i * shard_size
items = min(shard_size, dataset_total - skip)
pad = ("00" + str(i))[-3:]
shard_name = "{}_shard_{}".format(name_prefix, pad)
shard_artifact_root = "{}/shards/shard-{}".format(
artifact_root_base,
pad,
)
searchbench_round_op(
name = shard_name,
mode = "evaluate_matrix",
manifest = manifest,
artifact_root = shard_artifact_root,
dataset_config = dataset_config,
dataset_split = dataset_split,
dataset_skip = skip,
dataset_max_items = items,
dataset_distinct_repos = dataset_distinct_repos,
parallel_matches = parallel_matches,
matrix_material_concurrency = matrix_material_concurrency,
matrix_concurrent_roles = matrix_concurrent_roles,
matrix_force = matrix_force,
cost_plan = cost_plan,
round_label = round_label,
repo_root = repo_root,
plan_gate = True,
action_name = shard_name,
action_label = "//{}:{}".format(
native.package_name(),
shard_name,
),
next_action_label = "//{}:{}_shards".format(
native.package_name(),
name_prefix,
),
)
shard_targets.append(":{}".format(shard_name))
native.filegroup(
name = "{}_shards".format(name_prefix),
srcs = shard_targets,
visibility = ["PUBLIC"],
)
return shard_targets
# TODO: stop giving Buck2 responsibilities