mbox series

[Concept,00/20] buildman: Add distributed builds

Message ID 20260316154733.1587261-1-sjg@u-boot.org
Headers
Series buildman: Add distributed builds |

Message

Simon Glass March 16, 2026, 3:47 p.m. UTC 
  From: Simon Glass <sjg@chromium.org>

This series adds distributed build support to buildman, allowing board
builds to be spread across a pool of remote machines over SSH.

The first few patches prepare Builder for distributed use: adding
parameters for custom thread classes, signal handling, lazy worktree
setup, and splitting build_boards() so remote workers can queue boards
incrementally.

The machine module probes remote hosts over SSH to collect their
capabilities (CPUs, memory, load, toolchains) and manages toolchain
version checking and fetching.

The worker module runs on remote machines in --worker mode, accepting
JSON commands on stdin and streaming results on stdout. It reuses
Builder and BuilderThread with a custom subclass that sends results
over the SSH pipe instead of writing to disk.

The boss module manages SSH connections to workers, handling startup,
source push via git, demand-driven board dispatch, result collection
with timeouts, and clean shutdown on Ctrl-C.

The wire-up patch connects everything through control.py with --dist,
--use-machines, and --no-local flags.


Simon Glass (20):
  u_boot_pylib: Add stdin_data support to run_pipe()
  u_boot_pylib: Support passing modules to run_test_suites()
  buildman: Remove unused imports from builder.py
  buildman: Fix pylint warnings in test_builder.py
  buildman: Fix import order in control.py
  buildman: Initialise _timestamps to an empty deque
  buildman: Add thread_class param to Builder
  buildman: Add handle_signals param to Builder
  buildman: Add lazy thread setup to Builder
  buildman: Split build_boards() into two methods
  buildman: Add distributed-build attributes to Builder
  buildman: Cache the kconfig-changed check per commit
  buildman: Add dynamic job-count setting
  buildman: Pass NPROC to make for LTO parallelism
  buildman: Add remote machine probing for distributed builds
  buildman: Install toolchains on remote machines
  buildman: Add worker mode for distributed builds
  buildman: Add boss module for driving remote workers
  buildman: Wire up distributed builds with WorkerPool
  buildman: Document distributed builds in the README

 tools/buildman/boss.py          | 1507 ++++++++++++++++++
 tools/buildman/bsettings.py     |   10 +-
 tools/buildman/builder.py       |  228 ++-
 tools/buildman/builderthread.py |   94 +-
 tools/buildman/buildman.rst     |  132 ++
 tools/buildman/cmdline.py       |   30 +
 tools/buildman/control.py       |  313 +++-
 tools/buildman/func_test.py     |    9 +-
 tools/buildman/machine.py       |  923 +++++++++++
 tools/buildman/main.py          |    6 +
 tools/buildman/test.py          |    3 +
 tools/buildman/test_boss.py     | 2645 +++++++++++++++++++++++++++++++
 tools/buildman/test_builder.py  |  109 +-
 tools/buildman/test_machine.py  | 1080 +++++++++++++
 tools/buildman/test_worker.py   |  896 +++++++++++
 tools/buildman/worker.py        |  985 ++++++++++++
 tools/u_boot_pylib/command.py   |   13 +-
 tools/u_boot_pylib/test_util.py |   49 +-
 18 files changed, 8877 insertions(+), 155 deletions(-)
 create mode 100644 tools/buildman/boss.py
 create mode 100644 tools/buildman/machine.py
 create mode 100644 tools/buildman/test_boss.py
 create mode 100644 tools/buildman/test_machine.py
 create mode 100644 tools/buildman/test_worker.py
 create mode 100644 tools/buildman/worker.py