From patchwork Mon Mar 16 15:47:05 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 132 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1773676075; bh=llueSEmN0R6YByQNS/RbWmGa0Eh+Qwn7mLSAuCYZWqw=; h=From:To:Date:CC:Subject:List-Id:List-Archive:List-Help:List-Owner: List-Post:List-Subscribe:List-Unsubscribe:From; b=KM5CuzZ0s+zb1mrqNJcwHL5ezdj2qzad2SPpmIC4PqkSVFDPNh+CXD49UzEqPve/n d1z//xTuJ9VzURluFUy0sBqAplyARp2wdkhL6vzyqLtuay553Oy06U9uRuF8Z0jFI+ bBCdJrvgeWt/aWT+ga6gdOBczaS2McXmtR5c/FBkeZsuMCvfg+FgZ3NP7wkmEbOQ+o UcpsS45w4BSjzeD4LffXpNLW7FNfYwDSviCiZJNHD5IQqqLpEH1puNS6zt6DlE6Cch VKoJEiXDfaAL2PxE22c5/4f1UvP0l2SGtJDKfz6GfvnEY0HsJ9mzqZWDAyapAxu3Md yTdW105OgY2RA== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 59D8C6A081 for ; Mon, 16 Mar 2026 09:47:55 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id KjArlPk3va-q for ; Mon, 16 Mar 2026 09:47:55 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1773676075; bh=llueSEmN0R6YByQNS/RbWmGa0Eh+Qwn7mLSAuCYZWqw=; h=From:To:Date:CC:Subject:List-Id:List-Archive:List-Help:List-Owner: List-Post:List-Subscribe:List-Unsubscribe:From; b=KM5CuzZ0s+zb1mrqNJcwHL5ezdj2qzad2SPpmIC4PqkSVFDPNh+CXD49UzEqPve/n d1z//xTuJ9VzURluFUy0sBqAplyARp2wdkhL6vzyqLtuay553Oy06U9uRuF8Z0jFI+ bBCdJrvgeWt/aWT+ga6gdOBczaS2McXmtR5c/FBkeZsuMCvfg+FgZ3NP7wkmEbOQ+o UcpsS45w4BSjzeD4LffXpNLW7FNfYwDSviCiZJNHD5IQqqLpEH1puNS6zt6DlE6Cch VKoJEiXDfaAL2PxE22c5/4f1UvP0l2SGtJDKfz6GfvnEY0HsJ9mzqZWDAyapAxu3Md yTdW105OgY2RA== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 482216A07C for ; Mon, 16 Mar 2026 09:47:55 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1773676071; bh=QOwrxOlKEI9sV0Z7/zEp2DPjf15WthOvjd7mriIH//A=; h=From:To:Cc:Subject:Date:From; b=qaa2QWqpyK5aldwmc+YKwmRpS4CI5lltl+LF6E3xaQWHi9yGAJoHt85rnk+LB7LJD hYJp3rPvv9OlKLbz/zWeTE3h9uf/q/QFjD6DXrHcg7VZDCtGPMbgSEShzS3avXauNB j+B37zfmbn9ecRjgxhLWON8bZYLAuIw9Oi98I/B9p62acR8NoKUhqFOqLuBpZ5Z87O nXEJCxsUhQUGg788UmYQ0XwwJEPklFSvvwbOUyvcPT3OaJGqocG1p1NFBUluDfEwK1 LYVPyShg/+aE+paYIN6ytdb3LfhQ5BHamYEhl4buK2zHdhko74hgjyfhDZ4ZK3fZew Xz2Io33ilqjhg== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 560146A07C; Mon, 16 Mar 2026 09:47:51 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id MP6fzmgPfF7e; Mon, 16 Mar 2026 09:47:51 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1773676067; bh=drnPrSSY65r5gYt8V6RMPCRke+Bn8BFnS2JJW/RTov0=; h=From:To:Cc:Subject:Date:From; b=HWt+HEZBznjxhsTuY41l7KDwF/bA9SkdZCyWuZIx1X4RQRgrQI3W6aUpNjk7K2L2C dh75boOvj0eA3XMwbQF3OWsKc8veyuHgKxwahh+QeWEJ3TYx+79icRphzEQ0Y6ER0W 1w0rOJ8HeS0/wIZvy7uv4+6QzITsNcBndl+gnCB8eRcAVmREIb332PrT0MtppRBV7U /DfQ3yYwbkS4HljeI+f4jMvKi3qxTmndqXW+AtdWqPzWWl+igItanWgdsucNbM7OBJ Slly2qrPINv7gP3IJy4OvN8YPGC7Q20pUjaAiURVBCnbodAdLlP91c5mVmAcPqFN45 aaCoV46sOXysg== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id 955E86A073; Mon, 16 Mar 2026 09:47:47 -0600 (MDT) From: Simon Glass To: U-Boot Concept Date: Mon, 16 Mar 2026 09:47:05 -0600 Message-ID: <20260316154733.1587261-1-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 Message-ID-Hash: JIBBXTW3IUPLNOM7UJZ2J2NITJMD47UR X-Message-ID-Hash: JIBBXTW3IUPLNOM7UJZ2J2NITJMD47UR X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Simon Glass X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 00/20] buildman: Add distributed builds List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass This series adds distributed build support to buildman, allowing board builds to be spread across a pool of remote machines over SSH. The first few patches prepare Builder for distributed use: adding parameters for custom thread classes, signal handling, lazy worktree setup, and splitting build_boards() so remote workers can queue boards incrementally. The machine module probes remote hosts over SSH to collect their capabilities (CPUs, memory, load, toolchains) and manages toolchain version checking and fetching. The worker module runs on remote machines in --worker mode, accepting JSON commands on stdin and streaming results on stdout. It reuses Builder and BuilderThread with a custom subclass that sends results over the SSH pipe instead of writing to disk. The boss module manages SSH connections to workers, handling startup, source push via git, demand-driven board dispatch, result collection with timeouts, and clean shutdown on Ctrl-C. The wire-up patch connects everything through control.py with --dist, --use-machines, and --no-local flags. Simon Glass (20): u_boot_pylib: Add stdin_data support to run_pipe() u_boot_pylib: Support passing modules to run_test_suites() buildman: Remove unused imports from builder.py buildman: Fix pylint warnings in test_builder.py buildman: Fix import order in control.py buildman: Initialise _timestamps to an empty deque buildman: Add thread_class param to Builder buildman: Add handle_signals param to Builder buildman: Add lazy thread setup to Builder buildman: Split build_boards() into two methods buildman: Add distributed-build attributes to Builder buildman: Cache the kconfig-changed check per commit buildman: Add dynamic job-count setting buildman: Pass NPROC to make for LTO parallelism buildman: Add remote machine probing for distributed builds buildman: Install toolchains on remote machines buildman: Add worker mode for distributed builds buildman: Add boss module for driving remote workers buildman: Wire up distributed builds with WorkerPool buildman: Document distributed builds in the README tools/buildman/boss.py | 1507 ++++++++++++++++++ tools/buildman/bsettings.py | 10 +- tools/buildman/builder.py | 228 ++- tools/buildman/builderthread.py | 94 +- tools/buildman/buildman.rst | 132 ++ tools/buildman/cmdline.py | 30 + tools/buildman/control.py | 313 +++- tools/buildman/func_test.py | 9 +- tools/buildman/machine.py | 923 +++++++++++ tools/buildman/main.py | 6 + tools/buildman/test.py | 3 + tools/buildman/test_boss.py | 2645 +++++++++++++++++++++++++++++++ tools/buildman/test_builder.py | 109 +- tools/buildman/test_machine.py | 1080 +++++++++++++ tools/buildman/test_worker.py | 896 +++++++++++ tools/buildman/worker.py | 985 ++++++++++++ tools/u_boot_pylib/command.py | 13 +- tools/u_boot_pylib/test_util.py | 49 +- 18 files changed, 8877 insertions(+), 155 deletions(-) create mode 100644 tools/buildman/boss.py create mode 100644 tools/buildman/machine.py create mode 100644 tools/buildman/test_boss.py create mode 100644 tools/buildman/test_machine.py create mode 100644 tools/buildman/test_worker.py create mode 100644 tools/buildman/worker.py