From patchwork Mon Mar 16 15:47:14 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 2001 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1773676130; bh=QO5fJq3qTaZt/9cqnEEiQtlFJ+8/Mz3OLbgUqczqtIE=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=Hu8hv7edhfA7wWlfBoNBXVBbCRgZboHKmjl2wOSCjsksmLk1kkdMVUS5O5ZuLXX/w bIoBVUwV+3IdfDk0MhMeKaqpLiKYLs95/f7tCLazx812oru8WUTVtFhr77zIQjUHgT 2KiM4ZtlKHtxT3cHfvEB3vCW4pUrKvFkEyB1qEj53veMBmopVAQat7iQ9X0bpV3tKZ mHr7hGna9h9Bt8XUzvC5K1ZDrdxnd71vRRWR6mdFsoz66d8TwoqIPIrmBUuNN39Ay+ xK2Vy64uLjRziTvhvEqo6I8xqV1MgP7XS7mYOaw5ByaYKArhsItT8cvP5gfj+qGTSK ouRjSYdbNVjuw== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id BE07D6A096 for ; Mon, 16 Mar 2026 09:48:50 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 4hI18yrpPGnw for ; Mon, 16 Mar 2026 09:48:50 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1773676128; bh=QO5fJq3qTaZt/9cqnEEiQtlFJ+8/Mz3OLbgUqczqtIE=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=s6WUKfAVy7GDysQ2hwiT3756ZxtZqrTefXJh1zoIXEltH644GlxoT/mGutEBGGLMl zBescXLa6jw7+rI76Q5KkLIjMxJs2YlxNSEITMFFNw1cohg+a3YySoGjjs+vSHxiFr gdixgSaOAtZFeZLZBE2a5FutIILbaLFGycx93zK+93NpbLeALPBNDIly5sKeI2isP1 bhLkSz/5KpFFmfhM/slQyW/OoZLkv+0jDG5mzTeJtOXIbTVZPC5SsaTMfpvdbEU1Os okwCD4oxBGxOXrlu+hsjf7gha1Aoc8WbkYs4A2vNOlqXyUnk39w5/scjKAOy98zUX6 6VaX6ynTwYOpA== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id B669A6A061 for ; Mon, 16 Mar 2026 09:48:48 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1773676126; bh=yebeGejYyA8iLcSe6nVUvYrrACaQmG9rg7dgbn2lQo8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=wzoVVKGOGOVc/F01UnAkf9Q9u5MzbEv8HjUCYNrJlBneDn7oSE9aP4h4Eg7TTNRso 4hMy04/TOo9E/6DPS65ntKyHB6oWJ3IFTD998tBJoAKZsf6ZCynQXcFVIdZt/tKgPg SKbwJmzRcRQ35ohFJsZRPhdm+wUMv/w9cjPp99rEUMqoXGrXiCTroru0SZ1nXo0Wdn 15aA1lRxRLc65GvvABchV7HfJ62H5G04JbgTua+kxY3XGsRMyJIbABsG0CW+9SYbFe gT6cYL8ZiQpXQm3v7c5K9MW3UwQHIj/7BcmBBC/4PkMxs6GBte0tr9DxOYI9haJIS8 CABEfe09Iki4g== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 951036A081; Mon, 16 Mar 2026 09:48:46 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id RWh_QcYthQrD; Mon, 16 Mar 2026 09:48:46 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1773676124; bh=iYHGBBzPI+JZ1CYxEWpRLU7r4lFjjXBVawQfWUjoRjo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jmowwghuFg6oofZBrjo4NTWVB3k3Ue7QvHitKE5Kn942/5/tLo8IHnfTtKWamRIIJ W/sGNlWLHK+Vl0wd7+tuRs7w5eom5TdWHrOI1/6zdEpJaafP9zIfQuEXE4JU6oVX1l in3EuA25tGDPVwTUNLKBMGnrgCmQKMjhgjYyyy11qngHSBTdAQAjgd3GiP3ioM+GrX RZ3sbOWj5ZQCFgez+u83mnWkYJrpQ6eUO+ZLEgNAfLl2Mm1Mmc81AGAnrMwCTXCWLs umKGtyyqAP3jAE9tM1ncRfzndPKmZb8WAGpQO48IBxMkCXYycbLSl9UrmXn9ZQXHEZ YeoSZFRthnsLQ== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id 2EE3D6A089; Mon, 16 Mar 2026 09:48:44 -0600 (MDT) From: Simon Glass To: U-Boot Concept Date: Mon, 16 Mar 2026 09:47:14 -0600 Message-ID: <20260316154733.1587261-10-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260316154733.1587261-1-sjg@u-boot.org> References: <20260316154733.1587261-1-sjg@u-boot.org> MIME-Version: 1.0 Message-ID-Hash: VTKEF2JPUIPTK46UPOQFTKGG5LEVJ3X2 X-Message-ID-Hash: VTKEF2JPUIPTK46UPOQFTKGG5LEVJ3X2 X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Simon Glass X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 09/20] buildman: Add lazy thread setup to Builder List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass Setting up worktrees or clones for every thread sequentially in _prepare_working_space() is slow on machines with many threads (e.g. 256 on ruru), and the distributed-build worker does not need it since each thread sets up its own worktree on first use. Add a lazy_thread_setup parameter that skips the per-thread setup loop. Only the working directory is created and the git setup type (worktree vs clone) is determined via a new _detect_git_setup() method and stored in _setup_git, so that the deferred prepare_thread() calls can use it. Extract _detect_git_setup() from _prepare_working_space() so that both the eager and lazy paths share the same detection logic. Signed-off-by: Simon Glass --- tools/buildman/builder.py | 50 ++++++++++++++++++++++++++++------ tools/buildman/test_builder.py | 24 ++++++++++++++-- 2 files changed, 63 insertions(+), 11 deletions(-) diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py index 53ec604f654..8c83203cf8e 100644 --- a/tools/buildman/builder.py +++ b/tools/buildman/builder.py @@ -232,7 +232,7 @@ class Builder: in_tree=False, force_config_on_failure=False, make_func=None, dtc_skip=False, build_target=None, thread_class=builderthread.BuilderThread, - handle_signals=True): + handle_signals=True, lazy_thread_setup=False): """Create a new Builder object Args: @@ -310,6 +310,7 @@ class Builder: self.kconfig_reconfig = 0 self.force_build = False self.git_dir = git_dir + self._setup_git = False self._timestamp_count = 10 self._build_period_us = None self._complete_delay = None @@ -377,6 +378,7 @@ class Builder: # Note: baseline state for result summaries is now in ResultHandler self._thread_class = thread_class + self._lazy_thread_setup = lazy_thread_setup self._setup_threads(mrproper, per_board_out_dir, test_thread_exceptions) ignore_lines = ['(make.*Waiting for unfinished)', @@ -1047,6 +1049,18 @@ class Builder: return self._working_dir return os.path.join(self._working_dir, f'{max(thread_num, 0):02d}') + def prepare_thread(self, thread_num): + """Prepare a single thread's working directory on demand + + This can be called by a BuilderThread to lazily set up its + worktree/clone on first use, rather than doing all threads upfront. + Uses the git setup method determined by _detect_git_setup(). + + Args: + thread_num (int): Thread number (0, 1, ...) + """ + self._prepare_thread(thread_num, self._setup_git) + def _prepare_thread(self, thread_num, setup_git): """Prepare the working directory for a thread. @@ -1103,6 +1117,11 @@ class Builder: Set up the git repo for each thread. Creates a linked working tree if git-worktree is available, or clones the repo if it isn't. + When lazy_thread_setup is True, only the working directory and git + setup type are determined here. Each thread sets up its own + worktree/clone on first use via prepare_thread(), which avoids a + long sequential setup phase on machines with many threads. + Args: max_threads: Maximum number of threads we expect to need. If 0 then 1 is set up, since the main process still needs somewhere to @@ -1110,20 +1129,35 @@ class Builder: setup_git: True to set up a git worktree or a git clone """ builderthread.mkdir(self._working_dir) + + self._setup_git = self._detect_git_setup(setup_git) + + if self._lazy_thread_setup: + return + + # Always do at least one thread + for thread in range(max(max_threads, 1)): + self._prepare_thread(thread, self._setup_git) + + def _detect_git_setup(self, setup_git): + """Determine which git setup method to use + + Args: + setup_git: True to set up git, False to skip + + Returns: + str or False: 'worktree', 'clone', or False + """ if setup_git and self.git_dir: src_dir = os.path.abspath(self.git_dir) if gitutil.check_worktree_is_available(src_dir): - setup_git = 'worktree' # If we previously added a worktree but the directory for it # got deleted, we need to prune its files from the repo so # that we can check out another in its place. gitutil.prune_worktrees(src_dir) - else: - setup_git = 'clone' - - # Always do at least one thread - for thread in range(max(max_threads, 1)): - self._prepare_thread(thread, setup_git) + return 'worktree' + return 'clone' + return False def _get_output_space_removals(self): """Get the output directories ready to receive files. diff --git a/tools/buildman/test_builder.py b/tools/buildman/test_builder.py index 70a8b365f2a..48be83cf645 100644 --- a/tools/buildman/test_builder.py +++ b/tools/buildman/test_builder.py @@ -354,10 +354,28 @@ class TestPrepareWorkingSpace(unittest.TestCase): self.builder.git_dir = None self.builder._prepare_working_space(2, True) - # setup_git should remain True but git operations skipped + # _detect_git_setup returns False when git_dir is None self.assertEqual(mock_prepare_thread.call_count, 2) - mock_prepare_thread.assert_any_call(0, True) - mock_prepare_thread.assert_any_call(1, True) + mock_prepare_thread.assert_any_call(0, False) + mock_prepare_thread.assert_any_call(1, False) + + @mock.patch.object(builder.Builder, '_prepare_thread') + @mock.patch.object(gitutil, 'prune_worktrees') + @mock.patch.object(gitutil, 'check_worktree_is_available', + return_value=True) + @mock.patch.object(builderthread, 'mkdir') + def test_lazy_setup(self, _mock_mkdir, mock_check_worktree, + mock_prune, mock_prepare_thread): + """Test lazy_thread_setup skips upfront thread preparation""" + self.builder._lazy_thread_setup = True + self.builder._prepare_working_space(4, True) + + # Git setup type is detected so prepare_thread() can use it + # later, but no threads are prepared upfront + self.assertEqual(self.builder._setup_git, 'worktree') + mock_check_worktree.assert_called_once() + mock_prune.assert_called_once() + mock_prepare_thread.assert_not_called() class TestShowNotBuilt(unittest.TestCase):