From patchwork Mon Mar 16 15:47:18 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 2006 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1773676138; bh=E9CFJIpMl8qxFx7DVCCKFcpbXryxXOUcA9FQ58oOqSg=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=QPVr0YqakgN2n7WD1Aj1BUWz8X6NyJRI/SYIe/5jRBnanqKiY+tMOCUETrFCulMIs VqSWRE4pzwOKtD0gtbEodMc92eO83mETgHIV+LJ5x6dv3aBqEQxCDnUHXPaS83BSMS nYLOx6liakvqeUofI91TbtHaR4Mp4h8uJSgmNROrZNi87qnxrPpGGlFJVeVNlAp+82 MYDkgFX2v2jLHh2GUkt+LwgajD4K1rgDpu6C74M0PP88gr88U1bUSbc2DYQv14HUhU kp4mTJOQXn8+oOsQn6C80dRmU/7D7dqJgdTs7YASPDyBSCDzUL+SFCHaDhpd4fyRN5 tn7gQnG0hyK9w== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 6A7A96A0A5 for ; Mon, 16 Mar 2026 09:48:58 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id owaklQtfZ6KR for ; Mon, 16 Mar 2026 09:48:58 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1773676136; bh=E9CFJIpMl8qxFx7DVCCKFcpbXryxXOUcA9FQ58oOqSg=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=wiEH+LbCqJE688QHwUjbTjuABbOEwrkUe1F6+U1ORYSb0DBcXWm1lnS3Mp7PKsliF cIv2WGgRyy9raAMSzjcxfaCRlxwfw8j5iexHIiqfhDuRvCkwHlLUHw6chkuAzrjqFp WYhndSBihXWSiZ+bwX1K9Oy8xOraoodLXQNjqMXjxLmNBEoTbY0c4rJusJqDVlt2zD z9p7nNh+frgZCwUtVBK3aEWg6lFJ9edSSREo4Guw3r1ncDVfbRzG0/TuVFnoDusR7k GnH3cgMJbTq6Xlusfs56GnzAPFl8b/RJZokntehzMlX/t8213uNv2RKqTID9hEOvUi SMFI+FLqnfr/A== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id D5C646A09A for ; Mon, 16 Mar 2026 09:48:56 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1773676133; bh=Uwp2rRQI56Bl7megx4Z9wZcNZQpXj8Ye+KQ86xIA21c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PM7oF8BQbFr6+8uB9gVptVHEQQ6LH1rpoTl11hWd3CAUQPcKK5DwTJjWB+fszXxPz x8N5i8xnKHe3KQovsMNhS2Il2GxIPryLHTrgBoY3UTKBY8qLFEbI3CWFx57DYLtNTB ndj04DaTHisIV46pCFBENWv2pOpJuJI+i7EawhO1sSOy5Im0Xaa6PrW6yf+7NprQf+ bfzF2LL27sSDrok3k7EAKL/VYNDDlyksj2jLWs8R6Oyqhvq928UZOAyCr8vYR6Iwkj w7UjHTxGlNnAo8Yss4YhZoLXakjstakQWeVd6kP+2X7rXDigL33q2a6jNlnXPhttiT EyIL2atGFOSQA== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id B446F6A085; Mon, 16 Mar 2026 09:48:53 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id YKRmt74fqW3c; Mon, 16 Mar 2026 09:48:53 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1773676131; bh=qD4lnEus3TtaRWui8WnKwWwYj1NxHg5nOnU96btiqQo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hzl9+iHsNeeXexU2ejw5R26I2M9jzloAC57Ox6rqF2COv8EgLg5FKcv8t8PMoaGDy E15h2osS2YSlSkmYFmf895+H+w9kOXpUTwlnz/2Je94hMwuQtCITwCBUPU/t6ilj0y 5k1msJsywMsCKkHyE0AdGMIPf1CC4x74ep4MnnwW5qaTQBD2iwpCPyOLEmlW2nnuqR yvZno+ehXbzr4prBKBkCzCg66xq4mB/HAIp13iw1Y0aabsRsBG9SXRgmTAIr1TQNHt L255Ql6mASqge2OZUCqAhywhHhVppkc6Eko0CXqbRgzk4yV0kdnJmy4/WWIa9+dubA Gy7hK2L55Z2Iw== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id 2B56B6A086; Mon, 16 Mar 2026 09:48:51 -0600 (MDT) From: Simon Glass To: U-Boot Concept Date: Mon, 16 Mar 2026 09:47:18 -0600 Message-ID: <20260316154733.1587261-14-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260316154733.1587261-1-sjg@u-boot.org> References: <20260316154733.1587261-1-sjg@u-boot.org> MIME-Version: 1.0 Message-ID-Hash: VN52EEQSBWDYRBG6GHE35S7ZE4TGQ5WH X-Message-ID-Hash: VN52EEQSBWDYRBG6GHE35S7ZE4TGQ5WH X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Simon Glass X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 13/20] buildman: Add dynamic job-count setting List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass When many boards are in flight on a high-thread machine, each make gets only -j1 (nthreads / active_boards = 256/256), causing poor efficiency when the machine is mostly idle. Track the number of in-flight boards with a locked counter in BuilderThread.run(). When Builder.num_jobs is None (meaning dynamic mode), calculate -j as nthreads / active_boards. When the in-flight count drops below the baseline (max_boards or nthreads), ramp it up by a factor of two to use more of the available CPU power. Rename _num_threads and _active_lock to public since BuilderThread accesses them from a separate module. Signed-off-by: Simon Glass --- tools/buildman/builder.py | 16 ++++++++-------- tools/buildman/builderthread.py | 19 +++++++++++++++++-- 2 files changed, 25 insertions(+), 10 deletions(-) diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py index de895d43cfe..c4ec22dbc77 100644 --- a/tools/buildman/builder.py +++ b/tools/buildman/builder.py @@ -199,7 +199,7 @@ class Builder: _complete_delay: Expected delay until completion (timedelta) _next_delay_update: Next time we plan to display a progress update (datatime) - _num_threads: Number of builder threads to run + num_threads: Number of builder threads to run _opts: DisplayOptions for result output _re_make_err: Compiled regex for make error detection _restarting_config: True if 'Restart config' is detected in output @@ -304,11 +304,11 @@ class Builder: self.do_make = make_func or self.make self.gnu_make = gnu_make self.checkout = checkout - self._num_threads = num_threads + self.num_threads = num_threads self.num_jobs = num_jobs self.active_boards = 0 self.max_boards = 0 - self._active_lock = threading.Lock() + self.active_lock = threading.Lock() self._already_done = 0 self.kconfig_reconfig = 0 self.force_build = False @@ -403,11 +403,11 @@ class Builder: test_thread_exceptions (bool): True to make threads raise an exception instead of reporting their result (for tests) """ - if self._num_threads: + if self.num_threads: self._single_builder = None self.queue = queue.Queue() self.out_queue = queue.Queue() - for i in range(self._num_threads): + for i in range(self.num_threads): t = self._thread_class( self, i, mrproper, per_board_out_dir, test_exception=test_thread_exceptions) @@ -1227,7 +1227,7 @@ class Builder: self._result_handler.reset_result_summary(board_selected) builderthread.mkdir(self.base_dir, parents = True) - self._prepare_working_space(min(self._num_threads, len(board_selected)), + self._prepare_working_space(min(self.num_threads, len(board_selected)), board_selected and commits is not None) self._prepare_output_space() if not self._opts.ide: @@ -1247,7 +1247,7 @@ class Builder: job.adjust_cfg = self.adjust_cfg job.fragments = fragments job.step = self._step - if self._num_threads: + if self.num_threads: self.queue.put(job) else: self._single_builder.run_job(job) @@ -1268,7 +1268,7 @@ class Builder: - number of boards that issued warnings - list of thread exceptions raised """ - if self._num_threads: + if self.num_threads: term = threading.Thread(target=self.queue.join) term.daemon = True term.start() diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py index 13c98612c81..16534196d4d 100644 --- a/tools/buildman/builderthread.py +++ b/tools/buildman/builderthread.py @@ -332,8 +332,18 @@ class BuilderThread(threading.Thread): args.append('V=1') else: args.append('-s') - if self.builder.num_jobs is not None: - args.extend(['-j', str(self.builder.num_jobs)]) + num_jobs = self.builder.num_jobs + if num_jobs is None and self.builder.active_boards: + active = self.builder.active_boards + nthreads = self.builder.num_threads + baseline = self.builder.max_boards or nthreads + if active < baseline: + # Tail: ramp up 2x to compensate for make overhead + num_jobs = max(1, nthreads * 2 // active) + else: + num_jobs = max(1, nthreads // active) + if num_jobs is not None: + args.extend(['-j', str(num_jobs)]) if self.builder.warnings_as_errors: args.append('KCFLAGS=-Werror') args.append('HOSTCFLAGS=-Werror') @@ -1022,10 +1032,15 @@ class BuilderThread(threading.Thread): """ while True: job = self.builder.queue.get() + with self.builder.active_lock: + self.builder.active_boards += 1 try: self.run_job(job) except Exception as exc: # pylint: disable=W0718 print('Thread exception (use -T0 to run without threads):', exc) self.builder.thread_exceptions.append(exc) + finally: + with self.builder.active_lock: + self.builder.active_boards -= 1 self.builder.queue.task_done()