From patchwork Sun Mar 29 11:10:26 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 2064 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782696; bh=SyfS5rC2D80AT0+pnpictfYoz3R/hRIkBlfFfTewT18=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=ha3Esr9oKjSvo0U/+L/Bfi/Kb1aXsn0o6ZubEiuKeHHHfk9XhrnmaqKaM8zLAvtW1 OLy6bbMwZ69HLYWLK0xFGJYUaaLIvVI6Z96efJstxmNBRuW1t3VlsPJ2kEZQ5kFtLC IsyUqRmTQaFMN+drnebvvNpZhLi0rFyPVMAHspHKxaQufM1ZHQtctih2xTzl8NI/5V uz8PG+0r2MK0XTzJ8JTZ93+67FCaYbyuJlB85KBRUHeoWqZs4wUsnIMSzl1lwk0op+ +aNTG8QK7jlp56hk+KMGhZA8bg9v9nP0s5lPi/eOceZWTlV2NiAzIgHNur7ckYMxRB jsaxCBtCbAYww== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 3A6CE6A2BE for ; Sun, 29 Mar 2026 05:11:36 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 7Wj6tNZSP4gy for ; Sun, 29 Mar 2026 05:11:36 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782696; bh=SyfS5rC2D80AT0+pnpictfYoz3R/hRIkBlfFfTewT18=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=ha3Esr9oKjSvo0U/+L/Bfi/Kb1aXsn0o6ZubEiuKeHHHfk9XhrnmaqKaM8zLAvtW1 OLy6bbMwZ69HLYWLK0xFGJYUaaLIvVI6Z96efJstxmNBRuW1t3VlsPJ2kEZQ5kFtLC IsyUqRmTQaFMN+drnebvvNpZhLi0rFyPVMAHspHKxaQufM1ZHQtctih2xTzl8NI/5V uz8PG+0r2MK0XTzJ8JTZ93+67FCaYbyuJlB85KBRUHeoWqZs4wUsnIMSzl1lwk0op+ +aNTG8QK7jlp56hk+KMGhZA8bg9v9nP0s5lPi/eOceZWTlV2NiAzIgHNur7ckYMxRB jsaxCBtCbAYww== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 2267B6A2C0 for ; Sun, 29 Mar 2026 05:11:36 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782693; bh=ft/5ndk51bBVqph2s/IxzNA5vx7ntXcSahh5sb+iUAU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KiYVRCTZEyRftmhXwGecQzqROK0rzHDs7U7oyLVXYtxtcQDcv6AYXHEtrBpy07ypd Ikr1vIox1lds5AuB49sGUE5ghisQp/h5i92WH0ufI97uORsmepyNWl8s58oMMAsw15 bEHQFCYYQt711KO4kUsl4ORbHM2L3RE2W+CITD8+eElzI7qIt3lGeSO9UPFdoWF1DT tSVXIv6AbgoLdGVcApn5cLT4OdyizurBHf/Sq86CTa+q0o2VUR68MhspJfKH2JmQqV A9bsiSfaLNF8ZgFA7y4YoTahiMQ4DuhhZUWkj1rWVEgm0ly8ne/z/xoHOms+mNI+co gkpULXWzBf9Mg== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 334DB6A2BE; Sun, 29 Mar 2026 05:11:33 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id G-bw7rxm-ECg; Sun, 29 Mar 2026 05:11:33 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782688; bh=rs8xnrvtq9aWbdnidfyeSWNnzo7y0zHCKLYgllfkT2A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IBsfl52tY7FFqa6rfYNkI1/FSg4QZxzXNxBYC7ko89P96ciHTthgflcARcgtUbSIV q358CrfgyghQ8erJjyubpi0OpZK6tMR+Ierx3MDNte0LV+WM4ohNBr+DL39UprVBjl h+cnmpQyKIiXNXabKJpxrOHSkkqJvbShlBVgJxw2w0crZm5Dw1e3Y9OmQydSVxrC7W v45eC8/ulCE13X6r5RbkFqU0yy/HhqmKPPRVLVJm429wRrFuh3y8K4hEJkAxbUxbyA gqAMXxxt8YELX0FblktgagJT14lxZO8/h8+jTLb1kywegMRMe73CNOHgLndfedxnZ3 ns17GSt67jvkw== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id A4DA06A2BC; Sun, 29 Mar 2026 05:11:28 -0600 (MDT) From: Simon Glass To: U-Boot Concept Date: Sun, 29 Mar 2026 05:10:26 -0600 Message-ID: <20260329111037.1352652-2-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260329111037.1352652-1-sjg@u-boot.org> References: <20260329111037.1352652-1-sjg@u-boot.org> MIME-Version: 1.0 Message-ID-Hash: 5BNR54SEDHZD7BLRBJSDJIBA6S3ZU6MI X-Message-ID-Hash: 5BNR54SEDHZD7BLRBJSDJIBA6S3ZU6MI X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Simon Glass X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 1/7] qconfig: Use kconfiglib to build the CONFIG database List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass The -b option currently spawns two make subprocesses per board (make defconfig + make auto.conf), requiring cross-compiler toolchains for every architecture and taking several minutes for the full set of ~1500 boards. Replace this with direct kconfiglib evaluation. Each worker-process parses the Kconfig tree once, then loads each defconfig in its chunk using kconf.load_config() and reads the resolved CONFIG values via sym.config_string. This avoids all overhead related to subprocess and toolchains, reducing the full database build from minutes to under two seconds. The multiprocessing pattern follows the existing approach in buildman/boards.py, including handling of #include directives in defconfig files via the C preprocessor. Signed-off-by: Simon Glass --- doc/develop/qconfig.rst | 50 ++++++++++++--- tools/qconfig.py | 132 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 169 insertions(+), 13 deletions(-) diff --git a/doc/develop/qconfig.rst b/doc/develop/qconfig.rst index 3b995355967..423fb9118d8 100644 --- a/doc/develop/qconfig.rst +++ b/doc/develop/qconfig.rst @@ -24,14 +24,43 @@ You may need to install 'python3-asteval' for the 'asteval' module. How does it work? ----------------- -When building a database (`-b`), this tool runs configuration and builds -include/autoconf.mk for every defconfig. The config options defined in Kconfig -appear in the .config file (unless they are hidden because of unmet dependency.) -On the other hand, the config options defined by board headers are seen -in include/autoconf.mk. +Building a database +~~~~~~~~~~~~~~~~~~~ + +When building a database (`-b`), this tool evaluates the Kconfig tree directly +using kconfiglib (a pure-Python Kconfig implementation). For each defconfig, it +loads the file with ``kconf.load_config()``, resolves all symbol dependencies, +and collects the resulting CONFIG values. This is the same approach used by +``buildman`` when scanning board parameters. + +Multiple worker processes run in parallel (one per CPU core by default, +adjustable with ``-j``), each parsing the Kconfig tree once and then processing +its share of defconfigs. This completes in a few seconds for all ~1500 boards, +since no ``make`` subprocesses or cross-compiler toolchains are needed. + +Defconfig files containing ``#include`` directives are preprocessed with the +C preprocessor before loading, matching the behaviour of the build system. + +There are two known cosmetic differences compared with the old make-based +approach: + +- ``CONFIG_GCC_VERSION`` reflects the host compiler rather than each board's + cross-compiler, since no cross-compiler is invoked. + +- Backslash escape sequences in string values (e.g. ``\n``, ``\x1b``) may + differ slightly due to kconfiglib's unescape/escape round-trip. This affects + a handful of string CONFIGs such as ``CONFIG_AUTOBOOT_PROMPT``. + +Neither difference affects the usefulness of the database for finding CONFIG +combinations or computing imply relationships. + +Resyncing defconfigs +~~~~~~~~~~~~~~~~~~~~ When resyncing defconfigs (`-s`) the .config is synced by "make savedefconfig" -and the defconfig is updated with it. +and the defconfig is updated with it. This path still uses ``make`` +subprocesses and therefore requires appropriate cross-compiler toolchains (see +below). For faster processing, this tool is multi-threaded. It creates separate build directories where the out-of-tree build is run. The @@ -44,9 +73,12 @@ Note that `*.config` fragments are not supported. Toolchains ---------- -Appropriate toolchains are necessary to generate include/autoconf.mk -for all the architectures supported by U-Boot. Most of them are available -at the kernel.org site. This tool uses the same tools as +Toolchains are **not** needed for building the database (`-b`), since it +uses kconfiglib to evaluate Kconfig files directly in Python. + +For resyncing defconfigs (`-s`), appropriate toolchains are necessary to run +``make savedefconfig`` for all the architectures supported by U-Boot. Most of +them are available at the kernel.org site. This tool uses the same tools as :doc:`../build/buildman`, so you can use `buildman --fetch-arch` to fetch toolchains. diff --git a/tools/qconfig.py b/tools/qconfig.py index 70d53b5f8d9..4e06bfec88b 100755 --- a/tools/qconfig.py +++ b/tools/qconfig.py @@ -283,6 +283,129 @@ def scan_kconfig(): return kconfiglib.Kconfig() +def _scan_defconfigs_worker(srcdir, defconfigs, queue, error_queue): + """Worker process that scans defconfigs using kconfiglib + + Each worker creates its own Kconfig instance (parsing is done once per + process) then loads each defconfig in turn, collecting all CONFIG values. + + Args: + srcdir (str): Source-tree directory + defconfigs (list of str): Defconfig filenames to process, e.g. + ['sandbox_defconfig', 'snow_defconfig'] + queue (multiprocessing.Queue): Output queue for (defconfig, configs) + error_queue (multiprocessing.Queue): Output queue for failed defconfigs + """ + os.environ['srctree'] = srcdir + os.environ['UBOOTVERSION'] = 'dummy' + os.environ['KCONFIG_OBJDIR'] = '' + os.environ['CC'] = 'gcc' + kconf = kconfiglib.Kconfig(warn=False) + + for defconfig in defconfigs: + fname = os.path.join(srcdir, 'configs', defconfig) + try: + if b'#include' in tools.read_file(fname): + cpp = os.getenv('CPP', 'cpp').split() + cmd = cpp + ['-nostdinc', '-P', '-I', srcdir, + '-undef', '-x', 'assembler-with-cpp', fname] + stdout = subprocess.check_output(cmd, stderr=subprocess.DEVNULL) + tmp = tempfile.NamedTemporaryFile(prefix='qconfig-', delete=False) + tmp.write(stdout) + tmp.close() + kconf.load_config(tmp.name) + os.unlink(tmp.name) + else: + kconf.load_config(fname) + + configs = {} + for sym in kconf.unique_defined_syms: + conf = sym.config_string + if not conf or conf.startswith('#'): + continue + config, value = conf.rstrip('\n').split('=', 1) + configs[config] = value + queue.put((defconfig, configs)) + except Exception as exc: + error_queue.put((defconfig, str(exc))) + + +def do_build_db(args): + """Build the CONFIG database using kconfiglib instead of make + + This evaluates the Kconfig tree directly in Python for each defconfig, + avoiding the overhead of spawning make subprocesses and the need for + cross-compiler toolchains. + + Args: + args (Namespace): Program arguments (uses jobs, defconfigs, + defconfiglist, nocolour) + + Returns: + tuple: + config_db (dict): configs for each defconfig + Progress: progress indicator + """ + srcdir = os.getcwd() + + if args.defconfigs: + defconfigs = [os.path.basename(d) + for d in get_matched_defconfigs(args.defconfigs)] + elif args.defconfiglist: + defconfigs = [os.path.basename(d) + for d in get_matched_defconfigs(args.defconfiglist)] + else: + defconfigs = get_all_defconfigs() + + col = terminal.Color(terminal.COLOR_NEVER if args.nocolour + else terminal.COLOR_IF_TERMINAL) + progress = Progress(col, len(defconfigs)) + + jobs = args.jobs + total = len(defconfigs) + result_queue = multiprocessing.Queue() + error_queue = multiprocessing.Queue() + processes = [] + for i in range(jobs): + chunk = defconfigs[total * i // jobs:total * (i + 1) // jobs] + if not chunk: + continue + proc = multiprocessing.Process( + target=_scan_defconfigs_worker, + args=(srcdir, chunk, result_queue, error_queue)) + proc.start() + processes.append(proc) + + config_db = {} + remaining = total + while remaining: + # Drain both queues without blocking forever + found = False + while not result_queue.empty(): + defconfig, configs = result_queue.get() + config_db[defconfig] = configs + progress.inc(True) + progress.show() + remaining -= 1 + found = True + while not error_queue.empty(): + defconfig, msg = error_queue.get() + print(col.build(col.RED, f'{defconfig}: {msg}', bright=True), + file=sys.stderr) + progress.inc(False) + progress.show() + remaining -= 1 + found = True + if not found: + time.sleep(SLEEP_TIME) + + for proc in processes: + proc.join() + + progress.completed() + return config_db, progress + + # pylint: disable=R0903 class KconfigParser: """A parser of .config and include/autoconf.mk.""" @@ -1745,7 +1868,7 @@ def ensure_database(threads): dry_run=False, exit_on_error=False, jobs=threads, git_ref=None, defconfigs=None, defconfiglist=None, nocolour=False) - config_db, progress = move_config(args) + config_db, progress = do_build_db(args) write_db(config_db, progress) @@ -1772,13 +1895,14 @@ def main(): if args.find: return do_find_config(args.configs, args.list) + if args.build_db: + config_db, progress = do_build_db(args) + return write_db(config_db, progress) + config_db, progress = move_config(args) if args.commit: add_commit(args.configs) - - if args.build_db: - return write_db(config_db, progress) return move_done(progress)