From patchwork Sun Mar 29 11:10:26 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 2064 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782696; bh=SyfS5rC2D80AT0+pnpictfYoz3R/hRIkBlfFfTewT18=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=ha3Esr9oKjSvo0U/+L/Bfi/Kb1aXsn0o6ZubEiuKeHHHfk9XhrnmaqKaM8zLAvtW1 OLy6bbMwZ69HLYWLK0xFGJYUaaLIvVI6Z96efJstxmNBRuW1t3VlsPJ2kEZQ5kFtLC IsyUqRmTQaFMN+drnebvvNpZhLi0rFyPVMAHspHKxaQufM1ZHQtctih2xTzl8NI/5V uz8PG+0r2MK0XTzJ8JTZ93+67FCaYbyuJlB85KBRUHeoWqZs4wUsnIMSzl1lwk0op+ +aNTG8QK7jlp56hk+KMGhZA8bg9v9nP0s5lPi/eOceZWTlV2NiAzIgHNur7ckYMxRB jsaxCBtCbAYww== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 3A6CE6A2BE for ; Sun, 29 Mar 2026 05:11:36 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 7Wj6tNZSP4gy for ; Sun, 29 Mar 2026 05:11:36 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782696; bh=SyfS5rC2D80AT0+pnpictfYoz3R/hRIkBlfFfTewT18=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=ha3Esr9oKjSvo0U/+L/Bfi/Kb1aXsn0o6ZubEiuKeHHHfk9XhrnmaqKaM8zLAvtW1 OLy6bbMwZ69HLYWLK0xFGJYUaaLIvVI6Z96efJstxmNBRuW1t3VlsPJ2kEZQ5kFtLC IsyUqRmTQaFMN+drnebvvNpZhLi0rFyPVMAHspHKxaQufM1ZHQtctih2xTzl8NI/5V uz8PG+0r2MK0XTzJ8JTZ93+67FCaYbyuJlB85KBRUHeoWqZs4wUsnIMSzl1lwk0op+ +aNTG8QK7jlp56hk+KMGhZA8bg9v9nP0s5lPi/eOceZWTlV2NiAzIgHNur7ckYMxRB jsaxCBtCbAYww== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 2267B6A2C0 for ; Sun, 29 Mar 2026 05:11:36 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782693; bh=ft/5ndk51bBVqph2s/IxzNA5vx7ntXcSahh5sb+iUAU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KiYVRCTZEyRftmhXwGecQzqROK0rzHDs7U7oyLVXYtxtcQDcv6AYXHEtrBpy07ypd Ikr1vIox1lds5AuB49sGUE5ghisQp/h5i92WH0ufI97uORsmepyNWl8s58oMMAsw15 bEHQFCYYQt711KO4kUsl4ORbHM2L3RE2W+CITD8+eElzI7qIt3lGeSO9UPFdoWF1DT tSVXIv6AbgoLdGVcApn5cLT4OdyizurBHf/Sq86CTa+q0o2VUR68MhspJfKH2JmQqV A9bsiSfaLNF8ZgFA7y4YoTahiMQ4DuhhZUWkj1rWVEgm0ly8ne/z/xoHOms+mNI+co gkpULXWzBf9Mg== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 334DB6A2BE; Sun, 29 Mar 2026 05:11:33 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id G-bw7rxm-ECg; Sun, 29 Mar 2026 05:11:33 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782688; bh=rs8xnrvtq9aWbdnidfyeSWNnzo7y0zHCKLYgllfkT2A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IBsfl52tY7FFqa6rfYNkI1/FSg4QZxzXNxBYC7ko89P96ciHTthgflcARcgtUbSIV q358CrfgyghQ8erJjyubpi0OpZK6tMR+Ierx3MDNte0LV+WM4ohNBr+DL39UprVBjl h+cnmpQyKIiXNXabKJpxrOHSkkqJvbShlBVgJxw2w0crZm5Dw1e3Y9OmQydSVxrC7W v45eC8/ulCE13X6r5RbkFqU0yy/HhqmKPPRVLVJm429wRrFuh3y8K4hEJkAxbUxbyA gqAMXxxt8YELX0FblktgagJT14lxZO8/h8+jTLb1kywegMRMe73CNOHgLndfedxnZ3 ns17GSt67jvkw== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id A4DA06A2BC; Sun, 29 Mar 2026 05:11:28 -0600 (MDT) From: Simon Glass To: U-Boot Concept Date: Sun, 29 Mar 2026 05:10:26 -0600 Message-ID: <20260329111037.1352652-2-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260329111037.1352652-1-sjg@u-boot.org> References: <20260329111037.1352652-1-sjg@u-boot.org> MIME-Version: 1.0 Message-ID-Hash: 5BNR54SEDHZD7BLRBJSDJIBA6S3ZU6MI X-Message-ID-Hash: 5BNR54SEDHZD7BLRBJSDJIBA6S3ZU6MI X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Simon Glass X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 1/7] qconfig: Use kconfiglib to build the CONFIG database List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass The -b option currently spawns two make subprocesses per board (make defconfig + make auto.conf), requiring cross-compiler toolchains for every architecture and taking several minutes for the full set of ~1500 boards. Replace this with direct kconfiglib evaluation. Each worker-process parses the Kconfig tree once, then loads each defconfig in its chunk using kconf.load_config() and reads the resolved CONFIG values via sym.config_string. This avoids all overhead related to subprocess and toolchains, reducing the full database build from minutes to under two seconds. The multiprocessing pattern follows the existing approach in buildman/boards.py, including handling of #include directives in defconfig files via the C preprocessor. Signed-off-by: Simon Glass --- doc/develop/qconfig.rst | 50 ++++++++++++--- tools/qconfig.py | 132 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 169 insertions(+), 13 deletions(-) diff --git a/doc/develop/qconfig.rst b/doc/develop/qconfig.rst index 3b995355967..423fb9118d8 100644 --- a/doc/develop/qconfig.rst +++ b/doc/develop/qconfig.rst @@ -24,14 +24,43 @@ You may need to install 'python3-asteval' for the 'asteval' module. How does it work? ----------------- -When building a database (`-b`), this tool runs configuration and builds -include/autoconf.mk for every defconfig. The config options defined in Kconfig -appear in the .config file (unless they are hidden because of unmet dependency.) -On the other hand, the config options defined by board headers are seen -in include/autoconf.mk. +Building a database +~~~~~~~~~~~~~~~~~~~ + +When building a database (`-b`), this tool evaluates the Kconfig tree directly +using kconfiglib (a pure-Python Kconfig implementation). For each defconfig, it +loads the file with ``kconf.load_config()``, resolves all symbol dependencies, +and collects the resulting CONFIG values. This is the same approach used by +``buildman`` when scanning board parameters. + +Multiple worker processes run in parallel (one per CPU core by default, +adjustable with ``-j``), each parsing the Kconfig tree once and then processing +its share of defconfigs. This completes in a few seconds for all ~1500 boards, +since no ``make`` subprocesses or cross-compiler toolchains are needed. + +Defconfig files containing ``#include`` directives are preprocessed with the +C preprocessor before loading, matching the behaviour of the build system. + +There are two known cosmetic differences compared with the old make-based +approach: + +- ``CONFIG_GCC_VERSION`` reflects the host compiler rather than each board's + cross-compiler, since no cross-compiler is invoked. + +- Backslash escape sequences in string values (e.g. ``\n``, ``\x1b``) may + differ slightly due to kconfiglib's unescape/escape round-trip. This affects + a handful of string CONFIGs such as ``CONFIG_AUTOBOOT_PROMPT``. + +Neither difference affects the usefulness of the database for finding CONFIG +combinations or computing imply relationships. + +Resyncing defconfigs +~~~~~~~~~~~~~~~~~~~~ When resyncing defconfigs (`-s`) the .config is synced by "make savedefconfig" -and the defconfig is updated with it. +and the defconfig is updated with it. This path still uses ``make`` +subprocesses and therefore requires appropriate cross-compiler toolchains (see +below). For faster processing, this tool is multi-threaded. It creates separate build directories where the out-of-tree build is run. The @@ -44,9 +73,12 @@ Note that `*.config` fragments are not supported. Toolchains ---------- -Appropriate toolchains are necessary to generate include/autoconf.mk -for all the architectures supported by U-Boot. Most of them are available -at the kernel.org site. This tool uses the same tools as +Toolchains are **not** needed for building the database (`-b`), since it +uses kconfiglib to evaluate Kconfig files directly in Python. + +For resyncing defconfigs (`-s`), appropriate toolchains are necessary to run +``make savedefconfig`` for all the architectures supported by U-Boot. Most of +them are available at the kernel.org site. This tool uses the same tools as :doc:`../build/buildman`, so you can use `buildman --fetch-arch` to fetch toolchains. diff --git a/tools/qconfig.py b/tools/qconfig.py index 70d53b5f8d9..4e06bfec88b 100755 --- a/tools/qconfig.py +++ b/tools/qconfig.py @@ -283,6 +283,129 @@ def scan_kconfig(): return kconfiglib.Kconfig() +def _scan_defconfigs_worker(srcdir, defconfigs, queue, error_queue): + """Worker process that scans defconfigs using kconfiglib + + Each worker creates its own Kconfig instance (parsing is done once per + process) then loads each defconfig in turn, collecting all CONFIG values. + + Args: + srcdir (str): Source-tree directory + defconfigs (list of str): Defconfig filenames to process, e.g. + ['sandbox_defconfig', 'snow_defconfig'] + queue (multiprocessing.Queue): Output queue for (defconfig, configs) + error_queue (multiprocessing.Queue): Output queue for failed defconfigs + """ + os.environ['srctree'] = srcdir + os.environ['UBOOTVERSION'] = 'dummy' + os.environ['KCONFIG_OBJDIR'] = '' + os.environ['CC'] = 'gcc' + kconf = kconfiglib.Kconfig(warn=False) + + for defconfig in defconfigs: + fname = os.path.join(srcdir, 'configs', defconfig) + try: + if b'#include' in tools.read_file(fname): + cpp = os.getenv('CPP', 'cpp').split() + cmd = cpp + ['-nostdinc', '-P', '-I', srcdir, + '-undef', '-x', 'assembler-with-cpp', fname] + stdout = subprocess.check_output(cmd, stderr=subprocess.DEVNULL) + tmp = tempfile.NamedTemporaryFile(prefix='qconfig-', delete=False) + tmp.write(stdout) + tmp.close() + kconf.load_config(tmp.name) + os.unlink(tmp.name) + else: + kconf.load_config(fname) + + configs = {} + for sym in kconf.unique_defined_syms: + conf = sym.config_string + if not conf or conf.startswith('#'): + continue + config, value = conf.rstrip('\n').split('=', 1) + configs[config] = value + queue.put((defconfig, configs)) + except Exception as exc: + error_queue.put((defconfig, str(exc))) + + +def do_build_db(args): + """Build the CONFIG database using kconfiglib instead of make + + This evaluates the Kconfig tree directly in Python for each defconfig, + avoiding the overhead of spawning make subprocesses and the need for + cross-compiler toolchains. + + Args: + args (Namespace): Program arguments (uses jobs, defconfigs, + defconfiglist, nocolour) + + Returns: + tuple: + config_db (dict): configs for each defconfig + Progress: progress indicator + """ + srcdir = os.getcwd() + + if args.defconfigs: + defconfigs = [os.path.basename(d) + for d in get_matched_defconfigs(args.defconfigs)] + elif args.defconfiglist: + defconfigs = [os.path.basename(d) + for d in get_matched_defconfigs(args.defconfiglist)] + else: + defconfigs = get_all_defconfigs() + + col = terminal.Color(terminal.COLOR_NEVER if args.nocolour + else terminal.COLOR_IF_TERMINAL) + progress = Progress(col, len(defconfigs)) + + jobs = args.jobs + total = len(defconfigs) + result_queue = multiprocessing.Queue() + error_queue = multiprocessing.Queue() + processes = [] + for i in range(jobs): + chunk = defconfigs[total * i // jobs:total * (i + 1) // jobs] + if not chunk: + continue + proc = multiprocessing.Process( + target=_scan_defconfigs_worker, + args=(srcdir, chunk, result_queue, error_queue)) + proc.start() + processes.append(proc) + + config_db = {} + remaining = total + while remaining: + # Drain both queues without blocking forever + found = False + while not result_queue.empty(): + defconfig, configs = result_queue.get() + config_db[defconfig] = configs + progress.inc(True) + progress.show() + remaining -= 1 + found = True + while not error_queue.empty(): + defconfig, msg = error_queue.get() + print(col.build(col.RED, f'{defconfig}: {msg}', bright=True), + file=sys.stderr) + progress.inc(False) + progress.show() + remaining -= 1 + found = True + if not found: + time.sleep(SLEEP_TIME) + + for proc in processes: + proc.join() + + progress.completed() + return config_db, progress + + # pylint: disable=R0903 class KconfigParser: """A parser of .config and include/autoconf.mk.""" @@ -1745,7 +1868,7 @@ def ensure_database(threads): dry_run=False, exit_on_error=False, jobs=threads, git_ref=None, defconfigs=None, defconfiglist=None, nocolour=False) - config_db, progress = move_config(args) + config_db, progress = do_build_db(args) write_db(config_db, progress) @@ -1772,13 +1895,14 @@ def main(): if args.find: return do_find_config(args.configs, args.list) + if args.build_db: + config_db, progress = do_build_db(args) + return write_db(config_db, progress) + config_db, progress = move_config(args) if args.commit: add_commit(args.configs) - - if args.build_db: - return write_db(config_db, progress) return move_done(progress) From patchwork Sun Mar 29 11:10:27 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 2065 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782702; bh=iNLWLqKRD/EsCREXDCnA+umKKqH6pmajH8mbi3sXePI=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=llEnAsh6Hg1IKIEoiXg81kFdVYEU/8m7tppeJ7kgbNZviNh0C8CEM/BMs4ZzKtTjJ TbeiFnFcCpoxw7i32ABN3DePP5tRVDMYtQt/So1ZsBHh1TzpaU6psOwPZMO/Q0s58i dB+MWC9OBBlyigxFP2zMZLWJ3kzKzGWokJfbK2QmOBJODYGwSeN/pEUeytjpWX2xU8 etjELJtCCvUvXj8QgK37MemuOxi6h05NbPpU36r45sReVO73bQOFQI4J7MfRjI3bim t48j+HJehOn3KwanWBnKZyKDpDP7ggMESLyHbXChjzfcJ2UZJk91ldhnYqlpbjSoBN 5SSe+2j3IeP0g== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id D5E2C6A2BE for ; Sun, 29 Mar 2026 05:11:42 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id EnKwTIMKl_Iq for ; Sun, 29 Mar 2026 05:11:42 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782702; bh=iNLWLqKRD/EsCREXDCnA+umKKqH6pmajH8mbi3sXePI=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=llEnAsh6Hg1IKIEoiXg81kFdVYEU/8m7tppeJ7kgbNZviNh0C8CEM/BMs4ZzKtTjJ TbeiFnFcCpoxw7i32ABN3DePP5tRVDMYtQt/So1ZsBHh1TzpaU6psOwPZMO/Q0s58i dB+MWC9OBBlyigxFP2zMZLWJ3kzKzGWokJfbK2QmOBJODYGwSeN/pEUeytjpWX2xU8 etjELJtCCvUvXj8QgK37MemuOxi6h05NbPpU36r45sReVO73bQOFQI4J7MfRjI3bim t48j+HJehOn3KwanWBnKZyKDpDP7ggMESLyHbXChjzfcJ2UZJk91ldhnYqlpbjSoBN 5SSe+2j3IeP0g== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id C43116A2BF for ; Sun, 29 Mar 2026 05:11:42 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782700; bh=CKSPjZMX+KEVUje9nzFcSFP2ZCQr9U3Dqvl49OawdhU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HpjsOsGTRoykmWViFgRkZOBTUk+4aGxNbNSI1YvCJW1m6EUZONKvQ4oYNkKn/wDZk n4jCivyNlCL44LX1pAkMcdRJdJKDO1EPrcbemHyA/hKuWrVGbGk1tiRNUDNmmyKcCJ NHxAwUFHbRLWqhTM/1apPKE4gl8CHaSEpsjt/JgZd/sA2bQ2hrURLo5Ss9qlPMNtOf PRx8cTVU+okrIyEQmGta023s0pAuS+E/kewEigSFVicorgqe6CPXPzvGGRcQO1MqsH 7qu/uJowjVsWnvy9c8fJftyEg3dMd9XpSZulmFN65VBbgmNuWJMIm4L8whiVXbeujX EvAEcjFZY5FtQ== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id ACAD76A2BE; Sun, 29 Mar 2026 05:11:40 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id YQKdytCOlW_H; Sun, 29 Mar 2026 05:11:40 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782694; bh=xHIJVjaZIN7Uv03Xdv4Xi6Xo2VWNysJbXl7NLRY4GDQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iDeombw0t1gyaqbOquPZxe1/qJ8klqzeUfvCyKNfAiJBwBuG3KRMopoGcH+hl0d+Z wCZGXyt6FFznKQmSWn6pqBzhbufE8M9XGbowfXWrrj2/LgQEBSVal0wxVZm6LvrrhW mviHx8PFmpHiDtyq95mf5nN0FcPS++/tUhtYb6G4VPoX8Ka2EqfMlv+HfsWm0G7GCs epRe/1CJ9KkU/R7HaHk8lmZDqBzhQETeA3B6cgtLdOsdBJxMIJvxrzBVxGF3E92z3K EG9ExWTDmQoWILO3ppknJ6VLK4dV+Ehi+NUmjkkwza4itpNFyT6ZRqYVL7s9MQq1GB xrB47s4u9QfDw== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id 5B8D26A2BC; Sun, 29 Mar 2026 05:11:34 -0600 (MDT) From: Simon Glass To: U-Boot Concept Date: Sun, 29 Mar 2026 05:10:27 -0600 Message-ID: <20260329111037.1352652-3-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260329111037.1352652-1-sjg@u-boot.org> References: <20260329111037.1352652-1-sjg@u-boot.org> MIME-Version: 1.0 Message-ID-Hash: CVR5YH3HNXHGMKTWYGNWWF7A32CS3RNR X-Message-ID-Hash: CVR5YH3HNXHGMKTWYGNWWF7A32CS3RNR X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Simon Glass X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 2/7] qconfig: Auto-rebuild stale database for -f queries List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass The -f (find) option calls read_database() directly, which crashes if the database does not exist and silently returns stale data if Kconfig or defconfig files have changed since the last build. Add a db_is_current() check that compares the database timestamp against all Kconfig and defconfig files, similar to output_is_new() in buildman/boards.py. Use this in ensure_database() so that both -f and buildman's callers get automatic rebuilds when the database is stale. Wire do_find_config() through ensure_database() instead of read_database() so that -f benefits from this. Signed-off-by: Simon Glass --- tools/qconfig.py | 45 +++++++++++++++++++++++++++++++++++++-------- 1 file changed, 37 insertions(+), 8 deletions(-) diff --git a/tools/qconfig.py b/tools/qconfig.py index 4e06bfec88b..1e1f74a0c85 100755 --- a/tools/qconfig.py +++ b/tools/qconfig.py @@ -1289,9 +1289,11 @@ def find_config(dbase, config_list): return result -def do_find_config(config_list, list_format): +def do_find_config(config_list, list_format, jobs): """Find boards with a given combination of CONFIGs + Rebuilds the database automatically if it is missing or stale. + Args: config_list (list of str): List of CONFIG options to check (each a regex consisting of a config option, with or without a CONFIG_ prefix. If @@ -1299,11 +1301,12 @@ def do_find_config(config_list, list_format): otherwise it must be true) list_format (bool): True to write in 'list' format, one board name per line + jobs (int): Number of threads to use if the database needs rebuilding Returns: int: exit code (0 for success) """ - dbase = read_database() + dbase = ensure_database(jobs) out = find_config(dbase, config_list) if not list_format: print(f'{len(out)} matches') @@ -1840,11 +1843,37 @@ def do_tests(): return 0 +def db_is_current(): + """Check if the CONFIG database is up to date + + Returns: + bool: True if the database exists and is newer than all Kconfig and + defconfig files + """ + if not os.path.exists(CONFIG_DATABASE): + return False + + db_time = os.path.getctime(CONFIG_DATABASE) + + for dirpath, _, filenames in os.walk('configs'): + for fname in fnmatch.filter(filenames, '*_defconfig'): + if db_time < os.path.getctime(os.path.join(dirpath, fname)): + return False + + for dirpath, _, filenames in os.walk('.'): + for fname in filenames: + if fname.startswith('Kconfig'): + if db_time < os.path.getctime(os.path.join(dirpath, fname)): + return False + + return True + + def ensure_database(threads): - """Return a qconfig database so that Kconfig options can be queried + """Return a qconfig database, rebuilding it if stale or missing - If a database exists, it is assumed to be up-to-date. If not, one is built, - which can take a few minutes. + Checks whether the database is newer than all Kconfig and defconfig files. + If not, it is rebuilt automatically. Args: threads (int): Number of threads to use when processing @@ -1862,8 +1891,8 @@ def ensure_database(threads): key: CONFIG option value: set of boards using that option """ - if not os.path.exists(CONFIG_DATABASE): - print('Building qconfig.db database') + if not db_is_current(): + print('Building qconfig.db database...') args = Namespace(build_db=True, verbose=False, force_sync=False, dry_run=False, exit_on_error=False, jobs=threads, git_ref=None, defconfigs=None, defconfiglist=None, @@ -1893,7 +1922,7 @@ def main(): sys.exit(1) return 0 if args.find: - return do_find_config(args.configs, args.list) + return do_find_config(args.configs, args.list, args.jobs) if args.build_db: config_db, progress = do_build_db(args) From patchwork Sun Mar 29 11:10:28 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 2066 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782705; bh=Tzr/CXv5iQif/SsFKUHpLBbvEDygtkPeDMT0wKdDt68=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=HdGH6/Gc5492g22Ny4Y/fwPdowZ7N9RWYxzF8jFvzf5gjf2axtGJ+SOAMJyL713/M TmnaVXfAV/2DYAPJephZZeUVAFvGAjXao0Z3lCmw0b1qhyjGngmgkahRzYvFFUaKzF AaYsJeNPiCAizyUS98BugQ5ZWa7db9WyeQ+VsV1klLMA55plWxk6+iOjMO22kXhGc5 FnL3RtkMf0ntz2YWtDrLgB3/qYKy+Z+6sGWfxVTyofWy6cG1TdKg/AIHcyAhabmIWO zrznanAe5VPH2Uwp9wzrx3kdT73kja4ouV9FgpHIjgSaEw16x5ZlfoOwhLyW9y2YkY 3lw6iDNviXmDQ== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id DB76C6A2BE for ; Sun, 29 Mar 2026 05:11:45 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id aimC_VFRNSdz for ; Sun, 29 Mar 2026 05:11:45 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782704; bh=Tzr/CXv5iQif/SsFKUHpLBbvEDygtkPeDMT0wKdDt68=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=jO0tPqZncIlwTQnP3SJAhikHzGCJx1jZf2i+OduhgG6EQC1AtxG3UZ1H2o7KJyR02 pHutwlVdonVoFseIB0iG5uZzNaNSnegk9iVMPTvXSXWqgxAC9xnjylTzI8+qRpebrY M3wWRokSszOYDhkgudQwgA7xcJ+IJQ2n8GOsC0DK3Bo+4XXCWxPNCtdQKpcEYeY6X5 xHt81nu5kvymXJ1lpU0LDI69FIm2wV1imHpp55d0oumsZlrYFlUG7XbGD0/7/O/3fi A9Y7ty3/kaUU/RSJfzpw56dvfeABWeMhcTpMGEKKsf16Qjq7EeCVBH+UZUraY/ctnv LyeFy3L3O3Cdw== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id CDD5E6A2C1 for ; Sun, 29 Mar 2026 05:11:44 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782702; bh=NBm9qeJHlrScxZ1gk1spdp+I3R5bA3WkCzvdyjKHOb8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=REj8nQkY9+Vv+EAfTWvEY+1yHm7+SCc8C3ucTmALKjX/1tG1o5eiDrNTz67Cjp79l et5k7BcPVdd7UaHWkmaMNp6Fy0u2yyYi0gY7cPPa/qV8p4ujXDeqVHWIczHsox70PY twXYe0VONcHrPDKhl4tCQUmW4Jg/NnwDQun26rxPTJXbuaURXpUMtcwjJFTSZAEtxI TfPlj4e7pa7YYtVVqXkc8sdebWcDDkWo/9l/haR6UI7zF9pPSRdNfj4xWp+l5Hns/W hJRDZ7/R7LybwNFQ5qxYuYcQJikQVgSULO/lX2eevVlxr2Zew9FZCZVnje1/z7LXC5 gtG0qDTwaoFwA== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id A4B016A2BC; Sun, 29 Mar 2026 05:11:42 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id HBx6jcCzApR1; Sun, 29 Mar 2026 05:11:42 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782698; bh=c+Cd/kM9xl6YjE31tiVEwk41q09YtVS3BHCE2OGdW8U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NKSmUIJhw0IQbTpIGcHXaNSnXU48reNVLIIKpeZaw4yyzOnKyhdEMzFZH119m/eLi edRx6CD8U6DkfIJOL5AkKWemrUxyx9V25njMU7rpw1ScSeXQIgP231MeghlxfpNV8A VEHOazDSgo9sSNtR9XQ+KroPgRpjuX4TIVFNTzcLiL8H4M7u5WfxLUKeUjFi6oRhE7 3KUWC/OvzB4+qtAGcl2lEVTCS52OreAgvzEfZvu9rjW2YJ/RNzci4VzqoFgQ6qVRo7 1tZUf02fG85vTmaQvgf8HU/reMmNYXGtL+gDFfLJV8BuPcgxmyY8vY18k27JvcRLdo 9PWp/hO5s4IAw== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id B181B6A2C1; Sun, 29 Mar 2026 05:11:38 -0600 (MDT) From: Simon Glass To: U-Boot Concept Date: Sun, 29 Mar 2026 05:10:28 -0600 Message-ID: <20260329111037.1352652-4-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260329111037.1352652-1-sjg@u-boot.org> References: <20260329111037.1352652-1-sjg@u-boot.org> MIME-Version: 1.0 Message-ID-Hash: 4VLRVPHPPMF3MWC5PB2XTTV74JYGGWHG X-Message-ID-Hash: 4VLRVPHPPMF3MWC5PB2XTTV74JYGGWHG X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Simon Glass X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 3/7] qconfig: Use kconfiglib for defconfig sync List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass The -s option currently spawns two make subprocesses per board (make defconfig + make savedefconfig), requiring cross-compiler toolchains for every architecture. Replace this with kconfiglib's load_config() and write_min_config(), which produces output matching 'make savedefconfig'. This uses the same multiprocessing pattern as the -b optimisation, completing a full sync of ~1500 boards in under two seconds with no toolchain requirement. The -r (git-ref) option still uses the old make-based path, since it needs to build against a different source tree. Signed-off-by: Simon Glass --- tools/qconfig.py | 138 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 138 insertions(+) -- 2.43.0 diff --git a/tools/qconfig.py b/tools/qconfig.py index 1e1f74a0c85..133fea41212 100755 --- a/tools/qconfig.py +++ b/tools/qconfig.py @@ -406,6 +406,138 @@ def do_build_db(args): return config_db, progress +def _sync_defconfigs_worker(srcdir, defconfigs, result_queue, error_queue, + dry_run): + """Worker process that syncs defconfigs using kconfiglib + + For each defconfig, loads it via kconfiglib and writes a minimal config + (equivalent to 'make savedefconfig'), then compares with the original. + + Args: + srcdir (str): Source-tree directory + defconfigs (list of str): Defconfig filenames to process + result_queue (multiprocessing.Queue): Output queue for + (defconfig, updated) tuples + error_queue (multiprocessing.Queue): Output queue for failed defconfigs + dry_run (bool): If True, do not update defconfig files + """ + os.environ['srctree'] = srcdir + os.environ['UBOOTVERSION'] = 'dummy' + os.environ['KCONFIG_OBJDIR'] = '' + os.environ['CC'] = 'gcc' + kconf = kconfiglib.Kconfig(warn=False) + + for defconfig in defconfigs: + orig = os.path.join(srcdir, 'configs', defconfig) + try: + # Skip defconfigs with #include — savedefconfig mangles them + if b'#include' in tools.read_file(orig): + result_queue.put((defconfig, False, 'has #include')) + continue + + kconf.load_config(orig) + + tmp = tempfile.NamedTemporaryFile( + mode='w', prefix='qconfig-', suffix='_defconfig', + dir=os.path.join(srcdir, 'configs'), delete=False) + tmp.close() + kconf.write_min_config(tmp.name) + + updated = not filecmp.cmp(orig, tmp.name) + if updated and not dry_run: + shutil.move(tmp.name, orig) + else: + os.unlink(tmp.name) + result_queue.put((defconfig, updated, None)) + except Exception as exc: + error_queue.put((defconfig, str(exc))) + + +def do_sync_defconfigs(args): + """Sync defconfig files using kconfiglib instead of make + + Evaluates each defconfig through kconfiglib and writes a minimal config + (equivalent to 'make savedefconfig'), updating the original if it differs. + + Args: + args (Namespace): Program arguments (uses jobs, defconfigs, + defconfiglist, nocolour, dry_run, force_sync) + + Returns: + Progress: progress indicator + """ + srcdir = os.getcwd() + + if args.defconfigs: + defconfigs = [os.path.basename(d) + for d in get_matched_defconfigs(args.defconfigs)] + elif args.defconfiglist: + defconfigs = [os.path.basename(d) + for d in get_matched_defconfigs(args.defconfiglist)] + else: + defconfigs = get_all_defconfigs() + + col = terminal.Color(terminal.COLOR_NEVER if args.nocolour + else terminal.COLOR_IF_TERMINAL) + progress = Progress(col, len(defconfigs)) + + jobs = args.jobs + total = len(defconfigs) + result_queue = multiprocessing.Queue() + error_queue = multiprocessing.Queue() + processes = [] + for i in range(jobs): + chunk = defconfigs[total * i // jobs:total * (i + 1) // jobs] + if not chunk: + continue + proc = multiprocessing.Process( + target=_sync_defconfigs_worker, + args=(srcdir, chunk, result_queue, error_queue, args.dry_run)) + proc.start() + processes.append(proc) + + remaining = total + updated_count = 0 + while remaining: + found = False + while not result_queue.empty(): + defconfig, updated, msg = result_queue.get() + if updated: + updated_count += 1 + name = defconfig[:-len('_defconfig')] + log = col.build(col.BLUE, 'defconfig updated', bright=True) + if args.dry_run: + log = col.build(col.YELLOW, 'would update', bright=True) + print(f'{name.ljust(20)} {log}') + elif msg: + name = defconfig[:-len('_defconfig')] + log = col.build(col.RED, f'ignored: {msg}', bright=True) + print(f'{name.ljust(20)} {log}') + progress.inc(True) + progress.show() + remaining -= 1 + found = True + while not error_queue.empty(): + defconfig, msg = error_queue.get() + print(col.build(col.RED, f'{defconfig}: {msg}', bright=True), + file=sys.stderr) + progress.inc(False) + progress.show() + remaining -= 1 + found = True + if not found: + time.sleep(SLEEP_TIME) + + for proc in processes: + proc.join() + + progress.completed() + if updated_count: + print(col.build(col.BLUE, + f'{updated_count} defconfig(s) updated', bright=True)) + return progress + + # pylint: disable=R0903 class KconfigParser: """A parser of .config and include/autoconf.mk.""" @@ -1928,6 +2060,12 @@ def main(): config_db, progress = do_build_db(args) return write_db(config_db, progress) + if args.force_sync and not args.git_ref: + progress = do_sync_defconfigs(args) + if args.commit: + add_commit(args.configs) + return move_done(progress) + config_db, progress = move_config(args) if args.commit: From patchwork Sun Mar 29 11:10:29 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 2067 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782711; bh=lmyStjvQxb92Tn5LYP664frfQcRdYEQJH5EaXThukGI=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=pnTAr+EC52YbTW2O6phx2HC92szs38g5IjSfhEX08eVR3frJSS0q6Cxpc4QrjloyN 3ZuMWIywiMhEvnRKc9ZXhzcnb3+7uoFQ4uqHv5oWK5shNA0UNIlAQgBAfy/PSwSxnK gU0hHN8AQ7NP+FII3RmbbuBmwLs6hIcW5rZQnq8Sdf8zgxi4H7NJeMe0tJ/TZCfry0 uzGf+hgYlwgvIn5EYesNtiVsU47yHkxYudPfDMHWrU7L+lPjwijqgaAYLfMaM0TB77 IB+qgiXqutHn3KPu3vb/JE4dO0LSkOQMkIT2cC/6I358IYOLE2nG/10NeNz8Z1211/ N/NB+n/bL4+4g== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 813766A2BE for ; Sun, 29 Mar 2026 05:11:51 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 3_C-5lIzt3sM for ; Sun, 29 Mar 2026 05:11:51 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782711; bh=lmyStjvQxb92Tn5LYP664frfQcRdYEQJH5EaXThukGI=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=pnTAr+EC52YbTW2O6phx2HC92szs38g5IjSfhEX08eVR3frJSS0q6Cxpc4QrjloyN 3ZuMWIywiMhEvnRKc9ZXhzcnb3+7uoFQ4uqHv5oWK5shNA0UNIlAQgBAfy/PSwSxnK gU0hHN8AQ7NP+FII3RmbbuBmwLs6hIcW5rZQnq8Sdf8zgxi4H7NJeMe0tJ/TZCfry0 uzGf+hgYlwgvIn5EYesNtiVsU47yHkxYudPfDMHWrU7L+lPjwijqgaAYLfMaM0TB77 IB+qgiXqutHn3KPu3vb/JE4dO0LSkOQMkIT2cC/6I358IYOLE2nG/10NeNz8Z1211/ N/NB+n/bL4+4g== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 6891A67C16 for ; Sun, 29 Mar 2026 05:11:51 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782708; bh=YBc0jF/NNVd+KPq6Gxg0U3QywpLWlFbyT0M2i/rK97c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gloXJtYaNEQO0KdJY2rrpmIivZ4X3aNnDe+fjzXennXi3bCmO4LhOBmJPXxGXU9NB Y7oHKMaoCYSvRHXH4TYmGNerBASL5YPOmfugaI0yhaZ8AaKkstQgdYGZIX+UaGYHFz Oc4AfYfiNtpb/O95H5zi36q2bn18jETdTNYcqHyv3LA60fW7oDRopbK6vnc8/VuRw0 K2ky9SmYfoRlkVN0GyHlBYln+G4cwTnrHZrU6Fznfr48w66tLjcaSKUIap1jMzCUUG rBLJTHZ4EtUTPJVN3UulH9QgXa2l314WEf9U+0in1c5CrMKxkRdCA8nnl8OroWJZNk w1s8x4EwBfhTA== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 49DD46A2BE; Sun, 29 Mar 2026 05:11:48 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id u_V1mQjMdyk7; Sun, 29 Mar 2026 05:11:48 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782703; bh=vwNxRJpWHyaz1vQTzgW3o4FPUiYIPaPJ0wrl9uO5J2s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sg77Hk1w7Rd66Glg/q8zshml5PgqtN6L7sqwMo8SEYfpZGcqOTA0HDZgqk3dZSpwU n4yV/9zkezq/asZj9TgXSSKnRLWrby5rfEMIXXjxQME3lzWdMbIsUVXMbyh84/Wkj7 cy81mW7XFnAwCE953U3E11HUXstZrwED7Cw0mH/ptC0A00a1RSNkvyG1/s9DzILvGQ ji/hee1TUGuWPd7ZpIbei/Y++1s+5yS2Mnj1tf7ILWnLRcO7cGTq7C/6Dt+FOfY6dg HNoxG31S9vwXXlr/F47VZW1xiQKtrum2FIlMstsaPYHwOCuJS40/8jf31V0Z4gAwwE JDluWlp1ggqQg== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id E46E76A2BF; Sun, 29 Mar 2026 05:11:42 -0600 (MDT) From: Simon Glass To: U-Boot Concept Date: Sun, 29 Mar 2026 05:10:29 -0600 Message-ID: <20260329111037.1352652-5-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260329111037.1352652-1-sjg@u-boot.org> References: <20260329111037.1352652-1-sjg@u-boot.org> MIME-Version: 1.0 Message-ID-Hash: WTY4QOCWYJX67SNNDMNNILKEXOCEY2QW X-Message-ID-Hash: WTY4QOCWYJX67SNNDMNNILKEXOCEY2QW X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Simon Glass X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 4/7] buildman: Fix kconfiglib string-value escaping List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass The C Kconfig implementation (scripts/kconfig/confdata.c) stores string values from .config files verbatim -- no unescaping on read, no escaping on write. But kconfiglib's unescape() strips backslashes before ANY character (\n -> n, \x -> x), and escape() only re-adds them for \ and ". This means \n in a .config string silently becomes n after a round-trip. Fix unescape() to only handle \" (the only escape that matters for .config parsing), and escape() to only handle " -> \". This makes string values pass through unchanged, matching the C behaviour. Also fix _expand_str() (the Kconfig file string parser) to only consume \\, \", \', and \$ (meaningful Kconfig escapes), leaving \n and similar sequences as-is. Signed-off-by: Simon Glass --- doc/develop/qconfig.rst | 17 +++++------------ tools/buildman/kconfiglib.py | 32 ++++++++++++++++++++------------ 2 files changed, 25 insertions(+), 24 deletions(-) -- 2.43.0 diff --git a/doc/develop/qconfig.rst b/doc/develop/qconfig.rst index 423fb9118d8..431de08cff3 100644 --- a/doc/develop/qconfig.rst +++ b/doc/develop/qconfig.rst @@ -41,18 +41,11 @@ since no ``make`` subprocesses or cross-compiler toolchains are needed. Defconfig files containing ``#include`` directives are preprocessed with the C preprocessor before loading, matching the behaviour of the build system. -There are two known cosmetic differences compared with the old make-based -approach: - -- ``CONFIG_GCC_VERSION`` reflects the host compiler rather than each board's - cross-compiler, since no cross-compiler is invoked. - -- Backslash escape sequences in string values (e.g. ``\n``, ``\x1b``) may - differ slightly due to kconfiglib's unescape/escape round-trip. This affects - a handful of string CONFIGs such as ``CONFIG_AUTOBOOT_PROMPT``. - -Neither difference affects the usefulness of the database for finding CONFIG -combinations or computing imply relationships. +There is one known cosmetic difference compared with the old make-based +approach: ``CONFIG_GCC_VERSION`` reflects the host compiler rather than each +board's cross-compiler, since no cross-compiler is invoked. This does not +affect the usefulness of the database for finding CONFIG combinations or +computing imply relationships. Resyncing defconfigs ~~~~~~~~~~~~~~~~~~~~ diff --git a/tools/buildman/kconfiglib.py b/tools/buildman/kconfiglib.py index 27abbf9a7a1..cdaacf471d8 100644 --- a/tools/buildman/kconfiglib.py +++ b/tools/buildman/kconfiglib.py @@ -2727,10 +2727,16 @@ class Kconfig(object): return (s, match.end()) elif match.group() == "\\": - # Replace '\x' with 'x'. 'i' ends up pointing to the character - # after 'x', which allows macros to be canceled with '\$(foo)'. + # Replace '\x' with 'x' for characters that are meaningful + # escapes in Kconfig string literals: \\, \", \', and \$ + # (to cancel macro expansion). Other \ sequences + # like \n are left as-is, so that the stored value + # round-trips correctly through escape(). i = match.end() - s = s[:match.start()] + s[i:] + if i < len(s) and s[i] in "\\\"'$": + s = s[:match.start()] + s[i:] + else: + i = match.end() elif match.group() == "$(": # A macro call within the string @@ -6176,23 +6182,25 @@ def split_expr(expr, op): def escape(s): r""" - Escapes the string 's' in the same fashion as is done for display in - Kconfig format and when writing strings to a .config file. " and \ are - replaced by \" and \\, respectively. + Escapes the string 's' for writing to a .config file. Only " is escaped + (to \"), matching the symmetric unescape() behaviour. Backslash sequences + like \\ and \n are left as-is, since unescape() preserves them and the C + Kconfig implementation does not process them. """ - # \ must be escaped before " to avoid double escaping - return s.replace("\\", r"\\").replace('"', r'\"') + return s.replace('"', r'\"') def unescape(s): r""" - Unescapes the string 's'. \ followed by any character is replaced with just - that character. Used internally when reading .config files. + Unescapes the string 's'. \" is replaced with ". Other \ sequences, + including \\, are left as-is so that escape() can round-trip them + correctly. This matches the C Kconfig implementation which does not + unescape string values read from .config files. """ return _unescape_sub(r"\1", s) -# unescape() helper -_unescape_sub = re.compile(r"\\(.)").sub +# unescape() helper — only unescape \" +_unescape_sub = re.compile(r'\\(")').sub def standard_kconfig(description=None): From patchwork Sun Mar 29 11:10:30 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 2068 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782714; bh=WjBXEubOEPLft2SdqbOwKETZ7kK8Nrt1fHp30/Z7t00=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=g9vrrp5LLt8NiqXiBIHxKAgt1IwTVqyWqJ+L09DbvKoNFPiarQ+R/up2bgVpM9idr 1CVqE3UKiIJFunZSe7aAVD7DsNTjLyjRFTwHgnHXtO8NXjUITcyzXG0kRhn6acIbt4 XS9gcuchsbEiQnaKihwDpcr6vvL4RxyFevsPeI/p4MA/MfF8UAwdhA0dT5RpJXl1lb 6VH7mqS4MZBnse+Omwv7XspOfDw1/HtXCNDTYHRzpw1ZSRiQ/WEfi4fJn1IJRTbVkU ZuRnmIwY86qsIkfX/ClCs4JI0GbpHHdcx7yimI8bKrAfZNjOLx3Mv3Ev7MSagiLRb8 Q5CbX9YpMqR5w== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id ADBD967C16 for ; Sun, 29 Mar 2026 05:11:54 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id LgDdnrPg7T7Y for ; Sun, 29 Mar 2026 05:11:54 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782712; bh=WjBXEubOEPLft2SdqbOwKETZ7kK8Nrt1fHp30/Z7t00=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=KKHQR8cVHG3TFpMjMdb58JyT043zDXRop2R08VSaYVHVIgKUq293Lg8zi1fjfcRxR MFFwVqW/3dn3eRV5CItsqpu3x5LsyMNrg+lI+QNNrp1H6klJ42EbL3vzMknR9SPx8x 0ofry4Ay066ScVwo3OWrgjBQbn7v9Sfu1TmPY+E5+jMPbCmom5BT5IYxVkoRm2MAU7 9J+jNJVjmOX0VrgtgHDkDTYYWzPuPUeMVLMPOllwoL3iO499WlHrhNQh33cka1/kdl F4iEPJ444t3IHmoe0eZvt7PNxxBfRgXsEtRLLhSjka3tsLq0gtnJaC9rIyN4dbRX3o ib2icdtpv33iw== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id B56726A2C7 for ; Sun, 29 Mar 2026 05:11:52 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782708; bh=/l6j2GlpsVRJT8783ruJ6NO19ORETUa1gO0T9+eh2ZU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LWzqn9zOmvEePIcaQprCmc700FkZK+2VkqxeQJ13aBvgTBhQmn0OWGGE5kRNNKUI4 S9X76/8+Gf6htat74ZF1vTLylOMmWRKIhD/T1XEiG8c2ORdd66ZlN84QIARnBk3IEr eB+pazW7yWLLvXme80WNO9y0Z5Om9KQ6eBVoxqZMtonyPxFY7i7JgtWum6CIAN7A3+ MU3CLa7FkV19Q2nSHcv9nfCm58hMJNs6QluK0Wzpli5KOthxU/swtXGNDKrWC4Uk2/ 7dYU+8eZSp/p4Dp368WyB4rQ097R46IEI53g1/gaMTG9f5osMaMrvv2HwD+byKDqT0 jUH/CrauwZRPw== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 807766A2BF; Sun, 29 Mar 2026 05:11:48 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id oRWNdmMw5LXp; Sun, 29 Mar 2026 05:11:48 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782707; bh=EQco21QK+6k+6mADhVYtv/95cuhFv5SYsck9LvuOMCo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mWBYfQpvkrPjRV27/dxLYygAfkwexPoHsy5Iw3PmNlHZQUZpFumlq/ZeKrFrV4vgy dxObk0+Uy/jK5HQSJ8ktW/DbpacfQtNOgVjxmDjp5Dp4FL2AAE1lfcC8gMkAjT2piu mB0uNgwFmfyHAffIe9dg/827A2wbXDw0Rd9SmcI6F1n+ld4UPa52EvP/sfRLDAre+g GrLxpoG7689C3cScKpnwpIpZhT/N0EUivzY/G3nIy5r2QojBCTeWPcuJ9Dg0Pb4F0J ts+wXuMXBH7Fff9RxeG2aRLB2y8SF1bbZAzovvv7oU0TMlWqneCGzUvt3GtVFTq15O /ZIz2eW8+ppRQ== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id 639CE6A2BC; Sun, 29 Mar 2026 05:11:47 -0600 (MDT) From: Simon Glass To: U-Boot Concept Date: Sun, 29 Mar 2026 05:10:30 -0600 Message-ID: <20260329111037.1352652-6-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260329111037.1352652-1-sjg@u-boot.org> References: <20260329111037.1352652-1-sjg@u-boot.org> MIME-Version: 1.0 Message-ID-Hash: MBWU3MOU7DSV4MHJEFDBYVV4XE6OMSBG X-Message-ID-Hash: MBWU3MOU7DSV4MHJEFDBYVV4XE6OMSBG X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Simon Glass X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 5/7] qconfig: Handle #include defconfigs in kconfiglib sync List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass The -s option skips defconfigs that use #include directives because make savedefconfig does not preserve them. Handle these by computing the delta between the full expanded config and the base config provided by the included files, preserving the include structure while still syncing redundant CONFIGs out of the overlay. The approach is to preprocess just the include lines to get the base config, preprocess the full defconfig to get the combined config, run write_min_config() on both, and use full_min - base_min as the overlay delta. The original include lines are preserved at the top. Also refactor the cpp preprocessing into a shared helper used by both the database-build and sync workers. Signed-off-by: Simon Glass --- doc/develop/qconfig.rst | 45 +++---- tools/qconfig.py | 269 +++++++++++++++++++++++++++++++++++----- 2 files changed, 260 insertions(+), 54 deletions(-) -- 2.43.0 diff --git a/doc/develop/qconfig.rst b/doc/develop/qconfig.rst index 431de08cff3..4a85c78cbbb 100644 --- a/doc/develop/qconfig.rst +++ b/doc/develop/qconfig.rst @@ -41,39 +41,34 @@ since no ``make`` subprocesses or cross-compiler toolchains are needed. Defconfig files containing ``#include`` directives are preprocessed with the C preprocessor before loading, matching the behaviour of the build system. -There is one known cosmetic difference compared with the old make-based -approach: ``CONFIG_GCC_VERSION`` reflects the host compiler rather than each -board's cross-compiler, since no cross-compiler is invoked. This does not -affect the usefulness of the database for finding CONFIG combinations or -computing imply relationships. - Resyncing defconfigs ~~~~~~~~~~~~~~~~~~~~ -When resyncing defconfigs (`-s`) the .config is synced by "make savedefconfig" -and the defconfig is updated with it. This path still uses ``make`` -subprocesses and therefore requires appropriate cross-compiler toolchains (see -below). - -For faster processing, this tool is multi-threaded. It creates -separate build directories where the out-of-tree build is run. The -temporary build directories are automatically created and deleted as -needed. The number of threads are chosen based on the number of the CPU -cores of your system although you can change it via -j (--jobs) option. +When resyncing defconfigs (`-s`), the tool also uses kconfiglib. It loads +each defconfig with ``load_config()`` and writes a minimal config with +``write_min_config()`` (equivalent to ``make savedefconfig``). Defconfigs +that use ``#include`` directives are handled by computing the delta between +the full expanded config and the base provided by the included files, so the +include structure is preserved. -Note that `*.config` fragments are not supported. +The ``-r`` (git-ref) option still uses the old make-based path, since it +needs to build against a different source tree. Toolchains ---------- -Toolchains are **not** needed for building the database (`-b`), since it -uses kconfiglib to evaluate Kconfig files directly in Python. - -For resyncing defconfigs (`-s`), appropriate toolchains are necessary to run -``make savedefconfig`` for all the architectures supported by U-Boot. Most of -them are available at the kernel.org site. This tool uses the same tools as -:doc:`../build/buildman`, so you can use `buildman --fetch-arch` to fetch -toolchains. +Toolchains are **not** needed for ``-b`` or ``-s``, since both use +kconfiglib to evaluate Kconfig files directly in Python. The only +difference from using a real toolchain is that ``CONFIG_GCC_VERSION`` +reflects the host compiler rather than each board's cross-compiler. +This does not affect database queries, imply analysis, or defconfig +sync, since ``CONFIG_GCC_VERSION`` is a build-time value that never +appears in defconfig files or influences Kconfig defaults. + +The ``-r`` (git-ref) option still requires toolchains, as it falls back +to the make-based path. Most toolchains are available at the kernel.org +site. This tool uses the same tools as :doc:`../build/buildman`, so you +can use ``buildman --fetch-arch`` to fetch them. Examples diff --git a/tools/qconfig.py b/tools/qconfig.py index 133fea41212..c0215978319 100755 --- a/tools/qconfig.py +++ b/tools/qconfig.py @@ -283,6 +283,43 @@ def scan_kconfig(): return kconfiglib.Kconfig() +def _cpp_preprocess(srcdir, fname): + """Run the C preprocessor on a file to expand #include directives + + Args: + srcdir (str): Source-tree directory (used as include path) + fname (str): Path to the file to preprocess + + Returns: + str: Path to a temporary file with the preprocessed output. + Caller must delete it. + """ + cpp = os.getenv('CPP', 'cpp').split() + cmd = cpp + ['-nostdinc', '-P', '-I', srcdir, + '-undef', '-x', 'assembler-with-cpp', fname] + stdout = subprocess.check_output(cmd, stderr=subprocess.DEVNULL) + tmp = tempfile.NamedTemporaryFile(prefix='qconfig-', delete=False) + tmp.write(stdout) + tmp.close() + return tmp.name + + +def _load_defconfig(kconf, srcdir, fname): + """Load a defconfig, preprocessing #include directives if present + + Args: + kconf (kconfiglib.Kconfig): Kconfig instance + srcdir (str): Source-tree directory + fname (str): Path to the defconfig file + """ + if b'#include' in tools.read_file(fname): + tmp = _cpp_preprocess(srcdir, fname) + kconf.load_config(tmp) + os.unlink(tmp) + else: + kconf.load_config(fname) + + def _scan_defconfigs_worker(srcdir, defconfigs, queue, error_queue): """Worker process that scans defconfigs using kconfiglib @@ -305,18 +342,7 @@ def _scan_defconfigs_worker(srcdir, defconfigs, queue, error_queue): for defconfig in defconfigs: fname = os.path.join(srcdir, 'configs', defconfig) try: - if b'#include' in tools.read_file(fname): - cpp = os.getenv('CPP', 'cpp').split() - cmd = cpp + ['-nostdinc', '-P', '-I', srcdir, - '-undef', '-x', 'assembler-with-cpp', fname] - stdout = subprocess.check_output(cmd, stderr=subprocess.DEVNULL) - tmp = tempfile.NamedTemporaryFile(prefix='qconfig-', delete=False) - tmp.write(stdout) - tmp.close() - kconf.load_config(tmp.name) - os.unlink(tmp.name) - else: - kconf.load_config(fname) + _load_defconfig(kconf, srcdir, fname) configs = {} for sym in kconf.unique_defined_syms: @@ -406,6 +432,119 @@ def do_build_db(args): return config_db, progress +def _get_min_config_lines(kconf, fname): + """Get the set of minimal config lines for a defconfig + + Args: + kconf (kconfiglib.Kconfig): Kconfig instance (will be modified) + fname (str): Path to preprocessed defconfig (or plain defconfig) + + Returns: + set of str: Lines from write_min_config output (without header) + """ + kconf.load_config(fname) + tmp = tempfile.NamedTemporaryFile(mode='w', prefix='qconfig-mc-', + delete=False) + tmp.close() + kconf.write_min_config(tmp.name) + with open(tmp.name) as inf: + lines = set(inf.readlines()) + os.unlink(tmp.name) + return lines + + +def _sync_plain_defconfig(kconf, orig, dry_run): + """Sync a plain defconfig (no #include) + + Args: + kconf (kconfiglib.Kconfig): Kconfig instance + orig (str): Path to the original defconfig file + dry_run (bool): If True, do not update defconfig files + + Returns: + bool: True if the defconfig was (or would be) updated + """ + kconf.load_config(orig) + confdir = os.path.dirname(orig) + tmp = tempfile.NamedTemporaryFile( + mode='w', prefix='qconfig-', suffix='_defconfig', + dir=confdir, delete=False) + tmp.close() + kconf.write_min_config(tmp.name) + + updated = not filecmp.cmp(orig, tmp.name) + if updated and not dry_run: + shutil.move(tmp.name, orig) + else: + os.unlink(tmp.name) + return updated + + +def _sync_include_defconfig(kconf, srcdir, orig, dry_run): + """Sync a defconfig that uses #include directives + + Computes the minimal delta between the full config and the base config + provided by the included files, preserving the #include structure. + + Args: + kconf (kconfiglib.Kconfig): Kconfig instance + srcdir (str): Source-tree directory + orig (str): Path to the original defconfig file + dry_run (bool): If True, do not update defconfig files + + Returns: + bool: True if the defconfig was (or would be) updated + """ + # Get the full min_config (base + overlay) + full_tmp = _cpp_preprocess(srcdir, orig) + full_lines = _get_min_config_lines(kconf, full_tmp) + os.unlink(full_tmp) + + # Build a temp file with just the #include lines (no overlay CONFIGs) + # to get the base min_config + include_lines = [] + with open(orig, 'rb') as inf: + for line in inf: + if line.startswith(b'#include'): + include_lines.append(line) + + base_tmp = tempfile.NamedTemporaryFile(prefix='qconfig-base-', + suffix='_defconfig', + dir=os.path.dirname(orig), + delete=False) + base_tmp.writelines(include_lines) + base_tmp.close() + + base_pp = _cpp_preprocess(srcdir, base_tmp.name) + os.unlink(base_tmp.name) + base_lines = _get_min_config_lines(kconf, base_pp) + os.unlink(base_pp) + + # Delta = full - base + delta = sorted(full_lines - base_lines) + + # Build the new defconfig: #include lines + delta + # Preserve the separator (blank line or not) from the original + orig_text = tools.read_file(orig, binary=False) + last_include_idx = orig_text.rfind('#include') + after_include = orig_text[orig_text.index('\n', last_include_idx) + 1:] + sep = b'\n' if after_include.startswith('\n') else b'' + + new_content = b'' + for line in include_lines: + new_content += line + if delta: + new_content += sep + for line in delta: + new_content += line.encode() if isinstance(line, str) else line + + orig_content = tools.read_file(orig) + updated = new_content != orig_content + if updated and not dry_run: + tools.write_file(orig, new_content) + return updated + + def _sync_defconfigs_worker(srcdir, defconfigs, result_queue, error_queue, dry_run): """Worker process that syncs defconfigs using kconfiglib @@ -430,24 +569,14 @@ def _sync_defconfigs_worker(srcdir, defconfigs, result_queue, error_queue, for defconfig in defconfigs: orig = os.path.join(srcdir, 'configs', defconfig) try: - # Skip defconfigs with #include — savedefconfig mangles them - if b'#include' in tools.read_file(orig): - result_queue.put((defconfig, False, 'has #include')) - continue + raw = tools.read_file(orig) + has_include = b'#include' in raw - kconf.load_config(orig) - - tmp = tempfile.NamedTemporaryFile( - mode='w', prefix='qconfig-', suffix='_defconfig', - dir=os.path.join(srcdir, 'configs'), delete=False) - tmp.close() - kconf.write_min_config(tmp.name) - - updated = not filecmp.cmp(orig, tmp.name) - if updated and not dry_run: - shutil.move(tmp.name, orig) + if has_include: + updated = _sync_include_defconfig(kconf, srcdir, orig, + dry_run) else: - os.unlink(tmp.name) + updated = _sync_plain_defconfig(kconf, orig, dry_run) result_queue.put((defconfig, updated, None)) except Exception as exc: error_queue.put((defconfig, str(exc))) @@ -1965,8 +2094,90 @@ def move_done(progress): col.GREEN, f'{progress.total} processed ', bright=True)) return 0 +class SyncTests(unittest.TestCase): + """Tests for defconfig sync using kconfiglib""" + + @classmethod + def setUpClass(cls): + """Create a shared Kconfig instance for all tests""" + os.environ['srctree'] = os.getcwd() + os.environ['UBOOTVERSION'] = 'dummy' + os.environ['KCONFIG_OBJDIR'] = '' + os.environ['CC'] = 'gcc' + cls.kconf = kconfiglib.Kconfig(warn=False) + cls.srcdir = os.getcwd() + + def test_sync_plain_noop(self): + """Syncing an already-minimal defconfig produces no change""" + # sandbox_defconfig should already be synced if the tree is clean + orig = 'configs/sandbox_defconfig' + updated = _sync_plain_defconfig(self.kconf, orig, dry_run=True) + # This may or may not be updated depending on tree state, but + # it should not crash + self.assertIsInstance(updated, bool) + + def test_sync_include_preserves_structure(self): + """Syncing a #include defconfig preserves the #include lines""" + orig = 'configs/sandbox_nocmdline_defconfig' + if not os.path.exists(orig): + self.skipTest(f'{orig} not found') + + # Dry-run should not modify the file + content_before = tools.read_file(orig) + updated = _sync_include_defconfig(self.kconf, self.srcdir, orig, + dry_run=True) + content_after = tools.read_file(orig) + self.assertEqual(content_before, content_after) + + # The output should still start with #include + self.assertIn(b'#include', content_after) + + def test_sync_include_removes_redundant(self): + """Syncing a #include defconfig removes CONFIGs from the base""" + # Create a temp defconfig that includes sandbox and redundantly + # sets a CONFIG that sandbox already sets + with tempfile.NamedTemporaryFile( + mode='w', prefix='test-', suffix='_defconfig', + dir='configs', delete=False) as tmp: + tmp.write('#include "sandbox_defconfig"\n') + tmp.write('CONFIG_CMDLINE=y\n') + tmp_name = tmp.name + try: + updated = _sync_include_defconfig(self.kconf, self.srcdir, + tmp_name, dry_run=False) + self.assertTrue(updated) + with open(tmp_name) as inf: + result = inf.read() + # CONFIG_CMDLINE=y should be gone (it's in the base) + self.assertNotIn('CONFIG_CMDLINE=y', result) + # #include should still be there + self.assertIn('#include "sandbox_defconfig"', result) + finally: + os.unlink(tmp_name) + + def test_sync_include_keeps_override(self): + """Syncing a #include defconfig keeps CONFIGs that differ from base""" + # Create a temp defconfig that includes sandbox and disables CMDLINE + with tempfile.NamedTemporaryFile( + mode='w', prefix='test-', suffix='_defconfig', + dir='configs', delete=False) as tmp: + tmp.write('#include "sandbox_defconfig"\n') + tmp.write('# CONFIG_CMDLINE is not set\n') + tmp_name = tmp.name + try: + _sync_include_defconfig(self.kconf, self.srcdir, tmp_name, + dry_run=False) + with open(tmp_name) as inf: + result = inf.read() + # Disabling CMDLINE is an override — should be kept + self.assertIn('# CONFIG_CMDLINE is not set', result) + self.assertIn('#include "sandbox_defconfig"', result) + finally: + os.unlink(tmp_name) + + def do_tests(): - """Run doctests and unit tests (so far there are no unit tests)""" + """Run doctests and unit tests""" sys.argv = [sys.argv[0]] fail, _ = doctest.testmod() if fail: From patchwork Sun Mar 29 11:10:31 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 2069 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782715; bh=B7auIZB9W7myj6jwN8kMUlLY+mKxv3DTfUwqQMMKB+0=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=eLpP/uNBOW/pMinlZKjGCJgjxHwei/B8MsstNZnIhX5vPA5UDZHrP+Jd9AbkebrQW 8KxsBmvVENe84lQ9Erirt7lK9azZ3Kx7rtR5eGEpue9xAz+nUOP8aJLc1uke+lpcAy hG+dnoBlxqxGkIkHts21jV3vhNpUuQtgKJoXX0qln/PfDIrefE/BZP4ZxLZHh7lGll hzNnbu2YClEhH6ws6GfGQ3uQ2CPBMwy185b5kcMqD3tJZ8iChKLZhc/YvTK9AL6Bdw WVXvpOcC7YOhJfBLFa+xcS/8IEO3gLav8Lm+crga1CXwvvZVyMDMZLlyba0FMw4Ggp bVjYqBod58R9Q== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 26FD26A2D6 for ; Sun, 29 Mar 2026 05:11:55 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id WW6EuSmqbqqJ for ; Sun, 29 Mar 2026 05:11:55 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782713; bh=B7auIZB9W7myj6jwN8kMUlLY+mKxv3DTfUwqQMMKB+0=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=KbCiNztrOA03MiXEK3oMe4V3ddKZj0T0bF2otK/YomOjgPLE6iBiR7LAy7Z4Iz797 eN64i6jkT1VlnHjeDWzpzxLWnM7IgShMtcKao5Ixf63DqbuLwdHfp5wfQ/dk6Fpzg/ VlpzUzPQ1inPKSEcDCavf0yv4vNkYBRpGOfD5mzsIpjm7XjM4SzcRaq0Mvl9drIJd9 pnqbHIu4Ebg5Tb/O23MfPOkF+M9bKiHZXVTsejM7BgAHu83W59lMwYVnAGX8Xt0hIn w8TC4meV8DF59S5KzvDzMT0chMA3P3ptS/5PVtRSo6O6076N8K/6ak+dBAV44Sk06W S2hfGwF71UJUQ== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 740056A2C5 for ; Sun, 29 Mar 2026 05:11:53 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782709; bh=CZvYQkPYzLBCVIrJa3IfUSqqsgWWIDL+MlW9TlYL4HU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bEFKo9l4Rm3HNRWefB52W2hsjRrIpEjo0+H6NaY4MrzWzmCYztCOrPaGvgyxBK71j 5zGkmd/JzlKwfMF0R5IijXrSWwe3ONiqWnRYb70wLHjnpOcDaTgHvDLGXMVi9ctXuP WIV4swskiiAOuKV0VKSua5VN50hy/FHGZRqXvQv/btDYKzdT+tjnvRzz8UDPYbUIDi r5V2fPA0XJc2HbtfP8PTeLW5yJzx1+FsaLNTwsBPbsJBTEtnSJLAgfMgkg91Zb6Dc5 Acz8g40oyOpIhM30aQ+ImZskXD/qh7Gc5frnUhk4e7sOkS4+uOnxPQMFdOJM9x9FXm Oh3EOVWVl/S6A== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 659326A2BE; Sun, 29 Mar 2026 05:11:49 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id fycsByCjA_VC; Sun, 29 Mar 2026 05:11:49 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782708; bh=4vHZ8apzn15MYEKYcYqxS/XRSdoHJP/MmdznnCRwWlU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=O/C4uqFpffFTYsYrRxTJhiTBXW2B0y8BK7OKkd+f34I8n68rhwK/U5eAt1paq0p0a 3ge9qE46gxHB3h5p96w/5FBJWJ6Vl9f2Vo0pEsLJxVVR0RxNJINh8RIepfICWeclyx 8dsPJdfHcuvbI70iw7RrL0KxcAob0JL9HpQqusSeiMPgc0YuuJH0KKIp21aOCIaqHx BI0l2VEr7ndr1EsBE+/VlDeZB1mG871Q0ErUoORb42rAtHRWKJ0d55fi7f27oorJyD C+VV9AXO0ghKafdFDTRo6MwxK50NW0PQBzkMjZCJlg00VUH7ByCUbNUWkeQ4/8NyO1 IwgjIqXOGRrZw== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id B93916A2BC; Sun, 29 Mar 2026 05:11:48 -0600 (MDT) From: Simon Glass To: U-Boot Concept Date: Sun, 29 Mar 2026 05:10:31 -0600 Message-ID: <20260329111037.1352652-7-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260329111037.1352652-1-sjg@u-boot.org> References: <20260329111037.1352652-1-sjg@u-boot.org> MIME-Version: 1.0 Message-ID-Hash: C7VDPIJ3HBTMJ7XE46GZ3KGHIMVIXQ36 X-Message-ID-Hash: C7VDPIJ3HBTMJ7XE46GZ3KGHIMVIXQ36 X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Simon Glass X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 6/7] qconfig: Remove dead database-build code from move_config() List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass Now that -b uses do_build_db() and -s uses do_sync_defconfigs(), move_config() is only reached for -s -r (sync with git-ref). The build_db branch in Slot.poll(), do_add_to_db(), DatabaseThread, and the db_queue plumbing are all dead code in that path. Remove them, along with the now-unused queue and threading imports. Signed-off-by: Simon Glass --- tools/qconfig.py | 73 ++++++++---------------------------------------- 1 file changed, 12 insertions(+), 61 deletions(-) diff --git a/tools/qconfig.py b/tools/qconfig.py index c0215978319..443223cf9f4 100755 --- a/tools/qconfig.py +++ b/tools/qconfig.py @@ -18,13 +18,11 @@ import fnmatch import glob import multiprocessing import os -import queue import re import shutil import subprocess import sys import tempfile -import threading import time import unittest @@ -714,32 +712,6 @@ class KconfigParser: return arch -class DatabaseThread(threading.Thread): - """This thread processes results from Slot threads. - - It collects the data in the master config directary. There is only one - result thread, and this helps to serialise the build output. - """ - def __init__(self, config_db, db_queue): - """Set up a new result thread - - Args: - builder: Builder which will be sent each result - """ - threading.Thread.__init__(self) - self.config_db = config_db - self.db_queue= db_queue - - def run(self): - """Called to start up the result thread. - - We collect the next result job and pass it on to the build. - """ - while True: - defconfig, configs = self.db_queue.get() - self.config_db[defconfig] = configs - self.db_queue.task_done() - class Slot: @@ -751,19 +723,18 @@ class Slot: """ def __init__(self, toolchains, args, progress, devnull, make_cmd, - reference_src_dir, db_queue): + reference_src_dir): """Create a new process slot. Args: toolchains: Toolchains object containing toolchains. - args: Program arguments; this class uses build_db, verbose, + args: Program arguments; this class uses verbose, force_sync, dry_run, exit_on_error progress: A progress indicator. devnull: A file object of '/dev/null'. make_cmd: command name of GNU Make. reference_src_dir: Determine the true starting config state from this source tree. - db_queue: output queue to write config info for the database col (terminal.Color): Colour object """ self.toolchains = toolchains @@ -773,7 +744,6 @@ class Slot: self.devnull = devnull self.make_cmd = (make_cmd, 'O=' + self.build_dir) self.reference_src_dir = reference_src_dir - self.db_queue = db_queue self.col = progress.col self.parser = KconfigParser(self.build_dir) self.state = STATE_IDLE @@ -853,8 +823,6 @@ class Slot: if self.current_src_dir: self.current_src_dir = None self.do_defconfig() - elif self.args.build_db: - self.do_add_to_db() else: self.do_savedefconfig() elif self.state == STATE_SAVEDEFCONFIG: @@ -908,16 +876,6 @@ class Slot: cwd=self.current_src_dir) self.state = STATE_AUTOCONF - def do_add_to_db(self): - """Add the board to the database""" - configs = {} - for line in read_file(os.path.join(self.build_dir, AUTO_CONF_PATH)): - if line.startswith('CONFIG'): - config, value = line.split('=', 1) - configs[config] = value.rstrip() - self.db_queue.put([self.defconfig, configs]) - self.finish(True) - def do_savedefconfig(self): """Update the .config and run 'make savedefconfig'.""" if not self.args.force_sync: @@ -995,17 +953,16 @@ class Slot: class Slots: """Controller of the array of subprocess slots.""" - def __init__(self, toolchains, args, progress, reference_src_dir, db_queue): + def __init__(self, toolchains, args, progress, reference_src_dir): """Create a new slots controller. Args: toolchains (Toolchains): Toolchains object containing toolchains - args (Namespace): Program arguments; this class uses build_db, + args (Namespace): Program arguments; this class uses verbose, force_sync, dry_run, exit_on_error, jobs, progress (Progress): A progress indicator. reference_src_dir (str): Determine the true starting config state from this source tree (None for none) - db_queue (Queue): output queue to write config info for the database """ self.args = args self.slots = [] @@ -1015,7 +972,7 @@ class Slots: make_cmd = get_make_cmd() for _ in range(args.jobs): self.slots.append(Slot(toolchains, args, progress, devnull, - make_cmd, reference_src_dir, db_queue)) + make_cmd, reference_src_dir)) def add(self, defconfig): """Add a new subprocess if a vacant slot is found. @@ -1102,27 +1059,22 @@ class ReferenceSource: return self.src_dir def move_config(args): - """Build database or sync config options to defconfig files. + """Sync config options to defconfig files using make (legacy path). + + This is only used with -r (git-ref), which needs to build against a + different source tree. The normal -s path uses do_sync_defconfigs(). Args: - args (Namespace): Program arguments; this class uses build_db, + args (Namespace): Program arguments; this class uses verbose, force_sync, dry_run, exit_on_error, jobs, git_ref, defconfigs, defconfiglist, nocolour Returns: tuple: - config_db (dict of configs for each defconfig): - key: defconfig name, e.g. "MPC8548CDS_legacy_defconfig" - value: dict: - key: CONFIG option - value: Value of option + config_db (dict): Always empty (database build uses do_build_db()) Progress: Progress indicator """ config_db = {} - db_queue = queue.Queue() - dbt = DatabaseThread(config_db, db_queue) - dbt.daemon = True - dbt.start() check_clean_directory() bsettings.setup('') @@ -1148,7 +1100,7 @@ def move_config(args): col = terminal.Color(terminal.COLOR_NEVER if args.nocolour else terminal.COLOR_IF_TERMINAL) progress = Progress(col, len(defconfigs)) - slots = Slots(toolchains, args, progress, reference_src_dir, db_queue) + slots = Slots(toolchains, args, progress, reference_src_dir) # Main loop to process defconfig files: # Add a new subprocess into a vacant slot. @@ -1164,7 +1116,6 @@ def move_config(args): time.sleep(SLEEP_TIME) slots.write_failed_boards() - db_queue.join() progress.completed() return config_db, progress From patchwork Sun Mar 29 11:10:32 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 2070 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782720; bh=oT5hnmry/3AtNW2DkP4h60Yns7/eTH/AsWw2rBx76V4=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=VpxdUP7/zcvHM/mWy6awvTQ5su8FUmsa5Zc1RR//sdzP00WDp9uTbHXpBQ+TrVGMb 831O4gDw32BGVxRj8RdtZbYJK6K2oLnZvJ8ML80hOvpozz43AKRVfSH0L/Q388z+oS pBK8PZLCNRvID9vYIFem6wMmUNdL1X2MzQqUyGocul+mI0zIlv1Cv1A3RHEnhgkREi CbC7r78yIQIPJzy/vOUXhsxOKvUY0gNbjCP/cOCKv5yDbj5PVGOBkKblVa6FIYYmNg K1qOKF44oRKSBYw0QLgNdGb/BPD2T0zYvKmy3GsRonPpOg2GAZZAaoz9QSBZxmXAfC dTDt+y4xKk3/g== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 4AEEE6A2BF for ; Sun, 29 Mar 2026 05:12:00 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id e_LiZQkjDpYe for ; Sun, 29 Mar 2026 05:12:00 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782720; bh=oT5hnmry/3AtNW2DkP4h60Yns7/eTH/AsWw2rBx76V4=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=VpxdUP7/zcvHM/mWy6awvTQ5su8FUmsa5Zc1RR//sdzP00WDp9uTbHXpBQ+TrVGMb 831O4gDw32BGVxRj8RdtZbYJK6K2oLnZvJ8ML80hOvpozz43AKRVfSH0L/Q388z+oS pBK8PZLCNRvID9vYIFem6wMmUNdL1X2MzQqUyGocul+mI0zIlv1Cv1A3RHEnhgkREi CbC7r78yIQIPJzy/vOUXhsxOKvUY0gNbjCP/cOCKv5yDbj5PVGOBkKblVa6FIYYmNg K1qOKF44oRKSBYw0QLgNdGb/BPD2T0zYvKmy3GsRonPpOg2GAZZAaoz9QSBZxmXAfC dTDt+y4xKk3/g== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 317676A2BC for ; Sun, 29 Mar 2026 05:12:00 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782717; bh=3g7KgEzToW0hDhUZSKEZMfTAdAuwmgB82QxAdZgbDto=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eZ7T7r6RlKYCZbOPyDt2J6Vc6JJ8eksxk4E5f/2Ui9UK+r9rjCyeL9oCbmKE/FMxo 3SlKDJAB8FBVzd1JYBjOSLgp6exWei603OKtfZwndmUTrqKaydXHi731zKPxPKArMT xSgiJNZYa2QqOSmdJ1FxOkvzVbpbJjn4/jzDoK8mc4BlUNcBxnblN38o/XPD5rXWy/ e7qHrVGp9hOnBpRdjJCX02Lp2y2qs1Lueqbwpok9OMqB4aVxUnQ+hbovd+uL7EkPwX AQIX2g5S9Lg2Ys9vIbWbJJVj4jRWKiIyL5saAA30RhVFoejOvhLiQXKoixwAQC+ZPo qqEI/hZO1STOA== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id B627867C16; Sun, 29 Mar 2026 05:11:57 -0600 (MDT) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id VUm19F10R1P9; Sun, 29 Mar 2026 05:11:57 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1774782709; bh=P1osZmFNuC16Q7//FF8BVhX2UqQQdMV/jwyBnWG3ZLs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CThWyU2rSZw6xuQ7yxz/NPB+vSSyoH894zHWPd8d37qEowLh0OKoW0ETeWI+0UXUk 1gJsBFNehLkDrm1GSJ3RmFT5gSJLsseMF712kJ8JEyZhxYLlE/MqaCXkEIc8XL+eY/ ++W4IDykD6sC4FsVNqUgxBqzaTEaeVzWwI0GAfVD9i7Yjwt0wjEcA52tficzkd2Y1a N5s2OEimlM4+qVfZ/DJqi4wtPOhvLeFV2d/hlW5sWWnm1YoGDWDS31VMHLpOoq7Dxr nSpCRUOeu4ahXgDc1BM+JOdz4/WO9qjry7aY5cNv1HYEDMpj/PLbX5gaLqL1jMo0qi caQSB9idIhN7w== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id A48FC6A2BC; Sun, 29 Mar 2026 05:11:49 -0600 (MDT) From: Simon Glass To: U-Boot Concept Date: Sun, 29 Mar 2026 05:10:32 -0600 Message-ID: <20260329111037.1352652-8-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260329111037.1352652-1-sjg@u-boot.org> References: <20260329111037.1352652-1-sjg@u-boot.org> MIME-Version: 1.0 Message-ID-Hash: Y7CXTU5OYIRLJR3CLQ6IXNCF7T2RACOE X-Message-ID-Hash: Y7CXTU5OYIRLJR3CLQ6IXNCF7T2RACOE X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Simon Glass X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 7/7] qconfig: Use kconfiglib for -r (git-ref) sync path List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass The -r option still uses the old make-based move_config() path, which requires cross-compiler toolchains and is much slower. Handle -r in do_sync_defconfigs() instead: clone the reference commit, create a kconfiglib.Kconfig instance from that tree, load each defconfig against it, write a full .config, then load that into the current tree's Kconfig and write_min_config(). This gives the same result without any toolchain requirement. With move_config() now unreachable, remove it along with all the infrastructure it depended on: KconfigParser, Slot, Slots, DatabaseThread, get_make_cmd(), check_clean_directory(), and the STATE_* constants. Signed-off-by: Simon Glass --- doc/develop/qconfig.rst | 24 +- tools/qconfig.py | 526 +++++----------------------------------- 2 files changed, 72 insertions(+), 478 deletions(-) diff --git a/doc/develop/qconfig.rst b/doc/develop/qconfig.rst index 4a85c78cbbb..7fdbad8923d 100644 --- a/doc/develop/qconfig.rst +++ b/doc/develop/qconfig.rst @@ -51,25 +51,21 @@ that use ``#include`` directives are handled by computing the delta between the full expanded config and the base provided by the included files, so the include structure is preserved. -The ``-r`` (git-ref) option still uses the old make-based path, since it -needs to build against a different source tree. +The ``-r`` (git-ref) option also uses kconfiglib: it clones the reference +commit, creates a Kconfig instance from that tree, loads each defconfig +against it, then normalises the result against the current tree's Kconfig. +No toolchains are needed. Toolchains ---------- -Toolchains are **not** needed for ``-b`` or ``-s``, since both use -kconfiglib to evaluate Kconfig files directly in Python. The only +Toolchains are **not** needed for any qconfig operation. The only difference from using a real toolchain is that ``CONFIG_GCC_VERSION`` reflects the host compiler rather than each board's cross-compiler. This does not affect database queries, imply analysis, or defconfig sync, since ``CONFIG_GCC_VERSION`` is a build-time value that never appears in defconfig files or influences Kconfig defaults. -The ``-r`` (git-ref) option still requires toolchains, as it falls back -to the make-based path. Most toolchains are available at the kernel.org -site. This tool uses the same tools as :doc:`../build/buildman`, so you -can use ``buildman --fetch-arch`` to fetch them. - Examples -------- @@ -256,12 +252,12 @@ Available options the number of threads is the same as the number of CPU cores. -r, --git-ref - Specify the git ref to clone for building the autoconf.mk. If unspecified - use the CWD. This is useful for when changes to the Kconfig affect the - default values and you want to capture the state of the defconfig from + Specify the git ref to clone for evaluating the Kconfig tree. If + unspecified, use the CWD. This is useful when changes to Kconfig affect + default values and you want to capture the state of defconfigs from before that change was in effect. If in doubt, specify a ref pre-Kconfig - changes (use HEAD if Kconfig changes are not committed). Worst case it will - take a bit longer to run, but will always do the right thing. + changes (use HEAD if Kconfig changes are not committed). Worst case it + will take a bit longer to run, but will always do the right thing. -v, --verbose Show any build errors as boards are built diff --git a/tools/qconfig.py b/tools/qconfig.py index 443223cf9f4..86b2bed9283 100755 --- a/tools/qconfig.py +++ b/tools/qconfig.py @@ -26,22 +26,13 @@ import tempfile import time import unittest -from buildman import bsettings from buildman import kconfiglib -from buildman import toolchain from u_boot_pylib import terminal from u_boot_pylib.terminal import tprint from u_boot_pylib import tools -SHOW_GNU_MAKE = 'scripts/show-gnu-make' SLEEP_TIME=0.03 -STATE_IDLE = 0 -STATE_DEFCONFIG = 1 -STATE_AUTOCONF = 2 -STATE_SAVEDEFCONFIG = 3 - -AUTO_CONF_PATH = 'include/config/auto.conf' CONFIG_DATABASE = 'qconfig.db' FAILED_LIST = 'qconfig.failed' @@ -88,25 +79,6 @@ def check_top_directory(): if not os.path.exists(fname): sys.exit('Please run at the top of source directory.') -def check_clean_directory(): - """Exit if the source tree is not clean.""" - for fname in '.config', 'include/config': - if os.path.exists(fname): - sys.exit("source tree is not clean, please run 'make mrproper'") - -def get_make_cmd(): - """Get the command name of GNU Make. - - U-Boot needs GNU Make for building, but the command name is not - necessarily "make". (for example, "gmake" on FreeBSD). - Returns the most appropriate command name on your system. - """ - with subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE) as proc: - ret = proc.communicate() - if proc.returncode: - sys.exit('GNU Make not found') - return ret[0].rstrip() - def get_matched_defconfig(line): """Get the defconfig files that match a pattern @@ -544,12 +516,16 @@ def _sync_include_defconfig(kconf, srcdir, orig, dry_run): def _sync_defconfigs_worker(srcdir, defconfigs, result_queue, error_queue, - dry_run): + dry_run, ref_srcdir=None): """Worker process that syncs defconfigs using kconfiglib For each defconfig, loads it via kconfiglib and writes a minimal config (equivalent to 'make savedefconfig'), then compares with the original. + When ref_srcdir is set (the -r option), loads each defconfig against the + reference Kconfig tree first, writes a full .config, then loads that + .config into the current tree's Kconfig and writes a minimal config. + Args: srcdir (str): Source-tree directory defconfigs (list of str): Defconfig filenames to process @@ -557,24 +533,57 @@ def _sync_defconfigs_worker(srcdir, defconfigs, result_queue, error_queue, (defconfig, updated) tuples error_queue (multiprocessing.Queue): Output queue for failed defconfigs dry_run (bool): If True, do not update defconfig files + ref_srcdir (str or None): Reference source tree for -r option """ - os.environ['srctree'] = srcdir os.environ['UBOOTVERSION'] = 'dummy' os.environ['KCONFIG_OBJDIR'] = '' os.environ['CC'] = 'gcc' + + os.environ['srctree'] = srcdir kconf = kconfiglib.Kconfig(warn=False) + if ref_srcdir: + os.environ['srctree'] = ref_srcdir + ref_kconf = kconfiglib.Kconfig(warn=False) + os.environ['srctree'] = srcdir + for defconfig in defconfigs: orig = os.path.join(srcdir, 'configs', defconfig) try: - raw = tools.read_file(orig) - has_include = b'#include' in raw - - if has_include: - updated = _sync_include_defconfig(kconf, srcdir, orig, - dry_run) + if ref_srcdir: + # Load defconfig against the reference Kconfig tree, write + # a full .config, then load it into the current tree + ref_orig = os.path.join(ref_srcdir, 'configs', defconfig) + if not os.path.exists(ref_orig): + ref_orig = orig + _load_defconfig(ref_kconf, ref_srcdir, ref_orig) + tmp_config = tempfile.NamedTemporaryFile( + prefix='qconfig-cfg-', delete=False) + tmp_config.close() + ref_kconf.write_config(tmp_config.name) + kconf.load_config(tmp_config.name, replace=True) + os.unlink(tmp_config.name) else: - updated = _sync_plain_defconfig(kconf, orig, dry_run) + raw = tools.read_file(orig) + if b'#include' in raw: + updated = _sync_include_defconfig(kconf, srcdir, orig, + dry_run) + result_queue.put((defconfig, updated, None)) + continue + _load_defconfig(kconf, srcdir, orig) + + confdir = os.path.dirname(orig) + tmp = tempfile.NamedTemporaryFile( + mode='w', prefix='qconfig-', suffix='_defconfig', + dir=confdir, delete=False) + tmp.close() + kconf.write_min_config(tmp.name) + + updated = not filecmp.cmp(orig, tmp.name) + if updated and not dry_run: + shutil.move(tmp.name, orig) + else: + os.unlink(tmp.name) result_queue.put((defconfig, updated, None)) except Exception as exc: error_queue.put((defconfig, str(exc))) @@ -586,15 +595,25 @@ def do_sync_defconfigs(args): Evaluates each defconfig through kconfiglib and writes a minimal config (equivalent to 'make savedefconfig'), updating the original if it differs. + When -r (git-ref) is given, loads each defconfig against the Kconfig tree + from the reference commit first, then normalises it against the current + tree, capturing any Kconfig default changes. + Args: args (Namespace): Program arguments (uses jobs, defconfigs, - defconfiglist, nocolour, dry_run, force_sync) + defconfiglist, nocolour, dry_run, force_sync, git_ref) Returns: Progress: progress indicator """ srcdir = os.getcwd() + if args.git_ref: + reference_src = ReferenceSource(args.git_ref) + ref_srcdir = reference_src.get_dir() + else: + ref_srcdir = None + if args.defconfigs: defconfigs = [os.path.basename(d) for d in get_matched_defconfigs(args.defconfigs)] @@ -619,7 +638,8 @@ def do_sync_defconfigs(args): continue proc = multiprocessing.Process( target=_sync_defconfigs_worker, - args=(srcdir, chunk, result_queue, error_queue, args.dry_run)) + args=(srcdir, chunk, result_queue, error_queue, args.dry_run, + ref_srcdir)) proc.start() processes.append(proc) @@ -665,364 +685,6 @@ def do_sync_defconfigs(args): return progress -# pylint: disable=R0903 -class KconfigParser: - """A parser of .config and include/autoconf.mk.""" - - re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"') - re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"') - - def __init__(self, build_dir): - """Create a new parser. - - Args: - build_dir: Build directory. - """ - self.dotconfig = os.path.join(build_dir, '.config') - self.autoconf = os.path.join(build_dir, 'include', 'autoconf.mk') - self.spl_autoconf = os.path.join(build_dir, 'spl', 'include', - 'autoconf.mk') - self.config_autoconf = os.path.join(build_dir, AUTO_CONF_PATH) - self.defconfig = os.path.join(build_dir, 'defconfig') - - def get_arch(self): - """Parse .config file and return the architecture. - - Returns: - Architecture name (e.g. 'arm'). - """ - arch = '' - cpu = '' - for line in read_file(self.dotconfig): - m_arch = self.re_arch.match(line) - if m_arch: - arch = m_arch.group(1) - continue - m_cpu = self.re_cpu.match(line) - if m_cpu: - cpu = m_cpu.group(1) - - if not arch: - return None - - # fix-up for aarch64 - if arch == 'arm' and cpu == 'armv8': - arch = 'aarch64' - - return arch - - - -class Slot: - - """A slot to store a subprocess. - - Each instance of this class handles one subprocess. - This class is useful to control multiple threads - for faster processing. - """ - - def __init__(self, toolchains, args, progress, devnull, make_cmd, - reference_src_dir): - """Create a new process slot. - - Args: - toolchains: Toolchains object containing toolchains. - args: Program arguments; this class uses verbose, - force_sync, dry_run, exit_on_error - progress: A progress indicator. - devnull: A file object of '/dev/null'. - make_cmd: command name of GNU Make. - reference_src_dir: Determine the true starting config state from this - source tree. - col (terminal.Color): Colour object - """ - self.toolchains = toolchains - self.args = args - self.progress = progress - self.build_dir = tempfile.mkdtemp() - self.devnull = devnull - self.make_cmd = (make_cmd, 'O=' + self.build_dir) - self.reference_src_dir = reference_src_dir - self.col = progress.col - self.parser = KconfigParser(self.build_dir) - self.state = STATE_IDLE - self.failed_boards = set() - self.defconfig = None - self.log = [] - self.current_src_dir = None - self.proc = None - - def __del__(self): - """Delete the working directory - - This function makes sure the temporary directory is cleaned away - even if Python suddenly dies due to error. It should be done in here - because it is guaranteed the destructor is always invoked when the - instance of the class gets unreferenced. - - If the subprocess is still running, wait until it finishes. - """ - if self.state != STATE_IDLE: - while self.proc.poll() is None: - pass - shutil.rmtree(self.build_dir) - - def add(self, defconfig): - """Assign a new subprocess for defconfig and add it to the slot. - - If the slot is vacant, create a new subprocess for processing the - given defconfig and add it to the slot. Just returns False if - the slot is occupied (i.e. the current subprocess is still running). - - Args: - defconfig (str): defconfig name. - - Returns: - Return True on success or False on failure - """ - if self.state != STATE_IDLE: - return False - - self.defconfig = defconfig - self.log = [] - self.current_src_dir = self.reference_src_dir - self.do_defconfig() - return True - - def poll(self): - """Check the status of the subprocess and handle it as needed. - - Returns True if the slot is vacant (i.e. in idle state). - If the configuration is successfully finished, assign a new - subprocess to build include/autoconf.mk. - If include/autoconf.mk is generated, invoke the parser to - parse the .config and the include/autoconf.mk, moving - config options to the .config as needed. - If the .config was updated, run "make savedefconfig" to sync - it, update the original defconfig, and then set the slot back - to the idle state. - - Returns: - Return True if the subprocess is terminated, False otherwise - """ - if self.state == STATE_IDLE: - return True - - if self.proc.poll() is None: - return False - - if self.proc.poll() != 0: - self.handle_error() - elif self.state == STATE_DEFCONFIG: - if self.reference_src_dir and not self.current_src_dir: - self.do_savedefconfig() - else: - self.do_autoconf() - elif self.state == STATE_AUTOCONF: - if self.current_src_dir: - self.current_src_dir = None - self.do_defconfig() - else: - self.do_savedefconfig() - elif self.state == STATE_SAVEDEFCONFIG: - self.update_defconfig() - else: - sys.exit('Internal Error. This should not happen.') - - return self.state == STATE_IDLE - - def handle_error(self): - """Handle error cases.""" - - self.log.append(self.col.build(self.col.RED, 'Failed to process', - bright=True)) - if self.args.verbose: - for line in self.proc.stderr.read().decode().splitlines(): - self.log.append(self.col.build(self.col.CYAN, line, True)) - self.finish(False) - - def do_defconfig(self): - """Run 'make _defconfig' to create the .config file.""" - - cmd = list(self.make_cmd) - cmd.append(self.defconfig) - # pylint: disable=R1732 - self.proc = subprocess.Popen(cmd, stdout=self.devnull, - stderr=subprocess.PIPE, - cwd=self.current_src_dir) - self.state = STATE_DEFCONFIG - - def do_autoconf(self): - """Run 'make AUTO_CONF_PATH'.""" - - arch = self.parser.get_arch() - try: - tchain = self.toolchains.select(arch) - except ValueError: - self.log.append(self.col.build( - self.col.YELLOW, - f"Tool chain for '{arch}' is missing: do nothing")) - self.finish(False) - return - env = tchain.make_environment(False) - - cmd = list(self.make_cmd) - cmd.append('KCONFIG_IGNORE_DUPLICATES=1') - cmd.append(AUTO_CONF_PATH) - # pylint: disable=R1732 - self.proc = subprocess.Popen(cmd, stdout=self.devnull, env=env, - stderr=subprocess.PIPE, - cwd=self.current_src_dir) - self.state = STATE_AUTOCONF - - def do_savedefconfig(self): - """Update the .config and run 'make savedefconfig'.""" - if not self.args.force_sync: - self.finish(True) - return - - cmd = list(self.make_cmd) - cmd.append('savedefconfig') - # pylint: disable=R1732 - self.proc = subprocess.Popen(cmd, stdout=self.devnull, - stderr=subprocess.PIPE) - self.state = STATE_SAVEDEFCONFIG - - def update_defconfig(self): - """Update the input defconfig and go back to the idle state.""" - orig_defconfig = os.path.join('configs', self.defconfig) - new_defconfig = os.path.join(self.build_dir, 'defconfig') - updated = not filecmp.cmp(orig_defconfig, new_defconfig) - success = True - - if updated: - # Files with #include get mangled as savedefconfig doesn't know how to - # deal with them. Ignore them - success = b'#include' not in tools.read_file(orig_defconfig) - if success: - self.log.append( - self.col.build(self.col.BLUE, 'defconfig updated', - bright=True)) - else: - self.log.append( - self.col.build(self.col.RED, 'ignored due to #include', - bright=True)) - updated = False - - if not self.args.dry_run and updated: - shutil.move(new_defconfig, orig_defconfig) - self.finish(success) - - def finish(self, success): - """Display log along with progress and go to the idle state. - - Args: - success (bool): Should be True when the defconfig was processed - successfully, or False when it fails. - """ - # output at least 30 characters to hide the "* defconfigs out of *". - name = self.defconfig[:-len('_defconfig')] - if self.log: - - # Put the first log line on the first line - log = name.ljust(20) + ' ' + self.log[0] - - if len(self.log) > 1: - log += '\n' + '\n'.join([' ' + s for s in self.log[1:]]) - # Some threads are running in parallel. - # Print log atomically to not mix up logs from different threads. - print(log, file=(sys.stdout if success else sys.stderr)) - - if not success: - if self.args.exit_on_error: - sys.exit('Exit on error.') - # If --exit-on-error flag is not set, skip this board and continue. - # Record the failed board. - self.failed_boards.add(name) - - self.progress.inc(success) - self.progress.show() - self.state = STATE_IDLE - - def get_failed_boards(self): - """Returns a set of failed boards (defconfigs) in this slot. - """ - return self.failed_boards - -class Slots: - """Controller of the array of subprocess slots.""" - - def __init__(self, toolchains, args, progress, reference_src_dir): - """Create a new slots controller. - - Args: - toolchains (Toolchains): Toolchains object containing toolchains - args (Namespace): Program arguments; this class uses - verbose, force_sync, dry_run, exit_on_error, jobs, - progress (Progress): A progress indicator. - reference_src_dir (str): Determine the true starting config state - from this source tree (None for none) - """ - self.args = args - self.slots = [] - self.progress = progress - self.col = progress.col - devnull = subprocess.DEVNULL - make_cmd = get_make_cmd() - for _ in range(args.jobs): - self.slots.append(Slot(toolchains, args, progress, devnull, - make_cmd, reference_src_dir)) - - def add(self, defconfig): - """Add a new subprocess if a vacant slot is found. - - Args: - defconfig (str): defconfig name to be put into. - - Returns: - Return True on success or False on failure - """ - for slot in self.slots: - if slot.add(defconfig): - return True - return False - - def available(self): - """Check if there is a vacant slot. - - Returns: - Return True if at lease one vacant slot is found, False otherwise. - """ - for slot in self.slots: - if slot.poll(): - return True - return False - - def empty(self): - """Check if all slots are vacant. - - Returns: - Return True if all the slots are vacant, False otherwise. - """ - ret = True - for slot in self.slots: - if not slot.poll(): - ret = False - return ret - - def write_failed_boards(self): - """Show the results of processing""" - boards = set() - - for slot in self.slots: - boards |= slot.get_failed_boards() - - if boards: - boards = '\n'.join(sorted(boards)) + '\n' - write_file(FAILED_LIST, boards) - - class ReferenceSource: """Reference source against which original configs should be parsed.""" @@ -1058,67 +720,6 @@ class ReferenceSource: return self.src_dir -def move_config(args): - """Sync config options to defconfig files using make (legacy path). - - This is only used with -r (git-ref), which needs to build against a - different source tree. The normal -s path uses do_sync_defconfigs(). - - Args: - args (Namespace): Program arguments; this class uses - verbose, force_sync, dry_run, exit_on_error, jobs, git_ref, - defconfigs, defconfiglist, nocolour - - Returns: - tuple: - config_db (dict): Always empty (database build uses do_build_db()) - Progress: Progress indicator - """ - config_db = {} - - check_clean_directory() - bsettings.setup('') - - # Get toolchains to use - toolchains = toolchain.Toolchains() - toolchains.get_settings() - toolchains.scan(verbose=False) - - if args.git_ref: - reference_src = ReferenceSource(args.git_ref) - reference_src_dir = reference_src.get_dir() - else: - reference_src_dir = None - - if args.defconfigs: - defconfigs = get_matched_defconfigs(args.defconfigs) - elif args.defconfiglist: - defconfigs = get_matched_defconfigs(args.defconfiglist) - else: - defconfigs = get_all_defconfigs() - - col = terminal.Color(terminal.COLOR_NEVER if args.nocolour - else terminal.COLOR_IF_TERMINAL) - progress = Progress(col, len(defconfigs)) - slots = Slots(toolchains, args, progress, reference_src_dir) - - # Main loop to process defconfig files: - # Add a new subprocess into a vacant slot. - # Sleep if there is no available slot. - for defconfig in defconfigs: - while not slots.add(defconfig): - while not slots.available(): - # No available slot: sleep for a while - time.sleep(SLEEP_TIME) - - # wait until all the subprocesses finish - while not slots.empty(): - time.sleep(SLEEP_TIME) - - slots.write_failed_boards() - progress.completed() - return config_db, progress - def find_kconfig_rules(kconf, config, imply_config): """Check whether a config has a 'select' or 'imply' keyword @@ -2222,17 +1823,14 @@ def main(): config_db, progress = do_build_db(args) return write_db(config_db, progress) - if args.force_sync and not args.git_ref: + if args.force_sync: progress = do_sync_defconfigs(args) if args.commit: add_commit(args.configs) return move_done(progress) - config_db, progress = move_config(args) - - if args.commit: - add_commit(args.configs) - return move_done(progress) + parser.print_usage() + return 1 if __name__ == '__main__':