From patchwork Mon Nov 24 13:49:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 759 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992203; bh=Wx7WVrOnDtRT17L+DFcI4h5AjVNC9XOtpRqD9PFi9K0=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=FR4Ye/LUIcro8Vw166UJa3aEOcc6Z1Xzw7JSuAB6um64pyJiwWIFJE3x5i7xtC75i o9MtMGPy8lr6wrd0QK9lM3OuvQGZRJfawDO1ePqiXholXS60RsNjtBu/rurgzyxGbC Eya+d/NI3fB1/8u7ZG36m1t8SMlqCmUNA7/CpD1eQCBq720hKkWADIbOU6upeazYFT XQeF/QM30Zpy92pD3mJR/UdoY4JB+N8mHM36tzuUInnEhLv/qah46dk+reIMhfbTFY pKHrKxJaB4BTdpZVW2TbQvpfIz1u2gWW+xXWPfB+odPfQ8wM6U2maFEsx+LdFd+Q0I h7++IaKI1hs/w== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 5300F68688 for ; Mon, 24 Nov 2025 06:50:03 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id J5YT-9RCIJSQ for ; Mon, 24 Nov 2025 06:50:03 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992203; bh=Wx7WVrOnDtRT17L+DFcI4h5AjVNC9XOtpRqD9PFi9K0=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=FR4Ye/LUIcro8Vw166UJa3aEOcc6Z1Xzw7JSuAB6um64pyJiwWIFJE3x5i7xtC75i o9MtMGPy8lr6wrd0QK9lM3OuvQGZRJfawDO1ePqiXholXS60RsNjtBu/rurgzyxGbC Eya+d/NI3fB1/8u7ZG36m1t8SMlqCmUNA7/CpD1eQCBq720hKkWADIbOU6upeazYFT XQeF/QM30Zpy92pD3mJR/UdoY4JB+N8mHM36tzuUInnEhLv/qah46dk+reIMhfbTFY pKHrKxJaB4BTdpZVW2TbQvpfIz1u2gWW+xXWPfB+odPfQ8wM6U2maFEsx+LdFd+Q0I h7++IaKI1hs/w== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 3EDDB6874E for ; Mon, 24 Nov 2025 06:50:03 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992201; bh=m7aZN7yPNh6gg4KH6ZUZyYOa5FT6spxlMaaoAOOyxM4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CbXP0DvMBKCfrIQX4IWZLKaQealibsjA2vJuz/0Ua2CUrid/KEMOSkZIOfNXEI7uK mNxSuIXtYaIvVbsSPY9gaByr6gsOHbtzvY3HnM6+kPm/RYDaFLvPAs/NhqfQbX6rxX K12m3pK/SAWukTBIhL4hoj6RNWo2JX9MYm1Nbq1g8u6F84j2JgazqdNfI5RVcTmq9N +YuwBB3zJH9W9tkYsKaHT2Y7K+SSOM24XlxI0CeQJlhpN6vwtsjmqIHCIa9GDCevEo 2ZxPISt6+aSZ4fqsqWiwy+YCnbL7TmnIBiwWx28pGnNhpubMpluFSg4oTSwH/EQxk7 Mdrfo5PtPtlCg== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 281F368688; Mon, 24 Nov 2025 06:50:01 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id D1eCazIwyY7J; Mon, 24 Nov 2025 06:50:01 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992197; bh=HfQiD2XgN5G5iwJ05XcVjsPGHKS5wxkPFk64xfZN2ek=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YOHObe3JhEcwXPCuRUMcYch/0jnBtTghd0CaqqH5W8p277C4943Aul6R3NSyflSqQ couaApli8HksLfQKzipO66QYu8LTgX5lS6caarXYNCPUNDSn0JMBNLL0nXvrorqyLD ynWfNMl9NmJXousqUrz1LZExZE2cMLFOe2dBOGcmDC4RAelSwktN8mS+rCpBm0TEkv zTt5jlM59+3LPKwF5h7V+b9ceOkH7RHXlNGIhTTeelpb1XD+ClqTOhDsHxqGEE2tWP BJghOwa/JzWI99GcbMesN/wCj4SLkJBdKC9BS5mRkz/HvybQZoLZ7rxjEVVuHCUEtB zZmdA6klY0mBA== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id 044B86876A; Mon, 24 Nov 2025 06:49:56 -0700 (MST) From: Simon Glass To: U-Boot Concept Date: Mon, 24 Nov 2025 06:49:12 -0700 Message-ID: <20251124134932.1991031-2-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251124134932.1991031-1-sjg@u-boot.org> References: <20251124134932.1991031-1-sjg@u-boot.org> MIME-Version: 1.0 Message-ID-Hash: SBK4W4UH7AKORW4NPQ6NAK4XAHZBK2XA X-Message-ID-Hash: SBK4W4UH7AKORW4NPQ6NAK4XAHZBK2XA X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Simon Glass , Claude X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 1/9] u_boot_pylib: Add stderr parameter to tprint() List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass Add optional stderr parameter to tprint() to allow printing to stderr instead of stdout. Co-developed-by: Claude Signed-off-by: Simon Glass --- tools/u_boot_pylib/terminal.py | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/tools/u_boot_pylib/terminal.py b/tools/u_boot_pylib/terminal.py index 69c183e85e5..e62fa166dca 100644 --- a/tools/u_boot_pylib/terminal.py +++ b/tools/u_boot_pylib/terminal.py @@ -141,7 +141,7 @@ def trim_ascii_len(text, size): def tprint(text='', newline=True, colour=None, limit_to_line=False, - bright=True, back=None, col=None): + bright=True, back=None, col=None, stderr=False): """Handle a line of output to the terminal. In test mode this is recorded in a list. Otherwise it is output to the @@ -151,6 +151,7 @@ def tprint(text='', newline=True, colour=None, limit_to_line=False, text: Text to print newline: True to add a new line at the end of the text colour: Colour to use for the text + stderr: True to print to stderr instead of stdout """ global last_print_len @@ -161,14 +162,17 @@ def tprint(text='', newline=True, colour=None, limit_to_line=False, if not col: col = Color() text = col.build(colour, text, bright=bright, back=back) + + file = sys.stderr if stderr else sys.stdout + if newline: - print(text) + print(text, file=file) last_print_len = None else: if limit_to_line: cols = shutil.get_terminal_size().columns text = trim_ascii_len(text, cols) - print(text, end='', flush=True) + print(text, end='', flush=True, file=file) last_print_len = calc_ascii_len(text) def print_clear(): From patchwork Mon Nov 24 13:49:13 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 760 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992205; bh=31vJyaYIJ5U98sWceV7hIQi5vt4qj4qx1e9V7kjtEBI=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=nghwyEy2oHHXX7ONJYdHiXbuCBqcsQdxAMcAVxe5VSaMjDbXvqoQwClUhO5bZ4jP7 8FE+uzRq+1/zvXzoN7lrM4DlYAZCk5szH7MgaFrJSo5vydZAeMt995gZSnTjorRQNK aX2NLvgQ8UPcNtRWDAMBIdL9pUGW577NurTGqNh54FeQxPH8iSMEIC+u4ItMSv0rdo LsrT3reUqnNAMKq3smj7ijtu7Exa/dNoJHjqZpt37/NAUARt9roJGj6YSiJqdeF0cR Tiz4sG1uTNvsEgo5pq9drciWzxO1rxcZg1BK9P1IeKdRrQQLV59RlyY/byZqFXLsoA g2Lxe8uTbTWVQ== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id C4E686874E for ; Mon, 24 Nov 2025 06:50:05 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 3LngKyGw4u4I for ; Mon, 24 Nov 2025 06:50:05 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992203; bh=31vJyaYIJ5U98sWceV7hIQi5vt4qj4qx1e9V7kjtEBI=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=dKLsLbj7RNKsBHvU6z877IxIw/3jCQ/9i6g8AEkukbhXyUl/HYHzNOBbz5c+tIZjB vHpWOF7PARe+riKKy8S17F3wi293+nenwKLSEqf/NpwKkJwfAtatuAmIAElrsjwXWX 3f8XIMoF1o1xsy7CkSXCXEyFsM6edQNdIQtpNw1KMrIwAehIJcPqFew1Hqo5MWXX9g Y2viMQoKTHGS0630zGN0la0w3L6KdZS2vsMwO83fMNyqyfsJkgSnBP5aTY9xzZ44sW MlUGVCdyz/Cd0pkdecRHB8ZxDxmWJvxXE/ndvLmd4qEO9RYxK0/MYJRGv23j7BriPt HoR8wH4zJQoKw== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id BAC0C6873C for ; Mon, 24 Nov 2025 06:50:03 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992202; bh=hhBPzIXBXZOkyDbRTmJD+2hKK+2ZJ8MPgYPEqnk+TcE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=p0yeUKQUn8rITIk9k2o5oi1MQMCGdqocPAGOYqtSZapsR45e4HRgXpdTwDAoQiXTZ v1Btkt1q5zpCMxCBxIpou1svc7xtd/F0CIK95snc50z7vr1BE5eJ22LyJG/LGhpCLp MFJzF+g2l+rL92mdDAsOD32k6lvoLUCrGWRvcBknz8jaNUX+dSmKvUkK9c2MWtqVmD 963x0itoeu5MkoIB099Rh7zGD7QwmFjAdctpIknUoq7fpg742V+ds2b9dALS8mZGDN hlbxNeT3mlllr4u05Xb4mGo/gEHn5EwiavMf2UOPnwW4xgL9ofAAjSaPz6uI0atdUY GETGFtSand+9g== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 170F168688; Mon, 24 Nov 2025 06:50:02 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id DK_2vagWTUd6; Mon, 24 Nov 2025 06:50:02 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992201; bh=peOFa5QTsHhL2EG5awPVqdd7k9qgQHGiEsHAOG3ASwU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CFMuT6ZGxZBijCd6cXbFiSb0HW2w3r6Q/5IUlW4DTzfboxPpVen4I4FBDOnB63Cgt tve7j5Hv+eSyk7Grn1/1KCGpEyye0Uw8iZqTNu8W0NDs86iaAsHKZUMQ5h8Ucm3Azg wTSzAjKCP53mIn+71+M8AeTyOBCqefYuCKwNIYf05Uv7Ffbexog5OMbeW2ZOem00t/ gKG19xscTnjPou5/BSaIqEnVjRpWzktS2/9nryB9kxk9gArj9Wbh884Nc4LuORSyE1 xx2+88cZ2VnfjhP1ekukA8XRJRPMPPXNTCgDQgnZDKRmTZrFjtOf5t1zwgphZrDcRo v7ndjlMNDhOjw== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id 6BE8A6873F; Mon, 24 Nov 2025 06:50:01 -0700 (MST) From: Simon Glass To: U-Boot Concept Date: Mon, 24 Nov 2025 06:49:13 -0700 Message-ID: <20251124134932.1991031-3-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251124134932.1991031-1-sjg@u-boot.org> References: <20251124134932.1991031-1-sjg@u-boot.org> MIME-Version: 1.0 Message-ID-Hash: HLFZYMWKJYJBKHPAPS33IYPDXJCIGERL X-Message-ID-Hash: HLFZYMWKJYJBKHPAPS33IYPDXJCIGERL X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Simon Glass , Claude X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 2/9] u_boot_pylib: Use terminal.tprint() for output in tout List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass Refactor tout.py to use terminal.tprint() instead of direct print() calls. This provides better control over output formatting and supports the new stderr parameter. It also reduces code duplication. Co-developed-by: Claude Signed-off-by: Simon Glass --- tools/u_boot_pylib/tout.py | 24 +++++++----------------- 1 file changed, 7 insertions(+), 17 deletions(-) diff --git a/tools/u_boot_pylib/tout.py b/tools/u_boot_pylib/tout.py index ca72108d6bc..137b55edfd0 100644 --- a/tools/u_boot_pylib/tout.py +++ b/tools/u_boot_pylib/tout.py @@ -11,8 +11,6 @@ from u_boot_pylib import terminal # Output verbosity levels that we support FATAL, ERROR, WARNING, NOTICE, INFO, DETAIL, DEBUG = range(7) -in_progress = False - """ This class handles output of progress and other useful information to the user. It provides for simple verbosity level control and can @@ -46,11 +44,8 @@ def user_is_present(): def clear_progress(): """Clear any active progress message on the terminal.""" - global in_progress - if verbose > ERROR and stdout_is_tty and in_progress: - _stdout.write('\r%s\r' % (" " * len (_progress))) - _stdout.flush() - in_progress = False + if verbose > ERROR and stdout_is_tty: + terminal.print_clear() def progress(msg, warning=False, trailer='...'): """Display progress information. @@ -58,17 +53,14 @@ def progress(msg, warning=False, trailer='...'): Args: msg: Message to display. warning: True if this is a warning.""" - global in_progress clear_progress() if verbose > ERROR: _progress = msg + trailer if stdout_is_tty: col = _color.YELLOW if warning else _color.GREEN - _stdout.write('\r' + _color.build(col, _progress)) - _stdout.flush() - in_progress = True + terminal.tprint('\r' + _progress, newline=False, colour=col, col=_color) else: - _stdout.write(_progress + '\n') + terminal.tprint(_progress) def _output(level, msg, color=None): """Output a message to the terminal. @@ -81,12 +73,10 @@ def _output(level, msg, color=None): """ if verbose >= level: clear_progress() - if color: - msg = _color.build(color, msg) - if level < NOTICE: - print(msg, file=sys.stderr) + if level <= WARNING: + terminal.tprint(msg, colour=color, col=_color, stderr=True) else: - print(msg) + terminal.tprint(msg, colour=color, col=_color) if level == FATAL: sys.exit(1) From patchwork Mon Nov 24 13:49:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 761 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992208; bh=rAaNOBlHvST7IKizeMytsdHscqLGmOEjgctZMXkmznw=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=Rc/hdjNx976tPYXEGDUzjBsmjUOwXD1K+/GpNBxSssW3O+xSiFrnt++5l/iHTnVWn 8j8QvtSTQ2A55iqMMJKBbYFrHwHbw7bJbov6FC4oDerpNa1D9SKmrPjK3DRHEmJpEt DDn/6CL5HiFKwfLnnlBSefc94ZrFsyd2EA2vzgj4PAJm78VLKYkIV/3vOOzZkf21s7 2eqPJB0jc14UlummEKhiJI+bH2jvaflQ4AOH/Vi2hblnVitytKFIWBwak3Pst6Ldxk 9yBeIExLZlxV+Qhq/fxX5R+Ux9BC7Gc0EBWNJmXFjkLF7JQtoiuWzVs19S5mGtTV81 H0nwNArkUOT/w== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 4F54868765 for ; Mon, 24 Nov 2025 06:50:08 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id iw5GpA5OfbM7 for ; Mon, 24 Nov 2025 06:50:08 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992208; bh=rAaNOBlHvST7IKizeMytsdHscqLGmOEjgctZMXkmznw=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=Rc/hdjNx976tPYXEGDUzjBsmjUOwXD1K+/GpNBxSssW3O+xSiFrnt++5l/iHTnVWn 8j8QvtSTQ2A55iqMMJKBbYFrHwHbw7bJbov6FC4oDerpNa1D9SKmrPjK3DRHEmJpEt DDn/6CL5HiFKwfLnnlBSefc94ZrFsyd2EA2vzgj4PAJm78VLKYkIV/3vOOzZkf21s7 2eqPJB0jc14UlummEKhiJI+bH2jvaflQ4AOH/Vi2hblnVitytKFIWBwak3Pst6Ldxk 9yBeIExLZlxV+Qhq/fxX5R+Ux9BC7Gc0EBWNJmXFjkLF7JQtoiuWzVs19S5mGtTV81 H0nwNArkUOT/w== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 3C91468669 for ; Mon, 24 Nov 2025 06:50:08 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992207; bh=FQnka1A5HXssjmAQoi+FT1XkUAaxex4kBSFFBfuGFz8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GyVPZDdoLYioCAjse2wcx1AFEIAhgNj8ggwis/JaJI0qR2yRLKYvtCfaj9hpd55Vl uF1F19vnPKnEl9rdo2wKqhC8zB0zS7ZNgSvyAbsBUR0kTX0YxjACBcDoVORH41Ky+S pJsu+VhCkRB1TRINyKh9O2DzSKrUOHkm7qCd8pKfY0steV0SBzqPpDNbGJoVEQ3nwK 3HibXwnU6taLCEf0kZAnsafBeLhwd77FqBXUmMAvb672oi4TnZvhF43IyBBaXLQ5/I wD3LH5C0/P47/58WBrjnCk3ytILWi0WdBdeXZojzOILRXoXrHPHBNgRtgT+JB5VnCO SPjtsuINavSRA== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 199F86873C; Mon, 24 Nov 2025 06:50:07 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id 0MkYBbngI0hS; Mon, 24 Nov 2025 06:50:07 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992202; bh=YnN8wK4HSu2zVUpu6Du/23ltzvEBd2JOvyob2tML0JQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bJ0p34nHdPZGD7YUkwXfBP6r43EKY27+p9A2iHjSOoiJBGZ8exwQUAolePDJwzHoW pLf6Xkaha3bwAD0CtAXzgxQPqnDEQ7o0kUicmRBpHUQsj9iFnRTLdnOwLrYxBWGCHk ezXIhT0Zll7+HmPy6e0G4e9Wc9YDmgUwc3DD5Tn5YBTBk4DycwOvbXoIGnwlkY0uCl uw1miN2koBR9J86kozwB3fciJED3IifTsHc8Pn+kOyUmQr8Pi1qVrdoMTf53+k0c5e 4URG+YFfQWtLbr16VhaWVBYuWI8FWj8MwnvwmX8iCA9DvvVmp63+PPezrhfiAwGOul kYh1zaVXbhP4g== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id 58FEC6873F; Mon, 24 Nov 2025 06:50:02 -0700 (MST) From: Simon Glass To: U-Boot Concept Date: Mon, 24 Nov 2025 06:49:14 -0700 Message-ID: <20251124134932.1991031-4-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251124134932.1991031-1-sjg@u-boot.org> References: <20251124134932.1991031-1-sjg@u-boot.org> MIME-Version: 1.0 Message-ID-Hash: SLLV4YNOSLPVUODOLSV53ZCYETQX2MCC X-Message-ID-Hash: SLLV4YNOSLPVUODOLSV53ZCYETQX2MCC X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Simon Glass , Claude X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 3/9] buildman: Support comma-separated values in -a flag List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass Allow users to specify multiple config adjustments in a single -a argument using commas. This is more convenient than repeating -a multiple times. Examples: buildman -a FOO,~BAR buildman -a FOO,~BAR -a BAZ=123 Add tests to verify comma-separated values work correctly. Co-developed-by: Claude Signed-off-by: Simon Glass --- tools/buildman/buildman.rst | 13 +++++++++++++ tools/buildman/cfgutil.py | 19 ++++++++++++------- tools/buildman/cmdline.py | 3 ++- tools/buildman/test.py | 12 ++++++++++++ 4 files changed, 39 insertions(+), 8 deletions(-) diff --git a/tools/buildman/buildman.rst b/tools/buildman/buildman.rst index 487e9d67a4b..c0599757b0b 100644 --- a/tools/buildman/buildman.rst +++ b/tools/buildman/buildman.rst @@ -1307,6 +1307,19 @@ You can disable options by preceding them with tilde (~). You can specify the buildman -a CMD_SETEXPR_FMT -a ~CMDLINE +You can also use comma-separated values to specify multiple options in a single +argument: + +.. code-block:: bash + + buildman -a CMD_SETEXPR_FMT,~CMDLINE + +or mix both styles: + +.. code-block:: bash + + buildman -a CMD_SETEXPR_FMT,~CMDLINE -a BOOTSTD_FULL + Some options have values, in which case you can change them: .. code-block:: bash diff --git a/tools/buildman/cfgutil.py b/tools/buildman/cfgutil.py index a340e01cb6b..5bc97d33595 100644 --- a/tools/buildman/cfgutil.py +++ b/tools/buildman/cfgutil.py @@ -134,7 +134,7 @@ def convert_list_to_dict(adjust_cfg_list): Args: adjust_cfg_list (list of str): List of changes to make to .config file before building. Each is one of (where C is the config option with - or without the CONFIG_ prefix) + or without the CONFIG_ prefix). Items can be comma-separated. C to enable C ~C to disable C @@ -154,12 +154,17 @@ def convert_list_to_dict(adjust_cfg_list): ValueError: if an item in adjust_cfg_list has invalid syntax """ result = {} - for cfg in adjust_cfg_list or []: - m_cfg = RE_CFG.match(cfg) - if not m_cfg: - raise ValueError(f"Invalid CONFIG adjustment '{cfg}'") - negate, _, opt, val = m_cfg.groups() - result[opt] = f'%s{opt}%s' % (negate or '', val or '') + for item in adjust_cfg_list or []: + # Split by comma to support comma-separated values + for cfg in item.split(','): + cfg = cfg.strip() + if not cfg: + continue + m_cfg = RE_CFG.match(cfg) + if not m_cfg: + raise ValueError(f"Invalid CONFIG adjustment '{cfg}'") + negate, _, opt, val = m_cfg.groups() + result[opt] = f'%s{opt}%s' % (negate or '', val or '') return result diff --git a/tools/buildman/cmdline.py b/tools/buildman/cmdline.py index ad07e6cac39..b3c70daeca3 100644 --- a/tools/buildman/cmdline.py +++ b/tools/buildman/cmdline.py @@ -24,7 +24,8 @@ def add_upto_m(parser): """ # Available JqzZ parser.add_argument('-a', '--adjust-cfg', type=str, action='append', - help='Adjust the Kconfig settings in .config before building') + help='Adjust the Kconfig settings in .config before building. ' + + 'Supports comma-separated values') parser.add_argument('-A', '--print-prefix', action='store_true', help='Print the tool-chain prefix for a board (CROSS_COMPILE=)') parser.add_argument('-b', '--branch', type=str, diff --git a/tools/buildman/test.py b/tools/buildman/test.py index a134ac4f917..81e708d9bd6 100644 --- a/tools/buildman/test.py +++ b/tools/buildman/test.py @@ -780,6 +780,18 @@ class TestBuild(unittest.TestCase): 'CONFIG_ANNA="anna"']) self.assertEqual(expect, actual) + # Test comma-separated values + actual = cfgutil.convert_list_to_dict( + ['FRED,~MARY,JOHN=0x123', 'ALICE="alice"', + 'CONFIG_AMY,~CONFIG_ABE', 'CONFIG_MARK=0x456,CONFIG_ANNA="anna"']) + self.assertEqual(expect, actual) + + # Test mixed comma-separated and individual values + actual = cfgutil.convert_list_to_dict( + ['FRED,~MARY', 'JOHN=0x123', 'ALICE="alice",CONFIG_AMY', + '~CONFIG_ABE,CONFIG_MARK=0x456', 'CONFIG_ANNA="anna"']) + self.assertEqual(expect, actual) + def test_check_cfg_file(self): """Test check_cfg_file detects conflicts as expected""" # Check failure to disable CONFIG From patchwork Mon Nov 24 13:49:15 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 976 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1765999488; bh=9pDRlGoagOlKmkWGqlsPdUZzFN/UGwj49RgsTA62YFw=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=ou2AiDXAHw5LAnAsVun5zSz8T2G5DHjyrysOGzWFgk+TJDdpK01SEo6ClgmF1TZIS WNjHQGpoydjXTgbs3UM2/AnCkBUpLSg8LPZ+dACViCRAiPWa31ehHA0pnkrSYh0eB4 vtmkC4oFPI7F6B6s22gttdoBnD0UaRMCEXtgXys4atimAZK+DWvcrfkq5nYWsIbC0k SwYpmvepwSXCLpkF2no9B0npy1rBK8COdL6PVi6JOjSnecK+jinNL1x84xJHRSGnkf iyFeihswnD4sXWuIKWPV3UznSFY4bNH6PuufAAptrCUH0Cayj7q+lJ5WrL7K64ni6u NYna1xi+b5lBA== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id E853E68C17 for ; Wed, 17 Dec 2025 12:24:48 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id O9OTtvmTmK8h for ; Wed, 17 Dec 2025 12:24:48 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1765999488; bh=9pDRlGoagOlKmkWGqlsPdUZzFN/UGwj49RgsTA62YFw=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=ou2AiDXAHw5LAnAsVun5zSz8T2G5DHjyrysOGzWFgk+TJDdpK01SEo6ClgmF1TZIS WNjHQGpoydjXTgbs3UM2/AnCkBUpLSg8LPZ+dACViCRAiPWa31ehHA0pnkrSYh0eB4 vtmkC4oFPI7F6B6s22gttdoBnD0UaRMCEXtgXys4atimAZK+DWvcrfkq5nYWsIbC0k SwYpmvepwSXCLpkF2no9B0npy1rBK8COdL6PVi6JOjSnecK+jinNL1x84xJHRSGnkf iyFeihswnD4sXWuIKWPV3UznSFY4bNH6PuufAAptrCUH0Cayj7q+lJ5WrL7K64ni6u NYna1xi+b5lBA== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id D18B068C09 for ; Wed, 17 Dec 2025 12:24:48 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992211; bh=gExYPt+TjuLWmA5AUL0K+t7GdXYaC6GzXtVKN5dg/MA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=P7wwKHKt3r8zgvsYKuN4YSsvar57SlOSsFUnNqTNLsQyOMh3Gag4UyqmA/VQHj+aw Z1D1sEvnqti+3z05Wwx36DKNK2CG0o0bnOP+64ovpIF0ZoGYH7RNNwgzwiBjJcRvN3 RptIGK2FCGs45bw66H4XpQziPXyGldyEChaoq4lLiUHsxoMyOJhZ/oQH5zh/PsSIbx 2McaI6ybkZBSdNo8Zge+xX5n4qOt9biMxQL9IUzO9AmEJdTvITZZtn44HFLusuk/H4 7qHU0f+URbusakI4IuboImswT7RVv5hUlNA/ju8H/kQhOgZCQohWoPRfbJSdh6wbqL kZCMcGXwPV97g== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id D7E3068669; Mon, 24 Nov 2025 06:50:11 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id 4miVS8Ce7tXx; Mon, 24 Nov 2025 06:50:11 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992207; bh=jizjuVq+7rqx9LMhshNGnpPjOtssEz9Q5yzPTJlC62Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=N1CTIP4dQb2VuMhcbrecdVywpiO7ovt3mERu9dI6EpLd+Up4nwwnRe7gKaNBRTFmF R4bbNSbdUkp9nWcUpkPn7BjgWxG8GFk6iq/7rROeyYeLD1UjBZs3x8Ag57wsJHtths dxxKXumLRA8z9A/l7TbIj0OEQbmHQ6h1ZwmtMAncoDj759sC6x0Hws959KV+qVU4yd uiFvaPJRLpzxt6S9QyWCuh2df458wwnSsiQ08p+O/p5yPQjvUPV06+XANUpnX8jC2x mHqAGPuwl2iK3ppvQEyj4oV+5pdDKjeRa0hpjoyTx4C4twbHzlCVeKH0JxZYL3Gxt1 i5KHwawDbmW+w== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id 14AEB68688; Mon, 24 Nov 2025 06:50:07 -0700 (MST) From: Simon Glass To: U-Boot Concept Date: Mon, 24 Nov 2025 06:49:15 -0700 Message-ID: <20251124134932.1991031-5-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251124134932.1991031-1-sjg@u-boot.org> References: <20251124134932.1991031-1-sjg@u-boot.org> MIME-Version: 1.0 X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Hits: max-size X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; news-moderation; no-subject; digests; suspicious-header Message-ID-Hash: XQ4GDJNPN4C5Z3DRTLMSV55DYWGTS5BQ X-Message-ID-Hash: XQ4GDJNPN4C5Z3DRTLMSV55DYWGTS5BQ X-Mailman-Approved-At: Wed, 17 Dec 2025 19:24:47 -0700 CC: Simon Glass , Claude X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 4/9] codman: Add a new source-code analysis tool List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass Add a new tool called 'codman' (code manager) for analysing source code usage in U-Boot builds. This tool determines which files and lines of code are actually compiled based on the build configuration. Co-developed-by: Claude Signed-off-by: Simon Glass --- tools/codman/analyser.py | 76 +++++ tools/codman/codman | 1 + tools/codman/codman.py | 664 +++++++++++++++++++++++++++++++++++++++ tools/codman/output.py | 536 +++++++++++++++++++++++++++++++ 4 files changed, 1277 insertions(+) create mode 100644 tools/codman/analyser.py create mode 120000 tools/codman/codman create mode 100755 tools/codman/codman.py create mode 100644 tools/codman/output.py diff --git a/tools/codman/analyser.py b/tools/codman/analyser.py new file mode 100644 index 00000000000..2c0cc8b8855 --- /dev/null +++ b/tools/codman/analyser.py @@ -0,0 +1,76 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright 2025 Canonical Ltd +# +"""Base classes for source code analysis. + +This module provides base classes and data structures for analyzing which lines +in source files are active vs inactive. +""" + +import os +from collections import namedtuple + +# Named tuple for file analysis results +# Fields: +# total_lines: Total number of lines in the file +# active_lines: Number of lines that are active (not removed by +# preprocessor) +# inactive_lines: Number of lines that are inactive (removed by +# preprocessor) +# line_status: Dict mapping line numbers to status ('active', +# 'inactive', etc.) +FileResult = namedtuple('FileResult', + ['total_lines', 'active_lines', + 'inactive_lines', 'line_status']) + + +class Analyser: # pylint: disable=too-few-public-methods + """Base class for source code analysers. + + This class provides common initialisation for analysers that determine + which lines in source files are active vs inactive based on various + methods (preprocessor analysis, debug info, etc.). + """ + + def __init__(self, srcdir, keep_temps=False): + """Set up the analyser. + + Args: + srcdir (str): Path to source root directory + keep_temps (bool): If True, keep temporary files for debugging + """ + self.srcdir = srcdir + self.keep_temps = keep_temps + + def find_object_files(self, build_dir): + """Find all object files in the build directory. + + Args: + build_dir (str): Build directory to search + + Returns: + list: List of absolute paths to .o files + """ + obj_files = [] + for root, _, files in os.walk(build_dir): + for fname in files: + if fname.endswith('.o'): + obj_files.append(os.path.join(root, fname)) + return obj_files + + @staticmethod + def count_lines(file_path): + """Count the number of lines in a file. + + Args: + file_path (str): Path to file to count lines in + + Returns: + int: Number of lines in the file, or 0 on error + """ + try: + with open(file_path, 'r', encoding='utf-8', errors='ignore') as f: + return len(f.readlines()) + except IOError: + return 0 diff --git a/tools/codman/codman b/tools/codman/codman new file mode 120000 index 00000000000..e7e14ca1165 --- /dev/null +++ b/tools/codman/codman @@ -0,0 +1 @@ +codman.py \ No newline at end of file diff --git a/tools/codman/codman.py b/tools/codman/codman.py new file mode 100755 index 00000000000..dbd72b066c1 --- /dev/null +++ b/tools/codman/codman.py @@ -0,0 +1,664 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright 2025 Canonical Ltd +# +"""Analyse C source code usage in U-Boot builds. + +This script performs file-level and line-level analysis of U-Boot source code: +- File level: which files are compiled vs not compiled +- Line level: which lines within compiled files are active based on CONFIG_* + +This combines file-level analysis (which files are used) with optional +line-level analysis: which parts of each file are active based on the +preprocessor and Kconfig options. +""" + +import argparse +import fnmatch +import multiprocessing +import os +import re +import subprocess +import sys + +# Allow 'from patman import xxx to work' +# pylint: disable=C0413 +our_path = os.path.dirname(os.path.realpath(__file__)) +sys.path.append(os.path.join(our_path, '..')) + +# pylint: disable=wrong-import-position +from u_boot_pylib import terminal, tools, tout + +# Import analysis modules +import dwarf +import lsp +import output +import unifdef +# pylint: enable=wrong-import-position + +# Pattern to match .cmd files +RE_PATTERN = re.compile(r'^\..*\.cmd$') + +# Pattern to extract the source file from a .cmd file +RE_LINE = re.compile(r'^(saved)?cmd_[^ ]*\.o := (?P.* )' + r'(?P[^ ]*\.[cS]) *(;|$)') +RE_SOURCE = re.compile(r'^source_[^ ]*\.o := (?P[^ ]*\.[cS])') + +# Directories to exclude from analysis +EXCLUDE_DIRS = ['.git', 'Documentation', 'doc', 'scripts', 'tools'] + +# Default base directory for builds +BUILD_BASE = '/tmp/b' + + +def cmdfiles_in_dir(directory): + """Generate paths to all .cmd files under the directory""" + for dirpath, dirnames, filenames in os.walk(directory, topdown=True): + dirnames = [d for d in dirnames if d not in EXCLUDE_DIRS] + + for filename in filenames: + if RE_PATTERN.match(filename): + yield os.path.join(dirpath, filename) + + +def extract_source_from_cmdfile(cmdfile_path, srcdir): + """Extract the source file path from a .cmd file. + + Args: + cmdfile_path (str): Path to the .cmd file to parse. + srcdir (str): Root directory of the U-Boot source tree. + """ + with open(cmdfile_path, 'rt', encoding='utf-8') as f: + for line in f: + result = RE_SOURCE.match(line) + if result: + file_path = result.group('file_path') + abs_path = os.path.realpath(os.path.join(srcdir, file_path)) + if os.path.exists(abs_path): + return abs_path + + result = RE_LINE.match(line) + if result: + file_path = result.group('file_path') + abs_path = os.path.realpath(os.path.join(srcdir, file_path)) + if os.path.exists(abs_path): + return abs_path + + return None + + +def find_all_source_files(srcdir): + """Find all C/assembly/header source files in the source tree. + + Args: + srcdir (str): Root directory of the U-Boot source tree. + + Returns: + Set of absolute paths to all source files. + """ + tout.progress('Finding all source files...') + all_sources = set() + exclude_dirs = [os.path.join(srcdir, d) for d in EXCLUDE_DIRS] + + for dirpath, dirnames, filenames in os.walk(srcdir, topdown=True): + # Skip excluded directories + if any(dirpath.startswith(excl) for excl in exclude_dirs): + dirnames[:] = [] + continue + + for filename in filenames: + if filename.endswith(('.c', '.S', '.h')): + abs_path = os.path.realpath(os.path.join(dirpath, filename)) + all_sources.add(abs_path) + + tout.info(f'Found {len(all_sources)} total source files') + + return all_sources + + +def extract_deps_from_cmdfile(cmdfile_path): + """Extract all source file dependencies from a .cmd file. + + This includes the main source file and all headers it depends on. + + Args: + cmdfile_path (str): Path to the .cmd file to parse. + + Returns: + Set of absolute paths to source files (c/S/h) used. + """ + deps = set() + + with open(cmdfile_path, 'rt', encoding='utf-8') as f: + in_deps_section = False + for line in f: + # Look for deps_* := lines + if line.startswith('deps_'): + in_deps_section = True + continue + + # If we're in the deps section, extract file paths + if in_deps_section: + # Lines look like: /path/to/file.h \ + # or: $(wildcard include/config/foo.h) \ + if line.strip() == '': + in_deps_section = False + continue + + # Skip wildcard lines + if '$(wildcard' in line: + continue + + # Extract the file path + path = line.strip().rstrip('\\').strip() + if path and os.path.exists(path): + abs_path = os.path.realpath(path) + # Only include .c, .S, .h files + if abs_path.endswith(('.c', '.S', '.h')): + deps.add(abs_path) + + return deps + + +def resolve_wrapper_file(source_file): + """Check if a file is a wrapper that only includes another .c file. + + For example lib/libfdt/fdt_overlay.c which holds: + #include + #include "../../scripts/dtc/libfdt/fdt_overlay.c" + + Args: + source_file (str): Path to the source file + + Returns: + str: Path to the included .c file if this is a wrapper, else the + original file + """ + lines = tools.read_file(source_file, binary=False).splitlines() + + # Check if file only has #include directives (and comments/blank lines) + included_c_file = None + has_other_content = False + + for line in lines: + stripped = line.strip() + # Skip blank lines and comments + if not stripped or stripped.startswith('//') or \ + stripped.startswith('/*') or stripped.startswith('*'): + continue + + # Check for #include directive + if stripped.startswith('#include'): + # Extract the included file + match = re.search(r'#include\s+[<"]([^>"]+)[>"]', stripped) + if match: + included = match.group(1) + # Only track .c file includes (the actual source) + if included.endswith('.c'): + included_c_file = included + continue + + # Found non-include content + has_other_content = True + break + + # If we only found includes and one was a .c file, resolve it + if not has_other_content and included_c_file: + # Resolve relative to the wrapper file's directory + wrapper_dir = os.path.dirname(source_file) + resolved = os.path.realpath( + os.path.join(wrapper_dir, included_c_file)) + if os.path.exists(resolved): + return resolved + + return source_file + + +def _process_cmdfile(args): + """Process a single .cmd file to extract source files. + + This is a worker function for multiprocessing. + + Args: + args: Tuple of (cmdfile_path, srcdir, srcdir_real) + + Returns: + set: Set of absolute paths to source files found in this .cmd file + """ + cmdfile, srcdir, srcdir_real = args + sources = set() + + # Get the main source file (.c or .S) + source_file = extract_source_from_cmdfile(cmdfile, srcdir) + if source_file: + # Resolve wrapper files to their actual source + resolved = resolve_wrapper_file(source_file) + # Only include files within the source tree + if os.path.realpath(resolved).startswith(srcdir_real): + sources.add(resolved) + + # Get all dependencies (headers) + deps = extract_deps_from_cmdfile(cmdfile) + # Filter to only include files within the source tree + for dep in deps: + if os.path.realpath(dep).startswith(srcdir_real): + sources.add(dep) + + return sources + + +def find_used_sources(build_dir, srcdir, jobs=None): + """Find all source files used in the build. + + This includes both the compiled .c/.S files and all .h headers they depend + on. For wrapper files that only include another .c file, the included file + is returned instead. + + Only files within the source tree are included - system headers and + toolchain files are excluded. + + Args: + build_dir (str): Path to the build directory containing .cmd files + srcdir (str): Path to U-Boot source root directory + jobs (int): Number of parallel jobs (None = use all CPUs) + + Returns: + set: Set of absolute paths to all source files used in the build + """ + tout.progress('Finding used source files...') + srcdir_real = os.path.realpath(srcdir) + + # Collect all cmdfiles first + cmdfiles = list(cmdfiles_in_dir(build_dir)) + tout.progress(f'Processing {len(cmdfiles)} .cmd files...') + + # Prepare arguments for each worker + worker_args = [(cmdfile, srcdir, srcdir_real) for cmdfile in cmdfiles] + + # Use multiprocessing to process cmdfiles in parallel + if jobs is None: + jobs = multiprocessing.cpu_count() + + used_sources = set() + with multiprocessing.Pool(processes=jobs) as pool: + # Process cmdfiles in parallel + for sources in pool.imap_unordered(_process_cmdfile, worker_args, + chunksize=100): + used_sources.update(sources) + + tout.info(f'Found {len(used_sources)} used source files') + + return used_sources + + +def select_sources(srcdir, build_dir, filter_pattern, jobs=None): + """Find all and used source files, optionally applying a filter. + + Args: + srcdir (str): Root directory of the source tree + build_dir (str): Build directory path + filter_pattern (str): Optional wildcard pattern to filter files + (None to skip) + jobs (int): Number of parallel jobs (None = use all CPUs) + + Returns: + tuple: (all_sources, used_sources, skipped_sources) - sets of file paths + """ + all_sources = find_all_source_files(srcdir) + + # Find used source files + used_sources = find_used_sources(build_dir, srcdir, jobs) + + # Apply filter if specified + if filter_pattern: + all_sources = {f for f in all_sources + if fnmatch.fnmatch(os.path.basename(f), + filter_pattern) or + fnmatch.fnmatch(f, filter_pattern)} + used_sources = {f for f in used_sources + if fnmatch.fnmatch(os.path.basename(f), + filter_pattern) or + fnmatch.fnmatch(f, filter_pattern)} + tout.progress(f'After filter: {len(all_sources)} total, ' + + f'{len(used_sources)} used') + + # Calculate unused sources + skipped_sources = all_sources - used_sources + + return all_sources, used_sources, skipped_sources + + +def do_build(args): + """Set up and validate source and build directories. + + Args: + args (Namespace): Parsed command-line arguments + + Returns: + tuple: (srcdir, build_dir) on success + Calls tout.fatal() on failure + """ + srcdir = os.path.realpath(args.source) + + if not os.path.isdir(srcdir): + tout.fatal(f'Source directory does not exist: {srcdir}') + + # Determine build directory + if args.build_dir: + build_dir = os.path.realpath(args.build_dir) + else: + # Use default: build_base/ + build_dir = os.path.join(args.build_base, args.board) + + # If not skipping build, build it + if not args.no_build: + if args.board: + build_board(args.board, build_dir, srcdir, args.adjust, + args.use_dwarf) + # Note: build_board() calls tout.fatal() on failure which exits + + # Verify build directory exists + if not os.path.isdir(build_dir): + tout.fatal(f'Build directory does not exist: {build_dir}') + + tout.info(f'Analysing build in: {build_dir}') + tout.info(f'Source directory: {srcdir}') + + return srcdir, build_dir + + +def build_board(board, build_dir, srcdir, adjust_cfg=None, use_dwarf=False): + """Build a board using buildman. + + Args: + board (str): Board name to build + build_dir (str): Directory to build into + srcdir (str): U-Boot source directory + adjust_cfg (list): List of CONFIG adjustments + use_dwarf (bool): Enable CC_OPTIMIZE_FOR_DEBUG to prevent inlining + + Returns: + True on success (note: failures call tout.fatal() which exits) + """ + tout.info(f"Building board '{board}' with buildman...") + tout.info(f'Build directory: {build_dir}') + + # Enable CC_OPTIMIZE_FOR_DEBUG if using DWARF to prevent inlining + if use_dwarf: + adjust_cfg = list(adjust_cfg or []) + ['CC_OPTIMIZE_FOR_DEBUG'] + + if adjust_cfg: + # Count actual adjustments (handle comma-separated values) + num_adjustments = sum(len([x for x in item.split(',') if x.strip()]) + for item in adjust_cfg) + tout.progress(f'Building with {num_adjustments} Kconfig adjustments') + else: + tout.progress('Building') + + # Run buildman to build the board + # -L: disable LTO, -w: enable warnings, -o: output directory, + # -m: mrproper (clean), -I: show errors/warnings only (incremental) + cmd = ['buildman', '--board', board, '-L', '-w', '-m', '-I', '-o', + build_dir] + + # Add CONFIG adjustments if specified + if adjust_cfg: + for adj in adjust_cfg: + cmd.extend(['--adjust-cfg', adj]) + + try: + result = subprocess.run(cmd, cwd=srcdir, check=False, + capture_output=False, text=True) + if result.returncode != 0: + tout.fatal(f'buildman exited with code {result.returncode}') + return True + except FileNotFoundError: + tout.fatal('buildman not found. Please ensure buildman is in ' + 'your PATH.') + except OSError as e: + tout.fatal(f'Error running buildman: {e}') + return None + + +def parse_args(argv=None): + """Parse command-line arguments. + + Returns: + Parsed arguments object + """ + parser = argparse.ArgumentParser( + description='Analyse C source code usage in U-Boot builds', + epilog='Example: %(prog)s -b sandbox --stats') + + parser.add_argument('-s', '--source', type=str, default='.', + help='Path to U-Boot source directory ' + '(default: current directory)') + parser.add_argument('-b', '--board', type=str, default='sandbox', + help='Board name to build and analyse (default: sandbox)') + parser.add_argument('-B', '--build-dir', type=str, + help='Use existing build directory instead of building') + parser.add_argument('--build-base', type=str, default=BUILD_BASE, + help=f'Base directory for builds (default: {BUILD_BASE})') + parser.add_argument('-n', '--no-build', action='store_true', + help='Skip building, use existing build directory') + parser.add_argument('-a', '--adjust', type=str, action='append', + help='Adjust CONFIG options ' + '(e.g., -a CONFIG_FOO, -a ~CONFIG_BAR)') + parser.add_argument('-w', '--dwarf', action='store_true', + dest='use_dwarf', + help='Use DWARF debug info ' + '(more accurate, requires rebuild)') + parser.add_argument('-l', '--lsp', action='store_true', + dest='use_lsp', + help='Use clangd LSP to analyse inactive regions ' + '(requires clangd)') + parser.add_argument('-u', '--unifdef', type=str, default='unifdef', + help='Path to unifdef executable (default: unifdef)') + parser.add_argument('-j', '--jobs', type=int, metavar='N', + help='Number of parallel jobs (default: all CPUs)') + parser.add_argument('-i', '--include-headers', action='store_true', + help='Include header files in unifdef analysis') + parser.add_argument('-f', '--filter', type=str, metavar='PATTERN', + help='Filter files by wildcard pattern (e.g., *acpi*)') + parser.add_argument('-v', '--verbose', action='store_true', + help='Show verbose output') + parser.add_argument('-D', '--debug', action='store_true', + help='Enable debug mode') + + # Subcommands + subparsers = parser.add_subparsers(dest='cmd', help='Command to execute') + + # stats command (default) + stats = subparsers.add_parser('stats', + help='Show statistics about code usage') + stats.add_argument('--top', type=int, metavar='N', default=20, + help='Show top N files with most inactive code ' + '(default: 20)') + + # dirs command + dirs = subparsers.add_parser('dirs', help='Show directory breakdown') + dirs.add_argument('-s', '--subdirs', action='store_true', + help='Show breakdown by all subdirectories') + dirs.add_argument('-f', '--show-files', action='store_true', + help='Show individual files within directories') + dirs.add_argument('-e', '--show-empty', action='store_true', + help='Show directories with 0 lines used') + + # detail command + detail = subparsers.add_parser('detail', + help='Show line-by-line analysis of files') + detail.add_argument('files', nargs='+', metavar='FILE', + help='File(s) to analyse') + + # unused command + subparsers.add_parser('unused', help='List all unused source files') + + # used command + subparsers.add_parser('used', help='List all used source files') + + # summary command + subparsers.add_parser('summary', + help='Show per-file summary of active/inactive lines') + + # copy-used command + copy = subparsers.add_parser('copy-used', + help='Copy used source files to a directory') + copy.add_argument('dest_dir', metavar='DIR', + help='Destination directory') + + args = parser.parse_args(argv) + + # Default command is stats + if not args.cmd: + args.cmd = 'stats' + # Set default value for --top when stats is the default command + args.top = 20 + + # Map subcommand arguments to expected names + if args.cmd == 'detail': + args.detail = args.files + elif args.cmd == 'copy-used': + args.copy_used = args.dest_dir + else: + args.detail = None + args.copy_used = None + + # Validation + if args.no_build and args.adjust: + tout.warning('-a/--adjust ignored when using -n/--no-build') + + return args + + +def do_analysis(used, build_dir, srcdir, unifdef_path, include_headers, jobs, + use_lsp, keep_temps=False): + """Perform line-level analysis if requested. + + Args: + used (set): Set of used source files + build_dir (str): Build directory path + srcdir (str): Source directory path + unifdef_path (str): Path to unifdef executable (None to use DWARF/LSP) + include_headers (bool): Include header files in unifdef analysis + jobs (int): Number of parallel jobs + use_lsp (bool): Use LSP (clangd) instead of DWARF + keep_temps (bool): If True, keep temporary files for debugging + + Returns: + dict: Line-level analysis results, or None if not requested/failed + """ + if unifdef_path: + config_file = os.path.join(build_dir, '.config') + analyser = unifdef.UnifdefAnalyser(config_file, srcdir, used, + unifdef_path, include_headers, + keep_temps) + elif use_lsp: + analyser = lsp.LspAnalyser(build_dir, srcdir, used, keep_temps) + else: + analyser = dwarf.DwarfAnalyser(build_dir, srcdir, used, keep_temps) + return analyser.process(jobs) + + +def do_output(args, all_srcs, used, skipped, results, srcdir): + """Perform output operation based on command. + + Args: + args (argparse.Namespace): Parsed command-line arguments + all_srcs (set): All source files + used (set): Used source files + skipped (set): Unused source files + results (dict): Line-level analysis results (or None) + srcdir (str): Source directory path + + Returns: + bool: True on success, False on failure + """ + terminal.print_clear() + + # Execute the command + if args.cmd == 'detail': + # Show detail for each file, collecting missing files + missing = [] + shown = 0 + for fname in args.detail: + if output.show_file_detail(fname, results, srcdir): + shown += 1 + else: + missing.append(fname) + + # Show summary if any files were missing + if missing: + tout.warning(f'{len(missing)} file(s) not found in analysed ' + f"sources: {', '.join(missing)}") + + ok = shown > 0 + elif args.cmd == 'summary': + ok = output.show_file_summary(results, srcdir) + elif args.cmd == 'unused': + ok = output.list_unused_files(skipped, srcdir) + elif args.cmd == 'used': + ok = output.list_used_files(used, srcdir) + elif args.cmd == 'copy-used': + ok = output.copy_used_files(used, srcdir, args.copy_used) + elif args.cmd == 'dirs': + ok = output.show_dir_breakdown(all_srcs, used, results, srcdir, + args.subdirs, args.show_files, + args.show_empty) + else: + # stats (default) + ok = output.show_statistics(all_srcs, used, skipped, results, srcdir, + args.top) + + return ok + + +def main(argv=None): + """Main function. + + Args: + argv (list): Command-line arguments (default: sys.argv[1:]) + + Returns: + int: Exit code (0 for success, 1 for failure) + """ + tout.init(tout.NOTICE) + args = parse_args(argv) + + # Init tout based on verbosity flags + if args.debug: + tout.init(tout.DEBUG) + elif args.verbose: + tout.init(tout.INFO) + + srcdir, build_dir = do_build(args) + all_srcs, used, skipped = select_sources(srcdir, build_dir, args.filter, + args.jobs) + + # Determine which files to analyse + files_to_analyse = used + if args.cmd == 'detail': + # For detail command, only analyse the requested files + files_to_analyse = set() + for fname in args.detail: + abs_path = os.path.realpath(os.path.join(srcdir, fname)) + if abs_path in used: + files_to_analyse.add(abs_path) + + # Perform line-level analysis + unifdef_path = None if (args.use_dwarf or args.use_lsp) else args.unifdef + keep_temps = args.debug + results = do_analysis(files_to_analyse, build_dir, srcdir, unifdef_path, + args.include_headers, args.jobs, args.use_lsp, + keep_temps) + if results is None: + return 1 + + if not do_output(args, all_srcs, used, skipped, results, srcdir): + return 1 + + return 0 + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/tools/codman/output.py b/tools/codman/output.py new file mode 100644 index 00000000000..2b1d097fe26 --- /dev/null +++ b/tools/codman/output.py @@ -0,0 +1,536 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright 2025 Canonical Ltd +# +"""Output formatting and display functions for srcman. + +This module provides functions for displaying analysis results in various +formats: +- Statistics views (file-level and line-level) +- Directory breakdowns (top-level and subdirectories) +- Per-file summaries +- Detailed line-by-line views +- File listings (used/unused) +- File copying operations +""" + +import os +import shutil +import sys +from collections import defaultdict + +# Import from tools directory +sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..')) +from u_boot_pylib import terminal, tout # pylint: disable=wrong-import-position + + +class DirStats: # pylint: disable=too-few-public-methods + """Statistics for a directory. + + Attributes: + total: Total number of files in directory + used: Number of files used (compiled) + unused: Number of files not used + lines_total: Total lines of code in directory + lines_used: Number of active lines (after preprocessing) + files: List of file info dicts (for --show-files) + """ + def __init__(self): + self.total = 0 + self.used = 0 + self.unused = 0 + self.lines_total = 0 + self.lines_used = 0 + self.files = [] + + +def count_lines(file_path): + """Count lines in a file""" + try: + with open(file_path, 'r', encoding='utf-8', errors='ignore') as f: + return sum(1 for _ in f) + except IOError: + return 0 + + +def klocs(lines): + """Format line count in thousands, rounded to 1 decimal place. + + Args: + lines (int): Line count (e.g., 3500) + + Returns: + Formatted string in thousands (e.g., '3.5') + """ + kloc = round(lines / 1000, 1) + return f'{kloc:.1f}' + + +def percent(numerator, denominator): + """Calculate percentage, handling division by zero. + + Args: + numerator (int/float): The numerator + denominator (int/float): The denominator + + Returns: + float: Percentage (0-100), or 0 if denominator is 0 + """ + return 100 * numerator / denominator if denominator else 0 + + +def print_heading(text, width=70, char='='): + """Print a heading with separator lines. + + Args: + text (str): Heading text to display (empty for separator only) + width (int): Width of the separator line + char (str): Character to use for separator + """ + print(char * width) + if text: + print(text) + print(char * width) + + +def show_file_detail(detail_file, file_results, srcdir): + """Show detailed line-by-line analysis for a specific file. + + Args: + detail_file (str): Path to the file to show details for (relative or + absolute) + file_results (dict): Dictionary mapping file paths to analysis results + srcdir (str): Root directory of the source tree + + Returns: + True on success, False on error + """ + detail_path = os.path.realpath(detail_file) + if detail_path not in file_results: + # Try relative to source root + detail_path = os.path.realpath(os.path.join(srcdir, detail_file)) + + if detail_path in file_results: + result = file_results[detail_path] + rel_path = os.path.relpath(detail_path, srcdir) + + print_heading(f'DETAIL FOR: {rel_path}', width=70) + print(f'Total lines: {result.total_lines:6}') + pct_active = percent(result.active_lines, result.total_lines) + pct_inactive = percent(result.inactive_lines, result.total_lines) + print(f'Active lines: {result.active_lines:6} ({pct_active:.1f}%)') + print(f'Inactive lines: {result.inactive_lines:6} ' + + f'({pct_inactive:.1f}%)') + print() + + # Show the file with status annotations + with open(detail_path, 'r', encoding='utf-8', errors='ignore') as f: + lines = f.readlines() + + col = terminal.Color() + for line_num, line in enumerate(lines, 1): + status = result.line_status.get(line_num, 'unknown') + marker = '-' if status == 'inactive' else ' ' + prefix = f'{marker} {line_num:4} | ' + code = line.rstrip() + + if status == 'active': + # Normal color for active code + print(prefix + code) + else: + # Non-bright cyan for inactive code + print(prefix + col.build(terminal.Color.CYAN, code, + bright=False)) + return True + + # File not found - caller handles errors + return False + + +def show_file_summary(file_results, srcdir): + """Show per-file summary of line analysis. + + Args: + file_results (dict): Dictionary mapping file paths to analysis results + srcdir (str): Root directory of the source tree + + Returns: + bool: True on success + """ + print_heading('PER-FILE SUMMARY', width=90) + print(f"{'File':<50} {'Total':>8} {'Active':>8} " + f"{'Inactive':>8} {'%Active':>8}") + print('-' * 90) + + for source_file in sorted(file_results.keys()): + result = file_results[source_file] + rel_path = os.path.relpath(source_file, srcdir) + if len(rel_path) > 47: + rel_path = '...' + rel_path[-44:] + + pct_active = percent(result.active_lines, result.total_lines) + print(f'{rel_path:<50} {result.total_lines:>8} ' + f'{result.active_lines:>8} {result.inactive_lines:>8} ' + f'{pct_active:>7.1f}%') + + return True + + +def list_unused_files(skipped_sources, srcdir): + """List unused source files. + + Args: + skipped_sources (set of str): Set of unused source file paths (relative + to srcdir) + srcdir (str): Root directory of the source tree + + Returns: + bool: True on success + """ + print(f'Unused source files ({len(skipped_sources)}):') + for source_file in sorted(skipped_sources): + try: + rel_path = os.path.relpath(source_file, srcdir) + except ValueError: + rel_path = source_file + print(f' {rel_path}') + + return True + + +def list_used_files(used_sources, srcdir): + """List used source files. + + Args: + used_sources (set of str): Set of used source file paths (relative + to srcdir) + srcdir (str): Root directory of the source tree + + Returns: + bool: True on success + """ + print(f'Used source files ({len(used_sources)}):') + for source_file in sorted(used_sources): + try: + rel_path = os.path.relpath(source_file, srcdir) + except ValueError: + rel_path = source_file + print(f' {rel_path}') + + return True + + +def copy_used_files(used_sources, srcdir, dest_dir): + """Copy used source files to a destination directory, preserving structure. + + Args: + used_sources (set): Set of used source file paths (relative to srcdir) + srcdir (str): Root directory of the source tree + dest_dir (str): Destination directory for the copy + + Returns: + True on success, False if errors occurred + """ + if os.path.exists(dest_dir): + tout.error(f'Destination directory already exists: {dest_dir}') + return False + + tout.progress(f'Copying {len(used_sources)} used source files to ' + + f'{dest_dir}') + + copied_count = 0 + error_count = 0 + + for source_file in sorted(used_sources): + src_path = os.path.join(srcdir, source_file) + dest_path = os.path.join(dest_dir, source_file) + + try: + # Create parent directory if needed + dest_parent = os.path.dirname(dest_path) + os.makedirs(dest_parent, exist_ok=True) + + # Copy the file + shutil.copy2(src_path, dest_path) + copied_count += 1 + except IOError as e: + error_count += 1 + tout.error(f'Error copying {source_file}: {e}') + + tout.progress(f'Copied {copied_count} files to {dest_dir}') + if error_count: + tout.error(f'Failed to copy {error_count} files') + return False + + return True + + +def collect_dir_stats(all_sources, used_sources, file_results, srcdir, + by_subdirs, show_files): + """Collect statistics organized by directory. + + Args: + all_sources (set): Set of all source file paths + used_sources (set): Set of used source file paths + file_results (dict): Optional dict mapping file paths to line + analysis results (or None) + srcdir (str): Root directory of the source tree + by_subdirs (bool): If True, use full subdirectory paths; + otherwise top-level only + show_files (bool): If True, collect individual file info within + each directory + + Returns: + dict: Directory statistics keyed by directory path + """ + dir_stats = defaultdict(DirStats) + + for source_file in all_sources: + rel_path = os.path.relpath(source_file, srcdir) + + if by_subdirs: + # Use the full directory path (not including the filename) + dir_path = os.path.dirname(rel_path) + if not dir_path: + dir_path = '.' + else: + # Use only the top-level directory + dir_path = (rel_path.split(os.sep)[0] if os.sep in rel_path + else rel_path) + + line_count = count_lines(source_file) + dir_stats[dir_path].total += 1 + dir_stats[dir_path].lines_total += line_count + + if source_file in used_sources: + dir_stats[dir_path].used += 1 + # Use active line count if line-level analysis was performed + # Normalize path to match file_results keys (absolute paths) + abs_source = os.path.realpath(source_file) + + # Try to find the file in file_results + result = None + if file_results: + if abs_source in file_results: + result = file_results[abs_source] + elif source_file in file_results: + result = file_results[source_file] + + if result: + active_lines = result.active_lines + inactive_lines = result.inactive_lines + dir_stats[dir_path].lines_used += active_lines + # Store file info for --show-files (exclude .h files) + if show_files and not rel_path.endswith('.h'): + dir_stats[dir_path].files.append({ + 'path': rel_path, + 'total': line_count, + 'active': active_lines, + 'inactive': inactive_lines + }) + else: + # File not found in results - count all lines + tout.debug(f'File not in results (using full count): ' + f'{rel_path}') + dir_stats[dir_path].lines_used += line_count + if show_files and not rel_path.endswith('.h'): + dir_stats[dir_path].files.append({ + 'path': rel_path, + 'total': line_count, + 'active': line_count, + 'inactive': 0 + }) + else: + dir_stats[dir_path].unused += 1 + + return dir_stats + + +def print_dir_stats(dir_stats, file_results, by_subdirs, show_files, + show_empty): + """Print directory statistics table. + + Args: + dir_stats (dict): Directory statistics keyed by directory path + file_results (dict): Optional dict mapping file paths to line analysis + results (or None) + by_subdirs (bool): If True, show full subdirectory breakdown; otherwise + top-level only + show_files (bool): If True, show individual files within directories + show_empty (bool): If True, show directories with 0 lines used + """ + # Sort alphabetically by directory name + sorted_dirs = sorted(dir_stats.items(), key=lambda x: x[0]) + + for dir_path, stats in sorted_dirs: + # Skip subdirectories with 0 lines used unless --show-zero-lines is set + if by_subdirs and not show_empty and stats.lines_used == 0: + continue + + pct_used = percent(stats.used, stats.total) + pct_code = percent(stats.lines_used, stats.lines_total) + # Truncate long paths + display_path = dir_path + if len(display_path) > 37: + display_path = '...' + display_path[-34:] + print(f'{display_path:<40} {stats.total:>7} {stats.used:>7} ' + f'{pct_used:>6.0f} {pct_code:>6.0f} ' + f'{klocs(stats.lines_total):>8} {klocs(stats.lines_used):>7}') + + # Show individual files if requested + if show_files and stats.files: + # Sort files by inactive lines (descending) for line-level, or + # alphabetically otherwise + if file_results: + sorted_files = sorted(stats.files, key=lambda x: x['inactive'], + reverse=True) + else: + sorted_files = sorted(stats.files, key=lambda x: x['path']) + + for info in sorted_files: + filename = os.path.basename(info['path']) + if len(filename) > 35: + filename = filename[:32] + '...' + + if file_results: + # Show line-level details + pct_active = percent(info['active'], info['total']) + print(f" {filename:<38} {info['total']:>7} " + f"{info['active']:>7} {pct_active:>6.1f} " + f"{info['inactive']:>7}") + else: + # Show file-level only + print(f" {filename:<38} {info['total']:>7} lines") + + +def show_dir_breakdown(all_sources, used_sources, file_results, srcdir, + by_subdirs, show_files, show_empty): + """Show breakdown by directory (top-level or subdirectories). + + Args: + all_sources (set): Set of all source file paths + used_sources (set): Set of used source file paths + file_results (dict): Optional dict mapping file paths to line analysis + results (or None) + srcdir (str): Root directory of the source tree + by_subdirs (bool): If True, show full subdirectory breakdown; otherwise + top-level only + show_files (bool): If True, show individual files within each directory + show_empty (bool): If True, show directories with 0 lines used + + Returns: + bool: True on success + """ + # Width of the main table (Directory + Total + Used columns) + table_width = 87 + + print_heading('BREAKDOWN BY TOP-LEVEL DIRECTORY' if by_subdirs else '', + width=table_width) + print(f"{'Directory':<40} {'Files':>7} {'Used':>7} {'%Used':>6} " + + f"{'%Code':>6} {'kLOC':>8} {'Used':>7}") + print('-' * table_width) + + # Collect directory statistics + dir_stats = collect_dir_stats(all_sources, used_sources, file_results, + srcdir, by_subdirs, show_files) + + # Print directory statistics + print_dir_stats(dir_stats, file_results, by_subdirs, show_files, show_empty) + + print('-' * table_width) + total_lines_all = sum(count_lines(f) for f in all_sources) + # Calculate used lines: if we have file_results, use active_lines from there + # Otherwise, count all lines in used files + if file_results: + total_lines_used = sum(r.active_lines for r in file_results.values()) + else: + total_lines_used = sum(count_lines(f) for f in used_sources) + pct_files = percent(len(used_sources), len(all_sources)) + pct_code = percent(total_lines_used, total_lines_all) + print(f"{'TOTAL':<40} {len(all_sources):>7} {len(used_sources):>7} " + f"{pct_files:>6.0f} {pct_code:>6.0f} " + f"{klocs(total_lines_all):>8} {klocs(total_lines_used):>7}") + print_heading('', width=table_width) + print() + + return True + + +def show_statistics(all_sources, used_sources, skipped_sources, file_results, + srcdir, top_n): + """Show overall statistics about source file usage. + + Args: + all_sources (set of str): Set of all source file paths + used_sources (set of str): Set of used source file paths + skipped_sources (set of str): Set of unused source file paths + file_results (dict): Optional dict mapping file paths to line analysis + results + srcdir (str): Root directory of the source tree + top_n (int): Number of top files with most inactive code to show + + Returns: + bool: True on success + """ + # Calculate line counts - use file_results (DWARF/unifdef) if available + if file_results: + # Use active lines from analysis results + used_lines = sum(r.active_lines for r in file_results.values()) + else: + # Fall back to counting all lines in used files + used_lines = sum(count_lines(f) for f in used_sources) + + unused_lines = sum(count_lines(f) for f in skipped_sources) + total_lines = used_lines + unused_lines + + print_heading('FILE-LEVEL STATISTICS', width=70) + print(f'Total source files: {len(all_sources):6}') + used_pct = percent(len(used_sources), len(all_sources)) + print(f'Used source files: {len(used_sources):6} ({used_pct:.1f}%)') + unused_pct = percent(len(skipped_sources), len(all_sources)) + print(f'Unused source files: {len(skipped_sources):6} ' + + f'({unused_pct:.1f}%)') + print() + print(f'Total lines of code: {total_lines:6}') + used_lines_pct = percent(used_lines, total_lines) + print(f'Used lines of code: {used_lines:6} ({used_lines_pct:.1f}%)') + unused_lines_pct = percent(unused_lines, total_lines) + print(f'Unused lines of code: {unused_lines:6} ' + + f'({unused_lines_pct:.1f}%)') + print_heading('', width=70) + + # If line-level analysis was performed, show those stats too + if file_results: + print() + total_lines_analysed = sum(r.total_lines for r in file_results.values()) + active_lines = sum(r.active_lines for r in file_results.values()) + inactive_lines = sum(r.inactive_lines for r in file_results.values()) + + print_heading('LINE-LEVEL STATISTICS (within compiled files)', width=70) + print(f'Files analysed: {len(file_results):6}') + print(f'Total lines in used files:{total_lines_analysed:6}') + active_pct = percent(active_lines, total_lines_analysed) + print(f'Active lines: {active_lines:6} ' + + f'({active_pct:.1f}%)') + inactive_pct = percent(inactive_lines, total_lines_analysed) + print(f'Inactive lines: {inactive_lines:6} ' + + f'({inactive_pct:.1f}%)') + print_heading('', width=70) + print() + + # Show top files with most inactive code + files_by_inactive = sorted( + file_results.items(), + key=lambda x: x[1].inactive_lines, + reverse=True + ) + + print(f'TOP {top_n} FILES WITH MOST INACTIVE CODE:') + print('-' * 70) + for source_file, result in files_by_inactive[:top_n]: + rel_path = os.path.relpath(source_file, srcdir) + pct_inactive = percent(result.inactive_lines, result.total_lines) + print(f' {result.inactive_lines:5} inactive lines ' + + f'({pct_inactive:4.1f}%) - {rel_path}') + + return True From patchwork Mon Nov 24 13:49:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 762 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992218; bh=W/+yofnzNluKQRTaJMD3uWTXbBN8ydAKE1H6gZ2A5vk=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=J0eXzdbapMckUf48ShZMilB48dnKNgN4zNMoZYmvxuxVXjMmihBDDNqEwdxUaYBld LwYwJUktDALIqVgvtTVx7GZHmr97ymQxEuxfJDY8JFQbL5I9rq+UYqK/KTZBVtFbpq I+gzzfKngd69hqVfks6x9K+PxGlJRiD9jA59hw1S5DuqxLrgvsx38SM28r8BKg4gqL mrpxD9Lwu9R7tsG0jbVgRDxi61Yuts/kiMSJFyF1t+Q/xHzXcO7oGEjGLp7HVGVVrV qZr/UoNbO9vWhEGsKuYKLJ32vTI0GnT6bz2h+Lwk3q2+18vDDnaI9PfZ+X2SBizhrs FqFDYmifgsEow== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 556CB68785 for ; Mon, 24 Nov 2025 06:50:18 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 2ZJRfHwd9qjN for ; Mon, 24 Nov 2025 06:50:18 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992218; bh=W/+yofnzNluKQRTaJMD3uWTXbBN8ydAKE1H6gZ2A5vk=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=J0eXzdbapMckUf48ShZMilB48dnKNgN4zNMoZYmvxuxVXjMmihBDDNqEwdxUaYBld LwYwJUktDALIqVgvtTVx7GZHmr97ymQxEuxfJDY8JFQbL5I9rq+UYqK/KTZBVtFbpq I+gzzfKngd69hqVfks6x9K+PxGlJRiD9jA59hw1S5DuqxLrgvsx38SM28r8BKg4gqL mrpxD9Lwu9R7tsG0jbVgRDxi61Yuts/kiMSJFyF1t+Q/xHzXcO7oGEjGLp7HVGVVrV qZr/UoNbO9vWhEGsKuYKLJ32vTI0GnT6bz2h+Lwk3q2+18vDDnaI9PfZ+X2SBizhrs FqFDYmifgsEow== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 388D16873E for ; Mon, 24 Nov 2025 06:50:18 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992216; bh=inHNO3X6m/SQ8CXrF0LzUS4/8LHl/UPnhbqQKj4ZsN4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=J14tngffN1cvncmDS5QbBQwpPXtGjgNRyK8QHC294mJOaL1GniDJicHyua6gAY0Cv 3DcbLsFHgC65zS0t5NcEqEeIs4AHuNh1rjqNrV97P9JpUwShttO7JA3l0VDmmaJv7W eQRFm1TPNrWwfoHfK/igrNSxBsziwYEbzDEPzahHueU7QJ7cHtIO4ZjPL+4dw6dsCA HKI2HFOn5ZAqVgb3oGCUERVV/VrZI+DQnKYEldFNled3y7PxEXRSMUoz0oQEIEnMfS WnLyQd0GDtoMxRGA7r/DbhxRf8r/eblorDUnf7TyMAJb2au7cFqz0OzKBQqUMDYevr ltTZ6rCszC12A== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 664CE68669; Mon, 24 Nov 2025 06:50:16 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id moULYArEe-hj; Mon, 24 Nov 2025 06:50:16 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992212; bh=le0+sTXla3ZI32wlN1obVb5l0pXwQ9Im5Qgut4SA8VE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XHTyu6XIXLrgw3YISv+K0MA18ZWV89hVZcrnPPJKfTm/JWQ6FK+pZdHa3do3rzfa/ 1Ef2zg1qmHDLAWTngYPwlBSuwxtPgOOKhy35MMFtZ4d9UXUTL5XPkqbEZD5H0mNuzJ XmnzYHHWplPLayO3MCyXXzg5m7WaxT0D+nWAAbC0YbOGGMyZbs5IjE4+HFBKvbNnJR axgScCWBXtpZDSnZ6/L1Ml6++8WIpvbtlwbqc3CthgOthZevfoCwD42Pi9376FBWam zsYdtJY3A3fsPGwQP9mQNkAZAedaUmIza+Qz+pWu2jcwBGfpGG9UBtplQwthRXQEyq w2wUBKDBNwE/w== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id 246A568688; Mon, 24 Nov 2025 06:50:12 -0700 (MST) From: Simon Glass To: U-Boot Concept Date: Mon, 24 Nov 2025 06:49:16 -0700 Message-ID: <20251124134932.1991031-6-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251124134932.1991031-1-sjg@u-boot.org> References: <20251124134932.1991031-1-sjg@u-boot.org> MIME-Version: 1.0 Message-ID-Hash: LXSICGNHODBQ5OKJG5RMAK6QZRHVHV6R X-Message-ID-Hash: LXSICGNHODBQ5OKJG5RMAK6QZRHVHV6R X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Simon Glass , Claude X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 5/9] codman: Provide an unifdef analyser List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass Add a way to do static preprocessor analysis using unifdef, as a way of figuring out what code is actually used in the build. I have modified the unifdef tool as follows: - O(1) algorithm for symbol lookup, instead of O(n) - support for IS_ENABLED(), CONFIG_IS_ENABLED() The first patch was sent upstream. The others are U-Boot-specific so I have not submitted those. Please get in touch if you would like the patches. Co-developed-by: Claude Signed-off-by: Simon Glass --- tools/codman/unifdef.py | 429 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 429 insertions(+) create mode 100644 tools/codman/unifdef.py diff --git a/tools/codman/unifdef.py b/tools/codman/unifdef.py new file mode 100644 index 00000000000..560b323b460 --- /dev/null +++ b/tools/codman/unifdef.py @@ -0,0 +1,429 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright 2025 Canonical Ltd +# +"""Unifdef-based line-level analysis for source code. + +This module provides functionality to analyse which lines in source files +are active vs inactive based on CONFIG_* settings, using the unifdef tool. +""" + +import multiprocessing +import os +import re +import shutil +import subprocess +import tempfile +import time + +from buildman import kconfiglib +from u_boot_pylib import tout +from analyser import Analyser, FileResult + + +def load_config(config_file, srcdir='.'): + """Load CONFIG_* symbols from a .config file and Kconfig. + + Args: + config_file (str): Path to .config file + srcdir (str): Path to source directory (for Kconfig loading) + + Returns: + tuple: (config_dict, error_message) where config_dict is a dictionary + mapping CONFIG_* symbol names to values, and error_message is None + on success or an error string on failure + """ + config = {} + + # First, load from .config file + with open(config_file, 'r', encoding='utf-8') as f: + for line in f: + line = line.strip() + + # Skip comments and blank lines + if not line or line.startswith('#'): + # Check for "is not set" pattern + if ' is not set' in line: + # Extract CONFIG name: '# CONFIG_FOO is not set' + parts = line.split() + if len(parts) >= 2 and parts[1].startswith('CONFIG_'): + config_name = parts[1] + config[config_name] = None + continue + + # Parse CONFIG_* assignments + if '=' in line: + name, value = line.split('=', 1) + if name.startswith('CONFIG_'): + config[name] = value + + # Then, load all Kconfig symbols and set undefined ones to None + # Only do this if we have a Kconfig file (i.e., in a real U-Boot tree) + kconfig_path = os.path.join(srcdir, 'Kconfig') + if not os.path.exists(kconfig_path): + # No Kconfig - probably a test environment, just use .config values + return config, None + + try: + # Set environment variables needed by kconfiglib + old_srctree = os.environ.get('srctree') + old_ubootversion = os.environ.get('UBOOTVERSION') + old_objdir = os.environ.get('KCONFIG_OBJDIR') + + os.environ['srctree'] = srcdir + os.environ['UBOOTVERSION'] = 'dummy' + os.environ['KCONFIG_OBJDIR'] = '' + + # Load Kconfig + kconf = kconfiglib.Kconfig(warn=False) + + # Add all defined symbols that aren't already in config as None + # kconfiglib provides names without CONFIG_ prefix + for name in kconf.syms: + config_name = f'CONFIG_{name}' + if config_name not in config: + # Symbol is defined in Kconfig but not in .config + config[config_name] = None + + # Restore environment + if old_srctree is not None: + os.environ['srctree'] = old_srctree + elif 'srctree' in os.environ: + del os.environ['srctree'] + if old_ubootversion is not None: + os.environ['UBOOTVERSION'] = old_ubootversion + elif 'UBOOTVERSION' in os.environ: + del os.environ['UBOOTVERSION'] + if old_objdir is not None: + os.environ['KCONFIG_OBJDIR'] = old_objdir + elif 'KCONFIG_OBJDIR' in os.environ: + del os.environ['KCONFIG_OBJDIR'] + + tout.progress(f'Loaded {len(kconf.syms)} Kconfig symbols') + except (OSError, IOError, ValueError, ImportError) as e: + # Return error if kconfiglib fails - we need all symbols for accurate analysis + return None, f'Failed to load Kconfig symbols: {e}' + + return config, None + + +def match_lines(orig_lines, processed_output, source_file): + """Match original and processed lines to determine which are active. + + Parses #line directives from unifdef -n output to determine exactly which + lines from the original source are active vs inactive. + + Args: + orig_lines (list): List of original source lines + processed_output (str): Processed output from unifdef -n + source_file (str): Path to source file (for matching #line directives) + + Returns: + dict: Mapping of line numbers (1-indexed) to 'active'/'inactive' status + """ + total_lines = len(orig_lines) + line_status = {} + + # set up all lines as inactive + for i in range(1, total_lines + 1): + line_status[i] = 'inactive' + + # Parse #line directives to find which lines are active + # Format: #line '' + # When we see a #line directive, all following non-directive lines + # come from that line number onward in the original file + # If no #line directive appears at start, output starts at line 1 + current_line = 1 # Start at line 1 by default + line_pattern = re.compile(r'^#line (\d+) "(.+)"$') + source_basename = source_file.split('/')[-1] + + for output_line in processed_output.splitlines(): + # Check for #line directive + match = line_pattern.match(output_line) + if match: + line_num = int(match.group(1)) + file_path = match.group(2) + # Only track lines from our source file (unifdef may include + # #line directives from headers) + if file_path == source_file or file_path.endswith(source_basename): + current_line = line_num + else: + # This is a #line for a different file (e.g., header) + # Stop tracking until we see our file again + current_line = None + elif current_line is not None: + # This is a real line from the source file + if current_line <= total_lines: + line_status[current_line] = 'active' + current_line += 1 + + return line_status + + +def worker(args): + """Run unifdef on a source file to determine active/inactive lines. + + Uses unifdef with -k flag to process the file, then uses difflib to match + original lines to processed lines to determine which are active vs inactive. + + Args: + args (tuple): Tuple of (source_file, defs_file, unifdef_path, + track_lines) + + Returns: + Tuple of (source_file, total_lines, active_lines, inactive_lines, + line_status, error_msg) + line_status is a dict mapping line numbers to 'active'/'inactive', or + {} if not tracked + error_msg is None on success, or an error string on failure + """ + source_file, defs_file, unifdef_path, track_lines = args + + try: + with open(source_file, 'r', encoding='utf-8', errors='ignore') as f: + orig_lines = f.readlines() + + total_lines = len(orig_lines) + + # Run unifdef to process the file + # -n: add #line directives for tracking original line numbers + # -E: error on unterminated conditionals + # -f: use defs file + result = subprocess.run( + [unifdef_path, '-n', '-E', '-f', defs_file, source_file], + capture_output=True, + text=True, + encoding='utf-8', + errors='ignore', + check=False + ) + + if result.returncode > 1: + # Error running unifdef + # Check if it's an 'obfuscated' error - these are expected for + # complex macros + if 'Obfuscated' in result.stderr: + # Obfuscated error - unifdef still produces output, so + # continue processing (don't return early) + pass + else: + # Real error + error_msg = (f'unifdef failed on {source_file} with return ' + f'code {result.returncode}\nstderr: ' + f'{result.stderr}') + return (source_file, 0, 0, 0, {}, error_msg) + + # Parse unifdef output to determine which lines are active + if track_lines: + line_status = match_lines(orig_lines, result.stdout, source_file) + active_lines = len([s for s in line_status.values() + if s == 'active']) + else: + line_status = {} + # Count non-#line directive lines in output + active_lines = len([line for line in result.stdout.splitlines() + if not line.startswith('#line')]) + inactive_lines = total_lines - active_lines + + return (source_file, total_lines, active_lines, inactive_lines, + line_status, None) + except (OSError, IOError) as e: + # Failed to execute unifdef or read source file + error_msg = f'Failed to process {source_file}: {e}' + return (source_file, 0, 0, 0, {}, error_msg) + + +class UnifdefAnalyser(Analyser): + """Analyser that uses unifdef to determine active lines. + + This analyser handles the creation of a unifdef configuration file from + CONFIG_* symbols and provides methods to analyse source files. + + Attributes: + config (dict): Dictionary of CONFIG_* symbols and their values + unifdef_cfg (str): Path to temporary unifdef configuration file + """ + + def __init__(self, config_file, srcdir, used_sources, unifdef_path, + include_headers, keep_temps=False): + """Set up the analyser with config file path. + + Args: + config_file (str): Path to .config file + srcdir (str): Path to source root directory + used_sources (set): Set of source files that are compiled + unifdef_path (str): Path to unifdef executable + include_headers (bool): If True, include header files; otherwise + only .c and .S + keep_temps (bool): If True, keep temporary files for debugging + """ + super().__init__(srcdir, keep_temps) + self.config_file = config_file + self.used_sources = used_sources + self.unifdef_path = unifdef_path + self.include_headers = include_headers + self.unifdef_cfg = None + + def _create_unifdef_config(self, config): + """Create a temporary unifdef configuration file. + + Args: + config (dict): Dictionary mapping CONFIG_* names to values + + Creates a file with -D and -U directives for each CONFIG_* symbol + that can be passed to unifdef via -f flag. + """ + # Create temporary file for unifdef directives + fd, self.unifdef_cfg = tempfile.mkstemp(prefix='unifdef_', + suffix='.cfg') + + with os.fdopen(fd, 'w') as f: + for name, value in sorted(config.items()): + if value is None or value == '' or value == 'n': + # Symbol is not set - undefine it + f.write(f'#undef {name}\n') + elif value is True or value == 'y': + # Boolean CONFIG - define it as 1 + f.write(f'#define {name} 1\n') + elif value == 'm': + # Module - treat as not set for U-Boot + f.write(f'#undef {name}\n') + elif (isinstance(value, str) and value.startswith('"') and + value.endswith('"')): + # String value with quotes - use as-is + f.write(f'#define {name} {value}\n') + else: + # Numeric or other value + try: + # Try to parse as integer + int_val = int(value, 0) + f.write(f'#define {name} {int_val}\n') + except (ValueError, TypeError): + # Not an integer - escape and quote it + escaped_value = (str(value).replace('\\', '\\\\') + .replace('"', '\\"')) + f.write(f'#define {name} "{escaped_value}"\n') + + def __del__(self): + """Clean up temporary unifdef config file""" + if self.unifdef_cfg and os.path.exists(self.unifdef_cfg): + # Keep the file if requested + if self.keep_temps: + tout.debug(f'Keeping unifdef config file: {self.unifdef_cfg}') + return + try: + os.unlink(self.unifdef_cfg) + except OSError: + pass + + def process(self, jobs=None): + """Perform line-level analysis on used source files. + + Args: + jobs (int): Number of parallel jobs (None = use all CPUs) + + Returns: + Dictionary mapping source files to analysis results, or None on + error + """ + # Validate config file exists + if not os.path.exists(self.config_file): + tout.error(f'Config file not found: {self.config_file}') + return None + + # Check if unifdef exists (check both absolute path and PATH) + if os.path.isabs(self.unifdef_path): + # Absolute path - check if it exists + if not os.path.exists(self.unifdef_path): + tout.fatal(f'unifdef not found at: {self.unifdef_path}') + else: + # Relative path or command name - check PATH + unifdef_full = shutil.which(self.unifdef_path) + if not unifdef_full: + tout.fatal(f'unifdef not found in PATH: {self.unifdef_path}') + self.unifdef_path = unifdef_full + + # Load configuration + tout.progress('Loading configuration...') + config, error = load_config(self.config_file, self.srcdir) + if error: + tout.fatal(error) + tout.progress(f'Loaded {len(config)} config symbols') + + # Create unifdef config file + self._create_unifdef_config(config) + + tout.progress('Analysing preprocessor conditionals...') + file_results = {} + + # Filter sources to only .c and .S files unless include_headers is set + used_sources = self.used_sources + if not self.include_headers: + filtered_sources = {s for s in used_sources + if s.endswith('.c') or s.endswith('.S')} + excluded_count = len(used_sources) - len(filtered_sources) + if excluded_count > 0: + tout.progress(f'Excluding {excluded_count} header files ' + + '(use -i to include them)') + used_sources = filtered_sources + + # Count lines in defs file + with open(self.unifdef_cfg, 'r', encoding='utf-8') as f: + defs_lines = len(f.readlines()) + + # Use multiprocessing for parallel unifdef execution + # Prepare arguments for parallel processing + source_list = sorted(used_sources) + worker_args = [(source_file, self.unifdef_cfg, self.unifdef_path, True) + for source_file in source_list] + + tout.progress(f'Running unifdef on {len(source_list)} files...') + start_time = time.time() + + # If jobs=1, run directly without multiprocessing for easier debugging + if jobs == 1: + results = [worker(args) for args in worker_args] + else: + with multiprocessing.Pool(processes=jobs) as pool: + results = list(pool.imap(worker, worker_args, chunksize=10)) + elapsed_time = time.time() - start_time + + # Convert results to file_results dict and calculate totals + # Check for errors first + total_source_lines = 0 + errors = [] + for (source_file, total_lines, active_lines, inactive_lines, + line_status, error_msg) in results: + if error_msg: + errors.append(error_msg) + else: + file_results[source_file] = FileResult( + total_lines=total_lines, + active_lines=active_lines, + inactive_lines=inactive_lines, + line_status=line_status + ) + total_source_lines += total_lines + + # Report any errors + if errors: + for error in errors: + tout.error(error) + tout.fatal(f'unifdef failed on {len(errors)} file(s)') + + kloc = total_source_lines // 1000 + tout.info(f'Analysed {len(file_results)} files ({kloc} kLOC, ' + + f'{defs_lines} defs) in {elapsed_time:.1f} seconds') + tout.info(f'Unifdef directives file: {self.unifdef_cfg}') + + # Clean up temporary unifdef config file (unless in debug mode) + if tout.verbose >= tout.DEBUG: + tout.debug(f'Keeping unifdef directives file: {self.unifdef_cfg}') + else: + try: + os.unlink(self.unifdef_cfg) + tout.debug(f'Cleaned up {self.unifdef_cfg}') + except OSError as e: + tout.debug(f'Failed to clean up {self.unifdef_cfg}: {e}') + + return file_results From patchwork Mon Nov 24 13:49:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 763 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992223; bh=4DNKxQT1B7g0BNITIIIqmUKRuRujKuL0FR1WfKykv4A=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=pWUCFIa8NNUgTYk6+K2KAVxr8engyaDh+v58cZ9apKy2fExaKrmDz0sxiJkQ4fUKt MA4mmPI8uwrBA+Vgmqc12uWpqlOB0ca5DYMqHWsFd3Q2cyIhOXXN73M3zfGs8fKYkT Qluc77g6uQ6iZcDMjq+mLZAKlmuJTTJZ8QTY7p7eFXm/tgiL9RcYRVkdKgQP/uOuM2 SRKWbOE1fbXQaGDoQitiCxkmRsswhxkSel18ZOAiscQ9k8AurucfQLmDdpzXs3Ih1o L0wZ7Aue4uOqkXoTie9cin/R/uSntWnQ2hG4SHwq5pI3ojwmB+TTt4K7Jz1fIUwHdu JsMH9FjgBstsg== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id B75C268785 for ; Mon, 24 Nov 2025 06:50:23 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id G1ZW0JhNQFCK for ; Mon, 24 Nov 2025 06:50:23 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992223; bh=4DNKxQT1B7g0BNITIIIqmUKRuRujKuL0FR1WfKykv4A=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=pWUCFIa8NNUgTYk6+K2KAVxr8engyaDh+v58cZ9apKy2fExaKrmDz0sxiJkQ4fUKt MA4mmPI8uwrBA+Vgmqc12uWpqlOB0ca5DYMqHWsFd3Q2cyIhOXXN73M3zfGs8fKYkT Qluc77g6uQ6iZcDMjq+mLZAKlmuJTTJZ8QTY7p7eFXm/tgiL9RcYRVkdKgQP/uOuM2 SRKWbOE1fbXQaGDoQitiCxkmRsswhxkSel18ZOAiscQ9k8AurucfQLmDdpzXs3Ih1o L0wZ7Aue4uOqkXoTie9cin/R/uSntWnQ2hG4SHwq5pI3ojwmB+TTt4K7Jz1fIUwHdu JsMH9FjgBstsg== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id A6E0C6874E for ; Mon, 24 Nov 2025 06:50:23 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992220; bh=5TfOBQYs6z3B3jLHu4TWixDD0IZZLURzCLSB3J879zU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=s54yLXEuBPALQZpJNaB2yNgN9V7+fpvPyVNSoSWTN8ZLrdYKLmfPif2HZa+bWCfrp rk76+Tc331RutQwldYmiryfAoItqflogbkiOy457yNHcXvIj1oA5fI/3zwz8agvRNR /d3keSYVSi49ZRwBuPPjkbzXRmnupgHmQTQpyXdeNsSjB7J+VFpyfkKpVlAF4Y7DgP /UorwofWFZEy5j6+5P9zXq2p6wR9ft+LkpJ6GtCiTRPDJQMiPBI4KAXMdS37tRxv7g DDfLBp6niNTfiemhUKOpW54WLp1e009eq7YD0GSfl3Afksm59D/9yO5YRElO9WUwOt 4RE9e0xLpX7CA== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id D3AAC68669; Mon, 24 Nov 2025 06:50:20 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id xy1LWNeHAW4Y; Mon, 24 Nov 2025 06:50:20 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992216; bh=klv673KDp4lQYeMfpwPZ5i2GPZtN/5k/gLHlYoRAEKU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Mm+0gyxQqHaN9Hk0BeOr97qk/EdtXNrNiaV48F71YRuK9mfjy3dYtQeR35CtEsJ1D QFlqI6g0zNOsQ1MBbNdRUC/xTf+WJ40+tKchz+PJLQzmzHlT0Ty3mLKeoW54d7Thmu oeR7AbOsTGpAQZbF7g0GOea2m3b//5KGbJoKaguA6rCe75I7bKUfeOjYh1kaEJuFZt XF5o4tv7M3LsO8XWlHNa8UxU2uzCBjfLOtTdc9/D3xIpkyreBuwKY3/21ACIwtK4IM u1G61Driqi0MD2bUnsa+K6lSDK3k0YuiTQA8iJha4JozdhPxgt+/aLvV6q4YVuaMsh 0KNYLAsC9zGlw== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id A578168688; Mon, 24 Nov 2025 06:50:16 -0700 (MST) From: Simon Glass To: U-Boot Concept Date: Mon, 24 Nov 2025 06:49:17 -0700 Message-ID: <20251124134932.1991031-7-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251124134932.1991031-1-sjg@u-boot.org> References: <20251124134932.1991031-1-sjg@u-boot.org> MIME-Version: 1.0 Message-ID-Hash: NOQXNK3664H22MQFMM7KGKNW7NKZIV44 X-Message-ID-Hash: NOQXNK3664H22MQFMM7KGKNW7NKZIV44 X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Simon Glass , Claude X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 6/9] codman: Provide an dwarf analyser List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass Add a way to do static preprocessor analysis using debug information from compiled code. This reads the DWARF tables to determin which lines produced code. Co-developed-by: Claude Signed-off-by: Simon Glass --- tools/codman/dwarf.py | 200 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 200 insertions(+) create mode 100644 tools/codman/dwarf.py diff --git a/tools/codman/dwarf.py b/tools/codman/dwarf.py new file mode 100644 index 00000000000..adceac9d20a --- /dev/null +++ b/tools/codman/dwarf.py @@ -0,0 +1,200 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright 2025 Canonical Ltd +# +"""DWARF debug info-based line-level analysis for source code. + +This module provides functionality to analyse which lines in source files +were compiled by extracting line information from DWARF debug data in +object files. +""" + +import multiprocessing +import os +import subprocess +from collections import defaultdict + +from u_boot_pylib import tout +from analyser import Analyser, FileResult + + +def worker(args): + """Extract line numbers from DWARF debug info in an object file. + + Uses readelf --debug-dump=decodedline to get the line table, then parses + section headers and line entries to determine which source lines were + compiled into the object. + + Args: + args (tuple): Tuple of (obj_path, build_dir, srcdir) + + Returns: + tuple: (source_lines_dict, error_msg) where source_lines_dict is a + mapping of source file paths to sets of line numbers, and + error_msg is None on success or an error string on failure + """ + obj_path, build_dir, srcdir = args + source_lines = defaultdict(set) + + # Get the directory of the .o file relative to build_dir + rel_to_build = os.path.relpath(obj_path, build_dir) + obj_dir = os.path.dirname(rel_to_build) + + # Use readelf to extract decoded line information + try: + result = subprocess.run( + ['readelf', '--debug-dump=decodedline', obj_path], + capture_output=True, text=True, check=False, + encoding='utf-8', errors='ignore') + if result.returncode != 0: + error_msg = (f'readelf failed on {obj_path} with return code ' + f'{result.returncode}\nstderr: {result.stderr}') + return (source_lines, error_msg) + + # Parse the output + # Format is: Section header with full path, then data lines + current_file = None + for line in result.stdout.splitlines(): + # Skip header lines and empty lines + if not line or line.startswith('Contents of') or \ + line.startswith('File name') or line.strip() == '' or \ + line.startswith(' '): + continue + + # Look for section headers with full path (e.g., '/path/to/file.c:') + if line.endswith(':'): + header_path = line.rstrip(':') + # Try to resolve the path + if os.path.isabs(header_path): + # Absolute path in DWARF + abs_path = os.path.realpath(header_path) + else: + # Relative path - try relative to srcdir and obj_dir + abs_path = os.path.realpath( + os.path.join(srcdir, obj_dir, header_path)) + if not os.path.exists(abs_path): + abs_path = os.path.realpath( + os.path.join(srcdir, header_path)) + + if os.path.exists(abs_path): + current_file = abs_path + continue + + # Parse data lines - use current_file from section header + if current_file: + parts = line.split() + if len(parts) >= 2: + try: + line_num = int(parts[1]) + # Skip special line numbers (like '-') + if line_num > 0: + source_lines[current_file].add(line_num) + except (ValueError, IndexError): + continue + except (OSError, subprocess.SubprocessError) as e: + error_msg = f'Failed to execute readelf on {obj_path}: {e}' + return (source_lines, error_msg) + + return (source_lines, None) + + +# pylint: disable=too-few-public-methods +class DwarfAnalyser(Analyser): + """Analyser that uses DWARF debug info to determine active lines. + + This analyser extracts line number information from DWARF debug data in + compiled object files to determine which source lines generated code. + """ + def __init__(self, build_dir, srcdir, used_sources, keep_temps=False): + """Initialise the DWARF analyser. + + Args: + build_dir (str): Build directory containing .o files + srcdir (str): Path to source root directory + used_sources (set): Set of source files that are compiled + keep_temps (bool): If True, keep temporary files for debugging + """ + super().__init__(srcdir, keep_temps) + self.build_dir = build_dir + self.used_sources = used_sources + + def extract_lines(self, jobs=None): + """Extract used line numbers from DWARF debug info in object files. + + Args: + jobs (int): Number of parallel jobs (None = use all CPUs) + + Returns: + dict: Mapping of source file paths to sets of line numbers that + generated code + """ + # Find all .o files + obj_files = self.find_object_files(self.build_dir) + + if not obj_files: + return defaultdict(set) + + # Prepare arguments for parallel processing + args_list = [(obj_path, self.build_dir, self.srcdir) + for obj_path in obj_files] + + # Process in parallel + num_jobs = jobs if jobs else multiprocessing.cpu_count() + with multiprocessing.Pool(num_jobs) as pool: + results = pool.map(worker, args_list) + + # Merge results from all workers and check for errors + source_lines = defaultdict(set) + errors = [] + for result_dict, error_msg in results: + if error_msg: + errors.append(error_msg) + else: + for source_file, lines in result_dict.items(): + source_lines[source_file].update(lines) + + # Report any errors + if errors: + for error in errors: + tout.error(error) + tout.fatal(f'readelf failed on {len(errors)} object file(s)') + + return source_lines + + def process(self, jobs=None): + """Perform line-level analysis using DWARF debug info. + + Args: + jobs (int): Number of parallel jobs (None = use all CPUs) + + Returns: + dict: Mapping of source file paths to FileResult named tuples + """ + tout.progress('Extracting DWARF line information...') + dwarf_line_map = self.extract_lines(jobs) + + file_results = {} + for source_file in self.used_sources: + abs_path = os.path.realpath(source_file) + used_lines = dwarf_line_map.get(abs_path, set()) + + # Count total lines in the file + total_lines = self.count_lines(abs_path) + + active_lines = len(used_lines) + inactive_lines = total_lines - active_lines + + # Create line status dict + line_status = {} + for i in range(1, total_lines + 1): + line_status[i] = 'active' if i in used_lines else 'inactive' + + file_results[abs_path] = FileResult( + total_lines=total_lines, + active_lines=active_lines, + inactive_lines=inactive_lines, + line_status=line_status + ) + + tout.info(f'Analysed {len(file_results)} files using DWARF debug info') + return file_results From patchwork Mon Nov 24 13:49:18 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 764 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992227; bh=1UEslwyAK+zv61NrYDsMX6eGtqIAQUoSrSGXNuqfGmM=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=pY0DwcznTeokd2uT7lTmlnkvGfPs15m7EjmnClz8Vs4wgZJlmwdLRWGfCp2Sj0qk1 0J10P5LqAQHE3g3eLCbgvzMBq7dcVM3ikpzZ8Ro4a72CHjeGpRrrw0/J2ghdn2Ei0l hJRL+E13xP5bvsNssdz+XL7EqeCgdz2spd2Lk9RQu1rcXzZ3ihumza4RWe/ht9D7GU flEQUO6XsrzLWzrLTX3ZVDj9BszHka38Odjgs6xI8hsSCEqqtlf/7h3cGyR2rb+skc swCkxSMDLjcd0fAVmP06m1Qb5Ukt6j2Gq5LRrTZRjRsOksxLs8fbNdybB9H4qzTbje DVowtsf8LgllA== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 312AB68688 for ; Mon, 24 Nov 2025 06:50:27 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id ngNF2rbQMGcI for ; Mon, 24 Nov 2025 06:50:27 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992227; bh=1UEslwyAK+zv61NrYDsMX6eGtqIAQUoSrSGXNuqfGmM=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=pY0DwcznTeokd2uT7lTmlnkvGfPs15m7EjmnClz8Vs4wgZJlmwdLRWGfCp2Sj0qk1 0J10P5LqAQHE3g3eLCbgvzMBq7dcVM3ikpzZ8Ro4a72CHjeGpRrrw0/J2ghdn2Ei0l hJRL+E13xP5bvsNssdz+XL7EqeCgdz2spd2Lk9RQu1rcXzZ3ihumza4RWe/ht9D7GU flEQUO6XsrzLWzrLTX3ZVDj9BszHka38Odjgs6xI8hsSCEqqtlf/7h3cGyR2rb+skc swCkxSMDLjcd0fAVmP06m1Qb5Ukt6j2Gq5LRrTZRjRsOksxLs8fbNdybB9H4qzTbje DVowtsf8LgllA== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 1F3FD6874E for ; Mon, 24 Nov 2025 06:50:27 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992226; bh=AFlZTeL0cIoRJ2o54HC2m5LT6+Hydm7i0baBEqOtSN8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cTDF7NhPMFXW0sKlxNP8r70oN4BchK9QDVmLofyDzM+FsbwoPvpWJ29FyUQko5gSv mybQBVH3qYW0Eeu4SQ4ZfoW76jHaUFQ7AEkX1fhAQOB7qrh3N+x4OV8mrTi+++sufo aRSthb7vQHFsFQpBhNuzaXZmH5lb9ZwlSywA5RRnnk37/nV4Bnb9XboTXXdutZMlbD 7ddIdai7pOz4QQckKp3pkJaHq2YD11MzeJosDhsiGBw2KMQf2+TSfo+6xkQSdoH+PT bIx9uckz0dDWgI6p6aH9Wv40WBRwwBv6fUqcbYn1gqtB60Ic9hMAI34VMyX8+LI/oF DdblJA9wUVpRA== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 298AD6874E; Mon, 24 Nov 2025 06:50:26 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id ePFR8zb1szAt; Mon, 24 Nov 2025 06:50:26 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992221; bh=trRKzXsIf/Ck+jW9OwSxLG1V2jMxf8FiARyQ/1r09mI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cd5FZAurt3KaQFcRMi4nQ8WuD2Ed6qQ11ljbiiY54zoysw62j+0qsG1lNLx6QAVJk 4WjFT0/gRN1dfiAbf4ZGfg08Kc9qIfNif457OBpRHIwKvp7GVITOFL8Rnb197fbN5S 2zrHOzUlkK5IMiAAy9yA762dvlUq55HCGp1+O+dhMmJFrxTY9WqYWX2X8GvD0iUsdg pCIg+S7Y3HbSBHUI3aEjydybVIYdtMd7mBPhS3hJDWDyG0MO7VqPIMrA79rPrSA/Iy nRnQqwth89SyDQNn67SK+CJt+7SDLX7/VAyh6Xht8TK58Rn2u+Vwp6I3es1XZe6vl5 GGAzWZA/R//4g== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id 26FBE68688; Mon, 24 Nov 2025 06:50:21 -0700 (MST) From: Simon Glass To: U-Boot Concept Date: Mon, 24 Nov 2025 06:49:18 -0700 Message-ID: <20251124134932.1991031-8-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251124134932.1991031-1-sjg@u-boot.org> References: <20251124134932.1991031-1-sjg@u-boot.org> MIME-Version: 1.0 Message-ID-Hash: PIIIZOS4D5DEZLBUFUCTWQKODEOLGIQ6 X-Message-ID-Hash: PIIIZOS4D5DEZLBUFUCTWQKODEOLGIQ6 X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Simon Glass , Claude X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 7/9] codman: Begin an experimental lsp analyser List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass It is possible to use an LSP to determine which code is used, at least to some degree. Make a start on this, in the hope that future work may prove out the concept. So far I have not found this to be particularly useful, since it does not seem to handle IS_ENABLED() and similar macros when working out inactive regions. Co-developed-by: Claude Signed-off-by: Simon Glass --- tools/codman/lsp.py | 319 +++++++++++++++++++++++++++++++++++++ tools/codman/lsp_client.py | 225 ++++++++++++++++++++++++++ tools/codman/test_lsp.py | 153 ++++++++++++++++++ 3 files changed, 697 insertions(+) create mode 100644 tools/codman/lsp.py create mode 100644 tools/codman/lsp_client.py create mode 100755 tools/codman/test_lsp.py diff --git a/tools/codman/lsp.py b/tools/codman/lsp.py new file mode 100644 index 00000000000..143fe22a7e1 --- /dev/null +++ b/tools/codman/lsp.py @@ -0,0 +1,319 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright 2025 Canonical Ltd +# +"""LSP-based line-level analysis for source code. + +This module provides functionality to analyse which lines in source files +are active vs inactive based on preprocessor conditionals, using clangd's +inactive regions feature via the Language Server Protocol (LSP). +""" + +import concurrent.futures +import json +import multiprocessing +import os +import re +import tempfile +import time + +from u_boot_pylib import tools, tout +from analyser import Analyser, FileResult +from lsp_client import LspClient + + +def create_compile_commands(build_dir, srcdir): + """Create compile_commands.json using gen_compile_commands.py. + + Args: + build_dir (str): Build directory path + srcdir (str): Source directory path + + Returns: + list: List of compile command entries + """ + # Use the same pattern as gen_compile_commands.py + line_pattern = re.compile( + r'^(saved)?cmd_[^ ]*\.o := (?P.* )' + r'(?P[^ ]*\.[cS]) *(;|$)') + + compile_commands = [] + + # Walk through build directory looking for .cmd files + filename_matcher = re.compile(r'^\..*\.cmd$') + exclude_dirs = ['.git', 'Documentation', 'include', 'tools'] + + for dirpath, dirnames, filenames in os.walk(build_dir, topdown=True): + # Prune unwanted directories + dirnames = [d for d in dirnames if d not in exclude_dirs] + + for filename in filenames: + if not filename_matcher.match(filename): + continue + + cmd_file = os.path.join(dirpath, filename) + try: + with open(cmd_file, 'rt', encoding='utf-8') as f: + result = line_pattern.match(f.readline()) + if result: + command_prefix = result.group('command_prefix') + file_path = result.group('file_path') + + # Clean up command prefix (handle escaped #) + prefix = command_prefix.replace(r'\#', '#').replace( + '$(pound)', '#') + + # Get absolute path to source file + abs_path = os.path.realpath( + os.path.join(srcdir, file_path)) + if os.path.exists(abs_path): + compile_commands.append({ + 'directory': srcdir, + 'file': abs_path, + 'command': prefix + file_path, + }) + except (OSError, IOError): + continue + + return compile_commands + + +def worker(args): + """Analyse a single source file using clangd LSP. + + Args: + args (tuple): Tuple of (source_file, client) + where client is a shared LspClient instance + + Returns: + tuple: (source_file, inactive_regions, error_msg) + """ + source_file, client = args + + try: + # Read file content + content = tools.read_file(source_file, binary=False) + + # Open the document + client.notify('textDocument/didOpen', { + 'textDocument': { + 'uri': f'file://{source_file}', + 'languageId': 'c', + 'version': 1, + 'text': content + } + }) + + # Wait for clangd to process and send notifications + # Poll for inactive regions notification for this specific file + max_wait = 10 # seconds + start_time = time.time() + inactive_regions = None + + while time.time() - start_time < max_wait: + time.sleep(0.1) + + with client.lock: + notifications = list(client.notifications) + # Clear processed notifications to avoid buildup + client.notifications = [] + + for notif in notifications: + method = notif.get('method', '') + if method == 'textDocument/clangd.inactiveRegions': + params = notif.get('params', {}) + uri = params.get('uri', '') + # Check if this notification is for our file + if uri == f'file://{source_file}': + inactive_regions = params.get('inactiveRegions', []) + break + + if inactive_regions is not None: + break + + # Close the document to free resources + client.notify('textDocument/didClose', { + 'textDocument': { + 'uri': f'file://{source_file}' + } + }) + + if inactive_regions is None: + # No inactive regions notification received + # This could mean the file has no inactive code + inactive_regions = [] + + return (source_file, inactive_regions, None) + + except Exception as e: + return (source_file, None, str(e)) + + +class LspAnalyser(Analyser): # pylint: disable=too-few-public-methods + """Analyser that uses clangd LSP to determine active lines. + + This analyser uses the Language Server Protocol (LSP) with clangd to + identify inactive preprocessor regions in source files. + """ + + def __init__(self, build_dir, srcdir, used_sources, keep_temps=False): + """Set up the LSP analyser. + + Args: + build_dir (str): Build directory containing .o and .cmd files + srcdir (str): Path to source root directory + used_sources (set): Set of source files that are compiled + keep_temps (bool): If True, keep temporary files for debugging + """ + super().__init__(srcdir, keep_temps) + self.build_dir = build_dir + self.used_sources = used_sources + + def extract_inactive_regions(self, jobs=None): + """Extract inactive regions from source files using clangd. + + Args: + jobs (int): Number of parallel jobs (None = use all CPUs) + + Returns: + dict: Mapping of source file paths to lists of inactive regions + """ + # Create compile commands database + tout.progress('Building compile commands database...') + compile_commands = create_compile_commands(self.build_dir, self.srcdir) + + # Filter to only .c and .S files that we need to analyse + filtered_files = [] + for cmd in compile_commands: + source_file = cmd['file'] + if source_file in self.used_sources: + if source_file.endswith('.c') or source_file.endswith('.S'): + filtered_files.append(source_file) + + tout.progress(f'Found {len(filtered_files)} source files to analyse') + + if not filtered_files: + return {} + + inactive = {} + errors = [] + + # Create a single clangd instance and use it for all files + with tempfile.TemporaryDirectory() as tmpdir: + # Write compile commands database + compile_db = os.path.join(tmpdir, 'compile_commands.json') + with open(compile_db, 'w', encoding='utf-8') as f: + json.dump(compile_commands, f) + + # Start a single clangd server + tout.progress('Starting clangd server...') + with LspClient(['clangd', '--log=error', + f'--compile-commands-dir={tmpdir}']) as client: + result = client.init(f'file://{self.srcdir}') + if not result: + tout.error('Failed to start clangd') + return {} + + # Determine number of workers + if jobs is None: + jobs = min(multiprocessing.cpu_count(), len(filtered_files)) + elif jobs <= 0: + jobs = 1 + + tout.progress(f'Processing files with {jobs} workers...') + + # Use ThreadPoolExecutor to process files in parallel + # (threads share the same clangd client) + with concurrent.futures.ThreadPoolExecutor( + max_workers=jobs) as executor: + # Submit all tasks + future_to_file = { + executor.submit(worker, (source_file, client)): + source_file + for source_file in filtered_files + } + + # Collect results as they complete + completed = 0 + for future in concurrent.futures.as_completed(future_to_file): + source_file = future_to_file[future] + completed += 1 + tout.progress( + f'Processing {completed}/{len(filtered_files)}: ' + + f'{os.path.basename(source_file)}...') + + try: + source_file_result, inactive_regions, error_msg = ( + future.result()) + + if error_msg: + errors.append(f'{source_file}: {error_msg}') + elif inactive_regions is not None: + inactive[source_file_result] = ( + inactive_regions) + except Exception as exc: + errors.append(f'{source_file}: {exc}') + + # Report any errors + if errors: + for error in errors[:10]: # Show first 10 errors + tout.error(error) + if len(errors) > 10: + tout.error(f'... and {len(errors) - 10} more errors') + tout.warning(f'Failed to analyse {len(errors)} file(s) with LSP') + + return inactive + + def process(self, jobs=None): + """Perform line-level analysis using clangd LSP. + + Args: + jobs (int): Number of parallel jobs (None = use all CPUs) + + Returns: + dict: Mapping of source file paths to FileResult named tuples + """ + tout.progress('Extracting inactive regions using clangd LSP...') + inactive_regions_map = self.extract_inactive_regions(jobs) + + file_results = {} + for source_file in self.used_sources: + # Only process .c and .S files + if not (source_file.endswith('.c') or source_file.endswith('.S')): + continue + + abs_path = os.path.realpath(source_file) + inactive_regions = inactive_regions_map.get(abs_path, []) + + # Count total lines in the file + total_lines = self.count_lines(abs_path) + + # Create line status dict + line_status = {} + # Set up all lines as active + for i in range(1, total_lines + 1): + line_status[i] = 'active' + + # Mark inactive lines based on regions + # LSP uses 0-indexed line numbers + for region in inactive_regions: + start_line = region['start']['line'] + 1 + end_line = region['end']['line'] + 1 + # Mark lines as inactive (inclusive range) + for line_num in range(start_line, end_line + 1): + if line_num <= total_lines: + line_status[line_num] = 'inactive' + + inactive_lines = len([s for s in line_status.values() + if s == 'inactive']) + active_lines = total_lines - inactive_lines + + file_results[abs_path] = FileResult( + total_lines=total_lines, + active_lines=active_lines, + inactive_lines=inactive_lines, + line_status=line_status + ) + + tout.info(f'Analysed {len(file_results)} files using clangd LSP') + return file_results diff --git a/tools/codman/lsp_client.py b/tools/codman/lsp_client.py new file mode 100644 index 00000000000..954879a651e --- /dev/null +++ b/tools/codman/lsp_client.py @@ -0,0 +1,225 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright 2025 Canonical Ltd +# +"""Minimal LSP (Language Server Protocol) client for clangd. + +This module provides a simple JSON-RPC 2.0 client for communicating with +LSP servers like clangd. It focuses on the specific functionality needed +for analyzing inactive preprocessor regions. +""" + +import json +import subprocess +import threading +from typing import Any, Dict, Optional + + +class LspClient: + """Minimal LSP client for JSON-RPC 2.0 communication. + + This client handles the basic LSP protocol communication over + stdin/stdout with a language server process. + + Attributes: + process: The language server subprocess + next_id: Counter for JSON-RPC request IDs + responses: Dict mapping request IDs to response data + lock: Thread lock for response dictionary + reader_thread: Background thread reading server responses + """ + + def __init__(self, server_command): + """Init the LSP client and start the server. + + Args: + server_command (list): Command to start the LSP server + (e.g., ['clangd', '--log=error']) + """ + self.process = subprocess.Popen( + server_command, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + bufsize=0 + ) + self.next_id = 1 + self.responses = {} + self.notifications = [] + self.lock = threading.Lock() + self.running = True + + # Start background thread to read responses + self.reader_thread = threading.Thread(target=self._read_responses) + self.reader_thread.daemon = True + self.reader_thread.start() + + def _read_responses(self): + """Background thread to read responses from the server""" + while self.running and self.process.poll() is None: + try: + # Read headers + headers = {} + while True: + line = self.process.stdout.readline() + if not line or line == '\r\n' or line == '\n': + break + if ':' in line: + key, value = line.split(':', 1) + headers[key.strip()] = value.strip() + + if 'Content-Length' not in headers: + continue + + # Read content + content_length = int(headers['Content-Length']) + content = self.process.stdout.read(content_length) + + if not content: + break + + # Parse JSON + message = json.loads(content) + + # Store response or notification + with self.lock: + if 'id' in message: + # Response to a request + self.responses[message['id']] = message + else: + # Notification from server + self.notifications.append(message) + + except (json.JSONDecodeError, ValueError): + continue + except Exception: + break + + def _send_message(self, message: Dict[str, Any]): + """Send a JSON-RPC message to the server. + + Args: + message: JSON-RPC message dictionary + """ + content = json.dumps(message) + headers = f'Content-Length: {len(content)}\r\n\r\n' + self.process.stdin.write(headers + content) + self.process.stdin.flush() + + def request(self, method: str, params: Optional[Dict] = None, + timeout: int = 30) -> Optional[Dict]: + """Send a JSON-RPC request and wait for response. + + Args: + method: LSP method name (e.g., 'initialize') + params: Method parameters dictionary + timeout: Timeout in seconds (default: 30) + + Returns: + Response dictionary, or None on timeout/error + """ + request_id = self.next_id + self.next_id += 1 + + message = { + 'jsonrpc': '2.0', + 'id': request_id, + 'method': method, + } + if params: + message['params'] = params + + self._send_message(message) + + # Wait for response + import time + start_time = time.time() + while time.time() - start_time < timeout: + with self.lock: + if request_id in self.responses: + response = self.responses.pop(request_id) + if 'result' in response: + return response['result'] + if 'error' in response: + raise RuntimeError( + f"LSP error: {response['error']}") + return response + time.sleep(0.01) + + return None + + def notify(self, method: str, params: Optional[Dict] = None): + """Send a JSON-RPC notification (no response expected). + + Args: + method: LSP method name + params: Method parameters dictionary + """ + message = { + 'jsonrpc': '2.0', + 'method': method, + } + if params: + message['params'] = params + + self._send_message(message) + + def init(self, root_uri: str, capabilities: Optional[Dict] = None) -> Dict: + """Send initialize request to the server. + + Args: + root_uri: Workspace root URI (e.g., 'file:///path/to/workspace') + capabilities: Client capabilities dict + + Returns: + Server capabilities from initialize response + """ + if capabilities is None: + capabilities = { + 'textDocument': { + 'semanticTokens': { + 'requests': { + 'full': True + } + }, + 'publishDiagnostics': {}, + 'inactiveRegions': { + 'refreshSupport': False + } + } + } + + result = self.request('initialize', { + 'processId': None, + 'rootUri': root_uri, + 'capabilities': capabilities + }) + + # Send initialized notification + self.notify('initialized', {}) + + return result + + def shutdown(self): + """Shutdown the language server""" + self.request('shutdown') + self.notify('exit') + self.running = False + if self.process: + self.process.wait(timeout=5) + # Close file descriptors to avoid ResourceWarnings + if self.process.stdin: + self.process.stdin.close() + if self.process.stdout: + self.process.stdout.close() + if self.process.stderr: + self.process.stderr.close() + + def __enter__(self): + """Context manager entry""" + return self + + def __exit__(self, exc_type, exc_val, exc_tb): + """Context manager exit - ensure cleanup""" + self.shutdown() diff --git a/tools/codman/test_lsp.py b/tools/codman/test_lsp.py new file mode 100755 index 00000000000..1070ce655fb --- /dev/null +++ b/tools/codman/test_lsp.py @@ -0,0 +1,153 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0+ +# +# Copyright 2025 Canonical Ltd +# +"""Test script for LSP client with clangd""" + +import json +import os +import sys +import tempfile +import time + +# Add parent directory to path +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from lsp_client import LspClient # pylint: disable=wrong-import-position + + +def test_clangd(): + """Test basic clangd functionality""" + # Create a temporary directory with a simple C file + with tempfile.TemporaryDirectory() as tmpdir: + # Create a C file with CONFIG-style inactive code + test_file = os.path.join(tmpdir, 'test.c') + with open(test_file, 'w', encoding='utf-8') as f: + f.write('''#include + +// Simulate U-Boot style CONFIG options +#define CONFIG_FEATURE_A 1 + +void always_compiled(void) +{ + printf("Always here\\n"); +} + +#ifdef CONFIG_FEATURE_A +void feature_a_code(void) +{ + printf("Feature A enabled\\n"); +} +#endif + +#ifdef CONFIG_FEATURE_B +void feature_b_code(void) +{ + printf("Feature B enabled (THIS SHOULD BE INACTIVE)\\n"); +} +#endif + +#if 0 +void disabled_debug_code(void) +{ + printf("Debug code (INACTIVE)\\n"); +} +#endif +''') + + # Create compile_commands.json + compile_commands = [ + { + 'directory': tmpdir, + 'command': f'gcc -c {test_file}', + 'file': test_file + } + ] + compile_db = os.path.join(tmpdir, 'compile_commands.json') + with open(compile_db, 'w', encoding='utf-8') as f: + json.dump(compile_commands, f) + + # Create .clangd config to enable inactive regions + clangd_config = os.path.join(tmpdir, '.clangd') + with open(clangd_config, 'w', encoding='utf-8') as f: + f.write('''InactiveRegions: + Opacity: 0.55 +''') + + print(f'Created test file: {test_file}') + print(f'Created compile DB: {compile_db}') + print(f'Created clangd config: {clangd_config}') + + # Start clangd + print('\\nStarting clangd...') + with LspClient(['clangd', '--log=error', + f'--compile-commands-dir={tmpdir}']) as client: + print('Initialising...') + result = client.init(f'file://{tmpdir}') + print(f'Server capabilities: {result.get("capabilities", {}).keys()}') + + # Open the document + print(f'\\nOpening document: {test_file}') + with open(test_file, 'r', encoding='utf-8') as f: + content = f.read() + + client.notify('textDocument/didOpen', { + 'textDocument': { + 'uri': f'file://{test_file}', + 'languageId': 'c', + 'version': 1, + 'text': content + } + }) + + # Wait for clangd to index the file + print('\\nWaiting for clangd to index file...') + time.sleep(3) + + # Check for inactive regions notification + print('\\nChecking for inactive regions notification...') + with client.lock: + notifications = list(client.notifications) + + print(f'Received {len(notifications)} notifications:') + inactive_regions = None + for notif in notifications: + method = notif.get('method', 'unknown') + print(f' - {method}') + + # Look for the clangd inactive regions extension + if method == 'textDocument/clangd.inactiveRegions': + params = notif.get('params', {}) + inactive_regions = params.get('inactiveRegions', []) + print(f' Found {len(inactive_regions)} inactive regions!') + + if inactive_regions: + print('\\nInactive regions:') + for region in inactive_regions: + start = region['start'] + end = region['end'] + start_line = start['line'] + 1 # LSP is 0-indexed + end_line = end['line'] + 1 + print(f' Lines {start_line}-{end_line}') + else: + print('\\nNo inactive regions received (feature may not be enabled)') + + # Also show the file with line numbers for reference + print('\\nFile contents:') + for i, line in enumerate(content.split('\\n'), 1): + print(f'{i:3}: {line}') + + print('\\nTest completed!') + + # Check clangd stderr for any errors + print('\\n=== Clangd stderr output ===') + stderr_output = client.process.stderr.read() + if stderr_output: + print(stderr_output[:1000]) + else: + print('(no stderr output)') + + +if __name__ == '__main__': + test_clangd() From patchwork Mon Nov 24 13:49:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 765 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992231; bh=BONWzZ6Z8jmh1OC1CpAt8XWn2tFix1K2CGeI9Sojy+o=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=UeAzDjpDAktHlO/wQqPzG+NggM69/uG5LamxJk/TfdsddrHc003JuGs6bH+7UE9LK G8forPO5K/pfDZbC8q6uAuzqfGReutQEZNO2FiHZUi4hPAe4LVteRnaFyIsCWCcgcE GoY44I3cHf8QQotr8PsBvqGp/HQL7cjAYmecyZx5xKdlSj8LAJ+xEQgn3WuKiF8IXb 3pJQNNjjHWBtlRe+oEELiPqpFD2S8wBzwslmVKFDwamAmLszomyMr4jtQ7Yh7UnVVp 91ckgYrRrinsliuaJaIN7zcCN5FtNcs+TJPey0xrstbYefN24ylxIb7GCtDJ/wg/ot Pz/Mj0q9/SIhw== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id F07FF68768 for ; Mon, 24 Nov 2025 06:50:31 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 8LasDlprMHzQ for ; Mon, 24 Nov 2025 06:50:31 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992231; bh=BONWzZ6Z8jmh1OC1CpAt8XWn2tFix1K2CGeI9Sojy+o=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=UeAzDjpDAktHlO/wQqPzG+NggM69/uG5LamxJk/TfdsddrHc003JuGs6bH+7UE9LK G8forPO5K/pfDZbC8q6uAuzqfGReutQEZNO2FiHZUi4hPAe4LVteRnaFyIsCWCcgcE GoY44I3cHf8QQotr8PsBvqGp/HQL7cjAYmecyZx5xKdlSj8LAJ+xEQgn3WuKiF8IXb 3pJQNNjjHWBtlRe+oEELiPqpFD2S8wBzwslmVKFDwamAmLszomyMr4jtQ7Yh7UnVVp 91ckgYrRrinsliuaJaIN7zcCN5FtNcs+TJPey0xrstbYefN24ylxIb7GCtDJ/wg/ot Pz/Mj0q9/SIhw== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id DC9DC6874E for ; Mon, 24 Nov 2025 06:50:31 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992229; bh=nSwZnGLNm4MxSMvzXZULhjHQFUofErl2N8gwRT+etSk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZSjCJ0qb1oMKoEGgkcPRcotzGW44CxAJe6go2mI2ABcZ3dgzqYyuqWtSprtEmZMmV saSMVrX+ViULB8hx9OLdGE6p45Pg81bgrpJ6/+2uJjQRGzN1Stob4vEG35eGSeF1k3 4OZSKoMeEvMl9Z1ZYhNwo8yMR5G+XChValwLsCWuSJAVI2ayuK2bpS1VZhDN83dvxO Oaq4UZL2NCERC6AVup8IkJ7j2rFih5yJeVkqIO0d40enRTHhPkSuNkq5Hgl8GLIRzF fkT/03jkGSiLn5LwxbBGYgrRGhMv+kS73pAcHV4wq3qKthI1uwU6Hj54le1Ffa95V0 emBsUPQjE9E9w== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id DD7626874E; Mon, 24 Nov 2025 06:50:29 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id 63r-Fu8mVp3H; Mon, 24 Nov 2025 06:50:29 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992225; bh=aB50dJZ0RzXWdrD2KVm1SBvY0Xb9MM7qOWLXN4hDYNQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UYrM5TsN8jsTx/RGc7Hy+z7SQNL6Ft9+bEyamgeHSOf7u1zmbUN4qfOYCmLCsN3OI +c2m3ov3a2RKG5W8Y2pvs6CLUrmqqYVl0KWDKttLTwsSSwU6arWrwZ9kHzedbKI3Gy A1v/N7ZkSmgPMHJNbTL9lVALaXTcOj9f1Cak1Ucw3uhnpLgsHjkxRmJ30zy9P1Lw3l NykMC/QPahJ8ahzwbYlk3FEUSzdgGZ5VU3z2oEwP09VHIeHKecHgeCTGAu8y4SmGRx Hds4oI8Lvg4iIFupG/1eBqxa/tIRvqXlraOj5Poa86HOUj3pLD8EbJt+NvecLFDDCN zvprDfOMFeHtw== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id B296C6873C; Mon, 24 Nov 2025 06:50:25 -0700 (MST) From: Simon Glass To: U-Boot Concept Date: Mon, 24 Nov 2025 06:49:19 -0700 Message-ID: <20251124134932.1991031-9-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251124134932.1991031-1-sjg@u-boot.org> References: <20251124134932.1991031-1-sjg@u-boot.org> MIME-Version: 1.0 Message-ID-Hash: 5AZIKBJQ22AU4JSORRQURL4UUBHCPEFX X-Message-ID-Hash: 5AZIKBJQ22AU4JSORRQURL4UUBHCPEFX X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Simon Glass X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 8/9] codman: Add some basic tests List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass Add some rudimentary tests of the codman functionality. Signed-off-by: Simon Glass --- tools/codman/test_codman.py | 470 ++++++++++++++++++++++++++++++++++++ 1 file changed, 470 insertions(+) create mode 100755 tools/codman/test_codman.py diff --git a/tools/codman/test_codman.py b/tools/codman/test_codman.py new file mode 100755 index 00000000000..ed387c82472 --- /dev/null +++ b/tools/codman/test_codman.py @@ -0,0 +1,470 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0+ +# +# Copyright 2025 Canonical Ltd +# +"""Very basic tests for codman.py script""" + +import os +import shutil +import subprocess +import sys +import tempfile +import unittest + +# Test configuration +SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) + +# Import the module to test +sys.path.insert(0, SCRIPT_DIR) +sys.path.insert(0, os.path.join(SCRIPT_DIR, '..')) +# pylint: disable=wrong-import-position +from u_boot_pylib import terminal, tools +import output # pylint: disable=wrong-import-position +import codman # pylint: disable=wrong-import-position + + +class TestSourceUsage(unittest.TestCase): + """Test cases for codman.py""" + + def setUp(self): + """Set up test environment with fake source tree and build""" + self.test_dir = tempfile.mkdtemp(prefix='test_source_usage_') + self.src_dir = os.path.join(self.test_dir, 'src') + self.build_dir = os.path.join(self.test_dir, 'build') + os.makedirs(self.src_dir) + os.makedirs(self.build_dir) + + # Create fake source files + self._create_fake_sources() + + # Create fake Makefile + self._create_makefile() + + # Create fake .config + self._create_config() + + def tearDown(self): + """Clean up test environment""" + if os.path.exists(self.test_dir): + shutil.rmtree(self.test_dir) + + def _create_fake_sources(self): + """Create a fake source tree with various files""" + # Create directory structure + dirs = [ + 'common', + 'drivers/video', + 'drivers/serial', + 'lib', + 'arch/sandbox', + ] + for dir_path in dirs: + os.makedirs(os.path.join(self.src_dir, dir_path), exist_ok=True) + + # Create source files + # common/main.c - will be compiled + self._write_file('common/main.c', '''#include + +void board_init(void) +{ +#ifdef CONFIG_FEATURE_A + feature_a_init(); +#endif +#ifdef CONFIG_FEATURE_B + feature_b_init(); +#endif + common_init(); +} +''') + + # common/unused.c - will NOT be compiled + self._write_file('common/unused.c', '''#include + +void unused_function(void) +{ + /* This file is never compiled */ +} +''') + + # drivers/video/display.c - will be compiled + self._write_file('drivers/video/display.c', '''#include + +#ifdef CONFIG_VIDEO_LOGO +static void show_logo(void) +{ + /* Show boot logo */ +} +#endif + +void display_init(void) +{ +#ifdef CONFIG_VIDEO_LOGO + show_logo(); +#endif + /* Init display */ +} +''') + + # drivers/serial/serial.c - will be compiled + self._write_file('drivers/serial/serial.c', '''#include + +void serial_init(void) +{ + /* Init serial port */ +} +''') + + # lib/string.c - will be compiled + self._write_file('lib/string.c', '''#include + +int strlen(const char *s) +{ + int len = 0; + while (*s++) + len++; + return len; +} +''') + + # arch/sandbox/cpu.c - will be compiled + self._write_file('arch/sandbox/cpu.c', '''#include + +void cpu_init(void) +{ + /* Sandbox CPU init */ +} +''') + + # Create header files + self._write_file('include/common.h', '''#ifndef __COMMON_H +#define __COMMON_H +void board_init(void); +#endif +''') + + self._write_file('include/video.h', '''#ifndef __VIDEO_H +#define __VIDEO_H +void display_init(void); +#endif +''') + + self._write_file('include/serial.h', '''#ifndef __SERIAL_H +#define __SERIAL_H +void serial_init(void); +#endif +''') + + self._write_file('include/linux/string.h', '''#ifndef __LINUX_STRING_H +#define __LINUX_STRING_H +int strlen(const char *s); +#endif +''') + + def _create_makefile(self): + """Create a simple Makefile that generates .cmd files""" + makefile = f'''# Simple test Makefile +SRCDIR := {self.src_dir} +O ?= . +BUILD_DIR = $(O) + +# Compiler flags +CFLAGS := -Iinclude +ifeq ($(DEBUG),1) +CFLAGS += -g +endif + +# Source files to compile +OBJS = $(BUILD_DIR)/common/main.o \\ + $(BUILD_DIR)/drivers/video/display.o \\ + $(BUILD_DIR)/drivers/serial/serial.o \\ + $(BUILD_DIR)/lib/string.o \\ + $(BUILD_DIR)/arch/sandbox/cpu.o + +all: $(OBJS) +\t@echo "Build complete" + +# Rule to compile .c files +$(BUILD_DIR)/%.o: %.c +\t@mkdir -p $(dir $@) +\t@echo " CC $<" +\t@gcc $(CFLAGS) -c -o $@ $(SRCDIR)/$< +\t@echo "cmd_$@ := gcc $(CFLAGS) -c -o $@ $<" > $(dir $@).$(notdir $@).cmd +\t@echo "source_$@ := $(SRCDIR)/$<" >> $(dir $@).$(notdir $@).cmd +\t@echo "deps_$@ := \\\\" >> $(dir $@).$(notdir $@).cmd +\t@echo " $(SRCDIR)/$< \\\\" >> $(dir $@).$(notdir $@).cmd +\t@echo "" >> $(dir $@).$(notdir $@).cmd + +clean: +\t@rm -rf $(BUILD_DIR) + +.PHONY: all clean +''' + self._write_file('Makefile', makefile) + + def _create_config(self): + """Create a fake .config file""" + config = '''CONFIG_FEATURE_A=y +# CONFIG_FEATURE_B is not set +CONFIG_VIDEO_LOGO=y +''' + self._write_file(os.path.join(self.build_dir, '.config'), config) + + def _write_file(self, rel_path, content): + """Write a file relative to src_dir""" + if rel_path.startswith('/'): + # Absolute path for build dir files + file_path = rel_path + else: + file_path = os.path.join(self.src_dir, rel_path) + os.makedirs(os.path.dirname(file_path), exist_ok=True) + tools.write_file(file_path, content.encode('utf-8')) + + def _build(self, debug=False): + """Run the test build. + + Args: + debug (bool): If True, build with debug symbols (DEBUG=1) + """ + cmd = ['make', '-C', self.src_dir, f'O={self.build_dir}'] + if debug: + cmd.append('DEBUG=1') + result = subprocess.run(cmd, capture_output=True, text=True, + check=False) + if result.returncode != 0: + print(f'Build failed: {result.stderr}') + print(f'Build stdout: {result.stdout}') + self.fail('Test build failed') + + def test_basic_file_stats(self): + """Test basic file-level statistics""" + self._build() + + # Call select_sources() directly + _all_srcs, used, skipped = codman.select_sources( + self.src_dir, self.build_dir, None) + + # Verify counts - we have 5 compiled .c files + self.assertEqual(len(used), 5, + f'Expected 5 used files, got {len(used)}') + + # Should have 1 unused .c file (common/unused.c) + unused_c_files = [f for f in skipped if f.endswith('.c')] + self.assertEqual(len(unused_c_files), 1, + f'Expected 1 unused .c file, got {len(unused_c_files)}') + + # Check that specific files are in used set + used_basenames = {os.path.basename(f) for f in used} + self.assertIn('main.c', used_basenames) + self.assertIn('display.c', used_basenames) + self.assertIn('serial.c', used_basenames) + self.assertIn('string.c', used_basenames) + self.assertIn('cpu.c', used_basenames) + + # Check that unused.c is not in used set + self.assertNotIn('unused.c', used_basenames) + + def test_list_unused(self): + """Test listing unused files""" + self._build() + + _all_srcs, _used, skipped = codman.select_sources( + self.src_dir, self.build_dir, None) + + # Check that unused.c is in skipped set + skipped_basenames = {os.path.basename(f) for f in skipped} + self.assertIn('unused.c', skipped_basenames) + + # Check that used files are not in skipped set + self.assertNotIn('main.c', skipped_basenames) + self.assertNotIn('display.c', skipped_basenames) + + def test_by_dir(self): + """Test directory breakdown by collecting stats""" + self._build() + + all_srcs, used, _skipped = codman.select_sources( + self.src_dir, self.build_dir, None) + + # Collect directory stats + dir_stats = output.collect_dir_stats( + all_srcs, used, None, self.src_dir, False, False) + + # Should have stats for top-level directories + self.assertIn('common', dir_stats) + self.assertIn('drivers', dir_stats) + self.assertIn('lib', dir_stats) + self.assertIn('arch', dir_stats) + + # Check common directory has 2 files (main.c and unused.c) + self.assertEqual(dir_stats['common'].total, 2) + # Only 1 is used (main.c) + self.assertEqual(dir_stats['common'].used, 1) + + def test_subdirs(self): + """Test subdirectory breakdown""" + self._build() + + all_srcs, used, _skipped = codman.select_sources( + self.src_dir, self.build_dir, None) + + # Collect subdirectory stats (by_subdirs=True) + dir_stats = output.collect_dir_stats( + all_srcs, used, None, self.src_dir, True, False) + + # Should have stats for subdirectories + self.assertIn('drivers/video', dir_stats) + self.assertIn('drivers/serial', dir_stats) + self.assertIn('arch/sandbox', dir_stats) + + def test_filter(self): + """Test filtering by pattern""" + self._build() + + # Apply video filter + all_srcs, _used, _skipped = codman.select_sources( + self.src_dir, self.build_dir, '*video*') + + # Should only have video-related files + all_basenames = {os.path.basename(f) for f in all_srcs} + self.assertIn('display.c', all_basenames) + self.assertIn('video.h', all_basenames) + + # Should not have non-video files + self.assertNotIn('main.c', all_basenames) + self.assertNotIn('serial.c', all_basenames) + + def test_no_build_required(self): + """Test that analysis works with existing build""" + self._build() + + # Should work without building + all_srcs, used, _skipped = codman.select_sources( + self.src_dir, self.build_dir, None) + + # Verify we got results + self.assertGreater(len(all_srcs), 0) + self.assertGreater(len(used), 0) + + def test_do_analysis_unifdef(self): + """Test do_analysis() with unifdef""" + self._build() + + _all_srcs, used, _skipped = codman.select_sources( + self.src_dir, self.build_dir, None) + + # Run unifdef analysis + unifdef_path = shutil.which('unifdef') or '/usr/bin/unifdef' + results = codman.do_analysis(used, self.build_dir, self.src_dir, + unifdef_path, include_headers=False, + jobs=1, use_lsp=False) + + # Should get results + self.assertIsNotNone(results) + self.assertGreater(len(results), 0) + + # Check that results have the expected structure + for _file_path, result in results.items(): + self.assertGreater(result.total_lines, 0) + self.assertGreaterEqual(result.active_lines, 0) + self.assertGreaterEqual(result.inactive_lines, 0) + self.assertEqual(result.total_lines, + result.active_lines + result.inactive_lines) + + def test_do_analysis_dwarf(self): + """Test do_analysis() with DWARF""" + # Build with debug symbols + self._build(debug=True) + + _all_srcs, used, _skipped = codman.select_sources( + self.src_dir, self.build_dir, None) + + # Run DWARF analysis (unifdef_path=None) + results = codman.do_analysis(used, self.build_dir, self.src_dir, + unifdef_path=None, include_headers=False, + jobs=1, use_lsp=False) + + # Should get results + self.assertIsNotNone(results) + self.assertGreater(len(results), 0) + + # Check that results have the expected structure + for _file_path, result in results.items(): + self.assertGreater(result.total_lines, 0) + self.assertGreaterEqual(result.active_lines, 0) + self.assertGreaterEqual(result.inactive_lines, 0) + self.assertEqual(result.total_lines, + result.active_lines + result.inactive_lines) + + def test_do_analysis_unifdef_missing_config(self): + """Test do_analysis() with unifdef when config file is missing""" + self._build() + + _all_srcs, used, _skipped = codman.select_sources( + self.src_dir, self.build_dir, None) + + # Remove .config file + config_file = os.path.join(self.build_dir, '.config') + if os.path.exists(config_file): + os.remove(config_file) + + # Capture terminal output + with terminal.capture() as (_stdout, stderr): + # Run unifdef analysis - should return None + unifdef_path = shutil.which('unifdef') or '/usr/bin/unifdef' + results = codman.do_analysis(used, self.build_dir, self.src_dir, + unifdef_path, + include_headers=False, jobs=1, + use_lsp=False) + + # Should return None when config is missing + self.assertIsNone(results) + + # Check that error message was printed to stderr + error_text = stderr.getvalue() + self.assertIn('Config file not found', error_text) + self.assertIn('.config', error_text) + + def test_do_analysis_lsp(self): + """Test do_analysis() with LSP (clangd)""" + # Disabled for now + self.skipTest('LSP test disabled') + # Check if clangd is available + if not shutil.which('clangd'): + self.skipTest('clangd not found - skipping LSP test') + + # Build with compile commands + self._build() + + _all_srcs, used, _skipped = codman.select_sources( + self.src_dir, self.build_dir, None) + + # Run LSP analysis (unifdef_path=None, use_lsp=True) + results = codman.do_analysis(used, self.build_dir, self.src_dir, + unifdef_path=None, include_headers=False, + jobs=1, use_lsp=True) + + # Should get results + self.assertIsNotNone(results) + self.assertGreater(len(results), 0) + + # Check that results have the expected structure + for _file_path, result in results.items(): + self.assertGreater(result.total_lines, 0) + self.assertGreaterEqual(result.active_lines, 0) + self.assertGreaterEqual(result.inactive_lines, 0) + self.assertEqual(result.total_lines, + result.active_lines + result.inactive_lines) + + # Check specific file results + main_file = os.path.join(self.src_dir, 'common/main.c') + if main_file in results: + result = results[main_file] + # main.c has some conditional code, so should have some lines + self.assertGreater(result.total_lines, 0) + # Should have identified some active lines + self.assertGreater(result.active_lines, 0) + + +if __name__ == '__main__': + unittest.main(argv=['test_codman.py'], verbosity=2) From patchwork Mon Nov 24 13:49:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 766 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992235; bh=9T4LOoWPgbwaRQvINnwG4i0pbW8Pl7VorSgWyvTT+Ek=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=lnGIH1/uV52JJfRYNTTaSPdY36wxc4Me1LRMTAtNVswtHrM6SblcWBSSYG89abpv0 aW5qjsLh9Z8UdYy1XkKmcBFfHlMWxC4JOyajVVD/jlmFjRo+TXMMYwYP6ObhKhp54J 0j1oJ94aCZD06bAlCnnDzpmooTwmD2B0/udxw1IiEBnMcSqM1jo7ioyUbVbtdbH1/Z G3e6W+ueqPpCb+c8A9titKSaKsArKNeNXfjPnuNAgcXs1j7ytNebMgAiO83CpvPprl w3N45LNe/hhdxXvPpljbtYHGYhV0T6FFflmkGhBpivfTVa28xNMzJuswLpikrmGtBP 0q6ktRKcDGA6g== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 68C2668751 for ; Mon, 24 Nov 2025 06:50:35 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id z3FUC_5zDhIQ for ; Mon, 24 Nov 2025 06:50:35 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992235; bh=9T4LOoWPgbwaRQvINnwG4i0pbW8Pl7VorSgWyvTT+Ek=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=lnGIH1/uV52JJfRYNTTaSPdY36wxc4Me1LRMTAtNVswtHrM6SblcWBSSYG89abpv0 aW5qjsLh9Z8UdYy1XkKmcBFfHlMWxC4JOyajVVD/jlmFjRo+TXMMYwYP6ObhKhp54J 0j1oJ94aCZD06bAlCnnDzpmooTwmD2B0/udxw1IiEBnMcSqM1jo7ioyUbVbtdbH1/Z G3e6W+ueqPpCb+c8A9titKSaKsArKNeNXfjPnuNAgcXs1j7ytNebMgAiO83CpvPprl w3N45LNe/hhdxXvPpljbtYHGYhV0T6FFflmkGhBpivfTVa28xNMzJuswLpikrmGtBP 0q6ktRKcDGA6g== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 57CE16873C for ; Mon, 24 Nov 2025 06:50:35 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992234; bh=0Tn1gyFb82SkEHO8Im8NKVSkTIC0dby/gRDjW6bB8gE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=O+LGRuKQCyjnEey6vew8+dsy+XLtgXAVhiMrxPTLf9WINkajm+B6j7HYqe4wIaBtn fQjgo+Kfcc6aAT7Z/Ov069tAkeUmPT4N7M7qYLaHX3iBn9ugAEdAS2uVSkcc6nIdCy YB3l4Rh3EASZYwUZqI/7cEi5V2n16AA1VVM5ZElP/Nrzev+eVT45wSwTOagJSJEoVI 0AYHI3zaPMkZvmE74xyGk/UrtfgyVwFzBbu6bPCU0hL9M6Wz7Ucjs9GEVxbCmqQKTU pqjax+LaEW98qZGY1JGqPU4ddyIqPOhqwJinRqXyeE0UMi1dI1JAB76iAaNE6q0+Xt ldd3FeIe+BU5Q== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 5DCC568688; Mon, 24 Nov 2025 06:50:34 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id tXx7H2nlHnZW; Mon, 24 Nov 2025 06:50:34 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1763992230; bh=55cUvIwa94h+a+eo9M0vSgxSGQIm4BG0pyqpabQFUP0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BzSItjcIMHmYIgq7BbvMNqxXCBemzORzGBgBTLcduFBss63UaSvh9Yi8u7emc1wQq 4LAL35aj0OPysI3sN8smyOyMoeUOK6KKRrup/FcCEjVghUhfDPHnXN+6QrmKsnM8yc 7l5Y/pl+lXNUnrYgUHc0dxB4DjIayDNF0eMyTBoQbIHxVJGSkQxuPhCV8ew3YWkBxx NgZ38ud54LA/b++3ElWEyxs8J7MPGDiirG7cXR93cvhhY7cqPTvKICS2K49qd+/9iE LCjNFiDX3/RerRcWR+pCsLSRKgwbgwYqQC8eYXy2fskVqplnG8gJPQqMaTlwnmlnvn qQVaxor87tryw== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id 2F68F6873C; Mon, 24 Nov 2025 06:50:30 -0700 (MST) From: Simon Glass To: U-Boot Concept Date: Mon, 24 Nov 2025 06:49:20 -0700 Message-ID: <20251124134932.1991031-10-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251124134932.1991031-1-sjg@u-boot.org> References: <20251124134932.1991031-1-sjg@u-boot.org> MIME-Version: 1.0 Message-ID-Hash: ARUS77DLHIXN64NZLQDSYE3CCRMSRDWA X-Message-ID-Hash: ARUS77DLHIXN64NZLQDSYE3CCRMSRDWA X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Simon Glass X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 9/9] codman: Add documentation List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass Provide a description of the purpose of codman and some examples of how to use it. Signed-off-by: Simon Glass --- doc/develop/codman.rst | 1 + doc/develop/index.rst | 1 + tools/codman/codman.rst | 426 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 428 insertions(+) create mode 120000 doc/develop/codman.rst create mode 100644 tools/codman/codman.rst diff --git a/doc/develop/codman.rst b/doc/develop/codman.rst new file mode 120000 index 00000000000..a4f5c03d72d --- /dev/null +++ b/doc/develop/codman.rst @@ -0,0 +1 @@ +../../tools/codman/codman.rst \ No newline at end of file diff --git a/doc/develop/index.rst b/doc/develop/index.rst index 1a8e0168c67..d325ad23897 100644 --- a/doc/develop/index.rst +++ b/doc/develop/index.rst @@ -101,6 +101,7 @@ Refactoring checkpatch coccinelle + codman qconfig Code quality diff --git a/tools/codman/codman.rst b/tools/codman/codman.rst new file mode 100644 index 00000000000..d58bceb2101 --- /dev/null +++ b/tools/codman/codman.rst @@ -0,0 +1,426 @@ +.. SPDX-License-Identifier: GPL-2.0+ + +=================== +Codman code manager +=================== + +The codman tool analyses U-Boot builds to determine which source files and lines +of code are actually compiled and used. + +U-Boot is a massive project with thousands of files and nearly endless +configuration possibilities. A single board configuration might only compile a +small fraction of the total source tree. Codman can help answer questions like: + +* "I just enabled ``CONFIG_CMD_NET``, how much code did that actually add?" +* "How much code would I remove by disabling ``CONFIG_CMDLINE``? + +Simply searching for ``CONFIG_`` macros or header inclusions is tricky because +the build logic takes many forms: Makefile rules, #ifdefs, IS_ENABLED(), +CONFIG_IS_ENABLED() and static inlines. The end result is board-specific in any +case. + +Codman cuts through this complexity by analysing the actual build artifacts +generated by the compiler: + +#. Builds the specified board +#. Parses the ``.cmd`` files to find which source file were compiled. +#. Analyses the source code (with unifdef) or the object files (dwarf tables) + to figure out which files and lines were compiled. + +Usage +===== + +Basic usage, from within the U-Boot source tree:: + + ./tools/codman/codman.py -b [flags] [command-flags] + +Codman operations does out-of-tree builds, meaning that the object files end up +in a separate directory for each board. Use ``--build-base`` to set that. The +default is ``/tmp/b`` meaning that a sandbox build would end up in +``/tmp/b/sandbox``, for eaxmple. + +Relationship to LSPs +==================== + +LSPs can allow you to see unused code in your IDE, which is very handy for +interactive use. Codman is more about getting a broader picture, although it +does allow individual files to be listed. Codman does include a ``--lsp`` option +but this doesn't work particularly well. + +Commands +======== + +The basic functionality is accessed via these commands: + +* ``stats`` - Show statistics (default if no command given) +* ``dirs`` - Show directory breakdown +* ``unused`` - List unused files +* ``used`` - List used files +* ``summary`` - Show per-file summary +* ``detail ...`` - Show line-by-line analysis of one or more files +* ``copy-used `` - Copy used source files to a directory + + +This will build the board and show statistics about source file usage. + +Adjusting Configuration (-a) +============================ + +Sometimes you want to explore "what if" scenarios without manually editing +``defconfig`` files or running menuconfig. The ``-a`` (or ``--adjust``) option +allows you to modify the Kconfig configuration on the fly before the analysis +build runs. + +This is particularly useful for **impact analysis**: seeing exactly how much +code a specific feature adds to the build. + +Syntax +------ + +The `CONFIG_` prefix is optional. + +* ``-a CONFIG_OPTION``: Enable a boolean option (sets to 'y'). +* ``-a ~CONFIG_OPTION``: Disable an option. +* ``-a OPTION=val``: Set an option (``CONFIG_OPTION``) to a specific value. +* ``-a CONFIG_A,CONFIG_B``: Set multiple options (comma-separated). + +Examples +-------- + +**Check the impact of USB:** + +Enable the USB subsystem on the sandbox board and see how the code stats change:: + + codman -b sandbox -a CMD_USB stats + +**Disable Networking:** +See what code remains active when networking is explicitly disabled:: + + codman -b sandbox -a ~NET,NO_NET stats + +**Multiple Adjustments:** +Enable USB and USB storage together:: + + codman -b sandbox -a CONFIG_CMD_USB -a CONFIG_USB_STORAGE stats + +Common Options +============== + +Building: + +* ``-b, --board `` - Board to build and analyse (default: sandbox, uses buildman) +* ``-B, --build-dir `` - Use existing build directory instead of building +* ``--build-base `` - Base directory for builds (default: /tmp/b) +* ``-n, --no-build`` - Skip building, use existing build directory +* ``-a, --adjust `` - Adjust CONFIG options (see section above) + +Line-level analysis: + +* ``-w, --dwarf`` - Use DWARF debug info (most accurate, requires rebuild) +* ``-i, --include-headers`` - Include header files in unifdef analysis + +Filtering: + +* ``-f, --filter `` - Filter files by wildcard pattern (e.g., + ``*acpi*``) + +Output control: + +* ``-v, --verbose`` - Show verbose output +* ``-D, --debug`` - Enable debug mode +* ``--top `` - (for ``stats`` command) Show top N files with most inactive + code (default: 20) + +The ``dirs command`` has a few extra options: + +* ``-s, --subdirs`` - Show a breakdown by subdirectory +* ``-f, --show-files`` - Show individual files within directories (with ``-s``) +* ``-e, --show-empty`` - Show directories with 0 lines used + +Other: + +* ``-j, --jobs `` - Number of parallel jobs for line analysis + +How to use commands +=================== + +The following commands show the different ways to use codman. Commands are +specified as positional arguments after the global options. + +Basic Statistics (``stats``) +----------------------------- + +Show overall statistics for sandbox build:: + + $ codman -b qemu-x86 stats + ====================================================================== + FILE-LEVEL STATISTICS + ====================================================================== + Total source files: 14114 + Used source files: 1046 (7.4%) + Unused source files: 13083 (92.7%) + + Total lines of code: 3646331 + Used lines of code: 192543 (5.3%) + Unused lines of code: 3453788 (94.7%) + ====================================================================== + + ====================================================================== + LINE-LEVEL STATISTICS (within compiled files) + ====================================================================== + Files analysed: 504 + Total lines in used files:209915 + Active lines: 192543 (91.7%) + Inactive lines: 17372 (8.3%) + ====================================================================== + + TOP 20 FILES WITH MOST INACTIVE CODE: + ---------------------------------------------------------------------- + 2621 inactive lines (56.6%) - drivers/mtd/spi/spi-nor-core.c + 669 inactive lines (46.7%) - cmd/mem.c + 594 inactive lines (45.8%) - cmd/nvedit.c + 579 inactive lines (89.5%) - drivers/mtd/spi/spi-nor-ids.c + 488 inactive lines (27.4%) - net/net.c + ... + + +Directory Breakdown (``dirs``) +------------------------------ + +See which top-level directories contribute code:: + + codman dirs + +Output shows breakdown by directory:: + + BREAKDOWN BY TOP-LEVEL DIRECTORY + ================================================================================= + Directory Files Used %Used %Code kLOC Used + --------------------------------------------------------------------------------- + arch 234 156 67 72 12.3 8.9 + board 123 45 37 25 5.6 1.4 + cmd 89 67 75 81 3.4 2.8 + common 156 134 86 88 8.9 7.8 + ... + +For detailed subdirectory breakdown:: + + codman dirs --subdirs + +With ``--show-files``, also shows individual files within each directory:: + + codman dirs --subdirs --show-files + +You can also specify a file filter:: + + codman -b qemu-x86 -f "*acpi*" dirs -sf + ======================================================================================= + BREAKDOWN BY TOP-LEVEL DIRECTORY + ======================================================================================= + Directory Files Used %Used %Code kLOC Used + --------------------------------------------------------------------------------------- + arch/x86/include/asm 5 2 40 36 0.6 0.2 + arch/x86/lib 5 1 20 6 1.2 0.1 + acpi.c 65 65 100.0 0 + cmd 1 1 100 100 0.2 0.2 + acpi.c 216 215 99.5 1 + drivers/qfw 1 1 100 93 0.3 0.3 + qfw_acpi.c 332 309 93.1 23 + include/acpi 5 4 80 91 3.3 3.0 + include/dm 1 1 100 100 0.4 0.4 + include/power 1 1 100 100 0.2 0.2 + lib/acpi 13 3 23 14 3.9 0.5 + acpi_writer.c 131 63 48.1 68 + acpi_extra.c 181 177 97.8 4 + acpi.c 304 304 100.0 0 + lib/efi_loader 1 1 100 100 0.1 0.1 + efi_acpi.c 75 75 100.0 0 + --------------------------------------------------------------------------------------- + TOTAL 78 15 19 7 17.5 1.2 + ======================================================================================= + + +Detail View (``detail``) +------------------------ + +See exactly which lines are active/inactive in a specific file:: + + $ codman -b qemu-x86 detail common/main.c + ====================================================================== + DETAIL FOR: common/main.c + ====================================================================== + Total lines: 115 + Active lines: 93 (80.9%) + Inactive lines: 22 (19.1%) + + 1 | // SPDX-License-Identifier: GPL-2.0+ + 2 | /* + 3 | * (C) Copyright 2000 + 4 | * Wolfgang Denk, DENX Software Engineering, wd@denx.de. + 5 | */ + ... + 23 | + 24 | static void run_preboot_environment_command(void) + 25 | { + 26 | char *p; + 27 | + 28 | p = env_get("preboot"); + 29 | if (p != NULL) { + 30 | int prev = 0; + 31 | + - 32 | if (IS_ENABLED(CONFIG_AUTOBOOT_KEYED)) + - 33 | prev = disable_ctrlc(1); /* disable Ctrl-C checking */ + 34 | + 35 | run_command_list(p, -1, 0); + 36 | + - 37 | if (IS_ENABLED(CONFIG_AUTOBOOT_KEYED)) + - 38 | disable_ctrlc(prev); /* restore Ctrl-C checking */ + 39 | } + 40 | } + 41 | + + +Lines with a ``-`` marker are not included in the build. + +Unused Files (``unused``) +------------------------- + +Find all source files that weren't compiled:: + + $ codman -b qemu-x86 unused |head -15 + Finding all source files...... + Found 1043 used source files... + Loading configuration...... + Loaded 8913 Kconfig symbols... + Loaded 8913 config symbols... + Analysing preprocessor conditionals...... + Excluding 539 header files (use -i to include them)... + Running unifdef on 504 files...... + Unused source files (13083): + arch/arc/cpu/arcv1/ivt.S + arch/arc/cpu/arcv2/ivt.S + arch/arc/include/asm/arc-bcr.h + +Used Files (``used``) +--------------------- + +List all source files that were included in a build:: + + $ codman -b qemu-x86 used |head -15 + Finding all source files...... + Found 1046 used source files... + Loading configuration...... + Loaded 8913 Kconfig symbols... + Loaded 8913 config symbols... + Analysing preprocessor conditionals...... + Excluding 542 header files (use -i to include them)... + Running unifdef on 504 files...... + Used source files (1046): + arch/x86/cpu/call32.S + arch/x86/cpu/cpu.c + ... + + +Per-File Summary (``summary``) +------------------------------ + +Shows detailed per-file statistics (requires ``-w`` or ``-l``):: + + $ codman -b qemu-x86 summary + ========================================================================================== + PER-FILE SUMMARY + ========================================================================================== + File Total Active Inactive %Active + ------------------------------------------------------------------------------------------ + arch/x86/cpu/call32.S 61 61 0 100.0% + arch/x86/cpu/cpu.c 399 353 46 88.5% + arch/x86/cpu/cpu_x86.c 99 99 0 100.0% + arch/x86/cpu/i386/call64.S 92 92 0 100.0% + arch/x86/cpu/i386/cpu.c 649 630 19 97.1% + arch/x86/cpu/i386/interrupt.c 630 622 8 98.7% + arch/x86/cpu/i386/setjmp.S 65 65 0 100.0% + arch/x86/cpu/intel_common/cpu.c 325 325 0 100.0% + ... + +Copy Used Files (``copy-used``) +------------------------------- + +Extract only the source files used in a build:: + + codman copy-used /tmp/sandbox-sources + +This creates a directory tree with only the compiled files, useful for creating +minimal source distributions. + +Analysis Methods +================ + +The script supports several analysis methods with different trade-offs. + +Firstly, files are detected by looking for .cmd files in the build. This +requires a build to be present. Given the complexity of the Makefile rules, it +seems like a reasonable trade-off. These directories are excluded: + +* tools/ +* test/ +* scripts/ +* doc/ + +unifdef +------- + +For discovering used/unused code, the unifdef mechanism produces reasonable +results. This simulates the C preprocessor using the ``unifdef`` tool to +determine which lines are active based on CONFIG_* settings. + +**Note:** This requires a patched version of unifdef that supports U-Boot's +``IS_ENABLED()`` and ``CONFIG_IS_ENABLED()`` macros, which are commonly used +throughout the codebase. It also supports faster operation, reducing run time +by about 100x on the U-Boot code base. + +The tools: + +1. Reads .config to extract all CONFIG_* symbol definitions +2. Generates a unifdef configuration file with -D/-U directives +3. Runs ``unifdef -k -E`` on each source file to process conditionals, with + ``-E`` enabling the IS_ENABLED() support +4. Compares original vs. processed output using line-number information +5. Lines removed by unifdef are marked as inactive + +This method Uses multiprocessing for parallel analysis of source files, so it +runs faster if you have plenty of CPU cores (e.g. 3s on a 22-thread +Intel Ultra 7). + +The preprocessor-level view is quite helpful. It is also possible to see .h +files using the ``-i`` flag + +Since unifdef does fairly simplistic parsing it can be fooled and show wrong +results. + + +DWARF (``-w/--dwarf``) +---------------------- + +The DWARF analyser uses debug information embedded in compiled object files to +determine exactly which source lines generated machine code. This is arguably +more accurate than unifdef, but it won't count comments, declarations and +various other features that don't actually generate code. + +The DWARF analyser: + +1. Rebuilds with ``CC_OPTIMIZE_FOR_DEBUG`` to prevent aggressive inlining +2. For each .o file, runs ``readelf --debug-dump=decodedline`` to get line info +3. Parses the DWARF line number table to map source lines to code addresses +4. Aggregates results across all object files +5. Any source line that doesn't appear in the line table is marked inactive + +As with unifdef, this uses multiprocessing for parallel analysis of object +files. It achieves similar performance. + + +See Also +======== + +* :doc:`../build/buildman` - Tool for building multiple boards +* :doc:`qconfig` +* :doc:`checkpatch` - Code-style checking tool