From patchwork Wed Dec 17 02:27:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 948 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1765938547; bh=QWxIqUF/40dYM++oilT3wdnPLKOkBrXFLK7grB6LSX8=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=gifHg0yqmEKhgy0YQv5yeOicdWSvdSmmjeHohyONmHtMGCXxd7jdy6USJB9IOQ8+r nmoEoefqdloCcmRhtczMwmbzkZeu1g26qiuwA1bTK5CnesT+JfsK7PvZ0OTPTozAOF i2e0RGqZlcO/ccZuxskSSksJbfamdL69rAKnGAdtrWhUB4ct4XPnZs/SHx3J+Dv6S1 jza7abBli8g5uc0fJqUTrilGtWOeLZot/G2SVsCW+Kdfec9E+NDFOHJSxF/WrkX4FI PrchlUM9XK5E2ygJlowvrMDeWuBWwZJkRi0Da9wlxSOoGFv2/vJjxr4Y0cgwJlzh+m bJ1F/CPKoZ4oQ== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id A7F0E68ABD for ; Tue, 16 Dec 2025 19:29:07 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id IUTKgiQu6c51 for ; Tue, 16 Dec 2025 19:29:07 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1765938547; bh=QWxIqUF/40dYM++oilT3wdnPLKOkBrXFLK7grB6LSX8=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=gifHg0yqmEKhgy0YQv5yeOicdWSvdSmmjeHohyONmHtMGCXxd7jdy6USJB9IOQ8+r nmoEoefqdloCcmRhtczMwmbzkZeu1g26qiuwA1bTK5CnesT+JfsK7PvZ0OTPTozAOF i2e0RGqZlcO/ccZuxskSSksJbfamdL69rAKnGAdtrWhUB4ct4XPnZs/SHx3J+Dv6S1 jza7abBli8g5uc0fJqUTrilGtWOeLZot/G2SVsCW+Kdfec9E+NDFOHJSxF/WrkX4FI PrchlUM9XK5E2ygJlowvrMDeWuBWwZJkRi0Da9wlxSOoGFv2/vJjxr4Y0cgwJlzh+m bJ1F/CPKoZ4oQ== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 94E9C68AFD for ; Tue, 16 Dec 2025 19:29:07 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1765938546; bh=2fk594I7H/0xARxiIa22A4PHK5nt+LQM86RA3O8/2RA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZBQ0XPPbqUnNZKhrzeymON6jBwPmoHiHU2qunEVISdlw5u1cLZ6fn7e96bfWiIh97 7IuZc4P0V0XWuyKn2uBKcM2eRM+Q48R/jv4uaJuoGRC984ejZeCajdI77lb5DVbYBP TV/5ogq+/O2Yfxs4yepkxE9C4+KtE6umc+c9B/pDWYOaEBrInm7B7srwNSoyPCX+t2 la62Tf/umwNN4rvtaAr3exmnN5yQiCfbgGyTjRLgYk7APzagiKnVwQOZYPShsh6XLj +l4ieK3OZVRk3m5NgLyaaW+E7wv0jKJYg0drcOlsszfo9kTg03jCluATPJlL3hlcEp LCiBrnsIhck3A== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id 481E668ABD; Tue, 16 Dec 2025 19:29:06 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id N8gA1_yiyThn; Tue, 16 Dec 2025 19:29:06 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1765938542; bh=RjoAWzxGANKy0Fn5gF7QLDShn+w1g1HFbg8ozbs2hW4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BIvHPuA31g031U3N2vlxgERjEJ3+LmQ8z0PT7Z3i0wxurxKXMgpb27Etq3JRIA9jX lqGhROP7fZQrYnpcvgYFtzv7F0Go3cYJe+MQLbiORPZSNZeItFZAnn57LzmQy+8iLm LB+/VT4EIxkEsQ4j2GW2pLi9p0g/jSnQ8GKqgjtaR98Ktg3qDhDBAO4pJHPeMlmcjm sHQZPFZU5nNIBzUUFlT7+HbdOciJ3j90XHIr49PXcC/wZryVDJm0vM3GlGKYOWOvnp VOYOFnbQYBJKeBDofI1BwGjb04cJBrNuKixOxrZJc4L+AUgkRHX+69A6215lWWSizA LnU+W9Q2vgq/A== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id C8C686884F; Tue, 16 Dec 2025 19:29:01 -0700 (MST) From: Simon Glass To: U-Boot Concept Date: Tue, 16 Dec 2025 19:27:54 -0700 Message-ID: <20251217022823.392557-6-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251217022823.392557-1-sjg@u-boot.org> References: <20251217022823.392557-1-sjg@u-boot.org> MIME-Version: 1.0 Message-ID-Hash: UQ7GNRXA6MBUQS5SCDCTQHMNWZE34KXU X-Message-ID-Hash: UQ7GNRXA6MBUQS5SCDCTQHMNWZE34KXU X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Simon Glass , "Claude Opus 4 . 5" X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 05/24] pickman: Add test coverage support List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass Add support for running pickman tests with code coverage checking, similar to binman. Use test_util.run_test_suites() for running tests and test_util.run_test_coverage() for coverage checking. Options added to the 'test' command: -P: Number of processes for parallel test execution -T: Run with coverage checking -v: Verbosity level (0-4) The coverage check allows failures for modules that require external services (agent.py, gitlab_api.py, control.py, database.py) since these cannot easily achieve 100% coverage in unit tests. Also update test_util.py to recognize 'pickman' as using the 'test' subcommand like binman and patman. Co-developed-by: Claude Opus 4.5 Signed-off-by: Simon Glass --- tools/pickman/__main__.py | 67 ++++++++++++++++++++++++++++++++- tools/u_boot_pylib/test_util.py | 8 ++-- 2 files changed, 70 insertions(+), 5 deletions(-) diff --git a/tools/pickman/__main__.py b/tools/pickman/__main__.py index 3d311264f06..2d2366c1b80 100755 --- a/tools/pickman/__main__.py +++ b/tools/pickman/__main__.py @@ -9,6 +9,7 @@ import argparse import os import sys +import unittest # Allow 'from pickman import xxx' to work via symlink our_path = os.path.dirname(os.path.realpath(__file__)) @@ -16,6 +17,8 @@ sys.path.insert(0, os.path.join(our_path, '..')) # pylint: disable=wrong-import-position,import-error from pickman import control +from pickman import ftest +from u_boot_pylib import test_util def parse_args(argv): @@ -80,11 +83,65 @@ def parse_args(argv): poll_cmd.add_argument('-t', '--target', default='master', help='Target branch for MR (default: master)') - subparsers.add_parser('test', help='Run tests') + test_cmd = subparsers.add_parser('test', help='Run tests') + test_cmd.add_argument('-P', '--processes', type=int, + help='Number of processes to run tests (default: all)') + test_cmd.add_argument('-T', '--test-coverage', action='store_true', + help='Run tests and check for 100%% coverage') + test_cmd.add_argument('-v', '--verbosity', type=int, default=1, + help='Verbosity level (0-4, default: 1)') + test_cmd.add_argument('tests', nargs='*', help='Specific tests to run') return parser.parse_args(argv) +def get_test_classes(): + """Get all test classes from the ftest module. + + Returns: + list: List of test class objects + """ + return [getattr(ftest, name) for name in dir(ftest) + if name.startswith('Test') and + isinstance(getattr(ftest, name), type) and + issubclass(getattr(ftest, name), unittest.TestCase)] + + +def run_tests(processes, verbosity, test_name): + """Run the pickman test suite. + + Args: + processes (int): Number of processes for concurrent tests + verbosity (int): Verbosity level (0-4) + test_name (str): Specific test to run, or None for all + + Returns: + int: 0 if tests passed, 1 otherwise + """ + result = test_util.run_test_suites( + 'pickman', False, verbosity, False, False, processes, + test_name, None, get_test_classes()) + + return 0 if result.wasSuccessful() else 1 + + +def run_test_coverage(args): + """Run tests with coverage checking. + + Args: + args (list): Specific tests to run, or None for all + """ + # agent.py and gitlab_api.py require external services (Claude, GitLab) + # so they can't achieve 100% coverage in unit tests + test_util.run_test_coverage( + 'tools/pickman/pickman', None, + ['*test*', '*__main__.py', 'tools/u_boot_pylib/*'], + None, extra_args=None, args=args, + allow_failures=['tools/pickman/agent.py', + 'tools/pickman/gitlab_api.py', + 'tools/pickman/control.py']) + + def main(argv=None): """Main function to parse args and run commands. @@ -92,6 +149,14 @@ def main(argv=None): argv (list): Command line arguments (None for sys.argv[1:]) """ args = parse_args(argv) + + if args.cmd == 'test': + if args.test_coverage: + run_test_coverage(args.tests or None) + return 0 + test_name = args.tests[0] if args.tests else None + return run_tests(args.processes, args.verbosity, test_name) + return control.do_pickman(args) diff --git a/tools/u_boot_pylib/test_util.py b/tools/u_boot_pylib/test_util.py index 7bd12705557..b1c8740d883 100644 --- a/tools/u_boot_pylib/test_util.py +++ b/tools/u_boot_pylib/test_util.py @@ -57,7 +57,8 @@ def run_test_coverage(prog, filter_fname, exclude_list, build_dir, glob_list += exclude_list glob_list += ['*libfdt.py', '*/site-packages/*', '*/dist-packages/*'] glob_list += ['*concurrencytest*'] - test_cmd = 'test' if 'binman' in prog or 'patman' in prog else '-t' + use_test = 'binman' in prog or 'patman' in prog or 'pickman' in prog + test_cmd = 'test' if use_test else '-t' prefix = '' if build_dir: prefix = 'PYTHONPATH=$PYTHONPATH:%s/sandbox_spl/tools ' % build_dir @@ -91,13 +92,12 @@ def run_test_coverage(prog, filter_fname, exclude_list, build_dir, print(coverage) if coverage != '100%': print(stdout) - print("To get a report in 'htmlcov/index.html', type: python3-coverage html") + print("To get a report in 'htmlcov/index.html', type: " + "python3-coverage html") print('Coverage error: %s, but should be 100%%' % coverage) ok = False if not ok: if allow_failures: - # for line in lines: - # print('.', line, re.match(r'^(tools/.*py) *\d+ *(\d+) *(\d+)%$', line)) lines = [re.match(r'^(tools/.*py) *\d+ *(\d+) *\d+%$', line) for line in stdout.splitlines()] bad = []