From patchwork Mon Dec 29 16:06:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1105 Return-Path: X-Original-To: u-boot-concept@u-boot.org Delivered-To: u-boot-concept@u-boot.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1767024413; bh=JiAv5qqx5vn84wpf+Zx0eUNS3CvWihDBm2CwBj/NNOg=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=jUkkngwtk3MOz84lnZx3STnO4+7ONCI3nR6xQGApBJ8i32v+n73/WC7BL8MRpNKAT PvBE8mtaMrg19vIkJt3yTpjfSFDC9k+d0hIzsk3SB+W5RJLUv6uffHaWit7zafnB+X E+moNSY84Vt2/rzB/7BadxJsaFrskoPsU439IhpN65ex8Y7Tybfnr8Perqf1Nk6rRh BdoavGrO5ohmwknnEG7VMJXokvkBR1sHH1ZXLUnDBlcjb248sv74iecL2JWPAOR294 8rY2QkOjQX7+/g4oCzBP2vmfLHdhynBiWv1CvASj2UjjD7Nz9RXj23EGx6TpxLZI/C vk/tofDdcC1dQ== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id CA60568CAD for ; Mon, 29 Dec 2025 09:06:53 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id v_E0z5BjTqJe for ; Mon, 29 Dec 2025 09:06:53 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1767024413; bh=JiAv5qqx5vn84wpf+Zx0eUNS3CvWihDBm2CwBj/NNOg=; h=From:To:Date:In-Reply-To:References:CC:Subject:List-Id: List-Archive:List-Help:List-Owner:List-Post:List-Subscribe: List-Unsubscribe:From; b=jUkkngwtk3MOz84lnZx3STnO4+7ONCI3nR6xQGApBJ8i32v+n73/WC7BL8MRpNKAT PvBE8mtaMrg19vIkJt3yTpjfSFDC9k+d0hIzsk3SB+W5RJLUv6uffHaWit7zafnB+X E+moNSY84Vt2/rzB/7BadxJsaFrskoPsU439IhpN65ex8Y7Tybfnr8Perqf1Nk6rRh BdoavGrO5ohmwknnEG7VMJXokvkBR1sHH1ZXLUnDBlcjb248sv74iecL2JWPAOR294 8rY2QkOjQX7+/g4oCzBP2vmfLHdhynBiWv1CvASj2UjjD7Nz9RXj23EGx6TpxLZI/C vk/tofDdcC1dQ== Received: from mail.u-boot.org (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id B8B5D68F2D for ; Mon, 29 Dec 2025 09:06:53 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1767024411; bh=HR8n6FR/q5pkNytm4grZ1iLj3SQoDyyJzjxBRa36lX0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RviJJEnh8lOV8gIPvNIF8WNaDcj3nB22gKtqoMCIzKPwjhz5T5Nl4uElsqsiP/LpL oM7c5og0+YHVsU0y5geTtVchJ36qXASNvNcGS1drrsZzodE4teMnMN019crhdwelz0 sEnoHqMF1T2e0/tqyzdSWcYi6DZMS+9Pn0X0cTNqWQo31vynnoLCg0P7AhbkcgWZxH HIbqQ7XuWf3xfaPn/LUVgdZTszXThKFAVXbvbyvTAlACUSXXBpi4AXVTW2Ud7swOd5 i5KVs3zVFM5uIHzWMSmgoTCzdp8/ZU9HNc8ktpq4VMakVG6C9il4916vl8N2JaXfs3 BpdlIAjbJRd1Q== Received: from localhost (localhost [127.0.0.1]) by mail.u-boot.org (Postfix) with ESMTP id A18B668CAD; Mon, 29 Dec 2025 09:06:51 -0700 (MST) X-Virus-Scanned: Debian amavis at Received: from mail.u-boot.org ([127.0.0.1]) by localhost (mail.u-boot.org [127.0.0.1]) (amavis, port 10026) with ESMTP id fIlXqkcTOJA9; Mon, 29 Dec 2025 09:06:51 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=u-boot.org; s=default; t=1767024406; bh=CpQ3XVHnq4WAxXRWS4KkiN8oCLiO8GoTP8yP4Ee18bw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Pmfg10dTJktUsmkk7q3gw1DKwtsRbdbiWfLdMoG+XTSkund7ek1f4BaOdcEEIrJNO rGe1qJGh774W1LmRMtBRwtIdhaiPQuBvmWE115QLrqMyy4j4UjwDF/kWXk7o/+nI+Z taRux1+qJdpcXefMHM0daZ6pWdnZMXWCldbRgGQG/6sYAzIikgKIAznbxu7PfLVmtK ky5VOV9ADHzWf43cgSy5Q59oumhsTzyytTSZArqeIFtyAz8CqnPeB3/rgmgOV10eIO it44k+sl3uY0zZ7WA/fPdCj576Nt2ffwGgJQAwdTgd4EewrosPtDDy1ceAXw6vbjPr g+glf9Vf6hldw== Received: from u-boot.org (unknown [73.34.74.121]) by mail.u-boot.org (Postfix) with ESMTPSA id 64C2F64E2A; Mon, 29 Dec 2025 09:06:46 -0700 (MST) From: Simon Glass To: U-Boot Concept Date: Mon, 29 Dec 2025 09:06:05 -0700 Message-ID: <20251229160611.3899708-8-sjg@u-boot.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251229160611.3899708-1-sjg@u-boot.org> References: <20251229160611.3899708-1-sjg@u-boot.org> MIME-Version: 1.0 Message-ID-Hash: GDLA5J7GGNR6VKT5RE4KRXBNV65NMRMM X-Message-ID-Hash: GDLA5J7GGNR6VKT5RE4KRXBNV65NMRMM X-MailFrom: sjg@u-boot.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Heinrich Schuchardt , Simon Glass , "Claude Opus 4 . 5" X-Mailman-Version: 3.3.10 Precedence: list Subject: [Concept] [PATCH 7/9] test: Add a flag to emit per-test result lines List-Id: Discussion and patches related to U-Boot Concept Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Simon Glass The ut command shows test output but does not provide a machine-readable indication of whether each individual test passed or failed. External tools must rely on heuristics like scanning for failure patterns in the output. Add a -E flag that emits an explicit result line after each test: Result: PASS: test_name: file.c Result: FAIL: test_name: file.c This allows tools to reliably determine per-test pass/fail status without fragile pattern matching. The flag is optional to maintain backward compatibility with existing scripts. Co-developed-by: Claude Opus 4.5 Signed-off-by: Simon Glass --- doc/usage/cmd/ut.rst | 7 ++++++- include/test/test.h | 2 ++ test/cmd_ut.c | 8 +++++++- test/test-main.c | 9 +++++++++ 4 files changed, 24 insertions(+), 2 deletions(-) diff --git a/doc/usage/cmd/ut.rst b/doc/usage/cmd/ut.rst index 07cd970da6c..c40928dece9 100644 --- a/doc/usage/cmd/ut.rst +++ b/doc/usage/cmd/ut.rst @@ -11,7 +11,7 @@ Synopsis :: - ut [-fmr] [-R] [-I:] [ | all []] [...] + ut [-Efmr] [-R] [-I:] [ | all []] [...] ut [-s] info Description @@ -26,6 +26,11 @@ suite test Speciifes a particular test to run, within a suite, or all suites +-E + Emit a result line after each test, in the format + `Result: PASS|FAIL|SKIP: : `. This is useful for + automated parsing of test results. + -f, -m Force running of manual tests. Manual tests have the `_norun` suffix and are normally skipped because they require external setup (e.g., creating diff --git a/include/test/test.h b/include/test/test.h index 5ae90e39e00..74225a70e54 100644 --- a/include/test/test.h +++ b/include/test/test.h @@ -97,6 +97,7 @@ struct ut_arg { * @arg_count: Number of parsed arguments * @arg_error: Set if ut_str/int/bool() detects a type mismatch * @keep_record: Preserve console recording when ut_fail() is called + * @emit_result: Emit result line after each test completes * @priv: Private data for tests to use as needed */ struct unit_test_state { @@ -128,6 +129,7 @@ struct unit_test_state { int arg_count; bool arg_error; bool keep_record; + bool emit_result; char priv[UT_PRIV_SIZE]; }; diff --git a/test/cmd_ut.c b/test/cmd_ut.c index 37144242099..d6e591916ce 100644 --- a/test/cmd_ut.c +++ b/test/cmd_ut.c @@ -255,6 +255,7 @@ static int do_ut(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]) bool show_suites = false; bool force_run = false; bool keep_record = false; + bool emit_result = false; int runs_per_text = 1; struct suite *ste; char *name; @@ -269,6 +270,9 @@ static int do_ut(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]) for (str++; *str; str++) { switch (*str) { + case 'E': + emit_result = true; + break; case 'r': runs_per_text = dectoul(str + 1, NULL); goto next_arg; @@ -299,6 +303,7 @@ next_arg: ut_init_state(&uts); uts.keep_record = keep_record; + uts.emit_result = emit_result; name = argv[0]; select_name = cmd_arg1(argc, argv); @@ -344,7 +349,8 @@ next_arg: } U_BOOT_LONGHELP(ut, - "[-fmrs] [-R] [-I:] [ [...]] - run unit tests\n" + "[-Efmrs] [-R] [-I:] [ [...]] - run unit tests\n" + " -E Emit result line after each test\n" " -r Number of times to run each test\n" " -f/-m Force 'manual' tests to run as well\n" " -I Test to run after other tests have run\n" diff --git a/test/test-main.c b/test/test-main.c index c9e164da678..2524a154186 100644 --- a/test/test-main.c +++ b/test/test-main.c @@ -624,6 +624,7 @@ static int ut_run_test(struct unit_test_state *uts, struct unit_test *test, { const char *fname = strrchr(test->file, '/') + 1; const char *note = ""; + int old_fail_count; int ret; if ((test->flags & UTF_DM) && !uts->of_live) @@ -639,6 +640,7 @@ static int ut_run_test(struct unit_test_state *uts, struct unit_test *test, if (ret) return ret; + old_fail_count = uts->cur.fail_count; uts->arg_error = false; ret = test->func(uts); if (ret == -EAGAIN) @@ -650,6 +652,13 @@ static int ut_run_test(struct unit_test_state *uts, struct unit_test *test, ut_set_state(NULL); + if (uts->emit_result) { + bool passed = uts->cur.fail_count == old_fail_count; + + printf("Result: %s: %s: %s%s\n", passed ? "PASS" : "FAIL", + test_name, fname, note); + } + return 0; }