From nobody Sat Sep 27 19:27:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B153C2E11B5; Mon, 11 Aug 2025 18:28:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754936895; cv=none; b=DsFnbNvFZ8ltNY53WvKWV0ma+xbpRRkfiyic+C2/Wln6kgBPp/nijFLM4/i5DRKOJUkvJ8ecmuPtxj6s0Mq8BZ/yxaci4/c+PhZJc2EcPcyZkhc9iHb77TgXHwhOyGQGt5MVwT+CGnYxhwlVtuEb3k8H2e1oC1d8AZtxc7kTJjE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754936895; c=relaxed/simple; bh=v+uI9RgVhcnAWYgq7k6R6BzMyJMOzTZxFw5x6+b2bgY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lb1pVBruYsBwHIv+ax75otp/JoYB8Y7BMR7PqzpzkG9Z9kzlt0dxGpO1PY8qRY3lxjMKkIBqqs1YGl912Bvgk+bQJCKgHuxmNL+CLN4siip+NvwV90c0uF2q0e5VU8IEfdBMpwZKkJnqKvYjiqAdiOD3k+rr9+neKaWzKNXn/EQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=HBtPGm5R; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="HBtPGm5R" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2EE18C4CEF5; Mon, 11 Aug 2025 18:28:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1754936895; bh=v+uI9RgVhcnAWYgq7k6R6BzMyJMOzTZxFw5x6+b2bgY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HBtPGm5Rva7/ljD7ylUb1VsOUFBjTRwA5nu75yCSL7wEOt53SS/3LuD31Tiy5ggDx 1WUHofV9ytaje/F9HR2OjDi3ovBHYS8TuJd5jL+HtmYM+E7lMxRVbLpST0P15nQCBq VCKP/g2cWmJrJYZ2koIWMX3FMInjsTdSngqJhfsbUIgRN1742eanNCzJSh+JUtF/3q IDRAGtMheOj1zRJJLRzyLov6wXn8iR5cbgS+LqTm4IViRoTzQ1chXKnl9uSXWNJoW7 +Vyzd9BM8pDC7i71cLI7OR01Gt//khT/iAj2SN7kEaQfMETd2NxZs3fJTQfOWUOuqy 7q197TEVFiEOw== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, Ard Biesheuvel , linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, Eric Biggers Subject: [PATCH 1/3] kunit, lib/crypto: Move run_irq_test() to common header Date: Mon, 11 Aug 2025 11:26:29 -0700 Message-ID: <20250811182631.376302-2-ebiggers@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250811182631.376302-1-ebiggers@kernel.org> References: <20250811182631.376302-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Rename run_irq_test() to kunit_run_irq_test() and move it to a public header so that it can be reused by crc_kunit. Signed-off-by: Eric Biggers --- include/kunit/run-in-irq-context.h | 129 ++++++++++++++++++++++++++ lib/crypto/tests/hash-test-template.h | 123 +----------------------- 2 files changed, 133 insertions(+), 119 deletions(-) create mode 100644 include/kunit/run-in-irq-context.h diff --git a/include/kunit/run-in-irq-context.h b/include/kunit/run-in-irq-= context.h new file mode 100644 index 0000000000000..108e96433ea45 --- /dev/null +++ b/include/kunit/run-in-irq-context.h @@ -0,0 +1,129 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Helper function for testing code in interrupt contexts + * + * Copyright 2025 Google LLC + */ +#ifndef _KUNIT_RUN_IN_IRQ_CONTEXT_H +#define _KUNIT_RUN_IN_IRQ_CONTEXT_H + +#include +#include +#include +#include + +#define KUNIT_IRQ_TEST_HRTIMER_INTERVAL us_to_ktime(5) + +struct kunit_irq_test_state { + bool (*func)(void *test_specific_state); + void *test_specific_state; + bool task_func_reported_failure; + bool hardirq_func_reported_failure; + bool softirq_func_reported_failure; + unsigned long hardirq_func_calls; + unsigned long softirq_func_calls; + struct hrtimer timer; + struct work_struct bh_work; +}; + +static enum hrtimer_restart kunit_irq_test_timer_func(struct hrtimer *time= r) +{ + struct kunit_irq_test_state *state =3D + container_of(timer, typeof(*state), timer); + + WARN_ON_ONCE(!in_hardirq()); + state->hardirq_func_calls++; + + if (!state->func(state->test_specific_state)) + state->hardirq_func_reported_failure =3D true; + + hrtimer_forward_now(&state->timer, KUNIT_IRQ_TEST_HRTIMER_INTERVAL); + queue_work(system_bh_wq, &state->bh_work); + return HRTIMER_RESTART; +} + +static void kunit_irq_test_bh_work_func(struct work_struct *work) +{ + struct kunit_irq_test_state *state =3D + container_of(work, typeof(*state), bh_work); + + WARN_ON_ONCE(!in_serving_softirq()); + state->softirq_func_calls++; + + if (!state->func(state->test_specific_state)) + state->softirq_func_reported_failure =3D true; +} + +/* + * Helper function which repeatedly runs the given @func in task, softirq,= and + * hardirq context concurrently, and reports a failure to KUnit if any + * invocation of @func in any context returns false. @func is passed + * @test_specific_state as its argument. At most 3 invocations of @func w= ill + * run concurrently: one in each of task, softirq, and hardirq context. + * + * The main purpose of this interrupt context testing is to validate fallb= ack + * code paths that run in contexts where the normal code path cannot be us= ed, + * typically due to the FPU or vector registers already being in-use in ke= rnel + * mode. These code paths aren't covered when the test code is executed o= nly by + * the KUnit test runner thread in task context. The reason for the concu= rrency + * is because merely using hardirq context is not sufficient to reach a fa= llback + * code path on some architectures; the hardirq actually has to occur whil= e the + * FPU or vector unit was already in-use in kernel mode. + * + * Another purpose of this testing is to detect issues with the architectu= re's + * irq_fpu_usable() and kernel_fpu_begin/end() or equivalent functions, + * especially in softirq context when the softirq may have interrupted a t= ask + * already using kernel-mode FPU or vector (if the arch didn't prevent tha= t). + * Crypto functions are often executed in softirqs, so this is important. + */ +static inline void kunit_run_irq_test(struct kunit *test, bool (*func)(voi= d *), + int max_iterations, + void *test_specific_state) +{ + struct kunit_irq_test_state state =3D { + .func =3D func, + .test_specific_state =3D test_specific_state, + }; + unsigned long end_jiffies; + + /* + * Set up a hrtimer (the way we access hardirq context) and a work + * struct for the BH workqueue (the way we access softirq context). + */ + hrtimer_setup_on_stack(&state.timer, kunit_irq_test_timer_func, + CLOCK_MONOTONIC, HRTIMER_MODE_REL_HARD); + INIT_WORK_ONSTACK(&state.bh_work, kunit_irq_test_bh_work_func); + + /* Run for up to max_iterations or 1 second, whichever comes first. */ + end_jiffies =3D jiffies + HZ; + hrtimer_start(&state.timer, KUNIT_IRQ_TEST_HRTIMER_INTERVAL, + HRTIMER_MODE_REL_HARD); + for (int i =3D 0; i < max_iterations && !time_after(jiffies, end_jiffies); + i++) { + if (!func(test_specific_state)) + state.task_func_reported_failure =3D true; + } + + /* Cancel the timer and work. */ + hrtimer_cancel(&state.timer); + flush_work(&state.bh_work); + + /* Sanity check: the timer and BH functions should have been run. */ + KUNIT_EXPECT_GT_MSG(test, state.hardirq_func_calls, 0, + "Timer function was not called"); + KUNIT_EXPECT_GT_MSG(test, state.softirq_func_calls, 0, + "BH work function was not called"); + + /* Check for incorrect hash values reported from any context. */ + KUNIT_EXPECT_FALSE_MSG( + test, state.task_func_reported_failure, + "Incorrect hash values reported from task context"); + KUNIT_EXPECT_FALSE_MSG( + test, state.hardirq_func_reported_failure, + "Incorrect hash values reported from hardirq context"); + KUNIT_EXPECT_FALSE_MSG( + test, state.softirq_func_reported_failure, + "Incorrect hash values reported from softirq context"); +} + +#endif /* _KUNIT_RUN_IN_IRQ_CONTEXT_H */ diff --git a/lib/crypto/tests/hash-test-template.h b/lib/crypto/tests/hash-= test-template.h index f437a0a9ac6cd..61b43e62779fb 100644 --- a/lib/crypto/tests/hash-test-template.h +++ b/lib/crypto/tests/hash-test-template.h @@ -3,15 +3,13 @@ * Test cases for hash functions, including a benchmark. This is included= by * KUnit test suites that want to use it. See sha512_kunit.c for an examp= le. * * Copyright 2025 Google LLC */ +#include #include -#include -#include #include -#include =20 /* test_buf is a guarded buffer, i.e. &test_buf[TEST_BUF_LEN] is not mappe= d. */ #define TEST_BUF_LEN 16384 static u8 *test_buf; =20 @@ -317,123 +315,10 @@ static void test_hash_ctx_zeroization(struct kunit *= test) HASH_FINAL(&ctx, test_buf); KUNIT_ASSERT_MEMEQ_MSG(test, &ctx, zeroes, sizeof(ctx), "Hash context was not zeroized by finalization"); } =20 -#define IRQ_TEST_HRTIMER_INTERVAL us_to_ktime(5) - -struct hash_irq_test_state { - bool (*func)(void *test_specific_state); - void *test_specific_state; - bool task_func_reported_failure; - bool hardirq_func_reported_failure; - bool softirq_func_reported_failure; - unsigned long hardirq_func_calls; - unsigned long softirq_func_calls; - struct hrtimer timer; - struct work_struct bh_work; -}; - -static enum hrtimer_restart hash_irq_test_timer_func(struct hrtimer *timer) -{ - struct hash_irq_test_state *state =3D - container_of(timer, typeof(*state), timer); - - WARN_ON_ONCE(!in_hardirq()); - state->hardirq_func_calls++; - - if (!state->func(state->test_specific_state)) - state->hardirq_func_reported_failure =3D true; - - hrtimer_forward_now(&state->timer, IRQ_TEST_HRTIMER_INTERVAL); - queue_work(system_bh_wq, &state->bh_work); - return HRTIMER_RESTART; -} - -static void hash_irq_test_bh_work_func(struct work_struct *work) -{ - struct hash_irq_test_state *state =3D - container_of(work, typeof(*state), bh_work); - - WARN_ON_ONCE(!in_serving_softirq()); - state->softirq_func_calls++; - - if (!state->func(state->test_specific_state)) - state->softirq_func_reported_failure =3D true; -} - -/* - * Helper function which repeatedly runs the given @func in task, softirq,= and - * hardirq context concurrently, and reports a failure to KUnit if any - * invocation of @func in any context returns false. @func is passed - * @test_specific_state as its argument. At most 3 invocations of @func w= ill - * run concurrently: one in each of task, softirq, and hardirq context. - * - * The main purpose of this interrupt context testing is to validate fallb= ack - * code paths that run in contexts where the normal code path cannot be us= ed, - * typically due to the FPU or vector registers already being in-use in ke= rnel - * mode. These code paths aren't covered when the test code is executed o= nly by - * the KUnit test runner thread in task context. The reason for the concu= rrency - * is because merely using hardirq context is not sufficient to reach a fa= llback - * code path on some architectures; the hardirq actually has to occur whil= e the - * FPU or vector unit was already in-use in kernel mode. - * - * Another purpose of this testing is to detect issues with the architectu= re's - * irq_fpu_usable() and kernel_fpu_begin/end() or equivalent functions, - * especially in softirq context when the softirq may have interrupted a t= ask - * already using kernel-mode FPU or vector (if the arch didn't prevent tha= t). - * Crypto functions are often executed in softirqs, so this is important. - */ -static void run_irq_test(struct kunit *test, bool (*func)(void *), - int max_iterations, void *test_specific_state) -{ - struct hash_irq_test_state state =3D { - .func =3D func, - .test_specific_state =3D test_specific_state, - }; - unsigned long end_jiffies; - - /* - * Set up a hrtimer (the way we access hardirq context) and a work - * struct for the BH workqueue (the way we access softirq context). - */ - hrtimer_setup_on_stack(&state.timer, hash_irq_test_timer_func, - CLOCK_MONOTONIC, HRTIMER_MODE_REL_HARD); - INIT_WORK_ONSTACK(&state.bh_work, hash_irq_test_bh_work_func); - - /* Run for up to max_iterations or 1 second, whichever comes first. */ - end_jiffies =3D jiffies + HZ; - hrtimer_start(&state.timer, IRQ_TEST_HRTIMER_INTERVAL, - HRTIMER_MODE_REL_HARD); - for (int i =3D 0; i < max_iterations && !time_after(jiffies, end_jiffies); - i++) { - if (!func(test_specific_state)) - state.task_func_reported_failure =3D true; - } - - /* Cancel the timer and work. */ - hrtimer_cancel(&state.timer); - flush_work(&state.bh_work); - - /* Sanity check: the timer and BH functions should have been run. */ - KUNIT_EXPECT_GT_MSG(test, state.hardirq_func_calls, 0, - "Timer function was not called"); - KUNIT_EXPECT_GT_MSG(test, state.softirq_func_calls, 0, - "BH work function was not called"); - - /* Check for incorrect hash values reported from any context. */ - KUNIT_EXPECT_FALSE_MSG( - test, state.task_func_reported_failure, - "Incorrect hash values reported from task context"); - KUNIT_EXPECT_FALSE_MSG( - test, state.hardirq_func_reported_failure, - "Incorrect hash values reported from hardirq context"); - KUNIT_EXPECT_FALSE_MSG( - test, state.softirq_func_reported_failure, - "Incorrect hash values reported from softirq context"); -} - #define IRQ_TEST_DATA_LEN 256 #define IRQ_TEST_NUM_BUFFERS 3 /* matches max concurrency level */ =20 struct hash_irq_test1_state { u8 expected_hashes[IRQ_TEST_NUM_BUFFERS][HASH_SIZE]; @@ -467,11 +352,11 @@ static void test_hash_interrupt_context_1(struct kuni= t *test) rand_bytes(test_buf, IRQ_TEST_NUM_BUFFERS * IRQ_TEST_DATA_LEN); for (int i =3D 0; i < IRQ_TEST_NUM_BUFFERS; i++) HASH(&test_buf[i * IRQ_TEST_DATA_LEN], IRQ_TEST_DATA_LEN, state.expected_hashes[i]); =20 - run_irq_test(test, hash_irq_test1_func, 100000, &state); + kunit_run_irq_test(test, hash_irq_test1_func, 100000, &state); } =20 struct hash_irq_test2_hash_ctx { struct HASH_CTX hash_ctx; atomic_t in_use; @@ -498,11 +383,11 @@ static bool hash_irq_test2_func(void *state_) break; } if (WARN_ON_ONCE(ctx =3D=3D &state->ctxs[ARRAY_SIZE(state->ctxs)])) { /* * This should never happen, as the number of contexts is equal - * to the maximum concurrency level of run_irq_test(). + * to the maximum concurrency level of kunit_run_irq_test(). */ return false; } =20 if (ctx->step =3D=3D 0) { @@ -564,11 +449,11 @@ static void test_hash_interrupt_context_2(struct kuni= t *test) } if (remaining) state->update_lens[state->num_steps++] =3D remaining; state->num_steps +=3D 2; /* for init and final */ =20 - run_irq_test(test, hash_irq_test2_func, 250000, state); + kunit_run_irq_test(test, hash_irq_test2_func, 250000, state); } =20 #define UNKEYED_HASH_KUNIT_CASES \ KUNIT_CASE(test_hash_test_vectors), \ KUNIT_CASE(test_hash_all_lens_up_to_4096), \ --=20 2.50.1 From nobody Sat Sep 27 19:27:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C806D2E11C3; Mon, 11 Aug 2025 18:28:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754936895; cv=none; b=pQMaEGham8hYawJXv7WKKX0iJWDTqWfUMEbkgaP5OdsaRQIp73/i7RIqS6fgOzoE/Nu2QhcWOsWJ1pqA1X1ttov4lijnU5aU8AeLYPAyzgSn2jZqwx761lU/n9e630RQC6C2AcMUsorzol92cEWcK6XttSEnxnrPoQv6b838O4o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754936895; c=relaxed/simple; bh=uMhACYcWihYuwB7DWQQsjF40f0GjfQBRPtdS9pPrNeA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eFeSMaLGdeax5o7lf4KCTaJQso163hDL56JCgP/p755bvuYdABCEqgnKje3mak60SktFag6BxM2Ef1o5uDVfyOvdZR9V9SqW0h3zjtGzb+9+mc0jc8e4wqoYc9jNwYZ6ZlNSwXNWIskf8iwo8TMRdu0TYFPEUsSqPLM6DOSJOZ8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=qIB+1+Md; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="qIB+1+Md" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7CF66C4CEF6; Mon, 11 Aug 2025 18:28:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1754936895; bh=uMhACYcWihYuwB7DWQQsjF40f0GjfQBRPtdS9pPrNeA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qIB+1+MddbonxE/igP3HJeFr7AvgrTxjLhdEVgQq2CkqC/DE3GUTj206WglR1Jlqh ZjhMPL1XRo2sl6+T/B1K0wH4t2RibOIzvepC/MXJaqxxY6AnacPPZM/2u+fDD+AzKD 7Cu4/aQSv/kEpAgAkwF8p+D9NDg2qxc5/LV6nloN/onxoYL7F6Wv1fBqJE3PVPODmn IM9njFgS8u3Yk205hVcsSLyq4J/SxkQwuLK8pnEeANdcuqa88ved3QIhHCApiCdvtP P5yr0+AUCo8l7McqZ0vl4YBDo509giQElBglRbTDBquOr99cUVrNbEcKGwAjPSzZY9 VcykrMVVhJYnw== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, Ard Biesheuvel , linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, Eric Biggers Subject: [PATCH 2/3] lib/crc: crc_kunit: Test CRC computation in interrupt contexts Date: Mon, 11 Aug 2025 11:26:30 -0700 Message-ID: <20250811182631.376302-3-ebiggers@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250811182631.376302-1-ebiggers@kernel.org> References: <20250811182631.376302-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Test that if CRCs are computed in task, softirq, and hardirq context concurrently, then all results are as expected. Implement this using kunit_run_irq_test() which is also used by the lib/crypto/ tests. As with the corresponding lib/crypto/ tests, the purpose of this is to test fallback code paths and to exercise edge cases in the architecture's support for in-kernel FPU/SIMD/vector. Remove the code from crc_test() that sometimes disabled interrupts, as that was just an incomplete attempt to achieve similar test coverage. Signed-off-by: Eric Biggers --- lib/crc/tests/crc_kunit.c | 62 +++++++++++++++++++++++++++++++++------ 1 file changed, 53 insertions(+), 9 deletions(-) diff --git a/lib/crc/tests/crc_kunit.c b/lib/crc/tests/crc_kunit.c index f08d985d8860e..9a450e25ac811 100644 --- a/lib/crc/tests/crc_kunit.c +++ b/lib/crc/tests/crc_kunit.c @@ -4,10 +4,11 @@ * * Copyright 2024 Google LLC * * Author: Eric Biggers */ +#include #include #include #include #include #include @@ -139,19 +140,66 @@ static size_t generate_random_length(size_t max_lengt= h) break; } return len % (max_length + 1); } =20 +#define IRQ_TEST_DATA_LEN 512 +#define IRQ_TEST_NUM_BUFFERS 3 /* matches max concurrency level */ + +struct crc_irq_test_state { + const struct crc_variant *v; + u64 initial_crc; + u64 expected_crcs[IRQ_TEST_NUM_BUFFERS]; + atomic_t seqno; +}; + +/* + * Compute the CRC of one of the test messages and verify that it matches = the + * expected CRC from @state->expected_crcs. To increase the chance of det= ecting + * problems, cycle through multiple messages. + */ +static bool crc_irq_test_func(void *state_) +{ + struct crc_irq_test_state *state =3D state_; + const struct crc_variant *v =3D state->v; + u32 i =3D (u32)atomic_inc_return(&state->seqno) % IRQ_TEST_NUM_BUFFERS; + u64 actual_crc =3D v->func(state->initial_crc, + &test_buffer[i * IRQ_TEST_DATA_LEN], + IRQ_TEST_DATA_LEN); + + return actual_crc =3D=3D state->expected_crcs[i]; +} + +/* + * Test that if CRCs are computed in task, softirq, and hardirq context + * concurrently, then all results are as expected. + */ +static void crc_interrupt_context_test(struct kunit *test, + const struct crc_variant *v) +{ + struct crc_irq_test_state state =3D { + .v =3D v, + .initial_crc =3D generate_random_initial_crc(v), + }; + + for (int i =3D 0; i < IRQ_TEST_NUM_BUFFERS; i++) { + state.expected_crcs[i] =3D crc_ref( + v, state.initial_crc, + &test_buffer[i * IRQ_TEST_DATA_LEN], IRQ_TEST_DATA_LEN); + } + + kunit_run_irq_test(test, crc_irq_test_func, 100000, &state); +} + /* Test that v->func gives the same CRCs as a reference implementation. */ static void crc_test(struct kunit *test, const struct crc_variant *v) { size_t i; =20 for (i =3D 0; i < CRC_KUNIT_NUM_TEST_ITERS; i++) { u64 init_crc, expected_crc, actual_crc; size_t len, offset; - bool nosimd; =20 init_crc =3D generate_random_initial_crc(v); len =3D generate_random_length(CRC_KUNIT_MAX_LEN); =20 /* Generate a random offset. */ @@ -166,26 +214,22 @@ static void crc_test(struct kunit *test, const struct= crc_variant *v) =20 if (rand32() % 8 =3D=3D 0) /* Refresh the data occasionally. */ prandom_bytes_state(&rng, &test_buffer[offset], len); =20 - nosimd =3D rand32() % 8 =3D=3D 0; - /* * Compute the CRC, and verify that it equals the CRC computed * by a simple bit-at-a-time reference implementation. */ expected_crc =3D crc_ref(v, init_crc, &test_buffer[offset], len); - if (nosimd) - local_irq_disable(); actual_crc =3D v->func(init_crc, &test_buffer[offset], len); - if (nosimd) - local_irq_enable(); KUNIT_EXPECT_EQ_MSG(test, expected_crc, actual_crc, - "Wrong result with len=3D%zu offset=3D%zu nosimd=3D%d", - len, offset, nosimd); + "Wrong result with len=3D%zu offset=3D%zu", + len, offset); } + + crc_interrupt_context_test(test, v); } =20 static __always_inline void crc_benchmark(struct kunit *test, u64 (*crc_func)(u64 crc, const u8 *p, size_t len)) --=20 2.50.1 From nobody Sat Sep 27 19:27:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 255872E1C53; Mon, 11 Aug 2025 18:28:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754936896; cv=none; b=tBdfFz4d7Zi6Brj/wh/D9jYrQuVdPN+WY26mSuSX3oSEUOJAUGiOj5+iodV5xNlgkcl5zFB0HbdOdsy5Sne3jldd3gYMEbm+R/MUwaqU8ki0PFOhfhVDl4tFegqj+/icp2N/0j/p6EBl64A9nLnx6d+zea7qcHU6nw6ACkyLjZg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754936896; c=relaxed/simple; bh=R/c+sHj58hPXsXZ0wEu7sPVUYOmEp9NF5pVcd2XPhrk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=I8wh8W67wAV2h++I1IHo9kr9WtesLiMsPjzLoFB+Wyx42GL+PfRbOT9u/hKojxe4QCsmEAJcAZbJ57JHHyEsPI9p/oJVYhNIYw7310RVmL2bYYJgj71vB8vyB/JIlC1uoiherkQAeAqw3q+sjPvGTA5pUVwOHxibxvvQ+nOu2PE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=W8o5NsGI; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="W8o5NsGI" Received: by smtp.kernel.org (Postfix) with ESMTPSA id CB150C4CEED; Mon, 11 Aug 2025 18:28:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1754936896; bh=R/c+sHj58hPXsXZ0wEu7sPVUYOmEp9NF5pVcd2XPhrk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=W8o5NsGIjnD6Fikc+Kz+w8FS4JVodiTAEjEhMMlpT5RIW6ivG8H5FJh83Nc4tq/Lm LxfZQn1eF3Q9MssTKldhaiUQv0nBNFWe0mloJIzvF8ha4Im7bvR6AsaHo5OAWcOeU0 mWmxcF745Iorr1HB9ktFabBj8J3+V0cjbarnvOEy6OF4i71tTsy18dDjE9h4PYLUwk cXUBKCb7GQQc4gY6BBpqgG30ycrK36OWoH6OLx5CnTBEYJ26L7aJlUrxKuWC3yXRv3 mmdLGw52pB6Jq/585O+B3Rl8CDXSXh1IfmsKR4IpSP6qjRga6qTaDLR/poss1wAdSL 0gbdbI0FnhBSQ== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, Ard Biesheuvel , linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, Eric Biggers Subject: [PATCH 3/3] lib/crc: Use underlying functions instead of crypto_simd_usable() Date: Mon, 11 Aug 2025 11:26:31 -0700 Message-ID: <20250811182631.376302-4-ebiggers@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250811182631.376302-1-ebiggers@kernel.org> References: <20250811182631.376302-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Since crc_kunit now tests the fallback code paths without using crypto_simd_disabled_for_test, make the CRC code just use the underlying may_use_simd() and irq_fpu_usable() functions directly instead of crypto_simd_usable(). This eliminates an unnecessary layer. Take the opportunity to add likely() and unlikely() annotations as well. Signed-off-by: Eric Biggers --- lib/crc/arm/crc-t10dif.h | 6 ++---- lib/crc/arm/crc32.h | 6 ++---- lib/crc/arm64/crc-t10dif.h | 6 ++---- lib/crc/arm64/crc32.h | 11 ++++++----- lib/crc/powerpc/crc-t10dif.h | 5 +++-- lib/crc/powerpc/crc32.h | 5 +++-- lib/crc/x86/crc-pclmul-template.h | 3 +-- lib/crc/x86/crc32.h | 2 +- 8 files changed, 20 insertions(+), 24 deletions(-) diff --git a/lib/crc/arm/crc-t10dif.h b/lib/crc/arm/crc-t10dif.h index 2edf7e9681d05..1a969f4024d47 100644 --- a/lib/crc/arm/crc-t10dif.h +++ b/lib/crc/arm/crc-t10dif.h @@ -3,12 +3,10 @@ * Accelerated CRC-T10DIF using ARM NEON and Crypto Extensions instructions * * Copyright (C) 2016 Linaro Ltd */ =20 -#include - #include #include =20 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon); static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_pmull); @@ -21,19 +19,19 @@ asmlinkage void crc_t10dif_pmull8(u16 init_crc, const u= 8 *buf, size_t len, =20 static inline u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length) { if (length >=3D CRC_T10DIF_PMULL_CHUNK_SIZE) { if (static_branch_likely(&have_pmull)) { - if (crypto_simd_usable()) { + if (likely(may_use_simd())) { kernel_neon_begin(); crc =3D crc_t10dif_pmull64(crc, data, length); kernel_neon_end(); return crc; } } else if (length > CRC_T10DIF_PMULL_CHUNK_SIZE && static_branch_likely(&have_neon) && - crypto_simd_usable()) { + likely(may_use_simd())) { u8 buf[16] __aligned(16); =20 kernel_neon_begin(); crc_t10dif_pmull8(crc, data, length, buf); kernel_neon_end(); diff --git a/lib/crc/arm/crc32.h b/lib/crc/arm/crc32.h index 018007e162a2b..ae71aa60b7a74 100644 --- a/lib/crc/arm/crc32.h +++ b/lib/crc/arm/crc32.h @@ -5,12 +5,10 @@ * Copyright (C) 2016 Linaro Ltd */ =20 #include =20 -#include - #include #include #include =20 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_crc32); @@ -32,11 +30,11 @@ static inline u32 crc32_le_scalar(u32 crc, const u8 *p,= size_t len) } =20 static inline u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) { if (len >=3D PMULL_MIN_LEN + 15 && - static_branch_likely(&have_pmull) && crypto_simd_usable()) { + static_branch_likely(&have_pmull) && likely(may_use_simd())) { size_t n =3D -(uintptr_t)p & 15; =20 /* align p to 16-byte boundary */ if (n) { crc =3D crc32_le_scalar(crc, p, n); @@ -61,11 +59,11 @@ static inline u32 crc32c_scalar(u32 crc, const u8 *p, s= ize_t len) } =20 static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len) { if (len >=3D PMULL_MIN_LEN + 15 && - static_branch_likely(&have_pmull) && crypto_simd_usable()) { + static_branch_likely(&have_pmull) && likely(may_use_simd())) { size_t n =3D -(uintptr_t)p & 15; =20 /* align p to 16-byte boundary */ if (n) { crc =3D crc32c_scalar(crc, p, n); diff --git a/lib/crc/arm64/crc-t10dif.h b/lib/crc/arm64/crc-t10dif.h index c4521a7f1ee9b..435a990ec43c2 100644 --- a/lib/crc/arm64/crc-t10dif.h +++ b/lib/crc/arm64/crc-t10dif.h @@ -5,12 +5,10 @@ * Copyright (C) 2016 - 2017 Linaro Ltd */ =20 #include =20 -#include - #include #include =20 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_asimd); static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_pmull); @@ -23,19 +21,19 @@ asmlinkage u16 crc_t10dif_pmull_p64(u16 init_crc, const= u8 *buf, size_t len); =20 static inline u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length) { if (length >=3D CRC_T10DIF_PMULL_CHUNK_SIZE) { if (static_branch_likely(&have_pmull)) { - if (crypto_simd_usable()) { + if (likely(may_use_simd())) { kernel_neon_begin(); crc =3D crc_t10dif_pmull_p64(crc, data, length); kernel_neon_end(); return crc; } } else if (length > CRC_T10DIF_PMULL_CHUNK_SIZE && static_branch_likely(&have_asimd) && - crypto_simd_usable()) { + likely(may_use_simd())) { u8 buf[16]; =20 kernel_neon_begin(); crc_t10dif_pmull_p8(crc, data, length, buf); kernel_neon_end(); diff --git a/lib/crc/arm64/crc32.h b/lib/crc/arm64/crc32.h index 6e5dec45f05d2..31e649cd40a2f 100644 --- a/lib/crc/arm64/crc32.h +++ b/lib/crc/arm64/crc32.h @@ -3,12 +3,10 @@ #include #include #include #include =20 -#include - // The minimum input length to consider the 4-way interleaved code path static const size_t min_len =3D 1024; =20 asmlinkage u32 crc32_le_arm64(u32 crc, unsigned char const *p, size_t len); asmlinkage u32 crc32c_le_arm64(u32 crc, unsigned char const *p, size_t len= ); @@ -21,11 +19,12 @@ asmlinkage u32 crc32_be_arm64_4way(u32 crc, unsigned ch= ar const *p, size_t len); static inline u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) { if (!alternative_has_cap_likely(ARM64_HAS_CRC32)) return crc32_le_base(crc, p, len); =20 - if (len >=3D min_len && cpu_have_named_feature(PMULL) && crypto_simd_usab= le()) { + if (len >=3D min_len && cpu_have_named_feature(PMULL) && + likely(may_use_simd())) { kernel_neon_begin(); crc =3D crc32_le_arm64_4way(crc, p, len); kernel_neon_end(); =20 p +=3D round_down(len, 64); @@ -41,11 +40,12 @@ static inline u32 crc32_le_arch(u32 crc, const u8 *p, s= ize_t len) static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len) { if (!alternative_has_cap_likely(ARM64_HAS_CRC32)) return crc32c_base(crc, p, len); =20 - if (len >=3D min_len && cpu_have_named_feature(PMULL) && crypto_simd_usab= le()) { + if (len >=3D min_len && cpu_have_named_feature(PMULL) && + likely(may_use_simd())) { kernel_neon_begin(); crc =3D crc32c_le_arm64_4way(crc, p, len); kernel_neon_end(); =20 p +=3D round_down(len, 64); @@ -61,11 +61,12 @@ static inline u32 crc32c_arch(u32 crc, const u8 *p, siz= e_t len) static inline u32 crc32_be_arch(u32 crc, const u8 *p, size_t len) { if (!alternative_has_cap_likely(ARM64_HAS_CRC32)) return crc32_be_base(crc, p, len); =20 - if (len >=3D min_len && cpu_have_named_feature(PMULL) && crypto_simd_usab= le()) { + if (len >=3D min_len && cpu_have_named_feature(PMULL) && + likely(may_use_simd())) { kernel_neon_begin(); crc =3D crc32_be_arm64_4way(crc, p, len); kernel_neon_end(); =20 p +=3D round_down(len, 64); diff --git a/lib/crc/powerpc/crc-t10dif.h b/lib/crc/powerpc/crc-t10dif.h index 59e16804a6eae..e033d5f57bae2 100644 --- a/lib/crc/powerpc/crc-t10dif.h +++ b/lib/crc/powerpc/crc-t10dif.h @@ -4,12 +4,12 @@ * * Copyright 2017, Daniel Axtens, IBM Corporation. * [based on crc32c-vpmsum_glue.c] */ =20 +#include #include -#include #include #include #include #include =20 @@ -27,11 +27,12 @@ static inline u16 crc_t10dif_arch(u16 crci, const u8 *p= , size_t len) unsigned int prealign; unsigned int tail; u32 crc =3D crci; =20 if (len < (VECTOR_BREAKPOINT + VMX_ALIGN) || - !static_branch_likely(&have_vec_crypto) || !crypto_simd_usable()) + !static_branch_likely(&have_vec_crypto) || + unlikely(!may_use_simd())) return crc_t10dif_generic(crc, p, len); =20 if ((unsigned long)p & VMX_ALIGN_MASK) { prealign =3D VMX_ALIGN - ((unsigned long)p & VMX_ALIGN_MASK); crc =3D crc_t10dif_generic(crc, p, prealign); diff --git a/lib/crc/powerpc/crc32.h b/lib/crc/powerpc/crc32.h index 811cc2e6ed24d..cc8fa3913d4e0 100644 --- a/lib/crc/powerpc/crc32.h +++ b/lib/crc/powerpc/crc32.h @@ -1,8 +1,8 @@ // SPDX-License-Identifier: GPL-2.0-only +#include #include -#include #include #include #include #include =20 @@ -22,11 +22,12 @@ static inline u32 crc32c_arch(u32 crc, const u8 *p, siz= e_t len) { unsigned int prealign; unsigned int tail; =20 if (len < (VECTOR_BREAKPOINT + VMX_ALIGN) || - !static_branch_likely(&have_vec_crypto) || !crypto_simd_usable()) + !static_branch_likely(&have_vec_crypto) || + unlikely(!may_use_simd())) return crc32c_base(crc, p, len); =20 if ((unsigned long)p & VMX_ALIGN_MASK) { prealign =3D VMX_ALIGN - ((unsigned long)p & VMX_ALIGN_MASK); crc =3D crc32c_base(crc, p, prealign); diff --git a/lib/crc/x86/crc-pclmul-template.h b/lib/crc/x86/crc-pclmul-tem= plate.h index 35c950d7010c2..02744831c6fac 100644 --- a/lib/crc/x86/crc-pclmul-template.h +++ b/lib/crc/x86/crc-pclmul-template.h @@ -10,11 +10,10 @@ #ifndef _CRC_PCLMUL_TEMPLATE_H #define _CRC_PCLMUL_TEMPLATE_H =20 #include #include -#include #include #include "crc-pclmul-consts.h" =20 #define DECLARE_CRC_PCLMUL_FUNCS(prefix, crc_t) \ crc_t prefix##_pclmul_sse(crc_t crc, const u8 *p, size_t len, \ @@ -55,11 +54,11 @@ static inline bool have_avx512(void) * the dcache than the table-based code is, a 16-byte cutoff seems to work= well. */ #define CRC_PCLMUL(crc, p, len, prefix, consts, have_pclmulqdq) \ do { \ if ((len) >=3D 16 && static_branch_likely(&(have_pclmulqdq)) && \ - crypto_simd_usable()) { \ + likely(irq_fpu_usable())) { \ const void *consts_ptr; \ \ consts_ptr =3D (consts).fold_across_128_bits_consts; \ kernel_fpu_begin(); \ crc =3D static_call(prefix##_pclmul)((crc), (p), (len), \ diff --git a/lib/crc/x86/crc32.h b/lib/crc/x86/crc32.h index cea2c96d08d09..2c4a5976654ad 100644 --- a/lib/crc/x86/crc32.h +++ b/lib/crc/x86/crc32.h @@ -42,11 +42,11 @@ static inline u32 crc32c_arch(u32 crc, const u8 *p, siz= e_t len) =20 if (!static_branch_likely(&have_crc32)) return crc32c_base(crc, p, len); =20 if (IS_ENABLED(CONFIG_X86_64) && len >=3D CRC32C_PCLMUL_BREAKEVEN && - static_branch_likely(&have_pclmulqdq) && crypto_simd_usable()) { + static_branch_likely(&have_pclmulqdq) && likely(irq_fpu_usable())) { /* * Long length, the vector registers are usable, and the CPU is * 64-bit and supports both CRC32 and PCLMULQDQ instructions. * It is worthwhile to divide the data into multiple streams, * CRC them independently, and combine them using PCLMULQDQ. --=20 2.50.1