From nobody Mon Feb 9 05:53:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2916FEB64DD for ; Tue, 11 Jul 2023 14:33:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233191AbjGKOdz (ORCPT ); Tue, 11 Jul 2023 10:33:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43024 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232936AbjGKOdv (ORCPT ); Tue, 11 Jul 2023 10:33:51 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 297D712F for ; Tue, 11 Jul 2023 07:33:50 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5707177ff8aso62954757b3.2 for ; Tue, 11 Jul 2023 07:33:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689086029; x=1691678029; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=i17FLoNCi12ib3F8C35t83wNdppeBZG3dj7HY/hu0Qg=; b=POMOteOGWVpEMzAdxRIRnpR2K5d5Xn0jeyr4+I4tSok+qBnJD2BhtswqJcYb6EvIjq 40GDUnn2Q6K48/qpgxEh7qG8vJu4ed/WaVnvSqAgsOuIYHomu6lGzPpBwU3IYh/Hg8Ka H8ShOvmMYm+s0+19EmI89LN4icZW0QCk8qfQOIickWv3VOOX3XIQuQe33RwCss7CbaYh FfQyZExLgrVBAB+Zr6ljXSoERfdMiJqouw4MnCJDOId0qMgPyxIPEw2l17Nc2Y42HWo2 srS0dFDMODwvtcb2HbIp8cCvBfQgyofrVxPcTX2tJ3z1SUDQ2WmVEvBoHVOxSElbNjjD thCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689086029; x=1691678029; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=i17FLoNCi12ib3F8C35t83wNdppeBZG3dj7HY/hu0Qg=; b=GDaPGZD+Zxw49oUWjcPF5hBLmut1tWeBkDrv6n6EUDsEr9JdVV/vucuyZYxlxoo3zg aLSXs7w61NT2Wv2ir4pMzYR1G6Kfe51fgR04hWEQuSIY/sizcbW1Vj8+r4bbbw8s83Os mR560sSyzAK1pZNmzn42QPuLD4IP1lFZ8jzPKVNJ6+/w8QuFXdUMGvHsibZMsOXcpc63 1m5fipEMqnvV7O2Fo26/dn51IaTB+8OGG0xgIFXLvaeL3UCWNEEB1ZpYxA1y30LwFQyS ZEGYiNYDxiUIPCB9l2tcVJ+ZwiVc3ivpSysyt8cgbwjzjxtTtzyrvtQD6cT5MVCXSOWt IXhw== X-Gm-Message-State: ABy/qLb6yf5TeKX9eXykcbrAw8T+dmfY8vYq7yexYRKh49pE9XlY95P+ 7Sy/oOs7d6TQGvouIDYh/TveYD0AoPc= X-Google-Smtp-Source: APBJJlGQycuhFuaUSIxQ82QAIoJs6wHgHFp6GCXAoJqCmsJ1+Hc9x7Y5LOtqM6CF56P3mb9FHh3eNETgLIA= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:564d:3aaa:6b5f:4419]) (user=glider job=sendgmr) by 2002:a81:404d:0:b0:570:75a8:7867 with SMTP id m13-20020a81404d000000b0057075a87867mr127870ywn.4.1689086029340; Tue, 11 Jul 2023 07:33:49 -0700 (PDT) Date: Tue, 11 Jul 2023 16:33:28 +0200 In-Reply-To: <20230711143337.3086664-1-glider@google.com> Mime-Version: 1.0 References: <20230711143337.3086664-1-glider@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230711143337.3086664-2-glider@google.com> Subject: [PATCH 1/5] linux/bitqueue.h: add a KUnit test for bitqueue.h From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, yury.norov@gmail.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add tests checking that struct bitq correctly handles sub-byte values. Signed-off-by: Alexander Potapenko --- lib/Kconfig.debug | 8 ++ lib/Makefile | 1 + lib/test_bitqueue.c | 244 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 253 insertions(+) create mode 100644 lib/test_bitqueue.c diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index ce51d4dc6803e..a6598b2c250d5 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2678,6 +2678,14 @@ config SIPHASH_KUNIT_TEST This is intended to help people writing architecture-specific optimized versions. If unsure, say N. =20 +config BITQUEUE_KUNIT_TEST + tristate "Test " if !KUNIT_ALL_TESTS + depends on KUNIT + default KUNIT_ALL_TESTS + help + Enable this option to test the kernel's bit queue implementation + (). + config TEST_UDELAY tristate "udelay test driver" help diff --git a/lib/Makefile b/lib/Makefile index 876fcdeae34ec..7efb6aba31cf9 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -394,6 +394,7 @@ CFLAGS_fortify_kunit.o +=3D $(DISABLE_STRUCTLEAK_PLUGIN) obj-$(CONFIG_FORTIFY_KUNIT_TEST) +=3D fortify_kunit.o obj-$(CONFIG_STRSCPY_KUNIT_TEST) +=3D strscpy_kunit.o obj-$(CONFIG_SIPHASH_KUNIT_TEST) +=3D siphash_kunit.o +obj-$(CONFIG_BITQUEUE_KUNIT_TEST) +=3D test_bitqueue.o =20 obj-$(CONFIG_GENERIC_LIB_DEVMEM_IS_ALLOWED) +=3D devmem_is_allowed.o =20 diff --git a/lib/test_bitqueue.c b/lib/test_bitqueue.c new file mode 100644 index 0000000000000..aec04b3a5f068 --- /dev/null +++ b/lib/test_bitqueue.c @@ -0,0 +1,244 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Test cases for struct bitq, a simple bit queue. + */ + +#include +#include +#include + +/* Set up a bit queue containing @size bytes. */ +static void bitq_setup(struct bitq *it, size_t size) +{ + u8 *data =3D kmalloc(size, GFP_KERNEL); + + bitq_init(it, data, size); +} + +/* Tear down the bit queue. */ +static void bitq_teardown(struct bitq *it) +{ + kfree(it->data); + memset(it, 0, sizeof(*it)); +} + +/* Test that nothing can be popped from an empty queue. */ +static void test_empty(struct kunit *test) +{ + struct bitq it; + u8 val =3D 0; + + /* Allocate a two-byte queue. */ + bitq_setup(&it, 2); + + /* Queue is empty. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 8), -1); + bitq_teardown(&it); +} + +/* Test that simple byte-granular enqueue/dequeue operations work. */ +static void test_basic_enqueue_dequeue(struct kunit *test) +{ + struct bitq it; + u8 val =3D 0; + + /* Allocate a two-byte queue. */ + bitq_setup(&it, 2); + /* Enqueue two 8-bit values. */ + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0xaa, 8), 8); + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0xbb, 8), 8); + /* Cannot enqueue the third byte. */ + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 1, 8), -1); + /* Dequeue two bytes. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 8), 8); + KUNIT_EXPECT_EQ(test, val, 0xaa); + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 8), 8); + KUNIT_EXPECT_EQ(test, val, 0xbb); + + /* Queue is empty. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 8), -1); + bitq_teardown(&it); +} + +/* Test that values shorter than 8 bits can be enqueued and dequeued. */ +static void test_shorter_than_byte(struct kunit *test) +{ + struct bitq it; + u8 val =3D 0; + + /* Allocate a two-byte queue. */ + bitq_setup(&it, 2); + /* Enqueue two 0b101 values. */ + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0b101, 3), 3); + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0b101, 3), 3); + /* The first byte of the queue is now 0b10110100. */ + + /* Now dequeue three 2-bit values: 0b10, 0b11, 0b01. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 2), 2); + KUNIT_EXPECT_EQ(test, val, 0b10); + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 2), 2); + KUNIT_EXPECT_EQ(test, val, 0b11); + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 2), 2); + KUNIT_EXPECT_EQ(test, val, 0b01); + + /* Queue is empty. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 1), -1); + bitq_teardown(&it); +} + +/* Test that bits are carried over correctly if they do not fit. */ +static void test_carryover(struct kunit *test) +{ + struct bitq it; + u8 val =3D 0; + int i; + + /* Allocate a three-byte queue. */ + bitq_setup(&it, 3); + /* Enqueue 0b100 seven times. */ + for (i =3D 0; i < 7; i++) + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0b100, 3), 3); + /* Now dequeue three 7-bit values: 0b1001001, 0b0010010, 0b0100100. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 7), 7); + KUNIT_EXPECT_EQ(test, val, 0b1001001); + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 7), 7); + KUNIT_EXPECT_EQ(test, val, 0b0010010); + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 7), 7); + KUNIT_EXPECT_EQ(test, val, 0b0100100); + + /* Queue is empty. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 1), -1); + bitq_teardown(&it); +} + +/* + * Test case extracted from the EA0 tag compression algorithm, where + * carried over bits were accidentally written into the previous byte. + */ +static void test_carryover_ea0(struct kunit *test) +{ + struct bitq it; + u8 val =3D 0; + + /* Allocate a three-byte queue. */ + bitq_setup(&it, 3); + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0b100, 3), 3); + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0b1010, 4), 4); + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0b0000, 4), 4); + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0b1010, 4), 4); + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0b1011, 4), 4); + + /* Now dequeue two byte values: 0b10010100, 0b00010101. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 8), 8); + KUNIT_EXPECT_EQ(test, val, 0b10010100); + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 8), 8); + KUNIT_EXPECT_EQ(test, val, 0b00010101); + /* And the remaining 0b011. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 3), 3); + KUNIT_EXPECT_EQ(test, val, 0b011); + + /* Queue is empty. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 1), -1); + bitq_teardown(&it); +} + +/* Test that upper bits of the pushed value are discarded. */ +static void test_trim_upper_bits(struct kunit *test) +{ + struct bitq it; + u8 val =3D 0; + + /* Allocate a two-byte queue. */ + bitq_setup(&it, 2); + /* Enqueue two values that do not fit into 4 bits. */ + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0xab, 4), 4); + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0xab, 4), 4); + /* The first byte of the queue is now 0xbb. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 8), 8); + KUNIT_EXPECT_EQ(test, val, 0xbb); + + /* Queue is empty. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 1), -1); + bitq_teardown(&it); +} + +/* Another test for discarding the upper bits. */ +static void test_trim_upper_bits2(struct kunit *test) +{ + struct bitq it; + u8 val =3D 0; + + /* Allocate a two-byte queue. */ + bitq_setup(&it, 2); + /* Push seven zero bits. */ + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0, 7), 7); + /* Push a single 1 bit, but pass a bigger value to bitq_enqueue(). */ + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0xff, 1), 1); + /* The first byte of the queue is now 0x01. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 8), 8); + KUNIT_EXPECT_EQ(test, val, 0x01); + + /* Queue is empty. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 1), -1); + bitq_teardown(&it); +} + +/* Test that a NULL value can be used as output of bitq_dequeue() */ +static void test_dequeue_to_null(struct kunit *test) +{ + struct bitq it; + + /* Allocate a two-byte queue. */ + bitq_setup(&it, 2); + /* Enqueue a byte value. */ + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0xab, 8), 8); + /* Dequeue the byte, but discard its value. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, NULL, 8), 8); + + /* Queue is empty. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, NULL, 1), -1); + bitq_teardown(&it); +} + +/* Test that bitq_init_full works. */ +static void test_init_full(struct kunit *test) +{ + struct bitq it; + u8 data[2] =3D { 0xaa, 0xbb }; + u8 val =3D 0; + + /* Initialize a queue with the contents of @data */ + bitq_init_full(&it, data, 2); + /* Cannot enqueue anything else. */ + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 1, 8), -1); + /* Dequeue two bytes. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 8), 8); + KUNIT_EXPECT_EQ(test, val, 0xaa); + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 8), 8); + KUNIT_EXPECT_EQ(test, val, 0xbb); + + /* Queue is empty. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, NULL, 1), -1); +} + +static struct kunit_case bitq_test_cases[] =3D { + KUNIT_CASE(test_empty), + KUNIT_CASE(test_basic_enqueue_dequeue), + KUNIT_CASE(test_shorter_than_byte), + KUNIT_CASE(test_carryover), + KUNIT_CASE(test_carryover_ea0), + KUNIT_CASE(test_trim_upper_bits), + KUNIT_CASE(test_trim_upper_bits2), + KUNIT_CASE(test_dequeue_to_null), + KUNIT_CASE(test_init_full), + {} +}; + +static struct kunit_suite bitq_test_suite =3D { + .name =3D "bitq", + .test_cases =3D bitq_test_cases, +}; +kunit_test_suites(&bitq_test_suite); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Alexander Potapenko "); --=20 2.41.0.255.g8b1d071c50-goog From nobody Mon Feb 9 05:53:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00301EB64DC for ; Tue, 11 Jul 2023 14:34:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233312AbjGKOeA (ORCPT ); Tue, 11 Jul 2023 10:34:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43084 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233172AbjGKOdz (ORCPT ); Tue, 11 Jul 2023 10:33:55 -0400 Received: from mail-ed1-x549.google.com (mail-ed1-x549.google.com [IPv6:2a00:1450:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9BE7F9C for ; Tue, 11 Jul 2023 07:33:53 -0700 (PDT) Received: by mail-ed1-x549.google.com with SMTP id 4fb4d7f45d1cf-50daa85e940so3767500a12.0 for ; Tue, 11 Jul 2023 07:33:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689086032; x=1691678032; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=dufumGN0X5SXjxYpittS7pKflaiJdPliu9At6Sap++c=; b=75uvsIUF8pp1PFh46nfHCG3WKMmvQIBEGMIyyIIHShVAQquW6/g+m+PKuhZ9x6sQji iClp8scbJoKjVEXbhgaKgs8rCWdskSFEIYvo3JSyqpKiPkap0fUB+HL35ZgRzYpA0/CK iE5K9iqwV60CMTAvtPTJl/1mkcJKQKg3m2w+VjR6ju8JEAoevMvbnlXXjIX7t9+e2xEY ZcuxDv+Lb2ZS++s9EWQC4S/MI+LZAjAtZWuUo885GzT68wi616cFcfZ/lsCHwqsE/42r n86Xg5zOdfDTEByAFc8MmPMnUAw7KaxGf2TwZ75eD9yHw1cAySBhvnF16rk6SwZceK5W aa4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689086032; x=1691678032; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dufumGN0X5SXjxYpittS7pKflaiJdPliu9At6Sap++c=; b=ENhMePsbq2WYIMII8NTQp2p4JxYK1hN4ImjweEfbzJayDhdio6LlTz+6vJUKRUDCXl yoc+pSxlbfJS8FkE0PLCWYFqB+U1Gsk2lfkDZUAUyUzrVFigg+CJ9M/+bnl3U1vf+82E KD+jHxQs0MpHll79uQEkQtOklgIA2bHqbGowfAWzg9cAtCjc8mDyOHXobSK8XHMQm0el FLXaNJ937neoIAcvuLI7V9Y2RDoTSBrDgvMH1WjC9lmrhwO4LAXU2o2D/b3EgGcZECIg wK9eBu2gCYILViJhR6/mBZ9kRKooZxWbctxWSz9LJ1x46C5wEWng5S1eC1nRrEzMRf58 1DCg== X-Gm-Message-State: ABy/qLYgWm+N3f5BSv7Zol96rRi3+6RhBEemdOQSuh7B+4uCtMHzzuXz 3Q+Ii4JQOxQKJ5lXPvUaoFtLEGdRVnc= X-Google-Smtp-Source: APBJJlFYz/C1GDAbhGhoPueuxhQix1/9PyVpcPvrmHKitH+djEv3zoQpy4GpjnDGejdwdcFji6DbCjpZTyY= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:564d:3aaa:6b5f:4419]) (user=glider job=sendgmr) by 2002:a50:d786:0:b0:51e:23bd:df12 with SMTP id w6-20020a50d786000000b0051e23bddf12mr79993edi.7.1689086032177; Tue, 11 Jul 2023 07:33:52 -0700 (PDT) Date: Tue, 11 Jul 2023 16:33:29 +0200 In-Reply-To: <20230711143337.3086664-1-glider@google.com> Mime-Version: 1.0 References: <20230711143337.3086664-1-glider@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230711143337.3086664-3-glider@google.com> Subject: [PATCH 1/5] linux/bitqueue.h: add the bit queue implementation From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, yury.norov@gmail.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" struct bitq represents a bit queue with external storage. Its purpose is to easily pack sub-byte values, which can be used, for example, to implement RLE algorithms. Signed-off-by: Alexander Potapenko --- include/linux/bitqueue.h | 144 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 144 insertions(+) create mode 100644 include/linux/bitqueue.h diff --git a/include/linux/bitqueue.h b/include/linux/bitqueue.h new file mode 100644 index 0000000000000..c4393f703c697 --- /dev/null +++ b/include/linux/bitqueue.h @@ -0,0 +1,144 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * A simple bit queue which supports enqueueing/dequeueing of sub-byte val= ues. + * + * This can be used to pack complex bitfields into byte arrays. + */ +#ifndef _LINUX_BITQUEUE_H +#define _LINUX_BITQUEUE_H + +#include +#include + +/** + * struct bitq - represents a bit queue with external storage. + * @data: data buffer used by the queue. + * @size: size of @data in bytes. + * @bit_pos: current bit position. + */ +struct bitq { + u8 *data; + int size, bit_pos; +}; + +/** + * bitq_init - initialize an empty bit queue. + * @q: struct bitq to be initialized. + * @data: external data buffer to use. + * @size: capacity in bytes. + * + * Return: 0 in the case of success, -1 if either of the pointers is NULL. + */ +static inline int bitq_init(struct bitq *q, u8 *data, int size) +{ + if (!q || !data) + return -1; + q->data =3D data; + q->size =3D size; + memset(data, 0, size); + q->bit_pos =3D 0; + return 0; +} + +/** + * bitq_init_full - make a bit queue from an initialized byte array. + * @q: struct bitq to be initialized. + * @data: external data buffer to use. + * @size: capacity in bytes. + * + * Return: 0 in the case of success, -1 if either of the pointers is NULL. + */ +static inline int bitq_init_full(struct bitq *q, u8 *data, int size) +{ + if (!q || !data) + return -1; + q->data =3D data; + q->size =3D size; + q->bit_pos =3D q->size * 8; + return 0; +} + +/** + * bitq_enqueue - push up to 8 bits to the end of the queue. + * @q: struct bitq. + * @value: byte containing the value to be pushed. + * @bits: number of bits (1 to 8) to push. + * + * Return: number of bits pushed, or -1 in the case of an error. + */ +static inline int bitq_enqueue(struct bitq *q, u8 value, int bits) +{ + int byte_pos, left_in_byte, max_pos; + u8 hi, lo; + + if (!q || (bits < 1) || (bits > 8)) + return -1; + + max_pos =3D q->size * 8; + if ((max_pos - q->bit_pos) < bits) + return -1; + + left_in_byte =3D 8 - (q->bit_pos % 8); + byte_pos =3D q->bit_pos / 8; + /* Clamp @value. */ + value %=3D (1 << bits); + if (left_in_byte >=3D bits) { + /* @value fits into the current byte. */ + value <<=3D (left_in_byte - bits); + q->data[byte_pos] |=3D value; + } else { + /* + * @value needs to be split between the current and the + * following bytes. + */ + hi =3D value >> (bits - left_in_byte); + q->data[byte_pos] |=3D hi; + byte_pos++; + lo =3D value << (8 - (bits - left_in_byte)); + q->data[byte_pos] |=3D lo; + } + q->bit_pos +=3D bits; + return bits; +} + +/** + * bitq_dequeue - pop up to 8 bits from the beginning of the queue. + * @q: struct bitq. + * @value: u8* to store the popped value (can be NULL). + * @bits: number of bits (1 to 8) to pop. + * + * Return: number of bits popped, or -1 in the case of an error. + */ + +#include +static inline int bitq_dequeue(struct bitq *q, u8 *value, int bits) +{ + int rem_bits =3D 8 - bits, i; + u8 output; + + /* Invalid arguments. */ + if (!q || (bits < 1) || (bits > 8)) + return -1; + /* Not enough space to insert @bits. */ + if (q->bit_pos < bits) + return -1; + /* Take the first @bits bits from the first byte. */ + output =3D q->data[0]; + output >>=3D rem_bits; + if (value) + *value =3D output; + + /* + * Shift every byte in the queue to the left by @bits, carrying over to + * the previous byte. + */ + for (i =3D 0; i < q->size - 1; i++) { + q->data[i] =3D (q->data[i] << bits) | + (q->data[i + 1] >> rem_bits); + } + q->data[q->size - 1] <<=3D bits; + q->bit_pos -=3D bits; + return bits; +} + +#endif // _LINUX_BITQUEUE_H --=20 2.41.0.255.g8b1d071c50-goog From nobody Mon Feb 9 05:53:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B8DFEB64DD for ; Tue, 11 Jul 2023 14:34:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232690AbjGKOeF (ORCPT ); Tue, 11 Jul 2023 10:34:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43176 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233242AbjGKOd7 (ORCPT ); Tue, 11 Jul 2023 10:33:59 -0400 Received: from mail-ej1-x649.google.com (mail-ej1-x649.google.com [IPv6:2a00:1450:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6EA2B10EB for ; Tue, 11 Jul 2023 07:33:56 -0700 (PDT) Received: by mail-ej1-x649.google.com with SMTP id a640c23a62f3a-993d500699fso267416666b.1 for ; Tue, 11 Jul 2023 07:33:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689086035; x=1691678035; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/3xF8MGaf3YIHiv8WE5XhXRUvL32mDr7FE1dMJNOszw=; b=5eHPvBom+QAdA03a/wXMEx71lKCdM1JNPZcMDUneDGQ8Bsjo8FS6TZjzFwAGqlhwdx Q37xzjaugHsjhviYLYEhNoTLyztU3ndtHDDJu8Gjs8XJCTH7wVuDYo70BITzPUC7JQVe PgCGmsVZ2dUf0exHrvyC1V6M1eeIVxfiSUYRZ+jSYU3xVtKGsdninDG1adTF+7s5vZ+3 IdIoEh5aPv1yTwEbVCmolz3tl0iCaM7mQQfLCKYQJhaGtZvAKR6argzlJ4iefRXDjjXW LK0Vqh36E8yc8XFYya7Zibe7jOsk0/iuRxJTQ3Aiz7C9Pc2NallJMkW0TMHoX3V/rt1W +aow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689086035; x=1691678035; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/3xF8MGaf3YIHiv8WE5XhXRUvL32mDr7FE1dMJNOszw=; b=VSBzt78w+17rYM/NNaDRv7ptz+v7saJ95937uv2Keym1qBw3BoAxhVaXOUMNnQm/8V a9Z8S4R3eVyWVZ3duHdXVHfAhqnkDJfbgYFuVKIqG3YlyOXUwL1XcLkAV5Dwlnnq+C8e NA15n/y7lpmdBh5cxqRTG9YdJpVutzJbJJ+PmqtZ49OrjISmHGPNk5V7QcPC7mkszsFw M2z0M6euVL6SAS8ruuNCva7aXkoQ9WcJmmdC1uYx1Y90gnJ5A6WqOxan9p9xH/XoqpW5 HVwOuG7CYpESMQxwsChOFNh00sVxDWRjGaVDMNm9obmq9Nl2n7VUURas8fW87qX39gbh a20Q== X-Gm-Message-State: ABy/qLaSLZmbAkHQnckTYpFl4e2aYKVadpetYp+mlyv1/And34KqvpBb IEakZdMXDbVZ7B0fIOqObq1syUrVvqA= X-Google-Smtp-Source: APBJJlGY1TSKy0k5Wx3iLQE3LDVhp8beWNpJ4M9PuULbda4yF734Ga83YPAeQrqegE+KSz+mFgpZP7DdfrI= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:564d:3aaa:6b5f:4419]) (user=glider job=sendgmr) by 2002:a17:907:763a:b0:98e:2b63:9f78 with SMTP id jy26-20020a170907763a00b0098e2b639f78mr50605ejc.0.1689086034940; Tue, 11 Jul 2023 07:33:54 -0700 (PDT) Date: Tue, 11 Jul 2023 16:33:30 +0200 In-Reply-To: <20230711143337.3086664-1-glider@google.com> Mime-Version: 1.0 References: <20230711143337.3086664-1-glider@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230711143337.3086664-4-glider@google.com> Subject: [PATCH 2/5] arm64: mte: implement CONFIG_ARM64_MTE_COMP From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, yury.norov@gmail.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The config implements the EA0 algorithm suggested by Evgenii Stepanov to compress the memory tags for ARM MTE during swapping. The algorithm is based on RLE and specifically targets 128-byte buffers of tags corresponding to a single page. In the common case a buffer can be compressed into 63 bits, making it possible to store it without additional memory allocation. Suggested-by: Evgenii Stepanov Signed-off-by: Alexander Potapenko --- arch/arm64/Kconfig | 10 + arch/arm64/include/asm/mtecomp.h | 60 +++++ arch/arm64/mm/Makefile | 1 + arch/arm64/mm/mtecomp.c | 398 +++++++++++++++++++++++++++++++ 4 files changed, 469 insertions(+) create mode 100644 arch/arm64/include/asm/mtecomp.h create mode 100644 arch/arm64/mm/mtecomp.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 343e1e1cae10a..b25b584a0a9cb 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2065,6 +2065,16 @@ config ARM64_EPAN if the cpu does not implement the feature. endmenu # "ARMv8.7 architectural features" =20 +config ARM64_MTE_COMP + bool "Tag compression for ARM64 MTE" + default y + depends on ARM64_MTE + help + Enable tag compression support for ARM64 MTE. + + 128-byte tag buffers corresponding to 4K pages can be compressed using + the EA0 algorithm to save heap memory. + config ARM64_SVE bool "ARM Scalable Vector Extension support" default y diff --git a/arch/arm64/include/asm/mtecomp.h b/arch/arm64/include/asm/mtec= omp.h new file mode 100644 index 0000000000000..65a3730cc50d9 --- /dev/null +++ b/arch/arm64/include/asm/mtecomp.h @@ -0,0 +1,60 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASM_MTECOMP_H +#define __ASM_MTECOMP_H + +#include + +/* + * ea0_compress() - compress the given tag array. + * @tags: 128-byte array to read the tags from. + * + * Compresses the tags and returns a 64-bit opaque handle pointing to the + * tag storage. May allocate memory, which is freed by @ea0_release_handle= (). + */ +u64 ea0_compress(u8 *tags); + +/* + * ea0_decompress() - decompress the tag array addressed by the handle. + * @handle: handle returned by @ea0_decompress() + * @tags: 128-byte array to write the tags to. + * + * Reads the compressed data and writes it into the user-supplied tag arra= y. + * Returns true on success, false on error. + */ +bool ea0_decompress(u64 handle, u8 *tags); + +/* + * ea0_release_handle() - release the handle returned by ea0_compress(). + * @handle: handle returned by ea0_compress(). + */ +void ea0_release_handle(u64 handle); + +/* Functions below are exported for testing purposes. */ + +/* + * ea0_storage_size() - calculate the memory occupied by compressed tags. + * @handle: storage handle returned by ea0_compress. + */ +int ea0_storage_size(u64 handle); + +/* + * ea0_tags_to_ranges() - break @tags into arrays of tag ranges. + * @tags: 128-byte array containing 256 MTE tags. + * @out_tags: u8 array to store the tag of every range. + * @out_sizes: u16 array to store the size of every range. + * @out_len: length of @out_tags and @out_sizes (output parameter, initial= ly + * equal to lengths of out_tags[] and out_sizes[]). + */ +void ea0_tags_to_ranges(u8 *tags, u8 *out_tags, short *out_sizes, int *out= _len); + +/* + * ea0_ranges_to_tags() - fill @tags using given tag ranges. + * @r_tags: u8[256] containing the tag of every range. + * @r_sizes: u16[256] containing the size of every range. + * @r_len: length of @r_tags and @r_sizes. + * @tags: 128-byte array to write the tags to. + */ +void ea0_ranges_to_tags(u8 *r_tags, short *r_sizes, int r_len, u8 *tags); + +#endif // __ASM_MTECOMP_H diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index dbd1bc95967d0..46778f6dd83c2 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -10,6 +10,7 @@ obj-$(CONFIG_TRANS_TABLE) +=3D trans_pgd.o obj-$(CONFIG_TRANS_TABLE) +=3D trans_pgd-asm.o obj-$(CONFIG_DEBUG_VIRTUAL) +=3D physaddr.o obj-$(CONFIG_ARM64_MTE) +=3D mteswap.o +obj-$(CONFIG_ARM64_MTE_COMP) +=3D mtecomp.o KASAN_SANITIZE_physaddr.o +=3D n =20 obj-$(CONFIG_KASAN) +=3D kasan_init.o diff --git a/arch/arm64/mm/mtecomp.c b/arch/arm64/mm/mtecomp.c new file mode 100644 index 0000000000000..01f7d22665b49 --- /dev/null +++ b/arch/arm64/mm/mtecomp.c @@ -0,0 +1,398 @@ +// SPDX-License-Identifier: GPL-2.0-only + +/* + * MTE tag compression algorithm. + * Proposed by Evgenii Stepanov + */ + +/* + * EA0 stands for "Evgenii's Algorithm 0", as the initial proposal contain= ed two + * compression algorithms. + * + * The algorithm attempts to compress a 128-byte (MTE_GRANULES_PER_PAGE / = 2) + * array of tags into a smaller byte sequence that can be stored in a + * 16-, 32-, or 64-byte buffer. A special case is storing the tags inline = in + * an 8-byte pointer. + * + * We encapsulate tag storage memory management in this module, because it= is + * tightly coupled with the pointer representation. + * ea0_compress(*tags) takes a 128-byte buffer and returns an opaque val= ue + * that can be stored in Xarray + * ea_decompress(*ptr, *tags) takes the opaque value and loads the tags = into + * the provided 128-byte buffer. + * + * + * + * The compression algorithm works as follows. + * + * 1. The input array of 128 bytes is transformed into tag ranges (two arr= ays: + * @r_tags containing tag values and @r_sizes containing range lengths)= by + * ea0_tags_to_ranges(). Note that @r_sizes sums up to 256. + * + * 2. Depending on the number N of ranges, the following storage class is = picked: + * N <=3D 6: 8 bytes (inline case, no allocation required); + * 6 < N <=3D 11: 16 bytes + * 11 < N <=3D 23: 32 bytes + * 23 < N <=3D 46: 64 bytes + * 46 < N: 128 bytes (no compression will be performed) + * + * 3. The number of the largest element of @r_sizes is stored in @largest_= idx. + * The element itself is thrown away from @r_sizes, because it can be + * reconstructed from the sum of the remaining elements. Note that now = none + * of the remaining @r_sizes elements is greater than 127. + * + * 4. For the inline case, the following values are stored in the 8-byte h= andle: + * largest_idx : i4 + * r_tags[0..5] : i4 x 6 + * r_sizes[0..4] : i7 x 5 + * (if N is less than 6, @r_tags and @r_sizes are padded up with zero v= alues) + * + * Because @largest_idx is <=3D 5, bit 63 of the handle is always 0 (so= it can + * be stored in the Xarray), and bits 62..60 cannot all be 1, so it can= be + * distinguished from a kernel pointer. + * + * 5. For the out-of-line case, the storage is allocated from one of the + * "mte-tags-{16,32,64,128}" kmem caches. The resulting pointer is alig= ned + * on 8 bytes, so its bits 2..0 can be used to store the size class: + * - 0 for 128 bytes + * - 1 for 16 + * - 2 for 32 + * - 4 for 64. + * Bit 63 of the pointer is zeroed out, so that it can be stored in Xar= ray. + * + * 6. The data layout in the allocated storage is as follows: + * largest_idx : i6 + * r_tags[0..N] : i4 x N + * r_sizes[0..N-1] : i7 x (N-1) + * + * + * + * The decompression algorithm performs the steps below. + * + * 1. Decide if data is stored inline (bits 62..60 of the handle !=3D 0b11= 1) or + * out-of line. + * + * 2. For the inline case, treat the handle itself as the input buffer. + * + * 3. For the out-of-line case, look at bits 2..0 of the handle to underst= and + * the input buffer length. To obtain the pointer to the input buffer, = unset + * bits 2..0 of the handle and set bit 63. + * + * 4. If the input buffer is 128 byte long, copy its contents to the output + * buffer. + * + * 5. Otherwise, read @largest_idx, @r_tags and @r_sizes from the input bu= ffer. + * Calculate the removed largest element of @r_sizes: + * largest =3D 256 - sum(r_sizes) + * and insert it into @r_sizes at position @largest_idx. + * + * 6. While @r_sizes[i] > 0, add a 4-bit value @r_tags[i] to the output bu= ffer + * @r_sizes[i] times. + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +/* The handle must fit into an Xarray value. */ +#define HANDLE_MASK ~(BIT_ULL(63)) + +/* Out-of-line handles have 0b111 in bits 62..60. */ +#define NOINLINE_MASK (BIT_ULL(62) | BIT_ULL(61) | BIT_ULL(60)) + +/* Cache index is stored in the lowest pointer bits. */ +#define CACHE_ID_MASK (BIT_ULL(2) | BIT_ULL(1) | BIT_ULL(0)) + +/* Four separate caches to store out-of-line data. */ +#define NUM_CACHES 4 +static struct kmem_cache *mtecomp_caches[NUM_CACHES]; + +/* Translate allocation size into mtecomp_caches[] index. */ +static int ea0_size_to_cache_id(int len) +{ + switch (len) { + case 16: + return 1; + case 32: + return 2; + case 64: + return 3; + default: + return 0; + } +} + +/* Translate mtecomp_caches[] index into allocation size. */ +static int ea0_cache_id_to_size(int id) +{ + switch (id) { + case 1: + return 16; + case 2: + return 32; + case 3: + return 64; + default: + return 128; + } +} + +/* Transform tags into tag ranges. */ +void ea0_tags_to_ranges(u8 *tags, u8 *out_tags, short *out_sizes, int *out= _len) +{ + u8 prev_tag =3D 0xff; + int cur_idx =3D -1; + u8 cur_tag; + int i; + + memset(out_tags, 0, *out_len * sizeof(*out_tags)); + memset(out_sizes, 0, *out_len * sizeof(*out_sizes)); + for (i =3D 0; i < MTE_GRANULES_PER_PAGE; i++) { + cur_tag =3D tags[i / 2]; + if (i % 2) + cur_tag =3D cur_tag % 16; + else + cur_tag =3D cur_tag / 16; + if (cur_tag =3D=3D prev_tag) { + out_sizes[cur_idx]++; + } else { + cur_idx++; + prev_tag =3D cur_tag; + out_tags[cur_idx] =3D prev_tag; + out_sizes[cur_idx] =3D 1; + } + } + *out_len =3D cur_idx + 1; +} + +/* Transform tag ranges back into tags. */ +void ea0_ranges_to_tags(u8 *r_tags, short *r_sizes, int r_len, u8 *tags) +{ + struct bitq iter; + int i, j; + + bitq_init(&iter, tags, 128); + for (i =3D 0; i < r_len; i++) { + for (j =3D 0; j < r_sizes[i]; j++) + bitq_enqueue(&iter, r_tags[i], 4); + } +} + +/* Translate @num_ranges into the allocation size needed to hold them. */ +static int ea0_alloc_size(int num_ranges) +{ + if (num_ranges <=3D 6) + return 8; + if (num_ranges <=3D 11) + return 16; + if (num_ranges <=3D 23) + return 32; + if (num_ranges <=3D 46) + return 64; + return 128; +} + +/* Translate allocation size into maximum number of ranges that it can hol= d. */ +static int ea0_size_to_ranges(int size) +{ + switch (size) { + case 8: + return 6; + case 16: + return 11; + case 32: + return 23; + case 64: + return 46; + default: + return 0; + } +} + +/* Is the data stored inline in the handle itself? */ +static bool ea0_is_inline(u64 handle) +{ + return (handle & NOINLINE_MASK) !=3D NOINLINE_MASK; +} + +/* Get the size of the buffer backing @handle. */ +int ea0_storage_size(u64 handle) +{ + if (ea0_is_inline(handle)) + return 8; + return ea0_cache_id_to_size(handle & CACHE_ID_MASK); +} +EXPORT_SYMBOL(ea0_storage_size); + +/* Compress ranges into the buffer of the given length. */ +void ea0_compress_to_buf(int len, u8 *tags, short *sizes, u8 *buf, int buf= len) +{ + int largest_idx =3D -1, i; + short largest =3D 0; + struct bitq iter; + + bitq_init(&iter, buf, buflen); + for (i =3D 0; i < len; i++) { + if (i =3D=3D len) + break; + if (sizes[i] > largest) { + largest =3D sizes[i]; + largest_idx =3D i; + } + } + if (len <=3D 6) + /* Inline case, @buflen <=3D 8. */ + bitq_enqueue(&iter, largest_idx, 4); + else + bitq_enqueue(&iter, largest_idx, 6); + for (i =3D 0; i < len; i++) + bitq_enqueue(&iter, tags[i], 4); + for (i =3D len; i < ea0_size_to_ranges(buflen); i++) + bitq_enqueue(&iter, 0, 4); + for (i =3D 0; i < len; i++) { + if (i =3D=3D largest_idx) + continue; + bitq_enqueue(&iter, sizes[i], 7); + } +} + +/* Compress the data inline. */ +static u64 ea0_compress_inline(int len, u8 *tags, short *sizes) +{ + u64 result; + + ea0_compress_to_buf(len, tags, sizes, (u8 *)&result, sizeof(result)); + result =3D be64_to_cpu(result); + return result; +} + +/* Compress @tags and return a handle. */ +u64 ea0_compress(u8 *tags) +{ + int alloc_size, cache_id; + struct kmem_cache *cache; + short r_sizes[256]; + u8 r_tags[256]; + int r_len =3D ARRAY_SIZE(r_tags); + u8 *storage; + + ea0_tags_to_ranges(tags, r_tags, r_sizes, &r_len); + alloc_size =3D ea0_alloc_size(r_len); + if (alloc_size =3D=3D 8) + return ea0_compress_inline(r_len, r_tags, r_sizes); + cache_id =3D ea0_size_to_cache_id(alloc_size); + cache =3D mtecomp_caches[cache_id]; + storage =3D kmem_cache_alloc(cache, GFP_KERNEL); + if (alloc_size < 128) { + ea0_compress_to_buf(r_len, r_tags, r_sizes, storage, + alloc_size); + return ((u64)storage | cache_id) & HANDLE_MASK; + } + memcpy(storage, tags, alloc_size); + return (u64)storage & HANDLE_MASK; +} + +/* Decompress the contents of the given buffer into @tags. */ +static bool ea0_decompress_from_buf(u8 *buf, int buflen, u8 *tags) +{ + int bits, largest_idx, i, r_len =3D ea0_size_to_ranges(buflen); + short r_sizes[46], sum =3D 0; + u8 r_tags[46]; + struct bitq iter; + u8 val; + + bitq_init_full(&iter, buf, buflen); + bits =3D bitq_dequeue(&iter, &val, (buflen =3D=3D 8) ? 4 : 6); + if (bits =3D=3D -1) + return false; + largest_idx =3D val; + for (i =3D 0; i < r_len; i++) { + bits =3D bitq_dequeue(&iter, &val, 4); + if (bits =3D=3D -1) + return false; + r_tags[i] =3D val; + } + for (i =3D 0; i < r_len; i++) { + if (i =3D=3D largest_idx) + continue; + bits =3D bitq_dequeue(&iter, &val, 7); + if (bits =3D=3D -1) + return false; + if (!val) { + r_len =3D i; + break; + } + r_sizes[i] =3D val; + sum +=3D val; + } + if (sum >=3D 256) + return false; + r_sizes[largest_idx] =3D 256 - sum; + ea0_ranges_to_tags(r_tags, r_sizes, r_len, tags); + return true; +} + +/* Get pointer to the out-of-line storage from a handle. */ +static void *ea0_storage(u64 handle) +{ + if (ea0_is_inline(handle)) + return NULL; + return (void *)((handle & (~CACHE_ID_MASK)) | BIT_ULL(63)); +} + +/* Decompress tags from the buffer referenced by @handle. */ +bool ea0_decompress(u64 handle, u8 *tags) +{ + u8 *storage =3D ea0_storage(handle); + int size =3D ea0_storage_size(handle); + + if (size =3D=3D 128) { + memcpy(tags, storage, size); + return true; + } + if (size =3D=3D 8) { + handle =3D cpu_to_be64(handle); + return ea0_decompress_from_buf((u8 *)&handle, sizeof(handle), + tags); + } + return ea0_decompress_from_buf(storage, size, tags); +} +EXPORT_SYMBOL(ea0_decompress); + +/* Release the memory referenced by @handle. */ +void ea0_release_handle(u64 handle) +{ + void *storage =3D ea0_storage(handle); + int size =3D ea0_storage_size(handle); + struct kmem_cache *c; + + if (!handle || !storage) + return; + + c =3D mtecomp_caches[ea0_size_to_cache_id(size)]; + kmem_cache_free(c, storage); +} +EXPORT_SYMBOL(ea0_release_handle); + +/* Set up mtecomp_caches[]. */ +static int mtecomp_init(void) +{ + char name[16]; + int size; + int i; + + for (i =3D 0; i < NUM_CACHES; i++) { + size =3D ea0_cache_id_to_size(i); + snprintf(name, ARRAY_SIZE(name), "mte-tags-%d", size); + mtecomp_caches[i] =3D + kmem_cache_create(name, size, size, 0, NULL); + } + return 0; +} + +module_init(mtecomp_init); --=20 2.41.0.255.g8b1d071c50-goog From nobody Mon Feb 9 05:53:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D4A2C0015E for ; Tue, 11 Jul 2023 14:34:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232333AbjGKOeS (ORCPT ); Tue, 11 Jul 2023 10:34:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43088 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233242AbjGKOeG (ORCPT ); Tue, 11 Jul 2023 10:34:06 -0400 Received: from mail-ed1-x54a.google.com (mail-ed1-x54a.google.com [IPv6:2a00:1450:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 261B71712 for ; Tue, 11 Jul 2023 07:33:59 -0700 (PDT) Received: by mail-ed1-x54a.google.com with SMTP id 4fb4d7f45d1cf-51dd16f823bso3914674a12.0 for ; Tue, 11 Jul 2023 07:33:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689086037; x=1691678037; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=i17FLoNCi12ib3F8C35t83wNdppeBZG3dj7HY/hu0Qg=; b=bF7ks1NOiP3BF1aqWMehZ/RymXHgICxwrRGh0fmfl8YsDQilu2nmGiaXt7lq7KHWZ/ e9N/L9vmQVA2AkmiBicDeuxKhtL0clapuMiVkJRboNpfFhF0gnkJtLlA5nH221cM06C8 w8ghPuSDsktDOO4BW/h6HGUuU2I56zCEwXmGcY8/d5lUyTen5sfwTBWKA0JKu2d1nbhU OhUNrvMLxhBq2k+psa7klp3nulrB2kSJtBx2vVJ6EgDgszJ4V5GX848W2L0X5CZxd1FI i2VCWpnJ7RjEds/XnyZFZeXfg0WKbqcobeZg8o+VHatLpaakqsxWcz13vMxE+PonafDG DLeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689086037; x=1691678037; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=i17FLoNCi12ib3F8C35t83wNdppeBZG3dj7HY/hu0Qg=; b=bdJnqriFXscCG5FGz7tFPLW8v0mLDhTA0AhFdLAzRMKImmjw9eyr7MPUAErDSUNTvs qo+Z+Fh09+mnGrLR0V8/Vp/3HRmHPFpCKIIEwicEyZZKmu/BproKceLGkSX73Xu2PlsQ 3veaEZ1OKibMnWbA7h4AupExkpFgfDi8s5mKKMr4UCeLRrVxqWzvLOcccB66L4tpAbaO eIFWR7RSTwlOBGIYyCLQGqSFDO5GqB6AIXZgFlQ2laTVSX6jR4g7Sfs9pJEhQhcjPt3r RtnrCwl8YLhbTwXNpIkfrQbBl+8JepnsJ1DII+/NsiVM81XdNCbZxNuxHR41E0BuVYjr CrEQ== X-Gm-Message-State: ABy/qLaxAqPFwtk+pS87RgWx6nunGnjfjAXuWlZzc2bAIcUz3ddkme52 NUbO+/LgYRVAzh/QOypEU1ztgt+4xHc= X-Google-Smtp-Source: APBJJlGNWFJLgtlulnhBeOMdEwDi3KGdL3AjhF9WjdYEy0zcia1bCw7YygoOrzA5eHxyMc8FxCdKjfKZ7z0= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:564d:3aaa:6b5f:4419]) (user=glider job=sendgmr) by 2002:a50:a6c9:0:b0:51e:2514:309d with SMTP id f9-20020a50a6c9000000b0051e2514309dmr75741edc.7.1689086037613; Tue, 11 Jul 2023 07:33:57 -0700 (PDT) Date: Tue, 11 Jul 2023 16:33:31 +0200 In-Reply-To: <20230711143337.3086664-1-glider@google.com> Mime-Version: 1.0 References: <20230711143337.3086664-1-glider@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230711143337.3086664-5-glider@google.com> Subject: [PATCH 2/5] linux/bitqueue.h: add a KUnit test for bitqueue.h From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, yury.norov@gmail.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add tests checking that struct bitq correctly handles sub-byte values. Signed-off-by: Alexander Potapenko --- lib/Kconfig.debug | 8 ++ lib/Makefile | 1 + lib/test_bitqueue.c | 244 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 253 insertions(+) create mode 100644 lib/test_bitqueue.c diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index ce51d4dc6803e..a6598b2c250d5 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2678,6 +2678,14 @@ config SIPHASH_KUNIT_TEST This is intended to help people writing architecture-specific optimized versions. If unsure, say N. =20 +config BITQUEUE_KUNIT_TEST + tristate "Test " if !KUNIT_ALL_TESTS + depends on KUNIT + default KUNIT_ALL_TESTS + help + Enable this option to test the kernel's bit queue implementation + (). + config TEST_UDELAY tristate "udelay test driver" help diff --git a/lib/Makefile b/lib/Makefile index 876fcdeae34ec..7efb6aba31cf9 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -394,6 +394,7 @@ CFLAGS_fortify_kunit.o +=3D $(DISABLE_STRUCTLEAK_PLUGIN) obj-$(CONFIG_FORTIFY_KUNIT_TEST) +=3D fortify_kunit.o obj-$(CONFIG_STRSCPY_KUNIT_TEST) +=3D strscpy_kunit.o obj-$(CONFIG_SIPHASH_KUNIT_TEST) +=3D siphash_kunit.o +obj-$(CONFIG_BITQUEUE_KUNIT_TEST) +=3D test_bitqueue.o =20 obj-$(CONFIG_GENERIC_LIB_DEVMEM_IS_ALLOWED) +=3D devmem_is_allowed.o =20 diff --git a/lib/test_bitqueue.c b/lib/test_bitqueue.c new file mode 100644 index 0000000000000..aec04b3a5f068 --- /dev/null +++ b/lib/test_bitqueue.c @@ -0,0 +1,244 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Test cases for struct bitq, a simple bit queue. + */ + +#include +#include +#include + +/* Set up a bit queue containing @size bytes. */ +static void bitq_setup(struct bitq *it, size_t size) +{ + u8 *data =3D kmalloc(size, GFP_KERNEL); + + bitq_init(it, data, size); +} + +/* Tear down the bit queue. */ +static void bitq_teardown(struct bitq *it) +{ + kfree(it->data); + memset(it, 0, sizeof(*it)); +} + +/* Test that nothing can be popped from an empty queue. */ +static void test_empty(struct kunit *test) +{ + struct bitq it; + u8 val =3D 0; + + /* Allocate a two-byte queue. */ + bitq_setup(&it, 2); + + /* Queue is empty. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 8), -1); + bitq_teardown(&it); +} + +/* Test that simple byte-granular enqueue/dequeue operations work. */ +static void test_basic_enqueue_dequeue(struct kunit *test) +{ + struct bitq it; + u8 val =3D 0; + + /* Allocate a two-byte queue. */ + bitq_setup(&it, 2); + /* Enqueue two 8-bit values. */ + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0xaa, 8), 8); + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0xbb, 8), 8); + /* Cannot enqueue the third byte. */ + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 1, 8), -1); + /* Dequeue two bytes. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 8), 8); + KUNIT_EXPECT_EQ(test, val, 0xaa); + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 8), 8); + KUNIT_EXPECT_EQ(test, val, 0xbb); + + /* Queue is empty. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 8), -1); + bitq_teardown(&it); +} + +/* Test that values shorter than 8 bits can be enqueued and dequeued. */ +static void test_shorter_than_byte(struct kunit *test) +{ + struct bitq it; + u8 val =3D 0; + + /* Allocate a two-byte queue. */ + bitq_setup(&it, 2); + /* Enqueue two 0b101 values. */ + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0b101, 3), 3); + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0b101, 3), 3); + /* The first byte of the queue is now 0b10110100. */ + + /* Now dequeue three 2-bit values: 0b10, 0b11, 0b01. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 2), 2); + KUNIT_EXPECT_EQ(test, val, 0b10); + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 2), 2); + KUNIT_EXPECT_EQ(test, val, 0b11); + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 2), 2); + KUNIT_EXPECT_EQ(test, val, 0b01); + + /* Queue is empty. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 1), -1); + bitq_teardown(&it); +} + +/* Test that bits are carried over correctly if they do not fit. */ +static void test_carryover(struct kunit *test) +{ + struct bitq it; + u8 val =3D 0; + int i; + + /* Allocate a three-byte queue. */ + bitq_setup(&it, 3); + /* Enqueue 0b100 seven times. */ + for (i =3D 0; i < 7; i++) + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0b100, 3), 3); + /* Now dequeue three 7-bit values: 0b1001001, 0b0010010, 0b0100100. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 7), 7); + KUNIT_EXPECT_EQ(test, val, 0b1001001); + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 7), 7); + KUNIT_EXPECT_EQ(test, val, 0b0010010); + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 7), 7); + KUNIT_EXPECT_EQ(test, val, 0b0100100); + + /* Queue is empty. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 1), -1); + bitq_teardown(&it); +} + +/* + * Test case extracted from the EA0 tag compression algorithm, where + * carried over bits were accidentally written into the previous byte. + */ +static void test_carryover_ea0(struct kunit *test) +{ + struct bitq it; + u8 val =3D 0; + + /* Allocate a three-byte queue. */ + bitq_setup(&it, 3); + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0b100, 3), 3); + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0b1010, 4), 4); + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0b0000, 4), 4); + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0b1010, 4), 4); + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0b1011, 4), 4); + + /* Now dequeue two byte values: 0b10010100, 0b00010101. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 8), 8); + KUNIT_EXPECT_EQ(test, val, 0b10010100); + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 8), 8); + KUNIT_EXPECT_EQ(test, val, 0b00010101); + /* And the remaining 0b011. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 3), 3); + KUNIT_EXPECT_EQ(test, val, 0b011); + + /* Queue is empty. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 1), -1); + bitq_teardown(&it); +} + +/* Test that upper bits of the pushed value are discarded. */ +static void test_trim_upper_bits(struct kunit *test) +{ + struct bitq it; + u8 val =3D 0; + + /* Allocate a two-byte queue. */ + bitq_setup(&it, 2); + /* Enqueue two values that do not fit into 4 bits. */ + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0xab, 4), 4); + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0xab, 4), 4); + /* The first byte of the queue is now 0xbb. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 8), 8); + KUNIT_EXPECT_EQ(test, val, 0xbb); + + /* Queue is empty. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 1), -1); + bitq_teardown(&it); +} + +/* Another test for discarding the upper bits. */ +static void test_trim_upper_bits2(struct kunit *test) +{ + struct bitq it; + u8 val =3D 0; + + /* Allocate a two-byte queue. */ + bitq_setup(&it, 2); + /* Push seven zero bits. */ + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0, 7), 7); + /* Push a single 1 bit, but pass a bigger value to bitq_enqueue(). */ + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0xff, 1), 1); + /* The first byte of the queue is now 0x01. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 8), 8); + KUNIT_EXPECT_EQ(test, val, 0x01); + + /* Queue is empty. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 1), -1); + bitq_teardown(&it); +} + +/* Test that a NULL value can be used as output of bitq_dequeue() */ +static void test_dequeue_to_null(struct kunit *test) +{ + struct bitq it; + + /* Allocate a two-byte queue. */ + bitq_setup(&it, 2); + /* Enqueue a byte value. */ + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 0xab, 8), 8); + /* Dequeue the byte, but discard its value. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, NULL, 8), 8); + + /* Queue is empty. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, NULL, 1), -1); + bitq_teardown(&it); +} + +/* Test that bitq_init_full works. */ +static void test_init_full(struct kunit *test) +{ + struct bitq it; + u8 data[2] =3D { 0xaa, 0xbb }; + u8 val =3D 0; + + /* Initialize a queue with the contents of @data */ + bitq_init_full(&it, data, 2); + /* Cannot enqueue anything else. */ + KUNIT_EXPECT_EQ(test, bitq_enqueue(&it, 1, 8), -1); + /* Dequeue two bytes. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 8), 8); + KUNIT_EXPECT_EQ(test, val, 0xaa); + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, &val, 8), 8); + KUNIT_EXPECT_EQ(test, val, 0xbb); + + /* Queue is empty. */ + KUNIT_EXPECT_EQ(test, bitq_dequeue(&it, NULL, 1), -1); +} + +static struct kunit_case bitq_test_cases[] =3D { + KUNIT_CASE(test_empty), + KUNIT_CASE(test_basic_enqueue_dequeue), + KUNIT_CASE(test_shorter_than_byte), + KUNIT_CASE(test_carryover), + KUNIT_CASE(test_carryover_ea0), + KUNIT_CASE(test_trim_upper_bits), + KUNIT_CASE(test_trim_upper_bits2), + KUNIT_CASE(test_dequeue_to_null), + KUNIT_CASE(test_init_full), + {} +}; + +static struct kunit_suite bitq_test_suite =3D { + .name =3D "bitq", + .test_cases =3D bitq_test_cases, +}; +kunit_test_suites(&bitq_test_suite); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Alexander Potapenko "); --=20 2.41.0.255.g8b1d071c50-goog From nobody Mon Feb 9 05:53:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 723B7EB64DC for ; Tue, 11 Jul 2023 14:34:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233554AbjGKOeZ (ORCPT ); Tue, 11 Jul 2023 10:34:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43314 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233372AbjGKOeR (ORCPT ); Tue, 11 Jul 2023 10:34:17 -0400 Received: from mail-ed1-x549.google.com (mail-ed1-x549.google.com [IPv6:2a00:1450:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E64C61981 for ; Tue, 11 Jul 2023 07:34:04 -0700 (PDT) Received: by mail-ed1-x549.google.com with SMTP id 4fb4d7f45d1cf-51bee352ffcso3769817a12.1 for ; Tue, 11 Jul 2023 07:34:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689086043; x=1691678043; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/3xF8MGaf3YIHiv8WE5XhXRUvL32mDr7FE1dMJNOszw=; b=AZhcGN9+YgQGxT0b+gWryU1J+8KQz6Y3m8GxVKyhLqmKrQPeSmTOlv7IV78hSmnuil 8hfb7BDJZkzcpehUV/JCvu+xj6PGW2eClHWg513tiOz0LuIxqgLRkQLSKWMXIj/8PX3I B5EKn4EyYjv+hMI20N7MHjfngg+uZIZsAqvQr6EBKFmUOIeZkeu5r4tA4Aa52Jrwqsqa 5gKzwFYwFqXlEdg9Ufe8GFoiYuXL+/tItEP9dFjSoLBKIl17UxpxodYwLLe2PFyJQBRa JuKnV8cCcuEeAN5BqUDcMao02PwLtuwogColw8cCBpKBm/lWCRwkLdJauavmkgHMIjPb 4drg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689086043; x=1691678043; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/3xF8MGaf3YIHiv8WE5XhXRUvL32mDr7FE1dMJNOszw=; b=SLJRB4UuqaoPAG1SRS+7CJHqJRYRXWWUpBbCWCvbTFGG82nw9HKIwEexu/E+p3YVwH G1N2MZ5eaPc7ssmaWEcPr/Kb7KcMQQGSNn++KdGu9sUayDMfPLHk3trhZQiY9Uva+T+c pX+LzwQ8wGqWXogE+kr9e6DynVgVSsfhT3MFXqwgZdRb/+eo8sfF7z8/R2CEUqr2fHbj UUrMwl+wjxdm2MZIKIF6geqALPId2nCuimsAh/m/Z2LYw5mVYev6DSwkFI0wS6c+G1gq Up7aUv0K0sAQpMDu7Fe/PbAVv+vAyF/Ro00oP3VTr/VJWH5K9Q804nwfUHzltca81525 hNBA== X-Gm-Message-State: ABy/qLbWmGm2XZNI2CQDr/RzHufShE0pXROSCkdqSY1uTSEFyh1gyoqP XCwbtryVPT6nIhcOazAtaiBvSnsiKrY= X-Google-Smtp-Source: APBJJlGW5VcJHVTqJwpQ3fDS1Mx4H3SOEP8K0v5y+Mrm5kTD3AGvEuguwea+HFIg1ODLgDlDedLbfTXStZ0= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:564d:3aaa:6b5f:4419]) (user=glider job=sendgmr) by 2002:a50:d685:0:b0:51d:e3e6:cc6b with SMTP id r5-20020a50d685000000b0051de3e6cc6bmr77850edi.6.1689086042901; Tue, 11 Jul 2023 07:34:02 -0700 (PDT) Date: Tue, 11 Jul 2023 16:33:33 +0200 In-Reply-To: <20230711143337.3086664-1-glider@google.com> Mime-Version: 1.0 References: <20230711143337.3086664-1-glider@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230711143337.3086664-7-glider@google.com> Subject: [PATCH 3/5] arm64: mte: implement CONFIG_ARM64_MTE_COMP From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, yury.norov@gmail.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The config implements the EA0 algorithm suggested by Evgenii Stepanov to compress the memory tags for ARM MTE during swapping. The algorithm is based on RLE and specifically targets 128-byte buffers of tags corresponding to a single page. In the common case a buffer can be compressed into 63 bits, making it possible to store it without additional memory allocation. Suggested-by: Evgenii Stepanov Signed-off-by: Alexander Potapenko --- arch/arm64/Kconfig | 10 + arch/arm64/include/asm/mtecomp.h | 60 +++++ arch/arm64/mm/Makefile | 1 + arch/arm64/mm/mtecomp.c | 398 +++++++++++++++++++++++++++++++ 4 files changed, 469 insertions(+) create mode 100644 arch/arm64/include/asm/mtecomp.h create mode 100644 arch/arm64/mm/mtecomp.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 343e1e1cae10a..b25b584a0a9cb 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2065,6 +2065,16 @@ config ARM64_EPAN if the cpu does not implement the feature. endmenu # "ARMv8.7 architectural features" =20 +config ARM64_MTE_COMP + bool "Tag compression for ARM64 MTE" + default y + depends on ARM64_MTE + help + Enable tag compression support for ARM64 MTE. + + 128-byte tag buffers corresponding to 4K pages can be compressed using + the EA0 algorithm to save heap memory. + config ARM64_SVE bool "ARM Scalable Vector Extension support" default y diff --git a/arch/arm64/include/asm/mtecomp.h b/arch/arm64/include/asm/mtec= omp.h new file mode 100644 index 0000000000000..65a3730cc50d9 --- /dev/null +++ b/arch/arm64/include/asm/mtecomp.h @@ -0,0 +1,60 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASM_MTECOMP_H +#define __ASM_MTECOMP_H + +#include + +/* + * ea0_compress() - compress the given tag array. + * @tags: 128-byte array to read the tags from. + * + * Compresses the tags and returns a 64-bit opaque handle pointing to the + * tag storage. May allocate memory, which is freed by @ea0_release_handle= (). + */ +u64 ea0_compress(u8 *tags); + +/* + * ea0_decompress() - decompress the tag array addressed by the handle. + * @handle: handle returned by @ea0_decompress() + * @tags: 128-byte array to write the tags to. + * + * Reads the compressed data and writes it into the user-supplied tag arra= y. + * Returns true on success, false on error. + */ +bool ea0_decompress(u64 handle, u8 *tags); + +/* + * ea0_release_handle() - release the handle returned by ea0_compress(). + * @handle: handle returned by ea0_compress(). + */ +void ea0_release_handle(u64 handle); + +/* Functions below are exported for testing purposes. */ + +/* + * ea0_storage_size() - calculate the memory occupied by compressed tags. + * @handle: storage handle returned by ea0_compress. + */ +int ea0_storage_size(u64 handle); + +/* + * ea0_tags_to_ranges() - break @tags into arrays of tag ranges. + * @tags: 128-byte array containing 256 MTE tags. + * @out_tags: u8 array to store the tag of every range. + * @out_sizes: u16 array to store the size of every range. + * @out_len: length of @out_tags and @out_sizes (output parameter, initial= ly + * equal to lengths of out_tags[] and out_sizes[]). + */ +void ea0_tags_to_ranges(u8 *tags, u8 *out_tags, short *out_sizes, int *out= _len); + +/* + * ea0_ranges_to_tags() - fill @tags using given tag ranges. + * @r_tags: u8[256] containing the tag of every range. + * @r_sizes: u16[256] containing the size of every range. + * @r_len: length of @r_tags and @r_sizes. + * @tags: 128-byte array to write the tags to. + */ +void ea0_ranges_to_tags(u8 *r_tags, short *r_sizes, int r_len, u8 *tags); + +#endif // __ASM_MTECOMP_H diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index dbd1bc95967d0..46778f6dd83c2 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -10,6 +10,7 @@ obj-$(CONFIG_TRANS_TABLE) +=3D trans_pgd.o obj-$(CONFIG_TRANS_TABLE) +=3D trans_pgd-asm.o obj-$(CONFIG_DEBUG_VIRTUAL) +=3D physaddr.o obj-$(CONFIG_ARM64_MTE) +=3D mteswap.o +obj-$(CONFIG_ARM64_MTE_COMP) +=3D mtecomp.o KASAN_SANITIZE_physaddr.o +=3D n =20 obj-$(CONFIG_KASAN) +=3D kasan_init.o diff --git a/arch/arm64/mm/mtecomp.c b/arch/arm64/mm/mtecomp.c new file mode 100644 index 0000000000000..01f7d22665b49 --- /dev/null +++ b/arch/arm64/mm/mtecomp.c @@ -0,0 +1,398 @@ +// SPDX-License-Identifier: GPL-2.0-only + +/* + * MTE tag compression algorithm. + * Proposed by Evgenii Stepanov + */ + +/* + * EA0 stands for "Evgenii's Algorithm 0", as the initial proposal contain= ed two + * compression algorithms. + * + * The algorithm attempts to compress a 128-byte (MTE_GRANULES_PER_PAGE / = 2) + * array of tags into a smaller byte sequence that can be stored in a + * 16-, 32-, or 64-byte buffer. A special case is storing the tags inline = in + * an 8-byte pointer. + * + * We encapsulate tag storage memory management in this module, because it= is + * tightly coupled with the pointer representation. + * ea0_compress(*tags) takes a 128-byte buffer and returns an opaque val= ue + * that can be stored in Xarray + * ea_decompress(*ptr, *tags) takes the opaque value and loads the tags = into + * the provided 128-byte buffer. + * + * + * + * The compression algorithm works as follows. + * + * 1. The input array of 128 bytes is transformed into tag ranges (two arr= ays: + * @r_tags containing tag values and @r_sizes containing range lengths)= by + * ea0_tags_to_ranges(). Note that @r_sizes sums up to 256. + * + * 2. Depending on the number N of ranges, the following storage class is = picked: + * N <=3D 6: 8 bytes (inline case, no allocation required); + * 6 < N <=3D 11: 16 bytes + * 11 < N <=3D 23: 32 bytes + * 23 < N <=3D 46: 64 bytes + * 46 < N: 128 bytes (no compression will be performed) + * + * 3. The number of the largest element of @r_sizes is stored in @largest_= idx. + * The element itself is thrown away from @r_sizes, because it can be + * reconstructed from the sum of the remaining elements. Note that now = none + * of the remaining @r_sizes elements is greater than 127. + * + * 4. For the inline case, the following values are stored in the 8-byte h= andle: + * largest_idx : i4 + * r_tags[0..5] : i4 x 6 + * r_sizes[0..4] : i7 x 5 + * (if N is less than 6, @r_tags and @r_sizes are padded up with zero v= alues) + * + * Because @largest_idx is <=3D 5, bit 63 of the handle is always 0 (so= it can + * be stored in the Xarray), and bits 62..60 cannot all be 1, so it can= be + * distinguished from a kernel pointer. + * + * 5. For the out-of-line case, the storage is allocated from one of the + * "mte-tags-{16,32,64,128}" kmem caches. The resulting pointer is alig= ned + * on 8 bytes, so its bits 2..0 can be used to store the size class: + * - 0 for 128 bytes + * - 1 for 16 + * - 2 for 32 + * - 4 for 64. + * Bit 63 of the pointer is zeroed out, so that it can be stored in Xar= ray. + * + * 6. The data layout in the allocated storage is as follows: + * largest_idx : i6 + * r_tags[0..N] : i4 x N + * r_sizes[0..N-1] : i7 x (N-1) + * + * + * + * The decompression algorithm performs the steps below. + * + * 1. Decide if data is stored inline (bits 62..60 of the handle !=3D 0b11= 1) or + * out-of line. + * + * 2. For the inline case, treat the handle itself as the input buffer. + * + * 3. For the out-of-line case, look at bits 2..0 of the handle to underst= and + * the input buffer length. To obtain the pointer to the input buffer, = unset + * bits 2..0 of the handle and set bit 63. + * + * 4. If the input buffer is 128 byte long, copy its contents to the output + * buffer. + * + * 5. Otherwise, read @largest_idx, @r_tags and @r_sizes from the input bu= ffer. + * Calculate the removed largest element of @r_sizes: + * largest =3D 256 - sum(r_sizes) + * and insert it into @r_sizes at position @largest_idx. + * + * 6. While @r_sizes[i] > 0, add a 4-bit value @r_tags[i] to the output bu= ffer + * @r_sizes[i] times. + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +/* The handle must fit into an Xarray value. */ +#define HANDLE_MASK ~(BIT_ULL(63)) + +/* Out-of-line handles have 0b111 in bits 62..60. */ +#define NOINLINE_MASK (BIT_ULL(62) | BIT_ULL(61) | BIT_ULL(60)) + +/* Cache index is stored in the lowest pointer bits. */ +#define CACHE_ID_MASK (BIT_ULL(2) | BIT_ULL(1) | BIT_ULL(0)) + +/* Four separate caches to store out-of-line data. */ +#define NUM_CACHES 4 +static struct kmem_cache *mtecomp_caches[NUM_CACHES]; + +/* Translate allocation size into mtecomp_caches[] index. */ +static int ea0_size_to_cache_id(int len) +{ + switch (len) { + case 16: + return 1; + case 32: + return 2; + case 64: + return 3; + default: + return 0; + } +} + +/* Translate mtecomp_caches[] index into allocation size. */ +static int ea0_cache_id_to_size(int id) +{ + switch (id) { + case 1: + return 16; + case 2: + return 32; + case 3: + return 64; + default: + return 128; + } +} + +/* Transform tags into tag ranges. */ +void ea0_tags_to_ranges(u8 *tags, u8 *out_tags, short *out_sizes, int *out= _len) +{ + u8 prev_tag =3D 0xff; + int cur_idx =3D -1; + u8 cur_tag; + int i; + + memset(out_tags, 0, *out_len * sizeof(*out_tags)); + memset(out_sizes, 0, *out_len * sizeof(*out_sizes)); + for (i =3D 0; i < MTE_GRANULES_PER_PAGE; i++) { + cur_tag =3D tags[i / 2]; + if (i % 2) + cur_tag =3D cur_tag % 16; + else + cur_tag =3D cur_tag / 16; + if (cur_tag =3D=3D prev_tag) { + out_sizes[cur_idx]++; + } else { + cur_idx++; + prev_tag =3D cur_tag; + out_tags[cur_idx] =3D prev_tag; + out_sizes[cur_idx] =3D 1; + } + } + *out_len =3D cur_idx + 1; +} + +/* Transform tag ranges back into tags. */ +void ea0_ranges_to_tags(u8 *r_tags, short *r_sizes, int r_len, u8 *tags) +{ + struct bitq iter; + int i, j; + + bitq_init(&iter, tags, 128); + for (i =3D 0; i < r_len; i++) { + for (j =3D 0; j < r_sizes[i]; j++) + bitq_enqueue(&iter, r_tags[i], 4); + } +} + +/* Translate @num_ranges into the allocation size needed to hold them. */ +static int ea0_alloc_size(int num_ranges) +{ + if (num_ranges <=3D 6) + return 8; + if (num_ranges <=3D 11) + return 16; + if (num_ranges <=3D 23) + return 32; + if (num_ranges <=3D 46) + return 64; + return 128; +} + +/* Translate allocation size into maximum number of ranges that it can hol= d. */ +static int ea0_size_to_ranges(int size) +{ + switch (size) { + case 8: + return 6; + case 16: + return 11; + case 32: + return 23; + case 64: + return 46; + default: + return 0; + } +} + +/* Is the data stored inline in the handle itself? */ +static bool ea0_is_inline(u64 handle) +{ + return (handle & NOINLINE_MASK) !=3D NOINLINE_MASK; +} + +/* Get the size of the buffer backing @handle. */ +int ea0_storage_size(u64 handle) +{ + if (ea0_is_inline(handle)) + return 8; + return ea0_cache_id_to_size(handle & CACHE_ID_MASK); +} +EXPORT_SYMBOL(ea0_storage_size); + +/* Compress ranges into the buffer of the given length. */ +void ea0_compress_to_buf(int len, u8 *tags, short *sizes, u8 *buf, int buf= len) +{ + int largest_idx =3D -1, i; + short largest =3D 0; + struct bitq iter; + + bitq_init(&iter, buf, buflen); + for (i =3D 0; i < len; i++) { + if (i =3D=3D len) + break; + if (sizes[i] > largest) { + largest =3D sizes[i]; + largest_idx =3D i; + } + } + if (len <=3D 6) + /* Inline case, @buflen <=3D 8. */ + bitq_enqueue(&iter, largest_idx, 4); + else + bitq_enqueue(&iter, largest_idx, 6); + for (i =3D 0; i < len; i++) + bitq_enqueue(&iter, tags[i], 4); + for (i =3D len; i < ea0_size_to_ranges(buflen); i++) + bitq_enqueue(&iter, 0, 4); + for (i =3D 0; i < len; i++) { + if (i =3D=3D largest_idx) + continue; + bitq_enqueue(&iter, sizes[i], 7); + } +} + +/* Compress the data inline. */ +static u64 ea0_compress_inline(int len, u8 *tags, short *sizes) +{ + u64 result; + + ea0_compress_to_buf(len, tags, sizes, (u8 *)&result, sizeof(result)); + result =3D be64_to_cpu(result); + return result; +} + +/* Compress @tags and return a handle. */ +u64 ea0_compress(u8 *tags) +{ + int alloc_size, cache_id; + struct kmem_cache *cache; + short r_sizes[256]; + u8 r_tags[256]; + int r_len =3D ARRAY_SIZE(r_tags); + u8 *storage; + + ea0_tags_to_ranges(tags, r_tags, r_sizes, &r_len); + alloc_size =3D ea0_alloc_size(r_len); + if (alloc_size =3D=3D 8) + return ea0_compress_inline(r_len, r_tags, r_sizes); + cache_id =3D ea0_size_to_cache_id(alloc_size); + cache =3D mtecomp_caches[cache_id]; + storage =3D kmem_cache_alloc(cache, GFP_KERNEL); + if (alloc_size < 128) { + ea0_compress_to_buf(r_len, r_tags, r_sizes, storage, + alloc_size); + return ((u64)storage | cache_id) & HANDLE_MASK; + } + memcpy(storage, tags, alloc_size); + return (u64)storage & HANDLE_MASK; +} + +/* Decompress the contents of the given buffer into @tags. */ +static bool ea0_decompress_from_buf(u8 *buf, int buflen, u8 *tags) +{ + int bits, largest_idx, i, r_len =3D ea0_size_to_ranges(buflen); + short r_sizes[46], sum =3D 0; + u8 r_tags[46]; + struct bitq iter; + u8 val; + + bitq_init_full(&iter, buf, buflen); + bits =3D bitq_dequeue(&iter, &val, (buflen =3D=3D 8) ? 4 : 6); + if (bits =3D=3D -1) + return false; + largest_idx =3D val; + for (i =3D 0; i < r_len; i++) { + bits =3D bitq_dequeue(&iter, &val, 4); + if (bits =3D=3D -1) + return false; + r_tags[i] =3D val; + } + for (i =3D 0; i < r_len; i++) { + if (i =3D=3D largest_idx) + continue; + bits =3D bitq_dequeue(&iter, &val, 7); + if (bits =3D=3D -1) + return false; + if (!val) { + r_len =3D i; + break; + } + r_sizes[i] =3D val; + sum +=3D val; + } + if (sum >=3D 256) + return false; + r_sizes[largest_idx] =3D 256 - sum; + ea0_ranges_to_tags(r_tags, r_sizes, r_len, tags); + return true; +} + +/* Get pointer to the out-of-line storage from a handle. */ +static void *ea0_storage(u64 handle) +{ + if (ea0_is_inline(handle)) + return NULL; + return (void *)((handle & (~CACHE_ID_MASK)) | BIT_ULL(63)); +} + +/* Decompress tags from the buffer referenced by @handle. */ +bool ea0_decompress(u64 handle, u8 *tags) +{ + u8 *storage =3D ea0_storage(handle); + int size =3D ea0_storage_size(handle); + + if (size =3D=3D 128) { + memcpy(tags, storage, size); + return true; + } + if (size =3D=3D 8) { + handle =3D cpu_to_be64(handle); + return ea0_decompress_from_buf((u8 *)&handle, sizeof(handle), + tags); + } + return ea0_decompress_from_buf(storage, size, tags); +} +EXPORT_SYMBOL(ea0_decompress); + +/* Release the memory referenced by @handle. */ +void ea0_release_handle(u64 handle) +{ + void *storage =3D ea0_storage(handle); + int size =3D ea0_storage_size(handle); + struct kmem_cache *c; + + if (!handle || !storage) + return; + + c =3D mtecomp_caches[ea0_size_to_cache_id(size)]; + kmem_cache_free(c, storage); +} +EXPORT_SYMBOL(ea0_release_handle); + +/* Set up mtecomp_caches[]. */ +static int mtecomp_init(void) +{ + char name[16]; + int size; + int i; + + for (i =3D 0; i < NUM_CACHES; i++) { + size =3D ea0_cache_id_to_size(i); + snprintf(name, ARRAY_SIZE(name), "mte-tags-%d", size); + mtecomp_caches[i] =3D + kmem_cache_create(name, size, size, 0, NULL); + } + return 0; +} + +module_init(mtecomp_init); --=20 2.41.0.255.g8b1d071c50-goog From nobody Mon Feb 9 05:53:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 532A8EB64DC for ; Tue, 11 Jul 2023 14:34:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233167AbjGKOeW (ORCPT ); Tue, 11 Jul 2023 10:34:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233560AbjGKOeL (ORCPT ); Tue, 11 Jul 2023 10:34:11 -0400 Received: from mail-ed1-x549.google.com (mail-ed1-x549.google.com [IPv6:2a00:1450:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ACB4B10E7 for ; Tue, 11 Jul 2023 07:34:01 -0700 (PDT) Received: by mail-ed1-x549.google.com with SMTP id 4fb4d7f45d1cf-51836731bfbso3126702a12.3 for ; Tue, 11 Jul 2023 07:34:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689086040; x=1691678040; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6gUN4hkfqv1qmZBEmgq0ujJ8OMQMxN9ipJDwtCO0z/w=; b=ri7p9vV6wORs1YMnbVsQYCOsM+xoAZB3cF7gqHY0/TcF5npm8tbHpwX2wyWrRS5eOe cxnQFf/BJF4hXOinqViiCsfFKctuu14j0AismWmP79AnONqF62avtF9X2j2Kgx0KfPHd y2y23Qez4j3uD5gEqac0wu0yUTUNV2Vdrz6xX3ZoyX3lvrerkh7+JcGftmryZe+yxwHo jzKgxcF3xFaPgSC6kO7SKyBS5TiHWaPfb9+CunLBx5v1eVNxAyiAVBxi9tdKU2HSoUsc SdB3Ld1Ryvz/KmGIMbXAZ7M0qqYwv7XKFe+NCZaKr4WEG80RFBaN264nA/xvtXiyugeL HDJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689086040; x=1691678040; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6gUN4hkfqv1qmZBEmgq0ujJ8OMQMxN9ipJDwtCO0z/w=; b=iGkQ77KsagASh3jIeNACiWuaak6U20qBNCIEwrKJp8LizrKkOSqZNt0W45iw4YYRTs JxxKEI/91N7AM4ji05qGSs8cCS4utyqcDOVoO83bnba/bto/ibRvrWEkF8RjxH3Pmakv s8chgkmBaxf2V7qi5QYhWTb+noHr4O9KLCYXTXUgZf4PrMh9E3aPZwyxZtSkSMv46vop m0MgLs0wd3YwxWs20XVCAJnkL19FGGrOJGLHXYNifFhwlbLghQcK4orYDdIB4UozSA24 N3W4DiIku2UzfFbWXKymtAJvXnEOJv8PdF8O5rW/rcCrecH0yik2uto+SCoZno5tsEN4 Wvlg== X-Gm-Message-State: ABy/qLY1jc57xZKfobbKEYUIjAJ1GLBqIJXvRRUg0UVN0pxFH8WAp7Rb UDPVJ8eF7K+oZT4YvCb+WfAcm7cat/Q= X-Google-Smtp-Source: APBJJlEl5eZizLWdD0LG9EWoFPqQYZtN3FlxalPV8qr6GFGRoc0oBdk244stQXXlvHP/KKaa3FZUvVbs4+c= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:564d:3aaa:6b5f:4419]) (user=glider job=sendgmr) by 2002:a50:c252:0:b0:51e:1c91:3c7e with SMTP id t18-20020a50c252000000b0051e1c913c7emr82313edf.6.1689086040235; Tue, 11 Jul 2023 07:34:00 -0700 (PDT) Date: Tue, 11 Jul 2023 16:33:32 +0200 In-Reply-To: <20230711143337.3086664-1-glider@google.com> Mime-Version: 1.0 References: <20230711143337.3086664-1-glider@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230711143337.3086664-6-glider@google.com> Subject: [PATCH 3/5] arm64: mte: add a test for MTE tags compression From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, yury.norov@gmail.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Ensure that tag sequences containing alternating values are compressed to buffers of expected size and correctly decompressed afterwards. Signed-off-by: Alexander Potapenko --- arch/arm64/Kconfig | 10 ++ arch/arm64/mm/Makefile | 1 + arch/arm64/mm/test_mtecomp.c | 175 +++++++++++++++++++++++++++++++++++ 3 files changed, 186 insertions(+) create mode 100644 arch/arm64/mm/test_mtecomp.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b25b584a0a9cb..31fc50208b383 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2075,6 +2075,16 @@ config ARM64_MTE_COMP 128-byte tag buffers corresponding to 4K pages can be compressed using the EA0 algorithm to save heap memory. =20 +config ARM64_MTE_COMP_KUNIT_TEST + tristate "Test tag compression for ARM64 MTE" if !KUNIT_ALL_TESTS + default KUNIT_ALL_TESTS + depends on KUNIT && ARM64_MTE_COMP + help + Test EA0 compression algorithm enabled by CONFIG_ARM64_MTE_COMP. + + Ensure that tag sequences containing alternating values are compressed + to buffers of expected size and correctly decompressed afterwards. + config ARM64_SVE bool "ARM Scalable Vector Extension support" default y diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index 46778f6dd83c2..170dc62b010b9 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -11,6 +11,7 @@ obj-$(CONFIG_TRANS_TABLE) +=3D trans_pgd-asm.o obj-$(CONFIG_DEBUG_VIRTUAL) +=3D physaddr.o obj-$(CONFIG_ARM64_MTE) +=3D mteswap.o obj-$(CONFIG_ARM64_MTE_COMP) +=3D mtecomp.o +obj-$(CONFIG_ARM64_MTE_COMP_KUNIT_TEST) +=3D test_mtecomp.o KASAN_SANITIZE_physaddr.o +=3D n =20 obj-$(CONFIG_KASAN) +=3D kasan_init.o diff --git a/arch/arm64/mm/test_mtecomp.c b/arch/arm64/mm/test_mtecomp.c new file mode 100644 index 0000000000000..67bef6f28dac4 --- /dev/null +++ b/arch/arm64/mm/test_mtecomp.c @@ -0,0 +1,175 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Test cases for EA0, the compression algorithm for MTE tags. + */ + +#include +#include +#include +#include + +/* + * Test that ea0_tags_to_ranges() produces a single range for a zero-fille= d tag + * buffer. + */ +static void test_tags_to_ranges_zero(struct kunit *test) +{ + u8 tags[128], dtags[128]; + short r_sizes[256]; + int r_len =3D 256; + u8 r_tags[256]; + + memset(tags, 0, 128); + ea0_tags_to_ranges(tags, r_tags, r_sizes, &r_len); + KUNIT_EXPECT_EQ(test, r_len, 1); + KUNIT_EXPECT_EQ(test, r_tags[0], 0); + KUNIT_EXPECT_EQ(test, r_sizes[0], 256); + ea0_ranges_to_tags(r_tags, r_sizes, r_len, dtags); + KUNIT_EXPECT_EQ(test, memcmp(tags, dtags, 128), 0); +} + +/* + * Test that a small number of different tags is correctly transformed into + * ranges. + */ +static void test_tags_to_ranges_simple(struct kunit *test) +{ + u8 tags[128], dtags[128]; + const u8 ex_tags[] =3D { 0xa, 0x0, 0xa, 0xb, 0x0 }; + const short ex_sizes[] =3D { 1, 2, 2, 1, 250 }; + short r_sizes[256]; + int r_len =3D 256; + u8 r_tags[256]; + + memset(tags, 0, 128); + tags[0] =3D 0xa0; + tags[1] =3D 0x0a; + tags[2] =3D 0xab; + ea0_tags_to_ranges(tags, r_tags, r_sizes, &r_len); + KUNIT_EXPECT_EQ(test, r_len, 5); + KUNIT_EXPECT_EQ(test, memcmp(r_tags, ex_tags, sizeof(ex_tags)), 0); + KUNIT_EXPECT_EQ(test, memcmp(r_sizes, ex_sizes, sizeof(ex_sizes)), 0); + ea0_ranges_to_tags(r_tags, r_sizes, r_len, dtags); + KUNIT_EXPECT_EQ(test, memcmp(tags, dtags, 128), 0); +} + +/* Test that repeated 0xa0 byte produces 256 ranges of length 1. */ +static void test_tags_to_ranges_repeated(struct kunit *test) +{ + u8 tags[128], dtags[128]; + short r_sizes[256]; + int r_len =3D 256; + u8 r_tags[256]; + + memset(tags, 0xa0, 128); + ea0_tags_to_ranges(tags, r_tags, r_sizes, &r_len); + KUNIT_EXPECT_EQ(test, r_len, 256); + ea0_ranges_to_tags(r_tags, r_sizes, r_len, dtags); + KUNIT_EXPECT_EQ(test, memcmp(tags, dtags, 128), 0); +} + +/* Test that a zero-filled array is compressed into inline storage. */ +static void test_compress_zero(struct kunit *test) +{ + u8 tags[128], dtags[128]; + u64 handle; + + memset(tags, 0, 128); + handle =3D ea0_compress(tags); + KUNIT_EXPECT_EQ(test, handle & BIT_ULL(63), 0); + /* Tags are stored inline. */ + KUNIT_EXPECT_EQ(test, ea0_storage_size(handle), 8); + KUNIT_EXPECT_TRUE(test, ea0_decompress(handle, dtags)); + KUNIT_EXPECT_EQ(test, memcmp(tags, dtags, 128), 0); +} + +/* + * Test that a very small number of tag ranges ends up compressed into 8 b= ytes. + */ +static void test_compress_simple(struct kunit *test) +{ + u8 tags[128], dtags[128]; + u64 handle; + + memset(tags, 0, 128); + tags[0] =3D 0xa0; + tags[1] =3D 0x0a; + tags[2] =3D 0xab; + + handle =3D ea0_compress(tags); + KUNIT_EXPECT_EQ(test, handle & BIT_ULL(63), 0); + /* Tags are stored inline. */ + KUNIT_EXPECT_EQ(test, ea0_storage_size(handle), 8); + KUNIT_EXPECT_TRUE(test, ea0_decompress(handle, dtags)); + KUNIT_EXPECT_EQ(test, memcmp(tags, dtags, 128), 0); +} + +/* + * Generate a buffer that will contain @nranges of tag ranges, test that it + * compresses into @exp_size bytes and decompresses into the original tag + * sequence. + */ +static void compress_range_helper(struct kunit *test, int nranges, int exp= _size) +{ + u8 tags[128], dtags[128]; + u64 handle; + int i; + + memset(tags, 0, 128); + + if (nranges > 1) { + nranges--; + for (i =3D 0; i < nranges / 2; i++) + tags[i] =3D 0xab; + if (nranges % 2) + tags[nranges / 2] =3D 0xa0; + } + + handle =3D ea0_compress(tags); + KUNIT_EXPECT_EQ(test, handle & BIT_ULL(63), 0); + KUNIT_EXPECT_EQ(test, ea0_storage_size(handle), exp_size); + KUNIT_EXPECT_TRUE(test, ea0_decompress(handle, dtags)); + KUNIT_EXPECT_EQ(test, memcmp(tags, dtags, 128), 0); +} + +/* + * Test that every number of tag ranges is correctly compressed and + * decompressed. + */ +static void test_compress_ranges(struct kunit *test) +{ + int i, exp_size; + + for (i =3D 1; i <=3D 256; i++) { + if (i < 7) + exp_size =3D 8; + else if (i < 12) + exp_size =3D 16; + else if (i < 24) + exp_size =3D 32; + else if (i < 47) + exp_size =3D 64; + else + exp_size =3D 128; + compress_range_helper(test, i, exp_size); + } +} + +static struct kunit_case mtecomp_test_cases[] =3D { + KUNIT_CASE(test_tags_to_ranges_zero), + KUNIT_CASE(test_tags_to_ranges_simple), + KUNIT_CASE(test_tags_to_ranges_repeated), + KUNIT_CASE(test_compress_zero), + KUNIT_CASE(test_compress_simple), + KUNIT_CASE(test_compress_ranges), + {} +}; + +static struct kunit_suite mtecomp_test_suite =3D { + .name =3D "mtecomp", + .test_cases =3D mtecomp_test_cases, +}; +kunit_test_suites(&mtecomp_test_suite); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Alexander Potapenko "); --=20 2.41.0.255.g8b1d071c50-goog From nobody Mon Feb 9 05:53:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6129EB64DC for ; Tue, 11 Jul 2023 14:34:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231646AbjGKOeh (ORCPT ); Tue, 11 Jul 2023 10:34:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43552 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233024AbjGKOeV (ORCPT ); Tue, 11 Jul 2023 10:34:21 -0400 Received: from mail-ej1-x649.google.com (mail-ej1-x649.google.com [IPv6:2a00:1450:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9DADB19B5 for ; Tue, 11 Jul 2023 07:34:07 -0700 (PDT) Received: by mail-ej1-x649.google.com with SMTP id a640c23a62f3a-993dc6fbdaaso221924466b.0 for ; Tue, 11 Jul 2023 07:34:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689086045; x=1691678045; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6gUN4hkfqv1qmZBEmgq0ujJ8OMQMxN9ipJDwtCO0z/w=; b=iqdyD5qEEVva79Lf/V+wW3ZA7iXGjXI4gA4uaEkwjNIxeeq6oGFzUDP0IDO0SDu99s jh1edQmWl399fOTG/H19stm5Tebga67h+ewn6mHhgvV1NiGpSv2AH5CvHtT3FIfg47vs QIH8LuQUWEQLcq1D2VHN0BkoOhva8pjWB7vmDLAkBmpZXgO447yTqgpS0WoT3yKw1adU HuhooThEt1CnavhDw+RoxJtdA3LPncTNIsr8F854sXBGxZntSklAlD7O9JBkHpADQLWa vO36N9+amTyNCpYimISFNraYsJvwSeTModLQC7JHClhRIEXNr/c6RfmfewiRxMdByTqg Arhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689086045; x=1691678045; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6gUN4hkfqv1qmZBEmgq0ujJ8OMQMxN9ipJDwtCO0z/w=; b=VjyqRd0t92IJiPbppcDxkKKhwZV0ywbIgNxU4/w93ogSRM8pAXU6W5LD5Cz0awniIN AtjTt6NuCBcbBv1jw5J2MLYhrOTMXO2AjjoroP6hfIiCwJiXRsW0nqQSDA0FjIQ9oPjt xSxuDTLQX4xstlmrZIl/lRoQ5nxW/Zv/OUYfkNgRs6OGXcD7hcX8mRiTHLIyrljxoREy PuhkesfgGEuSsQu5EEvO/FH6W2nn/BnUa7sogKhWSMTG2zU2F5EqstYlkYaLqk2aIq8n nUTZUAxyrNfLZUNBtO873rY5ieReQmvyDo7TBwz6Y5Z+xD8yuzN1yXyEInQr28pN5hrO NF5Q== X-Gm-Message-State: ABy/qLZynsgSWiRjwJL2+qnNCNyU1tLZTR3DzpGozt/rfpcu1bJ0dH5H UOXj3Lh9uFwd31EKQf7lwAtbLhqglUQ= X-Google-Smtp-Source: APBJJlEWueh1U5SOOjLC0yewVqmX5Odwe1/rKpaZj3tXynzC1h8UAEyNblkwVzFFDg+KbXumcqfROgdrXjU= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:564d:3aaa:6b5f:4419]) (user=glider job=sendgmr) by 2002:a17:907:2711:b0:992:4ba4:dc47 with SMTP id w17-20020a170907271100b009924ba4dc47mr51200ejk.4.1689086045783; Tue, 11 Jul 2023 07:34:05 -0700 (PDT) Date: Tue, 11 Jul 2023 16:33:34 +0200 In-Reply-To: <20230711143337.3086664-1-glider@google.com> Mime-Version: 1.0 References: <20230711143337.3086664-1-glider@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230711143337.3086664-8-glider@google.com> Subject: [PATCH 4/5] arm64: mte: add a test for MTE tags compression From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, yury.norov@gmail.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Ensure that tag sequences containing alternating values are compressed to buffers of expected size and correctly decompressed afterwards. Signed-off-by: Alexander Potapenko --- arch/arm64/Kconfig | 10 ++ arch/arm64/mm/Makefile | 1 + arch/arm64/mm/test_mtecomp.c | 175 +++++++++++++++++++++++++++++++++++ 3 files changed, 186 insertions(+) create mode 100644 arch/arm64/mm/test_mtecomp.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b25b584a0a9cb..31fc50208b383 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2075,6 +2075,16 @@ config ARM64_MTE_COMP 128-byte tag buffers corresponding to 4K pages can be compressed using the EA0 algorithm to save heap memory. =20 +config ARM64_MTE_COMP_KUNIT_TEST + tristate "Test tag compression for ARM64 MTE" if !KUNIT_ALL_TESTS + default KUNIT_ALL_TESTS + depends on KUNIT && ARM64_MTE_COMP + help + Test EA0 compression algorithm enabled by CONFIG_ARM64_MTE_COMP. + + Ensure that tag sequences containing alternating values are compressed + to buffers of expected size and correctly decompressed afterwards. + config ARM64_SVE bool "ARM Scalable Vector Extension support" default y diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index 46778f6dd83c2..170dc62b010b9 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -11,6 +11,7 @@ obj-$(CONFIG_TRANS_TABLE) +=3D trans_pgd-asm.o obj-$(CONFIG_DEBUG_VIRTUAL) +=3D physaddr.o obj-$(CONFIG_ARM64_MTE) +=3D mteswap.o obj-$(CONFIG_ARM64_MTE_COMP) +=3D mtecomp.o +obj-$(CONFIG_ARM64_MTE_COMP_KUNIT_TEST) +=3D test_mtecomp.o KASAN_SANITIZE_physaddr.o +=3D n =20 obj-$(CONFIG_KASAN) +=3D kasan_init.o diff --git a/arch/arm64/mm/test_mtecomp.c b/arch/arm64/mm/test_mtecomp.c new file mode 100644 index 0000000000000..67bef6f28dac4 --- /dev/null +++ b/arch/arm64/mm/test_mtecomp.c @@ -0,0 +1,175 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Test cases for EA0, the compression algorithm for MTE tags. + */ + +#include +#include +#include +#include + +/* + * Test that ea0_tags_to_ranges() produces a single range for a zero-fille= d tag + * buffer. + */ +static void test_tags_to_ranges_zero(struct kunit *test) +{ + u8 tags[128], dtags[128]; + short r_sizes[256]; + int r_len =3D 256; + u8 r_tags[256]; + + memset(tags, 0, 128); + ea0_tags_to_ranges(tags, r_tags, r_sizes, &r_len); + KUNIT_EXPECT_EQ(test, r_len, 1); + KUNIT_EXPECT_EQ(test, r_tags[0], 0); + KUNIT_EXPECT_EQ(test, r_sizes[0], 256); + ea0_ranges_to_tags(r_tags, r_sizes, r_len, dtags); + KUNIT_EXPECT_EQ(test, memcmp(tags, dtags, 128), 0); +} + +/* + * Test that a small number of different tags is correctly transformed into + * ranges. + */ +static void test_tags_to_ranges_simple(struct kunit *test) +{ + u8 tags[128], dtags[128]; + const u8 ex_tags[] =3D { 0xa, 0x0, 0xa, 0xb, 0x0 }; + const short ex_sizes[] =3D { 1, 2, 2, 1, 250 }; + short r_sizes[256]; + int r_len =3D 256; + u8 r_tags[256]; + + memset(tags, 0, 128); + tags[0] =3D 0xa0; + tags[1] =3D 0x0a; + tags[2] =3D 0xab; + ea0_tags_to_ranges(tags, r_tags, r_sizes, &r_len); + KUNIT_EXPECT_EQ(test, r_len, 5); + KUNIT_EXPECT_EQ(test, memcmp(r_tags, ex_tags, sizeof(ex_tags)), 0); + KUNIT_EXPECT_EQ(test, memcmp(r_sizes, ex_sizes, sizeof(ex_sizes)), 0); + ea0_ranges_to_tags(r_tags, r_sizes, r_len, dtags); + KUNIT_EXPECT_EQ(test, memcmp(tags, dtags, 128), 0); +} + +/* Test that repeated 0xa0 byte produces 256 ranges of length 1. */ +static void test_tags_to_ranges_repeated(struct kunit *test) +{ + u8 tags[128], dtags[128]; + short r_sizes[256]; + int r_len =3D 256; + u8 r_tags[256]; + + memset(tags, 0xa0, 128); + ea0_tags_to_ranges(tags, r_tags, r_sizes, &r_len); + KUNIT_EXPECT_EQ(test, r_len, 256); + ea0_ranges_to_tags(r_tags, r_sizes, r_len, dtags); + KUNIT_EXPECT_EQ(test, memcmp(tags, dtags, 128), 0); +} + +/* Test that a zero-filled array is compressed into inline storage. */ +static void test_compress_zero(struct kunit *test) +{ + u8 tags[128], dtags[128]; + u64 handle; + + memset(tags, 0, 128); + handle =3D ea0_compress(tags); + KUNIT_EXPECT_EQ(test, handle & BIT_ULL(63), 0); + /* Tags are stored inline. */ + KUNIT_EXPECT_EQ(test, ea0_storage_size(handle), 8); + KUNIT_EXPECT_TRUE(test, ea0_decompress(handle, dtags)); + KUNIT_EXPECT_EQ(test, memcmp(tags, dtags, 128), 0); +} + +/* + * Test that a very small number of tag ranges ends up compressed into 8 b= ytes. + */ +static void test_compress_simple(struct kunit *test) +{ + u8 tags[128], dtags[128]; + u64 handle; + + memset(tags, 0, 128); + tags[0] =3D 0xa0; + tags[1] =3D 0x0a; + tags[2] =3D 0xab; + + handle =3D ea0_compress(tags); + KUNIT_EXPECT_EQ(test, handle & BIT_ULL(63), 0); + /* Tags are stored inline. */ + KUNIT_EXPECT_EQ(test, ea0_storage_size(handle), 8); + KUNIT_EXPECT_TRUE(test, ea0_decompress(handle, dtags)); + KUNIT_EXPECT_EQ(test, memcmp(tags, dtags, 128), 0); +} + +/* + * Generate a buffer that will contain @nranges of tag ranges, test that it + * compresses into @exp_size bytes and decompresses into the original tag + * sequence. + */ +static void compress_range_helper(struct kunit *test, int nranges, int exp= _size) +{ + u8 tags[128], dtags[128]; + u64 handle; + int i; + + memset(tags, 0, 128); + + if (nranges > 1) { + nranges--; + for (i =3D 0; i < nranges / 2; i++) + tags[i] =3D 0xab; + if (nranges % 2) + tags[nranges / 2] =3D 0xa0; + } + + handle =3D ea0_compress(tags); + KUNIT_EXPECT_EQ(test, handle & BIT_ULL(63), 0); + KUNIT_EXPECT_EQ(test, ea0_storage_size(handle), exp_size); + KUNIT_EXPECT_TRUE(test, ea0_decompress(handle, dtags)); + KUNIT_EXPECT_EQ(test, memcmp(tags, dtags, 128), 0); +} + +/* + * Test that every number of tag ranges is correctly compressed and + * decompressed. + */ +static void test_compress_ranges(struct kunit *test) +{ + int i, exp_size; + + for (i =3D 1; i <=3D 256; i++) { + if (i < 7) + exp_size =3D 8; + else if (i < 12) + exp_size =3D 16; + else if (i < 24) + exp_size =3D 32; + else if (i < 47) + exp_size =3D 64; + else + exp_size =3D 128; + compress_range_helper(test, i, exp_size); + } +} + +static struct kunit_case mtecomp_test_cases[] =3D { + KUNIT_CASE(test_tags_to_ranges_zero), + KUNIT_CASE(test_tags_to_ranges_simple), + KUNIT_CASE(test_tags_to_ranges_repeated), + KUNIT_CASE(test_compress_zero), + KUNIT_CASE(test_compress_simple), + KUNIT_CASE(test_compress_ranges), + {} +}; + +static struct kunit_suite mtecomp_test_suite =3D { + .name =3D "mtecomp", + .test_cases =3D mtecomp_test_cases, +}; +kunit_test_suites(&mtecomp_test_suite); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Alexander Potapenko "); --=20 2.41.0.255.g8b1d071c50-goog From nobody Mon Feb 9 05:53:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6770CC0015E for ; Tue, 11 Jul 2023 14:34:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233612AbjGKOeo (ORCPT ); Tue, 11 Jul 2023 10:34:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233626AbjGKOe2 (ORCPT ); Tue, 11 Jul 2023 10:34:28 -0400 Received: from mail-ej1-x649.google.com (mail-ej1-x649.google.com [IPv6:2a00:1450:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12B9F1BCD for ; Tue, 11 Jul 2023 07:34:09 -0700 (PDT) Received: by mail-ej1-x649.google.com with SMTP id a640c23a62f3a-98df34aa83aso799947766b.1 for ; Tue, 11 Jul 2023 07:34:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689086048; x=1691678048; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=j5KQ14CSTc8NpsFet5DkO7+Eg5yvGp+eL0uzYXahxFI=; b=iXh0XHR0rIGogedmzgelpLOrRb0uIzpw8b9s1nFY8Nb6/qdqQKgAhgySorTb4eXpEX trkwn7CjRdl9LZ3qFzgWwhfwnKIr2mEP82qOfkac24bPvfdLP0bgW5XO+eYDfwAWSeG0 G54d2KUpEVu/yW50J26RCv4jyu2/MU28uXVp06zOyK6gMOZJYzyNyhzUbhAyT65fSoog hB9wUuJd5c3gUEzs7IVlS8Z/hPscKxNwwLMh1ilhVTBAq4wLucqv06Uk2GX4YcdLhzRx HHMBxA4OWKMNLVqPEqoqRlWqd4tqwj9mhWiWw3emVbCItRbvukjzLC/Cy6ia+3qUH7K8 +fzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689086048; x=1691678048; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=j5KQ14CSTc8NpsFet5DkO7+Eg5yvGp+eL0uzYXahxFI=; b=E/P52C5rJKu40ZwdODvDkVCJuWB0kQY6+5NytnYvMm6Nf5PeISpyi89cOqigWPk20F 9QLWbhjqBD7gc8OxOvgTQwoUHO5mp3at0PLgatXIJG4w2vZ3KXLhSUuZpZCLpQJOETlv JW/PJ37WSMB8JN5E3qOx5kmrmp3sJJHQQrC72bIMAKS0w51K7S2gLt/a2RTQT5KYQDvk +gJY08vdJB0IVrOqEDHBmEB6q0FJIcQZkli4i+orbVtA/eL2bKzlLpXW1xlQk+4wrUW/ 0OZ/0ououJz0CGocYhJNJS/DmOIVYkluNi+4Wj+VqP3S7dPLlpm7O8LcCztmsBtzQzAe tShQ== X-Gm-Message-State: ABy/qLauxQoTCYmeUMkvdFUxQeLzTYp3ImJE0M2C/7ImGFGG1rX9QICy kXsSsfb/9vXxURXPKp2EHMWNTk69CDo= X-Google-Smtp-Source: APBJJlHozfsOe6ImlJVvRHKJJ0JhFSTsSYLEItORe7QvzMIgfatUXBXWbhDnx0jk/+iST4gofAzN6NPw06k= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:564d:3aaa:6b5f:4419]) (user=glider job=sendgmr) by 2002:a17:907:75f4:b0:991:b9b5:4027 with SMTP id jz20-20020a17090775f400b00991b9b54027mr87513ejc.5.1689086048546; Tue, 11 Jul 2023 07:34:08 -0700 (PDT) Date: Tue, 11 Jul 2023 16:33:35 +0200 In-Reply-To: <20230711143337.3086664-1-glider@google.com> Mime-Version: 1.0 References: <20230711143337.3086664-1-glider@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230711143337.3086664-9-glider@google.com> Subject: [PATCH 4/5] arm64: mte: add compression support to mteswap.c From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, yury.norov@gmail.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Define the internal mteswap.h interface: - _mte_alloc_and_save_tags() - _mte_free_saved_tags() - _mte_restore_tags() , that encapsulates saving tags for a struct page (together with memory allocation), restoring tags, and deleting the storage allocated for them. These functions accept opaque pointers, which may point to 128-byte tag buffers, as well as smaller buffers containing compressed tags, or have compressed tags stored directly in them. The existing code from mteswap.c operating with uncompressed tags is split away into mteswap_nocomp.c, and the newly introduced mteswap_comp.c provides compression with the EA0 algorithm. The latter implementation is picked if CONFIG_ARM64_MTE_COMP=3Dy. Soon after booting Android, tag compression saves ~2.5x memory previously spent by mteswap.c on tag allocations. With the growing uptime, the savings reach 20x and even more. Signed-off-by: Alexander Potapenko --- arch/arm64/mm/Makefile | 5 ++++ arch/arm64/mm/mteswap.c | 20 ++++++-------- arch/arm64/mm/mteswap.h | 12 ++++++++ arch/arm64/mm/mteswap_comp.c | 50 ++++++++++++++++++++++++++++++++++ arch/arm64/mm/mteswap_nocomp.c | 37 +++++++++++++++++++++++++ 5 files changed, 113 insertions(+), 11 deletions(-) create mode 100644 arch/arm64/mm/mteswap.h create mode 100644 arch/arm64/mm/mteswap_comp.c create mode 100644 arch/arm64/mm/mteswap_nocomp.c diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index 170dc62b010b9..46a798e2b67cb 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -11,6 +11,11 @@ obj-$(CONFIG_TRANS_TABLE) +=3D trans_pgd-asm.o obj-$(CONFIG_DEBUG_VIRTUAL) +=3D physaddr.o obj-$(CONFIG_ARM64_MTE) +=3D mteswap.o obj-$(CONFIG_ARM64_MTE_COMP) +=3D mtecomp.o +ifdef CONFIG_ARM64_MTE_COMP +obj-$(CONFIG_ARM64_MTE) +=3D mteswap_comp.o +else +obj-$(CONFIG_ARM64_MTE) +=3D mteswap_nocomp.o +endif obj-$(CONFIG_ARM64_MTE_COMP_KUNIT_TEST) +=3D test_mtecomp.o KASAN_SANITIZE_physaddr.o +=3D n =20 diff --git a/arch/arm64/mm/mteswap.c b/arch/arm64/mm/mteswap.c index cd508ba80ab1b..7e8ee43e0d389 100644 --- a/arch/arm64/mm/mteswap.c +++ b/arch/arm64/mm/mteswap.c @@ -7,6 +7,8 @@ #include #include =20 +#include "mteswap.h" + static DEFINE_XARRAY(mte_pages); =20 void *mte_allocate_tag_storage(void) @@ -27,20 +29,19 @@ int mte_save_tags(struct page *page) if (!page_mte_tagged(page)) return 0; =20 - tag_storage =3D mte_allocate_tag_storage(); + tag_storage =3D _mte_alloc_and_save_tags(page); + pr_err("HERE: %s:%d\n", __FILE__, __LINE__); // TODO if (!tag_storage) return -ENOMEM; =20 - mte_save_page_tags(page_address(page), tag_storage); - /* page_private contains the swap entry.val set in do_swap_page */ ret =3D xa_store(&mte_pages, page_private(page), tag_storage, GFP_KERNEL); if (WARN(xa_is_err(ret), "Failed to store MTE tags")) { - mte_free_tag_storage(tag_storage); + _mte_free_saved_tags(tag_storage); return xa_err(ret); } else if (ret) { /* Entry is being replaced, free the old entry */ - mte_free_tag_storage(ret); + _mte_free_saved_tags(ret); } =20 return 0; @@ -53,10 +54,7 @@ void mte_restore_tags(swp_entry_t entry, struct page *pa= ge) if (!tags) return; =20 - if (try_page_mte_tagging(page)) { - mte_restore_page_tags(page_address(page), tags); - set_page_mte_tagged(page); - } + _mte_restore_tags(tags, page); } =20 void mte_invalidate_tags(int type, pgoff_t offset) @@ -64,7 +62,7 @@ void mte_invalidate_tags(int type, pgoff_t offset) swp_entry_t entry =3D swp_entry(type, offset); void *tags =3D xa_erase(&mte_pages, entry.val); =20 - mte_free_tag_storage(tags); + _mte_free_saved_tags(tags); } =20 void mte_invalidate_tags_area(int type) @@ -78,7 +76,7 @@ void mte_invalidate_tags_area(int type) xa_lock(&mte_pages); xas_for_each(&xa_state, tags, last_entry.val - 1) { __xa_erase(&mte_pages, xa_state.xa_index); - mte_free_tag_storage(tags); + _mte_free_saved_tags(tags); } xa_unlock(&mte_pages); } diff --git a/arch/arm64/mm/mteswap.h b/arch/arm64/mm/mteswap.h new file mode 100644 index 0000000000000..bf25f2b3e75a4 --- /dev/null +++ b/arch/arm64/mm/mteswap.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef ARCH_ARM64_MM_MTESWAP_H_ +#define ARCH_ARM64_MM_MTESWAP_H_ + +#include + +void *_mte_alloc_and_save_tags(struct page *page); +void _mte_free_saved_tags(void *tags); +void _mte_restore_tags(void *tags, struct page *page); + +#endif // ARCH_ARM64_MM_MTESWAP_H_ diff --git a/arch/arm64/mm/mteswap_comp.c b/arch/arm64/mm/mteswap_comp.c new file mode 100644 index 0000000000000..bc2591f8d4f35 --- /dev/null +++ b/arch/arm64/mm/mteswap_comp.c @@ -0,0 +1,50 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* MTE tag storage management with EA0 compression. */ + +#include +#include +#include +#include +#include +#include +#include + +#include "mteswap.h" + +void *_mte_alloc_and_save_tags(struct page *page) +{ + u8 tags[128]; + u64 handle; + + mte_save_page_tags(page_address(page), tags); + handle =3D ea0_compress(tags); + return xa_mk_value(handle); +} + +void _mte_free_saved_tags(void *storage) +{ + unsigned long handle =3D xa_to_value(storage); + int size; + + if (!handle) + return; + size =3D ea0_storage_size(handle); + ea0_release_handle(handle); +} + +void _mte_restore_tags(void *tags, struct page *page) +{ + u64 handle =3D xa_to_value(tags); + u8 tags_decomp[128]; + + if (!handle) + return; + + if (try_page_mte_tagging(page)) { + if (!ea0_decompress(handle, tags_decomp)) + return; + mte_restore_page_tags(page_address(page), tags_decomp); + set_page_mte_tagged(page); + } +} diff --git a/arch/arm64/mm/mteswap_nocomp.c b/arch/arm64/mm/mteswap_nocomp.c new file mode 100644 index 0000000000000..efcdac88b342d --- /dev/null +++ b/arch/arm64/mm/mteswap_nocomp.c @@ -0,0 +1,37 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* MTE tag storage management without compression support. */ + +#include +#include +#include +#include +#include +#include + +#include "mteswap.h" + +void *_mte_alloc_and_save_tags(struct page *page) +{ + void *storage; + + storage =3D mte_allocate_tag_storage(); + if (!storage) + return NULL; + + mte_save_page_tags(page_address(page), storage); + return storage; +} + +void _mte_free_saved_tags(void *storage) +{ + mte_free_tag_storage(storage); +} + +void _mte_restore_tags(void *tags, struct page *page) +{ + if (try_page_mte_tagging(page)) { + mte_restore_page_tags(page_address(page), tags); + set_page_mte_tagged(page); + } +} --=20 2.41.0.255.g8b1d071c50-goog From nobody Mon Feb 9 05:53:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E00BC001DD for ; Tue, 11 Jul 2023 14:34:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233626AbjGKOer (ORCPT ); Tue, 11 Jul 2023 10:34:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43988 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233549AbjGKOee (ORCPT ); Tue, 11 Jul 2023 10:34:34 -0400 Received: from mail-ej1-x64a.google.com (mail-ej1-x64a.google.com [IPv6:2a00:1450:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 020871704 for ; Tue, 11 Jul 2023 07:34:12 -0700 (PDT) Received: by mail-ej1-x64a.google.com with SMTP id a640c23a62f3a-987e47d2e81so385572966b.1 for ; Tue, 11 Jul 2023 07:34:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689086051; x=1691678051; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=j5KQ14CSTc8NpsFet5DkO7+Eg5yvGp+eL0uzYXahxFI=; b=REFxOLn3yTkHXXbW63Pv3TKvhHmLym4lK7n4qAg1HhVqYHBz6Eygyf+ukgS+aoF7Ay FGPGtEGLPHN3UAOhSKs6RZPcuuvEvi3REVbiotM0gtXOI4psD1ewYPMkTsWifRHxEE7c ODSabgC3mrfJSuUFgXHqu28gWbpXZmRd0VSc++qeUL82LRuFMQBDVt39YY+Hl+YTOYS+ 9QImRBBwUfs4WhOyiSWe4jiBSOWjIjNvkytOGISZzdlVGzE2tNXPd6Mo5mGt/BYMG89V Fcpdqdueo74cUDPF7zBL3ajHYrFrGfxDwkUSszPa1TaF6xJtXn/ye+EZCfyEs3ocegWN xPCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689086051; x=1691678051; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=j5KQ14CSTc8NpsFet5DkO7+Eg5yvGp+eL0uzYXahxFI=; b=DGX9adL+i7Id7Ye6aroBcJhJWIuRRvvXZ4iF+AI1XjrLO+q25Mztnwi4l8E2CHVbCv IGlJZDaAgp53sk/Q6YkceQz+/HYIwRfwnfCNe4tp9w76U7/SmS5M2D1ZMKc+OJ7Z3xjx zWVaFwoFlCdFjKNM2t2SAhmAzs2B4QkrEvR2JChElZEtrhbWI3UMziguy7e+DZFfkkm+ PikPGrS9Ff+CX3fevr+b9+RzMWVtHLNHDwckID3waFgkq0nnsrfyR1pumjeR0mrxl0cw rBasAL0FICuVHFxkX6p5NGtafeBCk8fClhGQvxCJSFRC/46ClhBo2JQU/L6jB6uKtAAh KVrQ== X-Gm-Message-State: ABy/qLaMZVfS0hB0NW8TkyTcI64IftDS2J0/zXOr77uVy6Kf1ZQv573E ZOE5El3gpYfl165kfThRkrqJTRwftpA= X-Google-Smtp-Source: APBJJlEi0PgFIXfK3AJUMGBPN3szAfnwqAYopCk5Kg2fx0N0vfhegd6f9ZdvG0eTHAXWV2Nl1A9/xQ+VzWE= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:564d:3aaa:6b5f:4419]) (user=glider job=sendgmr) by 2002:a17:907:2c6a:b0:98e:2178:9b8f with SMTP id ib10-20020a1709072c6a00b0098e21789b8fmr53698ejc.2.1689086051116; Tue, 11 Jul 2023 07:34:11 -0700 (PDT) Date: Tue, 11 Jul 2023 16:33:36 +0200 In-Reply-To: <20230711143337.3086664-1-glider@google.com> Mime-Version: 1.0 References: <20230711143337.3086664-1-glider@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230711143337.3086664-10-glider@google.com> Subject: [PATCH 5/5] arm64: mte: add compression support to mteswap.c From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, yury.norov@gmail.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Define the internal mteswap.h interface: - _mte_alloc_and_save_tags() - _mte_free_saved_tags() - _mte_restore_tags() , that encapsulates saving tags for a struct page (together with memory allocation), restoring tags, and deleting the storage allocated for them. These functions accept opaque pointers, which may point to 128-byte tag buffers, as well as smaller buffers containing compressed tags, or have compressed tags stored directly in them. The existing code from mteswap.c operating with uncompressed tags is split away into mteswap_nocomp.c, and the newly introduced mteswap_comp.c provides compression with the EA0 algorithm. The latter implementation is picked if CONFIG_ARM64_MTE_COMP=3Dy. Soon after booting Android, tag compression saves ~2.5x memory previously spent by mteswap.c on tag allocations. With the growing uptime, the savings reach 20x and even more. Signed-off-by: Alexander Potapenko --- arch/arm64/mm/Makefile | 5 ++++ arch/arm64/mm/mteswap.c | 20 ++++++-------- arch/arm64/mm/mteswap.h | 12 ++++++++ arch/arm64/mm/mteswap_comp.c | 50 ++++++++++++++++++++++++++++++++++ arch/arm64/mm/mteswap_nocomp.c | 37 +++++++++++++++++++++++++ 5 files changed, 113 insertions(+), 11 deletions(-) create mode 100644 arch/arm64/mm/mteswap.h create mode 100644 arch/arm64/mm/mteswap_comp.c create mode 100644 arch/arm64/mm/mteswap_nocomp.c diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index 170dc62b010b9..46a798e2b67cb 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -11,6 +11,11 @@ obj-$(CONFIG_TRANS_TABLE) +=3D trans_pgd-asm.o obj-$(CONFIG_DEBUG_VIRTUAL) +=3D physaddr.o obj-$(CONFIG_ARM64_MTE) +=3D mteswap.o obj-$(CONFIG_ARM64_MTE_COMP) +=3D mtecomp.o +ifdef CONFIG_ARM64_MTE_COMP +obj-$(CONFIG_ARM64_MTE) +=3D mteswap_comp.o +else +obj-$(CONFIG_ARM64_MTE) +=3D mteswap_nocomp.o +endif obj-$(CONFIG_ARM64_MTE_COMP_KUNIT_TEST) +=3D test_mtecomp.o KASAN_SANITIZE_physaddr.o +=3D n =20 diff --git a/arch/arm64/mm/mteswap.c b/arch/arm64/mm/mteswap.c index cd508ba80ab1b..7e8ee43e0d389 100644 --- a/arch/arm64/mm/mteswap.c +++ b/arch/arm64/mm/mteswap.c @@ -7,6 +7,8 @@ #include #include =20 +#include "mteswap.h" + static DEFINE_XARRAY(mte_pages); =20 void *mte_allocate_tag_storage(void) @@ -27,20 +29,19 @@ int mte_save_tags(struct page *page) if (!page_mte_tagged(page)) return 0; =20 - tag_storage =3D mte_allocate_tag_storage(); + tag_storage =3D _mte_alloc_and_save_tags(page); + pr_err("HERE: %s:%d\n", __FILE__, __LINE__); // TODO if (!tag_storage) return -ENOMEM; =20 - mte_save_page_tags(page_address(page), tag_storage); - /* page_private contains the swap entry.val set in do_swap_page */ ret =3D xa_store(&mte_pages, page_private(page), tag_storage, GFP_KERNEL); if (WARN(xa_is_err(ret), "Failed to store MTE tags")) { - mte_free_tag_storage(tag_storage); + _mte_free_saved_tags(tag_storage); return xa_err(ret); } else if (ret) { /* Entry is being replaced, free the old entry */ - mte_free_tag_storage(ret); + _mte_free_saved_tags(ret); } =20 return 0; @@ -53,10 +54,7 @@ void mte_restore_tags(swp_entry_t entry, struct page *pa= ge) if (!tags) return; =20 - if (try_page_mte_tagging(page)) { - mte_restore_page_tags(page_address(page), tags); - set_page_mte_tagged(page); - } + _mte_restore_tags(tags, page); } =20 void mte_invalidate_tags(int type, pgoff_t offset) @@ -64,7 +62,7 @@ void mte_invalidate_tags(int type, pgoff_t offset) swp_entry_t entry =3D swp_entry(type, offset); void *tags =3D xa_erase(&mte_pages, entry.val); =20 - mte_free_tag_storage(tags); + _mte_free_saved_tags(tags); } =20 void mte_invalidate_tags_area(int type) @@ -78,7 +76,7 @@ void mte_invalidate_tags_area(int type) xa_lock(&mte_pages); xas_for_each(&xa_state, tags, last_entry.val - 1) { __xa_erase(&mte_pages, xa_state.xa_index); - mte_free_tag_storage(tags); + _mte_free_saved_tags(tags); } xa_unlock(&mte_pages); } diff --git a/arch/arm64/mm/mteswap.h b/arch/arm64/mm/mteswap.h new file mode 100644 index 0000000000000..bf25f2b3e75a4 --- /dev/null +++ b/arch/arm64/mm/mteswap.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef ARCH_ARM64_MM_MTESWAP_H_ +#define ARCH_ARM64_MM_MTESWAP_H_ + +#include + +void *_mte_alloc_and_save_tags(struct page *page); +void _mte_free_saved_tags(void *tags); +void _mte_restore_tags(void *tags, struct page *page); + +#endif // ARCH_ARM64_MM_MTESWAP_H_ diff --git a/arch/arm64/mm/mteswap_comp.c b/arch/arm64/mm/mteswap_comp.c new file mode 100644 index 0000000000000..bc2591f8d4f35 --- /dev/null +++ b/arch/arm64/mm/mteswap_comp.c @@ -0,0 +1,50 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* MTE tag storage management with EA0 compression. */ + +#include +#include +#include +#include +#include +#include +#include + +#include "mteswap.h" + +void *_mte_alloc_and_save_tags(struct page *page) +{ + u8 tags[128]; + u64 handle; + + mte_save_page_tags(page_address(page), tags); + handle =3D ea0_compress(tags); + return xa_mk_value(handle); +} + +void _mte_free_saved_tags(void *storage) +{ + unsigned long handle =3D xa_to_value(storage); + int size; + + if (!handle) + return; + size =3D ea0_storage_size(handle); + ea0_release_handle(handle); +} + +void _mte_restore_tags(void *tags, struct page *page) +{ + u64 handle =3D xa_to_value(tags); + u8 tags_decomp[128]; + + if (!handle) + return; + + if (try_page_mte_tagging(page)) { + if (!ea0_decompress(handle, tags_decomp)) + return; + mte_restore_page_tags(page_address(page), tags_decomp); + set_page_mte_tagged(page); + } +} diff --git a/arch/arm64/mm/mteswap_nocomp.c b/arch/arm64/mm/mteswap_nocomp.c new file mode 100644 index 0000000000000..efcdac88b342d --- /dev/null +++ b/arch/arm64/mm/mteswap_nocomp.c @@ -0,0 +1,37 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* MTE tag storage management without compression support. */ + +#include +#include +#include +#include +#include +#include + +#include "mteswap.h" + +void *_mte_alloc_and_save_tags(struct page *page) +{ + void *storage; + + storage =3D mte_allocate_tag_storage(); + if (!storage) + return NULL; + + mte_save_page_tags(page_address(page), storage); + return storage; +} + +void _mte_free_saved_tags(void *storage) +{ + mte_free_tag_storage(storage); +} + +void _mte_restore_tags(void *tags, struct page *page) +{ + if (try_page_mte_tagging(page)) { + mte_restore_page_tags(page_address(page), tags); + set_page_mte_tagged(page); + } +} --=20 2.41.0.255.g8b1d071c50-goog From nobody Mon Feb 9 05:53:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6535FEB64DC for ; Tue, 11 Jul 2023 14:34:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230018AbjGKOew (ORCPT ); Tue, 11 Jul 2023 10:34:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43822 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233417AbjGKOej (ORCPT ); Tue, 11 Jul 2023 10:34:39 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 42F1C1A3 for ; Tue, 11 Jul 2023 07:34:15 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-57745160c1dso65056867b3.2 for ; Tue, 11 Jul 2023 07:34:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689086054; x=1691678054; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=x2+pNKeYSTTkMHkj+CaenjYOksDIkk5vhNRhS7msv78=; b=rWmTYzGgcwZ9eiHk++iMkj2Y4zpLmf08k7Fa3FxbN2PPOiL0EOcRfmzpzmhxyA4emz Zg9vbMS+wwftUe4xhBLzikZhynzAc5rkXBa1gKdfPrw20WY6j7lwbreVixWTX2F+Q7GV MyzX+OLukg3yi87YVhK8m131BpFH4KFNk82gxBnH1oLyPQmyOci6MGj/rKMLxnB78zy7 0xjIe4k9etocylJpPsc3HYGBeTSh54bew9GyNu+UTR1qCleLfNcoc8QtaPrvdilWLAur sCWvVN7UZnFy5mwKJ4MrZdhTXXQQiUV2S92piY3NcUDDBVoBgWhQtNG73j7TeETUv+JI pB1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689086054; x=1691678054; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=x2+pNKeYSTTkMHkj+CaenjYOksDIkk5vhNRhS7msv78=; b=eyoElSU0aC8jYBBGjEcR9lSVK9LaL/PmHEjFsUhNnPMLNQAdS1ycmSZqtg4p1oah+x e1tNLqpZfZ9ia4WQnX4iRQ0wdWEHhFg/1XNuNZw/sqgvJdyi5bX7zPgFJiHT/+ebVIH1 +qnha1CmKzFrzb6d3qGwM9sVeh26zhCLAd29hPK64NUdc2XywLS6PKzVI+FXRxam/QzW VI2p3zhq5oQ6eX5dACK+kYODVb9dsCkZDLBIdNaR1wDEJ/AwKaseRBHCbakSN3QwKcRA 11J7mRIHHeiDPJGws+N2c1dicwWi/cbSwVa1ojHiK8mZobwW3pLK/owmPQopiWiNesYY R8sA== X-Gm-Message-State: ABy/qLYGiji2BUdZJUfeh5+wcCy5A5KEyuCzfm68fPDw3fiC3VE5FW3U 6Yya+g/vqnAWVXr2ZLxFH7T9x11rA/k= X-Google-Smtp-Source: APBJJlGMXFL68J/yseQS6WSJsGcs08hmpVoTnEriAaqp762LgxsoCDYkaMziTccnqGd7qTZ2i7dyeqWztoc= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:564d:3aaa:6b5f:4419]) (user=glider job=sendgmr) by 2002:a81:e304:0:b0:573:8316:8d04 with SMTP id q4-20020a81e304000000b0057383168d04mr130837ywl.4.1689086053931; Tue, 11 Jul 2023 07:34:13 -0700 (PDT) Date: Tue, 11 Jul 2023 16:33:37 +0200 In-Reply-To: <20230711143337.3086664-1-glider@google.com> Mime-Version: 1.0 References: <20230711143337.3086664-1-glider@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230711143337.3086664-11-glider@google.com> Subject: [PATCH 5/5] fixup mteswap From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, yury.norov@gmail.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" --- arch/arm64/mm/mteswap.c | 1 - 1 file changed, 1 deletion(-) diff --git a/arch/arm64/mm/mteswap.c b/arch/arm64/mm/mteswap.c index 7e8ee43e0d389..e072c1209fbb6 100644 --- a/arch/arm64/mm/mteswap.c +++ b/arch/arm64/mm/mteswap.c @@ -30,7 +30,6 @@ int mte_save_tags(struct page *page) return 0; =20 tag_storage =3D _mte_alloc_and_save_tags(page); - pr_err("HERE: %s:%d\n", __FILE__, __LINE__); // TODO if (!tag_storage) return -ENOMEM; =20 --=20 2.41.0.255.g8b1d071c50-goog