From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C53F3C433FE for ; Tue, 26 Apr 2022 16:44:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353122AbiDZQrg (ORCPT ); Tue, 26 Apr 2022 12:47:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353096AbiDZQrc (ORCPT ); Tue, 26 Apr 2022 12:47:32 -0400 Received: from mail-ej1-x64a.google.com (mail-ej1-x64a.google.com [IPv6:2a00:1450:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3B0E11815FB for ; Tue, 26 Apr 2022 09:44:24 -0700 (PDT) Received: by mail-ej1-x64a.google.com with SMTP id ga41-20020a1709070c2900b006f3b543907eso1186732ejc.0 for ; Tue, 26 Apr 2022 09:44:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=A3NTwB+RJ9VPcHTGR65MbBZ7w/cFn7M1ATQGzKSwE+o=; b=Ev6F5DoTQsSpCxCbmkM4Tk/VDlOb3s/m5trmUrtt8ohNngUKfb9OF00S9Mc2nr8vO/ IQzu1xDBWcfWryuhjSAG5ZlBXRwwzNyAPtvuMgv9hqu42UyoqcQ5XEuNIQ0sSyZ+pKNG nprOiwF/8Y6uwcYL1a16ovvRkuJ0Its6Mt6zvkHc2in2l3Y1ZetLaSNnUX8xEJI6/5bz L2BalAbeuA6NEVFDlHzBHWrt3POZUDMrT8BCTJ4wmAs/5Yaek0/+fGGtYHWXx1vHXyga CXr7UuHGTuK2cG9qbHwIW0QwOAeaZ8omncUskx4plnax4j1B4kX2WY7MzorNhLUuUpPW zBig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=A3NTwB+RJ9VPcHTGR65MbBZ7w/cFn7M1ATQGzKSwE+o=; b=mkH2LkJYh+rWw2V2md+COuWYXlUw3czxV6tQYZbUVmTDG46AqZ/b+7fux6MybwzNTW iiIEs/n7XbL5MNZRRg84kIuYgLc6NkpQ6SHIuq6+HsPOGWDaRRN+kgr8eSpsAaZXRPXx WrZYyp35UgkRJBTEBtpeY9s/3fhciSk6Np10f/hs2g0f+Srecd4jN2gRlDyZDABp7Z9Q hnxdzBiRvhME9WCC82Z1cqUm4w5n/ttpi7wPnwyuqTDFxgG8lI7grPElfh8Bbajkoe+A PBDZZ3TPo1jDziV9KYVvBLtPR38su4aziLiuW9SHMEEmBU6dYelrU+p1TLm1yd7NgdTu 1YvQ== X-Gm-Message-State: AOAM533nIF5p2o5Ox7ZmujNagmI8AZFijUX7Z1/J6Z0PQXfcjk/FbhFy 0OS8NtJx+Kl2w/nekJHOk09GXPxfOFo= X-Google-Smtp-Source: ABdhPJy2UkBps++xwJwtis+DecVA+28tEyDZziiiovCJIOEA9Q4lRIjJpV+/umWCIrcsFJtphuW7vqpus3Q= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a50:fe1a:0:b0:425:e276:5adf with SMTP id f26-20020a50fe1a000000b00425e2765adfmr13327701edt.284.1650991462573; Tue, 26 Apr 2022 09:44:22 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:30 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-2-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 01/46] x86: add missing include to sparsemem.h From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Dmitry Vyukov sparsemem.h:34:32: error: unknown type name 'phys_addr_t' extern int phys_to_target_node(phys_addr_t start); ^ sparsemem.h:36:39: error: unknown type name 'u64' extern int memory_add_physaddr_to_nid(u64 start); ^ Signed-off-by: Dmitry Vyukov Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/Ifae221ce85d870d8f8d17173bd4= 4d5cf9be2950f --- arch/x86/include/asm/sparsemem.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/x86/include/asm/sparsemem.h b/arch/x86/include/asm/sparse= mem.h index 6a9ccc1b2be5d..64df897c0ee30 100644 --- a/arch/x86/include/asm/sparsemem.h +++ b/arch/x86/include/asm/sparsemem.h @@ -2,6 +2,8 @@ #ifndef _ASM_X86_SPARSEMEM_H #define _ASM_X86_SPARSEMEM_H =20 +#include + #ifdef CONFIG_SPARSEMEM /* * generic non-linear memory support: --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7057C433F5 for ; Tue, 26 Apr 2022 16:45:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353177AbiDZQsU (ORCPT ); Tue, 26 Apr 2022 12:48:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40348 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348540AbiDZQrr (ORCPT ); Tue, 26 Apr 2022 12:47:47 -0400 Received: from mail-lj1-x249.google.com (mail-lj1-x249.google.com [IPv6:2a00:1450:4864:20::249]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 707BA191446 for ; Tue, 26 Apr 2022 09:44:27 -0700 (PDT) Received: by mail-lj1-x249.google.com with SMTP id 20-20020a05651c009400b002462f08f8d2so4825657ljq.2 for ; Tue, 26 Apr 2022 09:44:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=x9iYgpTxv0DnaRejeNz0igsc6c2qOQO28HLcQTucooU=; b=Gl3v87A3gpptfoX1Pv7tGOGPPJqWORLwAIxyRCfUwrY6EM/33Sv4bGLyko673V+5xF W4mSZxl9y8ecIksGz4CC6cMM37/vZPTybFun3kcP8rmLRp4jIpgojrsWED7hIGm2tK41 cIHamgbg09osnortXRg8+8xhIFhauRGd+cc35eZAYByb70I71asbchC3HhT2gEtPLrwF xHbkxkx9ilVJt+VW8y7uX1hEOh+kXUF62M21NT0ICSoUX6MwHxuJtZpqVVShleTmOQJ7 6zRLG6oUiR8di2OatEIxd1W+NN2xViGHvdecFvaeGc48EuCrlgSCU2KO294xezzXHnF7 M+aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=x9iYgpTxv0DnaRejeNz0igsc6c2qOQO28HLcQTucooU=; b=kP4qpYbc7//1MSz/8kPIbMbL58mmfXCgWt5TGfguWc/ye7RT7gZe6yY7Wl0holz587 fBv1W0w7RWPt/zFtrHfqBFN2WlySyr5pop2ZklqGTVY2nAm6oLC17sILYd+1x+2BocGR rCIWvAkKuJVZkzRZPQkJvVZR/oTFbq6a069K1tQ2fnamDj723ypgeqxI6vhTP4BMch6H Ftyl8+gSpriJ68rbUp1uTq31LVo5yOAk4L2/9NU1Eba/4B3FPZ58DCSAwEbMcjz3vzaW /lUkVauCaMEdA2RgKasI0T9yKycUEjSVD1p98yrENCWi0RdujAXjS+qCp72X/M+ilamg BMJg== X-Gm-Message-State: AOAM530Ivcf2TMiUDZyAsUsZQ+I54vOcDTfc9uQoIzPzrX8YkA7dt8xK 9bysr7vBPEgZlv1SCtCD67AxMJ5VZ9Q= X-Google-Smtp-Source: ABdhPJzXv4WU2IWezWx9nTDWae496L7Hc+FMPGMUtJ3yOHwUWfZiIbEVA68sUonu6ecYeBhxtBfXHrcPzyE= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:ac2:424e:0:b0:46b:9249:8ce3 with SMTP id m14-20020ac2424e000000b0046b92498ce3mr17070835lfl.282.1650991465442; Tue, 26 Apr 2022 09:44:25 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:31 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-3-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 02/46] stackdepot: reserve 5 extra bits in depot_stack_handle_t From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Some users (currently only KMSAN) may want to use spare bits in depot_stack_handle_t. Let them do so by adding @extra_bits to __stack_depot_save() to store arbitrary flags, and providing stack_depot_get_extra_bits() to retrieve those flags. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/I0587f6c777667864768daf07821= d594bce6d8ff9 --- include/linux/stackdepot.h | 8 ++++++++ lib/stackdepot.c | 29 ++++++++++++++++++++++++----- 2 files changed, 32 insertions(+), 5 deletions(-) diff --git a/include/linux/stackdepot.h b/include/linux/stackdepot.h index 17f992fe6355b..fd641d266bead 100644 --- a/include/linux/stackdepot.h +++ b/include/linux/stackdepot.h @@ -14,9 +14,15 @@ #include =20 typedef u32 depot_stack_handle_t; +/* + * Number of bits in the handle that stack depot doesn't use. Users may st= ore + * information in them. + */ +#define STACK_DEPOT_EXTRA_BITS 5 =20 depot_stack_handle_t __stack_depot_save(unsigned long *entries, unsigned int nr_entries, + unsigned int extra_bits, gfp_t gfp_flags, bool can_alloc); =20 /* @@ -41,6 +47,8 @@ depot_stack_handle_t stack_depot_save(unsigned long *entr= ies, unsigned int stack_depot_fetch(depot_stack_handle_t handle, unsigned long **entries); =20 +unsigned int stack_depot_get_extra_bits(depot_stack_handle_t handle); + int stack_depot_snprint(depot_stack_handle_t handle, char *buf, size_t siz= e, int spaces); =20 diff --git a/lib/stackdepot.c b/lib/stackdepot.c index bf5ba9af05009..6dc11a3b7b88e 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -42,7 +42,8 @@ #define STACK_ALLOC_OFFSET_BITS (STACK_ALLOC_ORDER + PAGE_SHIFT - \ STACK_ALLOC_ALIGN) #define STACK_ALLOC_INDEX_BITS (DEPOT_STACK_BITS - \ - STACK_ALLOC_NULL_PROTECTION_BITS - STACK_ALLOC_OFFSET_BITS) + STACK_ALLOC_NULL_PROTECTION_BITS - \ + STACK_ALLOC_OFFSET_BITS - STACK_DEPOT_EXTRA_BITS) #define STACK_ALLOC_SLABS_CAP 8192 #define STACK_ALLOC_MAX_SLABS \ (((1LL << (STACK_ALLOC_INDEX_BITS)) < STACK_ALLOC_SLABS_CAP) ? \ @@ -55,6 +56,7 @@ union handle_parts { u32 slabindex : STACK_ALLOC_INDEX_BITS; u32 offset : STACK_ALLOC_OFFSET_BITS; u32 valid : STACK_ALLOC_NULL_PROTECTION_BITS; + u32 extra : STACK_DEPOT_EXTRA_BITS; }; }; =20 @@ -73,6 +75,14 @@ static int next_slab_inited; static size_t depot_offset; static DEFINE_RAW_SPINLOCK(depot_lock); =20 +unsigned int stack_depot_get_extra_bits(depot_stack_handle_t handle) +{ + union handle_parts parts =3D { .handle =3D handle }; + + return parts.extra; +} +EXPORT_SYMBOL(stack_depot_get_extra_bits); + static bool init_stack_slab(void **prealloc) { if (!*prealloc) @@ -136,6 +146,7 @@ depot_alloc_stack(unsigned long *entries, int size, u32= hash, void **prealloc) stack->handle.slabindex =3D depot_index; stack->handle.offset =3D depot_offset >> STACK_ALLOC_ALIGN; stack->handle.valid =3D 1; + stack->handle.extra =3D 0; memcpy(stack->entries, entries, flex_array_size(stack, entries, size)); depot_offset +=3D required_size; =20 @@ -320,6 +331,7 @@ EXPORT_SYMBOL_GPL(stack_depot_fetch); * * @entries: Pointer to storage array * @nr_entries: Size of the storage array + * @extra_bits: Flags to store in unused bits of depot_stack_handle_t * @alloc_flags: Allocation gfp flags * @can_alloc: Allocate stack slabs (increased chance of failure if false) * @@ -331,6 +343,10 @@ EXPORT_SYMBOL_GPL(stack_depot_fetch); * If the stack trace in @entries is from an interrupt, only the portion u= p to * interrupt entry is saved. * + * Additional opaque flags can be passed in @extra_bits, stored in the unu= sed + * bits of the stack handle, and retrieved using stack_depot_get_extra_bit= s() + * without calling stack_depot_fetch(). + * * Context: Any context, but setting @can_alloc to %false is required if * alloc_pages() cannot be used from the current context. Current= ly * this is the case from contexts where neither %GFP_ATOMIC nor @@ -340,10 +356,11 @@ EXPORT_SYMBOL_GPL(stack_depot_fetch); */ depot_stack_handle_t __stack_depot_save(unsigned long *entries, unsigned int nr_entries, + unsigned int extra_bits, gfp_t alloc_flags, bool can_alloc) { struct stack_record *found =3D NULL, **bucket; - depot_stack_handle_t retval =3D 0; + union handle_parts retval =3D { .handle =3D 0 }; struct page *page =3D NULL; void *prealloc =3D NULL; unsigned long flags; @@ -427,9 +444,11 @@ depot_stack_handle_t __stack_depot_save(unsigned long = *entries, free_pages((unsigned long)prealloc, STACK_ALLOC_ORDER); } if (found) - retval =3D found->handle.handle; + retval.handle =3D found->handle.handle; fast_exit: - return retval; + retval.extra =3D extra_bits; + + return retval.handle; } EXPORT_SYMBOL_GPL(__stack_depot_save); =20 @@ -449,6 +468,6 @@ depot_stack_handle_t stack_depot_save(unsigned long *en= tries, unsigned int nr_entries, gfp_t alloc_flags) { - return __stack_depot_save(entries, nr_entries, alloc_flags, true); + return __stack_depot_save(entries, nr_entries, 0, alloc_flags, true); } EXPORT_SYMBOL_GPL(stack_depot_save); --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A3ADC4332F for ; Tue, 26 Apr 2022 16:44:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353125AbiDZQrv (ORCPT ); Tue, 26 Apr 2022 12:47:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40382 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353127AbiDZQrs (ORCPT ); Tue, 26 Apr 2022 12:47:48 -0400 Received: from mail-ej1-x64a.google.com (mail-ej1-x64a.google.com [IPv6:2a00:1450:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E27BC19146C for ; Tue, 26 Apr 2022 09:44:29 -0700 (PDT) Received: by mail-ej1-x64a.google.com with SMTP id 13-20020a170906328d00b006982d0888a4so9271381ejw.9 for ; Tue, 26 Apr 2022 09:44:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=nOhCPOtwmNXODLijzTSO8M7zxl8h4T8Tb26PN26Qq0M=; b=LZKEFbB8TPQO7kgBdpU2j4eYblKuzaR2Gi1xLMal4r01UVgMm32iuP23KamR8oVwXU A0Q2g9OquWtPZUG20Kv+3FzPplKg/UvdC5tL/zaE1RpHoBAA4uWQSRvxSviAcVNValYR atgp44CzDP4JuJCWMVAOztuYyp83xVUCTOdMgjPZtZT/I1G2Uvo4gRCy0vOJEAK515qh z6CoYJ90tMK9fDUC0vXsGNpavg5yM2TgwZG7Lr5lZNNSi4LNKj1E9yf0Md2SkqX4WdS2 da8nli8jJ9UvooA1jnZ8RZWFp4syKCdzroiMyC5ygmwL4OY2QmTJ4Viu/eSQqGFOkxbA m6yg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=nOhCPOtwmNXODLijzTSO8M7zxl8h4T8Tb26PN26Qq0M=; b=Juc/qqSWjmDspgN/0xyq8NtbE53IipLg1cmvB1b/Fg0GDJj85EnjMWYlYH6I8CwMWn oWRuL2HECakou7s0KkeOUwpXDhEeu3Ab6YwcKpcRpF9qkQr0cLPyv0qxZyjknYioPKlH Zx50Cgt3iIqKymIdZnwfFZBVCBMcVa6HgCFXkw6v3yx/g/u/h/s4szdtauIjFVq2Ih0z jA8+fHVo4JFRxi8yhZp0wVX5nqs3e0fM3nGLJtkiy5/zqbSfLHlNtZxKuehWFjb2AToV VB1ztZDpTU90bXkPQl0PAHB6VVdMBmxPORgYeU4FruXdJTWxL449vPC78mxWkqaI2K3k VCkg== X-Gm-Message-State: AOAM533v6eQks/lv7Pkrt5uYqyvv/VtdE3ILcjckgPjPGiDbOLiAZRgb QtqBK7+12bE2HIF5Duqx9UCEEZR6j/0= X-Google-Smtp-Source: ABdhPJwSu8anY6YzOERS1Ux22TtxUqNV58xGV4qRNVaMI4J71hTHmmbMpjG+iLMULK2X+phM8pbqjHM5UG4= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a17:907:7815:b0:6ce:5242:1280 with SMTP id la21-20020a170907781500b006ce52421280mr22103960ejc.217.1650991468011; Tue, 26 Apr 2022 09:44:28 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:32 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-4-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 03/46] kasan: common: adapt to the new prototype of __stack_depot_save() From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Pass extra_bits=3D0, as KASAN does not intend to store additional information in the stack handle. No functional change. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/I932d8f4f11a41b7483e0d570787= 44cc94697607a --- mm/kasan/common.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index d9079ec11f313..5d244746ac4fe 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -36,7 +36,7 @@ depot_stack_handle_t kasan_save_stack(gfp_t flags, bool c= an_alloc) unsigned int nr_entries; =20 nr_entries =3D stack_trace_save(entries, ARRAY_SIZE(entries), 0); - return __stack_depot_save(entries, nr_entries, flags, can_alloc); + return __stack_depot_save(entries, nr_entries, 0, flags, can_alloc); } =20 void kasan_set_track(struct kasan_track *track, gfp_t flags) --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 025FDC433FE for ; Tue, 26 Apr 2022 16:45:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232215AbiDZQsQ (ORCPT ); Tue, 26 Apr 2022 12:48:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40384 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353120AbiDZQrs (ORCPT ); Tue, 26 Apr 2022 12:47:48 -0400 Received: from mail-ej1-x64a.google.com (mail-ej1-x64a.google.com [IPv6:2a00:1450:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5481C191C67 for ; Tue, 26 Apr 2022 09:44:32 -0700 (PDT) Received: by mail-ej1-x64a.google.com with SMTP id go12-20020a1709070d8c00b006f009400732so9165153ejc.1 for ; Tue, 26 Apr 2022 09:44:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=jT7N8sHbqw5Bt63XokggrytDJOztxBUcMGiWNZUZnNs=; b=socyW/SWH6WVUElWDQ9so0Jec6CqBHZso2QGVjaGlPPUu+vTsdFC6iZP42U1gWVUW8 lw5Omaj6xQ6NIAx4hEPTVPQZVPhUnOOdq2KiTARpdZyQ3LitifZ9pKoTzPHYkhsda4Rc I3/96cm8sYfgVBN9+zu56yewNbGMMZugG3lvIt1FRn+zBDIMu2owM5cLnh00hlx9RTgJ 2Z6YBAV3FKrKFXGf4sOcstHFSQon+8TmbuzAsNLmJ8iTodBN45fyl/KZmz89vwrBf1cr dJFUsGH8Xd4MrMUpg558yBB4sq9J25PoMnfrF15rZOAHy5XHDqKNCFXnOKuvMldOAx0v Xc1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=jT7N8sHbqw5Bt63XokggrytDJOztxBUcMGiWNZUZnNs=; b=sQvd/EVuiBtniFrrGoAQ7Pn0sY/JVGPUvkj2tggVDAQ56IiPHCFQMyTLY32K3HkNzY qu+/QcsZxwXqhGkS5/kL+AG+dckk6GsBJaKXTQDrBRCkSXiAzrObAMWZ2QEsas3OCgKq SADYEC2JmnYNYh1XKGNMfTQ+ejESHJdLzlAP6cXaC69UzUbT4MffImyrQJxI8ST7kQhS ruqOMFM9nUCUx/H5k0pULDP7FAWOxrdfJoHtHWqJqrlsw7T/cTjWKeiCX8hAu+xbIZOQ 5qzy03y7CK3itjAM3qvpaoV0/7l3Mc/ffXbr9tuiTCFM2e8cvIgJCpHGHLwPS9Wbf4oL jy/g== X-Gm-Message-State: AOAM530f8VLwlEDef1RYE5+SMfDxKEDJI6PKeTPp1k8DZxnlhBzCY5qf VM5vmaVs/Ks3oOCwPxYWbj4WcGJdolY= X-Google-Smtp-Source: ABdhPJzVoY4VqalpQr3d8BPXtD+NaAh71WbLc1nD76HvvUm8mcb7U8Xkr1TFGyyyj1Et3ec1q26DZC4DL3U= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:3593:b0:425:dfd4:2947 with SMTP id y19-20020a056402359300b00425dfd42947mr14309082edc.137.1650991470646; Tue, 26 Apr 2022 09:44:30 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:33 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-5-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 04/46] instrumented.h: allow instrumenting both sides of copy_from_user() From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce instrument_copy_from_user_before() and instrument_copy_from_user_after() hooks to be invoked before and after the call to copy_from_user(). KASAN and KCSAN will be only using instrument_copy_from_user_before(), but for KMSAN we'll need to insert code after copy_from_user(). Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/I855034578f0b0f126734cbd734f= b4ae1d3a6af99 --- include/linux/instrumented.h | 21 +++++++++++++++++++-- include/linux/uaccess.h | 19 ++++++++++++++----- lib/iov_iter.c | 9 ++++++--- lib/usercopy.c | 3 ++- 4 files changed, 41 insertions(+), 11 deletions(-) diff --git a/include/linux/instrumented.h b/include/linux/instrumented.h index 42faebbaa202a..ee8f7d17d34f5 100644 --- a/include/linux/instrumented.h +++ b/include/linux/instrumented.h @@ -120,7 +120,7 @@ instrument_copy_to_user(void __user *to, const void *fr= om, unsigned long n) } =20 /** - * instrument_copy_from_user - instrument writes of copy_from_user + * instrument_copy_from_user_before - add instrumentation before copy_from= _user * * Instrument writes to kernel memory, that are due to copy_from_user (and * variants). The instrumentation should be inserted before the accesses. @@ -130,10 +130,27 @@ instrument_copy_to_user(void __user *to, const void *= from, unsigned long n) * @n number of bytes to copy */ static __always_inline void -instrument_copy_from_user(const void *to, const void __user *from, unsigne= d long n) +instrument_copy_from_user_before(const void *to, const void __user *from, = unsigned long n) { kasan_check_write(to, n); kcsan_check_write(to, n); } =20 +/** + * instrument_copy_from_user_after - add instrumentation after copy_from_u= ser + * + * Instrument writes to kernel memory, that are due to copy_from_user (and + * variants). The instrumentation should be inserted after the accesses. + * + * @to destination address + * @from source address + * @n number of bytes to copy + * @left number of bytes not copied (as returned by copy_from_user) + */ +static __always_inline void +instrument_copy_from_user_after(const void *to, const void __user *from, + unsigned long n, unsigned long left) +{ +} + #endif /* _LINUX_INSTRUMENTED_H */ diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h index 546179418ffa2..079bdea3b9dcd 100644 --- a/include/linux/uaccess.h +++ b/include/linux/uaccess.h @@ -58,20 +58,28 @@ static __always_inline __must_check unsigned long __copy_from_user_inatomic(void *to, const void __user *from, unsigned long= n) { - instrument_copy_from_user(to, from, n); + unsigned long res; + + instrument_copy_from_user_before(to, from, n); check_object_size(to, n, false); - return raw_copy_from_user(to, from, n); + res =3D raw_copy_from_user(to, from, n); + instrument_copy_from_user_after(to, from, n, res); + return res; } =20 static __always_inline __must_check unsigned long __copy_from_user(void *to, const void __user *from, unsigned long n) { + unsigned long res; + might_fault(); + instrument_copy_from_user_before(to, from, n); if (should_fail_usercopy()) return n; - instrument_copy_from_user(to, from, n); check_object_size(to, n, false); - return raw_copy_from_user(to, from, n); + res =3D raw_copy_from_user(to, from, n); + instrument_copy_from_user_after(to, from, n, res); + return res; } =20 /** @@ -115,8 +123,9 @@ _copy_from_user(void *to, const void __user *from, unsi= gned long n) unsigned long res =3D n; might_fault(); if (!should_fail_usercopy() && likely(access_ok(from, n))) { - instrument_copy_from_user(to, from, n); + instrument_copy_from_user_before(to, from, n); res =3D raw_copy_from_user(to, from, n); + instrument_copy_from_user_after(to, from, n, res); } if (unlikely(res)) memset(to + (n - res), 0, res); diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 6dd5330f7a995..fb19401c29c4f 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -159,13 +159,16 @@ static int copyout(void __user *to, const void *from,= size_t n) =20 static int copyin(void *to, const void __user *from, size_t n) { + size_t res =3D n; + if (should_fail_usercopy()) return n; if (access_ok(from, n)) { - instrument_copy_from_user(to, from, n); - n =3D raw_copy_from_user(to, from, n); + instrument_copy_from_user_before(to, from, n); + res =3D raw_copy_from_user(to, from, n); + instrument_copy_from_user_after(to, from, n, res); } - return n; + return res; } =20 static size_t copy_page_to_iter_iovec(struct page *page, size_t offset, si= ze_t bytes, diff --git a/lib/usercopy.c b/lib/usercopy.c index 7413dd300516e..1505a52f23a01 100644 --- a/lib/usercopy.c +++ b/lib/usercopy.c @@ -12,8 +12,9 @@ unsigned long _copy_from_user(void *to, const void __user= *from, unsigned long n unsigned long res =3D n; might_fault(); if (!should_fail_usercopy() && likely(access_ok(from, n))) { - instrument_copy_from_user(to, from, n); + instrument_copy_from_user_before(to, from, n); res =3D raw_copy_from_user(to, from, n); + instrument_copy_from_user_after(to, from, n, res); } if (unlikely(res)) memset(to + (n - res), 0, res); --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CDE8C433FE for ; Tue, 26 Apr 2022 16:44:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353116AbiDZQr6 (ORCPT ); Tue, 26 Apr 2022 12:47:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40396 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353139AbiDZQrs (ORCPT ); Tue, 26 Apr 2022 12:47:48 -0400 Received: from mail-ed1-x54a.google.com (mail-ed1-x54a.google.com [IPv6:2a00:1450:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B30BE194B0E for ; Tue, 26 Apr 2022 09:44:34 -0700 (PDT) Received: by mail-ed1-x54a.google.com with SMTP id eg38-20020a05640228a600b00425d61d0302so4457200edb.17 for ; Tue, 26 Apr 2022 09:44:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=n1CPd29HHKe3iLELt5x5zSxX1l0GlMUVKgGqbXsXPsc=; b=Ul2x8GzRzl3HHeVNodduisFS+GirD7ttp4YhxIfHtZ/wAm96wAOKW0A3UvjvpnRyGE dGIlLM7rLl8SOQncEfIhzWSYbpzWK4FIquce/uniGQa5s5Khmoip2xUEMKpML79SS2Nr fjFDfjY+DOzRzBepJfAOlaHW98Dbj79IrR8+XSS3SP0heVjAw3rhmHavoXW9MHBQKoZG wag5mhU0cWEiD2fbJEejotrtfue9XD44tRsrIfEYSpEORbTnfzF4DdsV8zcndXNrqw8l /BmYauMIADOumK6gR7O9BDkDjdwUZNDb4ZrqS03yp+ROELB3q+NAfod4pzAvjwcIAOW/ JTFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=n1CPd29HHKe3iLELt5x5zSxX1l0GlMUVKgGqbXsXPsc=; b=eiqS5UwvYza7zPJ2ZwGJxiTu81Pkib8v0a9KB9fAeeoCXMWhHeSt/TNkTlx961KG+2 EINNGEuCmbEUxM5Zu4bkmHHKUcHVJAPb45bX71rBroMUjEjqEFAiI4gwfSt6fYa4X6E0 Wcl+9I+yEVo1Yfucyatui5znjvwuoyj8f9fl4O2GrvaEpPQ7tN+BRGSdxIhA7ZC3lbGs xX6hRn17tGKWlJCDpEteWWoUMLHxscbejlxrXH1GasQb5lq1Da0oc6IRBQBadGE/c+dR KLz41AfKuaOPVIYzbRj6pCv+DmN5bs5PgUoJp8uByuCW5hpI9YYeiuNbBSC+9i+CfQuV nw9w== X-Gm-Message-State: AOAM531RQ+keJ0YSak5u1TLwfzpK3oJHrewLmxvSDEipvIHfNtE2mpZx pOUYd2fE3BLp5FtST9miu3iJVPk2UXE= X-Google-Smtp-Source: ABdhPJyMKZH4q2GhaVIalO1baxoI+dFp4AmHdfwHW+zpAMwIACuxCiWEQZTqeJmEu9W2AFPYPsdcZ1aQv28= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:1385:b0:413:2bc6:4400 with SMTP id b5-20020a056402138500b004132bc64400mr25986634edv.94.1650991473255; Tue, 26 Apr 2022 09:44:33 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:34 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-6-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 05/46] x86: asm: instrument usercopy in get_user() and __put_user_size() From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use hooks from instrumented.h to notify bug detection tools about usercopy events in get_user() and put_user_size(). It's still unclear how to instrument put_user(), which assumes that instrumentation code doesn't clobber RAX. Signed-off-by: Alexander Potapenko Reported-by: kernel test robot --- Link: https://linux-review.googlesource.com/id/Ia9f12bfe5832623250e20f1859f= df5cc485a2fce --- arch/x86/include/asm/uaccess.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index f78e2b3501a19..0373d52a0543e 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -5,6 +5,7 @@ * User space memory access functions */ #include +#include #include #include #include @@ -99,11 +100,13 @@ extern int __get_user_bad(void); int __ret_gu; \ register __inttype(*(ptr)) __val_gu asm("%"_ASM_DX); \ __chk_user_ptr(ptr); \ + instrument_copy_from_user_before((void *)&(x), ptr, sizeof(*(ptr))); \ asm volatile("call __" #fn "_%P4" \ : "=3Da" (__ret_gu), "=3Dr" (__val_gu), \ ASM_CALL_CONSTRAINT \ : "0" (ptr), "i" (sizeof(*(ptr)))); \ (x) =3D (__force __typeof__(*(ptr))) __val_gu; \ + instrument_copy_from_user_after((void *)&(x), ptr, sizeof(*(ptr)), 0); \ __builtin_expect(__ret_gu, 0); \ }) =20 @@ -248,7 +251,9 @@ extern void __put_user_nocheck_8(void); =20 #define __put_user_size(x, ptr, size, label) \ do { \ + __typeof__(*(ptr)) __pus_val =3D x; \ __chk_user_ptr(ptr); \ + instrument_copy_to_user(ptr, &(__pus_val), size); \ switch (size) { \ case 1: \ __put_user_goto(x, ptr, "b", "iq", label); \ @@ -286,6 +291,7 @@ do { \ #define __get_user_size(x, ptr, size, label) \ do { \ __chk_user_ptr(ptr); \ + instrument_copy_from_user_before((void *)&(x), ptr, size); \ switch (size) { \ case 1: { \ unsigned char x_u8__; \ @@ -305,6 +311,7 @@ do { \ default: \ (x) =3D __get_user_bad(); \ } \ + instrument_copy_from_user_after((void *)&(x), ptr, size, 0); \ } while (0) =20 #define __get_user_asm(x, addr, itype, ltype, label) \ --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CEADC433FE for ; Tue, 26 Apr 2022 16:44:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353150AbiDZQsD (ORCPT ); Tue, 26 Apr 2022 12:48:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353147AbiDZQrt (ORCPT ); Tue, 26 Apr 2022 12:47:49 -0400 Received: from mail-ed1-x54a.google.com (mail-ed1-x54a.google.com [IPv6:2a00:1450:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3B718191C60 for ; Tue, 26 Apr 2022 09:44:37 -0700 (PDT) Received: by mail-ed1-x54a.google.com with SMTP id w8-20020a50d788000000b00418e6810364so10517303edi.13 for ; Tue, 26 Apr 2022 09:44:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=cG4bw4Dybp9do4PvtZBZ2e2ITt4Wsjs9LSUgy4lmV+4=; b=RcuVhtgi/Tw1bMDtabWe38GwlUcwg4SUjFJCfkW2os/ElbHByzMYCT2RYzqn62C0bn cbc56TqAri4pL54mxEkGuVkiAaJJ9tXSwdeAGi+nY3slj4lCV4ZtmvWqeaNCS3QKANio WW7rLfKq9u9Kd3RXcYrkCGHS0aT05KrVMUMkow7J5zXgyXnRiQ3E/SReE2LZFqObeM2p smA+CIkkQF6Odog2q36ChSrXewTMVXM+93iY1Gau5URCGueQT5zBnFATmZOSwzbaRVC1 zWWjq4eJBwkMFtGAv2hVypAFZhEDp0HEN6QVTrNW6nJcUAwldpsVdCTpJbkD3ZSC1KHK QTEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=cG4bw4Dybp9do4PvtZBZ2e2ITt4Wsjs9LSUgy4lmV+4=; b=dygRiz9TQFJZMlHeuDyb59bwrVUME6alQw/eQkbf7IzVFN+/88Zo+HIZZSWMbI9pbn lOboZ2r1VExo+wjV3Gp4hcUZLULWnjkyn8NlAP9AtrFcUhL2AeGbGsfJT+7OJVFoY1Mj KfFB5184eaKyJtcLanOXnHReSrZjpK0OEhC+dkvJKSQzvD8a/9DzorPnegWwJxqBLcLe yhR4uHckzihwyvemYTmyN6RgGHZa0rUXPOazP7uYPfxquPiWILtOvpp6Hxz8VXDzom3v VZzazdO4E4ag6xNWGTNKr7INQg62pOwWwMEu6EB3zftKwj+w2Z7ron8BUrhiFfaW/1wN +GSw== X-Gm-Message-State: AOAM531boYWRaasfVelv/EcXQa7Lo9aQOkE+FFoSzSBVJOSEL9QDM8PZ XXrFKxssZBsv4FMwIapZPx9EeomxZbU= X-Google-Smtp-Source: ABdhPJwJq5G2rpZj9OKDdv47ee+grpfL1muZK+Akh8tj6QCkzJXGooM+Bx/fhzrHXRviMJo4c+0ZnBDuFgY= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:2809:b0:423:e123:5e40 with SMTP id h9-20020a056402280900b00423e1235e40mr25792114ede.84.1650991475654; Tue, 26 Apr 2022 09:44:35 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:35 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-7-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 06/46] asm-generic: instrument usercopy in cacheflush.h From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Notify memory tools about usercopy events in copy_to_user_page() and copy_from_user_page(). Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/Ic1ee8da1886325f46ad67f52176= f48c2c836c48f --- include/asm-generic/cacheflush.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/include/asm-generic/cacheflush.h b/include/asm-generic/cachefl= ush.h index 4f07afacbc239..0f63eb325025f 100644 --- a/include/asm-generic/cacheflush.h +++ b/include/asm-generic/cacheflush.h @@ -2,6 +2,8 @@ #ifndef _ASM_GENERIC_CACHEFLUSH_H #define _ASM_GENERIC_CACHEFLUSH_H =20 +#include + struct mm_struct; struct vm_area_struct; struct page; @@ -105,6 +107,7 @@ static inline void flush_cache_vunmap(unsigned long sta= rt, unsigned long end) #ifndef copy_to_user_page #define copy_to_user_page(vma, page, vaddr, dst, src, len) \ do { \ + instrument_copy_to_user(dst, src, len); \ memcpy(dst, src, len); \ flush_icache_user_page(vma, page, vaddr, len); \ } while (0) @@ -112,7 +115,11 @@ static inline void flush_cache_vunmap(unsigned long st= art, unsigned long end) =20 #ifndef copy_from_user_page #define copy_from_user_page(vma, page, vaddr, dst, src, len) \ - memcpy(dst, src, len) + do { \ + instrument_copy_from_user_before(dst, src, len); \ + memcpy(dst, src, len); \ + instrument_copy_from_user_after(dst, src, len, 0); \ + } while (0) #endif =20 #endif /* _ASM_GENERIC_CACHEFLUSH_H */ --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91834C433FE for ; Tue, 26 Apr 2022 16:45:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353155AbiDZQsG (ORCPT ); Tue, 26 Apr 2022 12:48:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40532 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353157AbiDZQrt (ORCPT ); Tue, 26 Apr 2022 12:47:49 -0400 Received: from mail-ed1-x549.google.com (mail-ed1-x549.google.com [IPv6:2a00:1450:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9D7E1971D8 for ; Tue, 26 Apr 2022 09:44:39 -0700 (PDT) Received: by mail-ed1-x549.google.com with SMTP id ch28-20020a0564021bdc00b00425cb227ab4so5255006edb.4 for ; Tue, 26 Apr 2022 09:44:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=p8FEo7s+Y/rcqoYNHQMIRhiXb9XntJYZxdO4j/h094A=; b=FN0NXTYm/4DM3BzRomEJlLBOdDZK2QtkvjCrwnxNu7ZCzEOc/oVhFdK4WXlDEVdirQ YuVhb8aZBGQP/vHgAYsLyv76/O97zW2Bf51WKCp789tm4Qjtf9zFBL3j57QvVDLDhnFw JWzYQwqOtyltmmcl+Arw6lRAwK/B1u8ETn2mV8K4IqKpBj4vuVhQCOItYU3wYbkHWE2J Z7jF6oHs4JLz+HXwY+ciXQdsTbDClhb4iTf+PvyW2JhTxjOung+RoUV/ib2RrncC3xwe fZCF7gC3F71l0WI97t3oSVGvki3Zm/4NN0wR3vTDsvenqe+v77M7MJ77kmQUAWJZ0U7L iRSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=p8FEo7s+Y/rcqoYNHQMIRhiXb9XntJYZxdO4j/h094A=; b=LSfyIY2i2aAPEJ0wZZapbDVOnqVtzyMBrN0aLbeHU2lg5N91q8lTUuP/MLwiVF2BSK Y2PjN+2GfJ8/ACwb05KbyZbBfr39JbhPFEbnsBEfnsYwLNpovT3J0Hn0afM1+zSpD6QP ck3/1RRWSvtkCmJ1D+pqKa9kcYDQeswnCRolV9LqTDadxLL5jZ/r5ob4cB7JRFtIROOG lEMT9S8WwU++CM/kER/QQAsGzNHVk0+VfM2vrXkpeDN0lq+MEKgb6QoK6PlqY2Wf03Bv vDZ9uQpFS3D4v2CPsDBYwGWMVHICuYMb+8af8EOncKejPY/Mbb2lfKdzTZl5aA0wlaVN SUHw== X-Gm-Message-State: AOAM532ZSXtGfy2HdUSy31+s24Xk8JwcSNVMqHQbnzEkOXmxj5z5D6Eq a/jgyjdkU/Gb89DmDneVWf9LzCjJaCI= X-Google-Smtp-Source: ABdhPJyCCyIfUafFOfp5MPZnTGpX5q6s2bnIOjoI2eskQhmcqE76RefRNF30puIR6F21nZAoW9kp50iu4zg= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:3553:b0:424:20d3:ae78 with SMTP id f19-20020a056402355300b0042420d3ae78mr25923509edd.290.1650991478143; Tue, 26 Apr 2022 09:44:38 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:36 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-8-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 07/46] kmsan: add ReST documentation From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add Documentation/dev-tools/kmsan.rst and reference it in the dev-tools index. Signed-off-by: Alexander Potapenko --- v2: -- added a note that KMSAN is not intended for production use Link: https://linux-review.googlesource.com/id/I751586f79418b95550a83c6035c= 650b5b01567cc --- Documentation/dev-tools/index.rst | 1 + Documentation/dev-tools/kmsan.rst | 414 ++++++++++++++++++++++++++++++ 2 files changed, 415 insertions(+) create mode 100644 Documentation/dev-tools/kmsan.rst diff --git a/Documentation/dev-tools/index.rst b/Documentation/dev-tools/in= dex.rst index 4621eac290f46..6b0663075dc04 100644 --- a/Documentation/dev-tools/index.rst +++ b/Documentation/dev-tools/index.rst @@ -24,6 +24,7 @@ Documentation/dev-tools/testing-overview.rst kcov gcov kasan + kmsan ubsan kmemleak kcsan diff --git a/Documentation/dev-tools/kmsan.rst b/Documentation/dev-tools/km= san.rst new file mode 100644 index 0000000000000..e116889da79d5 --- /dev/null +++ b/Documentation/dev-tools/kmsan.rst @@ -0,0 +1,414 @@ +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D +KernelMemorySanitizer (KMSAN) +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D + +KMSAN is a dynamic error detector aimed at finding uses of uninitialized +values. It is based on compiler instrumentation, and is quite similar to t= he +userspace `MemorySanitizer tool`_. + +An important note is that KMSAN is not intended for production use, becaus= e it +drastically increases kernel memory footprint and slows the whole system d= own. + +Example report +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Here is an example of a KMSAN report:: + + =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D + BUG: KMSAN: uninit-value in test_uninit_kmsan_check_memory+0x1be/0x380 [= kmsan_test] + test_uninit_kmsan_check_memory+0x1be/0x380 mm/kmsan/kmsan_test.c:273 + kunit_run_case_internal lib/kunit/test.c:333 + kunit_try_run_case+0x206/0x420 lib/kunit/test.c:374 + kunit_generic_run_threadfn_adapter+0x6d/0xc0 lib/kunit/try-catch.c:28 + kthread+0x721/0x850 kernel/kthread.c:327 + ret_from_fork+0x1f/0x30 ??:? + + Uninit was stored to memory at: + do_uninit_local_array+0xfa/0x110 mm/kmsan/kmsan_test.c:260 + test_uninit_kmsan_check_memory+0x1a2/0x380 mm/kmsan/kmsan_test.c:271 + kunit_run_case_internal lib/kunit/test.c:333 + kunit_try_run_case+0x206/0x420 lib/kunit/test.c:374 + kunit_generic_run_threadfn_adapter+0x6d/0xc0 lib/kunit/try-catch.c:28 + kthread+0x721/0x850 kernel/kthread.c:327 + ret_from_fork+0x1f/0x30 ??:? + + Local variable uninit created at: + do_uninit_local_array+0x4a/0x110 mm/kmsan/kmsan_test.c:256 + test_uninit_kmsan_check_memory+0x1a2/0x380 mm/kmsan/kmsan_test.c:271 + + Bytes 4-7 of 8 are uninitialized + Memory access of size 8 starts at ffff888083fe3da0 + + CPU: 0 PID: 6731 Comm: kunit_try_catch Tainted: G B E 5.16.= 0-rc3+ #104 + Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-2 04/= 01/2014 + =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D + + +The report says that the local variable ``uninit`` was created uninitializ= ed in +``do_uninit_local_array()``. The lower stack trace corresponds to the place +where this variable was created. + +The upper stack shows where the uninit value was used - in +``test_uninit_kmsan_check_memory()``. The tool shows the bytes which were = left +uninitialized in the local variable, as well as the stack where the value = was +copied to another memory location before use. + +Please note that KMSAN only reports an error when an uninitialized value is +actually used (e.g. in a condition or pointer dereference). A lot of +uninitialized values in the kernel are never used, and reporting them would +result in too many false positives. + +KMSAN and Clang +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +In order for KMSAN to work the kernel must be built with Clang, which so f= ar is +the only compiler that has KMSAN support. The kernel instrumentation pass = is +based on the userspace `MemorySanitizer tool`_. + +How to build +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +In order to build a kernel with KMSAN you will need a fresh Clang (14.0.0+= ). +Please refer to `LLVM documentation`_ for the instructions on how to build= Clang. + +Now configure and build the kernel with CONFIG_KMSAN enabled. + +How KMSAN works +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +KMSAN shadow memory +------------------- + +KMSAN associates a metadata byte (also called shadow byte) with every byte= of +kernel memory. A bit in the shadow byte is set iff the corresponding bit o= f the +kernel memory byte is uninitialized. Marking the memory uninitialized (i.e. +setting its shadow bytes to ``0xff``) is called poisoning, marking it +initialized (setting the shadow bytes to ``0x00``) is called unpoisoning. + +When a new variable is allocated on the stack, it is poisoned by default by +instrumentation code inserted by the compiler (unless it is a stack variab= le +that is immediately initialized). Any new heap allocation done without +``__GFP_ZERO`` is also poisoned. + +Compiler instrumentation also tracks the shadow values with the help from = the +runtime library in ``mm/kmsan/``. + +The shadow value of a basic or compound type is an array of bytes of the s= ame +length. When a constant value is written into memory, that memory is unpoi= soned. +When a value is read from memory, its shadow memory is also obtained and +propagated into all the operations which use that value. For every instruc= tion +that takes one or more values the compiler generates code that calculates = the +shadow of the result depending on those values and their shadows. + +Example:: + + int a =3D 0xff; // i.e. 0x000000ff + int b; + int c =3D a | b; + +In this case the shadow of ``a`` is ``0``, shadow of ``b`` is ``0xffffffff= ``, +shadow of ``c`` is ``0xffffff00``. This means that the upper three bytes of +``c`` are uninitialized, while the lower byte is initialized. + + +Origin tracking +--------------- + +Every four bytes of kernel memory also have a so-called origin assigned to +them. This origin describes the point in program execution at which the +uninitialized value was created. Every origin is associated with either the +full allocation stack (for heap-allocated memory), or the function contain= ing +the uninitialized variable (for locals). + +When an uninitialized variable is allocated on stack or heap, a new origin +value is created, and that variable's origin is filled with that value. +When a value is read from memory, its origin is also read and kept together +with the shadow. For every instruction that takes one or more values the o= rigin +of the result is one of the origins corresponding to any of the uninitiali= zed +inputs. If a poisoned value is written into memory, its origin is written = to the +corresponding storage as well. + +Example 1:: + + int a =3D 42; + int b; + int c =3D a + b; + +In this case the origin of ``b`` is generated upon function entry, and is +stored to the origin of ``c`` right before the addition result is written = into +memory. + +Several variables may share the same origin address, if they are stored in= the +same four-byte chunk. In this case every write to either variable updates = the +origin for all of them. We have to sacrifice precision in this case, becau= se +storing origins for individual bits (and even bytes) would be too costly. + +Example 2:: + + int combine(short a, short b) { + union ret_t { + int i; + short s[2]; + } ret; + ret.s[0] =3D a; + ret.s[1] =3D b; + return ret.i; + } + +If ``a`` is initialized and ``b`` is not, the shadow of the result would be +0xffff0000, and the origin of the result would be the origin of ``b``. +``ret.s[0]`` would have the same origin, but it will be never used, because +that variable is initialized. + +If both function arguments are uninitialized, only the origin of the second +argument is preserved. + +Origin chaining +~~~~~~~~~~~~~~~ + +To ease debugging, KMSAN creates a new origin for every store of an +uninitialized value to memory. The new origin references both its creation= stack +and the previous origin the value had. This may cause increased memory +consumption, so we limit the length of origin chains in the runtime. + +Clang instrumentation API +------------------------- + +Clang instrumentation pass inserts calls to functions defined in +``mm/kmsan/instrumentation.c`` into the kernel code. + +Shadow manipulation +~~~~~~~~~~~~~~~~~~~ + +For every memory access the compiler emits a call to a function that retur= ns a +pair of pointers to the shadow and origin addresses of the given memory:: + + typedef struct { + void *shadow, *origin; + } shadow_origin_ptr_t + + shadow_origin_ptr_t __msan_metadata_ptr_for_load_{1,2,4,8}(void *addr) + shadow_origin_ptr_t __msan_metadata_ptr_for_store_{1,2,4,8}(void *addr) + shadow_origin_ptr_t __msan_metadata_ptr_for_load_n(void *addr, uintptr_t= size) + shadow_origin_ptr_t __msan_metadata_ptr_for_store_n(void *addr, uintptr_= t size) + +The function name depends on the memory access size. + +The compiler makes sure that for every loaded value its shadow and origin +values are read from memory. When a value is stored to memory, its shadow = and +origin are also stored using the metadata pointers. + +Origin tracking +~~~~~~~~~~~~~~~ + +A special function is used to create a new origin value for a local variab= le and +set the origin of that variable to that value:: + + void __msan_poison_alloca(void *addr, uintptr_t size, char *descr) + +Access to per-task data +~~~~~~~~~~~~~~~~~~~~~~~~~ + +At the beginning of every instrumented function KMSAN inserts a call to +``__msan_get_context_state()``:: + + kmsan_context_state *__msan_get_context_state(void) + +``kmsan_context_state`` is declared in ``include/linux/kmsan.h``:: + + struct kmsan_context_state { + char param_tls[KMSAN_PARAM_SIZE]; + char retval_tls[KMSAN_RETVAL_SIZE]; + char va_arg_tls[KMSAN_PARAM_SIZE]; + char va_arg_origin_tls[KMSAN_PARAM_SIZE]; + u64 va_arg_overflow_size_tls; + char param_origin_tls[KMSAN_PARAM_SIZE]; + depot_stack_handle_t retval_origin_tls; + }; + +This structure is used by KMSAN to pass parameter shadows and origins betw= een +instrumented functions. + +String functions +~~~~~~~~~~~~~~~~ + +The compiler replaces calls to ``memcpy()``/``memmove()``/``memset()`` wit= h the +following functions. These functions are also called when data structures = are +initialized or copied, making sure shadow and origin values are copied alo= ngside +with the data:: + + void *__msan_memcpy(void *dst, void *src, uintptr_t n) + void *__msan_memmove(void *dst, void *src, uintptr_t n) + void *__msan_memset(void *dst, int c, uintptr_t n) + +Error reporting +~~~~~~~~~~~~~~~ + +For each pointer dereference and each condition the compiler emits a shadow +check that calls ``__msan_warning()`` in the case a poisoned value is being +used:: + + void __msan_warning(u32 origin) + +``__msan_warning()`` causes KMSAN runtime to print an error report. + +Inline assembly instrumentation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +KMSAN instruments every inline assembly output with a call to:: + + void __msan_instrument_asm_store(void *addr, uintptr_t size) + +, which unpoisons the memory region. + +This approach may mask certain errors, but it also helps to avoid a lot of +false positives in bitwise operations, atomics etc. + +Sometimes the pointers passed into inline assembly do not point to valid m= emory. +In such cases they are ignored at runtime. + +Disabling the instrumentation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A function can be marked with ``__no_kmsan_checks``. Doing so makes KMSAN +ignore uninitialized values in that function and mark its output as initia= lized. +As a result, the user will not get KMSAN reports related to that function. + +Another function attribute supported by KMSAN is ``__no_sanitize_memory``. +Applying this attribute to a function will result in KMSAN not instrumenti= ng it, +which can be helpful if we do not want the compiler to mess up some low-le= vel +code (e.g. that marked with ``noinstr``). + +This however comes at a cost: stack allocations from such functions will h= ave +incorrect shadow/origin values, likely leading to false positives. Functio= ns +called from non-instrumented code may also receive incorrect metadata for = their +parameters. + +As a rule of thumb, avoid using ``__no_sanitize_memory`` explicitly. + +It is also possible to disable KMSAN for a single file (e.g. main.o):: + + KMSAN_SANITIZE_main.o :=3D n + +or for the whole directory:: + + KMSAN_SANITIZE :=3D n + +in the Makefile. Think of this as applying ``__no_sanitize_memory`` to eve= ry +function in the file or directory. Most users won't need KMSAN_SANITIZE, u= nless +their code gets broken by KMSAN (e.g. runs at early boot time). + +Runtime library +--------------- + +The code is located in ``mm/kmsan/``. + +Per-task KMSAN state +~~~~~~~~~~~~~~~~~~~~ + +Every task_struct has an associated KMSAN task state that holds the KMSAN +context (see above) and a per-task flag disallowing KMSAN reports:: + + struct kmsan_context { + ... + bool allow_reporting; + struct kmsan_context_state cstate; + ... + } + + struct task_struct { + ... + struct kmsan_context kmsan; + ... + } + + +KMSAN contexts +~~~~~~~~~~~~~~ + +When running in a kernel task context, KMSAN uses ``current->kmsan.cstate`= ` to +hold the metadata for function parameters and return values. + +But in the case the kernel is running in the interrupt, softirq or NMI con= text, +where ``current`` is unavailable, KMSAN switches to per-cpu interrupt stat= e:: + + DEFINE_PER_CPU(struct kmsan_ctx, kmsan_percpu_ctx); + +Metadata allocation +~~~~~~~~~~~~~~~~~~~ + +There are several places in the kernel for which the metadata is stored. + +1. Each ``struct page`` instance contains two pointers to its shadow and +origin pages:: + + struct page { + ... + struct page *shadow, *origin; + ... + }; + +At boot-time, the kernel allocates shadow and origin pages for every avail= able +kernel page. This is done quite late, when the kernel address space is alr= eady +fragmented, so normal data pages may arbitrarily interleave with the metad= ata +pages. + +This means that in general for two contiguous memory pages their shadow/or= igin +pages may not be contiguous. So, if a memory access crosses the boundary +of a memory block, accesses to shadow/origin memory may potentially corrupt +other pages or read incorrect values from them. + +In practice, contiguous memory pages returned by the same ``alloc_pages()`` +call will have contiguous metadata, whereas if these pages belong to two +different allocations their metadata pages can be fragmented. + +For the kernel data (``.data``, ``.bss`` etc.) and percpu memory regions +there also are no guarantees on metadata contiguity. + +In the case ``__msan_metadata_ptr_for_XXX_YYY()`` hits the border between = two +pages with non-contiguous metadata, it returns pointers to fake shadow/ori= gin regions:: + + char dummy_load_page[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE))); + char dummy_store_page[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE))); + +``dummy_load_page`` is zero-initialized, so reads from it always yield zer= oes. +All stores to ``dummy_store_page`` are ignored. + +2. For vmalloc memory and modules, there is a direct mapping between the m= emory +range, its shadow and origin. KMSAN reduces the vmalloc area by 3/4, makin= g only +the first quarter available to ``vmalloc()``. The second quarter of the vm= alloc +area contains shadow memory for the first quarter, the third one holds the +origins. A small part of the fourth quarter contains shadow and origins fo= r the +kernel modules. Please refer to ``arch/x86/include/asm/pgtable_64_types.h`= ` for +more details. + +When an array of pages is mapped into a contiguous virtual memory space, t= heir +shadow and origin pages are similarly mapped into contiguous regions. + +3. For CPU entry area there are separate per-CPU arrays that hold its +metadata:: + + DEFINE_PER_CPU(char[CPU_ENTRY_AREA_SIZE], cpu_entry_area_shadow); + DEFINE_PER_CPU(char[CPU_ENTRY_AREA_SIZE], cpu_entry_area_origin); + +When calculating shadow and origin addresses for a given memory address, K= MSAN +checks whether the address belongs to the physical page range, the virtual= page +range or CPU entry area. + +Handling ``pt_regs`` +~~~~~~~~~~~~~~~~~~~~ + +Many functions receive a ``struct pt_regs`` holding the register state at a +certain point. Registers do not have (easily calculatable) shadow or origin +associated with them, so we assume they are always initialized. + +References +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +E. Stepanov, K. Serebryany. `MemorySanitizer: fast detector of uninitializ= ed +memory use in C++ +`_. +In Proceedings of CGO 2015. + +.. _MemorySanitizer tool: https://clang.llvm.org/docs/MemorySanitizer.html +.. _LLVM documentation: https://llvm.org/docs/GettingStarted.html --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA643C433F5 for ; Tue, 26 Apr 2022 16:45:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353167AbiDZQsM (ORCPT ); Tue, 26 Apr 2022 12:48:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40454 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353096AbiDZQru (ORCPT ); Tue, 26 Apr 2022 12:47:50 -0400 Received: from mail-ej1-x649.google.com (mail-ej1-x649.google.com [IPv6:2a00:1450:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 413421906BA for ; Tue, 26 Apr 2022 09:44:42 -0700 (PDT) Received: by mail-ej1-x649.google.com with SMTP id sa27-20020a1709076d1b00b006e8b357a2e7so9325546ejc.14 for ; Tue, 26 Apr 2022 09:44:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=/0H5A8X92xIp0p19j3P2M66qBbmfiGtveehlyZCJw24=; b=FvkMkIGRscjUzYZMvLVFQr9H/QRVOHYZw4hHknosvP6NOZtQxpNsZd0RtzhJXelcRc z988hAfskDDmoJnPIiI0jXNGvUulB8rFxBfA8ab5Sms91LUTaW4M13z8UFY9KvIBTi0U mydviyOj8aDprvPJfYeUMhjCpbeC8rH4hZxCn8OWbU3YolhExGVHn2JyusklOjF/6sO4 vYyxMokIMpAW+3L8FTBFk9+HgjrICCnt0AsWcRVYbtUA2ELiqXnXSxBNwEJZcwS3tI4o 91Z59BTGhx3wH4FUgP2JaAvwZ5iQeqRkzHHrNtLbtKEtxaZ4rzTmakhpwXtIFjy1dzme JxoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=/0H5A8X92xIp0p19j3P2M66qBbmfiGtveehlyZCJw24=; b=7O25rQT98YP6Rxfu6TYJ6A1FB8UfRV7Oyid2sXuGOD6Wotfeixo4ApuDuuXgj68RmA 8yGQpWWwbZ7C1APPKbAwjKcSAdbKI5HeOaUZUCow+Ey3jiy2A9EpikrMxeoO76NXaTQv 38ICYtPlz5EGIrqwId1wkecn24sByGGJcPksuLAT5z7VtF29KHdFPo0ARaOLNcDMGSvQ 5XUMy8V3JTkmXXXkE+tcVVxRkXBZTgVM4E9EhWFVB5oA9YXYj1x/9j7h/YgobpcJotwU t1gD5VIBJxMmVSqwtqGVoncRP3vl+LdIws2H/5XWu+aCJ7V7tcJUzxdqBzqaenFIqLaM LLow== X-Gm-Message-State: AOAM533jfSUs8XgY7MlSR4sH+JRKLSoo3RmP4PpQEvtdG3F/yaUdxTLU DGicIRsF2feT7k09NrSQUkwccV84V3s= X-Google-Smtp-Source: ABdhPJyznSX2i8WSeLQTTkki2cbZTho3ac9KkIH//+BcoF3wC20oGterW7mlNzbNsl8P2+ms/2tHQUoo1l8= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:aa7:d310:0:b0:425:f22f:763f with SMTP id p16-20020aa7d310000000b00425f22f763fmr9236955edq.163.1650991480781; Tue, 26 Apr 2022 09:44:40 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:37 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-9-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 08/46] kmsan: introduce __no_sanitize_memory and __no_kmsan_checks From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" __no_sanitize_memory is a function attribute that instructs KMSAN to skip a function during instrumentation. This is needed to e.g. implement the noinstr functions. __no_kmsan_checks is a function attribute that makes KMSAN ignore the uninitialized values coming from the function's inputs, and initialize the function's outputs. Functions marked with this attribute can't be inlined into functions not marked with it, and vice versa. This behavior is overridden by __always_inline. __SANITIZE_MEMORY__ is a macro that's defined iff the file is instrumented with KMSAN. This is not the same as CONFIG_KMSAN, which is defined for every file. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/I004ff0360c918d3cd8b18767ddd= 1381c6d3281be --- include/linux/compiler-clang.h | 23 +++++++++++++++++++++++ include/linux/compiler-gcc.h | 6 ++++++ 2 files changed, 29 insertions(+) diff --git a/include/linux/compiler-clang.h b/include/linux/compiler-clang.h index babb1347148c5..c561064921449 100644 --- a/include/linux/compiler-clang.h +++ b/include/linux/compiler-clang.h @@ -51,6 +51,29 @@ #define __no_sanitize_undefined #endif =20 +#if __has_feature(memory_sanitizer) +#define __SANITIZE_MEMORY__ +/* + * Unlike other sanitizers, KMSAN still inserts code into functions marked= with + * no_sanitize("kernel-memory"). Using disable_sanitizer_instrumentation + * provides the behavior consistent with other __no_sanitize_ attributes, + * guaranteeing that __no_sanitize_memory functions remain uninstrumented. + */ +#define __no_sanitize_memory __disable_sanitizer_instrumentation + +/* + * The __no_kmsan_checks attribute ensures that a function does not produce + * false positive reports by: + * - initializing all local variables and memory stores in this function; + * - skipping all shadow checks; + * - passing initialized arguments to this function's callees. + */ +#define __no_kmsan_checks __attribute__((no_sanitize("kernel-memory"))) +#else +#define __no_sanitize_memory +#define __no_kmsan_checks +#endif + /* * Support for __has_feature(coverage_sanitizer) was added in Clang 13 tog= ether * with no_sanitize("coverage"). Prior versions of Clang support coverage diff --git a/include/linux/compiler-gcc.h b/include/linux/compiler-gcc.h index 52299c957c98e..f1a7ce3f6e6fd 100644 --- a/include/linux/compiler-gcc.h +++ b/include/linux/compiler-gcc.h @@ -133,6 +133,12 @@ #define __SANITIZE_ADDRESS__ #endif =20 +/* + * GCC does not support KMSAN. + */ +#define __no_sanitize_memory +#define __no_kmsan_checks + /* * Turn individual warnings and errors on and off locally, depending * on version. --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B97D4C433EF for ; Tue, 26 Apr 2022 16:45:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353183AbiDZQsc (ORCPT ); Tue, 26 Apr 2022 12:48:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40824 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353132AbiDZQrx (ORCPT ); Tue, 26 Apr 2022 12:47:53 -0400 Received: from mail-lj1-x249.google.com (mail-lj1-x249.google.com [IPv6:2a00:1450:4864:20::249]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 47B14191446 for ; Tue, 26 Apr 2022 09:44:45 -0700 (PDT) Received: by mail-lj1-x249.google.com with SMTP id l8-20020a2ea808000000b0024da289e41dso4843014ljq.7 for ; Tue, 26 Apr 2022 09:44:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=ednioRVQi+463GyxSxojLtYXRmMb4k6reZrR8s7PDTk=; b=s7nDKeE2uO8tPBSHXhKtgeVd4KJBULrFKzji5KxbrLSqAsd9qwd5XqsiW4sxXc2Zuj /LDHzi6oYDxLjayBmNbbHHd8x60Z1q11m24TiSwhP8RYbreiv3W6YuCWDnMHnOA40Zg6 2XAh/SQC3o2WwNcd888BCR1ayDkPme9nLNf+yFJmAhhH+5m2makPEnGrUgDcpKDvSpAx uZ/EMSRFu7oZ7g4w8LFxCUmwK7SAeXqsUvws74aRat1Udaq+/e8oYjQBcJ/MQ0phtMYp ItWJ4UPus1+bGv6Mz6YHzcXh/jQw/SqfpbBVzObyfR2JQ5g2erl9ttRuMQ0dfEHG3Yf8 GoPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ednioRVQi+463GyxSxojLtYXRmMb4k6reZrR8s7PDTk=; b=io1OmFgsqAM6OdKPBiLUsx5B5NGYGAeOC+i0OLtlWZIkiRs8u2akNbtA2E3rJHd5Kl +04GS8uLj4QFrfO5yyfXNdeQbsGHcthEfRHQBlxHsmEjY4xK1gsaVKmTn2avPiOtSIFY CLlt6RPA200M51E5kXtjIQ6H2rGG/Ab1DM7CEucKP0ytGc89V8148qmLgI4ssKHV/ig3 DVktISon6qblLWctEiQMPIJiith7beVFX35TNttAvc606b3LS9Mln5dfTG9gq4gkKi5T j4S2BtJ0/SogB5JCdfZYIruPdvI5GoWvOJnlisynVZBzkbiSBSZrCP+14nhDHsuG7BWn fnVg== X-Gm-Message-State: AOAM532PLZpaEtbBHe//z4GmQAcj17ApIfiw83/OZQIPG6KlH28ESxDP d6P260aREUkB1Aj2Jphz5aKd9dMtqU8= X-Google-Smtp-Source: ABdhPJxHrJcKOCcTiJCxWbbHxqyDNc8yKbgnQVG+zhnL2F1NE0oryK8Ze862y3s98aS3FdAQ1AuU/NMiTCE= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6512:114f:b0:471:b097:4a29 with SMTP id m15-20020a056512114f00b00471b0974a29mr17572189lfg.93.1650991483323; Tue, 26 Apr 2022 09:44:43 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:38 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-10-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 09/46] kmsan: mark noinstr as __no_sanitize_memory From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" noinstr functions should never be instrumented, so make KMSAN skip them by applying the __no_sanitize_memory attribute. Signed-off-by: Alexander Potapenko --- v2: -- moved this patch earlier in the series per Mark Rutland's request Link: https://linux-review.googlesource.com/id/I3c9abe860b97b49bc0c8026918b= 17a50448dec0d --- include/linux/compiler_types.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h index 1c2c33ae1b37d..a9ba5edd8208b 100644 --- a/include/linux/compiler_types.h +++ b/include/linux/compiler_types.h @@ -227,7 +227,8 @@ struct ftrace_likely_data { /* Section for code which can't be instrumented at all */ #define noinstr \ noinline notrace __attribute((__section__(".noinstr.text"))) \ - __no_kcsan __no_sanitize_address __no_profile __no_sanitize_coverage + __no_kcsan __no_sanitize_address __no_profile __no_sanitize_coverage \ + __no_sanitize_memory =20 #endif /* __KERNEL__ */ =20 --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9E67C433F5 for ; Tue, 26 Apr 2022 16:45:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353205AbiDZQsk (ORCPT ); Tue, 26 Apr 2022 12:48:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41980 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353180AbiDZQs1 (ORCPT ); Tue, 26 Apr 2022 12:48:27 -0400 Received: from mail-ej1-x64a.google.com (mail-ej1-x64a.google.com [IPv6:2a00:1450:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B49EC191C63 for ; Tue, 26 Apr 2022 09:44:47 -0700 (PDT) Received: by mail-ej1-x64a.google.com with SMTP id go12-20020a1709070d8c00b006f009400732so9165457ejc.1 for ; Tue, 26 Apr 2022 09:44:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=H2XcU3OLx+0JTyyML32bzgUT5MJiV2tDmnYI11nI1dE=; b=m/GoUZBVtlZMRq+FF0SH5aCHlTGSvaapuxEDbVMKlEUnlusQPCCZ7ebjbcCdUrWYb5 a8Bnd3p86A4nE5NEzf9NbCXsFTEls9lTknebhOvAE06AdOtYHsT4/XF7t90n+nJ5iebk sRGdLEAOOW9bUwdhOZndjscgKwN1eOVOb9hwxXJ7DP2x/BYwqNzZ6yzf2ohW5qzq+FEZ g3mr3p0uqiTWn5KToQ7MIQy5yDLS3i1YymgzGDrCkkGyLo8f/tQJWjvGzdC5snsMrVUx RKkkljd+V3QJEWhwnCDg+P32Vbl+YhkiLRuywjeqifPG0/ok+r70FeWu7XYIUm7eYfDk 3sOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=H2XcU3OLx+0JTyyML32bzgUT5MJiV2tDmnYI11nI1dE=; b=NNFBuUQrZ+kAATzJtYPjMdyG2wmR87LbrBcRAwEPK4CXdg7Q3jQYHO8Fx/fYepV8pj gyIrWLKtjUNpmj1lpyc4JZkEmiKG1gzalAbA3/scqcpm2TkkG8Fg3UGIsR5lfa7ik3XF pceBtoXvWvUq4zX5BaDcRcg7zJzvCgU4vNZi+fGsQdR3q+fR2RYF2Rx3Ucxfj0YVU7F6 GeKpAq2eRTTojdGuA6TBg9q9+WWvQYfH3FnH1hDcVt1dLCIsUJ824Cj77CBtVHmt22eS RuTvdyvkCB/CdjnJb/sCTa5No3sBoDNLuHXsrM5YSn56r0P//+7Qbdvfy4dSoILPclwy jnqw== X-Gm-Message-State: AOAM532HFAf4aCst3QHhKUJBRuoH+IOruOXr7XAxJOkjeYG7XiopCJO9 cVS74CsP5w4hFzhR8TyVgge0ElUMua0= X-Google-Smtp-Source: ABdhPJxuZFKkNdAIYxCe6Wr3rHxB/+GLrdr6eXOf7wzKyJPWcdL4QfIWrMfSB3fnw8xzMi6hwbSucSjlH8I= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:4315:b0:426:155:e4a3 with SMTP id m21-20020a056402431500b004260155e4a3mr1554638edc.324.1650991485934; Tue, 26 Apr 2022 09:44:45 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:39 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-11-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 10/46] x86: kmsan: pgtable: reduce vmalloc space From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" KMSAN is going to use 3/4 of existing vmalloc space to hold the metadata, therefore we lower VMALLOC_END to make sure vmalloc() doesn't allocate past the first 1/4. Signed-off-by: Alexander Potapenko --- v2: -- added x86: to the title Link: https://linux-review.googlesource.com/id/I9d8b7f0a88a639f1263bc693cbd= 5c136626f7efd --- arch/x86/include/asm/pgtable_64_types.h | 41 ++++++++++++++++++++++++- arch/x86/mm/init_64.c | 2 +- 2 files changed, 41 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/pgtable_64_types.h b/arch/x86/include/asm= /pgtable_64_types.h index 91ac106545703..7f15d43754a34 100644 --- a/arch/x86/include/asm/pgtable_64_types.h +++ b/arch/x86/include/asm/pgtable_64_types.h @@ -139,7 +139,46 @@ extern unsigned int ptrs_per_p4d; # define VMEMMAP_START __VMEMMAP_BASE_L4 #endif /* CONFIG_DYNAMIC_MEMORY_LAYOUT */ =20 -#define VMALLOC_END (VMALLOC_START + (VMALLOC_SIZE_TB << 40) - 1) +#define VMEMORY_END (VMALLOC_START + (VMALLOC_SIZE_TB << 40) - 1) + +#ifndef CONFIG_KMSAN +#define VMALLOC_END VMEMORY_END +#else +/* + * In KMSAN builds vmalloc area is four times smaller, and the remaining 3= /4 + * are used to keep the metadata for virtual pages. The memory formerly + * belonging to vmalloc area is now laid out as follows: + * + * 1st quarter: VMALLOC_START to VMALLOC_END - new vmalloc area + * 2nd quarter: KMSAN_VMALLOC_SHADOW_START to + * VMALLOC_END+KMSAN_VMALLOC_SHADOW_OFFSET - vmalloc area sha= dow + * 3rd quarter: KMSAN_VMALLOC_ORIGIN_START to + * VMALLOC_END+KMSAN_VMALLOC_ORIGIN_OFFSET - vmalloc area ori= gins + * 4th quarter: KMSAN_MODULES_SHADOW_START to KMSAN_MODULES_ORIGIN_START + * - shadow for modules, + * KMSAN_MODULES_ORIGIN_START to + * KMSAN_MODULES_ORIGIN_START + MODULES_LEN - origins for mod= ules. + */ +#define VMALLOC_QUARTER_SIZE ((VMALLOC_SIZE_TB << 40) >> 2) +#define VMALLOC_END (VMALLOC_START + VMALLOC_QUARTER_SIZE - 1) + +/* + * vmalloc metadata addresses are calculated by adding shadow/origin offse= ts + * to vmalloc address. + */ +#define KMSAN_VMALLOC_SHADOW_OFFSET VMALLOC_QUARTER_SIZE +#define KMSAN_VMALLOC_ORIGIN_OFFSET (VMALLOC_QUARTER_SIZE << 1) + +#define KMSAN_VMALLOC_SHADOW_START (VMALLOC_START + KMSAN_VMALLOC_SHADOW_O= FFSET) +#define KMSAN_VMALLOC_ORIGIN_START (VMALLOC_START + KMSAN_VMALLOC_ORIGIN_O= FFSET) + +/* + * The shadow/origin for modules are placed one by one in the last 1/4 of + * vmalloc space. + */ +#define KMSAN_MODULES_SHADOW_START (VMALLOC_END + KMSAN_VMALLOC_ORIGIN_OFF= SET + 1) +#define KMSAN_MODULES_ORIGIN_START (KMSAN_MODULES_SHADOW_START + MODULES_L= EN) +#endif /* CONFIG_KMSAN */ =20 #define MODULES_VADDR (__START_KERNEL_map + KERNEL_IMAGE_SIZE) /* The module sections ends with the start of the fixmap */ diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 96d34ebb20a9e..fcea37beb3911 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -1287,7 +1287,7 @@ static void __init preallocate_vmalloc_pages(void) unsigned long addr; const char *lvl; =20 - for (addr =3D VMALLOC_START; addr <=3D VMALLOC_END; addr =3D ALIGN(addr += 1, PGDIR_SIZE)) { + for (addr =3D VMALLOC_START; addr <=3D VMEMORY_END; addr =3D ALIGN(addr += 1, PGDIR_SIZE)) { pgd_t *pgd =3D pgd_offset_k(addr); p4d_t *p4d; pud_t *pud; --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9AB04C433FE for ; Tue, 26 Apr 2022 16:45:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353236AbiDZQsr (ORCPT ); Tue, 26 Apr 2022 12:48:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43198 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353186AbiDZQs2 (ORCPT ); Tue, 26 Apr 2022 12:48:28 -0400 Received: from mail-ed1-x54a.google.com (mail-ed1-x54a.google.com [IPv6:2a00:1450:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 420FA1942EB for ; Tue, 26 Apr 2022 09:44:50 -0700 (PDT) Received: by mail-ed1-x54a.google.com with SMTP id c23-20020a50d657000000b00425d5162a0dso4685761edj.16 for ; Tue, 26 Apr 2022 09:44:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=SEm8RFoD3mzqXjcHWM9Z/XfNnVTnRxEBaTJIwFUC6FI=; b=V1UPpoK10uyPLSVeGxmKmWLpz7MVRMo+4uiBOQTE+btQjeiF7bX14i8xIN4hWxvfsI BKp6ezW66yD/fgl4XMpuBdON6QxtTZCTto768iFnXFc2LH7l5QwZpTox8pUyK0cZZMcU 9diZqYTWDNczn8waY+ysJcw4lWMp4tfA90aPMraK8ja0L3fYlvQlBZsap+cS3cIoYkUy i+K1sOjum2D1Snfi9ezBpkT3FqJQbpZ1ZkJb010JMdv/VFlzNAOneWXO4NmYGXJdCYB8 LxjKRUcmGqytLknAlHvn7Ijdu7TEekTSbAaIp3JV0enFaH8f7pcQ4jDbiUGAEReAXkN8 4fmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=SEm8RFoD3mzqXjcHWM9Z/XfNnVTnRxEBaTJIwFUC6FI=; b=3pnjIM053WohXYjmIU1aiXyXsXwFy8FYg2tlMI4ozvSKzEACB6wV+ccUBZKCx3TOFc sFa7Ek65Exvuhm8jtu9Jnn1I1hr6LPIw4TJ12ptQkww3cVaRNIWe3NqXUicuAtunFeIo Co5dHGpG4Q9OCHWWrvzo7N0ecqBLd94i2TUVGnEJBj6Ikh5VHqgPrAp+gOhpdnidILka L1Ja68pH4NZfeG1XTYDuCLvCGwClLmG3P8Sfjt+TdabZtYVATHfisebxH42ajjEsxoW4 Rg2U5bJfUZtqL/Le93hjlkefmDrPSd6VMp0ShCdU2zUQWAh3hsLx8WB4CDnajQ0+8SSL UHbw== X-Gm-Message-State: AOAM532Y64kaRlPHLT+21H2V83QcdLxfxUv3MYMyou6OP1cVJ1Izq+Uq Hzvm3b0S7wk08t12TPIvzI/bfOIVbBo= X-Google-Smtp-Source: ABdhPJzlxnxEzq09x8VRCHG67jl1BT+tZuNt6ukFhmOsg7s6+uxFS5/FlyCj/NdcpcGzVA6sRk/t/wnlBr4= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:35d2:b0:424:1eb0:45c2 with SMTP id z18-20020a05640235d200b004241eb045c2mr25704880edc.152.1650991488637; Tue, 26 Apr 2022 09:44:48 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:40 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-12-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 11/46] libnvdimm/pfn_dev: increase MAX_STRUCT_PAGE_SIZE From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" KMSAN adds extra metadata fields to struct page, so it does not fit into 64 bytes anymore. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/I353796acc6a850bfd7bb342aa1b= 63e616fc614f1 --- drivers/nvdimm/nd.h | 2 +- drivers/nvdimm/pfn_devs.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/nvdimm/nd.h b/drivers/nvdimm/nd.h index ec5219680092d..85ca5b4da3cf3 100644 --- a/drivers/nvdimm/nd.h +++ b/drivers/nvdimm/nd.h @@ -652,7 +652,7 @@ void devm_namespace_disable(struct device *dev, struct nd_namespace_common *ndns); #if IS_ENABLED(CONFIG_ND_CLAIM) /* max struct page size independent of kernel config */ -#define MAX_STRUCT_PAGE_SIZE 64 +#define MAX_STRUCT_PAGE_SIZE 128 int nvdimm_setup_pfn(struct nd_pfn *nd_pfn, struct dev_pagemap *pgmap); #else static inline int nvdimm_setup_pfn(struct nd_pfn *nd_pfn, diff --git a/drivers/nvdimm/pfn_devs.c b/drivers/nvdimm/pfn_devs.c index c31e184bfa45e..d51a3cd6581b1 100644 --- a/drivers/nvdimm/pfn_devs.c +++ b/drivers/nvdimm/pfn_devs.c @@ -784,7 +784,7 @@ static int nd_pfn_init(struct nd_pfn *nd_pfn) * when populating the vmemmap. This *should* be equal to * PMD_SIZE for most architectures. * - * Also make sure size of struct page is less than 64. We + * Also make sure size of struct page is less than 128. We * want to make sure we use large enough size here so that * we don't have a dynamic reserve space depending on * struct page size. But we also want to make sure we notice --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F46DC433EF for ; Tue, 26 Apr 2022 16:46:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353239AbiDZQtX (ORCPT ); Tue, 26 Apr 2022 12:49:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43226 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348540AbiDZQs3 (ORCPT ); Tue, 26 Apr 2022 12:48:29 -0400 Received: from mail-ej1-x649.google.com (mail-ej1-x649.google.com [IPv6:2a00:1450:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 05FCB191C59 for ; Tue, 26 Apr 2022 09:44:53 -0700 (PDT) Received: by mail-ej1-x649.google.com with SMTP id sa27-20020a1709076d1b00b006e8b357a2e7so9325794ejc.14 for ; Tue, 26 Apr 2022 09:44:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc:content-transfer-encoding; bh=I+h0bEkbop6PXGR8KD8WntxvfCzQLAueILixWq8NhjE=; b=E6xJ/acQBP6/CIvPzNzrog8j0UQtY4+YDiBy1Y2P5ryLVQxievpREa605KL0ckgMgL PYQkbBb5fcgwcFcwstuSgrInoccYcmP+VDNaeegRem0WsF/vhZPd5acgG9TOzU3RhjAn AGjLlOXeGa6b6rh+VnjJpCUnAyuL7Axz2gtuiWtIElXfuFiL4lEhdW4T6u5n758zGSWO 30IWtaJyj7Z0UF1Au0eAJ613Z/j/b/QSiA6d26n1IpLLUTiun8TRgiVuvA5bNQ/KZHJc srwcxlNKsw4p5c3AbeIMtMPfcvNfA3fr108gsqpEesCinMcoYLS9+vqMNLjF4y6686/k OMWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc:content-transfer-encoding; bh=I+h0bEkbop6PXGR8KD8WntxvfCzQLAueILixWq8NhjE=; b=jWr4asyBX+EiPR/WVETI0usQhdPR7XsFJbMl1lHiae/yJl+MCJjCTUoM3MmQRJe2RQ /WmhU8R4H7CdV7wzOuMaF0/i/BKsJwPBjYuhw47w9E8XMTsMc4x4F5xXpXieCjhCoKD+ b8ouCb06Z17cn/UeqM2A04ujmIrscgKh5U9S/wCNncfrkMvOzJkIPNe5aGG18FYcWO/S spOMkOKvzqdMKv7ekuscMF9MqIrVQkRSpi/l9uukxXKIxiIxKYoXMVqEgGkWnDizURUN YGooNItXwLRDp1oo2nJRVDcy4VsbkVnEcdxekSM26DLjuWGDLJiRq8ZFWcdRa/m+5ynZ Sbbg== X-Gm-Message-State: AOAM531zGVsOhN4nZIiUYUlkLRrk4uQnW9Kwoj0F3oLnhtfnh6ft6PGO 7jtw3QkUz5GoMz9bUM2pjFm2+FrdT80= X-Google-Smtp-Source: ABdhPJyIxePGxygd36SgHLHKsG1CV7OIL0ps+CaI4w8xpkljTDxyZ3AJbqgX8s7GXr5EC8WrmChC38hBy3M= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:3486:b0:425:f2c6:9695 with SMTP id v6-20020a056402348600b00425f2c69695mr8713048edc.2.1650991491493; Tue, 26 Apr 2022 09:44:51 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:41 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-13-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 12/46] kmsan: add KMSAN runtime core From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" For each memory location KernelMemorySanitizer maintains two types of metadata: 1. The so-called shadow of that location - =D0=B0 byte:byte mapping describ= ing whether or not individual bits of memory are initialized (shadow is 0) or not (shadow is 1). 2. The origins of that location - =D0=B0 4-byte:4-byte mapping containing 4-byte IDs of the stack traces where uninitialized values were created. Each struct page now contains pointers to two struct pages holding KMSAN metadata (shadow and origins) for the original struct page. Utility routines in mm/kmsan/core.c and mm/kmsan/shadow.c handle the metadata creation, addressing, copying and checking. mm/kmsan/report.c performs error reporting in the cases an uninitialized value is used in a way that leads to undefined behavior. KMSAN compiler instrumentation is responsible for tracking the metadata along with the kernel memory. mm/kmsan/instrumentation.c provides the implementation for instrumentation hooks that are called from files compiled with -fsanitize=3Dkernel-memory. To aid parameter passing (also done at instrumentation level), each task_struct now contains a struct kmsan_task_state used to track the metadata of function parameters and return values for that task. Finally, this patch provides CONFIG_KMSAN that enables KMSAN, and declares CFLAGS_KMSAN, which are applied to files compiled with KMSAN. The KMSAN_SANITIZE:=3Dn Makefile directive can be used to completely disable KMSAN instrumentation for certain files. Similarly, KMSAN_ENABLE_CHECKS:=3Dn disables KMSAN checks and makes newly created stack memory initialized. Users can also use functions from include/linux/kmsan-checks.h to mark certain memory regions as uninitialized or initialized (this is called "poisoning" and "unpoisoning") or check that a particular region is initialized. Signed-off-by: Alexander Potapenko --- v2: -- as requested by Greg K-H, moved hooks for different subsystems to respe= ctive patches, rewrote the patch description; -- addressed comments by Dmitry Vyukov; -- added a note about KMSAN being not intended for production use. -- fix case of unaligned dst in kmsan_internal_memmove_metadata() v3: -- print build IDs in reports where applicable -- drop redundant filter_irq_stacks(), unpoison the local passed to __stac= k_depot_save() -- remove a stray BUG() Link: https://linux-review.googlesource.com/id/I9b71bfe3425466c97159f9de006= 2e5e8e4fec866 --- Makefile | 1 + include/linux/kmsan-checks.h | 64 +++++ include/linux/kmsan.h | 47 ++++ include/linux/mm_types.h | 12 + include/linux/sched.h | 5 + lib/Kconfig.debug | 1 + lib/Kconfig.kmsan | 23 ++ mm/Makefile | 1 + mm/kmsan/Makefile | 18 ++ mm/kmsan/core.c | 458 +++++++++++++++++++++++++++++++++++ mm/kmsan/hooks.c | 66 +++++ mm/kmsan/instrumentation.c | 267 ++++++++++++++++++++ mm/kmsan/kmsan.h | 183 ++++++++++++++ mm/kmsan/report.c | 211 ++++++++++++++++ mm/kmsan/shadow.c | 186 ++++++++++++++ scripts/Makefile.kmsan | 1 + scripts/Makefile.lib | 9 + 17 files changed, 1553 insertions(+) create mode 100644 include/linux/kmsan-checks.h create mode 100644 include/linux/kmsan.h create mode 100644 lib/Kconfig.kmsan create mode 100644 mm/kmsan/Makefile create mode 100644 mm/kmsan/core.c create mode 100644 mm/kmsan/hooks.c create mode 100644 mm/kmsan/instrumentation.c create mode 100644 mm/kmsan/kmsan.h create mode 100644 mm/kmsan/report.c create mode 100644 mm/kmsan/shadow.c create mode 100644 scripts/Makefile.kmsan diff --git a/Makefile b/Makefile index c3ec1ea423797..d3c7dcd9f0fea 100644 --- a/Makefile +++ b/Makefile @@ -1009,6 +1009,7 @@ include-y :=3D scripts/Makefile.extrawarn include-$(CONFIG_DEBUG_INFO) +=3D scripts/Makefile.debug include-$(CONFIG_KASAN) +=3D scripts/Makefile.kasan include-$(CONFIG_KCSAN) +=3D scripts/Makefile.kcsan +include-$(CONFIG_KMSAN) +=3D scripts/Makefile.kmsan include-$(CONFIG_UBSAN) +=3D scripts/Makefile.ubsan include-$(CONFIG_KCOV) +=3D scripts/Makefile.kcov include-$(CONFIG_GCC_PLUGINS) +=3D scripts/Makefile.gcc-plugins diff --git a/include/linux/kmsan-checks.h b/include/linux/kmsan-checks.h new file mode 100644 index 0000000000000..a6522a0c28df9 --- /dev/null +++ b/include/linux/kmsan-checks.h @@ -0,0 +1,64 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * KMSAN checks to be used for one-off annotations in subsystems. + * + * Copyright (C) 2017-2022 Google LLC + * Author: Alexander Potapenko + * + */ + +#ifndef _LINUX_KMSAN_CHECKS_H +#define _LINUX_KMSAN_CHECKS_H + +#include + +#ifdef CONFIG_KMSAN + +/** + * kmsan_poison_memory() - Mark the memory range as uninitialized. + * @address: address to start with. + * @size: size of buffer to poison. + * @flags: GFP flags for allocations done by this function. + * + * Until other data is written to this range, KMSAN will treat it as + * uninitialized. Error reports for this memory will reference the call si= te of + * kmsan_poison_memory() as origin. + */ +void kmsan_poison_memory(const void *address, size_t size, gfp_t flags); + +/** + * kmsan_unpoison_memory() - Mark the memory range as initialized. + * @address: address to start with. + * @size: size of buffer to unpoison. + * + * Until other data is written to this range, KMSAN will treat it as + * initialized. + */ +void kmsan_unpoison_memory(const void *address, size_t size); + +/** + * kmsan_check_memory() - Check the memory range for being initialized. + * @address: address to start with. + * @size: size of buffer to check. + * + * If any piece of the given range is marked as uninitialized, KMSAN will = report + * an error. + */ +void kmsan_check_memory(const void *address, size_t size); + +#else + +static inline void kmsan_poison_memory(const void *address, size_t size, + gfp_t flags) +{ +} +static inline void kmsan_unpoison_memory(const void *address, size_t size) +{ +} +static inline void kmsan_check_memory(const void *address, size_t size) +{ +} + +#endif + +#endif /* _LINUX_KMSAN_CHECKS_H */ diff --git a/include/linux/kmsan.h b/include/linux/kmsan.h new file mode 100644 index 0000000000000..4e35f43eceaa9 --- /dev/null +++ b/include/linux/kmsan.h @@ -0,0 +1,47 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * KMSAN API for subsystems. + * + * Copyright (C) 2017-2022 Google LLC + * Author: Alexander Potapenko + * + */ +#ifndef _LINUX_KMSAN_H +#define _LINUX_KMSAN_H + +#include +#include +#include +#include +#include + +struct page; + +#ifdef CONFIG_KMSAN + +/* These constants are defined in the MSan LLVM instrumentation pass. */ +#define KMSAN_RETVAL_SIZE 800 +#define KMSAN_PARAM_SIZE 800 + +struct kmsan_context_state { + char param_tls[KMSAN_PARAM_SIZE]; + char retval_tls[KMSAN_RETVAL_SIZE]; + char va_arg_tls[KMSAN_PARAM_SIZE]; + char va_arg_origin_tls[KMSAN_PARAM_SIZE]; + u64 va_arg_overflow_size_tls; + char param_origin_tls[KMSAN_PARAM_SIZE]; + depot_stack_handle_t retval_origin_tls; +}; + +#undef KMSAN_PARAM_SIZE +#undef KMSAN_RETVAL_SIZE + +struct kmsan_ctx { + struct kmsan_context_state cstate; + int kmsan_in_runtime; + bool allow_reporting; +}; + +#endif + +#endif /* _LINUX_KMSAN_H */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 8834e38c06a4f..85c97a2145f7e 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -218,6 +218,18 @@ struct page { not kmapped, ie. highmem) */ #endif /* WANT_PAGE_VIRTUAL */ =20 +#ifdef CONFIG_KMSAN + /* + * KMSAN metadata for this page: + * - shadow page: every bit indicates whether the corresponding + * bit of the original page is initialized (0) or not (1); + * - origin page: every 4 bytes contain an id of the stack trace + * where the uninitialized value was created. + */ + struct page *kmsan_shadow; + struct page *kmsan_origin; +#endif + #ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS int _last_cpupid; #endif diff --git a/include/linux/sched.h b/include/linux/sched.h index a8911b1f35aad..9e53624cd73ac 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -1352,6 +1353,10 @@ struct task_struct { #endif #endif =20 +#ifdef CONFIG_KMSAN + struct kmsan_ctx kmsan_ctx; +#endif + #if IS_ENABLED(CONFIG_KUNIT) struct kunit *kunit_test; #endif diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 075cd25363ac3..b81670878acae 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -996,6 +996,7 @@ config DEBUG_STACKOVERFLOW =20 source "lib/Kconfig.kasan" source "lib/Kconfig.kfence" +source "lib/Kconfig.kmsan" =20 endmenu # "Memory Debugging" =20 diff --git a/lib/Kconfig.kmsan b/lib/Kconfig.kmsan new file mode 100644 index 0000000000000..199f79d031f94 --- /dev/null +++ b/lib/Kconfig.kmsan @@ -0,0 +1,23 @@ +config HAVE_ARCH_KMSAN + bool + +config HAVE_KMSAN_COMPILER + def_bool (CC_IS_CLANG && $(cc-option,-fsanitize=3Dkernel-memory -mllvm -m= san-disable-checks=3D1)) + +config KMSAN + bool "KMSAN: detector of uninitialized values use" + depends on HAVE_ARCH_KMSAN && HAVE_KMSAN_COMPILER + depends on SLUB && DEBUG_KERNEL && !KASAN && !KCSAN + depends on CC_IS_CLANG && CLANG_VERSION >=3D 140000 + select STACKDEPOT + select STACKDEPOT_ALWAYS_INIT + help + KernelMemorySanitizer (KMSAN) is a dynamic detector of uses of + uninitialized values in the kernel. It is based on compiler + instrumentation provided by Clang and thus requires Clang to build. + + An important note is that KMSAN is not intended for production use, + because it drastically increases kernel memory footprint and slows + the whole system down. + + See for more details. diff --git a/mm/Makefile b/mm/Makefile index 4cc13f3179a51..4da7eeaecc214 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -89,6 +89,7 @@ obj-$(CONFIG_SLAB) +=3D slab.o obj-$(CONFIG_SLUB) +=3D slub.o obj-$(CONFIG_KASAN) +=3D kasan/ obj-$(CONFIG_KFENCE) +=3D kfence/ +obj-$(CONFIG_KMSAN) +=3D kmsan/ obj-$(CONFIG_FAILSLAB) +=3D failslab.o obj-$(CONFIG_MEMTEST) +=3D memtest.o obj-$(CONFIG_MIGRATION) +=3D migrate.o diff --git a/mm/kmsan/Makefile b/mm/kmsan/Makefile new file mode 100644 index 0000000000000..a80dde1de7048 --- /dev/null +++ b/mm/kmsan/Makefile @@ -0,0 +1,18 @@ +obj-y :=3D core.o instrumentation.o hooks.o report.o shadow.o + +KMSAN_SANITIZE :=3D n +KCOV_INSTRUMENT :=3D n +UBSAN_SANITIZE :=3D n + +# Disable instrumentation of KMSAN runtime with other tools. +CC_FLAGS_KMSAN_RUNTIME :=3D -fno-stack-protector +CC_FLAGS_KMSAN_RUNTIME +=3D $(call cc-option,-fno-conserve-stack) +CC_FLAGS_KMSAN_RUNTIME +=3D -DDISABLE_BRANCH_PROFILING + +CFLAGS_REMOVE.o =3D $(CC_FLAGS_FTRACE) + +CFLAGS_core.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) +CFLAGS_hooks.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) +CFLAGS_instrumentation.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) +CFLAGS_report.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) +CFLAGS_shadow.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) diff --git a/mm/kmsan/core.c b/mm/kmsan/core.c new file mode 100644 index 0000000000000..933d864d9d467 --- /dev/null +++ b/mm/kmsan/core.c @@ -0,0 +1,458 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * KMSAN runtime library. + * + * Copyright (C) 2017-2022 Google LLC + * Author: Alexander Potapenko + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../slab.h" +#include "kmsan.h" + +/* + * Avoid creating too long origin chains, these are unlikely to participat= e in + * real reports. + */ +#define MAX_CHAIN_DEPTH 7 +#define NUM_SKIPPED_TO_WARN 10000 + +bool kmsan_enabled __read_mostly; + +/* + * Per-CPU KMSAN context to be used in interrupts, where current->kmsan is + * unavaliable. + */ +DEFINE_PER_CPU(struct kmsan_ctx, kmsan_percpu_ctx); + +void kmsan_internal_poison_memory(void *address, size_t size, gfp_t flags, + unsigned int poison_flags) +{ + u32 extra_bits =3D + kmsan_extra_bits(/*depth*/ 0, poison_flags & KMSAN_POISON_FREE); + bool checked =3D poison_flags & KMSAN_POISON_CHECK; + depot_stack_handle_t handle; + + handle =3D kmsan_save_stack_with_flags(flags, extra_bits); + kmsan_internal_set_shadow_origin(address, size, -1, handle, checked); +} + +void kmsan_internal_unpoison_memory(void *address, size_t size, bool check= ed) +{ + kmsan_internal_set_shadow_origin(address, size, 0, 0, checked); +} + +depot_stack_handle_t kmsan_save_stack_with_flags(gfp_t flags, + unsigned int extra) +{ + unsigned long entries[KMSAN_STACK_DEPTH]; + unsigned int nr_entries; + + nr_entries =3D stack_trace_save(entries, KMSAN_STACK_DEPTH, 0); + + /* Don't sleep (see might_sleep_if() in __alloc_pages_nodemask()). */ + flags &=3D ~__GFP_DIRECT_RECLAIM; + + return __stack_depot_save(entries, nr_entries, extra, flags, true); +} + +/* Copy the metadata following the memmove() behavior. */ +void kmsan_internal_memmove_metadata(void *dst, void *src, size_t n) +{ + depot_stack_handle_t old_origin =3D 0, new_origin =3D 0; + int src_slots, dst_slots, i, iter, step, skip_bits; + depot_stack_handle_t *origin_src, *origin_dst; + void *shadow_src, *shadow_dst; + u32 *align_shadow_src, shadow; + bool backwards; + + shadow_dst =3D kmsan_get_metadata(dst, KMSAN_META_SHADOW); + if (!shadow_dst) + return; + KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(dst, n)); + + shadow_src =3D kmsan_get_metadata(src, KMSAN_META_SHADOW); + if (!shadow_src) { + /* + * |src| is untracked: zero out destination shadow, ignore the + * origins, we're done. + */ + __memset(shadow_dst, 0, n); + return; + } + KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(src, n)); + + __memmove(shadow_dst, shadow_src, n); + + origin_dst =3D kmsan_get_metadata(dst, KMSAN_META_ORIGIN); + origin_src =3D kmsan_get_metadata(src, KMSAN_META_ORIGIN); + KMSAN_WARN_ON(!origin_dst || !origin_src); + src_slots =3D (ALIGN((u64)src + n, KMSAN_ORIGIN_SIZE) - + ALIGN_DOWN((u64)src, KMSAN_ORIGIN_SIZE)) / + KMSAN_ORIGIN_SIZE; + dst_slots =3D (ALIGN((u64)dst + n, KMSAN_ORIGIN_SIZE) - + ALIGN_DOWN((u64)dst, KMSAN_ORIGIN_SIZE)) / + KMSAN_ORIGIN_SIZE; + KMSAN_WARN_ON((src_slots < 1) || (dst_slots < 1)); + KMSAN_WARN_ON((src_slots - dst_slots > 1) || + (dst_slots - src_slots < -1)); + + backwards =3D dst > src; + i =3D backwards ? min(src_slots, dst_slots) - 1 : 0; + iter =3D backwards ? -1 : 1; + + align_shadow_src =3D + (u32 *)ALIGN_DOWN((u64)shadow_src, KMSAN_ORIGIN_SIZE); + for (step =3D 0; step < min(src_slots, dst_slots); step++, i +=3D iter) { + KMSAN_WARN_ON(i < 0); + shadow =3D align_shadow_src[i]; + if (i =3D=3D 0) { + /* + * If |src| isn't aligned on KMSAN_ORIGIN_SIZE, don't + * look at the first |src % KMSAN_ORIGIN_SIZE| bytes + * of the first shadow slot. + */ + skip_bits =3D ((u64)src % KMSAN_ORIGIN_SIZE) * 8; + shadow =3D (shadow >> skip_bits) << skip_bits; + } + if (i =3D=3D src_slots - 1) { + /* + * If |src + n| isn't aligned on + * KMSAN_ORIGIN_SIZE, don't look at the last + * |(src + n) % KMSAN_ORIGIN_SIZE| bytes of the + * last shadow slot. + */ + skip_bits =3D (((u64)src + n) % KMSAN_ORIGIN_SIZE) * 8; + shadow =3D (shadow << skip_bits) >> skip_bits; + } + /* + * Overwrite the origin only if the corresponding + * shadow is nonempty. + */ + if (origin_src[i] && (origin_src[i] !=3D old_origin) && shadow) { + old_origin =3D origin_src[i]; + new_origin =3D kmsan_internal_chain_origin(old_origin); + /* + * kmsan_internal_chain_origin() may return + * NULL, but we don't want to lose the previous + * origin value. + */ + if (!new_origin) + new_origin =3D old_origin; + } + if (shadow) + origin_dst[i] =3D new_origin; + else + origin_dst[i] =3D 0; + } + /* + * If dst_slots is greater than src_slots (i.e. + * dst_slots =3D=3D src_slots + 1), there is an extra origin slot at the + * beginning or end of the destination buffer, for which we take the + * origin from the previous slot. + * This is only done if the part of the source shadow corresponding to + * slot is non-zero. + * + * E.g. if we copy 8 aligned bytes that are marked as uninitialized + * and have origins o111 and o222, to an unaligned buffer with offset 1, + * these two origins are copied to three origin slots, so one of then + * needs to be duplicated, depending on the copy direction (@backwards) + * + * src shadow: |uuuu|uuuu|....| + * src origin: |o111|o222|....| + * + * backwards =3D 0: + * dst shadow: |.uuu|uuuu|u...| + * dst origin: |....|o111|o222| - fill the empty slot with o111 + * backwards =3D 1: + * dst shadow: |.uuu|uuuu|u...| + * dst origin: |o111|o222|....| - fill the empty slot with o222 + */ + if (src_slots < dst_slots) { + if (backwards) { + shadow =3D align_shadow_src[src_slots - 1]; + skip_bits =3D (((u64)dst + n) % KMSAN_ORIGIN_SIZE) * 8; + shadow =3D (shadow << skip_bits) >> skip_bits; + if (shadow) + /* src_slots > 0, therefore dst_slots is at least 2 */ + origin_dst[dst_slots - 1] =3D origin_dst[dst_slots - 2]; + } else { + shadow =3D align_shadow_src[0]; + skip_bits =3D ((u64)dst % KMSAN_ORIGIN_SIZE) * 8; + shadow =3D (shadow >> skip_bits) << skip_bits; + if (shadow) + origin_dst[0] =3D origin_dst[1]; + } + } +} + +depot_stack_handle_t kmsan_internal_chain_origin(depot_stack_handle_t id) +{ + unsigned long entries[3]; + u32 extra_bits; + int depth; + bool uaf; + + if (!id) + return id; + /* + * Make sure we have enough spare bits in |id| to hold the UAF bit and + * the chain depth. + */ + BUILD_BUG_ON((1 << STACK_DEPOT_EXTRA_BITS) <=3D (MAX_CHAIN_DEPTH << 1)); + + extra_bits =3D stack_depot_get_extra_bits(id); + depth =3D kmsan_depth_from_eb(extra_bits); + uaf =3D kmsan_uaf_from_eb(extra_bits); + + if (depth >=3D MAX_CHAIN_DEPTH) { + static atomic_long_t kmsan_skipped_origins; + long skipped =3D atomic_long_inc_return(&kmsan_skipped_origins); + + if (skipped % NUM_SKIPPED_TO_WARN =3D=3D 0) { + pr_warn("not chained %ld origins\n", skipped); + dump_stack(); + kmsan_print_origin(id); + } + return id; + } + depth++; + extra_bits =3D kmsan_extra_bits(depth, uaf); + + entries[0] =3D KMSAN_CHAIN_MAGIC_ORIGIN; + entries[1] =3D kmsan_save_stack_with_flags(GFP_ATOMIC, 0); + entries[2] =3D id; + /* + * @entries is a local var in non-instrumented code, so KMSAN does not + * know it is initialized. Explicitly unpoison it to avoid false + * positives when __stack_depot_save() passes it to instrumented code. + */ + kmsan_internal_unpoison_memory(entries, sizeof(entries), false); + return __stack_depot_save(entries, ARRAY_SIZE(entries), extra_bits, + GFP_ATOMIC, true); +} + +void kmsan_internal_set_shadow_origin(void *addr, size_t size, int b, + u32 origin, bool checked) +{ + u64 address =3D (u64)addr; + void *shadow_start; + u32 *origin_start; + size_t pad =3D 0; + int i; + + KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(addr, size)); + shadow_start =3D kmsan_get_metadata(addr, KMSAN_META_SHADOW); + if (!shadow_start) { + /* + * kmsan_metadata_is_contiguous() is true, so either all shadow + * and origin pages are NULL, or all are non-NULL. + */ + if (checked) { + pr_err("%s: not memsetting %ld bytes starting at %px, because the shado= w is NULL\n", + __func__, size, addr); + KMSAN_WARN_ON(true); + } + return; + } + __memset(shadow_start, b, size); + + if (!IS_ALIGNED(address, KMSAN_ORIGIN_SIZE)) { + pad =3D address % KMSAN_ORIGIN_SIZE; + address -=3D pad; + size +=3D pad; + } + size =3D ALIGN(size, KMSAN_ORIGIN_SIZE); + origin_start =3D + (u32 *)kmsan_get_metadata((void *)address, KMSAN_META_ORIGIN); + + for (i =3D 0; i < size / KMSAN_ORIGIN_SIZE; i++) + origin_start[i] =3D origin; +} + +struct page *kmsan_vmalloc_to_page_or_null(void *vaddr) +{ + struct page *page; + + if (!kmsan_internal_is_vmalloc_addr(vaddr) && + !kmsan_internal_is_module_addr(vaddr)) + return NULL; + page =3D vmalloc_to_page(vaddr); + if (pfn_valid(page_to_pfn(page))) + return page; + else + return NULL; +} + +void kmsan_internal_check_memory(void *addr, size_t size, const void *user= _addr, + int reason) +{ + depot_stack_handle_t cur_origin =3D 0, new_origin =3D 0; + unsigned long addr64 =3D (unsigned long)addr; + depot_stack_handle_t *origin =3D NULL; + unsigned char *shadow =3D NULL; + int cur_off_start =3D -1; + int i, chunk_size; + size_t pos =3D 0; + + if (!size) + return; + KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(addr, size)); + while (pos < size) { + chunk_size =3D min(size - pos, + PAGE_SIZE - ((addr64 + pos) % PAGE_SIZE)); + shadow =3D kmsan_get_metadata((void *)(addr64 + pos), + KMSAN_META_SHADOW); + if (!shadow) { + /* + * This page is untracked. If there were uninitialized + * bytes before, report them. + */ + if (cur_origin) { + kmsan_enter_runtime(); + kmsan_report(cur_origin, addr, size, + cur_off_start, pos - 1, user_addr, + reason); + kmsan_leave_runtime(); + } + cur_origin =3D 0; + cur_off_start =3D -1; + pos +=3D chunk_size; + continue; + } + for (i =3D 0; i < chunk_size; i++) { + if (!shadow[i]) { + /* + * This byte is unpoisoned. If there were + * poisoned bytes before, report them. + */ + if (cur_origin) { + kmsan_enter_runtime(); + kmsan_report(cur_origin, addr, size, + cur_off_start, pos + i - 1, + user_addr, reason); + kmsan_leave_runtime(); + } + cur_origin =3D 0; + cur_off_start =3D -1; + continue; + } + origin =3D kmsan_get_metadata((void *)(addr64 + pos + i), + KMSAN_META_ORIGIN); + KMSAN_WARN_ON(!origin); + new_origin =3D *origin; + /* + * Encountered new origin - report the previous + * uninitialized range. + */ + if (cur_origin !=3D new_origin) { + if (cur_origin) { + kmsan_enter_runtime(); + kmsan_report(cur_origin, addr, size, + cur_off_start, pos + i - 1, + user_addr, reason); + kmsan_leave_runtime(); + } + cur_origin =3D new_origin; + cur_off_start =3D pos + i; + } + } + pos +=3D chunk_size; + } + KMSAN_WARN_ON(pos !=3D size); + if (cur_origin) { + kmsan_enter_runtime(); + kmsan_report(cur_origin, addr, size, cur_off_start, pos - 1, + user_addr, reason); + kmsan_leave_runtime(); + } +} + +bool kmsan_metadata_is_contiguous(void *addr, size_t size) +{ + char *cur_shadow =3D NULL, *next_shadow =3D NULL, *cur_origin =3D NULL, + *next_origin =3D NULL; + u64 cur_addr =3D (u64)addr, next_addr =3D cur_addr + PAGE_SIZE; + depot_stack_handle_t *origin_p; + bool all_untracked =3D false; + + if (!size) + return true; + + /* The whole range belongs to the same page. */ + if (ALIGN_DOWN(cur_addr + size - 1, PAGE_SIZE) =3D=3D + ALIGN_DOWN(cur_addr, PAGE_SIZE)) + return true; + + cur_shadow =3D kmsan_get_metadata((void *)cur_addr, /*is_origin*/ false); + if (!cur_shadow) + all_untracked =3D true; + cur_origin =3D kmsan_get_metadata((void *)cur_addr, /*is_origin*/ true); + if (all_untracked && cur_origin) + goto report; + + for (; next_addr < (u64)addr + size; + cur_addr =3D next_addr, cur_shadow =3D next_shadow, + cur_origin =3D next_origin, next_addr +=3D PAGE_SIZE) { + next_shadow =3D kmsan_get_metadata((void *)next_addr, false); + next_origin =3D kmsan_get_metadata((void *)next_addr, true); + if (all_untracked) { + if (next_shadow || next_origin) + goto report; + if (!next_shadow && !next_origin) + continue; + } + if (((u64)cur_shadow =3D=3D ((u64)next_shadow - PAGE_SIZE)) && + ((u64)cur_origin =3D=3D ((u64)next_origin - PAGE_SIZE))) + continue; + goto report; + } + return true; + +report: + pr_err("%s: attempting to access two shadow page ranges.\n", __func__); + pr_err("Access of size %ld at %px.\n", size, addr); + pr_err("Addresses belonging to different ranges: %px and %px\n", + (void *)cur_addr, (void *)next_addr); + pr_err("page[0].shadow: %px, page[1].shadow: %px\n", cur_shadow, + next_shadow); + pr_err("page[0].origin: %px, page[1].origin: %px\n", cur_origin, + next_origin); + origin_p =3D kmsan_get_metadata(addr, KMSAN_META_ORIGIN); + if (origin_p) { + pr_err("Origin: %08x\n", *origin_p); + kmsan_print_origin(*origin_p); + } else { + pr_err("Origin: unavailable\n"); + } + return false; +} + +bool kmsan_internal_is_module_addr(void *vaddr) +{ + return ((u64)vaddr >=3D MODULES_VADDR) && ((u64)vaddr < MODULES_END); +} + +bool kmsan_internal_is_vmalloc_addr(void *addr) +{ + return ((u64)addr >=3D VMALLOC_START) && ((u64)addr < VMALLOC_END); +} diff --git a/mm/kmsan/hooks.c b/mm/kmsan/hooks.c new file mode 100644 index 0000000000000..4ac62fa67a02a --- /dev/null +++ b/mm/kmsan/hooks.c @@ -0,0 +1,66 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * KMSAN hooks for kernel subsystems. + * + * These functions handle creation of KMSAN metadata for memory allocation= s. + * + * Copyright (C) 2018-2022 Google LLC + * Author: Alexander Potapenko + * + */ + +#include +#include +#include +#include +#include +#include + +#include "../internal.h" +#include "../slab.h" +#include "kmsan.h" + +/* + * Instrumented functions shouldn't be called under + * kmsan_enter_runtime()/kmsan_leave_runtime(), because this will lead to + * skipping effects of functions like memset() inside instrumented code. + */ + +/* Functions from kmsan-checks.h follow. */ +void kmsan_poison_memory(const void *address, size_t size, gfp_t flags) +{ + if (!kmsan_enabled || kmsan_in_runtime()) + return; + kmsan_enter_runtime(); + /* The users may want to poison/unpoison random memory. */ + kmsan_internal_poison_memory((void *)address, size, flags, + KMSAN_POISON_NOCHECK); + kmsan_leave_runtime(); +} +EXPORT_SYMBOL(kmsan_poison_memory); + +void kmsan_unpoison_memory(const void *address, size_t size) +{ + unsigned long ua_flags; + + if (!kmsan_enabled || kmsan_in_runtime()) + return; + + ua_flags =3D user_access_save(); + kmsan_enter_runtime(); + /* The users may want to poison/unpoison random memory. */ + kmsan_internal_unpoison_memory((void *)address, size, + KMSAN_POISON_NOCHECK); + kmsan_leave_runtime(); + user_access_restore(ua_flags); +} +EXPORT_SYMBOL(kmsan_unpoison_memory); + +void kmsan_check_memory(const void *addr, size_t size) +{ + if (!kmsan_enabled) + return; + return kmsan_internal_check_memory((void *)addr, size, /*user_addr*/ 0, + REASON_ANY); +} +EXPORT_SYMBOL(kmsan_check_memory); diff --git a/mm/kmsan/instrumentation.c b/mm/kmsan/instrumentation.c new file mode 100644 index 0000000000000..fe062d123a76f --- /dev/null +++ b/mm/kmsan/instrumentation.c @@ -0,0 +1,267 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * KMSAN compiler API. + * + * This file implements __msan_XXX hooks that Clang inserts into the code + * compiled with -fsanitize=3Dkernel-memory. + * See Documentation/dev-tools/kmsan.rst for more information on how KMSAN + * instrumentation works. + * + * Copyright (C) 2017-2022 Google LLC + * Author: Alexander Potapenko + * + */ + +#include "kmsan.h" +#include +#include +#include + +static inline bool is_bad_asm_addr(void *addr, uintptr_t size, bool is_sto= re) +{ + if ((u64)addr < TASK_SIZE) + return true; + if (!kmsan_get_metadata(addr, KMSAN_META_SHADOW)) + return true; + return false; +} + +static inline struct shadow_origin_ptr +get_shadow_origin_ptr(void *addr, u64 size, bool store) +{ + unsigned long ua_flags =3D user_access_save(); + struct shadow_origin_ptr ret; + + ret =3D kmsan_get_shadow_origin_ptr(addr, size, store); + user_access_restore(ua_flags); + return ret; +} + +/* Get shadow and origin pointers for a memory load with non-standard size= . */ +struct shadow_origin_ptr __msan_metadata_ptr_for_load_n(void *addr, + uintptr_t size) +{ + return get_shadow_origin_ptr(addr, size, /*store*/ false); +} +EXPORT_SYMBOL(__msan_metadata_ptr_for_load_n); + +/* Get shadow and origin pointers for a memory store with non-standard siz= e. */ +struct shadow_origin_ptr __msan_metadata_ptr_for_store_n(void *addr, + uintptr_t size) +{ + return get_shadow_origin_ptr(addr, size, /*store*/ true); +} +EXPORT_SYMBOL(__msan_metadata_ptr_for_store_n); + +/* + * Declare functions that obtain shadow/origin pointers for loads and stor= es + * with fixed size. + */ +#define DECLARE_METADATA_PTR_GETTER(size) = \ + struct shadow_origin_ptr __msan_metadata_ptr_for_load_##size( \ + void *addr) \ + { \ + return get_shadow_origin_ptr(addr, size, /*store*/ false); \ + } \ + EXPORT_SYMBOL(__msan_metadata_ptr_for_load_##size); \ + struct shadow_origin_ptr __msan_metadata_ptr_for_store_##size( \ + void *addr) \ + { \ + return get_shadow_origin_ptr(addr, size, /*store*/ true); \ + } \ + EXPORT_SYMBOL(__msan_metadata_ptr_for_store_##size) + +DECLARE_METADATA_PTR_GETTER(1); +DECLARE_METADATA_PTR_GETTER(2); +DECLARE_METADATA_PTR_GETTER(4); +DECLARE_METADATA_PTR_GETTER(8); + +/* + * Handle a memory store performed by inline assembly. KMSAN conservatively + * attempts to unpoison the outputs of asm() directives to prevent false + * positives caused by missed stores. + */ +void __msan_instrument_asm_store(void *addr, uintptr_t size) +{ + unsigned long ua_flags; + + if (!kmsan_enabled || kmsan_in_runtime()) + return; + + ua_flags =3D user_access_save(); + /* + * Most of the accesses are below 32 bytes. The two exceptions so far + * are clwb() (64 bytes) and FPU state (512 bytes). + * It's unlikely that the assembly will touch more than 512 bytes. + */ + if (size > 512) { + WARN_ONCE(1, "assembly store size too big: %ld\n", size); + size =3D 8; + } + if (is_bad_asm_addr(addr, size, /*is_store*/ true)) { + user_access_restore(ua_flags); + return; + } + kmsan_enter_runtime(); + /* Unpoisoning the memory on best effort. */ + kmsan_internal_unpoison_memory(addr, size, /*checked*/ false); + kmsan_leave_runtime(); + user_access_restore(ua_flags); +} +EXPORT_SYMBOL(__msan_instrument_asm_store); + +/* Handle llvm.memmove intrinsic. */ +void *__msan_memmove(void *dst, const void *src, uintptr_t n) +{ + void *result; + + result =3D __memmove(dst, src, n); + if (!n) + /* Some people call memmove() with zero length. */ + return result; + if (!kmsan_enabled || kmsan_in_runtime()) + return result; + + kmsan_internal_memmove_metadata(dst, (void *)src, n); + + return result; +} +EXPORT_SYMBOL(__msan_memmove); + +/* Handle llvm.memcpy intrinsic. */ +void *__msan_memcpy(void *dst, const void *src, uintptr_t n) +{ + void *result; + + result =3D __memcpy(dst, src, n); + if (!n) + /* Some people call memcpy() with zero length. */ + return result; + + if (!kmsan_enabled || kmsan_in_runtime()) + return result; + + /* Using memmove instead of memcpy doesn't affect correctness. */ + kmsan_internal_memmove_metadata(dst, (void *)src, n); + + return result; +} +EXPORT_SYMBOL(__msan_memcpy); + +/* Handle llvm.memset intrinsic. */ +void *__msan_memset(void *dst, int c, uintptr_t n) +{ + void *result; + + result =3D __memset(dst, c, n); + if (!kmsan_enabled || kmsan_in_runtime()) + return result; + + kmsan_enter_runtime(); + /* + * Clang doesn't pass parameter metadata here, so it is impossible to + * use shadow of @c to set up the shadow for @dst. + */ + kmsan_internal_unpoison_memory(dst, n, /*checked*/ false); + kmsan_leave_runtime(); + + return result; +} +EXPORT_SYMBOL(__msan_memset); + +/* + * Create a new origin from an old one. This is done when storing an + * uninitialized value to memory. When reporting an error, KMSAN unrolls a= nd + * prints the whole chain of stores that preceded the use of this value. + */ +depot_stack_handle_t __msan_chain_origin(depot_stack_handle_t origin) +{ + depot_stack_handle_t ret =3D 0; + unsigned long ua_flags; + + if (!kmsan_enabled || kmsan_in_runtime()) + return ret; + + ua_flags =3D user_access_save(); + + /* Creating new origins may allocate memory. */ + kmsan_enter_runtime(); + ret =3D kmsan_internal_chain_origin(origin); + kmsan_leave_runtime(); + user_access_restore(ua_flags); + return ret; +} +EXPORT_SYMBOL(__msan_chain_origin); + +/* Poison a local variable when entering a function. */ +void __msan_poison_alloca(void *address, uintptr_t size, char *descr) +{ + depot_stack_handle_t handle; + unsigned long entries[4]; + unsigned long ua_flags; + + if (!kmsan_enabled || kmsan_in_runtime()) + return; + + ua_flags =3D user_access_save(); + entries[0] =3D KMSAN_ALLOCA_MAGIC_ORIGIN; + entries[1] =3D (u64)descr; + entries[2] =3D (u64)__builtin_return_address(0); + /* + * With frame pointers enabled, it is possible to quickly fetch the + * second frame of the caller stack without calling the unwinder. + * Without them, simply do not bother. + */ + if (IS_ENABLED(CONFIG_UNWINDER_FRAME_POINTER)) + entries[3] =3D (u64)__builtin_return_address(1); + else + entries[3] =3D 0; + + /* stack_depot_save() may allocate memory. */ + kmsan_enter_runtime(); + handle =3D stack_depot_save(entries, ARRAY_SIZE(entries), GFP_ATOMIC); + kmsan_leave_runtime(); + + kmsan_internal_set_shadow_origin(address, size, -1, handle, + /*checked*/ true); + user_access_restore(ua_flags); +} +EXPORT_SYMBOL(__msan_poison_alloca); + +/* Unpoison a local variable. */ +void __msan_unpoison_alloca(void *address, uintptr_t size) +{ + if (!kmsan_enabled || kmsan_in_runtime()) + return; + + kmsan_enter_runtime(); + kmsan_internal_unpoison_memory(address, size, /*checked*/ true); + kmsan_leave_runtime(); +} +EXPORT_SYMBOL(__msan_unpoison_alloca); + +/* + * Report that an uninitialized value with the given origin was used in a = way + * that constituted undefined behavior. + */ +void __msan_warning(u32 origin) +{ + if (!kmsan_enabled || kmsan_in_runtime()) + return; + kmsan_enter_runtime(); + kmsan_report(origin, /*address*/ 0, /*size*/ 0, + /*off_first*/ 0, /*off_last*/ 0, /*user_addr*/ 0, + REASON_ANY); + kmsan_leave_runtime(); +} +EXPORT_SYMBOL(__msan_warning); + +/* + * At the beginning of an instrumented function, obtain the pointer to + * `struct kmsan_context_state` holding the metadata for function paramete= rs. + */ +struct kmsan_context_state *__msan_get_context_state(void) +{ + return &kmsan_get_context()->cstate; +} +EXPORT_SYMBOL(__msan_get_context_state); diff --git a/mm/kmsan/kmsan.h b/mm/kmsan/kmsan.h new file mode 100644 index 0000000000000..bfe38789950a6 --- /dev/null +++ b/mm/kmsan/kmsan.h @@ -0,0 +1,183 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Functions used by the KMSAN runtime. + * + * Copyright (C) 2017-2022 Google LLC + * Author: Alexander Potapenko + * + */ + +#ifndef __MM_KMSAN_KMSAN_H +#define __MM_KMSAN_KMSAN_H + +#include +#include +#include +#include +#include +#include +#include +#include + +#define KMSAN_ALLOCA_MAGIC_ORIGIN 0xabcd0100 +#define KMSAN_CHAIN_MAGIC_ORIGIN 0xabcd0200 + +#define KMSAN_POISON_NOCHECK 0x0 +#define KMSAN_POISON_CHECK 0x1 +#define KMSAN_POISON_FREE 0x2 + +#define KMSAN_ORIGIN_SIZE 4 + +#define KMSAN_STACK_DEPTH 64 + +#define KMSAN_META_SHADOW (false) +#define KMSAN_META_ORIGIN (true) + +extern bool kmsan_enabled; +extern int panic_on_kmsan; + +/* + * KMSAN performs a lot of consistency checks that are currently enabled by + * default. BUG_ON is normally discouraged in the kernel, unless used for + * debugging, but KMSAN itself is a debugging tool, so it makes little sen= se to + * recover if something goes wrong. + */ +#define KMSAN_WARN_ON(cond) = \ + ({ \ + const bool __cond =3D WARN_ON(cond); \ + if (unlikely(__cond)) { \ + WRITE_ONCE(kmsan_enabled, false); \ + if (panic_on_kmsan) { \ + /* Can't call panic() here because */ \ + /* of uaccess checks.*/ \ + BUG(); \ + } \ + } \ + __cond; \ + }) + +/* + * A pair of metadata pointers to be returned by the instrumentation funct= ions. + */ +struct shadow_origin_ptr { + void *shadow, *origin; +}; + +struct shadow_origin_ptr kmsan_get_shadow_origin_ptr(void *addr, u64 size, + bool store); +void *kmsan_get_metadata(void *addr, bool is_origin); + +enum kmsan_bug_reason { + REASON_ANY, + REASON_COPY_TO_USER, + REASON_SUBMIT_URB, +}; + +void kmsan_print_origin(depot_stack_handle_t origin); + +/** + * kmsan_report() - Report a use of uninitialized value. + * @origin: Stack ID of the uninitialized value. + * @address: Address at which the memory access happens. + * @size: Memory access size. + * @off_first: Offset (from @address) of the first byte to be reported. + * @off_last: Offset (from @address) of the last byte to be reported. + * @user_addr: When non-NULL, denotes the userspace address to which the k= ernel + * is leaking data. + * @reason: Error type from enum kmsan_bug_reason. + * + * kmsan_report() prints an error message for a consequent group of bytes + * sharing the same origin. If an uninitialized value is used in a compari= son, + * this function is called once without specifying the addresses. When che= cking + * a memory range, KMSAN may call kmsan_report() multiple times with the s= ame + * @address, @size, @user_addr and @reason, but different @off_first and + * @off_last corresponding to different @origin values. + */ +void kmsan_report(depot_stack_handle_t origin, void *address, int size, + int off_first, int off_last, const void *user_addr, + enum kmsan_bug_reason reason); + +DECLARE_PER_CPU(struct kmsan_ctx, kmsan_percpu_ctx); + +static __always_inline struct kmsan_ctx *kmsan_get_context(void) +{ + return in_task() ? ¤t->kmsan_ctx : raw_cpu_ptr(&kmsan_percpu_ctx); +} + +/* + * When a compiler hook is invoked, it may make a call to instrumented code + * and eventually call itself recursively. To avoid that, we protect the + * runtime entry points with kmsan_enter_runtime()/kmsan_leave_runtime() a= nd + * exit the hook if kmsan_in_runtime() is true. + */ + +static __always_inline bool kmsan_in_runtime(void) +{ + if ((hardirq_count() >> HARDIRQ_SHIFT) > 1) + return true; + return kmsan_get_context()->kmsan_in_runtime; +} + +static __always_inline void kmsan_enter_runtime(void) +{ + struct kmsan_ctx *ctx; + + ctx =3D kmsan_get_context(); + KMSAN_WARN_ON(ctx->kmsan_in_runtime++); +} + +static __always_inline void kmsan_leave_runtime(void) +{ + struct kmsan_ctx *ctx =3D kmsan_get_context(); + + KMSAN_WARN_ON(--ctx->kmsan_in_runtime); +} + +depot_stack_handle_t kmsan_save_stack(void); +depot_stack_handle_t kmsan_save_stack_with_flags(gfp_t flags, + unsigned int extra_bits); + +/* + * Pack and unpack the origin chain depth and UAF flag to/from the extra b= its + * provided by the stack depot. + * The UAF flag is stored in the lowest bit, followed by the depth in the = upper + * bits. + * set_dsh_extra_bits() is responsible for clamping the value. + */ +static __always_inline unsigned int kmsan_extra_bits(unsigned int depth, + bool uaf) +{ + return (depth << 1) | uaf; +} + +static __always_inline bool kmsan_uaf_from_eb(unsigned int extra_bits) +{ + return extra_bits & 1; +} + +static __always_inline unsigned int kmsan_depth_from_eb(unsigned int extra= _bits) +{ + return extra_bits >> 1; +} + +/* + * kmsan_internal_ functions are supposed to be very simple and not requir= e the + * kmsan_in_runtime() checks. + */ +void kmsan_internal_memmove_metadata(void *dst, void *src, size_t n); +void kmsan_internal_poison_memory(void *address, size_t size, gfp_t flags, + unsigned int poison_flags); +void kmsan_internal_unpoison_memory(void *address, size_t size, bool check= ed); +void kmsan_internal_set_shadow_origin(void *address, size_t size, int b, + u32 origin, bool checked); +depot_stack_handle_t kmsan_internal_chain_origin(depot_stack_handle_t id); + +bool kmsan_metadata_is_contiguous(void *addr, size_t size); +void kmsan_internal_check_memory(void *addr, size_t size, const void *user= _addr, + int reason); +bool kmsan_internal_is_module_addr(void *vaddr); +bool kmsan_internal_is_vmalloc_addr(void *addr); + +struct page *kmsan_vmalloc_to_page_or_null(void *vaddr); + +#endif /* __MM_KMSAN_KMSAN_H */ diff --git a/mm/kmsan/report.c b/mm/kmsan/report.c new file mode 100644 index 0000000000000..f36fca452e313 --- /dev/null +++ b/mm/kmsan/report.c @@ -0,0 +1,211 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * KMSAN error reporting routines. + * + * Copyright (C) 2019-2022 Google LLC + * Author: Alexander Potapenko + * + */ + +#include +#include +#include +#include +#include + +#include "kmsan.h" + +static DEFINE_SPINLOCK(kmsan_report_lock); +#define DESCR_SIZE 128 +/* Protected by kmsan_report_lock */ +static char report_local_descr[DESCR_SIZE]; +int panic_on_kmsan __read_mostly; + +#ifdef MODULE_PARAM_PREFIX +#undef MODULE_PARAM_PREFIX +#endif +#define MODULE_PARAM_PREFIX "kmsan." +module_param_named(panic, panic_on_kmsan, int, 0); + +/* + * Skip internal KMSAN frames. + */ +static int get_stack_skipnr(const unsigned long stack_entries[], + int num_entries) +{ + int len, skip; + char buf[64]; + + for (skip =3D 0; skip < num_entries; ++skip) { + len =3D scnprintf(buf, sizeof(buf), "%ps", + (void *)stack_entries[skip]); + + /* Never show __msan_* or kmsan_* functions. */ + if ((strnstr(buf, "__msan_", len) =3D=3D buf) || + (strnstr(buf, "kmsan_", len) =3D=3D buf)) + continue; + + /* + * No match for runtime functions -- @skip entries to skip to + * get to first frame of interest. + */ + break; + } + + return skip; +} + +/* + * Currently the descriptions of locals generated by Clang look as follows: + * ----local_name@function_name + * We want to print only the name of the local, as other information in th= at + * description can be confusing. + * The meaningful part of the description is copied to a global buffer to = avoid + * allocating memory. + */ +static char *pretty_descr(char *descr) +{ + int i, pos =3D 0, len =3D strlen(descr); + + for (i =3D 0; i < len; i++) { + if (descr[i] =3D=3D '@') + break; + if (descr[i] =3D=3D '-') + continue; + report_local_descr[pos] =3D descr[i]; + if (pos + 1 =3D=3D DESCR_SIZE) + break; + pos++; + } + report_local_descr[pos] =3D 0; + return report_local_descr; +} + +void kmsan_print_origin(depot_stack_handle_t origin) +{ + unsigned long *entries =3D NULL, *chained_entries =3D NULL; + unsigned int nr_entries, chained_nr_entries, skipnr; + void *pc1 =3D NULL, *pc2 =3D NULL; + depot_stack_handle_t head; + unsigned long magic; + char *descr =3D NULL; + + if (!origin) + return; + + while (true) { + nr_entries =3D stack_depot_fetch(origin, &entries); + magic =3D nr_entries ? entries[0] : 0; + if ((nr_entries =3D=3D 4) && (magic =3D=3D KMSAN_ALLOCA_MAGIC_ORIGIN)) { + descr =3D (char *)entries[1]; + pc1 =3D (void *)entries[2]; + pc2 =3D (void *)entries[3]; + pr_err("Local variable %s created at:\n", + pretty_descr(descr)); + if (pc1) + pr_err(" %pSb\n", pc1); + if (pc2) + pr_err(" %pSb\n", pc2); + break; + } + if ((nr_entries =3D=3D 3) && (magic =3D=3D KMSAN_CHAIN_MAGIC_ORIGIN)) { + head =3D entries[1]; + origin =3D entries[2]; + pr_err("Uninit was stored to memory at:\n"); + chained_nr_entries =3D + stack_depot_fetch(head, &chained_entries); + kmsan_internal_unpoison_memory( + chained_entries, + chained_nr_entries * sizeof(*chained_entries), + /*checked*/ false); + skipnr =3D get_stack_skipnr(chained_entries, + chained_nr_entries); + stack_trace_print(chained_entries + skipnr, + chained_nr_entries - skipnr, 0); + pr_err("\n"); + continue; + } + pr_err("Uninit was created at:\n"); + if (nr_entries) { + skipnr =3D get_stack_skipnr(entries, nr_entries); + stack_trace_print(entries + skipnr, nr_entries - skipnr, + 0); + } else { + pr_err("(stack is not available)\n"); + } + break; + } +} + +void kmsan_report(depot_stack_handle_t origin, void *address, int size, + int off_first, int off_last, const void *user_addr, + enum kmsan_bug_reason reason) +{ + unsigned long stack_entries[KMSAN_STACK_DEPTH]; + int num_stack_entries, skipnr; + char *bug_type =3D NULL; + unsigned long flags, ua_flags; + bool is_uaf; + + if (!kmsan_enabled) + return; + if (!current->kmsan_ctx.allow_reporting) + return; + if (!origin) + return; + + current->kmsan_ctx.allow_reporting =3D false; + ua_flags =3D user_access_save(); + spin_lock_irqsave(&kmsan_report_lock, flags); + pr_err("=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D\n"); + is_uaf =3D kmsan_uaf_from_eb(stack_depot_get_extra_bits(origin)); + switch (reason) { + case REASON_ANY: + bug_type =3D is_uaf ? "use-after-free" : "uninit-value"; + break; + case REASON_COPY_TO_USER: + bug_type =3D is_uaf ? "kernel-infoleak-after-free" : + "kernel-infoleak"; + break; + case REASON_SUBMIT_URB: + bug_type =3D is_uaf ? "kernel-usb-infoleak-after-free" : + "kernel-usb-infoleak"; + break; + } + + num_stack_entries =3D + stack_trace_save(stack_entries, KMSAN_STACK_DEPTH, 1); + skipnr =3D get_stack_skipnr(stack_entries, num_stack_entries); + + pr_err("BUG: KMSAN: %s in %pSb\n", + bug_type, (void *)stack_entries[skipnr]); + stack_trace_print(stack_entries + skipnr, num_stack_entries - skipnr, + 0); + pr_err("\n"); + + kmsan_print_origin(origin); + + if (size) { + pr_err("\n"); + if (off_first =3D=3D off_last) + pr_err("Byte %d of %d is uninitialized\n", off_first, + size); + else + pr_err("Bytes %d-%d of %d are uninitialized\n", + off_first, off_last, size); + } + if (address) + pr_err("Memory access of size %d starts at %px\n", size, + address); + if (user_addr && reason =3D=3D REASON_COPY_TO_USER) + pr_err("Data copied to user address %px\n", user_addr); + pr_err("\n"); + dump_stack_print_info(KERN_ERR); + pr_err("=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D\n"); + add_taint(TAINT_BAD_PAGE, LOCKDEP_NOW_UNRELIABLE); + spin_unlock_irqrestore(&kmsan_report_lock, flags); + if (panic_on_kmsan) + panic("kmsan.panic set ...\n"); + user_access_restore(ua_flags); + current->kmsan_ctx.allow_reporting =3D true; +} diff --git a/mm/kmsan/shadow.c b/mm/kmsan/shadow.c new file mode 100644 index 0000000000000..de58cfbc55b9d --- /dev/null +++ b/mm/kmsan/shadow.c @@ -0,0 +1,186 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * KMSAN shadow implementation. + * + * Copyright (C) 2017-2022 Google LLC + * Author: Alexander Potapenko + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../internal.h" +#include "kmsan.h" + +#define shadow_page_for(page) ((page)->kmsan_shadow) + +#define origin_page_for(page) ((page)->kmsan_origin) + +static void *shadow_ptr_for(struct page *page) +{ + return page_address(shadow_page_for(page)); +} + +static void *origin_ptr_for(struct page *page) +{ + return page_address(origin_page_for(page)); +} + +static bool page_has_metadata(struct page *page) +{ + return shadow_page_for(page) && origin_page_for(page); +} + +static void set_no_shadow_origin_page(struct page *page) +{ + shadow_page_for(page) =3D NULL; + origin_page_for(page) =3D NULL; +} + +/* + * Dummy load and store pages to be used when the real metadata is unavail= able. + * There are separate pages for loads and stores, so that every load retur= ns a + * zero, and every store doesn't affect other loads. + */ +static char dummy_load_page[PAGE_SIZE] __aligned(PAGE_SIZE); +static char dummy_store_page[PAGE_SIZE] __aligned(PAGE_SIZE); + +/* + * Taken from arch/x86/mm/physaddr.h to avoid using an instrumented versio= n. + */ +static int kmsan_phys_addr_valid(unsigned long addr) +{ + if (IS_ENABLED(CONFIG_PHYS_ADDR_T_64BIT)) + return !(addr >> boot_cpu_data.x86_phys_bits); + else + return 1; +} + +/* + * Taken from arch/x86/mm/physaddr.c to avoid using an instrumented versio= n. + */ +static bool kmsan_virt_addr_valid(void *addr) +{ + unsigned long x =3D (unsigned long)addr; + unsigned long y =3D x - __START_KERNEL_map; + + /* use the carry flag to determine if x was < __START_KERNEL_map */ + if (unlikely(x > y)) { + x =3D y + phys_base; + + if (y >=3D KERNEL_IMAGE_SIZE) + return false; + } else { + x =3D y + (__START_KERNEL_map - PAGE_OFFSET); + + /* carry flag will be set if starting x was >=3D PAGE_OFFSET */ + if ((x > y) || !kmsan_phys_addr_valid(x)) + return false; + } + + return pfn_valid(x >> PAGE_SHIFT); +} + +static unsigned long vmalloc_meta(void *addr, bool is_origin) +{ + unsigned long addr64 =3D (unsigned long)addr, off; + + KMSAN_WARN_ON(is_origin && !IS_ALIGNED(addr64, KMSAN_ORIGIN_SIZE)); + if (kmsan_internal_is_vmalloc_addr(addr)) { + off =3D addr64 - VMALLOC_START; + return off + (is_origin ? KMSAN_VMALLOC_ORIGIN_START : + KMSAN_VMALLOC_SHADOW_START); + } + if (kmsan_internal_is_module_addr(addr)) { + off =3D addr64 - MODULES_VADDR; + return off + (is_origin ? KMSAN_MODULES_ORIGIN_START : + KMSAN_MODULES_SHADOW_START); + } + return 0; +} + +static struct page *virt_to_page_or_null(void *vaddr) +{ + if (kmsan_virt_addr_valid(vaddr)) + return virt_to_page(vaddr); + else + return NULL; +} + +struct shadow_origin_ptr kmsan_get_shadow_origin_ptr(void *address, u64 si= ze, + bool store) +{ + struct shadow_origin_ptr ret; + void *shadow; + + /* + * Even if we redirect this memory access to the dummy page, it will + * go out of bounds. + */ + KMSAN_WARN_ON(size > PAGE_SIZE); + + if (!kmsan_enabled || kmsan_in_runtime()) + goto return_dummy; + + KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(address, size)); + shadow =3D kmsan_get_metadata(address, KMSAN_META_SHADOW); + if (!shadow) + goto return_dummy; + + ret.shadow =3D shadow; + ret.origin =3D kmsan_get_metadata(address, KMSAN_META_ORIGIN); + return ret; + +return_dummy: + if (store) { + /* Ignore this store. */ + ret.shadow =3D dummy_store_page; + ret.origin =3D dummy_store_page; + } else { + /* This load will return zero. */ + ret.shadow =3D dummy_load_page; + ret.origin =3D dummy_load_page; + } + return ret; +} + +/* + * Obtain the shadow or origin pointer for the given address, or NULL if t= here's + * none. The caller must check the return value for being non-NULL if need= ed. + * The return value of this function should not depend on whether we're in= the + * runtime or not. + */ +void *kmsan_get_metadata(void *address, bool is_origin) +{ + u64 addr =3D (u64)address, pad, off; + struct page *page; + void *ret; + + if (is_origin && !IS_ALIGNED(addr, KMSAN_ORIGIN_SIZE)) { + pad =3D addr % KMSAN_ORIGIN_SIZE; + addr -=3D pad; + } + address =3D (void *)addr; + if (kmsan_internal_is_vmalloc_addr(address) || + kmsan_internal_is_module_addr(address)) + return (void *)vmalloc_meta(address, is_origin); + + page =3D virt_to_page_or_null(address); + if (!page) + return NULL; + if (!page_has_metadata(page)) + return NULL; + off =3D addr % PAGE_SIZE; + + ret =3D (is_origin ? origin_ptr_for(page) : shadow_ptr_for(page)) + off; + return ret; +} diff --git a/scripts/Makefile.kmsan b/scripts/Makefile.kmsan new file mode 100644 index 0000000000000..9793591f9855c --- /dev/null +++ b/scripts/Makefile.kmsan @@ -0,0 +1 @@ +export CFLAGS_KMSAN :=3D -fsanitize=3Dkernel-memory diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib index 9f69ecdd7977a..49e6e57fdf4c8 100644 --- a/scripts/Makefile.lib +++ b/scripts/Makefile.lib @@ -157,6 +157,15 @@ _c_flags +=3D $(if $(patsubst n%,, \ endif endif =20 +ifeq ($(CONFIG_KMSAN),y) +_c_flags +=3D $(if $(patsubst n%,, \ + $(KMSAN_SANITIZE_$(basetarget).o)$(KMSAN_SANITIZE)y), \ + $(CFLAGS_KMSAN)) +_c_flags +=3D $(if $(patsubst n%,, \ + $(KMSAN_ENABLE_CHECKS_$(basetarget).o)$(KMSAN_ENABLE_CHECKS)y), \ + , -mllvm -msan-disable-checks=3D1) +endif + ifeq ($(CONFIG_UBSAN),y) _c_flags +=3D $(if $(patsubst n%,, \ $(UBSAN_SANITIZE_$(basetarget).o)$(UBSAN_SANITIZE)$(CONFIG_UBSAN_SANITIZ= E_ALL)), \ --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A060C433F5 for ; Tue, 26 Apr 2022 16:46:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353226AbiDZQtO (ORCPT ); Tue, 26 Apr 2022 12:49:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43242 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353207AbiDZQsn (ORCPT ); Tue, 26 Apr 2022 12:48:43 -0400 Received: from mail-ed1-x54a.google.com (mail-ed1-x54a.google.com [IPv6:2a00:1450:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B8A90197D52 for ; Tue, 26 Apr 2022 09:44:55 -0700 (PDT) Received: by mail-ed1-x54a.google.com with SMTP id t25-20020a508d59000000b00425d86c2987so4107431edt.21 for ; Tue, 26 Apr 2022 09:44:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=W9bLVQHqrvNNLkakI5+eXR9eCGhjUZBWMw6dTDyY1eM=; b=BgZrayGSuVAfOhwVEgizv3zHVLAes+gixFFwYt44q0Anb09MOHmi4/yqVFg0co0xDn meP8eUM1Y1cp68Fqnny0uLKF0Qtln5gmkU4uSohGMZYlVmXcU+el8E0u30FueVIIajUM fV0/mRhuiJoD20Efo5TORsSuNROVI//1ETGTAUMCg4QXyCvHkSNJh0BZuJ+MEW//jk5W 5zxE3TyPvfyC/+M++78ZImLJvn13Sy60BHii4QN5AI+ODEg/JVu9RvQCDCnkT/RTmnsc 5z+2AC9k9WCqo7Z5jf84941KD9T2VCauRnuI4Jr4yEj//ttB99vhdrJTx2BaTgdSpkdR nqtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=W9bLVQHqrvNNLkakI5+eXR9eCGhjUZBWMw6dTDyY1eM=; b=qQWyqFOcfC84WktSDjWgnNIPnORMklsWAiyibRA3VE9lLhz6D3qz0N6QbS5W3/KhyO 2O7wwoSmBjT7gWsEbBulCEOtGORGepVvWLctKHKzZUohcCkogmvHK09g4Bba1kcvOuNF fApORSn4pXxvgd4ynDJahIPiB88BNQh+rlFDgnH8mtuC26qp45UmbZrQbQeGdfuIFTUv /UUD5+3SeDECj4enm4qgC+977eCAM6EVTJna4Wuji8Ri0xAnTs5JQ43WIqUASR6Pop8n U8ZAHdb0bHr8Y0GZ1fQ98ukyZfERyDqPAv41zq6Ns4020TCLDChMG9HKLVgYp6Oj5JX4 xBFQ== X-Gm-Message-State: AOAM532hZOF+678J9Z7eEdzgh6fGV/q6edz4C1+aGQgdlLmd9zyJaUgG mh1NbiKtQbPMX4LrYUS7ytUZl0Jpky4= X-Google-Smtp-Source: ABdhPJwCG6zbeNkNwZsSEqpPcp6mObXzGX3q9XLvEMJvGGRoqht5Amkh6vhwVXvGFlIqKIoGDyBdhsZSVt0= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:84a:b0:423:fe99:8c53 with SMTP id b10-20020a056402084a00b00423fe998c53mr25379562edz.195.1650991494081; Tue, 26 Apr 2022 09:44:54 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:42 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-14-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 13/46] kmsan: implement kmsan_init(), initialize READ_ONCE_NOCHECK() From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" kmsan_init() is a macro that takes a possibly uninitialized value and returns an initialized value of the same type. It can be used e.g. in cases when a value comes from non-instrumented code to avoid false positive reports. In particular, we use kmsan_init() in READ_ONCE_NOCHECK() so that it returns initialized values. This helps defeat false positives e.g. from leftover stack contents accessed by stack unwinders. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/Icd1260073666f944922f031bfb6= 762379ba1fa38 --- include/asm-generic/rwonce.h | 5 +++-- include/linux/kmsan-checks.h | 40 ++++++++++++++++++++++++++++++++++++ mm/kmsan/Makefile | 5 ++++- mm/kmsan/annotations.c | 28 +++++++++++++++++++++++++ 4 files changed, 75 insertions(+), 3 deletions(-) create mode 100644 mm/kmsan/annotations.c diff --git a/include/asm-generic/rwonce.h b/include/asm-generic/rwonce.h index 8d0a6280e9824..7cf993af8e1ea 100644 --- a/include/asm-generic/rwonce.h +++ b/include/asm-generic/rwonce.h @@ -25,6 +25,7 @@ #include #include #include +#include =20 /* * Yes, this permits 64-bit accesses on 32-bit architectures. These will @@ -69,14 +70,14 @@ unsigned long __read_once_word_nocheck(const void *addr) =20 /* * Use READ_ONCE_NOCHECK() instead of READ_ONCE() if you need to load a - * word from memory atomically but without telling KASAN/KCSAN. This is + * word from memory atomically but without telling KASAN/KCSAN/KMSAN. This= is * usually used by unwinding code when walking the stack of a running proc= ess. */ #define READ_ONCE_NOCHECK(x) \ ({ \ compiletime_assert(sizeof(x) =3D=3D sizeof(unsigned long), \ "Unsupported access size for READ_ONCE_NOCHECK()."); \ - (typeof(x))__read_once_word_nocheck(&(x)); \ + kmsan_init((typeof(x))__read_once_word_nocheck(&(x))); \ }) =20 static __no_kasan_or_inline diff --git a/include/linux/kmsan-checks.h b/include/linux/kmsan-checks.h index a6522a0c28df9..ecd8336190fc0 100644 --- a/include/linux/kmsan-checks.h +++ b/include/linux/kmsan-checks.h @@ -14,6 +14,44 @@ =20 #ifdef CONFIG_KMSAN =20 +/* + * Helper functions that mark the return value initialized. + * See mm/kmsan/annotations.c. + */ +u8 kmsan_init_1(u8 value); +u16 kmsan_init_2(u16 value); +u32 kmsan_init_4(u32 value); +u64 kmsan_init_8(u64 value); + +static inline void *kmsan_init_ptr(void *ptr) +{ + return (void *)kmsan_init_8((u64)ptr); +} + +static inline char kmsan_init_char(char value) +{ + return (u8)kmsan_init_1((u8)value); +} + +#define __decl_kmsan_init_type(type, fn) unsigned type : fn, signed type := fn + +/** + * kmsan_init - Make the value initialized. + * @val: 1-, 2-, 4- or 8-byte integer that may be treated as uninitialized= by + * KMSAN. + * + * Return: value of @val that KMSAN treats as initialized. + */ +#define kmsan_init(val) = \ + ( \ + (typeof(val))(_Generic((val), \ + __decl_kmsan_init_type(char, kmsan_init_1), \ + __decl_kmsan_init_type(short, kmsan_init_2), \ + __decl_kmsan_init_type(int, kmsan_init_4), \ + __decl_kmsan_init_type(long, kmsan_init_8), \ + char : kmsan_init_char, \ + void * : kmsan_init_ptr)(val))) + /** * kmsan_poison_memory() - Mark the memory range as uninitialized. * @address: address to start with. @@ -48,6 +86,8 @@ void kmsan_check_memory(const void *address, size_t size); =20 #else =20 +#define kmsan_init(value) (value) + static inline void kmsan_poison_memory(const void *address, size_t size, gfp_t flags) { diff --git a/mm/kmsan/Makefile b/mm/kmsan/Makefile index a80dde1de7048..73b705cbf75b9 100644 --- a/mm/kmsan/Makefile +++ b/mm/kmsan/Makefile @@ -1,9 +1,11 @@ -obj-y :=3D core.o instrumentation.o hooks.o report.o shadow.o +obj-y :=3D core.o instrumentation.o hooks.o report.o shadow.o annotations.o =20 KMSAN_SANITIZE :=3D n KCOV_INSTRUMENT :=3D n UBSAN_SANITIZE :=3D n =20 +KMSAN_SANITIZE_kmsan_annotations.o :=3D y + # Disable instrumentation of KMSAN runtime with other tools. CC_FLAGS_KMSAN_RUNTIME :=3D -fno-stack-protector CC_FLAGS_KMSAN_RUNTIME +=3D $(call cc-option,-fno-conserve-stack) @@ -11,6 +13,7 @@ CC_FLAGS_KMSAN_RUNTIME +=3D -DDISABLE_BRANCH_PROFILING =20 CFLAGS_REMOVE.o =3D $(CC_FLAGS_FTRACE) =20 +CFLAGS_annotations.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) CFLAGS_core.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) CFLAGS_hooks.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) CFLAGS_instrumentation.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) diff --git a/mm/kmsan/annotations.c b/mm/kmsan/annotations.c new file mode 100644 index 0000000000000..8ccde90bcd12b --- /dev/null +++ b/mm/kmsan/annotations.c @@ -0,0 +1,28 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * KMSAN annotations. + * + * The kmsan_init_SIZE functions reside in a separate translation unit to + * prevent inlining them. Clang may inline functions marked with + * __no_sanitize_memory attribute into functions without it, which effecti= vely + * results in ignoring the attribute. + * + * Copyright (C) 2017-2022 Google LLC + * Author: Alexander Potapenko + * + */ + +#include +#include + +#define DECLARE_KMSAN_INIT(size, t) = \ + __no_sanitize_memory t kmsan_init_##size(t value) \ + { \ + return value; \ + } \ + EXPORT_SYMBOL(kmsan_init_##size) + +DECLARE_KMSAN_INIT(1, u8); +DECLARE_KMSAN_INIT(2, u16); +DECLARE_KMSAN_INIT(4, u32); +DECLARE_KMSAN_INIT(8, u64); --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B77C8C433FE for ; Tue, 26 Apr 2022 16:46:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353246AbiDZQta (ORCPT ); Tue, 26 Apr 2022 12:49:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44082 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353210AbiDZQso (ORCPT ); Tue, 26 Apr 2022 12:48:44 -0400 Received: from mail-lf1-x14a.google.com (mail-lf1-x14a.google.com [IPv6:2a00:1450:4864:20::14a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E416F26D5 for ; Tue, 26 Apr 2022 09:44:58 -0700 (PDT) Received: by mail-lf1-x14a.google.com with SMTP id br31-20020a056512401f00b00471c57013ceso7095647lfb.3 for ; Tue, 26 Apr 2022 09:44:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=QRr/YK4fONn3HTG8Fp9y+tnnWkzL1O1MoaFJezJL3sE=; b=GRtnB1LL/0WxvgZxxJD6lyURNXu6W8fs61POQbIbc0oumZ7k69NQoqD3tuCVl5yHj/ Gx/zngh3XGQ7t3vv0vTrJzo1vEkQTQ/ZRo418CMleof5FGdj9l6StzKqWUtXQAScQwo5 7P5qB/cvRckWW+BL+XiMdliFEO5yd4aWG7n9leQq17xa//Gzq2wBQTSCFsGG0JWGpu0+ pqy91hZcYLp1jkzx4uiMbKUqjtEQl6oUHgMvxwlnBvwzVMZsM6AIx5QoYYlhWVc7tQxH mHkgIrHRenuJcacBwVLYMBUhS7EqeYlTJUvvLtw99sQakaUg4whscJWcJBjl5RDYQ691 p17g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=QRr/YK4fONn3HTG8Fp9y+tnnWkzL1O1MoaFJezJL3sE=; b=JF+C4Fz/DbfQmJ/kB5RsKN5dOT75OliVJJUaVBfChUyx5LMPHdu953ka92R4nC/S0n V+o1s1d3dIAXpo7AkjMk6r5cXnNxW3w6SzcLQG+xDB1JAxk9CBYj0zdMLXcrVpOG+D5i sb1xRHFfoMxtv6uZqO6ZyyCmxALgSWdxgnBIxopSKR6g80XPuSvBhAgQs8aeyCoUZNYo eaxIBrAGPVB2DIn0VBnfyelrtTK8JFQVxNDvcHwg40zWECSbJMieVcg4BZUpixIVVFpp y6pSuecWawTzVlWwCKzWulpPFChf1e2jYMSuqIv6CQfaxEvX/MyRej6szHxVJT9Slu4Y npEw== X-Gm-Message-State: AOAM530dVtkvyHDPCnQJjEemF7uxPlIXkYfVfoghNlqWRNdpFr398r3o zinAWRQEX8e4Rnn/EYR9/ic+Q+FdDa4= X-Google-Smtp-Source: ABdhPJxO/IzNek0yZm9+I1xLkkVDvFA7Rtx6U+bdK1jYJgO+mP11K/pbQde5ZHHXccF1PEslgOn1O7FKoX8= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6512:104a:b0:471:f0c2:99ee with SMTP id c10-20020a056512104a00b00471f0c299eemr14014612lfb.142.1650991496992; Tue, 26 Apr 2022 09:44:56 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:43 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-15-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 14/46] kmsan: disable instrumentation of unsupported common kernel code From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" EFI stub cannot be linked with KMSAN runtime, so we disable instrumentation for it. Instrumenting kcov, stackdepot or lockdep leads to infinite recursion caused by instrumentation hooks calling instrumented code again. This patch was previously part of "kmsan: disable KMSAN instrumentation for certain kernel parts", but was split away per Mark Rutland's request. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/I41ae706bd3474f074f6a870bfc3= f0f90e9c720f7 --- drivers/firmware/efi/libstub/Makefile | 1 + kernel/Makefile | 1 + kernel/locking/Makefile | 3 ++- lib/Makefile | 1 + 4 files changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/firmware/efi/libstub/Makefile b/drivers/firmware/efi/l= ibstub/Makefile index d0537573501e9..81432d0c904b1 100644 --- a/drivers/firmware/efi/libstub/Makefile +++ b/drivers/firmware/efi/libstub/Makefile @@ -46,6 +46,7 @@ GCOV_PROFILE :=3D n # Sanitizer runtimes are unavailable and cannot be linked here. KASAN_SANITIZE :=3D n KCSAN_SANITIZE :=3D n +KMSAN_SANITIZE :=3D n UBSAN_SANITIZE :=3D n OBJECT_FILES_NON_STANDARD :=3D y =20 diff --git a/kernel/Makefile b/kernel/Makefile index 847a82bfe0e3a..2a98e46479817 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -39,6 +39,7 @@ KCOV_INSTRUMENT_kcov.o :=3D n KASAN_SANITIZE_kcov.o :=3D n KCSAN_SANITIZE_kcov.o :=3D n UBSAN_SANITIZE_kcov.o :=3D n +KMSAN_SANITIZE_kcov.o :=3D n CFLAGS_kcov.o :=3D $(call cc-option, -fno-conserve-stack) -fno-stack-prote= ctor =20 # Don't instrument error handlers diff --git a/kernel/locking/Makefile b/kernel/locking/Makefile index d51cabf28f382..ea925731fa40f 100644 --- a/kernel/locking/Makefile +++ b/kernel/locking/Makefile @@ -5,8 +5,9 @@ KCOV_INSTRUMENT :=3D n =20 obj-y +=3D mutex.o semaphore.o rwsem.o percpu-rwsem.o =20 -# Avoid recursion lockdep -> KCSAN -> ... -> lockdep. +# Avoid recursion lockdep -> sanitizer -> ... -> lockdep. KCSAN_SANITIZE_lockdep.o :=3D n +KMSAN_SANITIZE_lockdep.o :=3D n =20 ifdef CONFIG_FUNCTION_TRACER CFLAGS_REMOVE_lockdep.o =3D $(CC_FLAGS_FTRACE) diff --git a/lib/Makefile b/lib/Makefile index 6b9ffc1bd1eed..caeb55f661726 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -269,6 +269,7 @@ obj-$(CONFIG_IRQ_POLL) +=3D irq_poll.o CFLAGS_stackdepot.o +=3D -fno-builtin obj-$(CONFIG_STACKDEPOT) +=3D stackdepot.o KASAN_SANITIZE_stackdepot.o :=3D n +KMSAN_SANITIZE_stackdepot.o :=3D n KCOV_INSTRUMENT_stackdepot.o :=3D n =20 obj-$(CONFIG_REF_TRACKER) +=3D ref_tracker.o --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC816C433F5 for ; Tue, 26 Apr 2022 16:46:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353273AbiDZQte (ORCPT ); Tue, 26 Apr 2022 12:49:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44210 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353191AbiDZQsq (ORCPT ); Tue, 26 Apr 2022 12:48:46 -0400 Received: from mail-ed1-x549.google.com (mail-ed1-x549.google.com [IPv6:2a00:1450:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 81E7822521 for ; Tue, 26 Apr 2022 09:45:01 -0700 (PDT) Received: by mail-ed1-x549.google.com with SMTP id n4-20020a5099c4000000b00418ed58d92fso10617444edb.0 for ; Tue, 26 Apr 2022 09:45:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=eiEr+2P23HBOM4McbqvmLa2JAqZ5nkaX8a1H8WzeBdc=; b=AH6wl3/h3jZOQ/PP2XRz3H+6yTFTXd6FbVvqWs3dSD1VQoD7oZ4P8RfDCTZSOTbJuF Dea0WX2BnUSJqwq6L7bGv/kGkAoJpJXcX90MvZJCD8V3DLZNs48DS11HIb7sInlZfzue DMGpTwc5BhsC0Yj7lwGhp9whIz5ydF7WZDmw1yJ41rcNpl0D6aJSuZ/x8E6uGXhPjIWY MZ1Sk37oKsYl4w4mevm5K1deWm/GnWAIBSURersc8GWz4STFmSMBvkmT8PLUk5DY0bqQ 4G5CpFQfFmOpU7OaJZK8mUaojJl5Tf4k8xlMFEdi9tJgBabQcMfO6BrSj4NGwPy+nPHk 6HFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=eiEr+2P23HBOM4McbqvmLa2JAqZ5nkaX8a1H8WzeBdc=; b=IAgnJ28lrGrvKDj9nQUuTsU04LBdjwKTl7OPsS0JzTKxvSPe3/1rTloQDBToZUKW// zn+XDL0B7ensyFK7597fD5b8QlnY8LcOIldx0Kclyx1LnbwKhFzWj6QkNWjI0750ewTu 8EPZaJJOBzZdYjmzyO7dk7za77Cq5vhsH8OqtsWMGkJQrymqKH/jFKNwJH+UUaVe2dCz hiq6K8cdHG4ChsQtkPS71Irje4rmrMIXm795vXcU6CGkfCK/fRH2IrJqmao1U+8grm9N wvPePv8upZqWpVQ8/iNJxnMWgM4unOIUDw3cVuGdLFXt8xIbAtGbBKrF880YC9v2nhSs rL7Q== X-Gm-Message-State: AOAM531iE98ju9k61+ZO37DEbKjElBlGhFxjE8TRawhW3JaFBm/CLKVT HqAnuvrbbdwmDL+uU8Tle2x+olLA5s8= X-Google-Smtp-Source: ABdhPJxDSlpkDTmdkRbr1nbB7ZI7Yjkrt1D8Ci5Kh7HlHhT8mmXkAZD7Y6I7F3vOvJUv/S/j20xf/QcJdwc= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:11cd:b0:425:ee49:58cb with SMTP id j13-20020a05640211cd00b00425ee4958cbmr10117861edw.157.1650991499654; Tue, 26 Apr 2022 09:44:59 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:44 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-16-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 15/46] MAINTAINERS: add entry for KMSAN From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add entry for KMSAN maintainers/reviewers. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/Ic5836c2bceb6b63f71a60d3327d= 18af3aa3dab77 --- MAINTAINERS | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 5e8c2f6117661..dc73b124971f1 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -10937,6 +10937,18 @@ F: kernel/kmod.c F: lib/test_kmod.c F: tools/testing/selftests/kmod/ =20 +KMSAN +M: Alexander Potapenko +R: Marco Elver +R: Dmitry Vyukov +L: kasan-dev@googlegroups.com +S: Maintained +F: Documentation/dev-tools/kmsan.rst +F: include/linux/kmsan*.h +F: lib/Kconfig.kmsan +F: mm/kmsan/ +F: scripts/Makefile.kmsan + KPROBES M: Naveen N. Rao M: Anil S Keshavamurthy --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8429DC433F5 for ; Tue, 26 Apr 2022 16:46:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353235AbiDZQti (ORCPT ); Tue, 26 Apr 2022 12:49:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353231AbiDZQsr (ORCPT ); Tue, 26 Apr 2022 12:48:47 -0400 Received: from mail-ed1-x549.google.com (mail-ed1-x549.google.com [IPv6:2a00:1450:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 098D31CFE3 for ; Tue, 26 Apr 2022 09:45:04 -0700 (PDT) Received: by mail-ed1-x549.google.com with SMTP id cf16-20020a0564020b9000b00425d543c75dso4623994edb.11 for ; Tue, 26 Apr 2022 09:45:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=xJ2BmwZfS9U11w9738T9IIYSomxf8dgaSR2dg4ENYSM=; b=M78kkNEBLhoz120Hsc5NTi+kQYw3vfJ7Ofe3VzxcySxmocKv/b6vGvNS2ffV8ofLMt Ub8EknwRYPUdh8/3USQbMH+y3P6HdZw7qv2b51j6EV4Ne6raJKPwvp8VRM5bVuYBUO11 PICCITdy7VOMLndrYZrqZxMq7AFtVRkmxAHYZcMztdTWIyZiptfTK5KbGLYXwpTzNJ9b 44ENadmhCTtn1832j4ypKDxdW1wwVPN+21DGY8Q2iniakVP1O83WAGPnnq32oyyj2k+x uLTUBglTyA1FuXmul5INuSVHXnYRELBW7ttl8FFdrkbwJugfs4E4SJpate5lkIid50pw bZHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=xJ2BmwZfS9U11w9738T9IIYSomxf8dgaSR2dg4ENYSM=; b=UuHnojTPUV7FUNM13qzUJNDiNC9U/CSmXEhl63KxT4luwgrFLfDVHxJh1pW6B/OzIb 1vAy4RrxmdOp8v74gYUVUlOkvIZFPymsGCbttjd9frR2EBSA3qRCB/QS2LdsIgc4aNim lnsqwdNDiy4uxoqzXh7k4eWMZtchh8jGQ9FW6Ng0MQYrN2XvJK3vgVtfmUcMfZMaT5dv BnnIuHHeR1t2T7VKvXJCbiymN/GSXipsIrpjWmcheUwGgvm57BSEwGm5pQFDbcy8j/S0 H2VB9eXoToCb5WCXc3JD00HAbnF0u0JLcQMOce3L1REnobwFZxCSrtofpQz8Jnry/wQc chuQ== X-Gm-Message-State: AOAM532GFehkm30eqRsjkWuEENmAQvZ/0c3bXBLjn9aCQtJK0F95WQ/+ LQV6CNG4Nu2kP9b11rVIimy1lx7XiHk= X-Google-Smtp-Source: ABdhPJwanrGXJn5S5Kb/uoAxmNWGHJmy0Y2Eba29AwNvQGec6+HVhIGagVjDS+qwYbcaVGcIOG9PfQHjjlI= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a17:906:4787:b0:6f3:7e2a:ebfd with SMTP id cw7-20020a170906478700b006f37e2aebfdmr14388383ejc.243.1650991502331; Tue, 26 Apr 2022 09:45:02 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:45 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-17-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 16/46] kmsan: mm: maintain KMSAN metadata for page operations From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Insert KMSAN hooks that make the necessary bookkeeping changes: - poison page shadow and origins in alloc_pages()/free_page(); - clear page shadow and origins in clear_page(), copy_user_highpage(); - copy page metadata in copy_highpage(), wp_page_copy(); - handle vmap()/vunmap()/iounmap(); Signed-off-by: Alexander Potapenko --- v2: -- move page metadata hooks implementation here -- remove call to kmsan_memblock_free_pages() v3: -- use PAGE_SHIFT in kmsan_ioremap_page_range() Link: https://linux-review.googlesource.com/id/I6d4f53a0e7eab46fa29f0348f30= 95d9f2e326850 --- arch/x86/include/asm/page_64.h | 13 ++++ arch/x86/mm/ioremap.c | 3 + include/linux/highmem.h | 3 + include/linux/kmsan.h | 123 +++++++++++++++++++++++++++++++++ mm/internal.h | 6 ++ mm/kmsan/hooks.c | 87 +++++++++++++++++++++++ mm/kmsan/shadow.c | 114 ++++++++++++++++++++++++++++++ mm/memory.c | 2 + mm/page_alloc.c | 11 +++ mm/vmalloc.c | 20 +++++- 10 files changed, 380 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index e9c86299b8351..36e270a8ea9a4 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -45,14 +45,27 @@ void clear_page_orig(void *page); void clear_page_rep(void *page); void clear_page_erms(void *page); =20 +/* This is an assembly header, avoid including too much of kmsan.h */ +#ifdef CONFIG_KMSAN +void kmsan_unpoison_memory(const void *addr, size_t size); +#endif +__no_sanitize_memory static inline void clear_page(void *page) { +#ifdef CONFIG_KMSAN + /* alternative_call_2() changes @page. */ + void *page_copy =3D page; +#endif alternative_call_2(clear_page_orig, clear_page_rep, X86_FEATURE_REP_GOOD, clear_page_erms, X86_FEATURE_ERMS, "=3DD" (page), "0" (page) : "cc", "memory", "rax", "rcx"); +#ifdef CONFIG_KMSAN + /* Clear KMSAN shadow for the pages that have it. */ + kmsan_unpoison_memory(page_copy, PAGE_SIZE); +#endif } =20 void copy_page(void *to, void *from); diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c index 17a492c273069..0da8608778221 100644 --- a/arch/x86/mm/ioremap.c +++ b/arch/x86/mm/ioremap.c @@ -17,6 +17,7 @@ #include #include #include +#include =20 #include #include @@ -474,6 +475,8 @@ void iounmap(volatile void __iomem *addr) return; } =20 + kmsan_iounmap_page_range((unsigned long)addr, + (unsigned long)addr + get_vm_area_size(p)); memtype_free(p->phys_addr, p->phys_addr + get_vm_area_size(p)); =20 /* Finally remove it */ diff --git a/include/linux/highmem.h b/include/linux/highmem.h index 39bb9b47fa9cd..3e1898a44d7e3 100644 --- a/include/linux/highmem.h +++ b/include/linux/highmem.h @@ -6,6 +6,7 @@ #include #include #include +#include #include #include #include @@ -277,6 +278,7 @@ static inline void copy_user_highpage(struct page *to, = struct page *from, vfrom =3D kmap_local_page(from); vto =3D kmap_local_page(to); copy_user_page(vto, vfrom, vaddr, to); + kmsan_unpoison_memory(page_address(to), PAGE_SIZE); kunmap_local(vto); kunmap_local(vfrom); } @@ -292,6 +294,7 @@ static inline void copy_highpage(struct page *to, struc= t page *from) vfrom =3D kmap_local_page(from); vto =3D kmap_local_page(to); copy_page(vto, vfrom); + kmsan_copy_page_meta(to, from); kunmap_local(vto); kunmap_local(vfrom); } diff --git a/include/linux/kmsan.h b/include/linux/kmsan.h index 4e35f43eceaa9..da41850b46cbd 100644 --- a/include/linux/kmsan.h +++ b/include/linux/kmsan.h @@ -42,6 +42,129 @@ struct kmsan_ctx { bool allow_reporting; }; =20 +/** + * kmsan_alloc_page() - Notify KMSAN about an alloc_pages() call. + * @page: struct page pointer returned by alloc_pages(). + * @order: order of allocated struct page. + * @flags: GFP flags used by alloc_pages() + * + * KMSAN marks 1<<@order pages starting at @page as uninitialized, unless + * @flags contain __GFP_ZERO. + */ +void kmsan_alloc_page(struct page *page, unsigned int order, gfp_t flags); + +/** + * kmsan_free_page() - Notify KMSAN about a free_pages() call. + * @page: struct page pointer passed to free_pages(). + * @order: order of deallocated struct page. + * + * KMSAN marks freed memory as uninitialized. + */ +void kmsan_free_page(struct page *page, unsigned int order); + +/** + * kmsan_copy_page_meta() - Copy KMSAN metadata between two pages. + * @dst: destination page. + * @src: source page. + * + * KMSAN copies the contents of metadata pages for @src into the metadata = pages + * for @dst. If @dst has no associated metadata pages, nothing happens. + * If @src has no associated metadata pages, @dst metadata pages are unpoi= soned. + */ +void kmsan_copy_page_meta(struct page *dst, struct page *src); + +/** + * kmsan_map_kernel_range_noflush() - Notify KMSAN about a vmap. + * @start: start of vmapped range. + * @end: end of vmapped range. + * @prot: page protection flags used for vmap. + * @pages: array of pages. + * @page_shift: page_shift passed to vmap_range_noflush(). + * + * KMSAN maps shadow and origin pages of @pages into contiguous ranges in + * vmalloc metadata address range. + */ +void kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end, + pgprot_t prot, struct page **pages, + unsigned int page_shift); + +/** + * kmsan_vunmap_kernel_range_noflush() - Notify KMSAN about a vunmap. + * @start: start of vunmapped range. + * @end: end of vunmapped range. + * + * KMSAN unmaps the contiguous metadata ranges created by + * kmsan_map_kernel_range_noflush(). + */ +void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end); + +/** + * kmsan_ioremap_page_range() - Notify KMSAN about a ioremap_page_range() = call. + * @addr: range start. + * @end: range end. + * @phys_addr: physical range start. + * @prot: page protection flags used for ioremap_page_range(). + * @page_shift: page_shift argument passed to vmap_range_noflush(). + * + * KMSAN creates new metadata pages for the physical pages mapped into the + * virtual memory. + */ +void kmsan_ioremap_page_range(unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int page_shift); + +/** + * kmsan_iounmap_page_range() - Notify KMSAN about a iounmap_page_range() = call. + * @start: range start. + * @end: range end. + * + * KMSAN unmaps the metadata pages for the given range and, unlike for + * vunmap_page_range(), also deallocates them. + */ +void kmsan_iounmap_page_range(unsigned long start, unsigned long end); + +#else + +static inline int kmsan_alloc_page(struct page *page, unsigned int order, + gfp_t flags) +{ + return 0; +} + +static inline void kmsan_free_page(struct page *page, unsigned int order) +{ +} + +static inline void kmsan_copy_page_meta(struct page *dst, struct page *src) +{ +} + +static inline void kmsan_vmap_pages_range_noflush(unsigned long start, + unsigned long end, + pgprot_t prot, + struct page **pages, + unsigned int page_shift) +{ +} + +static inline void kmsan_vunmap_range_noflush(unsigned long start, + unsigned long end) +{ +} + +static inline void kmsan_ioremap_page_range(unsigned long start, + unsigned long end, + phys_addr_t phys_addr, + pgprot_t prot, + unsigned int page_shift) +{ +} + +static inline void kmsan_iounmap_page_range(unsigned long start, + unsigned long end) +{ +} + #endif =20 #endif /* _LINUX_KMSAN_H */ diff --git a/mm/internal.h b/mm/internal.h index cf16280ce1321..3cf6fde8f02c4 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -744,8 +744,14 @@ int vmap_pages_range_noflush(unsigned long addr, unsig= ned long end, } #endif =20 +int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, + unsigned int page_shift); + void vunmap_range_noflush(unsigned long start, unsigned long end); =20 +void __vunmap_range_noflush(unsigned long start, unsigned long end); + int numa_migrate_prep(struct page *page, struct vm_area_struct *vma, unsigned long addr, int page_nid, int *flags); =20 diff --git a/mm/kmsan/hooks.c b/mm/kmsan/hooks.c index 4ac62fa67a02a..070756be70e3a 100644 --- a/mm/kmsan/hooks.c +++ b/mm/kmsan/hooks.c @@ -26,6 +26,93 @@ * skipping effects of functions like memset() inside instrumented code. */ =20 +static unsigned long vmalloc_shadow(unsigned long addr) +{ + return (unsigned long)kmsan_get_metadata((void *)addr, + KMSAN_META_SHADOW); +} + +static unsigned long vmalloc_origin(unsigned long addr) +{ + return (unsigned long)kmsan_get_metadata((void *)addr, + KMSAN_META_ORIGIN); +} + +void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end) +{ + __vunmap_range_noflush(vmalloc_shadow(start), vmalloc_shadow(end)); + __vunmap_range_noflush(vmalloc_origin(start), vmalloc_origin(end)); + flush_cache_vmap(vmalloc_shadow(start), vmalloc_shadow(end)); + flush_cache_vmap(vmalloc_origin(start), vmalloc_origin(end)); +} +EXPORT_SYMBOL(kmsan_vunmap_range_noflush); + +/* + * This function creates new shadow/origin pages for the physical pages ma= pped + * into the virtual memory. If those physical pages already had shadow/ori= gin, + * those are ignored. + */ +void kmsan_ioremap_page_range(unsigned long start, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int page_shift) +{ + gfp_t gfp_mask =3D GFP_KERNEL | __GFP_ZERO; + struct page *shadow, *origin; + unsigned long off =3D 0; + int i, nr; + + if (!kmsan_enabled || kmsan_in_runtime()) + return; + + nr =3D (end - start) / PAGE_SIZE; + kmsan_enter_runtime(); + for (i =3D 0; i < nr; i++, off +=3D PAGE_SIZE) { + shadow =3D alloc_pages(gfp_mask, 1); + origin =3D alloc_pages(gfp_mask, 1); + __vmap_pages_range_noflush( + vmalloc_shadow(start + off), + vmalloc_shadow(start + off + PAGE_SIZE), prot, &shadow, + PAGE_SHIFT); + __vmap_pages_range_noflush( + vmalloc_origin(start + off), + vmalloc_origin(start + off + PAGE_SIZE), prot, &origin, + PAGE_SHIFT); + } + flush_cache_vmap(vmalloc_shadow(start), vmalloc_shadow(end)); + flush_cache_vmap(vmalloc_origin(start), vmalloc_origin(end)); + kmsan_leave_runtime(); +} +EXPORT_SYMBOL(kmsan_ioremap_page_range); + +void kmsan_iounmap_page_range(unsigned long start, unsigned long end) +{ + unsigned long v_shadow, v_origin; + struct page *shadow, *origin; + int i, nr; + + if (!kmsan_enabled || kmsan_in_runtime()) + return; + + nr =3D (end - start) / PAGE_SIZE; + kmsan_enter_runtime(); + v_shadow =3D (unsigned long)vmalloc_shadow(start); + v_origin =3D (unsigned long)vmalloc_origin(start); + for (i =3D 0; i < nr; i++, v_shadow +=3D PAGE_SIZE, v_origin +=3D PAGE_SI= ZE) { + shadow =3D kmsan_vmalloc_to_page_or_null((void *)v_shadow); + origin =3D kmsan_vmalloc_to_page_or_null((void *)v_origin); + __vunmap_range_noflush(v_shadow, vmalloc_shadow(end)); + __vunmap_range_noflush(v_origin, vmalloc_origin(end)); + if (shadow) + __free_pages(shadow, 1); + if (origin) + __free_pages(origin, 1); + } + flush_cache_vmap(vmalloc_shadow(start), vmalloc_shadow(end)); + flush_cache_vmap(vmalloc_origin(start), vmalloc_origin(end)); + kmsan_leave_runtime(); +} +EXPORT_SYMBOL(kmsan_iounmap_page_range); + /* Functions from kmsan-checks.h follow. */ void kmsan_poison_memory(const void *address, size_t size, gfp_t flags) { diff --git a/mm/kmsan/shadow.c b/mm/kmsan/shadow.c index de58cfbc55b9d..8fe6a5ed05e67 100644 --- a/mm/kmsan/shadow.c +++ b/mm/kmsan/shadow.c @@ -184,3 +184,117 @@ void *kmsan_get_metadata(void *address, bool is_origi= n) ret =3D (is_origin ? origin_ptr_for(page) : shadow_ptr_for(page)) + off; return ret; } + +void kmsan_copy_page_meta(struct page *dst, struct page *src) +{ + if (!kmsan_enabled || kmsan_in_runtime()) + return; + if (!dst || !page_has_metadata(dst)) + return; + if (!src || !page_has_metadata(src)) { + kmsan_internal_unpoison_memory(page_address(dst), PAGE_SIZE, + /*checked*/ false); + return; + } + + kmsan_enter_runtime(); + __memcpy(shadow_ptr_for(dst), shadow_ptr_for(src), PAGE_SIZE); + __memcpy(origin_ptr_for(dst), origin_ptr_for(src), PAGE_SIZE); + kmsan_leave_runtime(); +} + +void kmsan_alloc_page(struct page *page, unsigned int order, gfp_t flags) +{ + bool initialized =3D (flags & __GFP_ZERO) || !kmsan_enabled; + struct page *shadow, *origin; + depot_stack_handle_t handle; + int pages =3D 1 << order; + int i; + + if (!page) + return; + + shadow =3D shadow_page_for(page); + origin =3D origin_page_for(page); + + if (initialized) { + __memset(page_address(shadow), 0, PAGE_SIZE * pages); + __memset(page_address(origin), 0, PAGE_SIZE * pages); + return; + } + + /* Zero pages allocated by the runtime should also be initialized. */ + if (kmsan_in_runtime()) + return; + + __memset(page_address(shadow), -1, PAGE_SIZE * pages); + kmsan_enter_runtime(); + handle =3D kmsan_save_stack_with_flags(flags, /*extra_bits*/ 0); + kmsan_leave_runtime(); + /* + * Addresses are page-aligned, pages are contiguous, so it's ok + * to just fill the origin pages with |handle|. + */ + for (i =3D 0; i < PAGE_SIZE * pages / sizeof(handle); i++) + ((depot_stack_handle_t *)page_address(origin))[i] =3D handle; +} + +void kmsan_free_page(struct page *page, unsigned int order) +{ + if (!kmsan_enabled || kmsan_in_runtime()) + return; + kmsan_enter_runtime(); + kmsan_internal_poison_memory(page_address(page), + PAGE_SIZE << compound_order(page), + GFP_KERNEL, + KMSAN_POISON_CHECK | KMSAN_POISON_FREE); + kmsan_leave_runtime(); +} + +void kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end, + pgprot_t prot, struct page **pages, + unsigned int page_shift) +{ + unsigned long shadow_start, origin_start, shadow_end, origin_end; + struct page **s_pages, **o_pages; + int nr, i, mapped; + + if (!kmsan_enabled) + return; + + shadow_start =3D vmalloc_meta((void *)start, KMSAN_META_SHADOW); + shadow_end =3D vmalloc_meta((void *)end, KMSAN_META_SHADOW); + if (!shadow_start) + return; + + nr =3D (end - start) / PAGE_SIZE; + s_pages =3D kcalloc(nr, sizeof(struct page *), GFP_KERNEL); + o_pages =3D kcalloc(nr, sizeof(struct page *), GFP_KERNEL); + if (!s_pages || !o_pages) + goto ret; + for (i =3D 0; i < nr; i++) { + s_pages[i] =3D shadow_page_for(pages[i]); + o_pages[i] =3D origin_page_for(pages[i]); + } + prot =3D __pgprot(pgprot_val(prot) | _PAGE_NX); + prot =3D PAGE_KERNEL; + + origin_start =3D vmalloc_meta((void *)start, KMSAN_META_ORIGIN); + origin_end =3D vmalloc_meta((void *)end, KMSAN_META_ORIGIN); + kmsan_enter_runtime(); + mapped =3D __vmap_pages_range_noflush(shadow_start, shadow_end, prot, + s_pages, page_shift); + KMSAN_WARN_ON(mapped); + mapped =3D __vmap_pages_range_noflush(origin_start, origin_end, prot, + o_pages, page_shift); + KMSAN_WARN_ON(mapped); + kmsan_leave_runtime(); + flush_tlb_kernel_range(shadow_start, shadow_end); + flush_tlb_kernel_range(origin_start, origin_end); + flush_cache_vmap(shadow_start, shadow_end); + flush_cache_vmap(origin_start, origin_end); + +ret: + kfree(s_pages); + kfree(o_pages); +} diff --git a/mm/memory.c b/mm/memory.c index 76e3af9639d93..04aa68acffeb3 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -52,6 +52,7 @@ #include #include #include +#include #include #include #include @@ -3032,6 +3033,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) put_page(old_page); return 0; } + kmsan_copy_page_meta(new_page, old_page); } =20 if (mem_cgroup_charge(page_folio(new_page), mm, GFP_KERNEL)) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0e42038382c12..98393e01e4259 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include @@ -1305,6 +1306,7 @@ static __always_inline bool free_pages_prepare(struct= page *page, VM_BUG_ON_PAGE(PageTail(page), page); =20 trace_mm_page_free(page, order); + kmsan_free_page(page, order); =20 if (unlikely(PageHWPoison(page)) && !order) { /* @@ -3696,6 +3698,14 @@ static struct page *rmqueue_pcplist(struct zone *pre= ferred_zone, /* * Allocate a page from the given zone. Use pcplists for order-0 allocatio= ns. */ + +/* + * Do not instrument rmqueue() with KMSAN. This function may call + * __msan_poison_alloca() through a call to set_pfnblock_flags_mask(). + * If __msan_poison_alloca() attempts to allocate pages for the stack depo= t, it + * may call rmqueue() again, which will result in a deadlock. + */ +__no_sanitize_memory static inline struct page *rmqueue(struct zone *preferred_zone, struct zone *zone, unsigned int order, @@ -5428,6 +5438,7 @@ struct page *__alloc_pages(gfp_t gfp, unsigned int or= der, int preferred_nid, } =20 trace_mm_page_alloc(page, order, alloc_gfp, ac.migratetype); + kmsan_alloc_page(page, order, alloc_gfp); =20 return page; } diff --git a/mm/vmalloc.c b/mm/vmalloc.c index cadfbb5155ea5..76a54007e5142 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -320,6 +320,9 @@ int ioremap_page_range(unsigned long addr, unsigned lon= g end, err =3D vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot), ioremap_max_page_shift); flush_cache_vmap(addr, end); + if (!err) + kmsan_ioremap_page_range(addr, end, phys_addr, prot, + ioremap_max_page_shift); return err; } =20 @@ -419,7 +422,7 @@ static void vunmap_p4d_range(pgd_t *pgd, unsigned long = addr, unsigned long end, * * This is an internal function only. Do not use outside mm/. */ -void vunmap_range_noflush(unsigned long start, unsigned long end) +void __vunmap_range_noflush(unsigned long start, unsigned long end) { unsigned long next; pgd_t *pgd; @@ -441,6 +444,12 @@ void vunmap_range_noflush(unsigned long start, unsigne= d long end) arch_sync_kernel_mappings(start, end); } =20 +void vunmap_range_noflush(unsigned long start, unsigned long end) +{ + kmsan_vunmap_range_noflush(start, end); + __vunmap_range_noflush(start, end); +} + /** * vunmap_range - unmap kernel virtual addresses * @addr: start of the VM area to unmap @@ -575,7 +584,7 @@ static int vmap_small_pages_range_noflush(unsigned long= addr, unsigned long end, * * This is an internal function only. Do not use outside mm/. */ -int vmap_pages_range_noflush(unsigned long addr, unsigned long end, +int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { unsigned int i, nr =3D (end - addr) >> PAGE_SHIFT; @@ -601,6 +610,13 @@ int vmap_pages_range_noflush(unsigned long addr, unsig= ned long end, return 0; } =20 +int vmap_pages_range_noflush(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift) +{ + kmsan_vmap_pages_range_noflush(addr, end, prot, pages, page_shift); + return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift); +} + /** * vmap_pages_range - map pages to a kernel virtual address * @addr: start of the VM area to map --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4A6BC433EF for ; Tue, 26 Apr 2022 16:46:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353254AbiDZQtm (ORCPT ); Tue, 26 Apr 2022 12:49:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44236 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353232AbiDZQsr (ORCPT ); Tue, 26 Apr 2022 12:48:47 -0400 Received: from mail-ed1-x54a.google.com (mail-ed1-x54a.google.com [IPv6:2a00:1450:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9D872275C6 for ; Tue, 26 Apr 2022 09:45:06 -0700 (PDT) Received: by mail-ed1-x54a.google.com with SMTP id cf16-20020a0564020b9000b00425d543c75dso4624062edb.11 for ; Tue, 26 Apr 2022 09:45:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=mDlUedrAZbNWp5ky59jIg6dQAqQa/OsCzADtdk11998=; b=YzWXjysgR42Q3TFVESmvN5pkk/chWv55MOWNS/6w5CgonTDe7EDOi6/dgov5acrkBk +eLpaTXxOwDMeDh/5NLJd2WaBWiomoLwEkXAYy/dGDvTLkfWpvwhhRdnoGBIUGURSa1L l9wqvDH8ai6JCL9QFA+PPkYwc0EQxgwx22SxZgPjsmRCAAgfTssPPlR7afWFGoadKtI2 Eron+27fhMxsLHiuvzn/2YPIzI2J1qzeOFB+DZVuNcwHxAUs4RAkE6xoNyDRZHC6y18/ xT82pJiOb+S+0kbNumQuG4h2OZz5fftBJtuJ9LtfEMIi/Y/tl9m44tBDJirEip3Wxd0t /VOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=mDlUedrAZbNWp5ky59jIg6dQAqQa/OsCzADtdk11998=; b=fttM4uDfCUqNKxFyWHXDcOjlbsuRg6WyeM8R5FDD5aPPIICgoglEUq+8MbGHKlK+p9 EFY0eodm9YVTnImcPePvgsZAzapmVLzKvtpM0ykHZ3fMyh5PXj/UDOUVR1okHZvSMLwU 5q0qLPCcVtCJBxjjq3QSZy+eN3t0SoTUBtJ6JKJxvpVQ2nep5tOiF5V0K0XpPgTucFSn V4TGvmECHl57FqinN/CaRjlAi2PoJQGDnDqZKtnMRBAfAetJOYqfwsHZ0wcF/+eykShI YEkpvVaiqAxzztz1fIDIBkeq6aWnF5SoFq8GDtRf4O/c2+P0uUGndGlEB6Ye9EbnYWmP iZig== X-Gm-Message-State: AOAM5325zV7/801e4Bv3UPZt6hD9RGDTNMjbtAASELPlf+ysPayjx6mv BwUWp3UTfym92rLAlsOssstidY7jYJA= X-Google-Smtp-Source: ABdhPJwjleNQ43dfYY89UGlYdB1Ku84OaO0hN9pyi13kzCfUmzv59quoT4hFyhlBNfthppVsEwYHULVNmjY= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:5114:b0:423:f33d:b3c with SMTP id m20-20020a056402511400b00423f33d0b3cmr25558297edd.199.1650991504954; Tue, 26 Apr 2022 09:45:04 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:46 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-18-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 17/46] kmsan: mm: call KMSAN hooks from SLUB code From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In order to report uninitialized memory coming from heap allocations KMSAN has to poison them unless they're created with __GFP_ZERO. It's handy that we need KMSAN hooks in the places where init_on_alloc/init_on_free initialization is performed. Signed-off-by: Alexander Potapenko --- v2: -- move the implementation of SLUB hooks here Link: https://linux-review.googlesource.com/id/I6954b386c5c5d7f99f48bb6cbcc= 74b75136ce86e --- include/linux/kmsan.h | 57 ++++++++++++++++++++++++++++++ mm/kmsan/hooks.c | 80 +++++++++++++++++++++++++++++++++++++++++++ mm/slab.h | 1 + mm/slub.c | 21 ++++++++++-- 4 files changed, 157 insertions(+), 2 deletions(-) diff --git a/include/linux/kmsan.h b/include/linux/kmsan.h index da41850b46cbd..ed3630068e2ef 100644 --- a/include/linux/kmsan.h +++ b/include/linux/kmsan.h @@ -16,6 +16,7 @@ #include =20 struct page; +struct kmem_cache; =20 #ifdef CONFIG_KMSAN =20 @@ -73,6 +74,44 @@ void kmsan_free_page(struct page *page, unsigned int ord= er); */ void kmsan_copy_page_meta(struct page *dst, struct page *src); =20 +/** + * kmsan_slab_alloc() - Notify KMSAN about a slab allocation. + * @s: slab cache the object belongs to. + * @object: object pointer. + * @flags: GFP flags passed to the allocator. + * + * Depending on cache flags and GFP flags, KMSAN sets up the metadata of t= he + * newly created object, marking it as initialized or uninitialized. + */ +void kmsan_slab_alloc(struct kmem_cache *s, void *object, gfp_t flags); + +/** + * kmsan_slab_free() - Notify KMSAN about a slab deallocation. + * @s: slab cache the object belongs to. + * @object: object pointer. + * + * KMSAN marks the freed object as uninitialized. + */ +void kmsan_slab_free(struct kmem_cache *s, void *object); + +/** + * kmsan_kmalloc_large() - Notify KMSAN about a large slab allocation. + * @ptr: object pointer. + * @size: object size. + * @flags: GFP flags passed to the allocator. + * + * Similar to kmsan_slab_alloc(), but for large allocations. + */ +void kmsan_kmalloc_large(const void *ptr, size_t size, gfp_t flags); + +/** + * kmsan_kfree_large() - Notify KMSAN about a large slab deallocation. + * @ptr: object pointer. + * + * Similar to kmsan_slab_free(), but for large allocations. + */ +void kmsan_kfree_large(const void *ptr); + /** * kmsan_map_kernel_range_noflush() - Notify KMSAN about a vmap. * @start: start of vmapped range. @@ -139,6 +178,24 @@ static inline void kmsan_copy_page_meta(struct page *d= st, struct page *src) { } =20 +static inline void kmsan_slab_alloc(struct kmem_cache *s, void *object, + gfp_t flags) +{ +} + +static inline void kmsan_slab_free(struct kmem_cache *s, void *object) +{ +} + +static inline void kmsan_kmalloc_large(const void *ptr, size_t size, + gfp_t flags) +{ +} + +static inline void kmsan_kfree_large(const void *ptr) +{ +} + static inline void kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end, pgprot_t prot, diff --git a/mm/kmsan/hooks.c b/mm/kmsan/hooks.c index 070756be70e3a..052e17b7a717d 100644 --- a/mm/kmsan/hooks.c +++ b/mm/kmsan/hooks.c @@ -26,6 +26,86 @@ * skipping effects of functions like memset() inside instrumented code. */ =20 +void kmsan_slab_alloc(struct kmem_cache *s, void *object, gfp_t flags) +{ + if (unlikely(object =3D=3D NULL)) + return; + if (!kmsan_enabled || kmsan_in_runtime()) + return; + /* + * There's a ctor or this is an RCU cache - do nothing. The memory + * status hasn't changed since last use. + */ + if (s->ctor || (s->flags & SLAB_TYPESAFE_BY_RCU)) + return; + + kmsan_enter_runtime(); + if (flags & __GFP_ZERO) + kmsan_internal_unpoison_memory(object, s->object_size, + KMSAN_POISON_CHECK); + else + kmsan_internal_poison_memory(object, s->object_size, flags, + KMSAN_POISON_CHECK); + kmsan_leave_runtime(); +} +EXPORT_SYMBOL(kmsan_slab_alloc); + +void kmsan_slab_free(struct kmem_cache *s, void *object) +{ + if (!kmsan_enabled || kmsan_in_runtime()) + return; + + /* RCU slabs could be legally used after free within the RCU period */ + if (unlikely(s->flags & (SLAB_TYPESAFE_BY_RCU | SLAB_POISON))) + return; + /* + * If there's a constructor, freed memory must remain in the same state + * until the next allocation. We cannot save its state to detect + * use-after-free bugs, instead we just keep it unpoisoned. + */ + if (s->ctor) + return; + kmsan_enter_runtime(); + kmsan_internal_poison_memory(object, s->object_size, GFP_KERNEL, + KMSAN_POISON_CHECK | KMSAN_POISON_FREE); + kmsan_leave_runtime(); +} +EXPORT_SYMBOL(kmsan_slab_free); + +void kmsan_kmalloc_large(const void *ptr, size_t size, gfp_t flags) +{ + if (unlikely(ptr =3D=3D NULL)) + return; + if (!kmsan_enabled || kmsan_in_runtime()) + return; + kmsan_enter_runtime(); + if (flags & __GFP_ZERO) + kmsan_internal_unpoison_memory((void *)ptr, size, + /*checked*/ true); + else + kmsan_internal_poison_memory((void *)ptr, size, flags, + KMSAN_POISON_CHECK); + kmsan_leave_runtime(); +} +EXPORT_SYMBOL(kmsan_kmalloc_large); + +void kmsan_kfree_large(const void *ptr) +{ + struct page *page; + + if (!kmsan_enabled || kmsan_in_runtime()) + return; + kmsan_enter_runtime(); + page =3D virt_to_head_page((void *)ptr); + KMSAN_WARN_ON(ptr !=3D page_address(page)); + kmsan_internal_poison_memory((void *)ptr, + PAGE_SIZE << compound_order(page), + GFP_KERNEL, + KMSAN_POISON_CHECK | KMSAN_POISON_FREE); + kmsan_leave_runtime(); +} +EXPORT_SYMBOL(kmsan_kfree_large); + static unsigned long vmalloc_shadow(unsigned long addr) { return (unsigned long)kmsan_get_metadata((void *)addr, diff --git a/mm/slab.h b/mm/slab.h index 95eb34174c1bb..1276b83656091 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -751,6 +751,7 @@ static inline void slab_post_alloc_hook(struct kmem_cac= he *s, memset(p[i], 0, s->object_size); kmemleak_alloc_recursive(p[i], s->object_size, 1, s->flags, flags); + kmsan_slab_alloc(s, p[i], flags); } =20 memcg_slab_post_alloc_hook(s, objcg, flags, size, p); diff --git a/mm/slub.c b/mm/slub.c index ed5c2c03a47aa..45082acaa6739 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -22,6 +22,7 @@ #include #include #include +#include #include #include #include @@ -357,18 +358,28 @@ static void prefetch_freepointer(const struct kmem_ca= che *s, void *object) prefetchw(object + s->offset); } =20 +/* + * When running under KMSAN, get_freepointer_safe() may return an uninitia= lized + * pointer value in the case the current thread loses the race for the next + * memory chunk in the freelist. In that case this_cpu_cmpxchg_double() in + * slab_alloc_node() will fail, so the uninitialized value won't be used, = but + * KMSAN will still check all arguments of cmpxchg because of imperfect + * handling of inline assembly. + * To work around this problem, use kmsan_init() to force initialize the + * return value of get_freepointer_safe(). + */ static inline void *get_freepointer_safe(struct kmem_cache *s, void *objec= t) { unsigned long freepointer_addr; void *p; =20 if (!debug_pagealloc_enabled_static()) - return get_freepointer(s, object); + return kmsan_init(get_freepointer(s, object)); =20 object =3D kasan_reset_tag(object); freepointer_addr =3D (unsigned long)object + s->offset; copy_from_kernel_nofault(&p, (void **)freepointer_addr, sizeof(p)); - return freelist_ptr(s, p, freepointer_addr); + return kmsan_init(freelist_ptr(s, p, freepointer_addr)); } =20 static inline void set_freepointer(struct kmem_cache *s, void *object, voi= d *fp) @@ -1683,6 +1694,7 @@ static inline void *kmalloc_large_node_hook(void *ptr= , size_t size, gfp_t flags) ptr =3D kasan_kmalloc_large(ptr, size, flags); /* As ptr might get tagged, call kmemleak hook after KASAN. */ kmemleak_alloc(ptr, size, 1, flags); + kmsan_kmalloc_large(ptr, size, flags); return ptr; } =20 @@ -1690,12 +1702,14 @@ static __always_inline void kfree_hook(void *x) { kmemleak_free(x); kasan_kfree_large(x); + kmsan_kfree_large(x); } =20 static __always_inline bool slab_free_hook(struct kmem_cache *s, void *x, bool init) { kmemleak_free_recursive(x, s->flags); + kmsan_slab_free(s, x); =20 debug_check_no_locks_freed(x, s->object_size); =20 @@ -3730,6 +3744,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t= flags, size_t size, */ slab_post_alloc_hook(s, objcg, flags, size, p, slab_want_init_on_alloc(flags, s)); + return i; error: slub_put_cpu_ptr(s->cpu_slab); @@ -5898,6 +5913,7 @@ static char *create_unique_id(struct kmem_cache *s) p +=3D sprintf(p, "%07u", s->size); =20 BUG_ON(p > name + ID_STR_LENGTH - 1); + kmsan_unpoison_memory(name, p - name); return name; } =20 @@ -5999,6 +6015,7 @@ static int sysfs_slab_alias(struct kmem_cache *s, con= st char *name) al->name =3D name; al->next =3D alias_list; alias_list =3D al; + kmsan_unpoison_memory(al, sizeof(struct saved_alias)); return 0; } =20 --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0C07C433F5 for ; Tue, 26 Apr 2022 16:47:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353335AbiDZQuT (ORCPT ); Tue, 26 Apr 2022 12:50:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43198 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353259AbiDZQs6 (ORCPT ); Tue, 26 Apr 2022 12:48:58 -0400 Received: from mail-ej1-x649.google.com (mail-ej1-x649.google.com [IPv6:2a00:1450:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 119EFDED3 for ; Tue, 26 Apr 2022 09:45:08 -0700 (PDT) Received: by mail-ej1-x649.google.com with SMTP id x2-20020a1709065ac200b006d9b316257fso9386592ejs.12 for ; Tue, 26 Apr 2022 09:45:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=d8pL6jrPvx0aomk5qPe/O8+76gWOKR1XrEif20OjnQY=; b=qHuNE5uJ2FDzh5HoUwOExyBkIzzUGi7u3r8cb1fmNh9JoURq8b/0KZazqEqktkONuv RSLViVqt3oajP2cDiy1OzrTVw7OGCCwD1o48WWrtmf3dphbUmtfJDJYJSX5YCXSpNfdF aXfERsMSgciWjwFSSbUv2J0vUdBe6KCUA+6Uqtn6lrlmCctTBymZW6DYKyoqZXJTtacd rnvL/jsjRqjNWe6PKbeI0qf9Va3S9UXaomO1/Ti1Zz7QraDNV7xVEPjV7ex+essI4Lra yDFX1T5khoEdzXqF+In8TAeLE7luEASOtTb9QCfHpF9L7qw2pr9BQJoomZYDCxZL6tCZ 7FYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=d8pL6jrPvx0aomk5qPe/O8+76gWOKR1XrEif20OjnQY=; b=HEuE5yOZIIImPe07BRznMWafyOsn9AFImX9YeJ2pUX6qsigJymBSgS12PaEUVw584I gLgxG9xEk80abHOzd4i+tnw0v+uJRPAeKnfi7BGg93JGt8IqXsv6YTPROVgvfRWKyUo1 G1siLEksMydiVWMCoSfYM/PTORukyjdcCSKuQjGVaVclkkOJ1m//qDQ1peEbt8o24/wv xx02gXm71VOn0uwrZIwaXb+1pigGvSG6k06lLkQXZUrszzcmFPJgOMlPV9wtPMqVi6WI ROhe7KpNqno3MbjtvZognuTL+i/DzQB0hSoDCjqOm0XwQXuPpg3MkEGTg5ssW4/tyc0w 65Lg== X-Gm-Message-State: AOAM5325cCAY0QN7ALHfuJfN8mZ4DSWDkWtaf4Y4U8R20Ndk9+qrbNlx 9GbsBJEfCxK7ir/F+3Nai7VkXzyYQ0s= X-Google-Smtp-Source: ABdhPJxFI4DglMrNp8eiqcTF6h41+ZsecBuhf95AwThCb0V12de/kX/qAKmHnmtUDGFTwaevSaUDv/yKfZQ= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:486:b0:413:bd00:4f3f with SMTP id k6-20020a056402048600b00413bd004f3fmr26069921edv.103.1650991507458; Tue, 26 Apr 2022 09:45:07 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:47 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-19-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 18/46] kmsan: handle task creation and exiting From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Tell KMSAN that a new task is created, so the tool creates a backing metadata structure for that task. Signed-off-by: Alexander Potapenko --- v2: -- move implementation of kmsan_task_create() and kmsan_task_exit() here Link: https://linux-review.googlesource.com/id/I0f41c3a1c7d66f7e14aabcfdfc7= c69addb945805 --- include/linux/kmsan.h | 17 +++++++++++++++++ kernel/exit.c | 2 ++ kernel/fork.c | 2 ++ mm/kmsan/core.c | 10 ++++++++++ mm/kmsan/hooks.c | 19 +++++++++++++++++++ mm/kmsan/kmsan.h | 2 ++ 6 files changed, 52 insertions(+) diff --git a/include/linux/kmsan.h b/include/linux/kmsan.h index ed3630068e2ef..dca42e0e91991 100644 --- a/include/linux/kmsan.h +++ b/include/linux/kmsan.h @@ -17,6 +17,7 @@ =20 struct page; struct kmem_cache; +struct task_struct; =20 #ifdef CONFIG_KMSAN =20 @@ -43,6 +44,14 @@ struct kmsan_ctx { bool allow_reporting; }; =20 +void kmsan_task_create(struct task_struct *task); + +/** + * kmsan_task_exit() - Notify KMSAN that a task has exited. + * @task: task about to finish. + */ +void kmsan_task_exit(struct task_struct *task); + /** * kmsan_alloc_page() - Notify KMSAN about an alloc_pages() call. * @page: struct page pointer returned by alloc_pages(). @@ -164,6 +173,14 @@ void kmsan_iounmap_page_range(unsigned long start, uns= igned long end); =20 #else =20 +static inline void kmsan_task_create(struct task_struct *task) +{ +} + +static inline void kmsan_task_exit(struct task_struct *task) +{ +} + static inline int kmsan_alloc_page(struct page *page, unsigned int order, gfp_t flags) { diff --git a/kernel/exit.c b/kernel/exit.c index f072959fcab7f..1784b7a741ddd 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -60,6 +60,7 @@ #include #include #include +#include #include #include #include @@ -741,6 +742,7 @@ void __noreturn do_exit(long code) WARN_ON(tsk->plug); =20 kcov_task_exit(tsk); + kmsan_task_exit(tsk); =20 coredump_task_exit(tsk); ptrace_event(PTRACE_EVENT_EXIT, code); diff --git a/kernel/fork.c b/kernel/fork.c index 9796897560ab1..a6178bd28c409 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -37,6 +37,7 @@ #include #include #include +#include #include #include #include @@ -1027,6 +1028,7 @@ static struct task_struct *dup_task_struct(struct tas= k_struct *orig, int node) tsk->worker_private =3D NULL; =20 kcov_task_init(tsk); + kmsan_task_create(tsk); kmap_local_fork(tsk); =20 #ifdef CONFIG_FAULT_INJECTION diff --git a/mm/kmsan/core.c b/mm/kmsan/core.c index 933d864d9d467..4b405abbb6c03 100644 --- a/mm/kmsan/core.c +++ b/mm/kmsan/core.c @@ -44,6 +44,16 @@ bool kmsan_enabled __read_mostly; */ DEFINE_PER_CPU(struct kmsan_ctx, kmsan_percpu_ctx); =20 +void kmsan_internal_task_create(struct task_struct *task) +{ + struct kmsan_ctx *ctx =3D &task->kmsan_ctx; + + __memset(ctx, 0, sizeof(struct kmsan_ctx)); + ctx->allow_reporting =3D true; + kmsan_internal_unpoison_memory(current_thread_info(), + sizeof(struct thread_info), false); +} + void kmsan_internal_poison_memory(void *address, size_t size, gfp_t flags, unsigned int poison_flags) { diff --git a/mm/kmsan/hooks.c b/mm/kmsan/hooks.c index 052e17b7a717d..43a529569053d 100644 --- a/mm/kmsan/hooks.c +++ b/mm/kmsan/hooks.c @@ -26,6 +26,25 @@ * skipping effects of functions like memset() inside instrumented code. */ =20 +void kmsan_task_create(struct task_struct *task) +{ + kmsan_enter_runtime(); + kmsan_internal_task_create(task); + kmsan_leave_runtime(); +} +EXPORT_SYMBOL(kmsan_task_create); + +void kmsan_task_exit(struct task_struct *task) +{ + struct kmsan_ctx *ctx =3D &task->kmsan_ctx; + + if (!kmsan_enabled || kmsan_in_runtime()) + return; + + ctx->allow_reporting =3D false; +} +EXPORT_SYMBOL(kmsan_task_exit); + void kmsan_slab_alloc(struct kmem_cache *s, void *object, gfp_t flags) { if (unlikely(object =3D=3D NULL)) diff --git a/mm/kmsan/kmsan.h b/mm/kmsan/kmsan.h index bfe38789950a6..a1b5900ffd97b 100644 --- a/mm/kmsan/kmsan.h +++ b/mm/kmsan/kmsan.h @@ -172,6 +172,8 @@ void kmsan_internal_set_shadow_origin(void *address, si= ze_t size, int b, u32 origin, bool checked); depot_stack_handle_t kmsan_internal_chain_origin(depot_stack_handle_t id); =20 +void kmsan_internal_task_create(struct task_struct *task); + bool kmsan_metadata_is_contiguous(void *addr, size_t size); void kmsan_internal_check_memory(void *addr, size_t size, const void *user= _addr, int reason); --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77EB9C4332F for ; Tue, 26 Apr 2022 16:47:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351422AbiDZQuY (ORCPT ); Tue, 26 Apr 2022 12:50:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44730 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353261AbiDZQs6 (ORCPT ); Tue, 26 Apr 2022 12:48:58 -0400 Received: from mail-ej1-x64a.google.com (mail-ej1-x64a.google.com [IPv6:2a00:1450:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 851E9DEA1 for ; Tue, 26 Apr 2022 09:45:11 -0700 (PDT) Received: by mail-ej1-x64a.google.com with SMTP id qw33-20020a1709066a2100b006f001832229so9338873ejc.4 for ; Tue, 26 Apr 2022 09:45:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=chRz9X+UoCgYtEKtJoTXT2tB633Q6kPzDish0ozWoJs=; b=iILPrkLPO23rfUUMvFSi4Z8er+bWDXpVUGW5WdqHRHVMWR6ksGloltRebKp+ZboRbt QDGCpX85b/0Qkl+38vYNDUD9JIgepW3D8ilhETtrTocxGWyK+r1AYQUDE4n0p+/wYlmc +cjjHVpxRswHVzBMoiapsUo2PnuCGD3h//7hXHdNCrq7yv12Xr7dPUFXuZn2YlpYkDIe lwg8HyX5Wczbjw3trREix2yESuYLL0ykOVKHx8fhUb4wFwtbgRnzyHxtzjBWFXxAHUwr iy1IY/QUZNxe2BPBRYT44V3X6pBMsGR9M5gPPvWc6U8RiiktG08RiBn63BI0OAZNzBBY SNWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=chRz9X+UoCgYtEKtJoTXT2tB633Q6kPzDish0ozWoJs=; b=Qm2TDoyaajvEmPSFXNdHXsxftzYxQAy7ZSj6GFmLc/mSQYzf1eAGJONwGQQAdMNjiL f57WgwYNmhDEb7blFCOk247YQ7FQ2TpGBdkVbgRdqkD/+w4zR/3k+ryYLDCd7Lm5e4gP 9SjMw22a93yqAn4ibOghBKP1v/w+LoPl1YrhOOvGdI0iCDuHbIU5/SN/6m/LjuB8kgqc Ads/jkLsXxoRAq0hpUxhD3y4rYShFGe1x5BiJB5SGEhPofa8RICsTyI9E7e5HjBwOQAy cAn1mlrKi86ryd8WN01hC5IyYO3/CmW64L+MIRA31ff//H2Pa1V7lEARW0mlz3z5nNT9 XMrw== X-Gm-Message-State: AOAM532xuAAxldd12Dz2ZDMXHh/KfX7jpw84CWaG0CiiPvRVnHfcd5Vb HhfopSolybGQk6CRXIrR+QNPHpPlyUU= X-Google-Smtp-Source: ABdhPJwRDzo6oVCot+pcNMGMkgKP0gMioynSPYX1iZPl03lH9XerXzgQUEWYQxlu8/hqLkIa5gsTio/2muM= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:5189:b0:423:f342:e0ce with SMTP id q9-20020a056402518900b00423f342e0cemr25740119edd.120.1650991509865; Tue, 26 Apr 2022 09:45:09 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:48 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-20-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 19/46] kmsan: init: call KMSAN initialization routines From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" kmsan_init_shadow() scans the mappings created at boot time and creates metadata pages for those mappings. When the memblock allocator returns pages to pagealloc, we reserve 2/3 of those pages and use them as metadata for the remaining 1/3. Once KMSAN starts, every page allocated by pagealloc has its associated shadow and origin pages. kmsan_initialize() initializes the bookkeeping for init_task and enables KMSAN. Signed-off-by: Alexander Potapenko --- v2: -- move mm/kmsan/init.c and kmsan_memblock_free_pages() to this patch -- print a warning that KMSAN is a debugging tool (per Greg K-H's request) Link: https://linux-review.googlesource.com/id/I7bc53706141275914326df23458= 81ffe0cdd16bd --- include/linux/kmsan.h | 48 +++++++++ init/main.c | 3 + mm/kmsan/Makefile | 3 +- mm/kmsan/init.c | 240 ++++++++++++++++++++++++++++++++++++++++++ mm/kmsan/kmsan.h | 3 + mm/kmsan/shadow.c | 36 +++++++ mm/page_alloc.c | 3 + 7 files changed, 335 insertions(+), 1 deletion(-) create mode 100644 mm/kmsan/init.c diff --git a/include/linux/kmsan.h b/include/linux/kmsan.h index dca42e0e91991..a5767c728a46b 100644 --- a/include/linux/kmsan.h +++ b/include/linux/kmsan.h @@ -52,6 +52,40 @@ void kmsan_task_create(struct task_struct *task); */ void kmsan_task_exit(struct task_struct *task); =20 +/** + * kmsan_init_shadow() - Initialize KMSAN shadow at boot time. + * + * Allocate and initialize KMSAN metadata for early allocations. + */ +void __init kmsan_init_shadow(void); + +/** + * kmsan_init_runtime() - Initialize KMSAN state and enable KMSAN. + */ +void __init kmsan_init_runtime(void); + +/** + * kmsan_memblock_free_pages() - handle freeing of memblock pages. + * @page: struct page to free. + * @order: order of @page. + * + * Freed pages are either returned to buddy allocator or held back to be u= sed + * as metadata pages. + */ +bool __init kmsan_memblock_free_pages(struct page *page, unsigned int orde= r); + +/** + * kmsan_task_create() - Initialize KMSAN state for the task. + * @task: task to initialize. + */ +void kmsan_task_create(struct task_struct *task); + +/** + * kmsan_task_exit() - Notify KMSAN that a task has exited. + * @task: task about to finish. + */ +void kmsan_task_exit(struct task_struct *task); + /** * kmsan_alloc_page() - Notify KMSAN about an alloc_pages() call. * @page: struct page pointer returned by alloc_pages(). @@ -173,6 +207,20 @@ void kmsan_iounmap_page_range(unsigned long start, uns= igned long end); =20 #else =20 +static inline void kmsan_init_shadow(void) +{ +} + +static inline void kmsan_init_runtime(void) +{ +} + +static inline bool kmsan_memblock_free_pages(struct page *page, + unsigned int order) +{ + return true; +} + static inline void kmsan_task_create(struct task_struct *task) { } diff --git a/init/main.c b/init/main.c index 98182c3c2c4b3..5c6937921c890 100644 --- a/init/main.c +++ b/init/main.c @@ -34,6 +34,7 @@ #include #include #include +#include #include #include #include @@ -835,6 +836,7 @@ static void __init mm_init(void) init_mem_debugging_and_hardening(); kfence_alloc_pool(); report_meminit(); + kmsan_init_shadow(); stack_depot_early_init(); mem_init(); mem_init_print_info(); @@ -852,6 +854,7 @@ static void __init mm_init(void) init_espfix_bsp(); /* Should be run after espfix64 is set up. */ pti_init(); + kmsan_init_runtime(); } =20 #ifdef CONFIG_RANDOMIZE_KSTACK_OFFSET diff --git a/mm/kmsan/Makefile b/mm/kmsan/Makefile index 73b705cbf75b9..f57a956cb1c8b 100644 --- a/mm/kmsan/Makefile +++ b/mm/kmsan/Makefile @@ -1,4 +1,4 @@ -obj-y :=3D core.o instrumentation.o hooks.o report.o shadow.o annotations.o +obj-y :=3D core.o instrumentation.o init.o hooks.o report.o shadow.o annot= ations.o =20 KMSAN_SANITIZE :=3D n KCOV_INSTRUMENT :=3D n @@ -16,6 +16,7 @@ CFLAGS_REMOVE.o =3D $(CC_FLAGS_FTRACE) CFLAGS_annotations.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) CFLAGS_core.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) CFLAGS_hooks.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) +CFLAGS_init.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) CFLAGS_instrumentation.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) CFLAGS_report.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) CFLAGS_shadow.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) diff --git a/mm/kmsan/init.c b/mm/kmsan/init.c new file mode 100644 index 0000000000000..45757d1390402 --- /dev/null +++ b/mm/kmsan/init.c @@ -0,0 +1,240 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * KMSAN initialization routines. + * + * Copyright (C) 2017-2021 Google LLC + * Author: Alexander Potapenko + * + */ + +#include "kmsan.h" + +#include +#include +#include + +#include "../internal.h" + +#define NUM_FUTURE_RANGES 128 +struct start_end_pair { + u64 start, end; +}; + +static struct start_end_pair start_end_pairs[NUM_FUTURE_RANGES] __initdata; +static int future_index __initdata; + +/* + * Record a range of memory for which the metadata pages will be created o= nce + * the page allocator becomes available. + */ +static void __init kmsan_record_future_shadow_range(void *start, void *end) +{ + u64 nstart =3D (u64)start, nend =3D (u64)end, cstart, cend; + bool merged =3D false; + int i; + + KMSAN_WARN_ON(future_index =3D=3D NUM_FUTURE_RANGES); + KMSAN_WARN_ON((nstart >=3D nend) || !nstart || !nend); + nstart =3D ALIGN_DOWN(nstart, PAGE_SIZE); + nend =3D ALIGN(nend, PAGE_SIZE); + + /* + * Scan the existing ranges to see if any of them overlaps with + * [start, end). In that case, merge the two ranges instead of + * creating a new one. + * The number of ranges is less than 20, so there is no need to organize + * them into a more intelligent data structure. + */ + for (i =3D 0; i < future_index; i++) { + cstart =3D start_end_pairs[i].start; + cend =3D start_end_pairs[i].end; + if ((cstart < nstart && cend < nstart) || + (cstart > nend && cend > nend)) + /* ranges are disjoint - do not merge */ + continue; + start_end_pairs[i].start =3D min(nstart, cstart); + start_end_pairs[i].end =3D max(nend, cend); + merged =3D true; + break; + } + if (merged) + return; + start_end_pairs[future_index].start =3D nstart; + start_end_pairs[future_index].end =3D nend; + future_index++; +} + +/* + * Initialize the shadow for existing mappings during kernel initializatio= n. + * These include kernel text/data sections, NODE_DATA and future ranges + * registered while creating other data (e.g. percpu). + * + * Allocations via memblock can be only done before slab is initialized. + */ +void __init kmsan_init_shadow(void) +{ + const size_t nd_size =3D roundup(sizeof(pg_data_t), PAGE_SIZE); + phys_addr_t p_start, p_end; + int nid; + u64 i; + + for_each_reserved_mem_range(i, &p_start, &p_end) + kmsan_record_future_shadow_range(phys_to_virt(p_start), + phys_to_virt(p_end)); + /* Allocate shadow for .data */ + kmsan_record_future_shadow_range(_sdata, _edata); + + for_each_online_node(nid) + kmsan_record_future_shadow_range( + NODE_DATA(nid), (char *)NODE_DATA(nid) + nd_size); + + for (i =3D 0; i < future_index; i++) + kmsan_init_alloc_meta_for_range( + (void *)start_end_pairs[i].start, + (void *)start_end_pairs[i].end); +} +EXPORT_SYMBOL(kmsan_init_shadow); + +struct page_pair { + struct page *shadow, *origin; +}; +static struct page_pair held_back[MAX_ORDER] __initdata; + +/* + * Eager metadata allocation. When the memblock allocator is freeing pages= to + * pagealloc, we use 2/3 of them as metadata for the remaining 1/3. + * We store the pointers to the returned blocks of pages in held_back[] gr= ouped + * by their order: when kmsan_memblock_free_pages() is called for the first + * time with a certain order, it is reserved as a shadow block, for the se= cond + * time - as an origin block. On the third time the incoming block receive= s its + * shadow and origin ranges from the previously saved shadow and origin bl= ocks, + * after which held_back[order] can be used again. + * + * At the very end there may be leftover blocks in held_back[]. They are + * collected later by kmsan_memblock_discard(). + */ +bool kmsan_memblock_free_pages(struct page *page, unsigned int order) +{ + struct page *shadow, *origin; + + if (!held_back[order].shadow) { + held_back[order].shadow =3D page; + return false; + } + if (!held_back[order].origin) { + held_back[order].origin =3D page; + return false; + } + shadow =3D held_back[order].shadow; + origin =3D held_back[order].origin; + kmsan_setup_meta(page, shadow, origin, order); + + held_back[order].shadow =3D NULL; + held_back[order].origin =3D NULL; + return true; +} + +#define MAX_BLOCKS 8 +struct smallstack { + struct page *items[MAX_BLOCKS]; + int index; + int order; +}; + +struct smallstack collect =3D { + .index =3D 0, + .order =3D MAX_ORDER, +}; + +static void smallstack_push(struct smallstack *stack, struct page *pages) +{ + KMSAN_WARN_ON(stack->index =3D=3D MAX_BLOCKS); + stack->items[stack->index] =3D pages; + stack->index++; +} +#undef MAX_BLOCKS + +static struct page *smallstack_pop(struct smallstack *stack) +{ + struct page *ret; + + KMSAN_WARN_ON(stack->index =3D=3D 0); + stack->index--; + ret =3D stack->items[stack->index]; + stack->items[stack->index] =3D NULL; + return ret; +} + +static void do_collection(void) +{ + struct page *page, *shadow, *origin; + + while (collect.index >=3D 3) { + page =3D smallstack_pop(&collect); + shadow =3D smallstack_pop(&collect); + origin =3D smallstack_pop(&collect); + kmsan_setup_meta(page, shadow, origin, collect.order); + __free_pages_core(page, collect.order); + } +} + +static void collect_split(void) +{ + struct smallstack tmp =3D { + .order =3D collect.order - 1, + .index =3D 0, + }; + struct page *page; + + if (!collect.order) + return; + while (collect.index) { + page =3D smallstack_pop(&collect); + smallstack_push(&tmp, &page[0]); + smallstack_push(&tmp, &page[1 << tmp.order]); + } + __memcpy(&collect, &tmp, sizeof(struct smallstack)); +} + +/* + * Memblock is about to go away. Split the page blocks left over in held_b= ack[] + * and return 1/3 of that memory to the system. + */ +static void kmsan_memblock_discard(void) +{ + int i; + + /* + * For each order=3DN: + * - push held_back[N].shadow and .origin to |collect|; + * - while there are >=3D 3 elements in |collect|, do garbage collection: + * - pop 3 ranges from |collect|; + * - use two of them as shadow and origin for the third one; + * - repeat; + * - split each remaining element from |collect| into 2 ranges of + * order=3DN-1, + * - repeat. + */ + collect.order =3D MAX_ORDER - 1; + for (i =3D MAX_ORDER - 1; i >=3D 0; i--) { + if (held_back[i].shadow) + smallstack_push(&collect, held_back[i].shadow); + if (held_back[i].origin) + smallstack_push(&collect, held_back[i].origin); + held_back[i].shadow =3D NULL; + held_back[i].origin =3D NULL; + do_collection(); + collect_split(); + } +} + +void __init kmsan_init_runtime(void) +{ + /* Assuming current is init_task */ + kmsan_internal_task_create(current); + kmsan_memblock_discard(); + pr_info("Starting KernelMemorySanitizer\n"); + pr_info("ATTENTION: KMSAN is a debugging tool! Do not use it on productio= n machines!\n"); + kmsan_enabled =3D true; +} +EXPORT_SYMBOL(kmsan_init_runtime); diff --git a/mm/kmsan/kmsan.h b/mm/kmsan/kmsan.h index a1b5900ffd97b..059f21c39ec1b 100644 --- a/mm/kmsan/kmsan.h +++ b/mm/kmsan/kmsan.h @@ -66,6 +66,7 @@ struct shadow_origin_ptr { struct shadow_origin_ptr kmsan_get_shadow_origin_ptr(void *addr, u64 size, bool store); void *kmsan_get_metadata(void *addr, bool is_origin); +void __init kmsan_init_alloc_meta_for_range(void *start, void *end); =20 enum kmsan_bug_reason { REASON_ANY, @@ -181,5 +182,7 @@ bool kmsan_internal_is_module_addr(void *vaddr); bool kmsan_internal_is_vmalloc_addr(void *addr); =20 struct page *kmsan_vmalloc_to_page_or_null(void *vaddr); +void kmsan_setup_meta(struct page *page, struct page *shadow, + struct page *origin, int order); =20 #endif /* __MM_KMSAN_KMSAN_H */ diff --git a/mm/kmsan/shadow.c b/mm/kmsan/shadow.c index 8fe6a5ed05e67..99cb9436eddc6 100644 --- a/mm/kmsan/shadow.c +++ b/mm/kmsan/shadow.c @@ -298,3 +298,39 @@ void kmsan_vmap_pages_range_noflush(unsigned long star= t, unsigned long end, kfree(s_pages); kfree(o_pages); } + +/* Allocate metadata for pages allocated at boot time. */ +void __init kmsan_init_alloc_meta_for_range(void *start, void *end) +{ + struct page *shadow_p, *origin_p; + void *shadow, *origin; + struct page *page; + u64 addr, size; + + start =3D (void *)ALIGN_DOWN((u64)start, PAGE_SIZE); + size =3D ALIGN((u64)end - (u64)start, PAGE_SIZE); + shadow =3D memblock_alloc(size, PAGE_SIZE); + origin =3D memblock_alloc(size, PAGE_SIZE); + for (addr =3D 0; addr < size; addr +=3D PAGE_SIZE) { + page =3D virt_to_page_or_null((char *)start + addr); + shadow_p =3D virt_to_page_or_null((char *)shadow + addr); + set_no_shadow_origin_page(shadow_p); + shadow_page_for(page) =3D shadow_p; + origin_p =3D virt_to_page_or_null((char *)origin + addr); + set_no_shadow_origin_page(origin_p); + origin_page_for(page) =3D origin_p; + } +} + +void kmsan_setup_meta(struct page *page, struct page *shadow, + struct page *origin, int order) +{ + int i; + + for (i =3D 0; i < (1 << order); i++) { + set_no_shadow_origin_page(&shadow[i]); + set_no_shadow_origin_page(&origin[i]); + shadow_page_for(&page[i]) =3D &shadow[i]; + origin_page_for(&page[i]) =3D &origin[i]; + } +} diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 98393e01e4259..35b1fedb2f09c 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1716,6 +1716,9 @@ void __init memblock_free_pages(struct page *page, un= signed long pfn, { if (early_page_uninitialised(pfn)) return; + if (!kmsan_memblock_free_pages(page, order)) + /* KMSAN will take care of these pages. */ + return; __free_pages_core(page, order); } =20 --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE7E1C433EF for ; Tue, 26 Apr 2022 16:47:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353291AbiDZQu2 (ORCPT ); Tue, 26 Apr 2022 12:50:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45020 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353286AbiDZQtF (ORCPT ); Tue, 26 Apr 2022 12:49:05 -0400 Received: from mail-ej1-x649.google.com (mail-ej1-x649.google.com [IPv6:2a00:1450:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4BDD6EA6 for ; Tue, 26 Apr 2022 09:45:14 -0700 (PDT) Received: by mail-ej1-x649.google.com with SMTP id dp12-20020a170906c14c00b006e7e8234ae2so9350162ejc.2 for ; Tue, 26 Apr 2022 09:45:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=PbBXRuKFLbtHp5AyM4yBRHn0ZnsWI8OdvT0jRFW7dGk=; b=UzHbOhn27qVpptDuuU81LcEBLt7rOLKE1Z4yNRMf0zkmHeofLcAdHL+SqDQLWH82Il 6cto3RZpExd7ZEnNnAe35UkzS2jkAjUX69Q2YDKqz0g6NCSq/WjRaGJHtw43EHu3cb2U 74H178/TCTR18+CzvmKyfFX9b2Z4X7YP6GK3JO9AiYQ2MlSw+zDsaJJUZQ1bJWeQtgsz KYwuUT517UeVfpvjEvnbP65ucrioTA1bLWKz7WGDz6RnAmYdszQ4/JzyWv4TAbKiuOGb sOlT+jbPA+6lfsFROSMY/3byYOL6djxZQq1uuRZWZI8XWXlEes7tAvr+I/ZTgHKJS0le Mwmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=PbBXRuKFLbtHp5AyM4yBRHn0ZnsWI8OdvT0jRFW7dGk=; b=c+pi3sGdEIBz/PdFn5TQLDM6DsdhCG13tVegFw1XXRVz0qtiNja/eRea7klYAPwlc+ cgaYDBJqIAYVyKEsS532x5qGFlEn6hbwyMvtYOCDTE+gJZ9rxkp8H+Hh5raYbw/FnYY0 mI6WsH2/p66X85eU+yKjkRbhn6AwzSxp8QPSuniYLSKCHWJc4X6FwQflge4RN/rXTxQX m7IRaxiTX//3eV1SNDpGiAIM+KWNv93oMTpKNNFZraEx3QOAHuFVhwwi0anoNolk5kAZ d79aokVVcpafgUwOMlREg4l8dnwLWfADOdqKBphFISzvXAh7pQMml6v9k0tdkiJdwXwT j4Ew== X-Gm-Message-State: AOAM530RCFppeg0S0BM+Q8ELKtGsKoEyzk4bRgyWncFBQLTvqDNRVLoK xey8RGAk1ySZuwgSPGSwvfo0VKNMe3U= X-Google-Smtp-Source: ABdhPJxJswKGNx1XbX2+/B2Rkoy0KYZvI3ec74susirQuoLEq7dCT4VYCkV8muuT7Bszjk5rbONHKLbz8bs= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:400b:b0:425:f59a:c221 with SMTP id d11-20020a056402400b00b00425f59ac221mr7821838eda.307.1650991512487; Tue, 26 Apr 2022 09:45:12 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:49 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-21-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 20/46] instrumented.h: add KMSAN support From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To avoid false positives, KMSAN needs to unpoison the data copied from the userspace. To detect infoleaks - check the memory buffer passed to copy_to_user(). Signed-off-by: Alexander Potapenko --- v2: -- move implementation of kmsan_copy_to_user() here Link: https://linux-review.googlesource.com/id/I43e93b9c02709e6be8d222342f1= b044ac8bdbaaf --- include/linux/instrumented.h | 5 ++++- include/linux/kmsan-checks.h | 19 ++++++++++++++++++ mm/kmsan/hooks.c | 38 ++++++++++++++++++++++++++++++++++++ 3 files changed, 61 insertions(+), 1 deletion(-) diff --git a/include/linux/instrumented.h b/include/linux/instrumented.h index ee8f7d17d34f5..c73c1b19e9227 100644 --- a/include/linux/instrumented.h +++ b/include/linux/instrumented.h @@ -2,7 +2,7 @@ =20 /* * This header provides generic wrappers for memory access instrumentation= that - * the compiler cannot emit for: KASAN, KCSAN. + * the compiler cannot emit for: KASAN, KCSAN, KMSAN. */ #ifndef _LINUX_INSTRUMENTED_H #define _LINUX_INSTRUMENTED_H @@ -10,6 +10,7 @@ #include #include #include +#include #include =20 /** @@ -117,6 +118,7 @@ instrument_copy_to_user(void __user *to, const void *fr= om, unsigned long n) { kasan_check_read(from, n); kcsan_check_read(from, n); + kmsan_copy_to_user(to, from, n, 0); } =20 /** @@ -151,6 +153,7 @@ static __always_inline void instrument_copy_from_user_after(const void *to, const void __user *from, unsigned long n, unsigned long left) { + kmsan_unpoison_memory(to, n - left); } =20 #endif /* _LINUX_INSTRUMENTED_H */ diff --git a/include/linux/kmsan-checks.h b/include/linux/kmsan-checks.h index ecd8336190fc0..aabaf1ba7c251 100644 --- a/include/linux/kmsan-checks.h +++ b/include/linux/kmsan-checks.h @@ -84,6 +84,21 @@ void kmsan_unpoison_memory(const void *address, size_t s= ize); */ void kmsan_check_memory(const void *address, size_t size); =20 +/** + * kmsan_copy_to_user() - Notify KMSAN about a data transfer to userspace. + * @to: destination address in the userspace. + * @from: source address in the kernel. + * @to_copy: number of bytes to copy. + * @left: number of bytes not copied. + * + * If this is a real userspace data transfer, KMSAN checks the bytes that = were + * actually copied to ensure there was no information leak. If @to belongs= to + * the kernel space (which is possible for compat syscalls), KMSAN just co= pies + * the metadata. + */ +void kmsan_copy_to_user(void __user *to, const void *from, size_t to_copy, + size_t left); + #else =20 #define kmsan_init(value) (value) @@ -98,6 +113,10 @@ static inline void kmsan_unpoison_memory(const void *ad= dress, size_t size) static inline void kmsan_check_memory(const void *address, size_t size) { } +static inline void kmsan_copy_to_user(void __user *to, const void *from, + size_t to_copy, size_t left) +{ +} =20 #endif =20 diff --git a/mm/kmsan/hooks.c b/mm/kmsan/hooks.c index 43a529569053d..1cdb4420977f1 100644 --- a/mm/kmsan/hooks.c +++ b/mm/kmsan/hooks.c @@ -212,6 +212,44 @@ void kmsan_iounmap_page_range(unsigned long start, uns= igned long end) } EXPORT_SYMBOL(kmsan_iounmap_page_range); =20 +void kmsan_copy_to_user(void __user *to, const void *from, size_t to_copy, + size_t left) +{ + unsigned long ua_flags; + + if (!kmsan_enabled || kmsan_in_runtime()) + return; + /* + * At this point we've copied the memory already. It's hard to check it + * before copying, as the size of actually copied buffer is unknown. + */ + + /* copy_to_user() may copy zero bytes. No need to check. */ + if (!to_copy) + return; + /* Or maybe copy_to_user() failed to copy anything. */ + if (to_copy <=3D left) + return; + + ua_flags =3D user_access_save(); + if ((u64)to < TASK_SIZE) { + /* This is a user memory access, check it. */ + kmsan_internal_check_memory((void *)from, to_copy - left, to, + REASON_COPY_TO_USER); + user_access_restore(ua_flags); + return; + } + /* Otherwise this is a kernel memory access. This happens when a compat + * syscall passes an argument allocated on the kernel stack to a real + * syscall. + * Don't check anything, just copy the shadow of the copied bytes. + */ + kmsan_internal_memmove_metadata((void *)to, (void *)from, + to_copy - left); + user_access_restore(ua_flags); +} +EXPORT_SYMBOL(kmsan_copy_to_user); + /* Functions from kmsan-checks.h follow. */ void kmsan_poison_memory(const void *address, size_t size, gfp_t flags) { --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 499CBC433F5 for ; Tue, 26 Apr 2022 16:47:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353341AbiDZQul (ORCPT ); Tue, 26 Apr 2022 12:50:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44156 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346708AbiDZQtG (ORCPT ); Tue, 26 Apr 2022 12:49:06 -0400 Received: from mail-ej1-x64a.google.com (mail-ej1-x64a.google.com [IPv6:2a00:1450:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0213D9FF0 for ; Tue, 26 Apr 2022 09:45:16 -0700 (PDT) Received: by mail-ej1-x64a.google.com with SMTP id sg44-20020a170907a42c00b006f3a40146e8so2520852ejc.19 for ; Tue, 26 Apr 2022 09:45:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=vFqZ9iNZJAVQIjd7MnY5jhYV4PJ/+WpYoNR2VRSWpiM=; b=eaD4tCJ/YLTDe4ajl2gQ/TiaUeEA4djHXSqlMDmsulc+qbwyzuVd2W9yINHg7fHnga 3Y+nBojMivWb/vlWu7aJjqdQ3YqnLBqv0yN1IxxX322BLlHzj3IM9ELCwxkKAp16qOhv LyADFdDfCUsE+lxz9NxlRdyYnWL3NGagYS8JkeULjygmzHGrjysJLw6McGcbkSgOELQg 1Rc7TqbJxh506aPuLFlQCeuB/YWI3RUvsBYyts3m6+c10BC++lIyRNin1VxoT8iWyPi+ 5eWh+n4wLJVVQzkWe/7aM9SIO+vr3tXqfKqj3v7+4FSCDACZ5wNHzbimfz+M1Qn6e6s2 Njyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=vFqZ9iNZJAVQIjd7MnY5jhYV4PJ/+WpYoNR2VRSWpiM=; b=7pB5awGjxyAPRCyheTGdpT/KgaJr024DmoF1B35WHQtdKRQ39TI5FMsYB7QDE2FKN1 Gekp34Bqb1CJATLDtJXzs9ecS1LwKqElVJDxCF0n4QVPk7DdcSHJWKcxwqmBA/q6ODPN bIhtx5nfmG8BocEXQoO8cFFLOdLUYSQnTaAdFDbEiOoYMtM3GSCnRXXXxrXmyqmys3sT YztVbp9XqJxKznDx9btrPJfDnr1Jfr2z5oAxfmwilYsUeCtwk7Up09RHXC9PDaTDufgM ES0CeQeCdfTsHDPtedYVVkFTzSZeedn2W3+K4g6en124XQddvl8nie3t2QPJuhHbmDSF Ijkw== X-Gm-Message-State: AOAM5320cLHi5pmVSMZmQQE54Pd/q+8+sBDNZMGhY/hIcM9kdoMFQA1f 1wX/hpsseOrraS6gd7D8F6yJbmQMFTM= X-Google-Smtp-Source: ABdhPJxSGoEtwF4OOq0yTBsFRqUEi4ydvzEhY/S3s2+/j9D6/negQ9CVvZDUHjNLSrRATTf6/ojzjnCmZdU= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a17:906:3e05:b0:6f3:a14a:fd3f with SMTP id k5-20020a1709063e0500b006f3a14afd3fmr7558438eji.640.1650991515225; Tue, 26 Apr 2022 09:45:15 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:50 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-22-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 21/46] kmsan: unpoison @tlb in arch_tlb_gather_mmu() From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This is a hack to reduce stackdepot pressure. struct mmu_gather contains 7 1-bit fields packed into a 32-bit unsigned int value. The remaining 25 bits remain uninitialized and are never used, but KMSAN updates the origin for them in zap_pXX_range() in mm/memory.c, thus creating very long origin chains. This is technically correct, but consumes too much memory. Unpoisoning the whole structure will prevent creating such chains. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/I76abee411b8323acfdbc29bc3a6= 0dca8cff2de77 --- mm/mmu_gather.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index afb7185ffdc45..2f3821268b311 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -1,6 +1,7 @@ #include #include #include +#include #include #include #include @@ -253,6 +254,15 @@ void tlb_flush_mmu(struct mmu_gather *tlb) static void __tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm, bool fullmm) { + /* + * struct mmu_gather contains 7 1-bit fields packed into a 32-bit + * unsigned int value. The remaining 25 bits remain uninitialized + * and are never used, but KMSAN updates the origin for them in + * zap_pXX_range() in mm/memory.c, thus creating very long origin + * chains. This is technically correct, but consumes too much memory. + * Unpoisoning the whole structure will prevent creating such chains. + */ + kmsan_unpoison_memory(tlb, sizeof(*tlb)); tlb->mm =3D mm; tlb->fullmm =3D fullmm; =20 --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66A4BC433F5 for ; Tue, 26 Apr 2022 16:47:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353348AbiDZQur (ORCPT ); Tue, 26 Apr 2022 12:50:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44176 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353196AbiDZQtG (ORCPT ); Tue, 26 Apr 2022 12:49:06 -0400 Received: from mail-ed1-x549.google.com (mail-ed1-x549.google.com [IPv6:2a00:1450:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6450E377D3 for ; Tue, 26 Apr 2022 09:45:19 -0700 (PDT) Received: by mail-ed1-x549.google.com with SMTP id l24-20020a056402231800b00410f19a3103so10611561eda.5 for ; Tue, 26 Apr 2022 09:45:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=8DiARL40b6+V/ufuvO47fdIAJou+WObIc+AUIejVSTY=; b=EUCnsUCRGBWdtTQAs7pEjigjtVlLNT5aUlwj+4ZqDn1AxMPgRR2DYXAW2EBPSLj5jq LSB/caN3RsCO5hDsl8In3F4gA8EH5xPbxv6LqVGHvoRzyZyXGYgANA68YhHsXXS+vnZ+ OcYOcvT0i08OEQfBjlycTxaFszB8WC9yK9ds1kfWnbajGrhHWXXsFXSdRLABRxG1dZSW Rhw88HYvuatIRRgwKt2yrWqq1vsLUCOLFOAIO8iKRMXZrvUjfGJ3c3X0qRnKvcsyUZMs Xb1mFmXQFpitzvw/5RfLlpk7s39iSkZsekeMIYoYzpQiwCw44UexPBJH9QY30S2Loi24 sKNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=8DiARL40b6+V/ufuvO47fdIAJou+WObIc+AUIejVSTY=; b=YjfZ5Zoxz4q7pRmFCqltws02b+DNGFuRwjxunP6GziWvGxLEzQk3RC5dYKR9JDrhJI 24QUrN91MWfZvG8h7ZuTW16Jruecp97rUOir80SU9LrHNxqmEhPhUppvbf7/FFY9fNh2 BkpuDKrXy1RMdMiYkUFWeFyfBUsIAj4LcqWkielDLa+j9/36u7znqGPvdLbUMV9sDyXO zYZnJylphglY8oUsml/zNb0yAdp5bxAmGlGbYBolLDAYQvidAJUFeM8Toe/9NSQODaU3 XTbGFHUM0b26Kmrp6ch4JwYhF+sUiwtZSrGsjJu159Z3VncPNkgbH571pFua6s53x8HG yeOA== X-Gm-Message-State: AOAM531R2Z/mmtUOaNZP6NDSVtpDXQ/2tDzr5Gtr0qws/ps8Y9vyyEdp rGOU+L8FOj+cBxLF1oqyIfwFNZiTcL8= X-Google-Smtp-Source: ABdhPJx8IprxR1s4FocdY4Bbtloh6XPKAEXkKiAjkB9XNvQcFCjh141Gp/J8iNuLq3R7S/RE8xQrIOLnKF8= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:aa7:c31a:0:b0:425:df3c:de8e with SMTP id l26-20020aa7c31a000000b00425df3cde8emr14404475edq.83.1650991517926; Tue, 26 Apr 2022 09:45:17 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:51 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-23-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 22/46] kmsan: add iomap support From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Functions from lib/iomap.c interact with hardware, so KMSAN must ensure that: - every read function returns an initialized value - every write function checks values before sending them to hardware. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/I45527599f09090aca046dfe1a26= df453adab100d --- lib/iomap.c | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/lib/iomap.c b/lib/iomap.c index fbaa3e8f19d6c..bdda1a42771b2 100644 --- a/lib/iomap.c +++ b/lib/iomap.c @@ -6,6 +6,7 @@ */ #include #include +#include =20 #include =20 @@ -70,26 +71,31 @@ static void bad_io_access(unsigned long port, const cha= r *access) #define mmio_read64be(addr) swab64(readq(addr)) #endif =20 +__no_sanitize_memory unsigned int ioread8(const void __iomem *addr) { IO_COND(addr, return inb(port), return readb(addr)); return 0xff; } +__no_sanitize_memory unsigned int ioread16(const void __iomem *addr) { IO_COND(addr, return inw(port), return readw(addr)); return 0xffff; } +__no_sanitize_memory unsigned int ioread16be(const void __iomem *addr) { IO_COND(addr, return pio_read16be(port), return mmio_read16be(addr)); return 0xffff; } +__no_sanitize_memory unsigned int ioread32(const void __iomem *addr) { IO_COND(addr, return inl(port), return readl(addr)); return 0xffffffff; } +__no_sanitize_memory unsigned int ioread32be(const void __iomem *addr) { IO_COND(addr, return pio_read32be(port), return mmio_read32be(addr)); @@ -142,18 +148,21 @@ static u64 pio_read64be_hi_lo(unsigned long port) return lo | (hi << 32); } =20 +__no_sanitize_memory u64 ioread64_lo_hi(const void __iomem *addr) { IO_COND(addr, return pio_read64_lo_hi(port), return readq(addr)); return 0xffffffffffffffffULL; } =20 +__no_sanitize_memory u64 ioread64_hi_lo(const void __iomem *addr) { IO_COND(addr, return pio_read64_hi_lo(port), return readq(addr)); return 0xffffffffffffffffULL; } =20 +__no_sanitize_memory u64 ioread64be_lo_hi(const void __iomem *addr) { IO_COND(addr, return pio_read64be_lo_hi(port), @@ -161,6 +170,7 @@ u64 ioread64be_lo_hi(const void __iomem *addr) return 0xffffffffffffffffULL; } =20 +__no_sanitize_memory u64 ioread64be_hi_lo(const void __iomem *addr) { IO_COND(addr, return pio_read64be_hi_lo(port), @@ -188,22 +198,32 @@ EXPORT_SYMBOL(ioread64be_hi_lo); =20 void iowrite8(u8 val, void __iomem *addr) { + /* Make sure uninitialized memory isn't copied to devices. */ + kmsan_check_memory(&val, sizeof(val)); IO_COND(addr, outb(val,port), writeb(val, addr)); } void iowrite16(u16 val, void __iomem *addr) { + /* Make sure uninitialized memory isn't copied to devices. */ + kmsan_check_memory(&val, sizeof(val)); IO_COND(addr, outw(val,port), writew(val, addr)); } void iowrite16be(u16 val, void __iomem *addr) { + /* Make sure uninitialized memory isn't copied to devices. */ + kmsan_check_memory(&val, sizeof(val)); IO_COND(addr, pio_write16be(val,port), mmio_write16be(val, addr)); } void iowrite32(u32 val, void __iomem *addr) { + /* Make sure uninitialized memory isn't copied to devices. */ + kmsan_check_memory(&val, sizeof(val)); IO_COND(addr, outl(val,port), writel(val, addr)); } void iowrite32be(u32 val, void __iomem *addr) { + /* Make sure uninitialized memory isn't copied to devices. */ + kmsan_check_memory(&val, sizeof(val)); IO_COND(addr, pio_write32be(val,port), mmio_write32be(val, addr)); } EXPORT_SYMBOL(iowrite8); @@ -239,24 +259,32 @@ static void pio_write64be_hi_lo(u64 val, unsigned lon= g port) =20 void iowrite64_lo_hi(u64 val, void __iomem *addr) { + /* Make sure uninitialized memory isn't copied to devices. */ + kmsan_check_memory(&val, sizeof(val)); IO_COND(addr, pio_write64_lo_hi(val, port), writeq(val, addr)); } =20 void iowrite64_hi_lo(u64 val, void __iomem *addr) { + /* Make sure uninitialized memory isn't copied to devices. */ + kmsan_check_memory(&val, sizeof(val)); IO_COND(addr, pio_write64_hi_lo(val, port), writeq(val, addr)); } =20 void iowrite64be_lo_hi(u64 val, void __iomem *addr) { + /* Make sure uninitialized memory isn't copied to devices. */ + kmsan_check_memory(&val, sizeof(val)); IO_COND(addr, pio_write64be_lo_hi(val, port), mmio_write64be(val, addr)); } =20 void iowrite64be_hi_lo(u64 val, void __iomem *addr) { + /* Make sure uninitialized memory isn't copied to devices. */ + kmsan_check_memory(&val, sizeof(val)); IO_COND(addr, pio_write64be_hi_lo(val, port), mmio_write64be(val, addr)); } @@ -328,14 +356,20 @@ static inline void mmio_outsl(void __iomem *addr, con= st u32 *src, int count) void ioread8_rep(const void __iomem *addr, void *dst, unsigned long count) { IO_COND(addr, insb(port,dst,count), mmio_insb(addr, dst, count)); + /* KMSAN must treat values read from devices as initialized. */ + kmsan_unpoison_memory(dst, count); } void ioread16_rep(const void __iomem *addr, void *dst, unsigned long count) { IO_COND(addr, insw(port,dst,count), mmio_insw(addr, dst, count)); + /* KMSAN must treat values read from devices as initialized. */ + kmsan_unpoison_memory(dst, count * 2); } void ioread32_rep(const void __iomem *addr, void *dst, unsigned long count) { IO_COND(addr, insl(port,dst,count), mmio_insl(addr, dst, count)); + /* KMSAN must treat values read from devices as initialized. */ + kmsan_unpoison_memory(dst, count * 4); } EXPORT_SYMBOL(ioread8_rep); EXPORT_SYMBOL(ioread16_rep); @@ -343,14 +377,20 @@ EXPORT_SYMBOL(ioread32_rep); =20 void iowrite8_rep(void __iomem *addr, const void *src, unsigned long count) { + /* Make sure uninitialized memory isn't copied to devices. */ + kmsan_check_memory(src, count); IO_COND(addr, outsb(port, src, count), mmio_outsb(addr, src, count)); } void iowrite16_rep(void __iomem *addr, const void *src, unsigned long coun= t) { + /* Make sure uninitialized memory isn't copied to devices. */ + kmsan_check_memory(src, count * 2); IO_COND(addr, outsw(port, src, count), mmio_outsw(addr, src, count)); } void iowrite32_rep(void __iomem *addr, const void *src, unsigned long coun= t) { + /* Make sure uninitialized memory isn't copied to devices. */ + kmsan_check_memory(src, count * 4); IO_COND(addr, outsl(port, src,count), mmio_outsl(addr, src, count)); } EXPORT_SYMBOL(iowrite8_rep); --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68456C433EF for ; Tue, 26 Apr 2022 16:48:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242512AbiDZQvL (ORCPT ); Tue, 26 Apr 2022 12:51:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43200 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353296AbiDZQtL (ORCPT ); Tue, 26 Apr 2022 12:49:11 -0400 Received: from mail-ej1-x649.google.com (mail-ej1-x649.google.com [IPv6:2a00:1450:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 11C6E3BBFD for ; Tue, 26 Apr 2022 09:45:22 -0700 (PDT) Received: by mail-ej1-x649.google.com with SMTP id i14-20020a17090639ce00b006dabe6a112fso9272160eje.13 for ; Tue, 26 Apr 2022 09:45:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=A4BOD5bYYCwqT6UysGwaTqw9NZkJge0TWQm9U6+WrG0=; b=khSFQmZPrfoPgxGEUABi/FVW8V9rw4+gcS76epQACJKqyg7hkcjwrrNZSEPqT8cKN9 ONCpTChsOEZ0awUUpbA5x5xayPRdehOMzeg0fmzK8U1O+otCttBP994AhBNpWeRijLhC uA0BFEVHSShFszO60gnvZ/g8XEEAvlCsHxSsQtTYtMUoAiTqbiLeIdThYaket6Dbkru7 YLDTtMsAFMm7WODYFRUO4LbdZSTFrmTAN4yb5OHT1IcBSYVtVTBthQ8DYmMN9xjdT8Y5 P4oaK9oxiUba69g6lk7NgMT+HmLDsQRoGLtP+vFJRnamxCbKjs/0LNdi6KunA8g68VkO EfjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=A4BOD5bYYCwqT6UysGwaTqw9NZkJge0TWQm9U6+WrG0=; b=o/6EEklGNHs3UoKQ5NpmvZiPHUr3KEVMdjsBbah/HagI+mN2RFrszMCasuVfcWxcW2 Ery03/Lu80GXuRqVxpIwLfS2y8lgnoTxsIwXMkan9sDCIHlSnyZ+PVljZHNmSImmej3l 4Ph1YcO5If15uAKnbRcO7bTbfJDajrebAii/yz1rzM2jFt1oVMJhqONhUX1RxvVL7lbe FipAiEScCQbzFOojAjAqIsDKmTruabtl85Q5vBhCcZQHHzmGLdZlw+4Tf0K059300iE9 12gl8SUuo3EsTQUjVwdXw6I/Sdjw8qLktahq2a1qgZIvGOmiPI6vr66ixC62UqU5QGjD eIJw== X-Gm-Message-State: AOAM533inHhX1GuZ5PodpRXlwmJZVu9+0Plit4YJa5+pxFlOsz66pYp3 wOMCvkZb/v0/gd/siYoNqC3p/m9NXbE= X-Google-Smtp-Source: ABdhPJxAop9pzi2nkfiGiO4x6mNOPP93/Bjni9kA40EEqg0TirQbaSSuyCXI2m0QfvecUipohdkmvzvyeto= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:254e:b0:424:244:faf with SMTP id l14-20020a056402254e00b0042402440fafmr25661420edb.260.1650991520418; Tue, 26 Apr 2022 09:45:20 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:52 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-24-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 23/46] Input: libps2: mark data received in __ps2_command() as initialized From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" KMSAN does not know that the device initializes certain bytes in ps2dev->cmdbuf. Call kmsan_unpoison_memory() to explicitly mark them as initialized. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/I2d26f6baa45271d37320d3f4a52= 8c39cb7e545f0 --- drivers/input/serio/libps2.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/input/serio/libps2.c b/drivers/input/serio/libps2.c index 250e213cc80c6..3e19344eda93c 100644 --- a/drivers/input/serio/libps2.c +++ b/drivers/input/serio/libps2.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -294,9 +295,11 @@ int __ps2_command(struct ps2dev *ps2dev, u8 *param, un= signed int command) =20 serio_pause_rx(ps2dev->serio); =20 - if (param) + if (param) { for (i =3D 0; i < receive; i++) param[i] =3D ps2dev->cmdbuf[(receive - 1) - i]; + kmsan_unpoison_memory(param, receive); + } =20 if (ps2dev->cmdcnt && (command !=3D PS2_CMD_RESET_BAT || ps2dev->cmdcnt !=3D 1)) { --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15428C433EF for ; Tue, 26 Apr 2022 16:48:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345322AbiDZQvH (ORCPT ); Tue, 26 Apr 2022 12:51:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44228 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353304AbiDZQtL (ORCPT ); Tue, 26 Apr 2022 12:49:11 -0400 Received: from mail-ed1-x54a.google.com (mail-ed1-x54a.google.com [IPv6:2a00:1450:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C04AA3C737 for ; Tue, 26 Apr 2022 09:45:24 -0700 (PDT) Received: by mail-ed1-x54a.google.com with SMTP id h7-20020a056402094700b00425a52983dfso8255694edz.8 for ; Tue, 26 Apr 2022 09:45:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=ae6KBGyLKJoXcxB2yOctfhYiWJk7lyt59+0aGNQoZFE=; b=kru0e6mwMpQdFsNTXlAfmTpSGH0ko9Z62aiquxpVYOR0JXp5EtPpYUXKeC/mmsN9Uv Mp3CAzX6GOJEixW0dclu6CRFgdkOwz2UImg1qNv1NPxGTD3yEIxR2rgv8ej3AjUD5RT0 1+4QmZncDejfeevrwV59cqIXS/SsyVIM1YxmMil8RxMiL6nOjlfyWrun6JkjBe06L8gX yPe6oH2Oe0cNHqZUNa5jyVwSkEa/2QoaQq9AU6Rl3YYAf/EnVNPhfP0bZRfTEIERvMVW XKOQ0Si/wyaghQgFEr588BSzsT8VRUjOfovii7JYjgVO78nBwq8Bg/+CpEfFdTD5/Zo5 GoAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ae6KBGyLKJoXcxB2yOctfhYiWJk7lyt59+0aGNQoZFE=; b=gL7Vwv3vUkd5GpvplZqYu1iPLQQFH/bSiHziUUOumScsoOfWrtqVSBvoG8j0f+Eou6 rMpp73O99sUfKuwdlXisAfFmjuCwFGI0/1daVI+SQwp2gkb+R0fl82bAsSTPdcGn0Xao bVb6czcfFwqWEurYWaAUnYRfM07zV9eTatEugdtY11ViZiSwwZZ5biI39yaBKGIPKw4o bHpS4b8otpSgcnzkxI3jatTelB1OQj0jBiXXcuXK97FgVG65iOPk8DKn3Y60RBF1CQw/ isDlR+evcgU+cYlaEZmtm4yawX3oIR7YwB18vM9jHvTJQVpvHNUrF2wf9ut3WQz1clOZ sJeA== X-Gm-Message-State: AOAM533zSpfk4hxcp2A2lfyZRdS2Rz6h1ZxloIa7K9PaudyVmr6yCqB8 QjkcistIOR8H9WVYAusJyUmFpQRjCi4= X-Google-Smtp-Source: ABdhPJwgCnNxvqRMEZQ8KvQ5jOB/J8AlGoWTaRunP9zD2mXvGus8OSk8rSp8TRR9z2UMTYCCAqyWXZmoFZQ= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:274b:b0:423:fe73:95a0 with SMTP id z11-20020a056402274b00b00423fe7395a0mr25532792edd.224.1650991523134; Tue, 26 Apr 2022 09:45:23 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:53 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-25-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 24/46] kmsan: dma: unpoison DMA mappings From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" KMSAN doesn't know about DMA memory writes performed by devices. We unpoison such memory when it's mapped to avoid false positive reports. Signed-off-by: Alexander Potapenko --- v2: -- move implementation of kmsan_handle_dma() and kmsan_handle_dma_sg() here Link: https://linux-review.googlesource.com/id/Ia162dc4c5a92e74d4686c1be32a= 4dfeffc5c32cd --- include/linux/kmsan.h | 41 +++++++++++++++++++++++++++++ kernel/dma/mapping.c | 9 ++++--- mm/kmsan/hooks.c | 61 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 108 insertions(+), 3 deletions(-) diff --git a/include/linux/kmsan.h b/include/linux/kmsan.h index a5767c728a46b..d8667161a10c8 100644 --- a/include/linux/kmsan.h +++ b/include/linux/kmsan.h @@ -9,6 +9,7 @@ #ifndef _LINUX_KMSAN_H #define _LINUX_KMSAN_H =20 +#include #include #include #include @@ -18,6 +19,7 @@ struct page; struct kmem_cache; struct task_struct; +struct scatterlist; =20 #ifdef CONFIG_KMSAN =20 @@ -205,6 +207,35 @@ void kmsan_ioremap_page_range(unsigned long addr, unsi= gned long end, */ void kmsan_iounmap_page_range(unsigned long start, unsigned long end); =20 +/** + * kmsan_handle_dma() - Handle a DMA data transfer. + * @page: first page of the buffer. + * @offset: offset of the buffer within the first page. + * @size: buffer size. + * @dir: one of possible dma_data_direction values. + * + * Depending on @direction, KMSAN: + * * checks the buffer, if it is copied to device; + * * initializes the buffer, if it is copied from device; + * * does both, if this is a DMA_BIDIRECTIONAL transfer. + */ +void kmsan_handle_dma(struct page *page, size_t offset, size_t size, + enum dma_data_direction dir); + +/** + * kmsan_handle_dma_sg() - Handle a DMA transfer using scatterlist. + * @sg: scatterlist holding DMA buffers. + * @nents: number of scatterlist entries. + * @dir: one of possible dma_data_direction values. + * + * Depending on @direction, KMSAN: + * * checks the buffers in the scatterlist, if they are copied to device; + * * initializes the buffers, if they are copied from device; + * * does both, if this is a DMA_BIDIRECTIONAL transfer. + */ +void kmsan_handle_dma_sg(struct scatterlist *sg, int nents, + enum dma_data_direction dir); + #else =20 static inline void kmsan_init_shadow(void) @@ -287,6 +318,16 @@ static inline void kmsan_iounmap_page_range(unsigned l= ong start, { } =20 +static inline void kmsan_handle_dma(struct page *page, size_t offset, + size_t size, enum dma_data_direction dir) +{ +} + +static inline void kmsan_handle_dma_sg(struct scatterlist *sg, int nents, + enum dma_data_direction dir) +{ +} + #endif =20 #endif /* _LINUX_KMSAN_H */ diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index db7244291b745..5d17d5d62166b 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -156,6 +156,7 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struc= t page *page, addr =3D dma_direct_map_page(dev, page, offset, size, dir, attrs); else addr =3D ops->map_page(dev, page, offset, size, dir, attrs); + kmsan_handle_dma(page, offset, size, dir); debug_dma_map_page(dev, page, offset, size, dir, addr, attrs); =20 return addr; @@ -194,11 +195,13 @@ static int __dma_map_sg_attrs(struct device *dev, str= uct scatterlist *sg, else ents =3D ops->map_sg(dev, sg, nents, dir, attrs); =20 - if (ents > 0) + if (ents > 0) { + kmsan_handle_dma_sg(sg, nents, dir); debug_dma_map_sg(dev, sg, nents, ents, dir, attrs); - else if (WARN_ON_ONCE(ents !=3D -EINVAL && ents !=3D -ENOMEM && - ents !=3D -EIO)) + } else if (WARN_ON_ONCE(ents !=3D -EINVAL && ents !=3D -ENOMEM && + ents !=3D -EIO)) { return -EIO; + } =20 return ents; } diff --git a/mm/kmsan/hooks.c b/mm/kmsan/hooks.c index 1cdb4420977f1..8a6947a2a2f22 100644 --- a/mm/kmsan/hooks.c +++ b/mm/kmsan/hooks.c @@ -10,9 +10,11 @@ */ =20 #include +#include #include #include #include +#include #include #include =20 @@ -250,6 +252,65 @@ void kmsan_copy_to_user(void __user *to, const void *f= rom, size_t to_copy, } EXPORT_SYMBOL(kmsan_copy_to_user); =20 +static void kmsan_handle_dma_page(const void *addr, size_t size, + enum dma_data_direction dir) +{ + switch (dir) { + case DMA_BIDIRECTIONAL: + kmsan_internal_check_memory((void *)addr, size, /*user_addr*/ 0, + REASON_ANY); + kmsan_internal_unpoison_memory((void *)addr, size, + /*checked*/ false); + break; + case DMA_TO_DEVICE: + kmsan_internal_check_memory((void *)addr, size, /*user_addr*/ 0, + REASON_ANY); + break; + case DMA_FROM_DEVICE: + kmsan_internal_unpoison_memory((void *)addr, size, + /*checked*/ false); + break; + case DMA_NONE: + break; + } +} + +/* Helper function to handle DMA data transfers. */ +void kmsan_handle_dma(struct page *page, size_t offset, size_t size, + enum dma_data_direction dir) +{ + u64 page_offset, to_go, addr; + + if (PageHighMem(page)) + return; + addr =3D (u64)page_address(page) + offset; + /* + * The kernel may occasionally give us adjacent DMA pages not belonging + * to the same allocation. Process them separately to avoid triggering + * internal KMSAN checks. + */ + while (size > 0) { + page_offset =3D addr % PAGE_SIZE; + to_go =3D min(PAGE_SIZE - page_offset, (u64)size); + kmsan_handle_dma_page((void *)addr, to_go, dir); + addr +=3D to_go; + size -=3D to_go; + } +} +EXPORT_SYMBOL(kmsan_handle_dma); + +void kmsan_handle_dma_sg(struct scatterlist *sg, int nents, + enum dma_data_direction dir) +{ + struct scatterlist *item; + int i; + + for_each_sg(sg, item, nents, i) + kmsan_handle_dma(sg_page(item), item->offset, item->length, + dir); +} +EXPORT_SYMBOL(kmsan_handle_dma_sg); + /* Functions from kmsan-checks.h follow. */ void kmsan_poison_memory(const void *address, size_t size, gfp_t flags) { --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57657C4332F for ; Tue, 26 Apr 2022 16:47:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353305AbiDZQvC (ORCPT ); Tue, 26 Apr 2022 12:51:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43278 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353303AbiDZQtL (ORCPT ); Tue, 26 Apr 2022 12:49:11 -0400 Received: from mail-ej1-x649.google.com (mail-ej1-x649.google.com [IPv6:2a00:1450:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5BB663D4AC for ; Tue, 26 Apr 2022 09:45:27 -0700 (PDT) Received: by mail-ej1-x649.google.com with SMTP id dp12-20020a170906c14c00b006e7e8234ae2so9350439ejc.2 for ; Tue, 26 Apr 2022 09:45:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=n12oscJoQy25ZeJEgKUppRaHioaFclURl8PsqxmYCmo=; b=PIyBhpOCM8Ov1/+Aan+YSMRBfpf54pHX8ytL+CV9/1wxPrduFClfDo2MC25BFRGqUr LC5Hr2Ef8nJt4G1Cr30fbnTHYjTJ5Gax8GaKjFXG0iL+r/c29mApiZT+RSwu1Gv2kkOz 34uVLpDsIsKTbG2q42gMn4a9Vr9GOVlaxiLfXa8b/4pWebY6TjT2x9JFCSEXRuOPPg3n b/OGc6pwoUP/SU7+PJ629KnVXCQHZctVc5m6qyPQPfIcXk7XKqS9AYp6qyxiblVFksU/ vePEsS5V3b8GJxiZEtgsjh4z7hZSSCwDY93JM9PnMnCLZhZrlsTx6bFvrt7KV12fLlqe re3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=n12oscJoQy25ZeJEgKUppRaHioaFclURl8PsqxmYCmo=; b=eDfHR9etxf7+bT1TNJL/Wk64k90nQKooUpqUmkxiaBJsKFqV7y3J077oZHDAKAZ/ok DAEGgB4eAveYLNC+DqT4fjz13O5cbfEUg+UDWRusF63BkiHQxO2OfhF/YkJDtgmPjQGY 80ILieXfEAldpzrrYmUl5LSAMuVmNkXkp0HmUMVM66kP9yeGngZ+HSVaciAhqmGakNSm s7xTQ87/Kg+SJTm2jCQc1ii6aAvaHxix+chtHPKI6Xpt6Dg0ho5dVoP2Rts82PuhZKxP uAYLlVfSOhOSvl2v96FxXSwHQY9tmEb4gOa4q9o/I1NDd9Y2dzYaWNNS0YsCIZKXOjGf WuAA== X-Gm-Message-State: AOAM5316n9sW0EtgukyUSrKbWUmFrUagP2J8JPfH7EHGjfOcpT9QrymI oXPFx68L+tefmQfsUNMMNbO46V1arHg= X-Google-Smtp-Source: ABdhPJxU8JwPhcVfaOXEOS2U2ep3mvLwmmcZHEz36qAom9R3kEeJAH/9Z6s5Sw7k1Wv/sYn9qCbZoM2w78E= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:5210:b0:423:de77:2a4d with SMTP id s16-20020a056402521000b00423de772a4dmr25186177edd.295.1650991525861; Tue, 26 Apr 2022 09:45:25 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:54 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-26-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 25/46] kmsan: virtio: check/unpoison scatterlist in vring_map_one_sg() From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If vring doesn't use the DMA API, KMSAN is unable to tell whether the memory is initialized by hardware. Explicitly call kmsan_handle_dma() from vring_map_one_sg() in this case to prevent false positives. Signed-off-by: Alexander Potapenko Acked-by: Michael S. Tsirkin --- Link: https://linux-review.googlesource.com/id/I211533ecb86a66624e151551f83= ddd749536b3af --- drivers/virtio/virtio_ring.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index cfb028ca238eb..faecd9e3d6560 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include =20 @@ -331,8 +332,15 @@ static dma_addr_t vring_map_one_sg(const struct vring_= virtqueue *vq, struct scatterlist *sg, enum dma_data_direction direction) { - if (!vq->use_dma_api) + if (!vq->use_dma_api) { + /* + * If DMA is not used, KMSAN doesn't know that the scatterlist + * is initialized by the hardware. Explicitly check/unpoison it + * depending on the direction. + */ + kmsan_handle_dma(sg_page(sg), sg->offset, sg->length, direction); return (dma_addr_t)sg_phys(sg); + } =20 /* * We can't use dma_map_sg, because we don't use scatterlists in --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BBE87C433EF for ; Tue, 26 Apr 2022 16:48:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353376AbiDZQvP (ORCPT ); Tue, 26 Apr 2022 12:51:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46888 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353268AbiDZQtd (ORCPT ); Tue, 26 Apr 2022 12:49:33 -0400 Received: from mail-ej1-x64a.google.com (mail-ej1-x64a.google.com [IPv6:2a00:1450:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D683311C26 for ; Tue, 26 Apr 2022 09:45:29 -0700 (PDT) Received: by mail-ej1-x64a.google.com with SMTP id nb10-20020a1709071c8a00b006e8f89863ceso9377303ejc.18 for ; Tue, 26 Apr 2022 09:45:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=kXck7gc8vUdG+uIpTKbHiyCLJU3V9DKcF3fYel7nQCw=; b=HKjl4po9qikfY5w7sANNOWF8wIYoTZhGhPfGpG2oRS5qaieTXeddZbMui9oEk30REF GfljSR8S1QIxGgAiHnxwjgEv6+EF2+8jJ7T7QZK2oSf8eJtoGjdxW0lf0sHwGI/lNoOh KSwqI4UvdZtVfWHWxZE/JKd/uRDNjbPmLMNeGT3iF7SCfBSOExZCHwBizTIx/FZ301Kf goPWSOE0bCzYKGYIAeB4b4w+/WYrsLZPjYvsQGeev90WIWm90jQbop08KYN+ktuukkjW eqLwPo+Y1By7UOL4GBv0clQvJfdJAgsm6ae+0ok9ios/vlSXb1rGqpnveoFRq063DiYQ GxtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=kXck7gc8vUdG+uIpTKbHiyCLJU3V9DKcF3fYel7nQCw=; b=qcNRpVIxnjW0uw8KhtoGP4cpinue1KBnCvGwOsdnLheU7bRtq4Pp0/IoumF1JlcZqd FekEX0QWJUKquIU20oKey/mIxBl1W3JwJ6CqtmfN+9cwCZkcvlpzXf61qwK2T4UB7mey 50vdMagvR1lNSqDd0Tmsstdhu7J4zyh6DIjf225Qg9Fyx6WxbhFCmVor1FGgNbnFOE/k CoQPnUbft1CDyQmhSfJgGp7r4IawmhfLEWPLVKGc/Vyggde/o0qIq6WBkeRinCLY9Vbt ECPweSDnJ8bKToiy6AQ9h2q5uztpZqZEnw2qJRgKtFIlNlyk102wDW8xcd+/N8lLuwRh vRXw== X-Gm-Message-State: AOAM531o7y8Ei8Ku2gWmtFRktsotpHwW8znuUuhUy/sMqZwg38JCy5W1 eHXHJvlTLcjavkAfWvXPR7VzFqUrNm4= X-Google-Smtp-Source: ABdhPJz2TfzAbBIJqBzng2cj5zaotZzRLvL6/wlz3cGmkr5GPkO9jG7+a46FVlyYEs+vMIVeqbruPJFqxPk= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a17:907:1b15:b0:6d7:13bd:dd62 with SMTP id mp21-20020a1709071b1500b006d713bddd62mr21844408ejc.673.1650991528244; Tue, 26 Apr 2022 09:45:28 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:55 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-27-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 26/46] kmsan: handle memory sent to/from USB From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Depending on the value of is_out kmsan_handle_urb() KMSAN either marks the data copied to the kernel from a USB device as initialized, or checks the data sent to the device for being initialized. Signed-off-by: Alexander Potapenko --- v2: -- move kmsan_handle_urb() implementation to this patch Link: https://linux-review.googlesource.com/id/Ifa67fb72015d4de14c30e971556= f99fc8b2ee506 --- drivers/usb/core/urb.c | 2 ++ include/linux/kmsan.h | 15 +++++++++++++++ mm/kmsan/hooks.c | 17 +++++++++++++++++ 3 files changed, 34 insertions(+) diff --git a/drivers/usb/core/urb.c b/drivers/usb/core/urb.c index 33d62d7e3929f..1fe3f23205624 100644 --- a/drivers/usb/core/urb.c +++ b/drivers/usb/core/urb.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -426,6 +427,7 @@ int usb_submit_urb(struct urb *urb, gfp_t mem_flags) URB_SETUP_MAP_SINGLE | URB_SETUP_MAP_LOCAL | URB_DMA_SG_COMBINED); urb->transfer_flags |=3D (is_out ? URB_DIR_OUT : URB_DIR_IN); + kmsan_handle_urb(urb, is_out); =20 if (xfertype !=3D USB_ENDPOINT_XFER_CONTROL && dev->state < USB_STATE_CONFIGURED) diff --git a/include/linux/kmsan.h b/include/linux/kmsan.h index d8667161a10c8..55f976b721566 100644 --- a/include/linux/kmsan.h +++ b/include/linux/kmsan.h @@ -20,6 +20,7 @@ struct page; struct kmem_cache; struct task_struct; struct scatterlist; +struct urb; =20 #ifdef CONFIG_KMSAN =20 @@ -236,6 +237,16 @@ void kmsan_handle_dma(struct page *page, size_t offset= , size_t size, void kmsan_handle_dma_sg(struct scatterlist *sg, int nents, enum dma_data_direction dir); =20 +/** + * kmsan_handle_urb() - Handle a USB data transfer. + * @urb: struct urb pointer. + * @is_out: data transfer direction (true means output to hardware). + * + * If @is_out is true, KMSAN checks the transfer buffer of @urb. Otherwise, + * KMSAN initializes the transfer buffer. + */ +void kmsan_handle_urb(const struct urb *urb, bool is_out); + #else =20 static inline void kmsan_init_shadow(void) @@ -328,6 +339,10 @@ static inline void kmsan_handle_dma_sg(struct scatterl= ist *sg, int nents, { } =20 +static inline void kmsan_handle_urb(const struct urb *urb, bool is_out) +{ +} + #endif =20 #endif /* _LINUX_KMSAN_H */ diff --git a/mm/kmsan/hooks.c b/mm/kmsan/hooks.c index 8a6947a2a2f22..9aecbf2825837 100644 --- a/mm/kmsan/hooks.c +++ b/mm/kmsan/hooks.c @@ -17,6 +17,7 @@ #include #include #include +#include =20 #include "../internal.h" #include "../slab.h" @@ -252,6 +253,22 @@ void kmsan_copy_to_user(void __user *to, const void *f= rom, size_t to_copy, } EXPORT_SYMBOL(kmsan_copy_to_user); =20 +/* Helper function to check an URB. */ +void kmsan_handle_urb(const struct urb *urb, bool is_out) +{ + if (!urb) + return; + if (is_out) + kmsan_internal_check_memory(urb->transfer_buffer, + urb->transfer_buffer_length, + /*user_addr*/ 0, REASON_SUBMIT_URB); + else + kmsan_internal_unpoison_memory(urb->transfer_buffer, + urb->transfer_buffer_length, + /*checked*/ false); +} +EXPORT_SYMBOL(kmsan_handle_urb); + static void kmsan_handle_dma_page(const void *addr, size_t size, enum dma_data_direction dir) { --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3E49C4332F for ; Tue, 26 Apr 2022 16:48:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350873AbiDZQv2 (ORCPT ); Tue, 26 Apr 2022 12:51:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43286 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353251AbiDZQtl (ORCPT ); Tue, 26 Apr 2022 12:49:41 -0400 Received: from mail-ed1-x549.google.com (mail-ed1-x549.google.com [IPv6:2a00:1450:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 720333EB8B for ; Tue, 26 Apr 2022 09:45:32 -0700 (PDT) Received: by mail-ed1-x549.google.com with SMTP id ee56-20020a056402293800b00425b0f5b9c6so7562125edb.9 for ; Tue, 26 Apr 2022 09:45:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=8+FTymzEXuQwrQ48VSTvyp6MK3t+ENPCVe5WmeEaVIc=; b=fcJ50fxU8vlV4Zt4ow5QLp80TC8c56IdxjhwU67LEgyBqe8j9VHcHGiTXWvty1WMSf 1bPBfjkXUa51JG42+wVa+w3aHgsOYSrC1f9U8SZ2jz031VP/HC88wQw4NfXbVOoGdtcl SS8jFJBwA5MYs/dZR6sjMvgZtOfXiYQhrCsRhuTSkWp5gWJqFxoT0Go1VP32lUvN66NF QMYm/Pv7rIuLPqSnr2MRY4/KkU+A6liqUHe5KRLS7lCFdZ3JcG6G4DdynX7jifdqwnNt rgjoa8UfdHnj1e0EoDPPBI7pDd0IfYihgK9kALl0k/qSo8xllk+StsshEgAnxbfsnzPI nRkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=8+FTymzEXuQwrQ48VSTvyp6MK3t+ENPCVe5WmeEaVIc=; b=N93IEr5CMKcp6N1eF2aXQTs4/nHx41WFefi0u6bEmc9ks8BJh0eYKoL1lLW86guuFu aMPcs6ymkIv0HzMidMFoP2A1pwaaJ72LpMkpl9Szcc5tFeY8014fybkeg4O3IUgaO9Ye NME3S1yvi3ZrWOBtQTCw+wrN26eHqAW+gQbSGVu+0yPEeanMhggvAVX7OEgDzqoitThk A5bDeq2OeTDVezRd+Gf+8AiFGdvTAfPeQcaxBEiCdOxLhwwXgOsceAeMvCxbaJkRC9iU KYZkI7L+TbY2d/ksEny0LgjKZJ9QzWIxELc5r8W2hAGwThvO2qBttbQVVXfiKEoQO61d mfMw== X-Gm-Message-State: AOAM533tzFRZWoJUXPyBU99D0h1bwrwtNoOo7inl06nKz7FixSzJ5ZW3 VR918PGlQ2GiBvgUZyElvfcGsFqO9SM= X-Google-Smtp-Source: ABdhPJzMNlQ7wgMMMKP2+ozorBxditMXEaVfv9SlYu5XvLbQQ4QNVo7xn2ciiuqzzcwq86sHWkzQRQWNmV4= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a50:eb87:0:b0:425:c3e2:17a9 with SMTP id y7-20020a50eb87000000b00425c3e217a9mr22640245edr.109.1650991530577; Tue, 26 Apr 2022 09:45:30 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:56 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-28-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 27/46] kmsan: instrumentation.h: add instrumentation_begin_with_regs() From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When calling KMSAN-instrumented functions from non-instrumented functions, function parameters may not be initialized properly, leading to false positive reports. In particular, this happens all the time when calling interrupt handlers from `noinstr` IDT entries. We introduce instrumentation_begin_with_regs(), which calls instrumentation_begin() and notifies KMSAN about the beginning of the potentially instrumented region by calling kmsan_instrumentation_begin(), which: - wipes the current KMSAN state at the beginning of the region, ensuring that the first call of an instrumented function receives initialized parameters (this is a pretty good approximation of having all other instrumented functions receive initialized parameters); - unpoisons the `struct pt_regs` set up by the non-instrumented assembly code. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/I0f5e3372e00bd5fe25ddbf286f7= 260aae9011858 --- include/linux/instrumentation.h | 6 ++++++ include/linux/kmsan.h | 11 +++++++++++ mm/kmsan/hooks.c | 16 ++++++++++++++++ 3 files changed, 33 insertions(+) diff --git a/include/linux/instrumentation.h b/include/linux/instrumentatio= n.h index 24359b4a96053..3bbce9d556381 100644 --- a/include/linux/instrumentation.h +++ b/include/linux/instrumentation.h @@ -15,6 +15,11 @@ }) #define instrumentation_begin() __instrumentation_begin(__COUNTER__) =20 +#define instrumentation_begin_with_regs(regs) do { \ + __instrumentation_begin(__COUNTER__); \ + kmsan_instrumentation_begin(regs); \ +} while (0) + /* * Because instrumentation_{begin,end}() can nest, objtool validation cons= iders * _begin() a +1 and _end() a -1 and computes a sum over the instructions. @@ -55,6 +60,7 @@ #define instrumentation_end() __instrumentation_end(__COUNTER__) #else # define instrumentation_begin() do { } while(0) +# define instrumentation_begin_with_regs(regs) kmsan_instrumentation_begin= (regs) # define instrumentation_end() do { } while(0) #endif =20 diff --git a/include/linux/kmsan.h b/include/linux/kmsan.h index 55f976b721566..209a5a2192e22 100644 --- a/include/linux/kmsan.h +++ b/include/linux/kmsan.h @@ -247,6 +247,13 @@ void kmsan_handle_dma_sg(struct scatterlist *sg, int n= ents, */ void kmsan_handle_urb(const struct urb *urb, bool is_out); =20 +/** + * kmsan_instrumentation_begin() - handle instrumentation_begin(). + * @regs: pointer to struct pt_regs that non-instrumented code passes to + * instrumented code. + */ +void kmsan_instrumentation_begin(struct pt_regs *regs); + #else =20 static inline void kmsan_init_shadow(void) @@ -343,6 +350,10 @@ static inline void kmsan_handle_urb(const struct urb *= urb, bool is_out) { } =20 +static inline void kmsan_instrumentation_begin(struct pt_regs *regs) +{ +} + #endif =20 #endif /* _LINUX_KMSAN_H */ diff --git a/mm/kmsan/hooks.c b/mm/kmsan/hooks.c index 9aecbf2825837..c20d105c143c1 100644 --- a/mm/kmsan/hooks.c +++ b/mm/kmsan/hooks.c @@ -366,3 +366,19 @@ void kmsan_check_memory(const void *addr, size_t size) REASON_ANY); } EXPORT_SYMBOL(kmsan_check_memory); + +void kmsan_instrumentation_begin(struct pt_regs *regs) +{ + struct kmsan_context_state *state =3D &kmsan_get_context()->cstate; + + if (state) + __memset(state, 0, sizeof(struct kmsan_context_state)); + if (!kmsan_enabled || !regs) + return; + /* + * @regs may reside in cpu_entry_area, for which KMSAN does not allocate + * metadata. Do not force an error in that case. + */ + kmsan_internal_unpoison_memory(regs, sizeof(*regs), /*checked*/ false); +} +EXPORT_SYMBOL(kmsan_instrumentation_begin); --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81F05C4332F for ; Tue, 26 Apr 2022 16:48:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353423AbiDZQvc (ORCPT ); Tue, 26 Apr 2022 12:51:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43226 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236211AbiDZQuQ (ORCPT ); Tue, 26 Apr 2022 12:50:16 -0400 Received: from mail-ed1-x54a.google.com (mail-ed1-x54a.google.com [IPv6:2a00:1450:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0512E3F89A for ; Tue, 26 Apr 2022 09:45:34 -0700 (PDT) Received: by mail-ed1-x54a.google.com with SMTP id ee56-20020a056402293800b00425b0f5b9c6so7562195edb.9 for ; Tue, 26 Apr 2022 09:45:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=7rWXomHmmGtw2qSW/Jql3Xky6Nc8Jl3o+xpl6HNe0X8=; b=q5MasDLtRuk73svjnlxTla0qBFDkgnt0PsXrVkFMN41i4z8Uc7BjsHa4IZlwyI4mOh PSjNtr6oT4aSfGpSW0YXGpJKMOZ3sk1QnOzt4PQcjsP2frb+KqFrSVrmWcDHhZkns/d/ pdEqJGCSd8xeQiS1/OQIYoiHjeol8c3Q+yW+/B+Knn6wOUS02dBp111b/1mBG/LNHKLX o57oj7KW5sVZwCE68ycL0SdOrbqmApIoCkh8dUgH6H+ygVwXbl7Gp7mJ6YMwZNc44jBv kf7eWkFxMSiWJtqo7427dYj43Sgw5gav9YAFNr+RIt6rHBJ0ZfsALEQce0T6+nWFX4OZ E6gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=7rWXomHmmGtw2qSW/Jql3Xky6Nc8Jl3o+xpl6HNe0X8=; b=DjKyxpK5i7NS9QvVCozV4MRIw57kJbsvvfFo8G5qcMGOYqIzFdCkFZsekjj4gN02xx tGDKECZo+09CrnXeil7l3MumegM6RVWnoLqQRlsNQFM20ymejZYyHTdfV1ZeX7qCcTGU Up13H/Ih6WMqmA1gJ9FwPyjW3Ho4TZtvps5LkP8zjAeXfa5VG5XDFECpeumZLjJa9VHP 8fDCxOD3SRL3j4iVMtKBBpg9f0dR6C1jNWR6LOf9Yf3HTXjgawB9NeW42+4zIimBJTrL 17Xmzjb3A5JkUHq+UGrrtCVhYvvgpimCqKmucFNtQuOlPNXIpBDp25itInbRYJaOnkgz +/3Q== X-Gm-Message-State: AOAM531SxfNRsKj8mfomflHVlJJNy+qiIr7SynnsFTqrVzhNx1Mi0Lfl dkirvqvxvXl6e5bXJmolvyhVGYoSxS4= X-Google-Smtp-Source: ABdhPJyHNp1hORgE0TcahA4SesrvDuNa9ln2vyqb/yiKW2MgJrSo3ZS2uZRSG/oHoxHKcWZIpPWb90g9a0g= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:27d1:b0:425:f92f:aac0 with SMTP id c17-20020a05640227d100b00425f92faac0mr5194069ede.409.1650991533278; Tue, 26 Apr 2022 09:45:33 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:57 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-29-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 28/46] kmsan: entry: handle register passing from uninstrumented code From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Replace instrumentation_begin() with instrumentation_begin_with_regs() to let KMSAN handle the non-instrumented code and unpoison pt_regs passed from the instrumented part. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/I7f0a9809b66bd85faae43142971= d0095771b7a42 --- kernel/entry/common.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/kernel/entry/common.c b/kernel/entry/common.c index 93c3b86e781c1..ce2324374882c 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -23,7 +23,7 @@ static __always_inline void __enter_from_user_mode(struct= pt_regs *regs) CT_WARN_ON(ct_state() !=3D CONTEXT_USER); user_exit_irqoff(); =20 - instrumentation_begin(); + instrumentation_begin_with_regs(regs); trace_hardirqs_off_finish(); instrumentation_end(); } @@ -105,7 +105,7 @@ noinstr long syscall_enter_from_user_mode(struct pt_reg= s *regs, long syscall) =20 __enter_from_user_mode(regs); =20 - instrumentation_begin(); + instrumentation_begin_with_regs(regs); local_irq_enable(); ret =3D __syscall_enter_from_user_work(regs, syscall); instrumentation_end(); @@ -116,7 +116,7 @@ noinstr long syscall_enter_from_user_mode(struct pt_reg= s *regs, long syscall) noinstr void syscall_enter_from_user_mode_prepare(struct pt_regs *regs) { __enter_from_user_mode(regs); - instrumentation_begin(); + instrumentation_begin_with_regs(regs); local_irq_enable(); instrumentation_end(); } @@ -290,7 +290,7 @@ void syscall_exit_to_user_mode_work(struct pt_regs *reg= s) =20 __visible noinstr void syscall_exit_to_user_mode(struct pt_regs *regs) { - instrumentation_begin(); + instrumentation_begin_with_regs(regs); __syscall_exit_to_user_mode_work(regs); instrumentation_end(); __exit_to_user_mode(); @@ -303,7 +303,7 @@ noinstr void irqentry_enter_from_user_mode(struct pt_re= gs *regs) =20 noinstr void irqentry_exit_to_user_mode(struct pt_regs *regs) { - instrumentation_begin(); + instrumentation_begin_with_regs(regs); exit_to_user_mode_prepare(regs); instrumentation_end(); __exit_to_user_mode(); @@ -351,7 +351,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs = *regs) */ lockdep_hardirqs_off(CALLER_ADDR0); rcu_irq_enter(); - instrumentation_begin(); + instrumentation_begin_with_regs(regs); trace_hardirqs_off_finish(); instrumentation_end(); =20 @@ -366,7 +366,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs = *regs) * in having another one here. */ lockdep_hardirqs_off(CALLER_ADDR0); - instrumentation_begin(); + instrumentation_begin_with_regs(regs); rcu_irq_enter_check_tick(); trace_hardirqs_off_finish(); instrumentation_end(); @@ -413,7 +413,7 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqent= ry_state_t state) * and RCU as the return to user mode path. */ if (state.exit_rcu) { - instrumentation_begin(); + instrumentation_begin_with_regs(regs); /* Tell the tracer that IRET will enable interrupts */ trace_hardirqs_on_prepare(); lockdep_hardirqs_on_prepare(CALLER_ADDR0); @@ -423,7 +423,7 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqent= ry_state_t state) return; } =20 - instrumentation_begin(); + instrumentation_begin_with_regs(regs); if (IS_ENABLED(CONFIG_PREEMPTION)) irqentry_exit_cond_resched(); =20 @@ -451,7 +451,7 @@ irqentry_state_t noinstr irqentry_nmi_enter(struct pt_r= egs *regs) lockdep_hardirq_enter(); rcu_nmi_enter(); =20 - instrumentation_begin(); + instrumentation_begin_with_regs(regs); trace_hardirqs_off_finish(); ftrace_nmi_enter(); instrumentation_end(); @@ -461,7 +461,7 @@ irqentry_state_t noinstr irqentry_nmi_enter(struct pt_r= egs *regs) =20 void noinstr irqentry_nmi_exit(struct pt_regs *regs, irqentry_state_t irq_= state) { - instrumentation_begin(); + instrumentation_begin_with_regs(regs); ftrace_nmi_exit(); if (irq_state.lockdep) { trace_hardirqs_on_prepare(); --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A18B1C433F5 for ; Tue, 26 Apr 2022 16:50:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353535AbiDZQxG (ORCPT ); Tue, 26 Apr 2022 12:53:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44044 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353325AbiDZQuR (ORCPT ); Tue, 26 Apr 2022 12:50:17 -0400 Received: from mail-ed1-x54a.google.com (mail-ed1-x54a.google.com [IPv6:2a00:1450:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5CDEC4199C for ; Tue, 26 Apr 2022 09:45:37 -0700 (PDT) Received: by mail-ed1-x54a.google.com with SMTP id h7-20020a056402094700b00425a52983dfso8256051edz.8 for ; Tue, 26 Apr 2022 09:45:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=16ohZwKZ9t3WAbi0BPPBDE5qeMRuLRROgr0ny4RlYME=; b=hd5fKa3yl1cD60GLBnVYHLFqj30FOh54rE2Oq7JHoLD+E1R0r+8ebWBFi4Ec6/yN6I 2VHfaD9u+6R+5pPvZC2vcp3AmeAUtR5OUzLIsbP+sGZGJheeX+7A9SWP7pQngMMz3qjL QTripPctFa88nzwXMYJ8UC69nodJsg64C33KTS2WMClfT1hp1HxZzTUL/pU2gMmMNlXH xX20l1qJhPIRfQXmmOUzJ/S7aHCFrZqOcjs3VcKTw2C+wh835SRNI9el2psQYkn3f7/G tK6Hzl1skO9u7qLGDR3Qbdn2jaaLVwP2o6+HcdbDhyAw87g3MyScAg+EKj1eX/Ob/MJy aLHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=16ohZwKZ9t3WAbi0BPPBDE5qeMRuLRROgr0ny4RlYME=; b=IGNYWfZFZZ8/i7PiYwwRKuNkm1pQMa5MFZ+nT6sOY7HwlqfsddJOaZV6Uwuc8yArRg IoT4wile8MQ0mhjoXpRcz4+9rM2C4MOiU6AACNCC+JD8DbV6xJmH9+jJCrQUll09e7bS PQcPBeEQegDg35I0dqe3lvDSMoInvSvNzKKs553XW750eDBD+c9kV+uGF3J4CSEtoDR2 kuNmfpAT3GWM5GaQYWPz9QGDuJwQ0txblXjZhWeUgqs/5R0fM4WIp0ddp2rf+kPFhSEK LuhbmfVPo1aAYdp6q5tZ/kC8HNNjrs2W181U2LJZO2oVSMY1mBP/Ii5h5oU0+shJaJeS /jSw== X-Gm-Message-State: AOAM531kX/QYzBF3L0eGp5Q7RIsClzXMzmvYHl9/oRPLTWmKd+1saTp9 LxEWmjh9mAhjVOwed5OCBpJnkKWAeyw= X-Google-Smtp-Source: ABdhPJwgIvBqqtb4+UwfAnrYuzq842nCIM4NCU7gD9HqbyhroCBAje/ekLduvR5oCdli1Ru0qSmFr0Qjhq0= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:5255:b0:425:e40a:c927 with SMTP id t21-20020a056402525500b00425e40ac927mr12754417edd.308.1650991535747; Tue, 26 Apr 2022 09:45:35 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:58 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-30-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 29/46] kmsan: add tests for KMSAN From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The testing module triggers KMSAN warnings in different cases and checks that the errors are properly reported, using console probes to capture the tool's output. Signed-off-by: Alexander Potapenko --- v2: -- add memcpy tests Link: https://linux-review.googlesource.com/id/I49c3f59014cc37fd13541c80beb= 0b75a75244650 --- lib/Kconfig.kmsan | 16 ++ mm/kmsan/Makefile | 4 + mm/kmsan/kmsan_test.c | 536 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 556 insertions(+) create mode 100644 mm/kmsan/kmsan_test.c diff --git a/lib/Kconfig.kmsan b/lib/Kconfig.kmsan index 199f79d031f94..a68fdb5ed5d92 100644 --- a/lib/Kconfig.kmsan +++ b/lib/Kconfig.kmsan @@ -21,3 +21,19 @@ config KMSAN the whole system down. =20 See for more details. + +if KMSAN + +config KMSAN_KUNIT_TEST + tristate "KMSAN integration test suite" if !KUNIT_ALL_TESTS + default KUNIT_ALL_TESTS + depends on TRACEPOINTS && KUNIT + help + Test suite for KMSAN, testing various error detection scenarios, + and checking that reports are correctly output to console. + + Say Y here if you want the test to be built into the kernel and run + during boot; say M if you want the test to build as a module; say N + if you are unsure. + +endif diff --git a/mm/kmsan/Makefile b/mm/kmsan/Makefile index f57a956cb1c8b..7be6a7e92394f 100644 --- a/mm/kmsan/Makefile +++ b/mm/kmsan/Makefile @@ -20,3 +20,7 @@ CFLAGS_init.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) CFLAGS_instrumentation.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) CFLAGS_report.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) CFLAGS_shadow.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) + +obj-$(CONFIG_KMSAN_KUNIT_TEST) +=3D kmsan_test.o +KMSAN_SANITIZE_kmsan_test.o :=3D y +CFLAGS_kmsan_test.o +=3D $(call cc-disable-warning, uninitialized) diff --git a/mm/kmsan/kmsan_test.c b/mm/kmsan/kmsan_test.c new file mode 100644 index 0000000000000..44bb2e0f87d81 --- /dev/null +++ b/mm/kmsan/kmsan_test.c @@ -0,0 +1,536 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Test cases for KMSAN. + * For each test case checks the presence (or absence) of generated report= s. + * Relies on 'console' tracepoint to capture reports as they appear in the + * kernel log. + * + * Copyright (C) 2021-2022, Google LLC. + * Author: Alexander Potapenko + * + */ + +#include +#include "kmsan.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static DEFINE_PER_CPU(int, per_cpu_var); + +/* Report as observed from console. */ +static struct { + spinlock_t lock; + bool available; + bool ignore; /* Stop console output collection. */ + char header[256]; +} observed =3D { + .lock =3D __SPIN_LOCK_UNLOCKED(observed.lock), +}; + +/* Probe for console output: obtains observed lines of interest. */ +static void probe_console(void *ignore, const char *buf, size_t len) +{ + unsigned long flags; + + if (observed.ignore) + return; + spin_lock_irqsave(&observed.lock, flags); + + if (strnstr(buf, "BUG: KMSAN: ", len)) { + /* + * KMSAN report and related to the test. + * + * The provided @buf is not NUL-terminated; copy no more than + * @len bytes and let strscpy() add the missing NUL-terminator. + */ + strscpy(observed.header, buf, + min(len + 1, sizeof(observed.header))); + WRITE_ONCE(observed.available, true); + observed.ignore =3D true; + } + spin_unlock_irqrestore(&observed.lock, flags); +} + +/* Check if a report related to the test exists. */ +static bool report_available(void) +{ + return READ_ONCE(observed.available); +} + +/* Information we expect in a report. */ +struct expect_report { + const char *error_type; /* Error type. */ + /* + * Kernel symbol from the error header, or NULL if no report is + * expected. + */ + const char *symbol; +}; + +/* Check observed report matches information in @r. */ +static bool report_matches(const struct expect_report *r) +{ + typeof(observed.header) expected_header; + unsigned long flags; + bool ret =3D false; + const char *end; + char *cur; + + /* Doubled-checked locking. */ + if (!report_available() || !r->symbol) + return (!report_available() && !r->symbol); + + /* Generate expected report contents. */ + + /* Title */ + cur =3D expected_header; + end =3D &expected_header[sizeof(expected_header) - 1]; + + cur +=3D scnprintf(cur, end - cur, "BUG: KMSAN: %s", r->error_type); + + scnprintf(cur, end - cur, " in %s", r->symbol); + /* The exact offset won't match, remove it; also strip module name. */ + cur =3D strchr(expected_header, '+'); + if (cur) + *cur =3D '\0'; + + spin_lock_irqsave(&observed.lock, flags); + if (!report_available()) + goto out; /* A new report is being captured. */ + + /* Finally match expected output to what we actually observed. */ + ret =3D strstr(observed.header, expected_header); +out: + spin_unlock_irqrestore(&observed.lock, flags); + + return ret; +} + +/* =3D=3D=3D=3D=3D Test cases =3D=3D=3D=3D=3D */ + +/* Prevent replacing branch with select in LLVM. */ +static noinline void check_true(char *arg) +{ + pr_info("%s is true\n", arg); +} + +static noinline void check_false(char *arg) +{ + pr_info("%s is false\n", arg); +} + +#define USE(x) = \ + do { \ + if (x) \ + check_true(#x); \ + else \ + check_false(#x); \ + } while (0) + +#define EXPECTATION_ETYPE_FN(e, reason, fn) = \ + struct expect_report e =3D { \ + .error_type =3D reason, \ + .symbol =3D fn, \ + } + +#define EXPECTATION_NO_REPORT(e) EXPECTATION_ETYPE_FN(e, NULL, NULL) +#define EXPECTATION_UNINIT_VALUE_FN(e, fn) = \ + EXPECTATION_ETYPE_FN(e, "uninit-value", fn) +#define EXPECTATION_UNINIT_VALUE(e) EXPECTATION_UNINIT_VALUE_FN(e, __func_= _) +#define EXPECTATION_USE_AFTER_FREE(e) = \ + EXPECTATION_ETYPE_FN(e, "use-after-free", __func__) + +/* Test case: ensure that kmalloc() returns uninitialized memory. */ +static void test_uninit_kmalloc(struct kunit *test) +{ + EXPECTATION_UNINIT_VALUE(expect); + int *ptr; + + kunit_info(test, "uninitialized kmalloc test (UMR report)\n"); + ptr =3D kmalloc(sizeof(int), GFP_KERNEL); + USE(*ptr); + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); +} + +/* + * Test case: ensure that kmalloc'ed memory becomes initialized after mems= et(). + */ +static void test_init_kmalloc(struct kunit *test) +{ + EXPECTATION_NO_REPORT(expect); + int *ptr; + + kunit_info(test, "initialized kmalloc test (no reports)\n"); + ptr =3D kmalloc(sizeof(int), GFP_KERNEL); + memset(ptr, 0, sizeof(int)); + USE(*ptr); + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); +} + +/* Test case: ensure that kzalloc() returns initialized memory. */ +static void test_init_kzalloc(struct kunit *test) +{ + EXPECTATION_NO_REPORT(expect); + int *ptr; + + kunit_info(test, "initialized kzalloc test (no reports)\n"); + ptr =3D kzalloc(sizeof(int), GFP_KERNEL); + USE(*ptr); + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); +} + +/* Test case: ensure that local variables are uninitialized by default. */ +static void test_uninit_stack_var(struct kunit *test) +{ + EXPECTATION_UNINIT_VALUE(expect); + volatile int cond; + + kunit_info(test, "uninitialized stack variable (UMR report)\n"); + USE(cond); + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); +} + +/* Test case: ensure that local variables with initializers are initialize= d. */ +static void test_init_stack_var(struct kunit *test) +{ + EXPECTATION_NO_REPORT(expect); + volatile int cond =3D 1; + + kunit_info(test, "initialized stack variable (no reports)\n"); + USE(cond); + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); +} + +static noinline void two_param_fn_2(int arg1, int arg2) +{ + USE(arg1); + USE(arg2); +} + +static noinline void one_param_fn(int arg) +{ + two_param_fn_2(arg, arg); + USE(arg); +} + +static noinline void two_param_fn(int arg1, int arg2) +{ + int init =3D 0; + + one_param_fn(init); + USE(arg1); + USE(arg2); +} + +static void test_params(struct kunit *test) +{ + EXPECTATION_UNINIT_VALUE_FN(expect, "two_param_fn"); + volatile int uninit, init =3D 1; + + kunit_info(test, + "uninit passed through a function parameter (UMR report)\n"); + two_param_fn(uninit, init); + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); +} + +static int signed_sum3(int a, int b, int c) +{ + return a + b + c; +} + +/* + * Test case: ensure that uninitialized values are tracked through function + * arguments. + */ +static void test_uninit_multiple_params(struct kunit *test) +{ + EXPECTATION_UNINIT_VALUE(expect); + volatile char b =3D 3, c; + volatile int a; + + kunit_info(test, "uninitialized local passed to fn (UMR report)\n"); + USE(signed_sum3(a, b, c)); + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); +} + +/* Helper function to make an array uninitialized. */ +static noinline void do_uninit_local_array(char *array, int start, int sto= p) +{ + volatile char uninit; + int i; + + for (i =3D start; i < stop; i++) + array[i] =3D uninit; +} + +/* + * Test case: ensure kmsan_check_memory() reports an error when checking + * uninitialized memory. + */ +static void test_uninit_kmsan_check_memory(struct kunit *test) +{ + EXPECTATION_UNINIT_VALUE_FN(expect, "test_uninit_kmsan_check_memory"); + volatile char local_array[8]; + + kunit_info( + test, + "kmsan_check_memory() called on uninit local (UMR report)\n"); + do_uninit_local_array((char *)local_array, 5, 7); + + kmsan_check_memory((char *)local_array, 8); + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); +} + +/* + * Test case: check that a virtual memory range created with vmap() from + * initialized pages is still considered as initialized. + */ +static void test_init_kmsan_vmap_vunmap(struct kunit *test) +{ + EXPECTATION_NO_REPORT(expect); + const int npages =3D 2; + struct page **pages; + void *vbuf; + int i; + + kunit_info(test, "pages initialized via vmap (no reports)\n"); + + pages =3D kmalloc_array(npages, sizeof(struct page), GFP_KERNEL); + for (i =3D 0; i < npages; i++) + pages[i] =3D alloc_page(GFP_KERNEL); + vbuf =3D vmap(pages, npages, VM_MAP, PAGE_KERNEL); + memset(vbuf, 0xfe, npages * PAGE_SIZE); + for (i =3D 0; i < npages; i++) + kmsan_check_memory(page_address(pages[i]), PAGE_SIZE); + + if (vbuf) + vunmap(vbuf); + for (i =3D 0; i < npages; i++) + if (pages[i]) + __free_page(pages[i]); + kfree(pages); + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); +} + +/* + * Test case: ensure that memset() can initialize a buffer allocated via + * vmalloc(). + */ +static void test_init_vmalloc(struct kunit *test) +{ + EXPECTATION_NO_REPORT(expect); + int npages =3D 8, i; + char *buf; + + kunit_info(test, "vmalloc buffer can be initialized (no reports)\n"); + buf =3D vmalloc(PAGE_SIZE * npages); + buf[0] =3D 1; + memset(buf, 0xfe, PAGE_SIZE * npages); + USE(buf[0]); + for (i =3D 0; i < npages; i++) + kmsan_check_memory(&buf[PAGE_SIZE * i], PAGE_SIZE); + vfree(buf); + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); +} + +/* Test case: ensure that use-after-free reporting works. */ +static void test_uaf(struct kunit *test) +{ + EXPECTATION_USE_AFTER_FREE(expect); + volatile int value; + volatile int *var; + + kunit_info(test, "use-after-free in kmalloc-ed buffer (UMR report)\n"); + var =3D kmalloc(80, GFP_KERNEL); + var[3] =3D 0xfeedface; + kfree((int *)var); + /* Copy the invalid value before checking it. */ + value =3D var[3]; + USE(value); + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); +} + +/* + * Test case: ensure that uninitialized values are propagated through per-= CPU + * memory. + */ +static void test_percpu_propagate(struct kunit *test) +{ + EXPECTATION_UNINIT_VALUE(expect); + volatile int uninit, check; + + kunit_info(test, + "uninit local stored to per_cpu memory (UMR report)\n"); + + this_cpu_write(per_cpu_var, uninit); + check =3D this_cpu_read(per_cpu_var); + USE(check); + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); +} + +/* + * Test case: ensure that passing uninitialized values to printk() leads t= o an + * error report. + */ +static void test_printk(struct kunit *test) +{ + EXPECTATION_UNINIT_VALUE_FN(expect, "number"); + volatile int uninit; + + kunit_info(test, "uninit local passed to pr_info() (UMR report)\n"); + pr_info("%px contains %d\n", &uninit, uninit); + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); +} + +/* + * Test case: ensure that memcpy() correctly copies uninitialized values b= etween + * aligned `src` and `dst`. + */ +static void test_memcpy_aligned_to_aligned(struct kunit *test) +{ + EXPECTATION_UNINIT_VALUE_FN(expect, "test_memcpy_aligned_to_aligned"); + volatile int uninit_src; + volatile int dst =3D 0; + + kunit_info(test, "memcpy()ing aligned uninit src to aligned dst (UMR repo= rt)\n"); + memcpy((void *)&dst, (void *)&uninit_src, sizeof(uninit_src)); + kmsan_check_memory((void *)&dst, sizeof(dst)); + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); +} + +/* + * Test case: ensure that memcpy() correctly copies uninitialized values b= etween + * aligned `src` and unaligned `dst`. + * + * Copying aligned 4-byte value to an unaligned one leads to touching two + * aligned 4-byte values. This test case checks that KMSAN correctly repor= ts an + * error on the first of the two values. + */ +static void test_memcpy_aligned_to_unaligned(struct kunit *test) +{ + EXPECTATION_UNINIT_VALUE_FN(expect, "test_memcpy_aligned_to_unaligned"); + volatile int uninit_src; + volatile char dst[8] =3D {0}; + + kunit_info(test, "memcpy()ing aligned uninit src to unaligned dst (UMR re= port)\n"); + memcpy((void *)&dst[1], (void *)&uninit_src, sizeof(uninit_src)); + kmsan_check_memory((void *)dst, 4); + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); +} + +/* + * Test case: ensure that memcpy() correctly copies uninitialized values b= etween + * aligned `src` and unaligned `dst`. + * + * Copying aligned 4-byte value to an unaligned one leads to touching two + * aligned 4-byte values. This test case checks that KMSAN correctly repor= ts an + * error on the second of the two values. + */ +static void test_memcpy_aligned_to_unaligned2(struct kunit *test) +{ + EXPECTATION_UNINIT_VALUE_FN(expect, "test_memcpy_aligned_to_unaligned2"); + volatile int uninit_src; + volatile char dst[8] =3D {0}; + + kunit_info(test, "memcpy()ing aligned uninit src to unaligned dst - part = 2 (UMR report)\n"); + memcpy((void *)&dst[1], (void *)&uninit_src, sizeof(uninit_src)); + kmsan_check_memory((void *)&dst[4], sizeof(uninit_src)); + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); +} + +static struct kunit_case kmsan_test_cases[] =3D { + KUNIT_CASE(test_uninit_kmalloc), + KUNIT_CASE(test_init_kmalloc), + KUNIT_CASE(test_init_kzalloc), + KUNIT_CASE(test_uninit_stack_var), + KUNIT_CASE(test_init_stack_var), + KUNIT_CASE(test_params), + KUNIT_CASE(test_uninit_multiple_params), + KUNIT_CASE(test_uninit_kmsan_check_memory), + KUNIT_CASE(test_init_kmsan_vmap_vunmap), + KUNIT_CASE(test_init_vmalloc), + KUNIT_CASE(test_uaf), + KUNIT_CASE(test_percpu_propagate), + KUNIT_CASE(test_printk), + KUNIT_CASE(test_memcpy_aligned_to_aligned), + KUNIT_CASE(test_memcpy_aligned_to_unaligned), + KUNIT_CASE(test_memcpy_aligned_to_unaligned2), + {}, +}; + +/* =3D=3D=3D=3D=3D End test cases =3D=3D=3D=3D=3D */ + +static int test_init(struct kunit *test) +{ + unsigned long flags; + + spin_lock_irqsave(&observed.lock, flags); + observed.header[0] =3D '\0'; + observed.ignore =3D false; + observed.available =3D false; + spin_unlock_irqrestore(&observed.lock, flags); + + return 0; +} + +static void test_exit(struct kunit *test) +{ +} + +static struct kunit_suite kmsan_test_suite =3D { + .name =3D "kmsan", + .test_cases =3D kmsan_test_cases, + .init =3D test_init, + .exit =3D test_exit, +}; +static struct kunit_suite *kmsan_test_suites[] =3D { &kmsan_test_suite, NU= LL }; + +static void register_tracepoints(struct tracepoint *tp, void *ignore) +{ + check_trace_callback_type_console(probe_console); + if (!strcmp(tp->name, "console")) + WARN_ON(tracepoint_probe_register(tp, probe_console, NULL)); +} + +static void unregister_tracepoints(struct tracepoint *tp, void *ignore) +{ + if (!strcmp(tp->name, "console")) + tracepoint_probe_unregister(tp, probe_console, NULL); +} + +/* + * We only want to do tracepoints setup and teardown once, therefore we ha= ve to + * customize the init and exit functions and cannot rely on kunit_test_sui= te(). + */ +static int __init kmsan_test_init(void) +{ + /* + * Because we want to be able to build the test as a module, we need to + * iterate through all known tracepoints, since the static registration + * won't work here. + */ + for_each_kernel_tracepoint(register_tracepoints, NULL); + return __kunit_test_suites_init(kmsan_test_suites); +} + +static void kmsan_test_exit(void) +{ + __kunit_test_suites_exit(kmsan_test_suites); + for_each_kernel_tracepoint(unregister_tracepoints, NULL); + tracepoint_synchronize_unregister(); +} + +late_initcall_sync(kmsan_test_init); +module_exit(kmsan_test_exit); + +MODULE_LICENSE("GPL v2"); +MODULE_AUTHOR("Alexander Potapenko "); --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93DF6C433FE for ; Tue, 26 Apr 2022 16:49:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353528AbiDZQxB (ORCPT ); Tue, 26 Apr 2022 12:53:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45026 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353284AbiDZQuS (ORCPT ); Tue, 26 Apr 2022 12:50:18 -0400 Received: from mail-ej1-x649.google.com (mail-ej1-x649.google.com [IPv6:2a00:1450:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 973AC4739C for ; Tue, 26 Apr 2022 09:45:39 -0700 (PDT) Received: by mail-ej1-x649.google.com with SMTP id o8-20020a170906974800b006f3a8be7502so2044232ejy.8 for ; Tue, 26 Apr 2022 09:45:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=bWOHkFAZUowQhXbgV/15IRTSfhe/pFIeo0hpg6VoqgU=; b=Y425LRYJ7pqp2qUV4Un5BKaPtlMc56oRipAYvk35bNiZHMqwQj5K1oHUY+3UtaBrm8 6F8LD23RH7puAqmN6VxxN3dzRPCX2o3jtXwChLes9m38UGHVO/PjltBdViFuvOO0/pv+ 9a5CV3vayhrQVrbmrEv1bjS0Nmz7caoEnvxuLCxERX2znPJk4SiPOZ9Qb7HS9LCHG3Lh qNuuAbPiqTn/Kfp6058UKfvV0NLpFCO10Egzm+8gacy2cKWG5HhTdR3sVdEPim8XyvUO oM/6bhYxYTS9e2sL18gV9sczR7Og8Y4lewCULG7qERx2rzD5bbV+C50LUjYRSUrnHoTG GPqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=bWOHkFAZUowQhXbgV/15IRTSfhe/pFIeo0hpg6VoqgU=; b=ihYk+y9wfGQu2tclZzzhdM7bqKD6byTpsOSpPh5Jvy2gqqEi6rL4k0SwSmhtLWkjAy teMw/EImpxPF9RPrZma8JX1UNFwmOuKKCA0HyU4afVcUFWfnkT4SDhY1XRZ8gafZ7w62 7knj+AstYV+LREyUIIUi668ryrE/BgHbx/mlq2QXtQa5Gzd3fwJtxuG3Y85noylLfa5v G4iUTiUDwqyiyDtf7xMCr8J40U3kL1orG7PnU6jX2yN5WIIouxsAsYO5UCAqdpZphIeS K28u5BiOl7rwdp/jvsXSJ8lymF/5vFvUTl8u7BiPCUOSVW7lFNuUPtutjENxeyzCt8cT 53sg== X-Gm-Message-State: AOAM533UDF3b83IjobGRfVMLWfLqCDyrh2UMHRyJd4S4hEPXZqlpfYKl q4SI2gvOScltDjjGtJnGKWBHn1kvkYY= X-Google-Smtp-Source: ABdhPJy8hRX/dOnDKYiAzHg5kT/4VfAMjL04J1HrqWNmAu8buMPT+aF4N+PIwLqQH/MNoEjw6XwUMlGRDw4= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:aa7:c789:0:b0:413:605d:8d17 with SMTP id n9-20020aa7c789000000b00413605d8d17mr25370617eds.100.1650991538395; Tue, 26 Apr 2022 09:45:38 -0700 (PDT) Date: Tue, 26 Apr 2022 18:42:59 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-31-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 30/46] kmsan: disable strscpy() optimization under KMSAN From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Disable the efficient 8-byte reading under KMSAN to avoid false positives. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/Iffd8336965e88fce915db2e6a9d= 6524422975f69 --- lib/string.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/lib/string.c b/lib/string.c index 485777c9da832..4ece4c7e7831b 100644 --- a/lib/string.c +++ b/lib/string.c @@ -197,6 +197,14 @@ ssize_t strscpy(char *dest, const char *src, size_t co= unt) max =3D 0; #endif =20 + /* + * read_word_at_a_time() below may read uninitialized bytes after the + * trailing zero and use them in comparisons. Disable this optimization + * under KMSAN to prevent false positive reports. + */ + if (IS_ENABLED(CONFIG_KMSAN)) + max =3D 0; + while (max >=3D sizeof(unsigned long)) { unsigned long c, data; =20 --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10FDBC433F5 for ; Tue, 26 Apr 2022 16:49:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237000AbiDZQw5 (ORCPT ); Tue, 26 Apr 2022 12:52:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45068 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353262AbiDZQuS (ORCPT ); Tue, 26 Apr 2022 12:50:18 -0400 Received: from mail-ed1-x549.google.com (mail-ed1-x549.google.com [IPv6:2a00:1450:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7ED9347AED for ; Tue, 26 Apr 2022 09:45:42 -0700 (PDT) Received: by mail-ed1-x549.google.com with SMTP id w8-20020a50d788000000b00418e6810364so10519023edi.13 for ; Tue, 26 Apr 2022 09:45:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=38v0n0Mu0Z4gUwOcZ/qVPNhGReXdywq1ed1xS1ZMc/s=; b=CJZchZCi7h9Kr6IqDDTDcDumDHdj5sj0cAnBKwNoFce2+vz2Yl0MST5GWeDoMSKZ5a NVfjA+ZhItY65EpbQfZDQUCrqWGpDgv9nhiufcLfyRUfcjRvhRPn5OWtwZ3RALS3H0KH LhLDzIa1Or/rBw7+3a9BvsHMp18Jf58zzFyw76E+xVBiJy1T4ARK87dTf7j5OKfhdDne Iai9CoJ2Q3R2SWuzH5JHoscZjh4ZBFKysHomae/gt81c6+HE2/qqUTttMsNQx4fHXJkp vLvPqFXYcALtcsBGrJo7cj8kPyJ2pPrXNttP82G8cBzD6ZB4Kfjj/ja2SDjIFA84dxep n0HQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=38v0n0Mu0Z4gUwOcZ/qVPNhGReXdywq1ed1xS1ZMc/s=; b=t4KqfbtuBBCru+oyfm+fS1sFojbLL3NpIRQPopzRWhbfqCooLXQw/fLmh6SAXrkohb 8wgLFMew/GhCjcBGKyBvSymnBo58JnBsdeOxuEYTxRcJZzJuSgNd1fzqX+6f94LtgWKa 2Ad4LGPwhG5bImvaGdz2rWEKc7TyZNskvH6slmdZD407azEM7F7s/xMKV4ooYYNGjhot xVYnaqqFzITzrATmAuHd4TlL9KmiYlK7oUHw8pWsUgDjzTm8fgtP6a1vLtHhoFnStqem 8mXNuQxZwJ+gIqUA5zkKB3lQYx6y7kI2eDLW6Ht5Bj/2mKZusymaRlP3vBr+P48eIJbo EeSg== X-Gm-Message-State: AOAM533wKmQ1SOKjMn3EnhMkA5UZSleNsNniPlqY1uk9fPny9m4JsX2V 9S579/VksZGkrhfuukKfVPK8lWtb62M= X-Google-Smtp-Source: ABdhPJxDQJ1KHzAU3KQwWTLZtud1ICLmCrTQkWtJUuVwEnyYS6TOqN9JDU2m4nyDsEQ80Q+Det4ouE9T2mY= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:4305:b0:423:f73b:4dd8 with SMTP id m5-20020a056402430500b00423f73b4dd8mr25672281edc.218.1650991540848; Tue, 26 Apr 2022 09:45:40 -0700 (PDT) Date: Tue, 26 Apr 2022 18:43:00 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-32-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 31/46] crypto: kmsan: disable accelerated configs under KMSAN From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" KMSAN is unable to understand when initialized values come from assembly. Disable accelerated configs in KMSAN builds to prevent false positive reports. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/Idb2334bf3a1b68b31b399709bae= faa763038cc50 --- crypto/Kconfig | 30 ++++++++++++++++++++++++++++++ drivers/net/Kconfig | 1 + 2 files changed, 31 insertions(+) diff --git a/crypto/Kconfig b/crypto/Kconfig index 41068811fd0e1..8078dbba8dd2c 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -297,6 +297,7 @@ config CRYPTO_CURVE25519 config CRYPTO_CURVE25519_X86 tristate "x86_64 accelerated Curve25519 scalar multiplication library" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_LIB_CURVE25519_GENERIC select CRYPTO_ARCH_HAVE_LIB_CURVE25519 =20 @@ -345,11 +346,13 @@ config CRYPTO_AEGIS128 config CRYPTO_AEGIS128_SIMD bool "Support SIMD acceleration for AEGIS-128" depends on CRYPTO_AEGIS128 && ((ARM || ARM64) && KERNEL_MODE_NEON) + depends on !KMSAN # avoid false positives from assembly default y =20 config CRYPTO_AEGIS128_AESNI_SSE2 tristate "AEGIS-128 AEAD algorithm (x86_64 AESNI+SSE2 implementation)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_AEAD select CRYPTO_SIMD help @@ -486,6 +489,7 @@ config CRYPTO_NHPOLY1305 config CRYPTO_NHPOLY1305_SSE2 tristate "NHPoly1305 hash function (x86_64 SSE2 implementation)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_NHPOLY1305 help SSE2 optimized implementation of the hash function used by the @@ -494,6 +498,7 @@ config CRYPTO_NHPOLY1305_SSE2 config CRYPTO_NHPOLY1305_AVX2 tristate "NHPoly1305 hash function (x86_64 AVX2 implementation)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_NHPOLY1305 help AVX2 optimized implementation of the hash function used by the @@ -607,6 +612,7 @@ config CRYPTO_CRC32C config CRYPTO_CRC32C_INTEL tristate "CRC32c INTEL hardware acceleration" depends on X86 + depends on !KMSAN # avoid false positives from assembly select CRYPTO_HASH help In Intel processor with SSE4.2 supported, the processor will @@ -647,6 +653,7 @@ config CRYPTO_CRC32 config CRYPTO_CRC32_PCLMUL tristate "CRC32 PCLMULQDQ hardware acceleration" depends on X86 + depends on !KMSAN # avoid false positives from assembly select CRYPTO_HASH select CRC32 help @@ -712,6 +719,7 @@ config CRYPTO_BLAKE2S config CRYPTO_BLAKE2S_X86 tristate "BLAKE2s digest algorithm (x86 accelerated version)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_LIB_BLAKE2S_GENERIC select CRYPTO_ARCH_HAVE_LIB_BLAKE2S =20 @@ -726,6 +734,7 @@ config CRYPTO_CRCT10DIF config CRYPTO_CRCT10DIF_PCLMUL tristate "CRCT10DIF PCLMULQDQ hardware acceleration" depends on X86 && 64BIT && CRC_T10DIF + depends on !KMSAN # avoid false positives from assembly select CRYPTO_HASH help For x86_64 processors with SSE4.2 and PCLMULQDQ supported, @@ -778,6 +787,7 @@ config CRYPTO_POLY1305 config CRYPTO_POLY1305_X86_64 tristate "Poly1305 authenticator algorithm (x86_64/SSE2/AVX2)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_LIB_POLY1305_GENERIC select CRYPTO_ARCH_HAVE_LIB_POLY1305 help @@ -866,6 +876,7 @@ config CRYPTO_SHA1 config CRYPTO_SHA1_SSSE3 tristate "SHA1 digest algorithm (SSSE3/AVX/AVX2/SHA-NI)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_SHA1 select CRYPTO_HASH help @@ -877,6 +888,7 @@ config CRYPTO_SHA1_SSSE3 config CRYPTO_SHA256_SSSE3 tristate "SHA256 digest algorithm (SSSE3/AVX/AVX2/SHA-NI)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_SHA256 select CRYPTO_HASH help @@ -889,6 +901,7 @@ config CRYPTO_SHA256_SSSE3 config CRYPTO_SHA512_SSSE3 tristate "SHA512 digest algorithm (SSSE3/AVX/AVX2)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_SHA512 select CRYPTO_HASH help @@ -1061,6 +1074,7 @@ config CRYPTO_WP512 config CRYPTO_GHASH_CLMUL_NI_INTEL tristate "GHASH hash function (CLMUL-NI accelerated)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_CRYPTD help This is the x86_64 CLMUL-NI accelerated implementation of @@ -1111,6 +1125,7 @@ config CRYPTO_AES_TI config CRYPTO_AES_NI_INTEL tristate "AES cipher algorithms (AES-NI)" depends on X86 + depends on !KMSAN # avoid false positives from assembly select CRYPTO_AEAD select CRYPTO_LIB_AES select CRYPTO_ALGAPI @@ -1235,6 +1250,7 @@ config CRYPTO_BLOWFISH_COMMON config CRYPTO_BLOWFISH_X86_64 tristate "Blowfish cipher algorithm (x86_64)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_SKCIPHER select CRYPTO_BLOWFISH_COMMON imply CRYPTO_CTR @@ -1265,6 +1281,7 @@ config CRYPTO_CAMELLIA config CRYPTO_CAMELLIA_X86_64 tristate "Camellia cipher algorithm (x86_64)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_SKCIPHER imply CRYPTO_CTR help @@ -1281,6 +1298,7 @@ config CRYPTO_CAMELLIA_X86_64 config CRYPTO_CAMELLIA_AESNI_AVX_X86_64 tristate "Camellia cipher algorithm (x86_64/AES-NI/AVX)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_SKCIPHER select CRYPTO_CAMELLIA_X86_64 select CRYPTO_SIMD @@ -1299,6 +1317,7 @@ config CRYPTO_CAMELLIA_AESNI_AVX_X86_64 config CRYPTO_CAMELLIA_AESNI_AVX2_X86_64 tristate "Camellia cipher algorithm (x86_64/AES-NI/AVX2)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_CAMELLIA_AESNI_AVX_X86_64 help Camellia cipher algorithm module (x86_64/AES-NI/AVX2). @@ -1344,6 +1363,7 @@ config CRYPTO_CAST5 config CRYPTO_CAST5_AVX_X86_64 tristate "CAST5 (CAST-128) cipher algorithm (x86_64/AVX)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_SKCIPHER select CRYPTO_CAST5 select CRYPTO_CAST_COMMON @@ -1367,6 +1387,7 @@ config CRYPTO_CAST6 config CRYPTO_CAST6_AVX_X86_64 tristate "CAST6 (CAST-256) cipher algorithm (x86_64/AVX)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_SKCIPHER select CRYPTO_CAST6 select CRYPTO_CAST_COMMON @@ -1400,6 +1421,7 @@ config CRYPTO_DES_SPARC64 config CRYPTO_DES3_EDE_X86_64 tristate "Triple DES EDE cipher algorithm (x86-64)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_SKCIPHER select CRYPTO_LIB_DES imply CRYPTO_CTR @@ -1457,6 +1479,7 @@ config CRYPTO_CHACHA20 config CRYPTO_CHACHA20_X86_64 tristate "ChaCha stream cipher algorithms (x86_64/SSSE3/AVX2/AVX-512VL)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_SKCIPHER select CRYPTO_LIB_CHACHA_GENERIC select CRYPTO_ARCH_HAVE_LIB_CHACHA @@ -1500,6 +1523,7 @@ config CRYPTO_SERPENT config CRYPTO_SERPENT_SSE2_X86_64 tristate "Serpent cipher algorithm (x86_64/SSE2)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_SKCIPHER select CRYPTO_SERPENT select CRYPTO_SIMD @@ -1519,6 +1543,7 @@ config CRYPTO_SERPENT_SSE2_X86_64 config CRYPTO_SERPENT_SSE2_586 tristate "Serpent cipher algorithm (i586/SSE2)" depends on X86 && !64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_SKCIPHER select CRYPTO_SERPENT select CRYPTO_SIMD @@ -1538,6 +1563,7 @@ config CRYPTO_SERPENT_SSE2_586 config CRYPTO_SERPENT_AVX_X86_64 tristate "Serpent cipher algorithm (x86_64/AVX)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_SKCIPHER select CRYPTO_SERPENT select CRYPTO_SIMD @@ -1558,6 +1584,7 @@ config CRYPTO_SERPENT_AVX_X86_64 config CRYPTO_SERPENT_AVX2_X86_64 tristate "Serpent cipher algorithm (x86_64/AVX2)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_SERPENT_AVX_X86_64 help Serpent cipher algorithm, by Anderson, Biham & Knudsen. @@ -1699,6 +1726,7 @@ config CRYPTO_TWOFISH_586 config CRYPTO_TWOFISH_X86_64 tristate "Twofish cipher algorithm (x86_64)" depends on (X86 || UML_X86) && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_ALGAPI select CRYPTO_TWOFISH_COMMON imply CRYPTO_CTR @@ -1716,6 +1744,7 @@ config CRYPTO_TWOFISH_X86_64 config CRYPTO_TWOFISH_X86_64_3WAY tristate "Twofish cipher algorithm (x86_64, 3-way parallel)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_SKCIPHER select CRYPTO_TWOFISH_COMMON select CRYPTO_TWOFISH_X86_64 @@ -1736,6 +1765,7 @@ config CRYPTO_TWOFISH_X86_64_3WAY config CRYPTO_TWOFISH_AVX_X86_64 tristate "Twofish cipher algorithm (x86_64/AVX)" depends on X86 && 64BIT + depends on !KMSAN # avoid false positives from assembly select CRYPTO_SKCIPHER select CRYPTO_SIMD select CRYPTO_TWOFISH_COMMON diff --git a/drivers/net/Kconfig b/drivers/net/Kconfig index b2a4f998c180e..fed89b6981759 100644 --- a/drivers/net/Kconfig +++ b/drivers/net/Kconfig @@ -76,6 +76,7 @@ config WIREGUARD tristate "WireGuard secure network tunnel" depends on NET && INET depends on IPV6 || !IPV6 + depends on !KMSAN # KMSAN doesn't support the crypto configs below select NET_UDP_TUNNEL select DST_CACHE select CRYPTO --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EBB0C433EF for ; Tue, 26 Apr 2022 16:48:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353456AbiDZQvg (ORCPT ); Tue, 26 Apr 2022 12:51:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45096 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346708AbiDZQuo (ORCPT ); Tue, 26 Apr 2022 12:50:44 -0400 Received: from mail-ed1-x54a.google.com (mail-ed1-x54a.google.com [IPv6:2a00:1450:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2615348302 for ; Tue, 26 Apr 2022 09:45:45 -0700 (PDT) Received: by mail-ed1-x54a.google.com with SMTP id l24-20020a056402231800b00410f19a3103so10612285eda.5 for ; Tue, 26 Apr 2022 09:45:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=KgfdOYR2+eRzwy3N1nFbXjWhZjMyNN/8KPMy2HvUIiA=; b=DiXJaiB8AJMG8+A1MDOZ3VuBShHUCaL1GlKQxcD6mV0R7Av/UajF571MXN7vKcH+xJ B9d0v8KhVCzlQ+z94pkBBtVLTSnlc2yIkyIXP0B+bFCdaqtmHA9pe8/5NYuKcmD5Vy3L fA34cdtJjSamNgh5dsmbUVd7b9zA7hnjKOjdt88/6jqxc/UG3ZHi4m5Nam5rWzqbzqPm cd06DBJ3ACVxlq9XrgpCh+wVKQori1dj0NvbQ7HSZw0NjHsIf9jtgKdLkxo1euBXeZpk ExnDMib7ea+byjuGF4JqXiPnkpgw3Zh2HRRI3cVSjxCjQ97crsBXKlz1IuJlZ96GONPP Cgcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=KgfdOYR2+eRzwy3N1nFbXjWhZjMyNN/8KPMy2HvUIiA=; b=BGou/RkgZRvdZWK5APnCgjMVwxJrQ0C2f6hhmhd91xVzoXAxo6a4toYA3/5jgUrgK4 15VHCK2Y6Mq+gs/YgaDZPrVqvvXjzWya0ofAqS7GPAeD5uz3MiEOnjS+ACrUweyBvVG3 wRUvxNBAF3CZmKesjiw2z67FJ8tYV1lEKhCRSJv6xR9Zj9y/JVaM377xDko3kVV541Ny 2Khl/8V74AqtRgTQrKGPDw4cDL/cQFL4H/2HrAzq6HXJ99LokU3Jl/exQQgKu06UigJY S61dACCbvfigUUNtJmWV2muhDjKD6F6vS3+7OA9bHTtoeyHaDoCq/N07SOfGhxv+LEnL /w/g== X-Gm-Message-State: AOAM532msI6eWpTZ3R3O7wHB5sQVdWSwMGmzZuDTO5ZTxY0GJbrEYh06 woA4h2mn1JpOivt4AxvZhvWolMXH6EQ= X-Google-Smtp-Source: ABdhPJzCT8fH32RoUVNaiZCyh2VQrj6uF5kLZlCQX+kBGFsZsTKhLxpnKoIswBSqwHZF0t1dzIj5uX6h2dU= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:26c7:b0:423:e5d6:b6c6 with SMTP id x7-20020a05640226c700b00423e5d6b6c6mr25480700edd.61.1650991543391; Tue, 26 Apr 2022 09:45:43 -0700 (PDT) Date: Tue, 26 Apr 2022 18:43:01 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-33-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 32/46] kmsan: disable physical page merging in biovec From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" KMSAN metadata for adjacent physical pages may not be adjacent, therefore accessing such pages together may lead to metadata corruption. We disable merging pages in biovec to prevent such corruptions. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/Iece16041be5ee47904fbc98121b= 105e5be5fea5c --- block/blk.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/block/blk.h b/block/blk.h index 8ccbc6e076369..95815ac559743 100644 --- a/block/blk.h +++ b/block/blk.h @@ -93,6 +93,13 @@ static inline bool biovec_phys_mergeable(struct request_= queue *q, phys_addr_t addr1 =3D page_to_phys(vec1->bv_page) + vec1->bv_offset; phys_addr_t addr2 =3D page_to_phys(vec2->bv_page) + vec2->bv_offset; =20 + /* + * Merging adjacent physical pages may not work correctly under KMSAN + * if their metadata pages aren't adjacent. Just disable merging. + */ + if (IS_ENABLED(CONFIG_KMSAN)) + return false; + if (addr1 + vec1->bv_len !=3D addr2) return false; if (xen_domain() && !xen_biovec_phys_mergeable(vec1, vec2->bv_page)) --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EC64C433FE for ; Tue, 26 Apr 2022 16:49:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353519AbiDZQwx (ORCPT ); Tue, 26 Apr 2022 12:52:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44176 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353345AbiDZQup (ORCPT ); Tue, 26 Apr 2022 12:50:45 -0400 Received: from mail-lj1-x24a.google.com (mail-lj1-x24a.google.com [IPv6:2a00:1450:4864:20::24a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A4F748315 for ; Tue, 26 Apr 2022 09:45:47 -0700 (PDT) Received: by mail-lj1-x24a.google.com with SMTP id 23-20020a05651c00d700b0024f12064717so1744783ljr.15 for ; Tue, 26 Apr 2022 09:45:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=GtiQESdTzDECedHCE4txXpUZH3/tDa3FQ07WXH5OYuQ=; b=UW3AswRaq6F3AUzKmXnr7cPkQ0g2XLtpFenEmpgWjfvp2fOauRXzdotOTeUNyWSyYs 4tmhw8qwzpN9wiAWPxJUMCVxV1rWcYuXS49a+1tCMi0Oc5SG780CX3I+mnAa6/MfcB77 6+gdBv5jHg6AKYvKP7kA5u9scJwMLXdv1i19bLKNouK41RBdBrSMPPJuqKL61o2W1+Ho pFIgSZPybW4jMIEebVa0EuG05Nils5rY1Bzl3+5j5IX62oqraNKPX3IyM6K8LuZYgWcI qaxMdheHooD/KCw39MnxjCE+Vv1BhdJnZkIOIVorHWlt4oOjs2wI7s6rPQyDKu5idJkG PQRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=GtiQESdTzDECedHCE4txXpUZH3/tDa3FQ07WXH5OYuQ=; b=aZ7R5N0q+z0Kt4BIG8z/gVrIBbtsGsyzzAyKs8bPHkpX/ScXKjoJoRtyEfLNWxbuVR c0XjUTZS/c2v82DzAnYdGlP1rymNZB0Xx1pSPhm9gT/8wVq658i7+JyJqixJU9BV/dxe u/GLSk1mPUJVmcxSKfllSCh6xrkH03wkkqvJTZBr2LrxSjQgH7dLuHvNRwFayVeb5LeL x6kXneO7DHm33gDBLhZpoYSWQK0Vqenp8GKshPbb8VOeqXzY0mMvWGVbTZo4OX9R0y6S m5afapPq9Hx6Y08FITXVhOdvA8sO825Czhnj5+AsgctYQM6wypQ2PnuxULxCV7+K45tA jTKw== X-Gm-Message-State: AOAM533lqKm+aHjc6FlxjP2y73osUEO/JvHU1oSQxCx1PGpBZzBnw+BG IVKgyRZW7EITygJfrmpa2kt0PpaKqoM= X-Google-Smtp-Source: ABdhPJwozf4GfzZ3k/EMYH1LpIZPBrUltfR1bzZ40SvgirQ7Xof252RdKVFQ5zP9UN/h/78LotXJK8Dsdf0= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6512:20c6:b0:471:fdba:1480 with SMTP id u6-20020a05651220c600b00471fdba1480mr10896844lfr.425.1650991546042; Tue, 26 Apr 2022 09:45:46 -0700 (PDT) Date: Tue, 26 Apr 2022 18:43:02 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-34-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 33/46] kmsan: block: skip bio block merging logic for KMSAN From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, Eric Biggers Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" KMSAN doesn't allow treating adjacent memory pages as such, if they were allocated by different alloc_pages() calls. The block layer however does so: adjacent pages end up being used together. To prevent this, make page_is_mergeable() return false under KMSAN. Suggested-by: Eric Biggers Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/Ie29cc2464c70032347c32ab2a22= e1e7a0b37b905 --- block/bio.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/block/bio.c b/block/bio.c index 4259125e16ab2..db56090c00bae 100644 --- a/block/bio.c +++ b/block/bio.c @@ -836,6 +836,8 @@ static inline bool page_is_mergeable(const struct bio_v= ec *bv, return false; =20 *same_page =3D ((vec_end_addr & PAGE_MASK) =3D=3D page_addr); + if (!*same_page && IS_ENABLED(CONFIG_KMSAN)) + return false; if (*same_page) return true; return (bv->bv_page + bv_end / PAGE_SIZE) =3D=3D (page + off / PAGE_SIZE); --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8AA81C433F5 for ; Tue, 26 Apr 2022 16:49:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353513AbiDZQwt (ORCPT ); Tue, 26 Apr 2022 12:52:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45134 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353350AbiDZQu6 (ORCPT ); Tue, 26 Apr 2022 12:50:58 -0400 Received: from mail-wm1-x349.google.com (mail-wm1-x349.google.com [IPv6:2a00:1450:4864:20::349]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0580148321 for ; Tue, 26 Apr 2022 09:45:50 -0700 (PDT) Received: by mail-wm1-x349.google.com with SMTP id n25-20020a05600c3b9900b0038ff033b654so4011471wms.0 for ; Tue, 26 Apr 2022 09:45:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=KHDnEJZlYbtguNecmw4Z0aoQP18cjcWXna880BlF2jg=; b=Ras6zX6zLyLhsIDtzjqqN9y4Bp5f6rPNvg/NlhbZeOJcSZ0W0h6WszebH9Jl+tDnIj 1d8SC1JokEP9rj0/Uz/B0C8vgBdM75W2cOctkHAhkLUnzcLWel+aJ8n1Z2d27uToZyPD NXcsYuwWmfXUsoUSNdeFpNkCp8Mdmw2X2PM8Z0Xq3w/pvHZee2tgfSDClqP6w/GG0G6Z Mkg5MkTefNZEIo51rh3WSUC/xRWyYaKD3+5kRfKXRqTWJTGdKD34V/dogH3TIl3qdspb r/9R17wsy7X98UcL39rrXIx8RR/Zx48ZWvJVe4g3RHHZ1p7AGTIwvBiDniEVZixdJ1br ihtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=KHDnEJZlYbtguNecmw4Z0aoQP18cjcWXna880BlF2jg=; b=j4qzgF48585Hids6yZc0joP9EM8ittLJyEKvWxGkpM5AkXfMGykvnBGp5i6UdkytQn UxGoVTKJVMs6ZaIf55tyqpSCsSEjiCy4xLriIuvZs3/Q+v6yd6X+YcJRKMzfF1xugkZ1 i6oIXRBeEUceTjfdC8DsV8BYVV6wn0B9rLc8SeVuWEfuA0dkU9oLnu6iqiFiDcsJfZm5 a90gNjE5JC56aP2vU0xi4a+qp4QFuCd9EGNYgF97d2ahAX0GhKVESfLJ55OnXcEh7ZpP 2gvWV+ICqV/rdIpfBONDNoquU37GKY1d2obWh6hMC7UHPmSaNyjHI70aLnK6j5Ikf59k z28A== X-Gm-Message-State: AOAM533zprVr2AqCGswieIKeRyqMm4C278ymkJE+euaLGaHAc45LaOvc y7vKeMdSxSyrNl5cGS0oTn2U/iDYYOk= X-Google-Smtp-Source: ABdhPJwEbc2ldIDEpaUEaYnMNUbLZOo12zW9lcP4NsRCoKnifcBnuhUmGJEZqd7cMdPo2nqMrlaeyXx5FlQ= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:adf:e289:0:b0:1e3:14ad:75fe with SMTP id v9-20020adfe289000000b001e314ad75femr18987161wri.685.1650991548482; Tue, 26 Apr 2022 09:45:48 -0700 (PDT) Date: Tue, 26 Apr 2022 18:43:03 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-35-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 34/46] kmsan: kcov: unpoison area->list in kcov_remote_area_put() From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" KMSAN does not instrument kernel/kcov.c for performance reasons (with CONFIG_KCOV=3Dy virtually every place in the kernel invokes kcov instrumentation). Therefore the tool may miss writes from kcov.c that initialize memory. When CONFIG_DEBUG_LIST is enabled, list pointers from kernel/kcov.c are passed to instrumented helpers in lib/list_debug.c, resulting in false positives. To work around these reports, we unpoison the contents of area->list after initializing it. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/Ie17f2ee47a7af58f5cdf716d585= ebf0769348a5a --- kernel/kcov.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/kernel/kcov.c b/kernel/kcov.c index b3732b2105930..9e38209a7e0a9 100644 --- a/kernel/kcov.c +++ b/kernel/kcov.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -152,6 +153,12 @@ static void kcov_remote_area_put(struct kcov_remote_ar= ea *area, INIT_LIST_HEAD(&area->list); area->size =3D size; list_add(&area->list, &kcov_remote_areas); + /* + * KMSAN doesn't instrument this file, so it may not know area->list + * is initialized. Unpoison it explicitly to avoid reports in + * kcov_remote_area_get(). + */ + kmsan_unpoison_memory(&area->list, sizeof(struct list_head)); } =20 static notrace bool check_kcov_mode(enum kcov_mode needed_mode, struct tas= k_struct *t) --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D102EC433EF for ; Tue, 26 Apr 2022 16:49:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353479AbiDZQwW (ORCPT ); Tue, 26 Apr 2022 12:52:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45470 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353219AbiDZQu7 (ORCPT ); Tue, 26 Apr 2022 12:50:59 -0400 Received: from mail-ed1-x549.google.com (mail-ed1-x549.google.com [IPv6:2a00:1450:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 972D44832B for ; Tue, 26 Apr 2022 09:45:52 -0700 (PDT) Received: by mail-ed1-x549.google.com with SMTP id cf16-20020a0564020b9000b00425d543c75dso4625358edb.11 for ; Tue, 26 Apr 2022 09:45:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=qfycs5WHoOJISCgVkz4aOXJI06Vi4G56am9I/vqcS5Q=; b=fRwDF91Cl56NtYnfVRrrwWq+ACk34XnpCPgWDJNhxkPf0+jyG4t7SUqyZkxziScoD6 nlS+/T0kxR6PQ7wQlPhgkRdK+OAhXkwXjvWT5GdxHtziDcUCsqgfbLBiAQg7R4lxOA+S b1h3WJe9vuCUveQdnRjvIleLSdoXjd5T81AR3B/d6sysyhegK2IbBH7cJDKBoPbngE01 jvw+rllOIeEdhzy+uxIADb7nx6kl7sAhRpV9EXPjMlnyR573JuNml4rWiL8PIe9a6qEW epw0ifH48szv7cq73qPvPsJ2sXpbRFaPAhln/IK2M8zSSWNmDvdbyCHkb2G+fh3x7wwt zbNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=qfycs5WHoOJISCgVkz4aOXJI06Vi4G56am9I/vqcS5Q=; b=oDu0iONaHPE/onrD7cf4x+iTm6TJkajUF6zI68YuEERd6hAxrpA/Crm829twaOpPD6 10fpczJz/W4AHE7/BdAfjH/DeHPT3qTHGErihK8nN1eQHoiez8WgEYVAwA302Lj3SjdP zSO/bgjU/oEdU1AIJVPsj9J3NGxrEYrUPV51CvKK5GkVxFbORBAjtMjR1HmvVU9aRNhW cPsbmPB5I1rppDDdt1Ap6fpUFL7ShJo9bt7OiIC/cql+em3kNamEmYflt2BPgFQfh74Z /2AjZjEHAvqbdhwKEehUqE6wy2ZA3GuSD7pL+4EddysMVTaD+oMmUASWGvpb5SqA4csH hXaw== X-Gm-Message-State: AOAM531EykexQRiwTrkBEDQ7jY8UTJ2paqQW2nkIal1f62WaqO5DDGL7 3Tq/WysNiZvbNjGupowhz7qZgA3cNsQ= X-Google-Smtp-Source: ABdhPJyO5IzITq1Md/msOkFe5QpCMXnvVTw7Eg5gyXSwTxjpxMvbx88JMdeRNdS+uJ66Z5Deghmk2VEGLtk= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a17:906:a08b:b0:6b9:2e20:f139 with SMTP id q11-20020a170906a08b00b006b92e20f139mr23252089ejy.463.1650991550999; Tue, 26 Apr 2022 09:45:50 -0700 (PDT) Date: Tue, 26 Apr 2022 18:43:04 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-36-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 35/46] security: kmsan: fix interoperability with auto-initialization From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Heap and stack initialization is great, but not when we are trying uses of uninitialized memory. When the kernel is built with KMSAN, having kernel memory initialization enabled may introduce false negatives. We disable CONFIG_INIT_STACK_ALL_PATTERN and CONFIG_INIT_STACK_ALL_ZERO under CONFIG_KMSAN, making it impossible to auto-initialize stack variables in KMSAN builds. We also disable CONFIG_INIT_ON_ALLOC_DEFAULT_ON and CONFIG_INIT_ON_FREE_DEFAULT_ON to prevent accidental use of heap auto-initialization. We however still let the users enable heap auto-initialization at boot-time (by setting init_on_alloc=3D1 or init_on_free=3D1), in which case a warning is printed. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/I86608dd867018683a14ae1870f1= 928ad925f42e9 --- mm/page_alloc.c | 4 ++++ security/Kconfig.hardening | 4 ++++ 2 files changed, 8 insertions(+) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 35b1fedb2f09c..4c89729cac7ac 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -849,6 +849,10 @@ void init_mem_debugging_and_hardening(void) else static_branch_disable(&init_on_free); =20 + if (IS_ENABLED(CONFIG_KMSAN) && + (_init_on_alloc_enabled_early || _init_on_free_enabled_early)) + pr_info("mem auto-init: please make sure init_on_alloc and init_on_free = are disabled when running KMSAN\n"); + #ifdef CONFIG_DEBUG_PAGEALLOC if (!debug_pagealloc_enabled()) return; diff --git a/security/Kconfig.hardening b/security/Kconfig.hardening index ded4d7c0d1322..d6cce64899d13 100644 --- a/security/Kconfig.hardening +++ b/security/Kconfig.hardening @@ -106,6 +106,7 @@ choice config INIT_STACK_ALL_PATTERN bool "pattern-init everything (strongest)" depends on CC_HAS_AUTO_VAR_INIT_PATTERN + depends on !KMSAN help Initializes everything on the stack (including padding) with a specific debug value. This is intended to eliminate @@ -124,6 +125,7 @@ choice config INIT_STACK_ALL_ZERO bool "zero-init everything (strongest and safest)" depends on CC_HAS_AUTO_VAR_INIT_ZERO + depends on !KMSAN help Initializes everything on the stack (including padding) with a zero value. This is intended to eliminate all @@ -218,6 +220,7 @@ config STACKLEAK_RUNTIME_DISABLE =20 config INIT_ON_ALLOC_DEFAULT_ON bool "Enable heap memory zeroing on allocation by default" + depends on !KMSAN help This has the effect of setting "init_on_alloc=3D1" on the kernel command line. This can be disabled with "init_on_alloc=3D0". @@ -230,6 +233,7 @@ config INIT_ON_ALLOC_DEFAULT_ON =20 config INIT_ON_FREE_DEFAULT_ON bool "Enable heap memory zeroing on free by default" + depends on !KMSAN help This has the effect of setting "init_on_free=3D1" on the kernel command line. This can be disabled with "init_on_free=3D0". --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00FE0C433FE for ; Tue, 26 Apr 2022 16:48:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353402AbiDZQwB (ORCPT ); Tue, 26 Apr 2022 12:52:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43280 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353352AbiDZQu7 (ORCPT ); Tue, 26 Apr 2022 12:50:59 -0400 Received: from mail-ej1-x649.google.com (mail-ej1-x649.google.com [IPv6:2a00:1450:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2110048331 for ; Tue, 26 Apr 2022 09:45:55 -0700 (PDT) Received: by mail-ej1-x649.google.com with SMTP id sa27-20020a1709076d1b00b006e8b357a2e7so9327244ejc.14 for ; Tue, 26 Apr 2022 09:45:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=UzBVkSVOyNRYl4KHuto6hQklAwc82x2pMcEIZC5XiS8=; b=owv8rcv7uy/IjMK2oDHnzdeRikUPc32v+DR6vPabqmMBgOdq0UhvmmpjftMy8ytPNt HM3P5/DQ+pT5uKlWjCFQLCWuI9NAXILUzfH3+gwTr92gjb0OskwwBjPetTXSTIPD/6OL 3KUV5a7F0Rw16/K17cTVV6HnNmzJ94eEPwwrT81RmiU5XvUiR8m/dPbUUMv64bYcnrfS 8JLEMyNSW8k05BHKUBKubnloTQr7UOc9bvvDZfi1MjSOdb0FNIpfyPVgIpYfdedOBZYB K6n0Di2ypZYYijr3xtes9AkKQrrbDkPvtqX7ow2f4ddywq7YCN1BIaM7a/np99bjjIjy k1tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=UzBVkSVOyNRYl4KHuto6hQklAwc82x2pMcEIZC5XiS8=; b=UI9xdGGYqsUHW/N+TBTrdbzEHGnF6f/FBglyFP3+tQegLU9uzQ7dCBrfnDJXnE1aoO VX+1pEV+JJmWR6PxahKYuASVhgLsKLoGyf+tizs5OkoqW1wqqN5rNxLvMHrIX6+mmqIN KKMci27H5Ak5g/XX6X6450KI8PDOzFRfL3yOrE69ZMPu5D6TULiofs/CMJliulDI5UMe KUOXjYk/xRfkK9PFr6MzO2KK/AOxG5+4z7Ry57JR9dkRPF/pPD9mGYGQ5sjxrfKPUcyD jMKyRAAn3cDFQmB3Wdgw+jLcJbaeVk+tCZUTS1C3aFR6KEZZql4+UXQbV9y/tt0jOt6O oSYA== X-Gm-Message-State: AOAM530RjUvi46TQDPvm8N3hk9sZHtcBIml127apYEfTDd3/USr1v8DW NCyBQg3uXKilTyLkvDIPinMxRF/a4rI= X-Google-Smtp-Source: ABdhPJyuq9+TYZNExRenypS/KX++WGny+sO+6oBDx1c907ZTGsidHormgyYiaYHSTUT0xaavjbt6mR9vHwU= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:54:b0:419:9b58:e305 with SMTP id f20-20020a056402005400b004199b58e305mr25365353edu.158.1650991553606; Tue, 26 Apr 2022 09:45:53 -0700 (PDT) Date: Tue, 26 Apr 2022 18:43:05 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-37-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 36/46] objtool: kmsan: list KMSAN API functions as uaccess-safe From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" KMSAN inserts API function calls in a lot of places (function entries and exits, local variables, memory accesses), so they may get called from the uaccess regions as well. KMSAN API functions are used to update the metadata (shadow/origin pages) for kernel memory accesses. The metadata pages for kernel pointers are also located in the kernel memory, so touching them is not a problem. For userspace pointers, no metadata is allocated. If an API function is supposed to read or modify the metadata, it does so for kernel pointers and ignores userspace pointers. If an API function is supposed to return a pair of metadata pointers for the instrumentation to use (like all __msan_metadata_ptr_for_TYPE_SIZE() functions do), it returns the allocated metadata for kernel pointers and special dummy buffers residing in the kernel memory for userspace pointers. As a result, none of KMSAN API functions perform userspace accesses, but since they might be called from UACCESS regions they use user_access_save/restore(). Signed-off-by: Alexander Potapenko --- v3: -- updated the patch description Link: https://linux-review.googlesource.com/id/I242bc9816273fecad4ea3d97739= 3784396bb3c35 --- tools/objtool/check.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/tools/objtool/check.c b/tools/objtool/check.c index bd0c2c828940a..44825a96adc7c 100644 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -1008,6 +1008,25 @@ static const char *uaccess_safe_builtin[] =3D { "__sanitizer_cov_trace_cmp4", "__sanitizer_cov_trace_cmp8", "__sanitizer_cov_trace_switch", + /* KMSAN */ + "kmsan_copy_to_user", + "kmsan_report", + "kmsan_unpoison_memory", + "__msan_chain_origin", + "__msan_get_context_state", + "__msan_instrument_asm_store", + "__msan_metadata_ptr_for_load_1", + "__msan_metadata_ptr_for_load_2", + "__msan_metadata_ptr_for_load_4", + "__msan_metadata_ptr_for_load_8", + "__msan_metadata_ptr_for_load_n", + "__msan_metadata_ptr_for_store_1", + "__msan_metadata_ptr_for_store_2", + "__msan_metadata_ptr_for_store_4", + "__msan_metadata_ptr_for_store_8", + "__msan_metadata_ptr_for_store_n", + "__msan_poison_alloca", + "__msan_warning", /* UBSAN */ "ubsan_type_mismatch_common", "__ubsan_handle_type_mismatch", --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74518C433EF for ; Tue, 26 Apr 2022 16:48:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353286AbiDZQvq (ORCPT ); Tue, 26 Apr 2022 12:51:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44228 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353354AbiDZQu7 (ORCPT ); Tue, 26 Apr 2022 12:50:59 -0400 Received: from mail-ed1-x549.google.com (mail-ed1-x549.google.com [IPv6:2a00:1450:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B11C34833B for ; Tue, 26 Apr 2022 09:45:57 -0700 (PDT) Received: by mail-ed1-x549.google.com with SMTP id dn26-20020a05640222fa00b00425e4b8efa9so3782638edb.1 for ; Tue, 26 Apr 2022 09:45:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Fx98fFr6n4NVKGBzav2AZaWX3VRzhlrRB+7m3pCHiSE=; b=XHfBp7Cl9xVO2N8a97yhZbWVDGYXssOGKoCjey19oDLs3NuPvKFqJlX77OVyMj94Gt 2jjyaOrHBMMVqPlV8iHx1esMbi+0FDIhnzF7Cykz5C7cq/uxOyLSo3F2oQbehHxHtWWq J0Gr9w+xpCyeNSmgjOjL6MaF2r1Jb+P68Kxo24lt3IECVMYNwPiYSWt8f1OWxVrhhtpd NO15cOsJnLjPuQEWdNOcUO0Xa9QVwX7bjyI3Sb7JSEshKJQJE+Pg0L1D80PFDwJ2uTlH nrRZFhYPzn8Skr3KSaxneh2u7R3qXEV++ExcIhKQSHekMMa2AQX+GQFmuIwyxjASi21j HPqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Fx98fFr6n4NVKGBzav2AZaWX3VRzhlrRB+7m3pCHiSE=; b=CP1RoppJKCsVGHZz1UW9wZff7LQD6AsJW41JGTZ2fXUsJ2vAIY5EKJHK1XnkRQYwZS DWEoTSVL3wk2a0LH32tAGx6tJgNA/V+7KBeU+PhrO0stolESH8yWrEZOxSnTpo5EX4cA LvsyDAxtWiupy0+r4fg1qXVN55XhEZOoYL0PJ2okTj++NHgEA1KDMIFgX+qlpbc44dYY jPbSsHeP5WWNojTRCrtrf7j/AHoJ2RaAT9KN88d2yCThhYo19KPt4dSLNpBRK51MWuSJ 9trFFkzrVQV8QftcYZEe5oDNzIXPjOUisSKqy/Op2xI283H6UTgSCrRLPFlA+5jMJjwU wY7w== X-Gm-Message-State: AOAM533t8dpP10I+a1DjE3bJbzp0j3MSQ8LYxAAMEz5+X7IJhATV1y0V WYCDdNEMD9cs7ALsrP0pSs8l5kXMnqo= X-Google-Smtp-Source: ABdhPJz3WWkOPsikmC1Af3Hb7X8sg1uvJjDWagM85J5s9I3Wdek2Bn8waW8w9Db/kDe/JuhNTPmAIxhWeNI= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:34d2:b0:423:e6c4:3e9 with SMTP id w18-20020a05640234d200b00423e6c403e9mr26328332edc.372.1650991556120; Tue, 26 Apr 2022 09:45:56 -0700 (PDT) Date: Tue, 26 Apr 2022 18:43:06 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-38-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 37/46] x86: kmsan: make READ_ONCE_TASK_STACK() return initialized values From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To avoid false positives, assume that reading from the task stack always produces initialized values. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/I9e2350bf3e88688dd83537e12a2= 3456480141997 --- arch/x86/include/asm/unwind.h | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/arch/x86/include/asm/unwind.h b/arch/x86/include/asm/unwind.h index 7cede4dc21f00..87acc90875b74 100644 --- a/arch/x86/include/asm/unwind.h +++ b/arch/x86/include/asm/unwind.h @@ -128,18 +128,19 @@ unsigned long unwind_recover_ret_addr(struct unwind_s= tate *state, } =20 /* - * This disables KASAN checking when reading a value from another task's s= tack, - * since the other task could be running on another CPU and could have poi= soned - * the stack in the meantime. + * This disables KASAN/KMSAN checking when reading a value from another ta= sk's + * stack, since the other task could be running on another CPU and could h= ave + * poisoned the stack in the meantime. Frame pointers are uninitialized by + * default, so for KMSAN we mark the return value initialized unconditiona= lly. */ -#define READ_ONCE_TASK_STACK(task, x) \ -({ \ - unsigned long val; \ - if (task =3D=3D current) \ - val =3D READ_ONCE(x); \ - else \ - val =3D READ_ONCE_NOCHECK(x); \ - val; \ +#define READ_ONCE_TASK_STACK(task, x) \ +({ \ + unsigned long val; \ + if (task =3D=3D current && !IS_ENABLED(CONFIG_KMSAN)) \ + val =3D READ_ONCE(x); \ + else \ + val =3D READ_ONCE_NOCHECK(x); \ + val; \ }) =20 static inline bool task_on_another_cpu(struct task_struct *task) --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F417BC433F5 for ; Tue, 26 Apr 2022 16:48:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353326AbiDZQvv (ORCPT ); Tue, 26 Apr 2022 12:51:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45480 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353357AbiDZQu7 (ORCPT ); Tue, 26 Apr 2022 12:50:59 -0400 Received: from mail-lj1-x24a.google.com (mail-lj1-x24a.google.com [IPv6:2a00:1450:4864:20::24a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC644222B7 for ; Tue, 26 Apr 2022 09:46:00 -0700 (PDT) Received: by mail-lj1-x24a.google.com with SMTP id v1-20020a2e2f01000000b0024da499933dso4805846ljv.19 for ; Tue, 26 Apr 2022 09:46:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=W69+Dls+nKFAD4lCWK/D6+Hgr/p3mJKYHKVaWr09RwM=; b=owm+BOhtu0whEqeXMLhBmz/yzdgp3Ss1TLbeF+Aik4tkycnXKmYZ3n1neSo8hb18hb BX0hjeRGCN6xXxA1vzrvFH0v9CoRUCtTXQLKrHwmCu+y1ae44cMIL85bcgh0tq9Lx1ES uePdvpJnw//wlpuLMvMpt4JuP6ldvIkHR8UsAm0u7Ixqgiie37sR8o/J92rLVnj0THtN xbPhuRh0HxLLsvpwoO6qui2q+WyPa1s+kNb5h9WxXtLlSB5Ib2oPyXAg/DQqG7CPduQ+ iUtgABXZYumPoT6y5CGoxk/KW2xPtKCKp9d94Q7dOcFmZ/oeIYyV2FFgehwWZr7RwBiW UUTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=W69+Dls+nKFAD4lCWK/D6+Hgr/p3mJKYHKVaWr09RwM=; b=pfKbLaUUYrtilkcAmpNLK9tziQ7vh+z015G9O7g67VbfhcZ5lAtphe2KEn1LssUwtF bYzUrSvYEWF5nyvBfmUfsaSCZwo1vXBsaf3lAPdIDKwq8ac8LLuIlx5kb88fyK2OGSeD zpTkPWTlKKyc3MCi0k6ppVymR6gn4O+yhqV2qI9OV+Ze1D7yhXS1Wnie9p1lETs1eAgz UpM0VS1LH8E+L+m1Y6pqIrDHR/nSnZWeK7JvXYXcMkA7EP6H+YJnZhJFxmmFgkaMn6Qx Qzkgr+Eo88/5w+7dJgPZG+gzIDspBhXNtFW+to41d97o/YnotVOD+MfhLZMufHlGWRpg oOnQ== X-Gm-Message-State: AOAM533+aHqb5y0kyZHyLW38pnvmwOs10iVM4FU7VBhUX7AVW2syBPtV AZidO5cxoZuq6ldvayd30kxz5flP24Q= X-Google-Smtp-Source: ABdhPJwEO/56WzEvUU6hiW7HK6Xy45a+LFdxcYVfERHbSU9DhLBh7fMmnT9aTpw3X2NOrWJtIFseLCU8PIU= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6512:25a4:b0:471:fbe9:8893 with SMTP id bf36-20020a05651225a400b00471fbe98893mr11183684lfb.147.1650991558811; Tue, 26 Apr 2022 09:45:58 -0700 (PDT) Date: Tue, 26 Apr 2022 18:43:07 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-39-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 38/46] x86: kmsan: disable instrumentation of unsupported code From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Instrumenting some files with KMSAN will result in kernel being unable to link, boot or crashing at runtime for various reasons (e.g. infinite recursion caused by instrumentation hooks calling instrumented code again). Completely omit KMSAN instrumentation in the following places: - arch/x86/boot and arch/x86/realmode/rm, as KMSAN doesn't work for i386; - arch/x86/entry/vdso, which isn't linked with KMSAN runtime; - three files in arch/x86/kernel - boot problems; - arch/x86/mm/cpu_entry_area.c - recursion. Signed-off-by: Alexander Potapenko --- v2: -- moved the patch earlier in the series so that KMSAN can compile -- split off the non-x86 part into a separate patch v3: -- added a comment to lib/Makefile Link: https://linux-review.googlesource.com/id/Id5e5c4a9f9d53c24a35ebb633b8= 14c414628d81b --- arch/x86/boot/Makefile | 1 + arch/x86/boot/compressed/Makefile | 1 + arch/x86/entry/vdso/Makefile | 3 +++ arch/x86/kernel/Makefile | 2 ++ arch/x86/kernel/cpu/Makefile | 1 + arch/x86/mm/Makefile | 2 ++ arch/x86/realmode/rm/Makefile | 1 + lib/Makefile | 2 ++ 8 files changed, 13 insertions(+) diff --git a/arch/x86/boot/Makefile b/arch/x86/boot/Makefile index b5aecb524a8aa..d5623232b763f 100644 --- a/arch/x86/boot/Makefile +++ b/arch/x86/boot/Makefile @@ -12,6 +12,7 @@ # Sanitizer runtimes are unavailable and cannot be linked for early boot c= ode. KASAN_SANITIZE :=3D n KCSAN_SANITIZE :=3D n +KMSAN_SANITIZE :=3D n OBJECT_FILES_NON_STANDARD :=3D y =20 # Kernel does not boot with kcov instrumentation here. diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/M= akefile index 6115274fe10fc..6e2e34d2655ce 100644 --- a/arch/x86/boot/compressed/Makefile +++ b/arch/x86/boot/compressed/Makefile @@ -20,6 +20,7 @@ # Sanitizer runtimes are unavailable and cannot be linked for early boot c= ode. KASAN_SANITIZE :=3D n KCSAN_SANITIZE :=3D n +KMSAN_SANITIZE :=3D n OBJECT_FILES_NON_STANDARD :=3D y =20 # Prevents link failures: __sanitizer_cov_trace_pc() is not linked in. diff --git a/arch/x86/entry/vdso/Makefile b/arch/x86/entry/vdso/Makefile index 693f8b9031fb8..4f835eaa03ec1 100644 --- a/arch/x86/entry/vdso/Makefile +++ b/arch/x86/entry/vdso/Makefile @@ -11,6 +11,9 @@ include $(srctree)/lib/vdso/Makefile =20 # Sanitizer runtimes are unavailable and cannot be linked here. KASAN_SANITIZE :=3D n +KMSAN_SANITIZE_vclock_gettime.o :=3D n +KMSAN_SANITIZE_vgetcpu.o :=3D n + UBSAN_SANITIZE :=3D n KCSAN_SANITIZE :=3D n OBJECT_FILES_NON_STANDARD :=3D y diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index c41ef42adbe8a..fcbf6cf875a90 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -33,6 +33,8 @@ KASAN_SANITIZE_sev.o :=3D n # With some compiler versions the generated code results in boot hangs, ca= used # by several compilation units. To be safe, disable all instrumentation. KCSAN_SANITIZE :=3D n +KMSAN_SANITIZE_head$(BITS).o :=3D n +KMSAN_SANITIZE_nmi.o :=3D n =20 OBJECT_FILES_NON_STANDARD_test_nx.o :=3D y =20 diff --git a/arch/x86/kernel/cpu/Makefile b/arch/x86/kernel/cpu/Makefile index 9661e3e802be5..f10a921ee7565 100644 --- a/arch/x86/kernel/cpu/Makefile +++ b/arch/x86/kernel/cpu/Makefile @@ -12,6 +12,7 @@ endif # If these files are instrumented, boot hangs during the first second. KCOV_INSTRUMENT_common.o :=3D n KCOV_INSTRUMENT_perf_event.o :=3D n +KMSAN_SANITIZE_common.o :=3D n =20 # As above, instrumenting secondary CPU boot code causes boot hangs. KCSAN_SANITIZE_common.o :=3D n diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index fe3d3061fc116..ada726784012f 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -12,6 +12,8 @@ KASAN_SANITIZE_mem_encrypt_identity.o :=3D n # Disable KCSAN entirely, because otherwise we get warnings that some func= tions # reference __initdata sections. KCSAN_SANITIZE :=3D n +# Avoid recursion by not calling KMSAN hooks for CEA code. +KMSAN_SANITIZE_cpu_entry_area.o :=3D n =20 ifdef CONFIG_FUNCTION_TRACER CFLAGS_REMOVE_mem_encrypt.o =3D -pg diff --git a/arch/x86/realmode/rm/Makefile b/arch/x86/realmode/rm/Makefile index 83f1b6a56449f..f614009d3e4e2 100644 --- a/arch/x86/realmode/rm/Makefile +++ b/arch/x86/realmode/rm/Makefile @@ -10,6 +10,7 @@ # Sanitizer runtimes are unavailable and cannot be linked here. KASAN_SANITIZE :=3D n KCSAN_SANITIZE :=3D n +KMSAN_SANITIZE :=3D n OBJECT_FILES_NON_STANDARD :=3D y =20 # Prevents link failures: __sanitizer_cov_trace_pc() is not linked in. diff --git a/lib/Makefile b/lib/Makefile index caeb55f661726..444c961f2f2e1 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -269,6 +269,8 @@ obj-$(CONFIG_IRQ_POLL) +=3D irq_poll.o CFLAGS_stackdepot.o +=3D -fno-builtin obj-$(CONFIG_STACKDEPOT) +=3D stackdepot.o KASAN_SANITIZE_stackdepot.o :=3D n +# In particular, instrumenting stackdepot.c with KMSAN will result in infi= nite +# recursion. KMSAN_SANITIZE_stackdepot.o :=3D n KCOV_INSTRUMENT_stackdepot.o :=3D n =20 --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 288E6C433EF for ; Tue, 26 Apr 2022 16:48:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353395AbiDZQv6 (ORCPT ); Tue, 26 Apr 2022 12:51:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45498 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353360AbiDZQvA (ORCPT ); Tue, 26 Apr 2022 12:51:00 -0400 Received: from mail-ed1-x549.google.com (mail-ed1-x549.google.com [IPv6:2a00:1450:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A0B24833A for ; Tue, 26 Apr 2022 09:46:03 -0700 (PDT) Received: by mail-ed1-x549.google.com with SMTP id b24-20020a50e798000000b0041631767675so10630033edn.23 for ; Tue, 26 Apr 2022 09:46:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=XLE4dDKZQivTtEUon/Tj4SL+cxx+nqfawlbFCYGovJc=; b=JMsNnrgA3RJaeGbSFsIbMJk1WKHAyC9mHh9w+Ezp2I0AryPzMo0hvP3hFHcM3Fes/j 0Od6aTn5hLCVFSLoC+QEN/Y51yJ/k3To1PPqp71oGx9SidpVIg7W/FFigKzrWKt5pZNf ObR3IfkU591CrM0QCB8Yyv69AzGBOduai/CQ5ZELgXkxhQBpzhytZFiVH0+MwZEQnsa3 V3yvl4lBVXMYXdY61XOr/QmKRJgkZHWIaUhgcwCtzjh2gZfAxHgEOuB8kzeynle/cRhb dXMQ2W1fkb0VB8Nx+GeffYe9rusz/mxzvp3Q0HOhGVl2s14bRkzEjCgm6Hi/UlxZ9QPZ fd4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=XLE4dDKZQivTtEUon/Tj4SL+cxx+nqfawlbFCYGovJc=; b=5Ra/e4QlsN++eNV7i8TFW51YOuE0yFvEPt17f7v2T3wDVKSPznn0hsiDb/6arspLa8 7Z2sF4S6MX0zs9c6KgAC4tudtqSCVwP4RZnFFbm+Gnvko3r8mIekfdeuhHaPob+rIEBy uvYcyyqd9DKgKulopVs8NBvwxxsHikmHs4lIygx49qKTux+fhA7HESfSVXddJuIGuUx1 jP6uFtsvlQKFnqp9vE+CEYn43TN3Cz6D8VTqKtVjn4l1biHbOwWdrGyFK8F29ETvGq0s cFVxGRrgIkaQL8Ey0ifqd7hSJfeD6Kc7xoxH7NHx6EeVU9HHtmF35XaDJI7sPA5U682U Dapw== X-Gm-Message-State: AOAM532u3/MuPqCHtXDbC/0Wwe5nmOofD5vHhP0V7VVo2eRRC3YsqY58 ko5kD4KQV90LT2SffHF+sKv+uvEEhAE= X-Google-Smtp-Source: ABdhPJw2dp1WxDXIXbxKtYrp2TPHwfmzrgTwJgDoJzBSZTqGhvOW8ABWyU8M8nHDx8YckvsgOVfYsGY3TJA= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a17:907:3e21:b0:6f3:bd59:1aa0 with SMTP id hp33-20020a1709073e2100b006f3bd591aa0mr1461947ejc.682.1650991561485; Tue, 26 Apr 2022 09:46:01 -0700 (PDT) Date: Tue, 26 Apr 2022 18:43:08 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-40-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 39/46] x86: kmsan: skip shadow checks in __switch_to() From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When instrumenting functions, KMSAN obtains the per-task state (mostly pointers to metadata for function arguments and return values) once per function at its beginning, using the `current` pointer. Every time the instrumented function calls another function, this state (`struct kmsan_context_state`) is updated with shadow/origin data of the passed and returned values. When `current` changes in the low-level arch code, instrumented code can not notice that, and will still refer to the old state, possibly corrupting it or using stale data. This may result in false positive reports. To deal with that, we need to apply __no_kmsan_checks to the functions performing context switching - this will result in skipping all KMSAN shadow checks and marking newly created values as initialized, preventing all false positive reports in those functions. False negatives are still possible, but we expect them to be rare and impersistent. Suggested-by: Marco Elver Signed-off-by: Alexander Potapenko --- v2: -- This patch was previously called "kmsan: skip shadow checks in files doing context switches". Per Mark Rutland's suggestion, we now only skip checks in low-level arch-specific code, as context switches in common code should be invisible to KMSAN. We also apply the checks to precisely the functions performing the context switch instead of the whole file. Link: https://linux-review.googlesource.com/id/I45e3ed9c5f66ee79b0409d1673d= 66ae419029bcb Replace KMSAN_ENABLE_CHECKS_process_64.o with __no_kmsan_checks --- arch/x86/kernel/process_64.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index e459253649be2..9952a4c7e1d20 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -553,6 +553,7 @@ void compat_start_thread(struct pt_regs *regs, u32 new_= ip, u32 new_sp, bool x32) * Kprobes not supported here. Set the probe on schedule instead. * Function graph tracer not supported too. */ +__no_kmsan_checks __visible __notrace_funcgraph struct task_struct * __switch_to(struct task_struct *prev_p, struct task_struct *next_p) { --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A427C433F5 for ; Tue, 26 Apr 2022 16:48:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353390AbiDZQvz (ORCPT ); Tue, 26 Apr 2022 12:51:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43222 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353252AbiDZQvA (ORCPT ); Tue, 26 Apr 2022 12:51:00 -0400 Received: from mail-ed1-x54a.google.com (mail-ed1-x54a.google.com [IPv6:2a00:1450:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E85D0289A7 for ; Tue, 26 Apr 2022 09:46:05 -0700 (PDT) Received: by mail-ed1-x54a.google.com with SMTP id t25-20020a508d59000000b00425d86c2987so4109343edt.21 for ; Tue, 26 Apr 2022 09:46:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=xs3mnLw1SF/DZ/Tq7NTZzrdEtEEoGjj1tkhOxGSNVPQ=; b=Dp8fDgewCtm7EfJ+NxSWPaqf4fjHXCb5jsKUHsqjOQlRwQkgkKZ/UWCSiVVrRs/uZW OQRNetbiqEwnG85Z0p1Sf25DydKgfbA8aQ0Bsb8u1giNx3oJsml+h06KXd12ZFl+7zlh 88EaOBCQpujgyWZhEQULbY92x5FazQAo2pGdWFAwltYHX4X7vprtu3R+YlyuxLW4bV7A kz4jOXTPx7d0TC785iMRbaV+fkvSej+QsvFkw8jbYuayp+vHkk5UYhSq7fmBnGbZjeRh ej2EyGUQvI2W9wRBjeiQzBogyryUJQVM1Tt85ycKl00a26JFtroHDFnq/8e7RZbUcfE0 4QnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=xs3mnLw1SF/DZ/Tq7NTZzrdEtEEoGjj1tkhOxGSNVPQ=; b=Mt3yn8Ct68Yw+ddK6FI3m6vUK0dUU1YdxsUTByt35K4XtQxeXCWd96MBC1oNA/xL+E JRJxkWADsQaB6PEvHILa4+6qD+DL7ZdTkAz5keOpMIxa2oXa+7bl9wxLmDw8TDIvFME6 i6qvaYgc+X2Jj6o9kclQ8plvVTog7uy8T+TEVRF+IvZbB7bkPcGE28VuOsIEQXOh4y6j etbMhMNHUmZaU3ZwPEh0u2xqgE/Rx+f/H0CMxMnSVcaflYcu/UlBCcbq0aHEkjQtuVAZ tBeWktuEXrGylY7t6xeXOlXCDAzoJ+mxlxB/POy+Dr1yXgeUcAPIACdxcsFPHmMW493w sYOQ== X-Gm-Message-State: AOAM533skDPiK/4SOVnfu44P1S9/Pl4j6rddhQKnLKTFwE12Bk+fJnUa wI+RZjPWF2BX9LPqHBksaaQw8Qrz+BQ= X-Google-Smtp-Source: ABdhPJyVycewaYAOLWq91OXZaqwuaaJYgCPy/6P6fQ6DCJX8eHTBHY2JtlW8ZAI35LArwnFF6hd2OCloeNo= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:aa7:c70f:0:b0:425:f70d:b34 with SMTP id i15-20020aa7c70f000000b00425f70d0b34mr7131646edq.306.1650991564200; Tue, 26 Apr 2022 09:46:04 -0700 (PDT) Date: Tue, 26 Apr 2022 18:43:09 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-41-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 40/46] x86: kmsan: handle open-coded assembly in lib/iomem.c From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" KMSAN cannot intercept memory accesses within asm() statements. That's why we add kmsan_unpoison_memory() and kmsan_check_memory() to hint it how to handle memory copied from/to I/O memory. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/Icb16bf17269087e475debf07a7f= e7d4bebc3df23 --- arch/x86/lib/iomem.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/x86/lib/iomem.c b/arch/x86/lib/iomem.c index 3e2f33fc33de2..e0411a3774d49 100644 --- a/arch/x86/lib/iomem.c +++ b/arch/x86/lib/iomem.c @@ -1,6 +1,7 @@ #include #include #include +#include =20 #define movs(type,to,from) \ asm volatile("movs" type:"=3D&D" (to), "=3D&S" (from):"0" (to), "1" (from= ):"memory") @@ -37,6 +38,8 @@ static void string_memcpy_fromio(void *to, const volatile= void __iomem *from, si n-=3D2; } rep_movs(to, (const void *)from, n); + /* KMSAN must treat values read from devices as initialized. */ + kmsan_unpoison_memory(to, n); } =20 static void string_memcpy_toio(volatile void __iomem *to, const void *from= , size_t n) @@ -44,6 +47,8 @@ static void string_memcpy_toio(volatile void __iomem *to,= const void *from, size if (unlikely(!n)) return; =20 + /* Make sure uninitialized memory isn't copied to devices. */ + kmsan_check_memory(from, n); /* Align any unaligned destination IO */ if (unlikely(1 & (unsigned long)to)) { movs("b", to, from); --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37BF2C433F5 for ; Tue, 26 Apr 2022 16:49:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353467AbiDZQwN (ORCPT ); Tue, 26 Apr 2022 12:52:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43218 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353366AbiDZQvB (ORCPT ); Tue, 26 Apr 2022 12:51:01 -0400 Received: from mail-ed1-x54a.google.com (mail-ed1-x54a.google.com [IPv6:2a00:1450:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9131E48391 for ; Tue, 26 Apr 2022 09:46:08 -0700 (PDT) Received: by mail-ed1-x54a.google.com with SMTP id l24-20020a056402231800b00410f19a3103so10612978eda.5 for ; Tue, 26 Apr 2022 09:46:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=+FBFHZ/D2CVM8zHnxvAf38XdMkZWjnGIg8UqYOIGTJc=; b=eLrL3mhFQT5kOKuWWPsXYa6Om2vKcm1WdEFQHStYaEyvCoapAOY8AFslQqrxPfYmIf rXp6G49IKCxSIadhwDU/svB9bjlSTioT2d+8nFtnyoxtGkFZ5hU6IZRnk3EvL9+LHqKl YImrZr+ZfLnQxJu37Qjyj/MDxN8AmGTCABbxmGUID5Rdgt4TWVzw9Gx7nZ/Cv3t0o4mj 7IL5zxZSeAJwNc5HAOQNmnltHzHmQdS3We85em43fb3OAi9L+VU3jozzjW4Z5BOBjLbl 9zuRJUkSroGkn/RtJMaMYsf10xB5Km6ymFu+cfmKhvrnUfSPObf8M//QzCvTkk4jAHil C3cA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=+FBFHZ/D2CVM8zHnxvAf38XdMkZWjnGIg8UqYOIGTJc=; b=IlI5NmikgGvl440+MzSypJNs0cCf9GrU7vqCyYX/Z58jMfvO9AAIzd1Xbzz+6z/X2r tQ7xlyU2e+t+gntxDQzelI9c4hLd0bFYkxOrO3e7nRb4WOU15UL8VBBbCgUCb2hHRDCK BqNUKar/vOFfrqAkVBmIk+JNueV2XQXKhwquJ6Dc0R7H5u8SIEjrAPO2/VU9Rf3YluzK W+RTFFxuYzbYROaWwO3T5dNdqWwawezy81/Lvoc+3U17YYKdWjeAdvTjuY9TZJJsEHWZ QZ/UCdQlldjPmVmGXNfrfDpCllry6Q8gCsGQrTQoEfpYFc62NAAE6vK8o3GjHtF0E8PF 2i5A== X-Gm-Message-State: AOAM531LWxqsn39OHMNOMK0u6no4tPDjW1bMm3asMcXZ0KaefCbH+Emh q9vq/h03ix7IEeQ6Mkv8VpL9RcnWWgc= X-Google-Smtp-Source: ABdhPJzl2WkvDMC8oytisA9bFCUfSdKFjxjddbCsinvPLHHRa3ZNDj4UDz7djBQ5AhjJmhi+1lV1TeExHmQ= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:aa7:c793:0:b0:408:4a69:90b4 with SMTP id n19-20020aa7c793000000b004084a6990b4mr25741991eds.58.1650991567128; Tue, 26 Apr 2022 09:46:07 -0700 (PDT) Date: Tue, 26 Apr 2022 18:43:10 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-42-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 41/46] x86: kmsan: use __msan_ string functions where possible. From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Unless stated otherwise (by explicitly calling __memcpy(), __memset() or __memmove()) we want all string functions to call their __msan_ versions (e.g. __msan_memcpy() instead of memcpy()), so that shadow and origin values are updated accordingly. Bootloader must still use the default string functions to avoid crashes. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/I7ca9bd6b4f5c9b9816404862ae8= 7ca7984395f33 --- arch/x86/include/asm/string_64.h | 23 +++++++++++++++++++++-- include/linux/fortify-string.h | 2 ++ 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/string_64.h b/arch/x86/include/asm/string= _64.h index 6e450827f677a..3b87d889b6e16 100644 --- a/arch/x86/include/asm/string_64.h +++ b/arch/x86/include/asm/string_64.h @@ -11,11 +11,23 @@ function. */ =20 #define __HAVE_ARCH_MEMCPY 1 +#if defined(__SANITIZE_MEMORY__) +#undef memcpy +void *__msan_memcpy(void *dst, const void *src, size_t size); +#define memcpy __msan_memcpy +#else extern void *memcpy(void *to, const void *from, size_t len); +#endif extern void *__memcpy(void *to, const void *from, size_t len); =20 #define __HAVE_ARCH_MEMSET +#if defined(__SANITIZE_MEMORY__) +extern void *__msan_memset(void *s, int c, size_t n); +#undef memset +#define memset __msan_memset +#else void *memset(void *s, int c, size_t n); +#endif void *__memset(void *s, int c, size_t n); =20 #define __HAVE_ARCH_MEMSET16 @@ -55,7 +67,13 @@ static inline void *memset64(uint64_t *s, uint64_t v, si= ze_t n) } =20 #define __HAVE_ARCH_MEMMOVE +#if defined(__SANITIZE_MEMORY__) +#undef memmove +void *__msan_memmove(void *dest, const void *src, size_t len); +#define memmove __msan_memmove +#else void *memmove(void *dest, const void *src, size_t count); +#endif void *__memmove(void *dest, const void *src, size_t count); =20 int memcmp(const void *cs, const void *ct, size_t count); @@ -64,8 +82,7 @@ char *strcpy(char *dest, const char *src); char *strcat(char *dest, const char *src); int strcmp(const char *cs, const char *ct); =20 -#if defined(CONFIG_KASAN) && !defined(__SANITIZE_ADDRESS__) - +#if (defined(CONFIG_KASAN) && !defined(__SANITIZE_ADDRESS__)) /* * For files that not instrumented (e.g. mm/slub.c) we * should use not instrumented version of mem* functions. @@ -73,7 +90,9 @@ int strcmp(const char *cs, const char *ct); =20 #undef memcpy #define memcpy(dst, src, len) __memcpy(dst, src, len) +#undef memmove #define memmove(dst, src, len) __memmove(dst, src, len) +#undef memset #define memset(s, c, n) __memset(s, c, n) =20 #ifndef __NO_FORTIFY diff --git a/include/linux/fortify-string.h b/include/linux/fortify-string.h index 295637a66c46b..fe48f77599e04 100644 --- a/include/linux/fortify-string.h +++ b/include/linux/fortify-string.h @@ -269,8 +269,10 @@ __FORTIFY_INLINE void fortify_memset_chk(__kernel_size= _t size, * __builtin_object_size() must be captured here to avoid evaluating argum= ent * side-effects further into the macro layers. */ +#ifndef CONFIG_KMSAN #define memset(p, c, s) __fortify_memset_chk(p, c, s, \ __builtin_object_size(p, 0), __builtin_object_size(p, 1)) +#endif =20 /* * To make sure the compiler can enforce protection against buffer overflo= ws, --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CF06C433EF for ; Tue, 26 Apr 2022 16:49:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345607AbiDZQwG (ORCPT ); Tue, 26 Apr 2022 12:52:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46936 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353368AbiDZQvB (ORCPT ); Tue, 26 Apr 2022 12:51:01 -0400 Received: from mail-ej1-x64a.google.com (mail-ej1-x64a.google.com [IPv6:2a00:1450:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1CA11483A7 for ; Tue, 26 Apr 2022 09:46:11 -0700 (PDT) Received: by mail-ej1-x64a.google.com with SMTP id hp17-20020a1709073e1100b006ef69a535c3so9361546ejc.10 for ; Tue, 26 Apr 2022 09:46:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=SGOwGjYq/2PwacKh0/9At0yFevcHrzg9FDlShmi+vnk=; b=fSZ4zfvDJJmqaIzU1XL0WfStwXr6OPa8V+ga0qTz0UlsIHyG9I6Fr6ZEbs9vmLiuhO ClbMCWmC+tznTOah7YoL3txE0BplWerCcj5Pt/KeycPvVB+sF+pTOE13LG7aV8CaSZd6 CtB276fuSEgHwCIg/10uWG7rSCD98aAcUKoO+LU4U8ks8bUI5/GdxxpoNac4ZhTkb0E1 grtFS0KyEl74V0H8NCr4jzX10gmxpamQ+wZyHKObaL9HWxab3p+XWx1x2TN57EUQX6a4 MV6fXbcDTcGpFO2Dt7YM5K3PDMZSDsLSt5fDbOOZGvHfv6pU1T6bZgqLE57A40M5bQwM 0uLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=SGOwGjYq/2PwacKh0/9At0yFevcHrzg9FDlShmi+vnk=; b=pbwXyN7tnReibsJKqwWs52IdzKC83Dxm8jxPAadEOgdnDffY2RwgUXZaTZhGqby4yN /jBrj4u6nGC983Gz83EFUZLIryUJpUsKnApXUOv6yeXPkvrD6L+FYvbOWpGFRVhAjLHl c8YVIHshObi7zHcYwx8WYWbZTtNe5HFIBO+U0Rg7RDptm7AZWCHXxsZtONW7LXhKnZ/O indwk2WIIofKI4YXRNRpMMx/DRTTBJ6cudzR7fQqatol5udFBxiQJxg89f3dMmdZ0Ilr /SABi3wZghp8318AGpv+cb8ilXlYSFeLx1RDxO1oNu5DhfasVyCFAmB8LZcMJBEOlJ9/ s0RQ== X-Gm-Message-State: AOAM531nSPw5BC7xWpFhS0so7NCJqOt8fLrtYyzJ35PDHozz6cUtdfnB 1QV3P0TmVXlucFY8rKDjyhvBsxT0dlo= X-Google-Smtp-Source: ABdhPJw9NptsR6xC76SX+4t0lKzp8AK1gzotC/X9LsmKWQ/MoVeg5tjW3Iafs+1NkdThy9tu1emAPZFww8w= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:84a:b0:423:fe99:8c53 with SMTP id b10-20020a056402084a00b00423fe998c53mr25385277edz.195.1650991569448; Tue, 26 Apr 2022 09:46:09 -0700 (PDT) Date: Tue, 26 Apr 2022 18:43:11 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-43-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 42/46] x86: kmsan: sync metadata pages on page fault From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" KMSAN assumes shadow and origin pages for every allocated page are accessible. For pages between [VMALLOC_START, VMALLOC_END] those metadata pages start at KMSAN_VMALLOC_SHADOW_START and KMSAN_VMALLOC_ORIGIN_START, therefore we must sync a bigger memory region. Signed-off-by: Alexander Potapenko --- v2: -- addressed reports from kernel test robot Link: https://linux-review.googlesource.com/id/Ia5bd541e54f1ecc11b86666c3ec= 87c62ac0bdfb8 --- arch/x86/mm/fault.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index d0074c6ed31a3..f2250a32a10ca 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -260,7 +260,7 @@ static noinline int vmalloc_fault(unsigned long address) } NOKPROBE_SYMBOL(vmalloc_fault); =20 -void arch_sync_kernel_mappings(unsigned long start, unsigned long end) +static void __arch_sync_kernel_mappings(unsigned long start, unsigned long= end) { unsigned long addr; =20 @@ -284,6 +284,27 @@ void arch_sync_kernel_mappings(unsigned long start, un= signed long end) } } =20 +void arch_sync_kernel_mappings(unsigned long start, unsigned long end) +{ + __arch_sync_kernel_mappings(start, end); +#ifdef CONFIG_KMSAN + /* + * KMSAN maintains two additional metadata page mappings for the + * [VMALLOC_START, VMALLOC_END) range. These mappings start at + * KMSAN_VMALLOC_SHADOW_START and KMSAN_VMALLOC_ORIGIN_START and + * have to be synced together with the vmalloc memory mapping. + */ + if (start >=3D VMALLOC_START && end < VMALLOC_END) { + __arch_sync_kernel_mappings( + start - VMALLOC_START + KMSAN_VMALLOC_SHADOW_START, + end - VMALLOC_START + KMSAN_VMALLOC_SHADOW_START); + __arch_sync_kernel_mappings( + start - VMALLOC_START + KMSAN_VMALLOC_ORIGIN_START, + end - VMALLOC_START + KMSAN_VMALLOC_ORIGIN_START); + } +#endif +} + static bool low_pfn(unsigned long pfn) { return pfn < max_low_pfn; --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24062C433EF for ; Tue, 26 Apr 2022 16:49:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353473AbiDZQwS (ORCPT ); Tue, 26 Apr 2022 12:52:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44704 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353369AbiDZQvB (ORCPT ); Tue, 26 Apr 2022 12:51:01 -0400 Received: from mail-wm1-x34a.google.com (mail-wm1-x34a.google.com [IPv6:2a00:1450:4864:20::34a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 91AC248384 for ; Tue, 26 Apr 2022 09:46:13 -0700 (PDT) Received: by mail-wm1-x34a.google.com with SMTP id m125-20020a1c2683000000b00391893a2febso8285568wmm.4 for ; Tue, 26 Apr 2022 09:46:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=moZadDgSRgKTD5MeByPAKBz5Srz/DkBZpKo67JL3wX4=; b=cLDKsrGhoorGl3SRzelVo1PAO7x1RBOazkv5nVbG4NDdcGaRorPOu46pd1i1ETj6hN h0UoOHWi4fDikLK9zi9pBoTbj7p+CVMiVtA02QDHib8yzyjhlFhzlbs1tKotmp8GIt+Z r2bf7wmpqHdZBIZZAh4b+ySUHf+iXX37i29FxIhHYiYKb5VpzMQRfnvof2MvgAVVQ1ZM l3T7Qhx+uQ95eC8Y0ZpGlX0jRuN6gIrOSGpaYLHVjLemQwrYlnZaTUKaYJkL5xFg5tCv Q6JZbOy6PYMZBzBVtPmEhtlM/aUknABDbunmxBkt5+imBTxAI6SQwQQpEK8i4YNCCxax ihqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=moZadDgSRgKTD5MeByPAKBz5Srz/DkBZpKo67JL3wX4=; b=QEFsSxMlowfb1SCR8DmMvPiRUVZCZzcQC0U6UW+Qpifj7vN3GNxTfnQlE44HqMIIyB GDRFT/tRsuRqRuWlG5PnDj6mfd/1gWZd9W2tk1GHUgFroYv8WQJdF6tbBkTUovkHUQSk lmEXfzE36/4NctJreqEnacMo/wn5LbfIqcTK5sOXGtI9+pJ8y+DfUy23451ZtYxBFzKX Cg/fGFIGqgKh5QJR7VvUXe3n/4g18poEpAfEN/sr4ghmUeDU5TSbTDkWaI4bE4fsjllN kZktW2NhU9l+HGyTFsjbJvEV/nwILONrSy7QbMDje2vel3CoqRbBYgAwPdFK17vRwmTH pxFQ== X-Gm-Message-State: AOAM5321JOdyzLebWHWSw67bg+RKFu3NDsy9r59iv3Bw613nOPbWaCmx u1C44O6od0/gD9glcioqTOKub1SWC0k= X-Google-Smtp-Source: ABdhPJz0bSNwPRs9Vl+Yh0K5Bv3M9FkaOEwa9c2Exw9QMq+QGEH5+ihGPtHZ0GCKCcmwZaaizGJA4g9/QAo= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:adf:d1ce:0:b0:20a:e668:8927 with SMTP id b14-20020adfd1ce000000b0020ae6688927mr3156284wrd.508.1650991572003; Tue, 26 Apr 2022 09:46:12 -0700 (PDT) Date: Tue, 26 Apr 2022 18:43:12 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-44-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 43/46] x86: kasan: kmsan: support CONFIG_GENERIC_CSUM on x86, enable it for KASAN/KMSAN From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This is needed to allow memory tools like KASAN and KMSAN see the memory accesses from the checksum code. Without CONFIG_GENERIC_CSUM the tools can't see memory accesses originating from handwritten assembly code. For KASAN it's a question of detecting more bugs, for KMSAN using the C implementation also helps avoid false positives originating from seemingly uninitialized checksum values. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/I3e95247be55b1112af59dbba07e= 8cbf34e50a581 --- arch/x86/Kconfig | 4 ++++ arch/x86/include/asm/checksum.h | 16 ++++++++++------ arch/x86/lib/Makefile | 2 ++ 3 files changed, 16 insertions(+), 6 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index b0142e01002e3..ee5e6fd65bf1d 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -319,6 +319,10 @@ config GENERIC_ISA_DMA def_bool y depends on ISA_DMA_API =20 +config GENERIC_CSUM + bool + default y if KMSAN || KASAN + config GENERIC_BUG def_bool y depends on BUG diff --git a/arch/x86/include/asm/checksum.h b/arch/x86/include/asm/checksu= m.h index bca625a60186c..6df6ece8a28ec 100644 --- a/arch/x86/include/asm/checksum.h +++ b/arch/x86/include/asm/checksum.h @@ -1,9 +1,13 @@ /* SPDX-License-Identifier: GPL-2.0 */ -#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER 1 -#define HAVE_CSUM_COPY_USER -#define _HAVE_ARCH_CSUM_AND_COPY -#ifdef CONFIG_X86_32 -# include +#ifdef CONFIG_GENERIC_CSUM +# include #else -# include +# define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER 1 +# define HAVE_CSUM_COPY_USER +# define _HAVE_ARCH_CSUM_AND_COPY +# ifdef CONFIG_X86_32 +# include +# else +# include +# endif #endif diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile index f76747862bd2e..7ba5f61d72735 100644 --- a/arch/x86/lib/Makefile +++ b/arch/x86/lib/Makefile @@ -65,7 +65,9 @@ ifneq ($(CONFIG_X86_CMPXCHG64),y) endif else obj-y +=3D iomap_copy_64.o +ifneq ($(CONFIG_GENERIC_CSUM),y) lib-y +=3D csum-partial_64.o csum-copy_64.o csum-wrappers_64.o +endif lib-y +=3D clear_page_64.o copy_page_64.o lib-y +=3D memmove_64.o memset_64.o lib-y +=3D copy_user_64.o --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E02A1C433F5 for ; Tue, 26 Apr 2022 16:49:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353484AbiDZQw0 (ORCPT ); Tue, 26 Apr 2022 12:52:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44042 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353253AbiDZQvZ (ORCPT ); Tue, 26 Apr 2022 12:51:25 -0400 Received: from mail-ej1-x64a.google.com (mail-ej1-x64a.google.com [IPv6:2a00:1450:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D9A5488AB for ; Tue, 26 Apr 2022 09:46:16 -0700 (PDT) Received: by mail-ej1-x64a.google.com with SMTP id hr35-20020a1709073fa300b006f3647cd980so5655265ejc.5 for ; Tue, 26 Apr 2022 09:46:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=qZO+Ar5AUQjZ17sp8HnLfJ3bnZDmnQ4HtehVjR2RSag=; b=Pt5+odsaPLVLA+BIkFaOGZ5z3uWJenSne3f73BJt4PbanWkfmtsct7CN0wAO2xN5gU Yhaxk8FOCzZhswyeeTNVL8OMhm9c/T3iPCztTqmlZxSpLpgMOs6YXH5MZ/fx+97iCGvz zddbne3cp5tk6W6voA6FNsecjeKs0yW1DQ4Tz30W1QM+yA9fVauckWd7D4HpMMiv8n/u VNAOuyzoM5s6YO+8DH4/mHKvxm5uilD1LTm5hSJS1BnACQmJdUQmedoVo1JHnrQYqad/ JlxuIlepoOSV3NlP0UfaMmhcA4Fr+PZ2URCMg+I0v/ZRX2C/u5IO3EAmXGa2X6k+fRQb I4UQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=qZO+Ar5AUQjZ17sp8HnLfJ3bnZDmnQ4HtehVjR2RSag=; b=PvvQOkeWxb6tin1r/OdBpIQ8hVIbIwFgBKLY/n662Oo1Yc2gzkT3RFavBCrcUDflLs To1X4x/SsydH6Gjq4FDxgOoM7VqwnFsKE92qeV1cla2+rLg0szhkMHP461cEk36yD6ck TFM8yhVnJC1HPGjZePU2ds4uXwzpo5psB26UvrvZmakkP7hLnlkT/0uBFeQ3c3deJrFi E2LDgaPSxmWx4JfBwiy8atnQY9a6ykOa4FT6CRl3yZ1uo6IwzOvjtkmzv+Rwq9TvxZPC 3YZhoFaRJADOVgggx8HQ+eTvA64+eAD/kYhejvkot2bEKIdmS94vZAYYzE9trd0JGX2p 3ynA== X-Gm-Message-State: AOAM533z3HeKgR6SbeNYFBG0MBPxRSsc9rRYE5TAiZBaOqNqPc/iHBBY KGTB7bXjAeFsaSx1+M5g0tTlTV2vNes= X-Google-Smtp-Source: ABdhPJwC/1Ho3LFpscyKKysPm+tD7uHD3uvu1NdPCEt5vju65j1NkGkMRTYyimkaio71xSi48PC422JlIc4= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:2689:b0:422:15c4:e17e with SMTP id w9-20020a056402268900b0042215c4e17emr26075746edd.33.1650991574553; Tue, 26 Apr 2022 09:46:14 -0700 (PDT) Date: Tue, 26 Apr 2022 18:43:13 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-45-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 44/46] x86: fs: kmsan: disable CONFIG_DCACHE_WORD_ACCESS From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, Andrey Konovalov Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" dentry_string_cmp() calls read_word_at_a_time(), which might read uninitialized bytes to optimize string comparisons. Disabling CONFIG_DCACHE_WORD_ACCESS should prohibit this optimization, as well as (probably) similar ones. Suggested-by: Andrey Konovalov Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/I4c0073224ac2897cafb8c037362= c49dda9cfa133 --- arch/x86/Kconfig | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index ee5e6fd65bf1d..3209073f96415 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -128,7 +128,9 @@ config X86 select CLKEVT_I8253 select CLOCKSOURCE_VALIDATE_LAST_CYCLE select CLOCKSOURCE_WATCHDOG - select DCACHE_WORD_ACCESS + # Word-size accesses may read uninitialized data past the trailing \0 + # in strings and cause false KMSAN reports. + select DCACHE_WORD_ACCESS if !KMSAN select DYNAMIC_SIGFRAME select EDAC_ATOMIC_SCRUB select EDAC_SUPPORT --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A883C433EF for ; Tue, 26 Apr 2022 16:49:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353411AbiDZQwf (ORCPT ); Tue, 26 Apr 2022 12:52:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43308 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353383AbiDZQvZ (ORCPT ); Tue, 26 Apr 2022 12:51:25 -0400 Received: from mail-ej1-x64a.google.com (mail-ej1-x64a.google.com [IPv6:2a00:1450:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F17748E52 for ; Tue, 26 Apr 2022 09:46:18 -0700 (PDT) Received: by mail-ej1-x64a.google.com with SMTP id ne12-20020a1709077b8c00b006f3aca1f2b2so1754527ejc.17 for ; Tue, 26 Apr 2022 09:46:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Jk0NMFQ+GIouP+k3atOjYdUnEj7isi+T1qF4OKInEvY=; b=q/8YNmW2FY+UOSSefHsoVD/+tXnxdP8E+Oyh/hfdIdafdhpVQDdRSLmWDfSSmsNJKL QEvct6PQ12lO5BkzKJzp9F/jBD+fRyuCDw1W1Y+eoR9Lna4fX3FqSnRcbQNFAi9810kd 1KmXYk///irlbShN5FXTZLud1oM/MoL/kXNVhmqlso/CJrmniNY23vigbiK+1zT/oLX5 u/p0nsK9LPxyHxzvCXLM9yXvRIIjEDWc9Qa0QoPtEJBwsG2B6/rqSW8o/qagH0NO8S4i baPLMv5+6abizA7u6ohuw2x4eEH69qUDUIfIZ/8RQI8GXT+Tdau+m7YEckOF8FW3Fe6p 4Cgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Jk0NMFQ+GIouP+k3atOjYdUnEj7isi+T1qF4OKInEvY=; b=M9Q5DnhErvohRymOMrX8Nfr5KpILPC85KW7mpwh1GsbmyISIEDkykACs8RzMEAS/Uv 2ZGM7HKthXf5p7eATF8+XUPLellO9sgaNL++LH+6nwvRxMQPhMW7vNzLhS3z3fbGHj6o 6+VeiiLWNidnjcrx+fHaX/+VwpDuWnTj8xHsJSQ0fe+4d06brKyGYfHwSsCWHg7XfJji hn6lA13BivpVZpuDplFI0n4POBt5ppVg1O9yznKP41YCDcug24vzRbktOgK9Bvcy6KfO XHVH6MbfosVqgZDSo5BEQ2ClS55Wl1lSHaNPOIYPVprz+3n9vqeggMNy9NDGY5o/unmK vTeg== X-Gm-Message-State: AOAM5329ziGM2669ZBzXQ2ouI7b6135jeA3vF/9qSl8pSqfHibm06edS 2kFwCSjFWBpjcCbh9L3Q6WavggjnH8Y= X-Google-Smtp-Source: ABdhPJzs34r07s5FrFzj+GFFR4J7VeroU6awpjT3W6Wa0hn8rVzciQSq9AlsjYylNyT2KboobjGXSIMLGNU= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a05:6402:330b:b0:425:eded:7cfe with SMTP id e11-20020a056402330b00b00425eded7cfemr10281416eda.357.1650991577116; Tue, 26 Apr 2022 09:46:17 -0700 (PDT) Date: Tue, 26 Apr 2022 18:43:14 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-46-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 45/46] x86: kmsan: handle register passing from uninstrumented code From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Replace instrumentation_begin() with instrumentation_begin_with_regs() to let KMSAN handle the non-instrumented code and unpoison pt_regs passed from the instrumented part. This is done to reduce the number of false positive reports. Signed-off-by: Alexander Potapenko --- v2: -- this patch was previously called "x86: kmsan: handle register passing from uninstrumented code". Instead of adding KMSAN-specific code to every instrumentation_begin()/instrumentation_end() section, we changed instrumentation_begin() to instrumentation_begin_with_regs() where applicable. Link: https://linux-review.googlesource.com/id/I435ec076cd21752c2f877f5da81= f5eced62a2ea4 --- arch/x86/entry/common.c | 3 ++- arch/x86/include/asm/idtentry.h | 10 +++++----- arch/x86/kernel/cpu/mce/core.c | 2 +- arch/x86/kernel/kvm.c | 2 +- arch/x86/kernel/nmi.c | 2 +- arch/x86/kernel/sev.c | 4 ++-- arch/x86/kernel/traps.c | 14 +++++++------- arch/x86/mm/fault.c | 2 +- 8 files changed, 20 insertions(+), 19 deletions(-) diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c index 6c2826417b337..047d157987859 100644 --- a/arch/x86/entry/common.c +++ b/arch/x86/entry/common.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -75,7 +76,7 @@ __visible noinstr void do_syscall_64(struct pt_regs *regs= , int nr) add_random_kstack_offset(); nr =3D syscall_enter_from_user_mode(regs, nr); =20 - instrumentation_begin(); + instrumentation_begin_with_regs(regs); =20 if (!do_syscall_x64(regs, nr) && !do_syscall_x32(regs, nr) && nr !=3D -1)= { /* Invalid system call, but still a system call. */ diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentr= y.h index 7924f27f5c8b1..172b9b6f90628 100644 --- a/arch/x86/include/asm/idtentry.h +++ b/arch/x86/include/asm/idtentry.h @@ -53,7 +53,7 @@ __visible noinstr void func(struct pt_regs *regs) \ { \ irqentry_state_t state =3D irqentry_enter(regs); \ \ - instrumentation_begin(); \ + instrumentation_begin_with_regs(regs); \ __##func (regs); \ instrumentation_end(); \ irqentry_exit(regs, state); \ @@ -100,7 +100,7 @@ __visible noinstr void func(struct pt_regs *regs, \ { \ irqentry_state_t state =3D irqentry_enter(regs); \ \ - instrumentation_begin(); \ + instrumentation_begin_with_regs(regs); \ __##func (regs, error_code); \ instrumentation_end(); \ irqentry_exit(regs, state); \ @@ -197,7 +197,7 @@ __visible noinstr void func(struct pt_regs *regs, \ irqentry_state_t state =3D irqentry_enter(regs); \ u32 vector =3D (u32)(u8)error_code; \ \ - instrumentation_begin(); \ + instrumentation_begin_with_regs(regs); \ kvm_set_cpu_l1tf_flush_l1d(); \ run_irq_on_irqstack_cond(__##func, regs, vector); \ instrumentation_end(); \ @@ -237,7 +237,7 @@ __visible noinstr void func(struct pt_regs *regs) \ { \ irqentry_state_t state =3D irqentry_enter(regs); \ \ - instrumentation_begin(); \ + instrumentation_begin_with_regs(regs); \ kvm_set_cpu_l1tf_flush_l1d(); \ run_sysvec_on_irqstack_cond(__##func, regs); \ instrumentation_end(); \ @@ -264,7 +264,7 @@ __visible noinstr void func(struct pt_regs *regs) \ { \ irqentry_state_t state =3D irqentry_enter(regs); \ \ - instrumentation_begin(); \ + instrumentation_begin_with_regs(regs); \ __irq_enter_raw(); \ kvm_set_cpu_l1tf_flush_l1d(); \ __##func (regs); \ diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c index 981496e6bc0e4..e5acff54f7d55 100644 --- a/arch/x86/kernel/cpu/mce/core.c +++ b/arch/x86/kernel/cpu/mce/core.c @@ -1376,7 +1376,7 @@ static void queue_task_work(struct mce *m, char *msg,= void (*func)(struct callba /* Handle unconfigured int18 (should never happen) */ static noinstr void unexpected_machine_check(struct pt_regs *regs) { - instrumentation_begin(); + instrumentation_begin_with_regs(regs); pr_err("CPU#%d: Unexpected int18 (Machine Check)\n", smp_processor_id()); instrumentation_end(); diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index 8b1c45c9cda87..3df82a51ab1b5 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -250,7 +250,7 @@ noinstr bool __kvm_handle_async_pf(struct pt_regs *regs= , u32 token) return false; =20 state =3D irqentry_enter(regs); - instrumentation_begin(); + instrumentation_begin_with_regs(regs); =20 /* * If the host managed to inject an async #PF into an interrupt diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c index e73f7df362f5d..5078417e16ec1 100644 --- a/arch/x86/kernel/nmi.c +++ b/arch/x86/kernel/nmi.c @@ -328,7 +328,7 @@ static noinstr void default_do_nmi(struct pt_regs *regs) =20 __this_cpu_write(last_nmi_rip, regs->ip); =20 - instrumentation_begin(); + instrumentation_begin_with_regs(regs); =20 handled =3D nmi_handle(NMI_LOCAL, regs); __this_cpu_add(nmi_stats.normal, handled); diff --git a/arch/x86/kernel/sev.c b/arch/x86/kernel/sev.c index e6d316a01fdd4..9bfc29fc9c983 100644 --- a/arch/x86/kernel/sev.c +++ b/arch/x86/kernel/sev.c @@ -1330,7 +1330,7 @@ DEFINE_IDTENTRY_VC_KERNEL(exc_vmm_communication) =20 irq_state =3D irqentry_nmi_enter(regs); =20 - instrumentation_begin(); + instrumentation_begin_with_regs(regs); =20 if (!vc_raw_handle_exception(regs, error_code)) { /* Show some debug info */ @@ -1362,7 +1362,7 @@ DEFINE_IDTENTRY_VC_USER(exc_vmm_communication) } =20 irqentry_enter_from_user_mode(regs); - instrumentation_begin(); + instrumentation_begin_with_regs(regs); =20 if (!vc_raw_handle_exception(regs, error_code)) { /* diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index 1563fb9950059..9d3c9c4de94d3 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -305,7 +305,7 @@ static noinstr bool handle_bug(struct pt_regs *regs) /* * All lies, just get the WARN/BUG out. */ - instrumentation_begin(); + instrumentation_begin_with_regs(regs); /* * Since we're emulating a CALL with exceptions, restore the interrupt * state to what it was at the exception site. @@ -336,7 +336,7 @@ DEFINE_IDTENTRY_RAW(exc_invalid_op) return; =20 state =3D irqentry_enter(regs); - instrumentation_begin(); + instrumentation_begin_with_regs(regs); handle_invalid_op(regs); instrumentation_end(); irqentry_exit(regs, state); @@ -490,7 +490,7 @@ DEFINE_IDTENTRY_DF(exc_double_fault) #endif =20 irqentry_nmi_enter(regs); - instrumentation_begin(); + instrumentation_begin_with_regs(regs); notify_die(DIE_TRAP, str, regs, error_code, X86_TRAP_DF, SIGSEGV); =20 tsk->thread.error_code =3D error_code; @@ -820,14 +820,14 @@ DEFINE_IDTENTRY_RAW(exc_int3) */ if (user_mode(regs)) { irqentry_enter_from_user_mode(regs); - instrumentation_begin(); + instrumentation_begin_with_regs(regs); do_int3_user(regs); instrumentation_end(); irqentry_exit_to_user_mode(regs); } else { irqentry_state_t irq_state =3D irqentry_nmi_enter(regs); =20 - instrumentation_begin(); + instrumentation_begin_with_regs(regs); if (!do_int3(regs)) die("int3", regs, 0); instrumentation_end(); @@ -1026,7 +1026,7 @@ static __always_inline void exc_debug_kernel(struct p= t_regs *regs, */ unsigned long dr7 =3D local_db_save(); irqentry_state_t irq_state =3D irqentry_nmi_enter(regs); - instrumentation_begin(); + instrumentation_begin_with_regs(regs); =20 /* * If something gets miswired and we end up here for a user mode @@ -1105,7 +1105,7 @@ static __always_inline void exc_debug_user(struct pt_= regs *regs, */ =20 irqentry_enter_from_user_mode(regs); - instrumentation_begin(); + instrumentation_begin_with_regs(regs); =20 /* * Start the virtual/ptrace DR6 value with just the DR_STEP mask diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index f2250a32a10ca..676e394f1af5b 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1557,7 +1557,7 @@ DEFINE_IDTENTRY_RAW_ERRORCODE(exc_page_fault) */ state =3D irqentry_enter(regs); =20 - instrumentation_begin(); + instrumentation_begin_with_regs(regs); handle_page_fault(regs, error_code, address); instrumentation_end(); =20 --=20 2.36.0.rc2.479.g8af0fa9b8e-goog From nobody Sun May 10 18:32:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDF4DC433EF for ; Tue, 26 Apr 2022 16:49:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353182AbiDZQwa (ORCPT ); Tue, 26 Apr 2022 12:52:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45026 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353256AbiDZQv0 (ORCPT ); Tue, 26 Apr 2022 12:51:26 -0400 Received: from mail-ed1-x54a.google.com (mail-ed1-x54a.google.com [IPv6:2a00:1450:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2EE0C48E6A for ; Tue, 26 Apr 2022 09:46:21 -0700 (PDT) Received: by mail-ed1-x54a.google.com with SMTP id b24-20020a50e798000000b0041631767675so10630518edn.23 for ; Tue, 26 Apr 2022 09:46:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=XPVyte1DHdZzGVqIYLZisC9WWn/NV5PTOOmKGsNYVqI=; b=tfexPWSj1PSjUb4y32u6HDb/QnE+2ju4Q3VQ+kF29tn7ADhoO3fHaJy9KBX3jTsEd8 BhBWx/VeORcURqli+p+9+1QH8obAqShcA6ErLkkeVUmnfCO3qJEDT5UirMbYVSK2rKI7 EHcv+GYbS6ck2ZGcmgknDsZroKT9WTKBy/bFVemoqQAXMDxG6HC+soFmb8T4ca/gXcrO ftqfgIZ/fqbniMJnPu4e11czzALuk3B6wUJb7Az/LpjxMDJcInKVf5CafrOSxVa2JPzU ++NO1MFfBz4ggnwPlPZhMJNmDFkkEcXO47/KdD9d+fQq4LH0V9JjN+pdxl1pwz5u4gCv AAQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=XPVyte1DHdZzGVqIYLZisC9WWn/NV5PTOOmKGsNYVqI=; b=YCMVk/4xsxqLcekq4UbfM/rjt7ecaFplueal5/IZMHKWCzgzu+31vPXeqdIXTMe8+j qITpp1+KiFRxTOO5NvtOIfpqL58WcxwQxNso3t+zLK9oI2QZRETxFonHg6imifk98wxE i+z0vTGLI7OIaADjI1QxaxgSYFzfp1gOCJIDRcLjNLSpAg5CVCse8j654xLoeBJRSKj5 zGhG9z2rn827Q9yJsln1V8gdrptnAeoUSjBdN7Qn40P1usbZGtloA30aUEaEIvs57Win 5TnRNkPowqWa+0Q6mhceEv+kWGxhJNSt0PG6cuaoZfY1feoO3ZHjjHZBlmIqJxsKezqW tMAw== X-Gm-Message-State: AOAM530VDlqXHT1O8zLSv06F791pqP3P1TRI9FXEaCBIAi+H2UhqQfsN kqDsjbu8S0mpkLzk5R12KG2IJ/i4mAg= X-Google-Smtp-Source: ABdhPJxd69pRVvXv3tLXpgiAhti/608pH1mq1OSHk1HEp8P9KBrw2Up9R6Lp+Ti78ee6VL9a6lRzmQRfcX0= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:d580:abeb:bf6d:5726]) (user=glider job=sendgmr) by 2002:a17:906:478b:b0:6db:8b6e:d5de with SMTP id cw11-20020a170906478b00b006db8b6ed5demr22938586ejc.161.1650991579678; Tue, 26 Apr 2022 09:46:19 -0700 (PDT) Date: Tue, 26 Apr 2022 18:43:15 +0200 In-Reply-To: <20220426164315.625149-1-glider@google.com> Message-Id: <20220426164315.625149-47-glider@google.com> Mime-Version: 1.0 References: <20220426164315.625149-1-glider@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v3 46/46] x86: kmsan: enable KMSAN builds for x86 From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make KMSAN usable by adding the necessary Kconfig bits. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/I1d295ce8159ce15faa496d20089= d953a919c125e --- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 3209073f96415..592f5ca2017c2 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -168,6 +168,7 @@ config X86 select HAVE_ARCH_KASAN if X86_64 select HAVE_ARCH_KASAN_VMALLOC if X86_64 select HAVE_ARCH_KFENCE + select HAVE_ARCH_KMSAN if X86_64 select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS if MMU select HAVE_ARCH_MMAP_RND_COMPAT_BITS if MMU && COMPAT --=20 2.36.0.rc2.479.g8af0fa9b8e-goog