From nobody Sat Nov 30 05:26:08 2024 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAB897D3F4 for ; Wed, 11 Sep 2024 06:45:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726037154; cv=none; b=XE3gx/wPRsLUXoFeXAVxJrcS5End/2r03b3yJx3CzoJpMn4dVgk8w66MFe2SFs9IWFJOK1Q93GlL2E+1FTQ+eXkR9mDtscQYf5DYHCMU30gY92kyh8aPLMjoFezTfw5FyHkaX6H1PRPF8KHKVy0GPEntBftUnTfom+6fsw2Apgo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726037154; c=relaxed/simple; bh=dM2adnqSdhHnHwASvqcFoX4V3rfZ14QoQNpiYKbsRgQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=MhXagNs35el3vu2GNHdTRcrdJcX1+9cHvTXpKl8Rhfn/7wH0TjuKBC7Os0AhIHtgxETLSS14P1aApFCvnSd9tzlQzIbLjFgQd6l7YZBHyxQhkOIw9cNMcTW98P2ElnWTDFLJno3XaJWsxuIci73LdwIgykmqRf+OA6wbD1qXMUM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=lERXujxi; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="lERXujxi" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1726037152; x=1757573152; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dM2adnqSdhHnHwASvqcFoX4V3rfZ14QoQNpiYKbsRgQ=; b=lERXujxiEE5lGVLbU8Eer02M/WcS1IHRbEmD/cJcZGxq1+4sPUVFCIbZ dMs8dPF6+brvSneeNr4qk0reuNILj9EKA6PPzCLfjHBFKOgjcQP7Ol13/ 9iCmSJL6t3ZEcsgXYM+HNDQNtResrSNOvZwVefpg7ppUTYuIE0dcj6PSV AIe82cu4WO5aid5ptKbWdOgCdt46232UWL0nhje1Px0KJjduUzGuitFkq uialQHs2hro28gKGZztGDoqDsWL5Aq0bOL6UH6otUxABHSUTLaB1n1mJg u5+vHsa3s6CLEpem0jkJ15jDWk+1U1CQirr+ZM1cfmN2IIocMLJp0B2AA g==; X-CSE-ConnectionGUID: RwOPG8JyQGSFKioEYsXnpQ== X-CSE-MsgGUID: /73sftPrQu65GqYvQtxBfw== X-IronPort-AV: E=McAfee;i="6700,10204,11191"; a="36172986" X-IronPort-AV: E=Sophos;i="6.10,219,1719903600"; d="scan'208";a="36172986" Received: from orviesa007.jf.intel.com ([10.64.159.147]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Sep 2024 23:45:52 -0700 X-CSE-ConnectionGUID: wcILiiZ3QmaqfhSIVIcfHw== X-CSE-MsgGUID: fx8JhJANRZCRTz/LvKdSHg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,219,1719903600"; d="scan'208";a="67771485" Received: from feng-clx.sh.intel.com ([10.239.159.50]) by orviesa007.jf.intel.com with ESMTP; 10 Sep 2024 23:45:41 -0700 From: Feng Tang To: Vlastimil Babka , Andrew Morton , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Andrey Konovalov , Marco Elver , Shuah Khan , David Gow , Danilo Krummrich , Alexander Potapenko , Andrey Ryabinin , Dmitry Vyukov , Vincenzo Frascino Cc: linux-mm@kvack.org, kasan-dev@googlegroups.com, linux-kernel@vger.kernel.org, Feng Tang Subject: [PATCH v2 1/5] mm/kasan: Don't store metadata inside kmalloc object when slub_debug_orig_size is on Date: Wed, 11 Sep 2024 14:45:31 +0800 Message-Id: <20240911064535.557650-2-feng.tang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240911064535.557650-1-feng.tang@intel.com> References: <20240911064535.557650-1-feng.tang@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For a kmalloc object, when both kasan and slub redzone sanity check are enabled, they could both manipulate its data space like storing kasan free meta data and setting up kmalloc redzone, and may affect accuracy of that object's 'orig_size'. As an accurate 'orig_size' will be needed by some function like krealloc() soon, save kasan's free meta data in slub's metadata area instead of inside object when 'orig_size' is enabled. This will make it easier to maintain/understand the code. Size wise, when these two options are both enabled, the slub meta data space is already huge, and this just slightly increase the overall size. Signed-off-by: Feng Tang Acked-by: Andrey Konovalov --- mm/kasan/generic.c | 7 +++++-- mm/slab.h | 6 ++++++ mm/slub.c | 17 ----------------- 3 files changed, 11 insertions(+), 19 deletions(-) diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c index 6310a180278b..8b9e348113b1 100644 --- a/mm/kasan/generic.c +++ b/mm/kasan/generic.c @@ -392,9 +392,12 @@ void kasan_cache_create(struct kmem_cache *cache, unsi= gned int *size, * 1. Object is SLAB_TYPESAFE_BY_RCU, which means that it can * be touched after it was freed, or * 2. Object has a constructor, which means it's expected to - * retain its content until the next allocation. + * retain its content until the next allocation, or + * 3. It is from a kmalloc cache which enables the debug option + * to store original size. */ - if ((cache->flags & SLAB_TYPESAFE_BY_RCU) || cache->ctor) { + if ((cache->flags & SLAB_TYPESAFE_BY_RCU) || cache->ctor || + slub_debug_orig_size(cache)) { cache->kasan_info.free_meta_offset =3D *size; *size +=3D sizeof(struct kasan_free_meta); goto free_meta_added; diff --git a/mm/slab.h b/mm/slab.h index f22fb760b286..f72a8849b988 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -689,6 +689,12 @@ void __kmem_obj_info(struct kmem_obj_info *kpp, void *= object, struct slab *slab) void __check_heap_object(const void *ptr, unsigned long n, const struct slab *slab, bool to_user); =20 +static inline bool slub_debug_orig_size(struct kmem_cache *s) +{ + return (kmem_cache_debug_flags(s, SLAB_STORE_USER) && + (s->flags & SLAB_KMALLOC)); +} + #ifdef CONFIG_SLUB_DEBUG void skip_orig_size_check(struct kmem_cache *s, const void *object); #endif diff --git a/mm/slub.c b/mm/slub.c index 21f71cb6cc06..87c95f170f13 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -230,12 +230,6 @@ static inline bool kmem_cache_debug(struct kmem_cache = *s) return kmem_cache_debug_flags(s, SLAB_DEBUG_FLAGS); } =20 -static inline bool slub_debug_orig_size(struct kmem_cache *s) -{ - return (kmem_cache_debug_flags(s, SLAB_STORE_USER) && - (s->flags & SLAB_KMALLOC)); -} - void *fixup_red_left(struct kmem_cache *s, void *p) { if (kmem_cache_debug_flags(s, SLAB_RED_ZONE)) @@ -760,21 +754,10 @@ static inline void set_orig_size(struct kmem_cache *s, void *object, unsigned int orig_size) { void *p =3D kasan_reset_tag(object); - unsigned int kasan_meta_size; =20 if (!slub_debug_orig_size(s)) return; =20 - /* - * KASAN can save its free meta data inside of the object at offset 0. - * If this meta data size is larger than 'orig_size', it will overlap - * the data redzone in [orig_size+1, object_size]. Thus, we adjust - * 'orig_size' to be as at least as big as KASAN's meta data. - */ - kasan_meta_size =3D kasan_metadata_size(s, true); - if (kasan_meta_size > orig_size) - orig_size =3D kasan_meta_size; - p +=3D get_info_end(s); p +=3D sizeof(struct track) * 2; =20 --=20 2.34.1 From nobody Sat Nov 30 05:26:09 2024 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2655F80604 for ; Wed, 11 Sep 2024 06:46:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726037176; cv=none; b=YYM1ACotUU5qju/bU7cCVsE+T8iGetqpgqOiIOdH+RS98TTZRFGRJuZwRjSqTQ+Z2OeyA4wAFH2Ff+JuUw3GJMFmr8uIcj8RMVbK5ceq4BSRhpXRxN9ccsv5nAHoxi4EVPPq+60tyaoLGnZ591wLUekHy15S4vTh2cADBNuo9UQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726037176; c=relaxed/simple; bh=Q3PpnMMnKbBe2tGhLr5Y8UQvjMtp65YK3pg5VOQk5Ag=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=frqF/Z4ahlVtq1PKLuhOe1OH6FGJJ9J+K/bt2Q6AYAAh6Ch/8ynz7D2K9w0cgdTn9ivUmLbISn7IxD1ipgzj4SAVV/PxULIqMhTbnyaIKxazmsPvcZLX2TKv4ByOFqo/DDY0Cg3+pb0UbL8hZ+1jSHUnw3Q82mNpc95x/NZM1t8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=U13N2Ppw; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="U13N2Ppw" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1726037175; x=1757573175; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Q3PpnMMnKbBe2tGhLr5Y8UQvjMtp65YK3pg5VOQk5Ag=; b=U13N2PpwWWJ9ARVgvbe/V8DjBGtANSSM9ayTKbWKJC2DmqUxN9MVb4U5 wl16WYlDH7kX5BKcTVEPt7pOGyM2eyQ/aZE+ZHUAL95X66QtlMGB/026D v4dMZYsV/+sep/9M9O+HyD5wxbR7XLq3Zzv/NS3al5sSyGf3S21iBUXps s4PBGaWWc+iaYfkwzrsyOTZznf2dS4SVptIAi3SqjcYMrUciZMStUeQaG KBFZkMPCVVDdRLHdG9WruamypiFKVYAJ/EW8e2fKV07cOrylj8tk9BbWA o9kBNN0nCfa+l6Is4XWXN0Nnx9QgvRp6qq8BUw3dI3SY2Ea4QclET+PT0 g==; X-CSE-ConnectionGUID: ernaZeNfRwSau4+XhE7CBQ== X-CSE-MsgGUID: 4lH4vjrHSICVFThBQITE/Q== X-IronPort-AV: E=McAfee;i="6700,10204,11191"; a="36173002" X-IronPort-AV: E=Sophos;i="6.10,219,1719903600"; d="scan'208";a="36173002" Received: from orviesa007.jf.intel.com ([10.64.159.147]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Sep 2024 23:46:00 -0700 X-CSE-ConnectionGUID: TlC0q2XjRdmz/RXKu5H2YA== X-CSE-MsgGUID: ouEYEDaZRMK+unGJAsKkGQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,219,1719903600"; d="scan'208";a="67771491" Received: from feng-clx.sh.intel.com ([10.239.159.50]) by orviesa007.jf.intel.com with ESMTP; 10 Sep 2024 23:45:50 -0700 From: Feng Tang To: Vlastimil Babka , Andrew Morton , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Andrey Konovalov , Marco Elver , Shuah Khan , David Gow , Danilo Krummrich , Alexander Potapenko , Andrey Ryabinin , Dmitry Vyukov , Vincenzo Frascino Cc: linux-mm@kvack.org, kasan-dev@googlegroups.com, linux-kernel@vger.kernel.org, Feng Tang Subject: [PATCH v2 2/5] mm/slub: Consider kfence case for get_orig_size() Date: Wed, 11 Sep 2024 14:45:32 +0800 Message-Id: <20240911064535.557650-3-feng.tang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240911064535.557650-1-feng.tang@intel.com> References: <20240911064535.557650-1-feng.tang@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When 'orig_size' of kmalloc object is enabled by debug option, it should either contains the actual requested size or the cache's 'object_size'. But it's not true if that object is a kfence-allocated one, and its 'orig_size' in metadata could be zero or other values. This is not a big issue for current 'orig_size' usage, as init_object() and check_object() during alloc/free process will be skipped for kfence addresses. As 'orig_size' will be used by some function block like krealloc(), handle it by returning the 'object_size' in get_orig_size() for kfence addresses. Signed-off-by: Feng Tang --- mm/slub.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/slub.c b/mm/slub.c index 87c95f170f13..021991e17287 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -768,7 +768,7 @@ static inline unsigned int get_orig_size(struct kmem_ca= che *s, void *object) { void *p =3D kasan_reset_tag(object); =20 - if (!slub_debug_orig_size(s)) + if (!slub_debug_orig_size(s) || is_kfence_address(object)) return s->object_size; =20 p +=3D get_info_end(s); --=20 2.34.1 From nobody Sat Nov 30 05:26:09 2024 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE5367DA6E for ; Wed, 11 Sep 2024 06:46:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726037176; cv=none; b=uh8lOmS/O60zis+0Dq3jVF1dT5FOZR1rI7vtIkeMzt9NNBIfjJQkqroMmZQU5ArkQKROCNWSfgDASeoLr7KL1U+z6HAE/MF5J4DyqFag+6Cw5Ahdhhg4wM48YHpgFaedHjLScLFq6RwiPNrwffL6D92yxLAYHaA6b8/uWSPoEZ8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726037176; c=relaxed/simple; bh=Ag0o94U2BMb2lFyFC32W4EFoRg3K+ZMunlGHmbIXB5s=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=SwZ7NcQ4seFnUSM6h3AfVCHiusDcjHRQQVp+WQOaRhh7RIZxarFBVX8T+0kZYrjxlXm+L34/xDAkZ9g1WwQrgHPQwTo4Pw5u/qqX+UJ55RohmdgO/mtE54QR4FeIhh9LmTwA0zcUXx+C2WQtcICkHo9anVovq+GvjCfTFwtU/Mw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=nbAem6My; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="nbAem6My" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1726037174; x=1757573174; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Ag0o94U2BMb2lFyFC32W4EFoRg3K+ZMunlGHmbIXB5s=; b=nbAem6My0wwQaVqgOTOiWV+7dRfqZdcDA7oTXxXWe3FoJlOLv0+vE327 1qOqV/r09hPHvtYICZ8abmYQ1qgVeAUPU1SGijz3WJhaLVpmoY2HpRahQ GP7D2JApE3GkRS4Bsygldhuhu6DXKNS68sfmIGL9g1jJfQrIDa6JGhrzM kaY57//H16zLU7yFCz9Urcab4X1X766bnwgi6yZbYrqR85p6YYkzJ6yzk UQvZ/rSpfPo1Y8UFCB8FLvM4VoEHtY+PtmniaZgnd7O4RIxmbwPi/37Jn Riw10TMON/e8CQgBHCiLm56JW0bJKKlGTBcyYFZLKnVMtsMMJdsCYBahl w==; X-CSE-ConnectionGUID: vj2vtykyQu+ow8+jrdJqLA== X-CSE-MsgGUID: CNgdb46XS2KUmO2l3xIzug== X-IronPort-AV: E=McAfee;i="6700,10204,11191"; a="36173015" X-IronPort-AV: E=Sophos;i="6.10,219,1719903600"; d="scan'208";a="36173015" Received: from orviesa007.jf.intel.com ([10.64.159.147]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Sep 2024 23:46:11 -0700 X-CSE-ConnectionGUID: IgyzjmIvSCmBHDIIIAmEIw== X-CSE-MsgGUID: u/rfH00VRWC+1lv/88CKcQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,219,1719903600"; d="scan'208";a="67771497" Received: from feng-clx.sh.intel.com ([10.239.159.50]) by orviesa007.jf.intel.com with ESMTP; 10 Sep 2024 23:45:55 -0700 From: Feng Tang To: Vlastimil Babka , Andrew Morton , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Andrey Konovalov , Marco Elver , Shuah Khan , David Gow , Danilo Krummrich , Alexander Potapenko , Andrey Ryabinin , Dmitry Vyukov , Vincenzo Frascino Cc: linux-mm@kvack.org, kasan-dev@googlegroups.com, linux-kernel@vger.kernel.org, Feng Tang Subject: [PATCH v2 3/5] mm/slub: Move krealloc() and related code to slub.c Date: Wed, 11 Sep 2024 14:45:33 +0800 Message-Id: <20240911064535.557650-4-feng.tang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240911064535.557650-1-feng.tang@intel.com> References: <20240911064535.557650-1-feng.tang@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This is a preparation for the following refactoring of krealloc(), for more efficient function calling as it will call some internal functions defined in slub.c. Signed-off-by: Feng Tang --- mm/slab_common.c | 84 ------------------------------------------------ mm/slub.c | 84 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 84 insertions(+), 84 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index af6b14769fbd..5734b61a106f 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1185,90 +1185,6 @@ module_init(slab_proc_init); =20 #endif /* CONFIG_SLUB_DEBUG */ =20 -static __always_inline __realloc_size(2) void * -__do_krealloc(const void *p, size_t new_size, gfp_t flags) -{ - void *ret; - size_t ks; - - /* Check for double-free before calling ksize. */ - if (likely(!ZERO_OR_NULL_PTR(p))) { - if (!kasan_check_byte(p)) - return NULL; - ks =3D ksize(p); - } else - ks =3D 0; - - /* If the object still fits, repoison it precisely. */ - if (ks >=3D new_size) { - /* Zero out spare memory. */ - if (want_init_on_alloc(flags)) { - kasan_disable_current(); - memset((void *)p + new_size, 0, ks - new_size); - kasan_enable_current(); - } - - p =3D kasan_krealloc((void *)p, new_size, flags); - return (void *)p; - } - - ret =3D kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _= RET_IP_); - if (ret && p) { - /* Disable KASAN checks as the object's redzone is accessed. */ - kasan_disable_current(); - memcpy(ret, kasan_reset_tag(p), ks); - kasan_enable_current(); - } - - return ret; -} - -/** - * krealloc - reallocate memory. The contents will remain unchanged. - * @p: object to reallocate memory for. - * @new_size: how many bytes of memory are required. - * @flags: the type of memory to allocate. - * - * If @p is %NULL, krealloc() behaves exactly like kmalloc(). If @new_size - * is 0 and @p is not a %NULL pointer, the object pointed to is freed. - * - * If __GFP_ZERO logic is requested, callers must ensure that, starting wi= th the - * initial memory allocation, every subsequent call to this API for the sa= me - * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible= that - * __GFP_ZERO is not fully honored by this API. - * - * This is the case, since krealloc() only knows about the bucket size of = an - * allocation (but not the exact size it was allocated with) and hence - * implements the following semantics for shrinking and growing buffers wi= th - * __GFP_ZERO. - * - * new bucket - * 0 size size - * |--------|----------------| - * | keep | zero | - * - * In any case, the contents of the object pointed to are preserved up to = the - * lesser of the new and old sizes. - * - * Return: pointer to the allocated memory or %NULL in case of error - */ -void *krealloc_noprof(const void *p, size_t new_size, gfp_t flags) -{ - void *ret; - - if (unlikely(!new_size)) { - kfree(p); - return ZERO_SIZE_PTR; - } - - ret =3D __do_krealloc(p, new_size, flags); - if (ret && kasan_reset_tag(p) !=3D kasan_reset_tag(ret)) - kfree(p); - - return ret; -} -EXPORT_SYMBOL(krealloc_noprof); - /** * kfree_sensitive - Clear sensitive information in memory before freeing * @p: object to free memory of diff --git a/mm/slub.c b/mm/slub.c index 021991e17287..c1796f9dd30f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4712,6 +4712,90 @@ void kfree(const void *object) } EXPORT_SYMBOL(kfree); =20 +static __always_inline __realloc_size(2) void * +__do_krealloc(const void *p, size_t new_size, gfp_t flags) +{ + void *ret; + size_t ks; + + /* Check for double-free before calling ksize. */ + if (likely(!ZERO_OR_NULL_PTR(p))) { + if (!kasan_check_byte(p)) + return NULL; + ks =3D ksize(p); + } else + ks =3D 0; + + /* If the object still fits, repoison it precisely. */ + if (ks >=3D new_size) { + /* Zero out spare memory. */ + if (want_init_on_alloc(flags)) { + kasan_disable_current(); + memset((void *)p + new_size, 0, ks - new_size); + kasan_enable_current(); + } + + p =3D kasan_krealloc((void *)p, new_size, flags); + return (void *)p; + } + + ret =3D kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _= RET_IP_); + if (ret && p) { + /* Disable KASAN checks as the object's redzone is accessed. */ + kasan_disable_current(); + memcpy(ret, kasan_reset_tag(p), ks); + kasan_enable_current(); + } + + return ret; +} + +/** + * krealloc - reallocate memory. The contents will remain unchanged. + * @p: object to reallocate memory for. + * @new_size: how many bytes of memory are required. + * @flags: the type of memory to allocate. + * + * If @p is %NULL, krealloc() behaves exactly like kmalloc(). If @new_size + * is 0 and @p is not a %NULL pointer, the object pointed to is freed. + * + * If __GFP_ZERO logic is requested, callers must ensure that, starting wi= th the + * initial memory allocation, every subsequent call to this API for the sa= me + * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible= that + * __GFP_ZERO is not fully honored by this API. + * + * This is the case, since krealloc() only knows about the bucket size of = an + * allocation (but not the exact size it was allocated with) and hence + * implements the following semantics for shrinking and growing buffers wi= th + * __GFP_ZERO. + * + * new bucket + * 0 size size + * |--------|----------------| + * | keep | zero | + * + * In any case, the contents of the object pointed to are preserved up to = the + * lesser of the new and old sizes. + * + * Return: pointer to the allocated memory or %NULL in case of error + */ +void *krealloc_noprof(const void *p, size_t new_size, gfp_t flags) +{ + void *ret; + + if (unlikely(!new_size)) { + kfree(p); + return ZERO_SIZE_PTR; + } + + ret =3D __do_krealloc(p, new_size, flags); + if (ret && kasan_reset_tag(p) !=3D kasan_reset_tag(ret)) + kfree(p); + + return ret; +} +EXPORT_SYMBOL(krealloc_noprof); + struct detached_freelist { struct slab *slab; void *tail; --=20 2.34.1 From nobody Sat Nov 30 05:26:09 2024 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 162AF127E18 for ; Wed, 11 Sep 2024 06:46:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726037174; cv=none; b=hPHsdVmzXdmmBLfl5Qv8KQSBBkwROqFq0K4MunTzEcPyKAdFGBTTwkrQ+DkKBYCWDKWzQOnwnakKdrxx8J7XIsXn12GHd1TkT6wOHBJ5YBv0rwVHgmhuQYyUy6dP5YeqiElJb4tzv9Lw4L8R3J4QJ5DNSEOunQeefv+39A2d8rM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726037174; c=relaxed/simple; bh=ZHBD5nxjCn39VoHzOY+ge2IVDv8unPJPEMRErbHaUA4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ZxdBD4Fj4rijmXAOCDoNi6JB0mopBObor5bN83s5L8FydpCyJQ6rN5CNrZvNsDsSIzb+L5LIfI0EdsXPgUWfFSfVTOHPNal8YYOP5yRgsBvWFZh58UjvGSYeXgY9fQkH7lXyZDYySgrLnVWwvn5HFtguQzQNxk6479Hcr9jzO+I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=mAtxeU2f; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="mAtxeU2f" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1726037173; x=1757573173; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ZHBD5nxjCn39VoHzOY+ge2IVDv8unPJPEMRErbHaUA4=; b=mAtxeU2f8Z7WmpquHWY4nyvM1zqRIlqVBl49pGlAl5nPXPdlfSbdv/+M iVjgPTQvxuzlyZYljW594Jgel6PO1SteSfp74AcjfwDE9JbvAECLNZukv btTJu2Zo9qeQRvqlhjlzmdrrILhyp8H/olExK9B0wwyaH+yE/ZYtNXw8r 2I8tTZ3m3HrDZTZBxIP7JzXRBwTL2CaGQ4hS6iamYjLA6QwIjbDc3UWh+ sW2oxqIxZr24oE0RGsoz8X2+00p3zeJ35hYcfmO7LVECPVl0yh0Dj9pOr Rxxn+Yk7t7bQfQsDaGZTWuzzqaPRp5DoJB4U+6I8K3GjwdDIHqBWTaAlH Q==; X-CSE-ConnectionGUID: D4TUMG7lTlSUbwFk9MpS3g== X-CSE-MsgGUID: mCbQPIbHSgirw1c+H4EPZg== X-IronPort-AV: E=McAfee;i="6700,10204,11191"; a="36173029" X-IronPort-AV: E=Sophos;i="6.10,219,1719903600"; d="scan'208";a="36173029" Received: from orviesa007.jf.intel.com ([10.64.159.147]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Sep 2024 23:46:11 -0700 X-CSE-ConnectionGUID: v543qwycT2WvE2geGxwVvA== X-CSE-MsgGUID: Ll8xYtUdRRW3q2YxBIkspA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,219,1719903600"; d="scan'208";a="67771506" Received: from feng-clx.sh.intel.com ([10.239.159.50]) by orviesa007.jf.intel.com with ESMTP; 10 Sep 2024 23:46:00 -0700 From: Feng Tang To: Vlastimil Babka , Andrew Morton , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Andrey Konovalov , Marco Elver , Shuah Khan , David Gow , Danilo Krummrich , Alexander Potapenko , Andrey Ryabinin , Dmitry Vyukov , Vincenzo Frascino Cc: linux-mm@kvack.org, kasan-dev@googlegroups.com, linux-kernel@vger.kernel.org, Feng Tang Subject: [PATCH v2 4/5] mm/slub: Improve redzone check and zeroing for krealloc() Date: Wed, 11 Sep 2024 14:45:34 +0800 Message-Id: <20240911064535.557650-5-feng.tang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240911064535.557650-1-feng.tang@intel.com> References: <20240911064535.557650-1-feng.tang@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For current krealloc(), one problem is its caller doesn't pass the old request size, say the object is 64 bytes kmalloc one, but caller may only requested 48 bytes. Then when krealloc() shrinks or grows in the same object, or allocate a new bigger object, it lacks this 'original size' information to do accurate data preserving or zeroing (when __GFP_ZERO is set). Thus with slub debug redzone and object tracking enabled, parts of the object after krealloc() might contain redzone data instead of zeroes, which is violating the __GFP_ZERO guarantees. Good thing is in this case, kmalloc caches do have this 'orig_size' feature. So solve the problem by utilize 'org_size' to do accurate data zeroing and preserving. Suggested-by: Vlastimil Babka Signed-off-by: Feng Tang --- mm/slub.c | 54 ++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 38 insertions(+), 16 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index c1796f9dd30f..e0fb0a26c796 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4717,33 +4717,51 @@ __do_krealloc(const void *p, size_t new_size, gfp_t= flags) { void *ret; size_t ks; + int orig_size =3D 0; + struct kmem_cache *s; =20 - /* Check for double-free before calling ksize. */ + /* Check for double-free. */ if (likely(!ZERO_OR_NULL_PTR(p))) { if (!kasan_check_byte(p)) return NULL; - ks =3D ksize(p); + + s =3D virt_to_cache(p); + orig_size =3D get_orig_size(s, (void *)p); + ks =3D s->object_size; } else ks =3D 0; =20 - /* If the object still fits, repoison it precisely. */ - if (ks >=3D new_size) { - /* Zero out spare memory. */ - if (want_init_on_alloc(flags)) { - kasan_disable_current(); + /* If the object doesn't fit, allocate a bigger one */ + if (new_size > ks) + goto alloc_new; + + /* Zero out spare memory. */ + if (want_init_on_alloc(flags)) { + kasan_disable_current(); + if (orig_size < new_size) + memset((void *)p + orig_size, 0, new_size - orig_size); + else memset((void *)p + new_size, 0, ks - new_size); - kasan_enable_current(); - } + kasan_enable_current(); + } =20 - p =3D kasan_krealloc((void *)p, new_size, flags); - return (void *)p; + if (slub_debug_orig_size(s) && !is_kfence_address(p)) { + set_orig_size(s, (void *)p, new_size); + if (s->flags & SLAB_RED_ZONE && new_size < ks) + memset_no_sanitize_memory((void *)p + new_size, + SLUB_RED_ACTIVE, ks - new_size); } =20 + p =3D kasan_krealloc((void *)p, new_size, flags); + return (void *)p; + +alloc_new: ret =3D kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _= RET_IP_); if (ret && p) { /* Disable KASAN checks as the object's redzone is accessed. */ kasan_disable_current(); - memcpy(ret, kasan_reset_tag(p), ks); + if (orig_size) + memcpy(ret, kasan_reset_tag(p), orig_size); kasan_enable_current(); } =20 @@ -4764,16 +4782,20 @@ __do_krealloc(const void *p, size_t new_size, gfp_t= flags) * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible= that * __GFP_ZERO is not fully honored by this API. * - * This is the case, since krealloc() only knows about the bucket size of = an - * allocation (but not the exact size it was allocated with) and hence - * implements the following semantics for shrinking and growing buffers wi= th - * __GFP_ZERO. + * When slub_debug_orig_size() is off, krealloc() only knows about the buc= ket + * size of an allocation (but not the exact size it was allocated with) and + * hence implements the following semantics for shrinking and growing buff= ers + * with __GFP_ZERO. * * new bucket * 0 size size * |--------|----------------| * | keep | zero | * + * Otherwise, the original allocation size 'orig_size' could be used to + * precisely clear the requested size, and the new size will also be stored + * as the new 'orig_size'. + * * In any case, the contents of the object pointed to are preserved up to = the * lesser of the new and old sizes. * --=20 2.34.1 From nobody Sat Nov 30 05:26:09 2024 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9E0BD136338 for ; Wed, 11 Sep 2024 06:46:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726037178; cv=none; b=cYpdXQPHYsuYGH/1qw5O/3jGu4kd8Bm1mY2QgS92un6FFWxOaVlWO57mgdu5tm+Et6/9MVZqFZ5qalyP2Rs35xa91j8o3jBoMPshjaCofkFo4MC43jEfJwjcxObKmf/bV5QVnE5z4WTNcqkisyVKM3T1pPbHE2KjAXitIP/Z/XA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726037178; c=relaxed/simple; bh=spRk1KgMCNNaHSmFuRFHc+16C4JN3bxZV6odFONV/Uw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=H4RK89VP5qcCAcQNmJgQtXgpzfxrPwa6PoaTyuxUHfACflr7el+GdNFl46nBkjL/QePTD4+oKaV02hAZWnRwzvGW4uoHt3plfqXnGInbNfLbyt/cXNWNjIY5HFtMNgjd0T+bZ/C5fblY7UQR6fwJ+nPk5H7vUMU5HpWDNgrBgzk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=YAhp+rC0; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="YAhp+rC0" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1726037176; x=1757573176; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=spRk1KgMCNNaHSmFuRFHc+16C4JN3bxZV6odFONV/Uw=; b=YAhp+rC0hG3OevF7ddcm0cTK3KWJCPHwd4E14L4cKM0pZevAhbqdwvWc +QKajrSY6loRxZW20N8/cMD3MreJVqGSYrjM84gAdy7rtOJZtluecFvK4 5+S7YHJMLFQ4RO7H3Pk65NValRiMro+MbltqCxeKRdVdoRzPlk89AcCQz brAHgV0wlpJMxAXKg+yMA7nE+iaS7vnJo7ehUpNRk45nqk7Tw/U2nxICS EVksxm/EQN7ac/lG3hqfox7kjs1iNzB6nslYcDEG6Gz3s4HWO97QFNDx3 glZ2ZBFND/1BzQ+QqtvLZ5YS/9/WRYKuy/LEuRUCGYpN8Maso4KjGMscn A==; X-CSE-ConnectionGUID: 8Izt4nHdQ+eZlj+q0z0koQ== X-CSE-MsgGUID: mNMGoDDgRYma3vYkRfX+Ag== X-IronPort-AV: E=McAfee;i="6700,10204,11191"; a="36173051" X-IronPort-AV: E=Sophos;i="6.10,219,1719903600"; d="scan'208";a="36173051" Received: from orviesa007.jf.intel.com ([10.64.159.147]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Sep 2024 23:46:12 -0700 X-CSE-ConnectionGUID: tvwWZ/Y9QRKy+QIHoKa/xw== X-CSE-MsgGUID: V2Tkj4x9QrGrTc/GWUmeNQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,219,1719903600"; d="scan'208";a="67771512" Received: from feng-clx.sh.intel.com ([10.239.159.50]) by orviesa007.jf.intel.com with ESMTP; 10 Sep 2024 23:46:05 -0700 From: Feng Tang To: Vlastimil Babka , Andrew Morton , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Andrey Konovalov , Marco Elver , Shuah Khan , David Gow , Danilo Krummrich , Alexander Potapenko , Andrey Ryabinin , Dmitry Vyukov , Vincenzo Frascino Cc: linux-mm@kvack.org, kasan-dev@googlegroups.com, linux-kernel@vger.kernel.org, Feng Tang Subject: [PATCH v2 5/5] mm/slub, kunit: Add testcase for krealloc redzone and zeroing Date: Wed, 11 Sep 2024 14:45:35 +0800 Message-Id: <20240911064535.557650-6-feng.tang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240911064535.557650-1-feng.tang@intel.com> References: <20240911064535.557650-1-feng.tang@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Danilo Krummrich raised issue about krealloc+GFP_ZERO [1], and Vlastimil suggested to add some test case which can sanity test the kmalloc-redzone and zeroing by utilizing the kmalloc's 'orig_size' debug feature. It covers the grow and shrink case of krealloc() re-using current kmalloc object, and the case of re-allocating a new bigger object. [1]. https://lore.kernel.org/lkml/20240812223707.32049-1-dakr@kernel.org/ Suggested-by: Vlastimil Babka Signed-off-by: Feng Tang Reviewed-by: Danilo Krummrich --- Hi Danilo, I keep your Reviewed-By tag, as I think this v2 mostly changes what kmalloc slab to be used. Let me know if you want it dropped, thanks. lib/slub_kunit.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/lib/slub_kunit.c b/lib/slub_kunit.c index 6e3a1e5a7142..b3d158f38b98 100644 --- a/lib/slub_kunit.c +++ b/lib/slub_kunit.c @@ -186,6 +186,47 @@ static void test_leak_destroy(struct kunit *test) KUNIT_EXPECT_EQ(test, 1, slab_errors); } =20 +static void test_krealloc_redzone_zeroing(struct kunit *test) +{ + u8 *p; + int i; + struct kmem_cache *s =3D test_kmem_cache_create("TestSlub_krealloc", 64, + SLAB_KMALLOC|SLAB_STORE_USER|SLAB_RED_ZONE); + + p =3D __kmalloc_cache_noprof(s, GFP_KERNEL, 48); + memset(p, 0xff, 48); + + kasan_disable_current(); + OPTIMIZER_HIDE_VAR(p); + + /* Test shrink */ + p =3D krealloc(p, 40, GFP_KERNEL | __GFP_ZERO); + for (i =3D 40; i < 64; i++) + KUNIT_EXPECT_EQ(test, p[i], SLUB_RED_ACTIVE); + + /* Test grow within the same 64B kmalloc object */ + p =3D krealloc(p, 56, GFP_KERNEL | __GFP_ZERO); + for (i =3D 40; i < 56; i++) + KUNIT_EXPECT_EQ(test, p[i], 0); + for (i =3D 56; i < 64; i++) + KUNIT_EXPECT_EQ(test, p[i], SLUB_RED_ACTIVE); + + validate_slab_cache(s); + KUNIT_EXPECT_EQ(test, 0, slab_errors); + + memset(p, 0xff, 56); + /* Test grow with allocating a bigger 128B object */ + p =3D krealloc(p, 112, GFP_KERNEL | __GFP_ZERO); + for (i =3D 0; i < 56; i++) + KUNIT_EXPECT_EQ(test, p[i], 0xff); + for (i =3D 56; i < 112; i++) + KUNIT_EXPECT_EQ(test, p[i], 0); + + kfree(p); + kasan_enable_current(); + kmem_cache_destroy(s); +} + static int test_init(struct kunit *test) { slab_errors =3D 0; @@ -208,6 +249,7 @@ static struct kunit_case test_cases[] =3D { KUNIT_CASE(test_kmalloc_redzone_access), KUNIT_CASE(test_kfree_rcu), KUNIT_CASE(test_leak_destroy), + KUNIT_CASE(test_krealloc_redzone_zeroing), {} }; =20 --=20 2.34.1