From nobody Tue Jun 23 13:09:43 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19693C433EF for ; Fri, 4 Mar 2022 06:35:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238471AbiCDGg0 (ORCPT ); Fri, 4 Mar 2022 01:36:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42698 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238492AbiCDGgP (ORCPT ); Fri, 4 Mar 2022 01:36:15 -0500 Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F19BF18CC5A for ; Thu, 3 Mar 2022 22:34:44 -0800 (PST) Received: by mail-pj1-x102a.google.com with SMTP id cx5so6621347pjb.1 for ; Thu, 03 Mar 2022 22:34:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TVdYQpXFu0DWrIT8aY3Z6KXTWb1Uch1rx+HdjmDRFcE=; b=LsASEabx8/b3sjb/G+RHgJLKsqZzcmnsQpo7M0GFhPCRRcuFrPFU7DM1g1X9MwHM21 PCXAVQsHx+MY8cnF0ZNt6zIq0IoAZxJ+YDgCqdORaHNDX7Fz8BwvxFmSI3Z+xQzxLh33 BUyMjlWWgmy5H4BIxoTuTqf3UE5ZxGD7UwaXXV0XkkZ/t7lFPk5JmQnFpv5BflK6/4Xa rKVoS4CylYDxBNLDH1+HfxM1pKl0ZoCw1xFctB8g1uqKLagtbseDlW1WFBCZiHTwGO2N mZrCNFxqr5JLCxO3V2BqG1pVkczUxsM4IWh7nFxGKarULxMOmvPb/3v9bRDemWr79jBL YoEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TVdYQpXFu0DWrIT8aY3Z6KXTWb1Uch1rx+HdjmDRFcE=; b=Kc9u92y18j8+96p4YFV/joqk8WCw/fzdmOO07x+EEs9h7Ty+WnVO0qzCzUwz3ZuwJf RemP4U0cdyzCpAjj8N5P8zjyNgteLS4HWLsPpi1xupAhzXGnwzTwpJTwsXJ/a7p3/Krv 9NfyUAGF5ts7hitpGkXPj05AQR/wlSW1eoTNPl21Ahft7pG/LSvEBd0hofs7yTNFQXvG zXqzkx06xtuEuK1+PLY+yKS/8psStH8ocvCY0mp4stSCAUPou+lFEpm5NFIFn6Q/GVyE F1KjwBiNhk9scR9PTeMHxZWCY6pTIMhoS3f0v3pyt2BjhuHEjYC00Gf8l+qIzug7jSao mA8A== X-Gm-Message-State: AOAM530y7ansV+mWQRBhEcLjaAVZumuhTOZoFsMlm4OdRYB69zO4Mg+W FmoNPAtO1+v1BFH9+GlBR2w= X-Google-Smtp-Source: ABdhPJwrb7HJV9HhwNvNZ4oa0OmTF8RoFPyx59jmPxqApPtPntlDOJEcyLgJkbpsXbKA494+Pn+H8w== X-Received: by 2002:a17:90a:9306:b0:1bc:9256:5477 with SMTP id p6-20020a17090a930600b001bc92565477mr9339370pjo.170.1646375684474; Thu, 03 Mar 2022 22:34:44 -0800 (PST) Received: from ip-172-31-19-208.ap-northeast-1.compute.internal (ec2-18-181-137-102.ap-northeast-1.compute.amazonaws.com. [18.181.137.102]) by smtp.gmail.com with ESMTPSA id v10-20020a056a00148a00b004e0f420dd90sm4900007pfu.40.2022.03.03.22.34.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Mar 2022 22:34:44 -0800 (PST) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: linux-mm@kvack.org Cc: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Marco Elver , Matthew WilCox , Roman Gushchin , linux-kernel@vger.kernel.org, 42.hyeyoo@gmail.com Subject: [PATCH v2 1/5] mm/slab: kmalloc: pass requests larger than order-1 page to page allocator Date: Fri, 4 Mar 2022 06:34:23 +0000 Message-Id: <20220304063427.372145-2-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20220304063427.372145-1-42.hyeyoo@gmail.com> References: <20220304063427.372145-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" There is not much benefit for serving large objects in kmalloc(). Let's pass large requests to page allocator like SLUB for better maintenance of common code. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- include/linux/slab.h | 35 ++++++++++++++++------------------- mm/slab.c | 31 +++++++++++++++++++++++++++---- mm/slab.h | 19 +++++++++++++++++++ mm/slub.c | 19 ------------------- 4 files changed, 62 insertions(+), 42 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 37bde99b74af..e7b3330db4f3 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -224,29 +224,19 @@ void kmem_dump_obj(void *object); * Kmalloc array related definitions */ =20 -#ifdef CONFIG_SLAB /* - * The largest kmalloc size supported by the SLAB allocators is - * 32 megabyte (2^25) or the maximum allocatable page order if that is - * less than 32 MB. - * - * WARNING: Its not easy to increase this value since the allocators have - * to do various tricks to work around compiler limitations in order to - * ensure proper constant folding. + * SLAB and SLUB directly allocates requests fitting in to an order-1 page + * (PAGE_SIZE*2). Larger requests are passed to the page allocator. */ -#define KMALLOC_SHIFT_HIGH ((MAX_ORDER + PAGE_SHIFT - 1) <=3D 25 ? \ - (MAX_ORDER + PAGE_SHIFT - 1) : 25) -#define KMALLOC_SHIFT_MAX KMALLOC_SHIFT_HIGH +#ifdef CONFIG_SLAB +#define KMALLOC_SHIFT_HIGH (PAGE_SHIFT + 1) +#define KMALLOC_SHIFT_MAX (MAX_ORDER + PAGE_SHIFT - 1) #ifndef KMALLOC_SHIFT_LOW #define KMALLOC_SHIFT_LOW 5 #endif #endif =20 #ifdef CONFIG_SLUB -/* - * SLUB directly allocates requests fitting in to an order-1 page - * (PAGE_SIZE*2). Larger requests are passed to the page allocator. - */ #define KMALLOC_SHIFT_HIGH (PAGE_SHIFT + 1) #define KMALLOC_SHIFT_MAX (MAX_ORDER + PAGE_SHIFT - 1) #ifndef KMALLOC_SHIFT_LOW @@ -564,15 +554,15 @@ static __always_inline __alloc_size(1) void *kmalloc_= large(size_t size, gfp_t fl * Try really hard to succeed the allocation but fail * eventually. */ +#ifndef CONFIG_SLOB static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t fl= ags) { if (__builtin_constant_p(size)) { -#ifndef CONFIG_SLOB unsigned int index; -#endif + if (size > KMALLOC_MAX_CACHE_SIZE) return kmalloc_large(size, flags); -#ifndef CONFIG_SLOB + index =3D kmalloc_index(size); =20 if (!index) @@ -581,10 +571,17 @@ static __always_inline __alloc_size(1) void *kmalloc(= size_t size, gfp_t flags) return kmem_cache_alloc_trace( kmalloc_caches[kmalloc_type(flags)][index], flags, size); -#endif } return __kmalloc(size, flags); } +#else +static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t fl= ags) +{ + if (__builtin_constant_p(size) && size > KMALLOC_MAX_CACHE_SIZE) + return kmalloc_large(size, flags); + return __kmalloc(size, flags); +} +#endif =20 static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp= _t flags, int node) { diff --git a/mm/slab.c b/mm/slab.c index ddf5737c63d9..570af6dc3478 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3624,7 +3624,8 @@ __do_kmalloc_node(size_t size, gfp_t flags, int node,= unsigned long caller) void *ret; =20 if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) - return NULL; + return kmalloc_large(size, flags); + cachep =3D kmalloc_slab(size, flags); if (unlikely(ZERO_OR_NULL_PTR(cachep))) return cachep; @@ -3685,7 +3686,8 @@ static __always_inline void *__do_kmalloc(size_t size= , gfp_t flags, void *ret; =20 if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) - return NULL; + return kmalloc_large(size, flags); + cachep =3D kmalloc_slab(size, flags); if (unlikely(ZERO_OR_NULL_PTR(cachep))) return cachep; @@ -3739,14 +3741,21 @@ void kmem_cache_free_bulk(struct kmem_cache *orig_s= , size_t size, void **p) { struct kmem_cache *s; size_t i; + struct folio *folio; =20 local_irq_disable(); for (i =3D 0; i < size; i++) { void *objp =3D p[i]; =20 - if (!orig_s) /* called via kfree_bulk */ + if (!orig_s) { + /* called via kfree_bulk */ + folio =3D virt_to_folio(objp); + if (unlikely(!folio_test_slab(folio))) { + free_large_kmalloc(folio, objp); + continue; + } s =3D virt_to_cache(objp); - else + } else s =3D cache_from_obj(orig_s, objp); if (!s) continue; @@ -3776,11 +3785,20 @@ void kfree(const void *objp) { struct kmem_cache *c; unsigned long flags; + struct folio *folio; + void *object =3D (void *) objp; =20 trace_kfree(_RET_IP_, objp); =20 if (unlikely(ZERO_OR_NULL_PTR(objp))) return; + + folio =3D virt_to_folio(objp); + if (unlikely(!folio_test_slab(folio))) { + free_large_kmalloc(folio, object); + return; + } + local_irq_save(flags); kfree_debugcheck(objp); c =3D virt_to_cache(objp); @@ -4211,12 +4229,17 @@ void __check_heap_object(const void *ptr, unsigned = long n, size_t __ksize(const void *objp) { struct kmem_cache *c; + struct folio *folio; size_t size; =20 BUG_ON(!objp); if (unlikely(objp =3D=3D ZERO_SIZE_PTR)) return 0; =20 + folio =3D virt_to_folio(objp); + if (!folio_test_slab(folio)) + return folio_size(folio); + c =3D virt_to_cache(objp); size =3D c ? c->object_size : 0; =20 diff --git a/mm/slab.h b/mm/slab.h index c7f2abc2b154..31e98beb47a3 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -664,6 +664,25 @@ static inline struct kmem_cache *cache_from_obj(struct= kmem_cache *s, void *x) print_tracking(cachep, x); return cachep; } + +static __always_inline void kfree_hook(void *x) +{ + kmemleak_free(x); + kasan_kfree_large(x); +} + +static inline void free_large_kmalloc(struct folio *folio, void *object) +{ + unsigned int order =3D folio_order(folio); + + if (WARN_ON_ONCE(order =3D=3D 0)) + pr_warn_once("object pointer: 0x%p\n", object); + + kfree_hook(object); + mod_lruvec_page_state(folio_page(folio, 0), NR_SLAB_UNRECLAIMABLE_B, + -(PAGE_SIZE << order)); + __free_pages(folio_page(folio, 0), order); +} #endif /* CONFIG_SLOB */ =20 static inline size_t slab_ksize(const struct kmem_cache *s) diff --git a/mm/slub.c b/mm/slub.c index 261474092e43..04fd084f4709 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1686,12 +1686,6 @@ static inline void *kmalloc_large_node_hook(void *pt= r, size_t size, gfp_t flags) return ptr; } =20 -static __always_inline void kfree_hook(void *x) -{ - kmemleak_free(x); - kasan_kfree_large(x); -} - static __always_inline bool slab_free_hook(struct kmem_cache *s, void *x, bool init) { @@ -3535,19 +3529,6 @@ struct detached_freelist { struct kmem_cache *s; }; =20 -static inline void free_large_kmalloc(struct folio *folio, void *object) -{ - unsigned int order =3D folio_order(folio); - - if (WARN_ON_ONCE(order =3D=3D 0)) - pr_warn_once("object pointer: 0x%p\n", object); - - kfree_hook(object); - mod_lruvec_page_state(folio_page(folio, 0), NR_SLAB_UNRECLAIMABLE_B, - -(PAGE_SIZE << order)); - __free_pages(folio_page(folio, 0), order); -} - /* * This function progressively scans the array with free objects (with * a limited look ahead) and extract objects belonging to the same --=20 2.33.1 From nobody Tue Jun 23 13:09:43 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AE2AC433F5 for ; Fri, 4 Mar 2022 06:35:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238517AbiCDGga (ORCPT ); Fri, 4 Mar 2022 01:36:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44562 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238497AbiCDGgP (ORCPT ); Fri, 4 Mar 2022 01:36:15 -0500 Received: from mail-pf1-x432.google.com (mail-pf1-x432.google.com [IPv6:2607:f8b0:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BE3F518DAAD for ; Thu, 3 Mar 2022 22:34:47 -0800 (PST) Received: by mail-pf1-x432.google.com with SMTP id g1so6862322pfv.1 for ; Thu, 03 Mar 2022 22:34:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sl+tnBHk3Om3exFNoQL1k9gm+o8msiRC3imZlWXFCFQ=; b=RqVWgGr9J9yOyOmhf/M7Rf5rVoQmBacywljxXUt7owe+MoiBcPp5It1tMurj0PjnXd Ouqe/V6NBZrcvo6w7ZMuEdxfbsjAkSoFES01WeJvCt6D3HdrZvRJXFnMbIvexDn/FyhD iBQzMeqSvC0nBRzJ88nbEK9TTmcnpWFpAxvynTA7kLxRKnDcYMzhsuN3zD9VWmGIrmOM vhEzX4e6v7tTBdZfWd55w0xwJa+3Yup/4gmOEjWsj/dO0PJNxiG4madqTytAua4HdpvS 04mkkhHJhosVGbv66Ql6zJ2KJ/C0GcJtbwewtqAQtFtaP+l7OfjN6qUJehTr+x3SRHe9 UhgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sl+tnBHk3Om3exFNoQL1k9gm+o8msiRC3imZlWXFCFQ=; b=g9Pd08BtkrnkKkvselJ0RqlOR8gfLmktibbFUn8VOdoLgtZEQk4ySts6Gc0a+26akt 3OeVnda8yVni7SG6yNFSdb4ETS/y3zxjTHDNMOqxkg5Bra6y6uSUQlvV41zQGAzLpitW 4ZILNqMrtpfvgKPXbS8Ye9OROZe5ND3zdG3kg2LyPeyDZbTQ1BPLywJp6qfTd0lhc2F2 77P00LIlaYzTildqD6gtRsLuwkRYE/ICQAdi0uFv89Bn3lfgtemfqZ81GU5BxGbqN4aL qpToaxpGXWAV8rBu9Nu2BYgWuqFlPZA7XU7wpSgbrU2seMdPA7YOldl/sTgWGSbYV3k/ k0Rg== X-Gm-Message-State: AOAM533GzZSMiPqr4W55b2stKTDqewPmmqxYe31CCFo9YG8bJGiC5uR0 ElElTJl6iXsD4CT/uhlpSr61LAtbyeeGCw== X-Google-Smtp-Source: ABdhPJxAx/kaLkY9NudzLA8xsz3v1HTz5vDFFp5IWgY5okd3A/kN+0kX2WWhmiiBZ7RvUzsi5ADCrg== X-Received: by 2002:a63:cd02:0:b0:378:9b24:4a9 with SMTP id i2-20020a63cd02000000b003789b2404a9mr20323397pgg.327.1646375687212; Thu, 03 Mar 2022 22:34:47 -0800 (PST) Received: from ip-172-31-19-208.ap-northeast-1.compute.internal (ec2-18-181-137-102.ap-northeast-1.compute.amazonaws.com. [18.181.137.102]) by smtp.gmail.com with ESMTPSA id v10-20020a056a00148a00b004e0f420dd90sm4900007pfu.40.2022.03.03.22.34.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Mar 2022 22:34:47 -0800 (PST) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: linux-mm@kvack.org Cc: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Marco Elver , Matthew WilCox , Roman Gushchin , linux-kernel@vger.kernel.org, 42.hyeyoo@gmail.com Subject: [PATCH v2 2/5] mm/sl[au]b: unify __ksize() Date: Fri, 4 Mar 2022 06:34:24 +0000 Message-Id: <20220304063427.372145-3-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20220304063427.372145-1-42.hyeyoo@gmail.com> References: <20220304063427.372145-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Now that SLAB passes large requests to page allocator like SLUB, Unify __ksize(). Only SLOB need to implement own version of __ksize() because it stores size in object header for kmalloc objects. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Vlastimil Babka --- mm/slab.c | 30 ------------------------------ mm/slab_common.c | 27 +++++++++++++++++++++++++++ mm/slub.c | 16 ---------------- 3 files changed, 27 insertions(+), 46 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 570af6dc3478..3ddf2181d8e4 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -4216,33 +4216,3 @@ void __check_heap_object(const void *ptr, unsigned l= ong n, usercopy_abort("SLAB object", cachep->name, to_user, offset, n); } #endif /* CONFIG_HARDENED_USERCOPY */ - -/** - * __ksize -- Uninstrumented ksize. - * @objp: pointer to the object - * - * Unlike ksize(), __ksize() is uninstrumented, and does not provide the s= ame - * safety checks as ksize() with KASAN instrumentation enabled. - * - * Return: size of the actual memory used by @objp in bytes - */ -size_t __ksize(const void *objp) -{ - struct kmem_cache *c; - struct folio *folio; - size_t size; - - BUG_ON(!objp); - if (unlikely(objp =3D=3D ZERO_SIZE_PTR)) - return 0; - - folio =3D virt_to_folio(objp); - if (!folio_test_slab(folio)) - return folio_size(folio); - - c =3D virt_to_cache(objp); - size =3D c ? c->object_size : 0; - - return size; -} -EXPORT_SYMBOL(__ksize); diff --git a/mm/slab_common.c b/mm/slab_common.c index 23f2ab0713b7..1d2f92e871d2 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1245,6 +1245,33 @@ void kfree_sensitive(const void *p) } EXPORT_SYMBOL(kfree_sensitive); =20 +#ifndef CONFIG_SLOB +/** + * __ksize -- Uninstrumented ksize. + * @objp: pointer to the object + * + * Unlike ksize(), __ksize() is uninstrumented, and does not provide the s= ame + * safety checks as ksize() with KASAN instrumentation enabled. + * + * Return: size of the actual memory used by @objp in bytes + */ +size_t __ksize(const void *object) +{ + struct folio *folio; + + if (unlikely(object =3D=3D ZERO_SIZE_PTR)) + return 0; + + folio =3D virt_to_folio(object); + + if (unlikely(!folio_test_slab(folio))) + return folio_size(folio); + + return slab_ksize(folio_slab(folio)->slab_cache); +} +EXPORT_SYMBOL(__ksize); +#endif + /** * ksize - get the actual amount of memory allocated for a given object * @objp: Pointer to the object diff --git a/mm/slub.c b/mm/slub.c index 04fd084f4709..6f0ebadd8f30 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4507,22 +4507,6 @@ void __check_heap_object(const void *ptr, unsigned l= ong n, } #endif /* CONFIG_HARDENED_USERCOPY */ =20 -size_t __ksize(const void *object) -{ - struct folio *folio; - - if (unlikely(object =3D=3D ZERO_SIZE_PTR)) - return 0; - - folio =3D virt_to_folio(object); - - if (unlikely(!folio_test_slab(folio))) - return folio_size(folio); - - return slab_ksize(folio_slab(folio)->slab_cache); -} -EXPORT_SYMBOL(__ksize); - void kfree(const void *x) { struct folio *folio; --=20 2.33.1 From nobody Tue Jun 23 13:09:43 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D68CC433F5 for ; Fri, 4 Mar 2022 06:35:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238570AbiCDGge (ORCPT ); Fri, 4 Mar 2022 01:36:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43594 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238481AbiCDGgQ (ORCPT ); Fri, 4 Mar 2022 01:36:16 -0500 Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E77FE18E3E8 for ; Thu, 3 Mar 2022 22:34:50 -0800 (PST) Received: by mail-pj1-x102f.google.com with SMTP id m11-20020a17090a7f8b00b001beef6143a8so7164480pjl.4 for ; Thu, 03 Mar 2022 22:34:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kPOOk68I4eD0EZtqHYKQaSrGBjBqSrtM4mnuv4oRjyQ=; b=nDQzt1phNSGQYKWCNk0gp2OnBhDzbFsLtVoMQZYprv1JockjSyVxMyhajm1xm7lr7V AWmFM71q4nLmK+IbMdcN4MrfIWcWUSusSJ2Ak1sFRxCRVgo9+yRWwtHiBLiO5kN6l0Cf yLLWwtivZxP28j4U1PbUfMvuJgkWTHpSgrnltIZ0HxRydMzgP58+XIo8ze8XNWAnBf2r jxc5GN7nej6IJEd/CXES7lSAJZ89+/JWxuMRhSr1m6iMiAkoym1w3oFIxqXIbLz27+YQ I4aOWtlXWIArSgaMdhm6eVGUNNI2Nn3Ars/MW3LDXBiy2RoYV6zkphnKYaYNQ1rT3Fwa B46Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kPOOk68I4eD0EZtqHYKQaSrGBjBqSrtM4mnuv4oRjyQ=; b=pXentCywoqSr30KC+dN7fPjSaIpW+wsbuf3xJWY4bN/79g1HTfOIpMzXKSyA5UhXnJ JVYSCOI6e/ay0FqX4EkbFxDneVXTJ+O+HHuRv1vrWFWg+0kYnuPcqykm6W6zn85Q7gH+ rg5KzmB0OJM3w8eQIyrpL5WTJiGOWolvvcoH6d12ZaCde9vjcaQiW14aPBt0jDoo4Dz3 TcA3MQNzl8Q7aLqFpN6JXEmeXy1UgpBn/wCumMmmyRoAy9GSO60VnGRaUxcR+9ajOW9n QF0gHnFH+K3Y5DsNUKuaSMra+Xrt3qromqQPGpl1XT+kMBERwmvpLY6dXnu/oIXsspzS UPjA== X-Gm-Message-State: AOAM530dqYRpBVFjSjE9EqxbTkkU13hroxjZRNUBMnKCt+/i3d3uO6Kl kfKIbcRT6sTtVHFxiG75QAU= X-Google-Smtp-Source: ABdhPJx6Xk2mOdK3KCJoE/feOSg8DDkvCzteGHm0Tcc9N7Tq32/YEHDYtA7ATrAkhbz8I8OkNz0/Dg== X-Received: by 2002:a17:90a:eb07:b0:1be:f354:9404 with SMTP id j7-20020a17090aeb0700b001bef3549404mr9122314pjz.90.1646375689983; Thu, 03 Mar 2022 22:34:49 -0800 (PST) Received: from ip-172-31-19-208.ap-northeast-1.compute.internal (ec2-18-181-137-102.ap-northeast-1.compute.amazonaws.com. [18.181.137.102]) by smtp.gmail.com with ESMTPSA id v10-20020a056a00148a00b004e0f420dd90sm4900007pfu.40.2022.03.03.22.34.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Mar 2022 22:34:49 -0800 (PST) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: linux-mm@kvack.org Cc: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Marco Elver , Matthew WilCox , Roman Gushchin , linux-kernel@vger.kernel.org, 42.hyeyoo@gmail.com Subject: [PATCH v2 3/5] mm/sl[auo]b: move definition of __ksize() to mm/slab.h Date: Fri, 4 Mar 2022 06:34:25 +0000 Message-Id: <20220304063427.372145-4-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20220304063427.372145-1-42.hyeyoo@gmail.com> References: <20220304063427.372145-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" __ksize() is only called by KASAN. Remove export symbol and move definition to mm/slab.h as we don't want to grow its callers. [ willy@infradead.org: Move definition to mm/slab.h and reduce comments ] Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Vlastimil Babka --- include/linux/slab.h | 1 - mm/slab.h | 2 ++ mm/slab_common.c | 9 +-------- mm/slob.c | 1 - 4 files changed, 3 insertions(+), 10 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index e7b3330db4f3..d2b896553315 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -182,7 +182,6 @@ int kmem_cache_shrink(struct kmem_cache *s); void * __must_check krealloc(const void *objp, size_t new_size, gfp_t flag= s) __alloc_size(2); void kfree(const void *objp); void kfree_sensitive(const void *objp); -size_t __ksize(const void *objp); size_t ksize(const void *objp); #ifdef CONFIG_PRINTK bool kmem_valid_obj(void *object); diff --git a/mm/slab.h b/mm/slab.h index 31e98beb47a3..79b319d58504 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -685,6 +685,8 @@ static inline void free_large_kmalloc(struct folio *fol= io, void *object) } #endif /* CONFIG_SLOB */ =20 +size_t __ksize(const void *objp); + static inline size_t slab_ksize(const struct kmem_cache *s) { #ifndef CONFIG_SLUB diff --git a/mm/slab_common.c b/mm/slab_common.c index 1d2f92e871d2..b126fc7247b9 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1247,13 +1247,7 @@ EXPORT_SYMBOL(kfree_sensitive); =20 #ifndef CONFIG_SLOB /** - * __ksize -- Uninstrumented ksize. - * @objp: pointer to the object - * - * Unlike ksize(), __ksize() is uninstrumented, and does not provide the s= ame - * safety checks as ksize() with KASAN instrumentation enabled. - * - * Return: size of the actual memory used by @objp in bytes + * __ksize -- Uninstrumented ksize. Only called by KASAN. */ size_t __ksize(const void *object) { @@ -1269,7 +1263,6 @@ size_t __ksize(const void *object) =20 return slab_ksize(folio_slab(folio)->slab_cache); } -EXPORT_SYMBOL(__ksize); #endif =20 /** diff --git a/mm/slob.c b/mm/slob.c index 60c5842215f1..d8af6c54f133 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -588,7 +588,6 @@ size_t __ksize(const void *block) m =3D (unsigned int *)(block - align); return SLOB_UNITS(*m) * SLOB_UNIT; } -EXPORT_SYMBOL(__ksize); =20 int __kmem_cache_create(struct kmem_cache *c, slab_flags_t flags) { --=20 2.33.1 From nobody Tue Jun 23 13:09:43 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60913C433EF for ; Fri, 4 Mar 2022 06:35:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238576AbiCDGgh (ORCPT ); Fri, 4 Mar 2022 01:36:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44804 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238472AbiCDGgS (ORCPT ); Fri, 4 Mar 2022 01:36:18 -0500 Received: from mail-pf1-x432.google.com (mail-pf1-x432.google.com [IPv6:2607:f8b0:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 70F8018E430 for ; Thu, 3 Mar 2022 22:34:53 -0800 (PST) Received: by mail-pf1-x432.google.com with SMTP id d17so6886071pfl.0 for ; Thu, 03 Mar 2022 22:34:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Ks5pvMqQR8K26b0NEwSqAgaMYIVmHGUi4tZdn97VHAs=; b=mn/MfoEj7ex4cmWohzuiwMeU/dZ0xaA4vmjw8XtRuG2VBO7jFKIZkJ6oC7nxEnlPCU 1WZ1wLqzmp4zRfAh1HWUQLkVtbW7dmuR6q4FHPMIIzZ0NAiL9IGsbUVoomTUyxiac2CG F5DiPVczqJKVJNP55QPIQxeP2up+AbCNUuNTSJDU5nuBFZjf5FbOGiL2wHoCDaUA+ZS2 SMuXS50A+ZEdGex9kFFp9bU5DwfFMmhd/sUA088joOH3oLz+XBULwpkKsts4Hv/XUCTP M4oXr61f00V6ykH++i/cZx4YKwEsoqxKtenjUKqtj+ekJxDsba+46cUsSS6vTOg1nwy0 faqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ks5pvMqQR8K26b0NEwSqAgaMYIVmHGUi4tZdn97VHAs=; b=RtSDGQZiUHV6AvZZ9PPX7M3ZGJ2xgeyvt8QELydqN1QO3dMy8ln0PMID+JsIP6uLJX cD9bVxc466kBxrlwu3PCk2rSPUjhbG92PjHk2JIMskQQivdqFjIapQczG80i6AOS5I6K MDUenhd6rUgnvXNOCJl3EbmUFmj9PUlpgDSexRzcwPVbe3GX/OtdIimu5IaWMYbTh0X3 jO/7yZWXb+sMmDfdYLsGTW68hVMdkvF7fT7+7GaXGWVpP/iQnqVjB/J++CBn3+NjNc7C lkIBwWSLKBwRiPBuZg8maGeGmo6XHsonmwGqKhmUkbMgT4t7+oQuSwXcH1r7UV2T0MYP fpTA== X-Gm-Message-State: AOAM5311M9AxZQdYx/kbx6W5InEsHQQUAqFgxYZ5pNdcSiHhZn3qBycU 5GH1ceowmyWMompZblOWrg4= X-Google-Smtp-Source: ABdhPJyEhvk8aBpqhMLNfeybmgloUiwg+1evBGGHCq6W0x0KqLIVEGGOo3A5PVK/zxqLLXfIyyj3nw== X-Received: by 2002:a63:df0e:0:b0:378:4f83:496f with SMTP id u14-20020a63df0e000000b003784f83496fmr27591895pgg.560.1646375692737; Thu, 03 Mar 2022 22:34:52 -0800 (PST) Received: from ip-172-31-19-208.ap-northeast-1.compute.internal (ec2-18-181-137-102.ap-northeast-1.compute.amazonaws.com. [18.181.137.102]) by smtp.gmail.com with ESMTPSA id v10-20020a056a00148a00b004e0f420dd90sm4900007pfu.40.2022.03.03.22.34.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Mar 2022 22:34:52 -0800 (PST) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: linux-mm@kvack.org Cc: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Marco Elver , Matthew WilCox , Roman Gushchin , linux-kernel@vger.kernel.org, 42.hyeyoo@gmail.com Subject: [PATCH v2 4/5] mm/slub: limit number of node partial slabs only in cache creation Date: Fri, 4 Mar 2022 06:34:26 +0000 Message-Id: <20220304063427.372145-5-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20220304063427.372145-1-42.hyeyoo@gmail.com> References: <20220304063427.372145-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" SLUB sets number of minimum partial slabs for node (min_partial) using set_min_partial(). SLUB holds at least min_partial slabs even if they're empty to avoid excessive use of page allocator. set_min_partial() limits value of min_partial limits value of min_partial MIN_PARTIAL and MAX_PARTIAL. As set_min_partial() can be called by min_partial_store() too, Only limit value of min_partial in kmem_cache_open() so that it can be changed to value that a user wants. [ rientjes@google.com: Fold set_min_partial() into its callers ] Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Vlastimil Babka --- mm/slub.c | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 6f0ebadd8f30..f9ae983a3dc6 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3981,15 +3981,6 @@ static int init_kmem_cache_nodes(struct kmem_cache *= s) return 1; } =20 -static void set_min_partial(struct kmem_cache *s, unsigned long min) -{ - if (min < MIN_PARTIAL) - min =3D MIN_PARTIAL; - else if (min > MAX_PARTIAL) - min =3D MAX_PARTIAL; - s->min_partial =3D min; -} - static void set_cpu_partial(struct kmem_cache *s) { #ifdef CONFIG_SLUB_CPU_PARTIAL @@ -4196,7 +4187,8 @@ static int kmem_cache_open(struct kmem_cache *s, slab= _flags_t flags) * The larger the object size is, the more slabs we want on the partial * list to avoid pounding the page allocator excessively. */ - set_min_partial(s, ilog2(s->size) / 2); + s->min_partial =3D min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); + s->min_partial =3D max_t(unsigned long, MIN_PARTIAL, s->min_partial); =20 set_cpu_partial(s); =20 @@ -5361,7 +5353,7 @@ static ssize_t min_partial_store(struct kmem_cache *s= , const char *buf, if (err) return err; =20 - set_min_partial(s, min); + s->min_partial =3D min; return length; } SLAB_ATTR(min_partial); --=20 2.33.1 From nobody Tue Jun 23 13:09:43 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 527B8C433EF for ; Fri, 4 Mar 2022 06:35:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235465AbiCDGgl (ORCPT ); Fri, 4 Mar 2022 01:36:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44964 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238482AbiCDGgT (ORCPT ); Fri, 4 Mar 2022 01:36:19 -0500 Received: from mail-pj1-x1033.google.com (mail-pj1-x1033.google.com [IPv6:2607:f8b0:4864:20::1033]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C124B18F226 for ; Thu, 3 Mar 2022 22:34:55 -0800 (PST) Received: by mail-pj1-x1033.google.com with SMTP id mv5-20020a17090b198500b001bf2a039831so546029pjb.5 for ; Thu, 03 Mar 2022 22:34:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PgzWIJiGbElIzAA/+g3O4R+XIWRWlKNYEXwjILpxSlg=; b=VcsP531XOAZHoSzYutr+YrhPB/02f1REI7a2paVkswi+VQ1RJWPGg8afSzCgogHzyW dOUmDbKrKlQ2pIGPipEijTjuCJ2N6uNcjHbyMswPnO5gSGm6KKC8lgnMd5CDOze45fm6 ox2otFm8rAec4bBP8ZSzRbgS3yv+isnJSeAwbZSwGe0Xkpa/AIWIQs8BoVspdQDH/ejc RRFaT/TeqUqitkGA1WFWDZ75k4/o1UNUPoYyOumZB47EYuTWitcDPlm20xlJHkYfJCpF UAvLaQjJ1EFDEDIzDPfSx00I4XvsOPzMJ4rQuu9LeeLw8WyY2JgBU6nlJxw2wLfrXsCY DdbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PgzWIJiGbElIzAA/+g3O4R+XIWRWlKNYEXwjILpxSlg=; b=Xj7FTaJC07cVddU5dswXuBgerXmqWRnKSAf2GlJzBmknMC0pYEaMRyr70mcvWW/QeP zoYhszVC/fs7221dw5xpU+JXcqCaiF4HfgQG5AQe4nsGrcfB0giivGpgjX8gADjj6ldf G3VTHgGBqgGHTCE8mVaoQGvPmwS1tgAV9BhfHZDetR2fIsvqmCWTHDQ/B8oYbrd3H6Do rneuaxqdgPQX75dvMtg3QvahHAVFGfDQvs82LZIuZ3riC+w1fRNy+m+4C7gcV6PyDBjs WcSBJIGMdFJJhkVzS54TfTa4T3/9naXm/Q9Ax0cddT3/HdBPMDL+flyANGJTSRw+xC5s 3IHQ== X-Gm-Message-State: AOAM531mjlqVwQaCnvHDUj9RK+snT5bPzmAk/VMyb+aE3qzdF0pM40Iy To6SAi2TD269eFQll8eObeo= X-Google-Smtp-Source: ABdhPJxUr2XDiJRBOKg+kXq0iUgKNnV8N4aWOKmy0bwGTHBAdBDB2wi3hQmPcbrds4di6UaI00Dbuw== X-Received: by 2002:a17:902:c401:b0:14f:b686:e6ff with SMTP id k1-20020a170902c40100b0014fb686e6ffmr15079443plk.45.1646375695469; Thu, 03 Mar 2022 22:34:55 -0800 (PST) Received: from ip-172-31-19-208.ap-northeast-1.compute.internal (ec2-18-181-137-102.ap-northeast-1.compute.amazonaws.com. [18.181.137.102]) by smtp.gmail.com with ESMTPSA id v10-20020a056a00148a00b004e0f420dd90sm4900007pfu.40.2022.03.03.22.34.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Mar 2022 22:34:55 -0800 (PST) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: linux-mm@kvack.org Cc: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Marco Elver , Matthew WilCox , Roman Gushchin , linux-kernel@vger.kernel.org, 42.hyeyoo@gmail.com Subject: [PATCH v2 5/5] mm/slub: refactor deactivate_slab() Date: Fri, 4 Mar 2022 06:34:27 +0000 Message-Id: <20220304063427.372145-6-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20220304063427.372145-1-42.hyeyoo@gmail.com> References: <20220304063427.372145-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Simplify deactivate_slab() by unlocking n->list_lock and retrying cmpxchg_double() when cmpxchg_double() fails, and perform add_{partial,full} only when it succeed. Releasing and taking n->list_lock again here is not harmful as SLUB avoids deactivating slabs as much as possible. [ vbabka@suse.cz: perform add_{partial,full} when cmpxchg_double() succeed. ] Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slub.c | 81 ++++++++++++++++++++++--------------------------------- 1 file changed, 32 insertions(+), 49 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index f9ae983a3dc6..c1a693ec5874 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2344,8 +2344,8 @@ static void deactivate_slab(struct kmem_cache *s, str= uct slab *slab, { enum slab_modes { M_NONE, M_PARTIAL, M_FULL, M_FREE }; struct kmem_cache_node *n =3D get_node(s, slab_nid(slab)); - int lock =3D 0, free_delta =3D 0; - enum slab_modes l =3D M_NONE, m =3D M_NONE; + int free_delta =3D 0; + enum slab_modes mode =3D M_NONE; void *nextfree, *freelist_iter, *freelist_tail; int tail =3D DEACTIVATE_TO_HEAD; unsigned long flags =3D 0; @@ -2387,14 +2387,10 @@ static void deactivate_slab(struct kmem_cache *s, s= truct slab *slab, * Ensure that the slab is unfrozen while the list presence * reflects the actual number of objects during unfreeze. * - * We setup the list membership and then perform a cmpxchg - * with the count. If there is a mismatch then the slab - * is not unfrozen but the slab is on the wrong list. - * - * Then we restart the process which may have to remove - * the slab from the list that we just put it on again - * because the number of objects in the slab may have - * changed. + * We first perform cmpxchg holding lock and insert to list + * when it succeed. If there is mismatch then slub is not + * unfrozen and number of objects in the slab may have changed. + * Then release lock and retry cmpxchg again. */ redo: =20 @@ -2414,57 +2410,44 @@ static void deactivate_slab(struct kmem_cache *s, s= truct slab *slab, new.frozen =3D 0; =20 if (!new.inuse && n->nr_partial >=3D s->min_partial) - m =3D M_FREE; + mode =3D M_FREE; else if (new.freelist) { - m =3D M_PARTIAL; - if (!lock) { - lock =3D 1; - /* - * Taking the spinlock removes the possibility that - * acquire_slab() will see a slab that is frozen - */ - spin_lock_irqsave(&n->list_lock, flags); - } - } else { - m =3D M_FULL; - if (kmem_cache_debug_flags(s, SLAB_STORE_USER) && !lock) { - lock =3D 1; - /* - * This also ensures that the scanning of full - * slabs from diagnostic functions will not see - * any frozen slabs. - */ - spin_lock_irqsave(&n->list_lock, flags); - } + mode =3D M_PARTIAL; + /* + * Taking the spinlock removes the possibility that + * acquire_slab() will see a slab that is frozen + */ + spin_lock_irqsave(&n->list_lock, flags); + } else if (kmem_cache_debug_flags(s, SLAB_STORE_USER)) { + mode =3D M_FULL; + /* + * This also ensures that the scanning of full + * slabs from diagnostic functions will not see + * any frozen slabs. + */ + spin_lock_irqsave(&n->list_lock, flags); } =20 - if (l !=3D m) { - if (l =3D=3D M_PARTIAL) - remove_partial(n, slab); - else if (l =3D=3D M_FULL) - remove_full(s, n, slab); - - if (m =3D=3D M_PARTIAL) - add_partial(n, slab, tail); - else if (m =3D=3D M_FULL) - add_full(s, n, slab); - } =20 - l =3D m; if (!cmpxchg_double_slab(s, slab, old.freelist, old.counters, new.freelist, new.counters, - "unfreezing slab")) + "unfreezing slab")) { + if (mode =3D=3D M_PARTIAL || mode =3D=3D M_FULL) + spin_unlock_irqrestore(&n->list_lock, flags); goto redo; + } =20 - if (lock) - spin_unlock_irqrestore(&n->list_lock, flags); =20 - if (m =3D=3D M_PARTIAL) + if (mode =3D=3D M_PARTIAL) { + add_partial(n, slab, tail); + spin_unlock_irqrestore(&n->list_lock, flags); stat(s, tail); - else if (m =3D=3D M_FULL) + } else if (mode =3D=3D M_FULL) { + add_full(s, n, slab); + spin_unlock_irqrestore(&n->list_lock, flags); stat(s, DEACTIVATE_FULL); - else if (m =3D=3D M_FREE) { + } else if (mode =3D=3D M_FREE) { stat(s, DEACTIVATE_EMPTY); discard_slab(s, slab); stat(s, FREE_SLAB); --=20 2.33.1