From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EA3901DBB20 for ; Mon, 7 Oct 2024 16:49:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319796; cv=none; b=CKKZ7IJN+9v7nFL/nejOZm8WNWTc9XQ5AF7Y/jU0CcWuZTBi+HU29C2TZH6NzFwYgU+mqUFwYLRQtRKL+w5UmPo0Ano1VmR9MN/R/S/DwckSWU11SiDJ4rHIUcfyvs2RIDa7Ro9qMQnCOsvG46JhlhVzxHjGclZ/j2LJNZW+Yrw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319796; c=relaxed/simple; bh=k0DnZZu43ruD+bdSb4T6LCSi2wCBz0Hja4zGcT8s0mw=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=TtUh138q++0d0HGnJjE3YKZKWL1Ie/H4stDdjtAqBZ4hmPcvBhqT9gcLJJt3bctelsEDMnmqTBGjIAlGDaCYrH1yOsQ++g/CaouYIADf9X3g0QFzXQG/7OJY5U72Jl31Aaze8oH9sLK+CDsMEsxJJETV20/OM8N4e8T4+8OZIdQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=JNtOPjbB; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=+9ulgnTo; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="JNtOPjbB"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="+9ulgnTo" Message-ID: <20241007164913.009849239@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319792; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=R5gfZFizKr+dgdbeMxiwbggyU0Kkmipobr0FHuBYryc=; b=JNtOPjbB05KL37x7osBOUnibEtZ0i5bRN9nWMo/ujao+h5nBl6489z5mON7lNW+T7cpHeB GbVVDMibe/rsINJyReKEZ1fQXuMAzRatD5fsJvR7eZ3cM1eRjCHLTQLMWG2GQQuH7c6zfe 3VHxUaI9vC2rHJ/htoVLa8ztxLWObbkPU4P/ub1KQ/8C88kGuQ19kXvcQoZdPhqQi+DGY3 eHbFsLx+HMctKzr0gbJale1ctxmpvOXkBbeRzTSVbTUUnbwFlwKXcbgPesCrINRJdhdl3t oY99tnLnzCypKv1gYbi2+H1vR8V42KHt94O9Y/uAA1sQHwFN0ND7waK9zn3VrA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319792; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=R5gfZFizKr+dgdbeMxiwbggyU0Kkmipobr0FHuBYryc=; b=+9ulgnToVXOlDSURnoX/ZbH5raqwQq7Xd8GL06L+KTH74o9Vh89sco2soxc14pN6rnrt7f JiNjlAnzpO68p0DA== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 01/25] debugobjects: Delete a piece of redundant code References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:49:52 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Zhen Lei The statically allocated objects are all located in obj_static_pool[], the whole memory of obj_static_pool[] will be reclaimed later. Therefore, there is no need to split the remaining statically nodes in list obj_pool into isolated ones, no one will use them anymore. Just write INIT_HLIST_HEAD(&obj_pool) is enough. Since hlist_move_list() directly discards the old list, even this can be omitted. Signed-off-by: Zhen Lei Signed-off-by: Thomas Gleixner Link: https://lore.kernel.org/all/20240911083521.2257-2-thunder.leizhen@hua= wei.com --- lib/debugobjects.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -1325,10 +1325,10 @@ static int __init debug_objects_replace_ * active object references. */ =20 - /* Remove the statically allocated objects from the pool */ - hlist_for_each_entry_safe(obj, tmp, &obj_pool, node) - hlist_del(&obj->node); - /* Move the allocated objects to the pool */ + /* + * Replace the statically allocated objects list with the allocated + * objects list. + */ hlist_move_list(&objects, &obj_pool); =20 /* Replace the active object references */ From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7446F1DB352 for ; Mon, 7 Oct 2024 16:49:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319797; cv=none; b=D32Z6aKJkm6jgNyxH4fMcl9/JLWB9D65810HiKBCIYFPGqsTlGqLLTQQkn1S3iwG2vdvYwqOanFfJgKCjsMcNW97EnvTP35udM9WfLawc7Xkly3cotjxvbNGFyNE3sLkLefLv4rMF6FFLuTtaGNKnPYZVWwhfep/IqXI6tZMVmE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319797; c=relaxed/simple; bh=DxyC468MIKBKRepgOYqobCuLab+7VZZ8uQul0v7O7v8=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=gBhy5amRcKX7HLuh4gO4IItH1kHrspJ+62/tw5B3F0rOySzZZSbsPoVv6l870aw9D1oMqZQHeXwEpY8LK0MeN3DIs9nayrVp+hmeEIDAFHnGRe1kJ+FL5zen5SF5egn81HJwBP7RcWHQ+5D7+bb9+QvhK2ExXqu8oqFZpT1VqGo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=1vEf7qlL; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=D3w7uJPD; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="1vEf7qlL"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="D3w7uJPD" Message-ID: <20241007164913.073653668@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319793; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=mavBQBTQJ1+PBT60+7EBJrSCYznE7ddqxiJvXAlj+sw=; b=1vEf7qlLAaQO9cJSVc/+ikcdrtg+Z9VGqz//7c/tt5UWgaVCm9PKuY0nZmktp8gN7D7W6p Csyvo/gRgEX0D4F4EhSHK8PWzsoh6kngEN8uy94jAeq0hV+/Jz5BLfgtE9PpbMKd9n9ZOD A2CDjo9zUQ1R3FcG+43FzK2fSDiyz4ufwQxSwTgfY4WSqtCoFKYBO1k9l0BoNntXYaaij8 zquqzVK+3zLVPUYYEDXxqIEFqiK8F0TCCv8IJOQ7S9GZQmMRqWjLCQdWRI598V8gUnAN1q hl2K/WFw726tdOYqcFSgmp4Rl4cwNSVu07n5JXHdNRW3QB9MhvCi9qMCYG1SeQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319793; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=mavBQBTQJ1+PBT60+7EBJrSCYznE7ddqxiJvXAlj+sw=; b=D3w7uJPDosW5AcAtJP9qG70VmjfBW1tj1Sr4dyM8dFvJyP/+DZ+FygqVEIR5T+Hr6rzQKG 9wf/a/scicx3I9Dw== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 02/25] debugobjects: Collect newly allocated objects in a list to reduce lock contention References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:49:53 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Zhen Lei Collect the newly allocated debug objects in a list outside the lock, so that the lock held time and the potential lock contention is reduced. Signed-off-by: Zhen Lei Signed-off-by: Thomas Gleixner Link: https://lore.kernel.org/all/20240911083521.2257-3-thunder.leizhen@hua= wei.com --- lib/debugobjects.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -161,23 +161,25 @@ static void fill_pool(void) return; =20 while (READ_ONCE(obj_pool_free) < debug_objects_pool_min_level) { - struct debug_obj *new[ODEBUG_BATCH_SIZE]; + struct debug_obj *new, *last =3D NULL; + HLIST_HEAD(head); int cnt; =20 for (cnt =3D 0; cnt < ODEBUG_BATCH_SIZE; cnt++) { - new[cnt] =3D kmem_cache_zalloc(obj_cache, gfp); - if (!new[cnt]) + new =3D kmem_cache_zalloc(obj_cache, gfp); + if (!new) break; + hlist_add_head(&new->node, &head); + if (!last) + last =3D new; } if (!cnt) return; =20 raw_spin_lock_irqsave(&pool_lock, flags); - while (cnt) { - hlist_add_head(&new[--cnt]->node, &obj_pool); - debug_objects_allocated++; - WRITE_ONCE(obj_pool_free, obj_pool_free + 1); - } + hlist_splice_init(&head, &last->node, &obj_pool); + debug_objects_allocated +=3D cnt; + WRITE_ONCE(obj_pool_free, obj_pool_free + cnt); raw_spin_unlock_irqrestore(&pool_lock, flags); } } From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 173211DC19E for ; Mon, 7 Oct 2024 16:49:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319798; cv=none; b=EkxxH+EkhaCF4lZRuAGF6MR1P6VBq8sYF9fZFDly7wFhzBc5L0sjtJyJXwFuzY06DrAfzMxzHQ5CImJVc8AEbH2vNUFYHZ84wz8GvVE0scmbg9Xe9EyRL9sHzk4iCIz60QsWVRxrbj4U+ak7SYrDNXKouC53gvdOPyUY55LFx3I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319798; c=relaxed/simple; bh=CDFExI07QVdtD4rjmNZ/S8LS9EUlOppWlLJr4Y2xIM4=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=MDrDfTpE+daRdXqlhTdah3MUQYWqP6m9H7PgKrQ+ArYoF0ubeJHmasPrnwQdCsR4g/U6c5plsSbtlFJkXUidEuZ7d2cZOfHrALNyEZG5x2AsE29xF6UV5APXfVLTmpH+WtVexqVWTcVs0E+OFcX3nkf86dqsvfxeYk10NdZbbWQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=I5voKT0C; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=GUfWTlem; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="I5voKT0C"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="GUfWTlem" Message-ID: <20241007164913.137021337@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319795; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=rwVsd0p+SkppHSCskbEcnj79RSfR5XhMLN6cQUML+8o=; b=I5voKT0CnI1KLxFAO/JD1uJsWX8dpRPaD6qGJ3shQHp9gCN2XQYqPrOZVJqxvUl3a0rWuJ wFOSwEYtp9Fu84+PrJ3bvL4Q3iftIEklxXrWyFkGBKKgD4VpGv2WHwFOhbU2l6Mio9Lnmt dPT3t96vP8RPtvd6a7blOT+G7+yfqlhhaQeUzAx0+QmD9KJxO4LPUwjBo1Y+XXH6o5wk69 uN/nvLrl5s8b0UUzgQFtXaK64I1L6mKuAwm39BXIhpMOSjIiZwY66/KaPgIflhjfVsu/aB iCM+o54d3uJ5vmJFbta2b0Y44qW4Nqc/i8Axfma13ys9Pmpm5agMGf/Rq71NDQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319795; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=rwVsd0p+SkppHSCskbEcnj79RSfR5XhMLN6cQUML+8o=; b=GUfWTlem6G6KUraRtn0r1DJgovJNlPsZeh4GoiBf89ZBNQYQbAr6lGtFcvZBUn9apr2DNm XoAGYgJnf8okDJBg== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 03/25] debugobjects: Dont destroy kmem cache in init() References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:49:54 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" debug_objects_mem_init() is invoked from mm_core_init() before work queues are available. If debug_objects_mem_init() destroys the kmem cache in the error path it causes an Oops in __queue_work(): Oops: Oops: 0000 [#1] PREEMPT SMP PTI RIP: 0010:__queue_work+0x35/0x6a0 queue_work_on+0x66/0x70 flush_all_cpus_locked+0xdf/0x1a0 __kmem_cache_shutdown+0x2f/0x340 kmem_cache_destroy+0x4e/0x150 mm_core_init+0x9e/0x120 start_kernel+0x298/0x800 x86_64_start_reservations+0x18/0x30 x86_64_start_kernel+0xc5/0xe0 common_startup_64+0x12c/0x138 Further the object cache pointer is used in various places to check for early boot operation. It is exposed before the replacments for the static boot time objects are allocated and the self test operates on it. This can be avoided by: 1) Running the self test with the static boot objects 2) Exposing it only after the replacement objects have been added to the pool. Signed-off-by: Thomas Gleixner --- lib/debugobjects.c | 68 +++++++++++++++++++++++++++---------------------= ----- 1 file changed, 35 insertions(+), 33 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -1211,7 +1211,7 @@ static __initconst const struct debug_ob =20 static __initdata struct self_test obj =3D { .static_init =3D 0 }; =20 -static void __init debug_objects_selftest(void) +static bool __init debug_objects_selftest(void) { int fixups, oldfixups, warnings, oldwarnings; unsigned long flags; @@ -1280,9 +1280,10 @@ static void __init debug_objects_selftes descr_test =3D NULL; =20 local_irq_restore(flags); + return !!debug_objects_enabled; } #else -static inline void debug_objects_selftest(void) { } +static inline bool debug_objects_selftest(void) { return true; } #endif =20 /* @@ -1302,18 +1303,21 @@ void __init debug_objects_early_init(voi } =20 /* - * Convert the statically allocated objects to dynamic ones: + * Convert the statically allocated objects to dynamic ones. + * debug_objects_mem_init() is called early so only one CPU is up and + * interrupts are disabled, which means it is safe to replace the active + * object references. */ -static int __init debug_objects_replace_static_objects(void) +static bool __init debug_objects_replace_static_objects(struct kmem_cache = *cache) { struct debug_bucket *db =3D obj_hash; - struct hlist_node *tmp; struct debug_obj *obj, *new; + struct hlist_node *tmp; HLIST_HEAD(objects); int i, cnt =3D 0; =20 for (i =3D 0; i < ODEBUG_POOL_SIZE; i++) { - obj =3D kmem_cache_zalloc(obj_cache, GFP_KERNEL); + obj =3D kmem_cache_zalloc(cache, GFP_KERNEL); if (!obj) goto free; hlist_add_head(&obj->node, &objects); @@ -1322,12 +1326,6 @@ static int __init debug_objects_replace_ debug_objects_allocated +=3D i; =20 /* - * debug_objects_mem_init() is now called early that only one CPU is up - * and interrupts have been disabled, so it is safe to replace the - * active object references. - */ - - /* * Replace the statically allocated objects list with the allocated * objects list. */ @@ -1347,15 +1345,14 @@ static int __init debug_objects_replace_ } } =20 - pr_debug("%d of %d active objects replaced\n", - cnt, obj_pool_used); - return 0; + pr_debug("%d of %d active objects replaced\n", cnt, obj_pool_used); + return true; free: hlist_for_each_entry_safe(obj, tmp, &objects, node) { hlist_del(&obj->node); - kmem_cache_free(obj_cache, obj); + kmem_cache_free(cache, obj); } - return -ENOMEM; + return false; } =20 /* @@ -1366,6 +1363,7 @@ static int __init debug_objects_replace_ */ void __init debug_objects_mem_init(void) { + struct kmem_cache *cache; int cpu, extras; =20 if (!debug_objects_enabled) @@ -1380,29 +1378,33 @@ void __init debug_objects_mem_init(void) for_each_possible_cpu(cpu) INIT_HLIST_HEAD(&per_cpu(percpu_obj_pool.free_objs, cpu)); =20 - obj_cache =3D kmem_cache_create("debug_objects_cache", - sizeof (struct debug_obj), 0, - SLAB_DEBUG_OBJECTS | SLAB_NOLEAKTRACE, - NULL); + if (!debug_objects_selftest()) + return; + + cache =3D kmem_cache_create("debug_objects_cache", sizeof (struct debug_o= bj), 0, + SLAB_DEBUG_OBJECTS | SLAB_NOLEAKTRACE, NULL); =20 - if (!obj_cache || debug_objects_replace_static_objects()) { + if (!cache || !debug_objects_replace_static_objects(cache)) { debug_objects_enabled =3D 0; - kmem_cache_destroy(obj_cache); - pr_warn("out of memory.\n"); + pr_warn("Out of memory.\n"); return; - } else - debug_objects_selftest(); - -#ifdef CONFIG_HOTPLUG_CPU - cpuhp_setup_state_nocalls(CPUHP_DEBUG_OBJ_DEAD, "object:offline", NULL, - object_cpu_offline); -#endif + } =20 /* - * Increase the thresholds for allocating and freeing objects - * according to the number of possible CPUs available in the system. + * Adjust the thresholds for allocating and freeing objects + * according to the number of possible CPUs available in the + * system. */ extras =3D num_possible_cpus() * ODEBUG_BATCH_SIZE; debug_objects_pool_size +=3D extras; debug_objects_pool_min_level +=3D extras; + + /* Everything worked. Expose the cache */ + obj_cache =3D cache; + +#ifdef CONFIG_HOTPLUG_CPU + cpuhp_setup_state_nocalls(CPUHP_DEBUG_OBJ_DEAD, "object:offline", NULL, + object_cpu_offline); +#endif + return; } From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A8E6C1DC1B8 for ; Mon, 7 Oct 2024 16:49:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319799; cv=none; b=RcupH9KmPRDmlNxcTAuHEtUawFIEObUpOfHlNFM8nSu30CkH2bahCDIp1k5AVH5B+vw3jyGhM2l1Ld4IzyP6++g2eiIobBiBjRkHvzV15VfhI8jhgrbjU2gg1n8UaDpcvCVB/k5fpzAn1GN7kSKb1QRQiL6SVLmbdj/kIE5twHU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319799; c=relaxed/simple; bh=gNky2JPv/g//hDu0oNfoJPwDLycaraDu9NRDZgc4NCI=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=CG9FjvJhvkLfZzE9ynju4TlYcSj7nULSAfGadFPtqrVMso1m9OKJj4cusBZuPyavJg2Ih8mTediCBNVXs02xDLAS5GwB7BI9hl3ePbjdxSAafeXXC794zGc2NEfqyM0Q/NINMXEb/ExmS8uvFWxbQpd78kcV88BKs9T8XZ4SUr8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=MThGZPqW; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=Wq5Ym2T+; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="MThGZPqW"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="Wq5Ym2T+" Message-ID: <20241007164913.200379308@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319796; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=CDVx47swvYRM3WGljRSXbOAUqFkI8FfkKJLHbg/0eM8=; b=MThGZPqW8sXwCuitR9YA6mIUfJUpwCUi0hFPMyGXz8pM5TwQC7t6mBmed18MmNcomUY7cE 65PnqtWyj0ej4+cdRFUccO0XchdTf2hd5HF7Nl7+A8m5al5LuxKUhujZKz+2dQ95n7+Kqs wYBQXGbZzEMyHWrlSAf5BMdIbDt9XVcdBpssmNifhwNWa3mubwegiZHKDLi9BT9kxFuPdS 2g6w6ai5Mr4Y1UEqDC4rQZSANkptj+RmcM7cy1Ul5KFgD+g6JVnDKm7MajF/Dqhetx07pv MZUGYrYl0x3nMQIrF7GyNFeGJ1LnXskjnCvXQZJ/pMCNJHoxm7wgxJy7XNDLuA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319796; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=CDVx47swvYRM3WGljRSXbOAUqFkI8FfkKJLHbg/0eM8=; b=Wq5Ym2T+ky2cDXJV+n6ZYv1rD4yDiHeim2JKyj9Ka5ovrEvTe6QDWu1xgaptZ3k1/Wc8+A 9u8ivtk8U9k+luCw== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 04/25] debugobjects: Remove pointless hlist initialization References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:49:56 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" It's BSS zero initialized. Signed-off-by: Thomas Gleixner Reviewed-by: Zhen Lei --- lib/debugobjects.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -1364,20 +1364,11 @@ static bool __init debug_objects_replace void __init debug_objects_mem_init(void) { struct kmem_cache *cache; - int cpu, extras; + int extras; =20 if (!debug_objects_enabled) return; =20 - /* - * Initialize the percpu object pools - * - * Initialization is not strictly necessary, but was done for - * completeness. - */ - for_each_possible_cpu(cpu) - INIT_HLIST_HEAD(&per_cpu(percpu_obj_pool.free_objs, cpu)); - if (!debug_objects_selftest()) return; From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 87C491DCB03 for ; Mon, 7 Oct 2024 16:49:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319801; cv=none; b=YlaPJfqfl36+/0HVPHsD6aJQsVkaYqoMVlv8UkzL3GHiY3DLIkFfvnKU3mWwXP8AN1pRVFRE6GX7/0Ev5ErqjKfhAMtbyuhzEjsi2ET/totIUWWGmeIhdkX4qIR+7UvmqqBj4cAeoj+Dy6p/33qpeOog72YMIQTXHm/dJLKy98U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319801; c=relaxed/simple; bh=2V1uwOf9Lz5kfl8cAFnIf1hL1Ej6o91RjqpnkrVfhWI=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=JDlKH7NuCI6LnVkyGkceY14FbcelULOXjBs/O8gM6T6B96wmJwfZyqA/Bs2jLTJasHTeDhLhih11DWp9yb/kaJjJYdW8Izlp30B8Pi6sHMzH+jPgNt1tEB3UZfvbj9OsUbX0oysTC856y/ePP2JYZx2qrirkUy8miKQzGbMt9LQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=H+yJyBTQ; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=G/Y/BTVC; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="H+yJyBTQ"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="G/Y/BTVC" Message-ID: <20241007164913.263960570@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319797; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=MtciriuSO8ml4j4OAkJX3p/tvIF+cexwWzSghsv9eHk=; b=H+yJyBTQ+13VMbrAlsejigARsspFDkVphs+AIlMAMa0r3kRFcmPw+Ob0TKPVHjMRG7U575 azrn6wxbISAx2/Boiw479TgeVJk6jLvmNPMGUASCgM6sJ06stmmuQyYLHyn3b6ei8wUZNn 8N9tbrGsPowGGev2NHfW7KQP/34Av8uuFluk55UjLT+yFmmqw8t+aTFAZZKFCZ00qX8mdx 9yETXwSxizEgdEcBam2NUasvdAtEDSD1LjwY+IjjA5JFZRco/J/po6P2qG/emtFFY7z/M2 qNFVTsRDPCUxV7lRNfrWph4HGx60dwRxTGwI1Cq7/6kpH532mGQzyp5SdCiZPQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319797; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=MtciriuSO8ml4j4OAkJX3p/tvIF+cexwWzSghsv9eHk=; b=G/Y/BTVCpimC8aVVLx+4dq1l+puOMFl2C4Llvw5iilR8Pgeh0cLvIEx5CG1L6qafBH9ltR dCktcPowCLNLheAg== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 05/25] debugobjects: Dont free objects directly on CPU hotplug References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:49:57 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Freeing the per CPU pool of the unplugged CPU directly is suboptimal as the objects can be reused in the real pool if there is room. Aside of that this gets the accounting wrong. Use the regular free path, which allows reuse and has the accounting correc= t. Signed-off-by: Thomas Gleixner Reviewed-by: Zhen Lei --- lib/debugobjects.c | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -430,27 +430,28 @@ static void free_object(struct debug_obj } =20 #ifdef CONFIG_HOTPLUG_CPU -static int object_cpu_offline(unsigned int cpu) +static void put_objects(struct hlist_head *list) { - struct debug_percpu_free *percpu_pool; struct hlist_node *tmp; struct debug_obj *obj; - unsigned long flags; =20 - /* Remote access is safe as the CPU is dead already */ - percpu_pool =3D per_cpu_ptr(&percpu_obj_pool, cpu); - hlist_for_each_entry_safe(obj, tmp, &percpu_pool->free_objs, node) { + /* + * Using free_object() puts the objects into reuse or schedules + * them for freeing and it get's all the accounting correct. + */ + hlist_for_each_entry_safe(obj, tmp, list, node) { hlist_del(&obj->node); - kmem_cache_free(obj_cache, obj); + free_object(obj); } +} =20 - raw_spin_lock_irqsave(&pool_lock, flags); - obj_pool_used -=3D percpu_pool->obj_free; - debug_objects_freed +=3D percpu_pool->obj_free; - raw_spin_unlock_irqrestore(&pool_lock, flags); - - percpu_pool->obj_free =3D 0; +static int object_cpu_offline(unsigned int cpu) +{ + /* Remote access is safe as the CPU is dead already */ + struct debug_percpu_free *pcp =3D per_cpu_ptr(&percpu_obj_pool, cpu); =20 + put_objects(&pcp->free_objs); + pcp->obj_free =3D 0; return 0; } #endif From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E33BC1DC745 for ; Mon, 7 Oct 2024 16:50:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319802; cv=none; b=ECJlAJsY2Ve1e4V4smyzT2mESMsRsXYJ1rwEDMqFVWmRhvFFend9XV11oIGG8EoookYl6YNGwNxULkVt23Lg0b2v0+Y59nU+cvehkXDpJvX3rr06tDcO6y3yyFBjBfJeyGQVyK3Vlrq94wLhxQ3XazrN3KnZXyUtjFESSuKYRlY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319802; c=relaxed/simple; bh=dNGVc4bYaNKNEvCSBdFCpq1aOrxV8/ArUQ6Icm0wUQ4=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=Rtxc9jmgBFsDRLEGjNfSZ8e9nLEp7TC4LLO/cIuleWOBypzXx9zCcgXMw1qzHlIbkBbuY2EnKhHfk94GLz8k/SpJ0V9ihwuGv13sVIiuRtHHiSbdX8QE978j7WRai+g/E1MldUApGj530NeBGXnDp+apJSqNIGSFIsPyNslqSIs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=kU3uOOxT; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=vLf79YYx; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="kU3uOOxT"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="vLf79YYx" Message-ID: <20241007164913.326834268@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319798; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=NzOkvYQgrh58j6+yH6duWn05tqHa4WhiuqidyYO6bOY=; b=kU3uOOxTAQleQYG85E0qxa+Pn0hwlabW+kxAUlNWfwgsiALnmJyqFTJx/qE2tKd/7gVLzU my4l/idtWobv3+J/Mvyk38TIoloRrpt9MejxBrb4E1/OCohfI20A+EjK0S6E2EJGw5QklA gl02Va0sq3JiQL0M2GTuoqPJEXiMaMu4bLMpg7oQQXTTP+ORMqRPZYkXPN328YRXvNy8GN ImRFcm8GwhedA5QxcvIjX84aBb4lEWi9w77OGKTO28I4O1INFSYXNpgjjOqAX0eaUy2sKU 05QQN8tY5UXXw42SA9cBfnU2MYNO6aFgxhNdvB166hZYfHdqA221rdtbr6rOwA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319798; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=NzOkvYQgrh58j6+yH6duWn05tqHa4WhiuqidyYO6bOY=; b=vLf79YYxNSScHjNhamjUmH1b0IM2FjoQx6/rkCQ4gQ9ElxCoxzGrM9oFbzS2t83LbB3trx UZqcRlmDyV2M2HDQ== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 06/25] debugobjects: Reuse put_objects() on OOM References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:49:58 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Reuse the helper function instead of having a open coded copy. Signed-off-by: Thomas Gleixner Reviewed-by: Zhen Lei --- lib/debugobjects.c | 26 +++++++------------------- 1 file changed, 7 insertions(+), 19 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -429,7 +429,6 @@ static void free_object(struct debug_obj } } =20 -#ifdef CONFIG_HOTPLUG_CPU static void put_objects(struct hlist_head *list) { struct hlist_node *tmp; @@ -445,6 +444,7 @@ static void put_objects(struct hlist_hea } } =20 +#ifdef CONFIG_HOTPLUG_CPU static int object_cpu_offline(unsigned int cpu) { /* Remote access is safe as the CPU is dead already */ @@ -456,31 +456,19 @@ static int object_cpu_offline(unsigned i } #endif =20 -/* - * We run out of memory. That means we probably have tons of objects - * allocated. - */ +/* Out of memory. Free all objects from hash */ static void debug_objects_oom(void) { struct debug_bucket *db =3D obj_hash; - struct hlist_node *tmp; HLIST_HEAD(freelist); - struct debug_obj *obj; - unsigned long flags; - int i; =20 pr_warn("Out of memory. ODEBUG disabled\n"); =20 - for (i =3D 0; i < ODEBUG_HASH_SIZE; i++, db++) { - raw_spin_lock_irqsave(&db->lock, flags); - hlist_move_list(&db->list, &freelist); - raw_spin_unlock_irqrestore(&db->lock, flags); - - /* Now free them */ - hlist_for_each_entry_safe(obj, tmp, &freelist, node) { - hlist_del(&obj->node); - free_object(obj); - } + for (int i =3D 0; i < ODEBUG_HASH_SIZE; i++, db++) { + scoped_guard(raw_spinlock_irqsave, &db->lock) + hlist_move_list(&db->list, &freelist); + + put_objects(&freelist); } } From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C88A31DCB30 for ; Mon, 7 Oct 2024 16:50:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319803; cv=none; b=uDdyMMkGwKdnsp8uoGKFXyNcArOJGvpW5slDoKo7YIrfKq3znHxPzgnfauSOkyHw9Zse2yuYU005TOxL3q3WbmtHnVQ5F1oE/adBiB1qvaqQY70Gy3dPNngOLM6fjCmCswh/u9fvBrjxb/VLdN0RhkywjthGO2N2BSI+37QEz8I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319803; c=relaxed/simple; bh=aEu6hJ9U7PO6szkgs96fkDOHN4VNXLaykYx7x//6BQ8=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=FF83TLZXyiXJ1Zv3JZN0FHBun8T162H4bJRSTGAbPJ5KMlK4heIL3sXZlKnyKBIB9fAysoLePfvuUMYcYa5MtTxsQ2p6Nzcn5f29icTp9o4ZXs55uq3e85hwcb3I3J6m1oXp5xihv8KWXS/m4EFe0wCqmM2sujuRVTYEY9qdY54= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=xeKJzp0J; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=TsI2GUa9; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="xeKJzp0J"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="TsI2GUa9" Message-ID: <20241007164913.390511021@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319800; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=ogJg4brbf+ZHDcbh3aDpvZl7gvfFvI3DSNAZrNum05Y=; b=xeKJzp0JjdX+cqCjLkFcbdnon6N04WRSupArA1ZolC1VlD5PD7nhT7zY2UADaObguz3KGG fFJdyNhRH5+tZn451ho2AfnqWCeQA0e0RZ5AiOaPaiVJhSN2tnEsrdpQBVM3OUI+m4RanO E/Tvf1kRR41ezIyAwsHVTSoHO9ZYa+AVRDu5yjfXXsjbGI6Luzc/mYrKM8LXW9jtrXJGjg MqKEPVjQOmVe8/dnQlOXuIAxySfmOvftt+eQeoFmDKMz+W/CIBdGeUtarrsVFatGmu1Vuw JzGcx0faD+e3sKEjh1b2MdNiJcQGdF+w41pqkQ8p5Li8qi0Kamu/B8u7OiP80A== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319800; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=ogJg4brbf+ZHDcbh3aDpvZl7gvfFvI3DSNAZrNum05Y=; b=TsI2GUa9/51zEpBk+eorG65abOheUyI4yniE4ffXL+tEqu0LNfJedtzxDuVF9daDIZ6R1E 83nd6TjBi48TuxAg== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 07/25] debugobjects: Remove pointless debug printk References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:49:59 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" It has zero value. Signed-off-by: Thomas Gleixner Reviewed-by: Zhen Lei --- lib/debugobjects.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -1303,7 +1303,7 @@ static bool __init debug_objects_replace struct debug_obj *obj, *new; struct hlist_node *tmp; HLIST_HEAD(objects); - int i, cnt =3D 0; + int i; =20 for (i =3D 0; i < ODEBUG_POOL_SIZE; i++) { obj =3D kmem_cache_zalloc(cache, GFP_KERNEL); @@ -1330,11 +1330,8 @@ static bool __init debug_objects_replace /* copy object data */ *new =3D *obj; hlist_add_head(&new->node, &db->list); - cnt++; } } - - pr_debug("%d of %d active objects replaced\n", cnt, obj_pool_used); return true; free: hlist_for_each_entry_safe(obj, tmp, &objects, node) { From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CBF4D1DD527 for ; Mon, 7 Oct 2024 16:50:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319804; cv=none; b=Vua9aYlhi5qa33no70S64AMnZUo9dFUXSFGA6/5pQrWJSku7MvBcfVqU4QVwVHMVKyXHGUegs5n46Iy0crHSC5u3v+T0ULD6mtrTRnT3wD/bLqYU5AcXp+v/ufeTcT/Aheeikjstrrr+qdAQXINKbhMc5mD4m9rZmjOjPDPszvQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319804; c=relaxed/simple; bh=huwrr8yoIiLO+sycKuHEO61YqWvJ5wMCX42IOJL6d0U=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=GDPHhW8kay+CGxyLQRSgrqMmC/6Pc/KloMm7hZNAKuwcJq6iPijShapF8ID/StgKZ1otOYUecNbZwWCLcSoFSl1Q2Mq13qxaxLBL3TipHJYdYnyCYiKiDj6lfAFuvtmMy7iMf6vlLS5MbjUL2Qe5j8m+gml6gI8DcbbFY1B6OQU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=J94TJ8OT; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=52ymK6iS; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="J94TJ8OT"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="52ymK6iS" Message-ID: <20241007164913.453912357@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319801; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=3yz7XTTQgqOO6f/M5A+HCXYlu+Zj5Ag/dtYwShwS6fQ=; b=J94TJ8OTOuRHEGjzTKATV3UBkqhiXBbAFhsigEKq7PWDX3T9abMNr/UMT6oNeyugPWHj0O KZNeWnLB3qOthPsASJb5Rdo+UHKYs1PfQEPDWQdVT1FFYVlqsQXYJujIfSmM1PqAfYQVnH GDKBB2Ut7IdcFNtxqb2VopQc91sG/czF5JsExRoYkYnK5MhscxbAXmzGjKVAbGI9HYQWzt p7EkenFvFpPLtUDDNBpUXlfxQMwra2UFgse7rgdmYkFmo/5QB6c32xurVSNH9pZg75onHO zSzONmlI6zon0Psw4HRIg58Hg6D2WvwYY2U9wXDlD/p3LFYLaJGoZ2cTxhyqvQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319801; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=3yz7XTTQgqOO6f/M5A+HCXYlu+Zj5Ag/dtYwShwS6fQ=; b=52ymK6iSLfchVJuFo65BgMT/qhPoylOecdRa7zOdF2wRQ9fUinNUeTiStoET/ioNniJ3Pf y/CGyiSq9U6aXICw== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 08/25] debugobjects: Provide and use free_object_list() References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:50:01 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move the loop to free a list of objects into a helper function so it can be reused later. Signed-off-by: Thomas Gleixner --- lib/debugobjects.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -125,6 +125,20 @@ static const char *obj_states[ODEBUG_STA [ODEBUG_STATE_NOTAVAILABLE] =3D "not available", }; =20 +static void free_object_list(struct hlist_head *head) +{ + struct hlist_node *tmp; + struct debug_obj *obj; + int cnt =3D 0; + + hlist_for_each_entry_safe(obj, tmp, head, node) { + hlist_del(&obj->node); + kmem_cache_free(obj_cache, obj); + cnt++; + } + debug_objects_freed +=3D cnt; +} + static void fill_pool(void) { gfp_t gfp =3D __GFP_HIGH | __GFP_NOWARN; @@ -286,7 +300,6 @@ alloc_object(void *addr, struct debug_bu */ static void free_obj_work(struct work_struct *work) { - struct hlist_node *tmp; struct debug_obj *obj; unsigned long flags; HLIST_HEAD(tofree); @@ -323,15 +336,11 @@ static void free_obj_work(struct work_st */ if (obj_nr_tofree) { hlist_move_list(&obj_to_free, &tofree); - debug_objects_freed +=3D obj_nr_tofree; WRITE_ONCE(obj_nr_tofree, 0); } raw_spin_unlock_irqrestore(&pool_lock, flags); =20 - hlist_for_each_entry_safe(obj, tmp, &tofree, node) { - hlist_del(&obj->node); - kmem_cache_free(obj_cache, obj); - } + free_object_list(&tofree); } =20 static void __free_object(struct debug_obj *obj) @@ -1334,6 +1343,7 @@ static bool __init debug_objects_replace } return true; free: + /* Can't use free_object_list() as the cache is not populated yet */ hlist_for_each_entry_safe(obj, tmp, &objects, node) { hlist_del(&obj->node); kmem_cache_free(cache, obj); From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1506B1DD526 for ; Mon, 7 Oct 2024 16:50:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319806; cv=none; b=l17SrgdIYTKDUiGTWrkiNKZeemqAFtqnhWCUFAI2j69UkKsIQ+m69XhyOpgLqBGKDAPFnhiKuhBc5rL/GyUCXYbz02xKfJXkw7MSYazegg4B4pfd13GRg+egokE0LNUlzIwc1jwvWqQ0avyB0ytJ18ZQi/PMPH0Nu0tBRPJLxOw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319806; c=relaxed/simple; bh=BDvn3gp1vcsg9Kvk8ehmQgzJPA6KnCVv5EqekyqqmB4=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=lwgzyuqlZG0tJWgvTUfvV9ir7YLR4NDMiwjKH/4Aae3+QUTNZTrB/sG8Nvgb2R8GuBtwEKHvxqTPQzaxEpTTrnsZKL03OWHYP13Zk8WVjuDmOmlIACUxz4sEYv/R3/vnlQ0EzvKFUPOkLTvCQBDfyitDdjV3m0MgO6j/rDANasc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=wVSVv/bj; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=0wmAidJn; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="wVSVv/bj"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="0wmAidJn" Message-ID: <20241007164913.518175013@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319802; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=x8QWV5IHGvnzH9KgJaFacufBWJ4RoY7ciLDJXLETEmU=; b=wVSVv/bjsHzP9tZ9Q7UJF/pdjpOhP2wJJvsjKZY1S89z3CtsDCpy4dhEZPp9p0tVvifJqk QqQHD/w/+d8lMWFw+QV6a2xqWAkrABbdSLydRvZk8eHPcoqgDM68f24r7/dj8kboPE/uiX IO0z/ttIcVibuIVH58MWIuxOs8LLiq79PAeJqpKLZ0kXbrOTHlpliI3T7nwJYnB6CBKrfW ubhenrSbOPC37IZaGOrKCnJVsosPKC/yvtWhcSzJWgGNks7e2HcION2ap8+GwUMhC+b8aa 4efS40jenayvbp6VGTlnMWEYfwtzTF/sMebjAGAaqWVE+/ivFCdsHxIq7cEtqA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319802; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=x8QWV5IHGvnzH9KgJaFacufBWJ4RoY7ciLDJXLETEmU=; b=0wmAidJnoNM6mUoIxLAcMsUZMH+B+2H8tnlWuHD2/rwmM0B8X5fa/MMlqO32wnbEr9yLsH q2DLRrNAfuFSWRCg== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 09/25] debugobjects: Make debug_objects_enabled bool References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:50:02 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make it what it is. Signed-off-by: Thomas Gleixner Reviewed-by: Zhen Lei --- lib/debugobjects.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -82,7 +82,7 @@ static int __data_racy debug_objects_m static int __data_racy __maybe_unused debug_objects_maxchecked __read_most= ly; static int __data_racy debug_objects_fixups __read_mostly; static int __data_racy debug_objects_warnings __read_mostly; -static int __data_racy debug_objects_enabled __read_mostly +static bool __data_racy debug_objects_enabled __read_mostly =3D CONFIG_DEBUG_OBJECTS_ENABLE_DEFAULT; static int debug_objects_pool_size __ro_after_init =3D ODEBUG_POOL_SIZE; @@ -103,17 +103,16 @@ static DECLARE_DELAYED_WORK(debug_obj_wo =20 static int __init enable_object_debug(char *str) { - debug_objects_enabled =3D 1; + debug_objects_enabled =3D true; return 0; } +early_param("debug_objects", enable_object_debug); =20 static int __init disable_object_debug(char *str) { - debug_objects_enabled =3D 0; + debug_objects_enabled =3D false; return 0; } - -early_param("debug_objects", enable_object_debug); early_param("no_debug_objects", disable_object_debug); =20 static const char *obj_states[ODEBUG_STATE_MAX] =3D { @@ -592,7 +591,7 @@ static struct debug_obj *lookup_object_o } =20 /* Out of memory. Do the cleanup outside of the locked region */ - debug_objects_enabled =3D 0; + debug_objects_enabled =3D false; return NULL; } =20 @@ -1194,7 +1193,7 @@ check_results(void *addr, enum debug_obj out: raw_spin_unlock_irqrestore(&db->lock, flags); if (res) - debug_objects_enabled =3D 0; + debug_objects_enabled =3D false; return res; } =20 @@ -1278,7 +1277,7 @@ static bool __init debug_objects_selftes descr_test =3D NULL; =20 local_irq_restore(flags); - return !!debug_objects_enabled; + return debug_objects_enabled; } #else static inline bool debug_objects_selftest(void) { return true; } @@ -1372,7 +1371,7 @@ void __init debug_objects_mem_init(void) SLAB_DEBUG_OBJECTS | SLAB_NOLEAKTRACE, NULL); =20 if (!cache || !debug_objects_replace_static_objects(cache)) { - debug_objects_enabled =3D 0; + debug_objects_enabled =3D false; pr_warn("Out of memory.\n"); return; } From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 579521DD86A for ; Mon, 7 Oct 2024 16:50:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319807; cv=none; b=D7azhi+V7eTb0N3ay8MGvRaaOYfsTOg12X6C7hrS2a16XE256pn0pmc2iY4y8DdhXnZ2DG0703C1F1cqLv+ziUpPiEW3ebR4vwMlH59CFfRe7g+cIyLeo9uFSYyi/xNyxELrJ7IdG4eUAI5BL4C99twB2R4ZaiNzl3nXrLgLl8g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319807; c=relaxed/simple; bh=w5Qo7dTzecPxFEos9vuCxqFrLH3Wjssljj2B81D2lOQ=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=elC6W8zQqyDCFZJzsmV+2le56u5GFYz2BXGUNEZ1L+m57iEhdzmsOAI274MyDDyvjxlLf1FL33LZFKOQH7TlcXEkiz8EgbU2E7QZFBuvRhecPUBxXHKLQ4Y8hddTJ9hcoZtvJOqLdhiUjIMIqDtvJjOp4zbgyxbq9+eLaQ2CKws= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=HAN1yodO; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=CU+HAOE9; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="HAN1yodO"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="CU+HAOE9" Message-ID: <20241007164913.582118421@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319803; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=MV84TMCcv1W+qbGMeugJJXIH6Qz86ucpwvsGCV7iSjc=; b=HAN1yodOQzfeRGKA2JOdhV1r2+zsUK8DGiFBCJDXdqbeXlb1UJXvrb0QJrwLAw5NUncnpg Tv3eUJr8GDxmpkWGtMVv2URRbcxK72s9V8PRWWMWB4/KrnflsOjFqE3QhJWptdRDuNXhVu 5fMUTLUmpHur1ZK3zW/ASWacBXSgMA7f6BQT7bNTYMtdUdck1/1R0qwiipz4rTaUOxEafq y8PxHscwvlI71uJTPPQAYDxFGrPAhN9RIOSrLj7cWtCXJPKPeCvkncmhwuwsQhoG8Pu0sV l3QWZa8yH+zBgaLjKig0uRouyuCcgGdQOu1gycVGH+Vf4M9MWAfuj1gzSgHwfA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319803; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=MV84TMCcv1W+qbGMeugJJXIH6Qz86ucpwvsGCV7iSjc=; b=CU+HAOE9/r0XuV5Tu46pgjZwXMsdTALHeqnhbu3ytc9D/OfM1vb6OjetIrNQCuR0pN3ZMg IIGjVl0F8ItIIgBg== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 10/25] debugobjects: Reduce parallel pool fill attempts References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:50:03 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Zhen Lei The contention on the global pool_lock can be massive when the global pool needs to be refilled and many CPUs try to handle this. Address this by: - splitting the refill from free list and allocation. Refill from free list has no constraints vs. the context on RT, so it can be tried outside of the RT specific preemptible() guard - Let only one CPU handle the free list - Let only one CPU do allocations unless the pool level is below half of the minimum fill level. Suggested-by: Thomas Gleixner Signed-off-by: Zhen Lei Signed-off-by: Thomas Gleixner Link: https://lore.kernel.org/all/20240911083521.2257-4-thunder.leizhen@hua= wei.com- -- lib/debugobjects.c | 84 +++++++++++++++++++++++++++++++++++++-----------= ----- 1 file changed, 59 insertions(+), 25 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -138,14 +138,10 @@ static void free_object_list(struct hlis debug_objects_freed +=3D cnt; } =20 -static void fill_pool(void) +static void fill_pool_from_freelist(void) { - gfp_t gfp =3D __GFP_HIGH | __GFP_NOWARN; + static unsigned long state; struct debug_obj *obj; - unsigned long flags; - - if (likely(READ_ONCE(obj_pool_free) >=3D debug_objects_pool_min_level)) - return; =20 /* * Reuse objs from the global obj_to_free list; they will be @@ -154,32 +150,58 @@ static void fill_pool(void) * obj_nr_tofree is checked locklessly; the READ_ONCE() pairs with * the WRITE_ONCE() in pool_lock critical sections. */ - if (READ_ONCE(obj_nr_tofree)) { - raw_spin_lock_irqsave(&pool_lock, flags); - /* - * Recheck with the lock held as the worker thread might have - * won the race and freed the global free list already. - */ - while (obj_nr_tofree && (obj_pool_free < debug_objects_pool_min_level)) { - obj =3D hlist_entry(obj_to_free.first, typeof(*obj), node); - hlist_del(&obj->node); - WRITE_ONCE(obj_nr_tofree, obj_nr_tofree - 1); - hlist_add_head(&obj->node, &obj_pool); - WRITE_ONCE(obj_pool_free, obj_pool_free + 1); - } - raw_spin_unlock_irqrestore(&pool_lock, flags); + if (!READ_ONCE(obj_nr_tofree)) + return; + + /* + * Prevent the context from being scheduled or interrupted after + * setting the state flag; + */ + guard(irqsave)(); + + /* + * Avoid lock contention on &pool_lock and avoid making the cache + * line exclusive by testing the bit before attempting to set it. + */ + if (test_bit(0, &state) || test_and_set_bit(0, &state)) + return; + + guard(raw_spinlock)(&pool_lock); + /* + * Recheck with the lock held as the worker thread might have + * won the race and freed the global free list already. + */ + while (obj_nr_tofree && (obj_pool_free < debug_objects_pool_min_level)) { + obj =3D hlist_entry(obj_to_free.first, typeof(*obj), node); + hlist_del(&obj->node); + WRITE_ONCE(obj_nr_tofree, obj_nr_tofree - 1); + hlist_add_head(&obj->node, &obj_pool); + WRITE_ONCE(obj_pool_free, obj_pool_free + 1); } + clear_bit(0, &state); +} =20 - if (unlikely(!obj_cache)) +static void fill_pool(void) +{ + static atomic_t cpus_allocating; + + /* + * Avoid allocation and lock contention when: + * - One other CPU is already allocating + * - the global pool has not reached the critical level yet + */ + if (READ_ONCE(obj_pool_free) > (debug_objects_pool_min_level / 2) && + atomic_read(&cpus_allocating)) return; =20 + atomic_inc(&cpus_allocating); while (READ_ONCE(obj_pool_free) < debug_objects_pool_min_level) { struct debug_obj *new, *last =3D NULL; HLIST_HEAD(head); int cnt; =20 for (cnt =3D 0; cnt < ODEBUG_BATCH_SIZE; cnt++) { - new =3D kmem_cache_zalloc(obj_cache, gfp); + new =3D kmem_cache_zalloc(obj_cache, __GFP_HIGH | __GFP_NOWARN); if (!new) break; hlist_add_head(&new->node, &head); @@ -187,14 +209,14 @@ static void fill_pool(void) last =3D new; } if (!cnt) - return; + break; =20 - raw_spin_lock_irqsave(&pool_lock, flags); + guard(raw_spinlock_irqsave)(&pool_lock); hlist_splice_init(&head, &last->node, &obj_pool); debug_objects_allocated +=3D cnt; WRITE_ONCE(obj_pool_free, obj_pool_free + cnt); - raw_spin_unlock_irqrestore(&pool_lock, flags); } + atomic_dec(&cpus_allocating); } =20 /* @@ -597,6 +619,18 @@ static struct debug_obj *lookup_object_o =20 static void debug_objects_fill_pool(void) { + if (unlikely(!obj_cache)) + return; + + if (likely(READ_ONCE(obj_pool_free) >=3D debug_objects_pool_min_level)) + return; + + /* Try reusing objects from obj_to_free_list */ + fill_pool_from_freelist(); + + if (likely(READ_ONCE(obj_pool_free) >=3D debug_objects_pool_min_level)) + return; + /* * On RT enabled kernels the pool refill must happen in preemptible * context -- for !RT kernels we rely on the fact that spinlock_t and From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 452E61DD883 for ; Mon, 7 Oct 2024 16:50:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319808; cv=none; b=qnRu0QwHVzSFd5JoN1iaIAMxLRG0vMKeWC8HL5sWoaXF5ePICLmJM/MBIAO3rVDezD3ebXPUZ7DdQ/0W5ghKrqAL6+wpuPe/baFAwSkS4v/Hfh8kwBCe3haKfFKiE7Rg/xsZuoaCJO4eA1ICRdd1oCwtluwupXRlNw5vy8+MqNY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319808; c=relaxed/simple; bh=HPimt2xTQLdqBe3t+oeb1naUoy67PZu59PLjIHhEhW4=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=Tik+I/rdFkWXqgQ1b0ECC6xu71/0ZdYsagbZqFiJmQiDBhttybMI+5Q/3qMukPPr5JJ7aZyjvHk4AVMxBV7kK0k5RSq1nhEBgXA010k48R8pAbyK5bLlOzyZ4wsx+/+cZzZq2+YfEXdHaXHBjDJ4tAfQG+RkQq2nKJZsHimM4Zw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=Uh33rLh+; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=oaCZDy3I; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="Uh33rLh+"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="oaCZDy3I" Message-ID: <20241007164913.646171170@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319804; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=TF7x6lIDjivX15b47JvgETzLuxNAFhB3o0FVEA9+89g=; b=Uh33rLh+N6VNySlg2Lpa0yDxZ6z03034koBGBGl+pwxwCf5SAqKL+l/cuH7UNtG3WDOwea i+Pj1eR1vHOXh6W8nFYuMk5ixYIjs1DkzhT3dwA2SVqOIIMaI/VEylqXcd9h+cNk1MAXx1 XAtZV+DAcEpkYUzAgrrc1Xo/DfjMn9beWkW5zuqkBFegFlZYrgaa0NUhOCFOpoMtmgFpPd dXTKiPZ4yf2K/lHe/YuPagSRviHt6XdzJY88KQW9HeLcaGS8lJUkd96J+PH1IUE5+rE7xj pt4v1jiL1MGlHXkzWYXpcQ+vzzmJQMIiZzQNvlsczWVcB0i41OmNascnWS/tFg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319804; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=TF7x6lIDjivX15b47JvgETzLuxNAFhB3o0FVEA9+89g=; b=oaCZDy3IgNT6dxlKnAZF1FG2r6qqAU+bGTJgtsNE37leC93nO4kQknBv0dL643Ga376od8 TlCH5v1G7xbE6IBg== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 11/25] debugobjects: Move pools into a datastructure References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:50:04 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The contention on the global pool lock can be reduced by strict batch processing where batches of objects are moved from one list head to another instead of moving them object by object. This also reduces the cache footprint because it avoids the list walk and dirties at maximum three cache lines instead of potentially up to eighteen. To prepare for that, move the hlist head and related counters into a struct. No functional change. Signed-off-by: Thomas Gleixner Reviewed-by: Zhen Lei --- lib/debugobjects.c | 140 +++++++++++++++++++++++++++++-------------------= ----- 1 file changed, 78 insertions(+), 62 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -52,6 +52,11 @@ struct debug_percpu_free { int obj_free; }; =20 +struct obj_pool { + struct hlist_head objects; + unsigned int cnt; +} ____cacheline_aligned; + static DEFINE_PER_CPU(struct debug_percpu_free, percpu_obj_pool); =20 static struct debug_bucket obj_hash[ODEBUG_HASH_SIZE]; @@ -60,8 +65,8 @@ static struct debug_obj obj_static_pool =20 static DEFINE_RAW_SPINLOCK(pool_lock); =20 -static HLIST_HEAD(obj_pool); -static HLIST_HEAD(obj_to_free); +static struct obj_pool pool_global; +static struct obj_pool pool_to_free; =20 /* * Because of the presence of percpu free pools, obj_pool_free will @@ -71,12 +76,9 @@ static HLIST_HEAD(obj_to_free); * can be off. */ static int __data_racy obj_pool_min_free =3D ODEBUG_POOL_SIZE; -static int __data_racy obj_pool_free =3D ODEBUG_POOL_SIZE; static int obj_pool_used; static int __data_racy obj_pool_max_used; static bool obj_freeing; -/* The number of objs on the global free list */ -static int obj_nr_tofree; =20 static int __data_racy debug_objects_maxchain __read_mostly; static int __data_racy __maybe_unused debug_objects_maxchecked __read_most= ly; @@ -124,6 +126,21 @@ static const char *obj_states[ODEBUG_STA [ODEBUG_STATE_NOTAVAILABLE] =3D "not available", }; =20 +static __always_inline unsigned int pool_count(struct obj_pool *pool) +{ + return READ_ONCE(pool->cnt); +} + +static inline bool pool_global_should_refill(void) +{ + return READ_ONCE(pool_global.cnt) < debug_objects_pool_min_level; +} + +static inline bool pool_global_must_refill(void) +{ + return READ_ONCE(pool_global.cnt) < (debug_objects_pool_min_level / 2); +} + static void free_object_list(struct hlist_head *head) { struct hlist_node *tmp; @@ -146,11 +163,8 @@ static void fill_pool_from_freelist(void /* * Reuse objs from the global obj_to_free list; they will be * reinitialized when allocating. - * - * obj_nr_tofree is checked locklessly; the READ_ONCE() pairs with - * the WRITE_ONCE() in pool_lock critical sections. */ - if (!READ_ONCE(obj_nr_tofree)) + if (!pool_count(&pool_to_free)) return; =20 /* @@ -171,12 +185,12 @@ static void fill_pool_from_freelist(void * Recheck with the lock held as the worker thread might have * won the race and freed the global free list already. */ - while (obj_nr_tofree && (obj_pool_free < debug_objects_pool_min_level)) { - obj =3D hlist_entry(obj_to_free.first, typeof(*obj), node); + while (pool_to_free.cnt && (pool_global.cnt < debug_objects_pool_min_leve= l)) { + obj =3D hlist_entry(pool_to_free.objects.first, typeof(*obj), node); hlist_del(&obj->node); - WRITE_ONCE(obj_nr_tofree, obj_nr_tofree - 1); - hlist_add_head(&obj->node, &obj_pool); - WRITE_ONCE(obj_pool_free, obj_pool_free + 1); + WRITE_ONCE(pool_to_free.cnt, pool_to_free.cnt - 1); + hlist_add_head(&obj->node, &pool_global.objects); + WRITE_ONCE(pool_global.cnt, pool_global.cnt + 1); } clear_bit(0, &state); } @@ -190,12 +204,11 @@ static void fill_pool(void) * - One other CPU is already allocating * - the global pool has not reached the critical level yet */ - if (READ_ONCE(obj_pool_free) > (debug_objects_pool_min_level / 2) && - atomic_read(&cpus_allocating)) + if (!pool_global_must_refill() && atomic_read(&cpus_allocating)) return; =20 atomic_inc(&cpus_allocating); - while (READ_ONCE(obj_pool_free) < debug_objects_pool_min_level) { + while (pool_global_should_refill()) { struct debug_obj *new, *last =3D NULL; HLIST_HEAD(head); int cnt; @@ -212,9 +225,9 @@ static void fill_pool(void) break; =20 guard(raw_spinlock_irqsave)(&pool_lock); - hlist_splice_init(&head, &last->node, &obj_pool); + hlist_splice_init(&head, &last->node, &pool_global.objects); debug_objects_allocated +=3D cnt; - WRITE_ONCE(obj_pool_free, obj_pool_free + cnt); + WRITE_ONCE(pool_global.cnt, pool_global.cnt + cnt); } atomic_dec(&cpus_allocating); } @@ -268,10 +281,10 @@ alloc_object(void *addr, struct debug_bu } =20 raw_spin_lock(&pool_lock); - obj =3D __alloc_object(&obj_pool); + obj =3D __alloc_object(&pool_global.objects); if (obj) { obj_pool_used++; - WRITE_ONCE(obj_pool_free, obj_pool_free - 1); + WRITE_ONCE(pool_global.cnt, pool_global.cnt - 1); =20 /* * Looking ahead, allocate one batch of debug objects and @@ -283,22 +296,21 @@ alloc_object(void *addr, struct debug_bu for (i =3D 0; i < ODEBUG_BATCH_SIZE; i++) { struct debug_obj *obj2; =20 - obj2 =3D __alloc_object(&obj_pool); + obj2 =3D __alloc_object(&pool_global.objects); if (!obj2) break; - hlist_add_head(&obj2->node, - &percpu_pool->free_objs); + hlist_add_head(&obj2->node, &percpu_pool->free_objs); percpu_pool->obj_free++; obj_pool_used++; - WRITE_ONCE(obj_pool_free, obj_pool_free - 1); + WRITE_ONCE(pool_global.cnt, pool_global.cnt - 1); } } =20 if (obj_pool_used > obj_pool_max_used) obj_pool_max_used =3D obj_pool_used; =20 - if (obj_pool_free < obj_pool_min_free) - obj_pool_min_free =3D obj_pool_free; + if (pool_global.cnt < obj_pool_min_free) + obj_pool_min_free =3D pool_global.cnt; } raw_spin_unlock(&pool_lock); =20 @@ -329,7 +341,7 @@ static void free_obj_work(struct work_st if (!raw_spin_trylock_irqsave(&pool_lock, flags)) return; =20 - if (obj_pool_free >=3D debug_objects_pool_size) + if (pool_global.cnt >=3D debug_objects_pool_size) goto free_objs; =20 /* @@ -339,12 +351,12 @@ static void free_obj_work(struct work_st * may be gearing up to use more and more objects, don't free any * of them until the next round. */ - while (obj_nr_tofree && obj_pool_free < debug_objects_pool_size) { - obj =3D hlist_entry(obj_to_free.first, typeof(*obj), node); + while (pool_to_free.cnt && pool_global.cnt < debug_objects_pool_size) { + obj =3D hlist_entry(pool_to_free.objects.first, typeof(*obj), node); hlist_del(&obj->node); - hlist_add_head(&obj->node, &obj_pool); - WRITE_ONCE(obj_pool_free, obj_pool_free + 1); - WRITE_ONCE(obj_nr_tofree, obj_nr_tofree - 1); + hlist_add_head(&obj->node, &pool_global.objects); + WRITE_ONCE(pool_to_free.cnt, pool_to_free.cnt - 1); + WRITE_ONCE(pool_global.cnt, pool_global.cnt + 1); } raw_spin_unlock_irqrestore(&pool_lock, flags); return; @@ -355,9 +367,9 @@ static void free_obj_work(struct work_st * list. Move remaining free objs to a temporary list to free the * memory outside the pool_lock held region. */ - if (obj_nr_tofree) { - hlist_move_list(&obj_to_free, &tofree); - WRITE_ONCE(obj_nr_tofree, 0); + if (pool_to_free.cnt) { + hlist_move_list(&pool_to_free.objects, &tofree); + WRITE_ONCE(pool_to_free.cnt, 0); } raw_spin_unlock_irqrestore(&pool_lock, flags); =20 @@ -400,45 +412,45 @@ static void __free_object(struct debug_o =20 free_to_obj_pool: raw_spin_lock(&pool_lock); - work =3D (obj_pool_free > debug_objects_pool_size) && obj_cache && - (obj_nr_tofree < ODEBUG_FREE_WORK_MAX); + work =3D (pool_global.cnt > debug_objects_pool_size) && obj_cache && + (pool_to_free.cnt < ODEBUG_FREE_WORK_MAX); obj_pool_used--; =20 if (work) { - WRITE_ONCE(obj_nr_tofree, obj_nr_tofree + 1); - hlist_add_head(&obj->node, &obj_to_free); + WRITE_ONCE(pool_to_free.cnt, pool_to_free.cnt + 1); + hlist_add_head(&obj->node, &pool_to_free.objects); if (lookahead_count) { - WRITE_ONCE(obj_nr_tofree, obj_nr_tofree + lookahead_count); + WRITE_ONCE(pool_to_free.cnt, pool_to_free.cnt + lookahead_count); obj_pool_used -=3D lookahead_count; while (lookahead_count) { hlist_add_head(&objs[--lookahead_count]->node, - &obj_to_free); + &pool_to_free.objects); } } =20 - if ((obj_pool_free > debug_objects_pool_size) && - (obj_nr_tofree < ODEBUG_FREE_WORK_MAX)) { + if ((pool_global.cnt > debug_objects_pool_size) && + (pool_to_free.cnt < ODEBUG_FREE_WORK_MAX)) { int i; =20 /* * Free one more batch of objects from obj_pool. */ for (i =3D 0; i < ODEBUG_BATCH_SIZE; i++) { - obj =3D __alloc_object(&obj_pool); - hlist_add_head(&obj->node, &obj_to_free); - WRITE_ONCE(obj_pool_free, obj_pool_free - 1); - WRITE_ONCE(obj_nr_tofree, obj_nr_tofree + 1); + obj =3D __alloc_object(&pool_global.objects); + hlist_add_head(&obj->node, &pool_to_free.objects); + WRITE_ONCE(pool_global.cnt, pool_global.cnt - 1); + WRITE_ONCE(pool_to_free.cnt, pool_to_free.cnt + 1); } } } else { - WRITE_ONCE(obj_pool_free, obj_pool_free + 1); - hlist_add_head(&obj->node, &obj_pool); + WRITE_ONCE(pool_global.cnt, pool_global.cnt + 1); + hlist_add_head(&obj->node, &pool_global.objects); if (lookahead_count) { - WRITE_ONCE(obj_pool_free, obj_pool_free + lookahead_count); + WRITE_ONCE(pool_global.cnt, pool_global.cnt + lookahead_count); obj_pool_used -=3D lookahead_count; while (lookahead_count) { hlist_add_head(&objs[--lookahead_count]->node, - &obj_pool); + &pool_global.objects); } } } @@ -453,7 +465,7 @@ static void __free_object(struct debug_o static void free_object(struct debug_obj *obj) { __free_object(obj); - if (!READ_ONCE(obj_freeing) && READ_ONCE(obj_nr_tofree)) { + if (!READ_ONCE(obj_freeing) && pool_count(&pool_to_free)) { WRITE_ONCE(obj_freeing, true); schedule_delayed_work(&debug_obj_work, ODEBUG_FREE_WORK_DELAY); } @@ -622,13 +634,13 @@ static void debug_objects_fill_pool(void if (unlikely(!obj_cache)) return; =20 - if (likely(READ_ONCE(obj_pool_free) >=3D debug_objects_pool_min_level)) + if (likely(!pool_global_should_refill())) return; =20 /* Try reusing objects from obj_to_free_list */ fill_pool_from_freelist(); =20 - if (likely(READ_ONCE(obj_pool_free) >=3D debug_objects_pool_min_level)) + if (likely(!pool_global_should_refill())) return; =20 /* @@ -1040,7 +1052,7 @@ static void __debug_check_no_obj_freed(c debug_objects_maxchecked =3D objs_checked; =20 /* Schedule work to actually kmem_cache_free() objects */ - if (!READ_ONCE(obj_freeing) && READ_ONCE(obj_nr_tofree)) { + if (!READ_ONCE(obj_freeing) && pool_count(&pool_to_free)) { WRITE_ONCE(obj_freeing, true); schedule_delayed_work(&debug_obj_work, ODEBUG_FREE_WORK_DELAY); } @@ -1066,12 +1078,12 @@ static int debug_stats_show(struct seq_f seq_printf(m, "max_checked :%d\n", debug_objects_maxchecked); seq_printf(m, "warnings :%d\n", debug_objects_warnings); seq_printf(m, "fixups :%d\n", debug_objects_fixups); - seq_printf(m, "pool_free :%d\n", READ_ONCE(obj_pool_free) + obj_percp= u_free); + seq_printf(m, "pool_free :%d\n", pool_count(&pool_global) + obj_percp= u_free); seq_printf(m, "pool_pcp_free :%d\n", obj_percpu_free); seq_printf(m, "pool_min_free :%d\n", obj_pool_min_free); seq_printf(m, "pool_used :%d\n", obj_pool_used - obj_percpu_free); seq_printf(m, "pool_max_used :%d\n", obj_pool_max_used); - seq_printf(m, "on_free_list :%d\n", READ_ONCE(obj_nr_tofree)); + seq_printf(m, "on_free_list :%d\n", pool_count(&pool_to_free)); seq_printf(m, "objs_allocated:%d\n", debug_objects_allocated); seq_printf(m, "objs_freed :%d\n", debug_objects_freed); return 0; @@ -1330,7 +1342,9 @@ void __init debug_objects_early_init(voi raw_spin_lock_init(&obj_hash[i].lock); =20 for (i =3D 0; i < ODEBUG_POOL_SIZE; i++) - hlist_add_head(&obj_static_pool[i].node, &obj_pool); + hlist_add_head(&obj_static_pool[i].node, &pool_global.objects); + + pool_global.cnt =3D ODEBUG_POOL_SIZE; } =20 /* @@ -1354,21 +1368,23 @@ static bool __init debug_objects_replace hlist_add_head(&obj->node, &objects); } =20 - debug_objects_allocated +=3D i; + debug_objects_allocated =3D ODEBUG_POOL_SIZE; + pool_global.cnt =3D ODEBUG_POOL_SIZE; =20 /* * Replace the statically allocated objects list with the allocated * objects list. */ - hlist_move_list(&objects, &obj_pool); + hlist_move_list(&objects, &pool_global.objects); =20 /* Replace the active object references */ for (i =3D 0; i < ODEBUG_HASH_SIZE; i++, db++) { hlist_move_list(&db->list, &objects); =20 hlist_for_each_entry(obj, &objects, node) { - new =3D hlist_entry(obj_pool.first, typeof(*obj), node); + new =3D hlist_entry(pool_global.objects.first, typeof(*obj), node); hlist_del(&new->node); + pool_global.cnt--; /* copy object data */ *new =3D *obj; hlist_add_head(&new->node, &db->list); From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C5F9D1DD9A3 for ; Mon, 7 Oct 2024 16:50:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319809; cv=none; b=GY1mPbDctM+H7VdCJpgZ8pI5HIWN10NlshAe2CwZVPFTIierUY/FWVnNUrZNLToLbYXjJq+Vy287Lep2E4nISz2c9InTAlDR53nXwr9M46HqRAhH3kwOTeJ94Z6RD99nwuvJcpb/IO8Jd99LQrq8hx90ZztXUPmWloG5H9sgaA0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319809; c=relaxed/simple; bh=PjgQRkzOq7ZOYamgeX2qnTV4RUfzvLmT7KF6nmn07+0=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=rbjwDOksATPfDHfXooaTI2XoW2838WkSrNdh/cInCsqbFgvlyHOak1rAnLIhRETs2JLuJ9wdz0gVrMPOwBJtbpwAfILSasqWvEN9Xbv/+tjdk/7HgJ4lgkYoEgH5vcJk2AbGbIhxEZL3xTwbmyD9Lij5jteerOBHtImFdyiJk3o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=46G04xuk; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=OYrjoKh3; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="46G04xuk"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="OYrjoKh3" Message-ID: <20241007164913.708939081@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319806; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=03yKkOdmGQrxY+R+fCwNxN1bQA9mKL+hWQ/PugmBeKQ=; b=46G04xuk+StWd4yiNgwOLPlonc8M+rcnKyZ1zp/WTAYJ9Y1XhytJgeSvTXrPLSGfZicY0b QkSEfhwofhhp3ypLP2T809fk1yzmj15/rbcAhmApnffPblWSUwEJXi9rCqv+4VCnO4ILTN fEpabdj/E6Y0OrCLwn6DGLahaliDVhijf0LAMkq4Fuit9XZXQk8HxYC4enJTxh3E5PV0Uv df/f9odu8QQgYEQbO7UrcFfYVrqSozwKRFTeoEeKtb/PGV8aaL47SBddI7CiLmanc8by02 wN44Rj/Bf5+UXkL/cTgFpsSYPXDkugtBt4PA4UHDBR45TQx0bz780X3rSVmcaw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319806; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=03yKkOdmGQrxY+R+fCwNxN1bQA9mKL+hWQ/PugmBeKQ=; b=OYrjoKh3Em/nXn/oyBg9hNa53Bcv4IU4oDY8+X50eIVfUKOzNZGGEbWNwQ7j13U/fZTzfC wyrYXGgGSpjk3yBQ== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 12/25] debugobjects: Use separate list head for boot pool References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:50:05 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" There is no point to handle the statically allocated objects during early boot in the actual pool list. This phase does not require accounting, so all of the related complexity can be avoided. Signed-off-by: Thomas Gleixner Reviewed-by: Zhen Lei --- lib/debugobjects.c | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -68,6 +68,8 @@ static DEFINE_RAW_SPINLOCK(pool_lock); static struct obj_pool pool_global; static struct obj_pool pool_to_free; =20 +static HLIST_HEAD(pool_boot); + /* * Because of the presence of percpu free pools, obj_pool_free will * under-count those in the percpu free pools. Similarly, obj_pool_used @@ -278,6 +280,9 @@ alloc_object(void *addr, struct debug_bu percpu_pool->obj_free--; goto init_obj; } + } else { + obj =3D __alloc_object(&pool_boot); + goto init_obj; } =20 raw_spin_lock(&pool_lock); @@ -381,12 +386,14 @@ static void __free_object(struct debug_o struct debug_obj *objs[ODEBUG_BATCH_SIZE]; struct debug_percpu_free *percpu_pool; int lookahead_count =3D 0; - unsigned long flags; bool work; =20 - local_irq_save(flags); - if (!obj_cache) - goto free_to_obj_pool; + guard(irqsave)(); + + if (unlikely(!obj_cache)) { + hlist_add_head(&obj->node, &pool_boot); + return; + } =20 /* * Try to free it into the percpu pool first. @@ -395,7 +402,6 @@ static void __free_object(struct debug_o if (percpu_pool->obj_free < ODEBUG_POOL_PERCPU_SIZE) { hlist_add_head(&obj->node, &percpu_pool->free_objs); percpu_pool->obj_free++; - local_irq_restore(flags); return; } =20 @@ -410,7 +416,6 @@ static void __free_object(struct debug_o percpu_pool->obj_free--; } =20 -free_to_obj_pool: raw_spin_lock(&pool_lock); work =3D (pool_global.cnt > debug_objects_pool_size) && obj_cache && (pool_to_free.cnt < ODEBUG_FREE_WORK_MAX); @@ -455,7 +460,6 @@ static void __free_object(struct debug_o } } raw_spin_unlock(&pool_lock); - local_irq_restore(flags); } =20 /* @@ -1341,10 +1345,9 @@ void __init debug_objects_early_init(voi for (i =3D 0; i < ODEBUG_HASH_SIZE; i++) raw_spin_lock_init(&obj_hash[i].lock); =20 + /* Keep early boot simple and add everything to the boot list */ for (i =3D 0; i < ODEBUG_POOL_SIZE; i++) - hlist_add_head(&obj_static_pool[i].node, &pool_global.objects); - - pool_global.cnt =3D ODEBUG_POOL_SIZE; + hlist_add_head(&obj_static_pool[i].node, &pool_boot); } =20 /* @@ -1372,10 +1375,11 @@ static bool __init debug_objects_replace pool_global.cnt =3D ODEBUG_POOL_SIZE; =20 /* - * Replace the statically allocated objects list with the allocated - * objects list. + * Move the allocated objects to the global pool and disconnect the + * boot pool. */ hlist_move_list(&objects, &pool_global.objects); + pool_boot.first =3D NULL; =20 /* Replace the active object references */ for (i =3D 0; i < ODEBUG_HASH_SIZE; i++, db++) { From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B56A71DD893 for ; Mon, 7 Oct 2024 16:50:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319810; cv=none; b=hgvCEpdCHWFlgj9yg+4YLLGWxkyKqzJioIOujiob7SCiJ43br/dvmxBmoBGpnyZhxbtYIKsNBP3bmfvS3vUw54gqgSeG/tpMCTdIQE3wzr9t5m274Gj3Op9uefwvueawIixUywyjbcBny8/J2LeroRmsW+0RwlJzkThPhcfRi0I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319810; c=relaxed/simple; bh=p28R/DTSeAkAp/F1zxRG0aYLBOZX/REDwdHwMFyI7zc=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=PEUN6Q7j/qc2FZ3D1O9eTyju02F65d91TQ9i8+vg/YK6uagrm8XuD/qAMKYS+WDTllStG0j5Liw6O21rQb9k+jIsL+sO1IMai8lgWEoC4Z90WcFQPmr0XHP40KQjix2SM2+fSqS5D86MGfE3dkJO1wAVzLp9SworxifKQURsmSk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=1wlWvI3H; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=JkExd2fU; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="1wlWvI3H"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="JkExd2fU" Message-ID: <20241007164913.770595795@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319807; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=XsISTO1FmLtHeTAitVxsUtQMSisv5BnD16nh7PYhYH8=; b=1wlWvI3Ha1Vf1Fu/x+amzIDjzbUqZY+FCwJEqBpjv9kUfDb4gf8WXHBNmqO5VZrdXhTrMb HKJ3yC94uheWLwTAny2U7x89294pOsh4szSL7BWtIuj9vqban5JQPoxl9w64EhPwdLiU6K Aq3OVB5RbzKFqprV4LreYkN8K338InH0p3QL6Gg16zBEA6nZXWPlwkyXKlD/kqn9nPAkgT JFUIui6+JHH8c7fSLHxT1c4CIL8iOnY9jh7VRW9Lq/+FaDXVdZKYjt5rd8q7wVKLk6cxXG Ab4ttHRh3tms43kiSNKK8UJKvqt1irVH/sdq70FPBctVEfHayKedDPIsCruccg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319807; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=XsISTO1FmLtHeTAitVxsUtQMSisv5BnD16nh7PYhYH8=; b=JkExd2fUtkkn0p3cJ7UlbIi5pA1x4YIl6M9ut0bo5Ipvn5pDCtWjIwQhWStVWLlyffJbzZ n2GR4LSYPLjAwiDw== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 13/25] debugobjects: Rename and tidy up per CPU pools References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:50:07 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No point in having a separate data structure. Reuse struct obj_pool and tidy up the code. Signed-off-by: Thomas Gleixner Reviewed-by: Zhen Lei --- lib/debugobjects.c | 43 +++++++++++++++++-------------------------- 1 file changed, 17 insertions(+), 26 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -43,21 +43,12 @@ struct debug_bucket { raw_spinlock_t lock; }; =20 -/* - * Debug object percpu free list - * Access is protected by disabling irq - */ -struct debug_percpu_free { - struct hlist_head free_objs; - int obj_free; -}; - struct obj_pool { struct hlist_head objects; unsigned int cnt; } ____cacheline_aligned; =20 -static DEFINE_PER_CPU(struct debug_percpu_free, percpu_obj_pool); +static DEFINE_PER_CPU(struct obj_pool, pool_pcpu); =20 static struct debug_bucket obj_hash[ODEBUG_HASH_SIZE]; =20 @@ -271,13 +262,13 @@ static struct debug_obj *__alloc_object( static struct debug_obj * alloc_object(void *addr, struct debug_bucket *b, const struct debug_obj_de= scr *descr) { - struct debug_percpu_free *percpu_pool =3D this_cpu_ptr(&percpu_obj_pool); + struct obj_pool *percpu_pool =3D this_cpu_ptr(&pool_pcpu); struct debug_obj *obj; =20 if (likely(obj_cache)) { - obj =3D __alloc_object(&percpu_pool->free_objs); + obj =3D __alloc_object(&percpu_pool->objects); if (obj) { - percpu_pool->obj_free--; + percpu_pool->cnt--; goto init_obj; } } else { @@ -304,8 +295,8 @@ alloc_object(void *addr, struct debug_bu obj2 =3D __alloc_object(&pool_global.objects); if (!obj2) break; - hlist_add_head(&obj2->node, &percpu_pool->free_objs); - percpu_pool->obj_free++; + hlist_add_head(&obj2->node, &percpu_pool->objects); + percpu_pool->cnt++; obj_pool_used++; WRITE_ONCE(pool_global.cnt, pool_global.cnt - 1); } @@ -384,7 +375,7 @@ static void free_obj_work(struct work_st static void __free_object(struct debug_obj *obj) { struct debug_obj *objs[ODEBUG_BATCH_SIZE]; - struct debug_percpu_free *percpu_pool; + struct obj_pool *percpu_pool; int lookahead_count =3D 0; bool work; =20 @@ -398,10 +389,10 @@ static void __free_object(struct debug_o /* * Try to free it into the percpu pool first. */ - percpu_pool =3D this_cpu_ptr(&percpu_obj_pool); - if (percpu_pool->obj_free < ODEBUG_POOL_PERCPU_SIZE) { - hlist_add_head(&obj->node, &percpu_pool->free_objs); - percpu_pool->obj_free++; + percpu_pool =3D this_cpu_ptr(&pool_pcpu); + if (percpu_pool->cnt < ODEBUG_POOL_PERCPU_SIZE) { + hlist_add_head(&obj->node, &percpu_pool->objects); + percpu_pool->cnt++; return; } =20 @@ -410,10 +401,10 @@ static void __free_object(struct debug_o * of objects from the percpu pool and free them as well. */ for (; lookahead_count < ODEBUG_BATCH_SIZE; lookahead_count++) { - objs[lookahead_count] =3D __alloc_object(&percpu_pool->free_objs); + objs[lookahead_count] =3D __alloc_object(&percpu_pool->objects); if (!objs[lookahead_count]) break; - percpu_pool->obj_free--; + percpu_pool->cnt--; } =20 raw_spin_lock(&pool_lock); @@ -494,10 +485,10 @@ static void put_objects(struct hlist_hea static int object_cpu_offline(unsigned int cpu) { /* Remote access is safe as the CPU is dead already */ - struct debug_percpu_free *pcp =3D per_cpu_ptr(&percpu_obj_pool, cpu); + struct obj_pool *pcp =3D per_cpu_ptr(&pool_pcpu, cpu); =20 - put_objects(&pcp->free_objs); - pcp->obj_free =3D 0; + put_objects(&pcp->objects); + pcp->cnt =3D 0; return 0; } #endif @@ -1076,7 +1067,7 @@ static int debug_stats_show(struct seq_f int cpu, obj_percpu_free =3D 0; =20 for_each_possible_cpu(cpu) - obj_percpu_free +=3D per_cpu(percpu_obj_pool.obj_free, cpu); + obj_percpu_free +=3D per_cpu(pool_pcpu.cnt, cpu); =20 seq_printf(m, "max_chain :%d\n", debug_objects_maxchain); seq_printf(m, "max_checked :%d\n", debug_objects_maxchecked); From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 17E881DC193 for ; Mon, 7 Oct 2024 16:50:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319811; cv=none; b=m0Z6n6EUFxyGxd5xtPmijEhMCUMox0UutibVe9W8u0nbQxapDlOS4ODYcXLDpjMJ0aBgPEdgnaM7Q2qg2YDH/MHwZW8AmpbrQJk1QafPs8yGhzWjxpzYVkQQ0VkgdfUFwj3ZCocW1bIBdmNUAkJgK1m+vNKWFeEXuTE1t57UIZI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319811; c=relaxed/simple; bh=MJTHpZF1pjqou6oxpAbuBTFXweqN1Swg6SPfZvdrumk=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=YMUA21C6692dPscioQVJw+kQOeL61OJwa9Og5CvB0voeii9Q0z2i7QUfy7J06Qg5MLn4dtQoqlkCJRdFo+lkJK3+0CIM7zNpvchLumLB0UxJlOoDvQLLYhLcZiUxCqAqTH6EkNMNZdb0+8RkY/dwMfqBXfQ+QyIpUhivj7gw3NQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=YoVCqQIz; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=J/bis10t; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="YoVCqQIz"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="J/bis10t" Message-ID: <20241007164913.831908427@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319808; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=EcwY8nQO7dHZBlYc/nlcYihBCfrOTqL4tvXELl0Nu6I=; b=YoVCqQIz+PNiFE7Ntbj9kJe39LK+F83CCUtp3rvIQotIGibv6NQVIxEhz4B/135c6YCj6h x9UCI3wOiAy/NNdQhHz5GwTPjePwRIboRXFOISyAmSG0iSiwDo6XNRHws+j1R84N5AmKJ+ we9Cz7LwYn1s/tNRykzCpfdmgPYhBq1OSdRuyq78ZFGMaEeAdyENVrj0aCBmBpoFioyS6J 38bdml9KYKOJJuZmWh8uVT/Q0O6IxJqZcKyJNlQvYsCLdh+J6etSgjeM6YwLvoRquE1B1i T9IqmtViwvfCV0rjVmLva2Y1GG4OpdeKbO0RfV1fzrhoHPMj+8L4JQpzN6QJGg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319808; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=EcwY8nQO7dHZBlYc/nlcYihBCfrOTqL4tvXELl0Nu6I=; b=J/bis10tbnJF5m3xErQsidvk4g5HSoJbC1X5h0bf8/wl67/MTukn0IHCGVWVyuyq+yyZlJ eHsuiJbX/Q67lNAA== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 14/25] debugobjects: Move min/max count into pool struct References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:50:08 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Having the accounting in the datastructure is better in terms of cache lines and allows more optimizations later on. Signed-off-by: Thomas Gleixner Reviewed-by: Zhen Lei --- lib/debugobjects.c | 55 +++++++++++++++++++++++++++++-------------------= ----- 1 file changed, 31 insertions(+), 24 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -46,9 +46,14 @@ struct debug_bucket { struct obj_pool { struct hlist_head objects; unsigned int cnt; + unsigned int min_cnt; + unsigned int max_cnt; } ____cacheline_aligned; =20 -static DEFINE_PER_CPU(struct obj_pool, pool_pcpu); + +static DEFINE_PER_CPU_ALIGNED(struct obj_pool, pool_pcpu) =3D { + .max_cnt =3D ODEBUG_POOL_PERCPU_SIZE, +}; =20 static struct debug_bucket obj_hash[ODEBUG_HASH_SIZE]; =20 @@ -56,8 +61,14 @@ static struct debug_obj obj_static_pool =20 static DEFINE_RAW_SPINLOCK(pool_lock); =20 -static struct obj_pool pool_global; -static struct obj_pool pool_to_free; +static struct obj_pool pool_global =3D { + .min_cnt =3D ODEBUG_POOL_MIN_LEVEL, + .max_cnt =3D ODEBUG_POOL_SIZE, +}; + +static struct obj_pool pool_to_free =3D { + .max_cnt =3D UINT_MAX, +}; =20 static HLIST_HEAD(pool_boot); =20 @@ -79,13 +90,9 @@ static int __data_racy debug_objects_f static int __data_racy debug_objects_warnings __read_mostly; static bool __data_racy debug_objects_enabled __read_mostly =3D CONFIG_DEBUG_OBJECTS_ENABLE_DEFAULT; -static int debug_objects_pool_size __ro_after_init - =3D ODEBUG_POOL_SIZE; -static int debug_objects_pool_min_level __ro_after_init - =3D ODEBUG_POOL_MIN_LEVEL; =20 -static const struct debug_obj_descr *descr_test __read_mostly; -static struct kmem_cache *obj_cache __ro_after_init; +static const struct debug_obj_descr *descr_test __read_mostly; +static struct kmem_cache *obj_cache __ro_after_init; =20 /* * Track numbers of kmem_cache_alloc()/free() calls done. @@ -124,14 +131,14 @@ static __always_inline unsigned int pool return READ_ONCE(pool->cnt); } =20 -static inline bool pool_global_should_refill(void) +static __always_inline bool pool_should_refill(struct obj_pool *pool) { - return READ_ONCE(pool_global.cnt) < debug_objects_pool_min_level; + return pool_count(pool) < pool->min_cnt; } =20 -static inline bool pool_global_must_refill(void) +static __always_inline bool pool_must_refill(struct obj_pool *pool) { - return READ_ONCE(pool_global.cnt) < (debug_objects_pool_min_level / 2); + return pool_count(pool) < pool->min_cnt / 2; } =20 static void free_object_list(struct hlist_head *head) @@ -178,7 +185,7 @@ static void fill_pool_from_freelist(void * Recheck with the lock held as the worker thread might have * won the race and freed the global free list already. */ - while (pool_to_free.cnt && (pool_global.cnt < debug_objects_pool_min_leve= l)) { + while (pool_to_free.cnt && (pool_global.cnt < pool_global.min_cnt)) { obj =3D hlist_entry(pool_to_free.objects.first, typeof(*obj), node); hlist_del(&obj->node); WRITE_ONCE(pool_to_free.cnt, pool_to_free.cnt - 1); @@ -197,11 +204,11 @@ static void fill_pool(void) * - One other CPU is already allocating * - the global pool has not reached the critical level yet */ - if (!pool_global_must_refill() && atomic_read(&cpus_allocating)) + if (!pool_must_refill(&pool_global) && atomic_read(&cpus_allocating)) return; =20 atomic_inc(&cpus_allocating); - while (pool_global_should_refill()) { + while (pool_should_refill(&pool_global)) { struct debug_obj *new, *last =3D NULL; HLIST_HEAD(head); int cnt; @@ -337,7 +344,7 @@ static void free_obj_work(struct work_st if (!raw_spin_trylock_irqsave(&pool_lock, flags)) return; =20 - if (pool_global.cnt >=3D debug_objects_pool_size) + if (pool_global.cnt >=3D pool_global.max_cnt) goto free_objs; =20 /* @@ -347,7 +354,7 @@ static void free_obj_work(struct work_st * may be gearing up to use more and more objects, don't free any * of them until the next round. */ - while (pool_to_free.cnt && pool_global.cnt < debug_objects_pool_size) { + while (pool_to_free.cnt && pool_global.cnt < pool_global.max_cnt) { obj =3D hlist_entry(pool_to_free.objects.first, typeof(*obj), node); hlist_del(&obj->node); hlist_add_head(&obj->node, &pool_global.objects); @@ -408,7 +415,7 @@ static void __free_object(struct debug_o } =20 raw_spin_lock(&pool_lock); - work =3D (pool_global.cnt > debug_objects_pool_size) && obj_cache && + work =3D (pool_global.cnt > pool_global.max_cnt) && obj_cache && (pool_to_free.cnt < ODEBUG_FREE_WORK_MAX); obj_pool_used--; =20 @@ -424,7 +431,7 @@ static void __free_object(struct debug_o } } =20 - if ((pool_global.cnt > debug_objects_pool_size) && + if ((pool_global.cnt > pool_global.max_cnt) && (pool_to_free.cnt < ODEBUG_FREE_WORK_MAX)) { int i; =20 @@ -629,13 +636,13 @@ static void debug_objects_fill_pool(void if (unlikely(!obj_cache)) return; =20 - if (likely(!pool_global_should_refill())) + if (likely(!pool_should_refill(&pool_global))) return; =20 /* Try reusing objects from obj_to_free_list */ fill_pool_from_freelist(); =20 - if (likely(!pool_global_should_refill())) + if (likely(!pool_should_refill(&pool_global))) return; =20 /* @@ -1427,8 +1434,8 @@ void __init debug_objects_mem_init(void) * system. */ extras =3D num_possible_cpus() * ODEBUG_BATCH_SIZE; - debug_objects_pool_size +=3D extras; - debug_objects_pool_min_level +=3D extras; + pool_global.max_cnt +=3D extras; + pool_global.min_cnt +=3D extras; =20 /* Everything worked. Expose the cache */ obj_cache =3D cache; From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BE34D1DDA20 for ; Mon, 7 Oct 2024 16:50:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319813; cv=none; b=cGPtBk6IaZfqihvrCp45e/tf5z0RvpFEuA7u1sYg2MTPOgdEV9EID3GcnvfT0nSUtnPyu/BIi+hnDKhywiUaJQdaN22D1ZugnLdaJHZouc4e41D0P72G6MaV99v7RRFmOddxkc2X361snAUKAYnNPYlw9oG54URdydEV5/IeFjg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319813; c=relaxed/simple; bh=TG4W13Y2+J182jBOPq36tFXGWnSxPrWxhQ8KT35JGzQ=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=rWbs9ZY04kD4/jirYw9sUaIlk77Y4mHQdSSgGHY4mz1H+lv074OcuEvqsO0Fa4+BhhdtS18ZYkAO0IFDYZIxKQIYk4EuigmZz/1KWe2KmmVzbfTYwwEOQDA9dEFm5oU6GrF0QQyOiIQjncjyb0fy34vpJFdurl0kstv46mYqH0k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=YiqDAhfL; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=l2NT/9Qy; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="YiqDAhfL"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="l2NT/9Qy" Message-ID: <20241007164913.893554162@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319809; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=p/HKRAt0Ud86tNjgylPgGhmPpraXRWuyB3Oxip/mEds=; b=YiqDAhfL93Ned49EPNDl2MGOnFacZkA4r1GMeZffvUuhK/8YBNmg8zn8iCmZ/xXRbn5day gTNjATUrRoSVfDdQYgx+fr83Fip8m5ebwnc8KSFudljrr0yKpEYUn1z1T0dEAg/jj53br3 30jY1fTZRiynRuWdnP+7un/ToIbfvp4PFTcL4KXWel5gloLyljUZMVNCCrBjZHxOIQEmbI QoyWzzZFsONpaJGC2GfKZu3h6QtU2aLmVRURAYG3nU5eTHA+bZkd1x3u6tolTtfHuS/R29 7XQnzSU6ZKORWfGWvSJ73yIoXjHNZ367eMtS+opczQLiJOautWX9FX7ODxJLNg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319809; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=p/HKRAt0Ud86tNjgylPgGhmPpraXRWuyB3Oxip/mEds=; b=l2NT/9Qy4Y2fvCux6kBCYZt1oyY4ft6DjbPp0+CallgPiHcd3H4mn88FDclepyXMLCvP1i VVNClPKRe0GnixBg== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 15/25] debugobjects: Rework object allocation References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:50:09 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The current allocation scheme tries to allocate from the per CPU pool first. If that fails it allocates one object from the global pool and then refills the per CPU pool from the global pool. That is in the way of switching the pool management to batch mode as the global pool needs to be a strict stack of batches, which does not allow to allocate single objects. Rework the code to refill the per CPU pool first and then allocate the object from the refilled batch. Also try to allocate from the to free pool first to avoid freeing and reallocating objects. Signed-off-by: Thomas Gleixner Reviewed-by: Zhen Lei --- lib/debugobjects.c | 144 +++++++++++++++++++++++++-----------------------= ----- 1 file changed, 69 insertions(+), 75 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -141,6 +141,64 @@ static __always_inline bool pool_must_re return pool_count(pool) < pool->min_cnt / 2; } =20 +static bool pool_move_batch(struct obj_pool *dst, struct obj_pool *src) +{ + if (dst->cnt + ODEBUG_BATCH_SIZE > dst->max_cnt || !src->cnt) + return false; + + for (int i =3D 0; i < ODEBUG_BATCH_SIZE && src->cnt; i++) { + struct hlist_node *node =3D src->objects.first; + + WRITE_ONCE(src->cnt, src->cnt - 1); + WRITE_ONCE(dst->cnt, dst->cnt + 1); + + hlist_del(node); + hlist_add_head(node, &dst->objects); + } + return true; +} + +static struct debug_obj *__alloc_object(struct hlist_head *list) +{ + struct debug_obj *obj; + + if (unlikely(!list->first)) + return NULL; + + obj =3D hlist_entry(list->first, typeof(*obj), node); + hlist_del(&obj->node); + return obj; +} + +static struct debug_obj *pcpu_alloc(void) +{ + struct obj_pool *pcp =3D this_cpu_ptr(&pool_pcpu); + + lockdep_assert_irqs_disabled(); + + for (;;) { + struct debug_obj *obj =3D __alloc_object(&pcp->objects); + + if (likely(obj)) { + pcp->cnt--; + return obj; + } + + guard(raw_spinlock)(&pool_lock); + if (!pool_move_batch(pcp, &pool_to_free)) { + if (!pool_move_batch(pcp, &pool_global)) + return NULL; + } + obj_pool_used +=3D pcp->cnt; + + if (obj_pool_used > obj_pool_max_used) + obj_pool_max_used =3D obj_pool_used; + + if (pool_global.cnt < obj_pool_min_free) + obj_pool_min_free =3D pool_global.cnt; + } +} + static void free_object_list(struct hlist_head *head) { struct hlist_node *tmp; @@ -158,7 +216,6 @@ static void free_object_list(struct hlis static void fill_pool_from_freelist(void) { static unsigned long state; - struct debug_obj *obj; =20 /* * Reuse objs from the global obj_to_free list; they will be @@ -180,17 +237,11 @@ static void fill_pool_from_freelist(void if (test_bit(0, &state) || test_and_set_bit(0, &state)) return; =20 - guard(raw_spinlock)(&pool_lock); - /* - * Recheck with the lock held as the worker thread might have - * won the race and freed the global free list already. - */ - while (pool_to_free.cnt && (pool_global.cnt < pool_global.min_cnt)) { - obj =3D hlist_entry(pool_to_free.objects.first, typeof(*obj), node); - hlist_del(&obj->node); - WRITE_ONCE(pool_to_free.cnt, pool_to_free.cnt - 1); - hlist_add_head(&obj->node, &pool_global.objects); - WRITE_ONCE(pool_global.cnt, pool_global.cnt + 1); + /* Avoid taking the lock when there is no work to do */ + while (pool_should_refill(&pool_global) && pool_count(&pool_to_free)) { + guard(raw_spinlock)(&pool_lock); + /* Move a batch if possible */ + pool_move_batch(&pool_global, &pool_to_free); } clear_bit(0, &state); } @@ -251,74 +302,17 @@ static struct debug_obj *lookup_object(v return NULL; } =20 -/* - * Allocate a new object from the hlist - */ -static struct debug_obj *__alloc_object(struct hlist_head *list) +static struct debug_obj *alloc_object(void *addr, struct debug_bucket *b, + const struct debug_obj_descr *descr) { - struct debug_obj *obj =3D NULL; - - if (list->first) { - obj =3D hlist_entry(list->first, typeof(*obj), node); - hlist_del(&obj->node); - } - - return obj; -} - -static struct debug_obj * -alloc_object(void *addr, struct debug_bucket *b, const struct debug_obj_de= scr *descr) -{ - struct obj_pool *percpu_pool =3D this_cpu_ptr(&pool_pcpu); struct debug_obj *obj; =20 - if (likely(obj_cache)) { - obj =3D __alloc_object(&percpu_pool->objects); - if (obj) { - percpu_pool->cnt--; - goto init_obj; - } - } else { + if (likely(obj_cache)) + obj =3D pcpu_alloc(); + else obj =3D __alloc_object(&pool_boot); - goto init_obj; - } - - raw_spin_lock(&pool_lock); - obj =3D __alloc_object(&pool_global.objects); - if (obj) { - obj_pool_used++; - WRITE_ONCE(pool_global.cnt, pool_global.cnt - 1); - - /* - * Looking ahead, allocate one batch of debug objects and - * put them into the percpu free pool. - */ - if (likely(obj_cache)) { - int i; - - for (i =3D 0; i < ODEBUG_BATCH_SIZE; i++) { - struct debug_obj *obj2; - - obj2 =3D __alloc_object(&pool_global.objects); - if (!obj2) - break; - hlist_add_head(&obj2->node, &percpu_pool->objects); - percpu_pool->cnt++; - obj_pool_used++; - WRITE_ONCE(pool_global.cnt, pool_global.cnt - 1); - } - } - - if (obj_pool_used > obj_pool_max_used) - obj_pool_max_used =3D obj_pool_used; - - if (pool_global.cnt < obj_pool_min_free) - obj_pool_min_free =3D pool_global.cnt; - } - raw_spin_unlock(&pool_lock); =20 -init_obj: - if (obj) { + if (likely(obj)) { obj->object =3D addr; obj->descr =3D descr; obj->state =3D ODEBUG_STATE_NONE; From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 55FF71DDA34 for ; Mon, 7 Oct 2024 16:50:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319813; cv=none; b=VsQHM6QA/YF9MbKoIn/LGoaaOJoybP3839EknHfNvXYIbCIlsvM8h3YCZj6fVt306nxndHtOL/vlWOJMdWM7yP8shGRzPBGN5t7Pk3O3Yfw7ZNaezuSSHFpLNBd4Klrow4xAXSlREq4MfHBLu5NlAnhSdetckwTQ+7WPMR1SGD8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319813; c=relaxed/simple; bh=R8YmF0GjRlr0cM8252feIJU66w97dlkN40Ue5mmiqu8=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=kztzrEMFuv1N0yFKPhcVq9pFRTCreTW7ixcXrJmMAvZgo2SL82AmjvrIvrlATLD65FZ4UNz6ETSvGq9RSWoxx2Eu3AhyfegkSkzbFWri1Pq1qiR4iEqUjHB8ERrH76EqzaPfaqE3ysSZf2+EmYHTDTsNmxr2LT42vuNHYHNNfks= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=K1cT0xsA; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=sXuiqRvP; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="K1cT0xsA"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="sXuiqRvP" Message-ID: <20241007164913.955542307@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319811; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=8NTqjYqyTRgkGxIw/yZ5nqyqj1AMPCRe3Sjf3y1Rql0=; b=K1cT0xsA8fjsDmISnqkVGv7mwVJ7nBxXqh/XM1RI9rB+Exjm7X+0S3NFNGRBBiMbFR8gHD zwvA693CHZzJbtmNRW2gUDtDCOj5fTJarHxZvrncSxWHtF/UqQ0po98qz/Od8pwLApo7w6 9llGzle09wDUhDy1ZSqMoUyrBqM64O0Xm/vweqbZi+rgoa6dBUTkRLMNzS6TUYGezCMRAE cSQxzWsbIrWnE6xRbGw5wrBXmhm3461sMd3vLPCp8C+i2DoRR/+MoKKffmxhn33oOt8ME0 xtr/IQ8i9uJzqmlEQ/lbA7PnjkJorN+kUehi1pU2iDHvwt6rImhmMhVpr824fw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319811; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=8NTqjYqyTRgkGxIw/yZ5nqyqj1AMPCRe3Sjf3y1Rql0=; b=sXuiqRvPoug9UDEQq4kNiHavrlkzMhPe+NLOcYRBOmtuLOIfyBZBNOsM/X6EuO8NoJlIr1 fx3cRWZTnfPyBwDA== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 16/25] debugobjects: Rework object freeing References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:50:10 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" __free_object() is uncomprehensibly complex. The same can be achieved by: 1) Adding the object to the per CPU pool 2) If that pool is full, move a batch of objects into the global pool or if the global pool is full into the to free pool This also prepares for batch processing. Signed-off-by: Thomas Gleixner Reviewed-by: Zhen Lei --- lib/debugobjects.c | 99 ++++++++++++------------------------------------= ----- 1 file changed, 24 insertions(+), 75 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -199,6 +199,27 @@ static struct debug_obj *pcpu_alloc(void } } =20 +static void pcpu_free(struct debug_obj *obj) +{ + struct obj_pool *pcp =3D this_cpu_ptr(&pool_pcpu); + + lockdep_assert_irqs_disabled(); + + hlist_add_head(&obj->node, &pcp->objects); + pcp->cnt++; + + /* Pool full ? */ + if (pcp->cnt < ODEBUG_POOL_PERCPU_SIZE) + return; + + /* Remove a batch from the per CPU pool */ + guard(raw_spinlock)(&pool_lock); + /* Try to fit the batch into the pool_global first */ + if (!pool_move_batch(&pool_global, pcp)) + pool_move_batch(&pool_to_free, pcp); + obj_pool_used -=3D ODEBUG_BATCH_SIZE; +} + static void free_object_list(struct hlist_head *head) { struct hlist_node *tmp; @@ -375,83 +396,11 @@ static void free_obj_work(struct work_st =20 static void __free_object(struct debug_obj *obj) { - struct debug_obj *objs[ODEBUG_BATCH_SIZE]; - struct obj_pool *percpu_pool; - int lookahead_count =3D 0; - bool work; - guard(irqsave)(); - - if (unlikely(!obj_cache)) { + if (likely(obj_cache)) + pcpu_free(obj); + else hlist_add_head(&obj->node, &pool_boot); - return; - } - - /* - * Try to free it into the percpu pool first. - */ - percpu_pool =3D this_cpu_ptr(&pool_pcpu); - if (percpu_pool->cnt < ODEBUG_POOL_PERCPU_SIZE) { - hlist_add_head(&obj->node, &percpu_pool->objects); - percpu_pool->cnt++; - return; - } - - /* - * As the percpu pool is full, look ahead and pull out a batch - * of objects from the percpu pool and free them as well. - */ - for (; lookahead_count < ODEBUG_BATCH_SIZE; lookahead_count++) { - objs[lookahead_count] =3D __alloc_object(&percpu_pool->objects); - if (!objs[lookahead_count]) - break; - percpu_pool->cnt--; - } - - raw_spin_lock(&pool_lock); - work =3D (pool_global.cnt > pool_global.max_cnt) && obj_cache && - (pool_to_free.cnt < ODEBUG_FREE_WORK_MAX); - obj_pool_used--; - - if (work) { - WRITE_ONCE(pool_to_free.cnt, pool_to_free.cnt + 1); - hlist_add_head(&obj->node, &pool_to_free.objects); - if (lookahead_count) { - WRITE_ONCE(pool_to_free.cnt, pool_to_free.cnt + lookahead_count); - obj_pool_used -=3D lookahead_count; - while (lookahead_count) { - hlist_add_head(&objs[--lookahead_count]->node, - &pool_to_free.objects); - } - } - - if ((pool_global.cnt > pool_global.max_cnt) && - (pool_to_free.cnt < ODEBUG_FREE_WORK_MAX)) { - int i; - - /* - * Free one more batch of objects from obj_pool. - */ - for (i =3D 0; i < ODEBUG_BATCH_SIZE; i++) { - obj =3D __alloc_object(&pool_global.objects); - hlist_add_head(&obj->node, &pool_to_free.objects); - WRITE_ONCE(pool_global.cnt, pool_global.cnt - 1); - WRITE_ONCE(pool_to_free.cnt, pool_to_free.cnt + 1); - } - } - } else { - WRITE_ONCE(pool_global.cnt, pool_global.cnt + 1); - hlist_add_head(&obj->node, &pool_global.objects); - if (lookahead_count) { - WRITE_ONCE(pool_global.cnt, pool_global.cnt + lookahead_count); - obj_pool_used -=3D lookahead_count; - while (lookahead_count) { - hlist_add_head(&objs[--lookahead_count]->node, - &pool_global.objects); - } - } - } - raw_spin_unlock(&pool_lock); } =20 /* From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 02A881DDC21 for ; Mon, 7 Oct 2024 16:50:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319817; cv=none; b=So00KbARvAIGh7fGJDH3U5vLbr2M2W16co88lnENSrDstUg/jXg8lvqXdZ/Fcy8zVGxEjxHOcC1TEjVGiKFyuckTRFR6hX8x/SvKli1ERysEnWtdkIIr1hZFqn6SXE8LFd7znOFtKVkNIl4vmLB/aNTpeVVlV5eLdPcvQ/HG71g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319817; c=relaxed/simple; bh=yXUwgX8jkz1jw+eUIpPrYKnGu4sx71nK1S57F3E70jc=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=J+QiUd9ci3whOdbVgVz1DEQKBBQK+Sp75jiqa3ahty4Ra3HhM5PU/72LIaGoVEPslkAm3d/c4v5HcHCmGYEGkZp5QNPPb2wOf0R4HiNu+5gA/Lh2rp702MjCGyDyFBDPZ01PbiEReSTFTkmiZ/U9lsoVgaVMuAgxMrb3vgC25Sw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=ALqW3Ajs; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=hAZEPf+d; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="ALqW3Ajs"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="hAZEPf+d" Message-ID: <20241007164914.015906394@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319812; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=msOIqIj5y9aBSnK6QnRcLQeyJq+tPIAMJdXBn04Aajc=; b=ALqW3AjsumRSbCUtVML5FnoN8weMnnRKu06CFVdt69A/nxsuoopFAiNCi4zNgH8+2fs1gA 7dwoSy2SUyHbaKBBb6hZI7beFOXY4It9v9UJBo16hkblDBtSu/a6Zu2TlpOSYxSDz50Q9H kI4T+Yr3UgtFxjTYhAECYiT3aHyjC7B+XGPDgT3ukA4LUHDkaCdArndzI9EHBpI0MqRktW i5e/HxCOgLxuqiizJjw7h00qDjd3W5wKRm+KWG/3Prr+v4q+6BhwcEPJ5lQgCDE3cP3S74 +97qd0Bv/JxIMSb4H+iJVAhlprDkl2c62S0hTDLkLZ8owjpF7RmbsiHfLOogCA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319812; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=msOIqIj5y9aBSnK6QnRcLQeyJq+tPIAMJdXBn04Aajc=; b=hAZEPf+dONWHP4mM7ERTBuBcw6pdD1veaZU+eIGoranmLCs7YJVbk7Re6uNmVBU7+xRqRw 6xhRA5cDvNhAayCw== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 17/25] debugobjects: Rework free_object_work() References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:50:12 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Convert it to batch processing with intermediate helper functions. This reduces the final changes for batch processing. Signed-off-by: Thomas Gleixner Reviewed-by: Zhen Lei --- lib/debugobjects.c | 80 +++++++++++++++++++++++++-----------------------= ----- 1 file changed, 38 insertions(+), 42 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -35,7 +35,7 @@ * frequency of 10Hz and about 1024 objects for each freeing operation. * So it is freeing at most 10k debug objects per second. */ -#define ODEBUG_FREE_WORK_MAX 1024 +#define ODEBUG_FREE_WORK_MAX (1024 / ODEBUG_BATCH_SIZE) #define ODEBUG_FREE_WORK_DELAY DIV_ROUND_UP(HZ, 10) =20 struct debug_bucket { @@ -158,6 +158,21 @@ static bool pool_move_batch(struct obj_p return true; } =20 +static bool pool_pop_batch(struct hlist_head *head, struct obj_pool *src) +{ + if (!src->cnt) + return false; + + for (int i =3D 0; src->cnt && i < ODEBUG_BATCH_SIZE; i++) { + struct hlist_node *node =3D src->objects.first; + + WRITE_ONCE(src->cnt, src->cnt - 1); + hlist_del(node); + hlist_add_head(node, head); + } + return true; +} + static struct debug_obj *__alloc_object(struct hlist_head *list) { struct debug_obj *obj; @@ -343,55 +358,36 @@ static struct debug_obj *alloc_object(vo return obj; } =20 -/* - * workqueue function to free objects. - * - * To reduce contention on the global pool_lock, the actual freeing of - * debug objects will be delayed if the pool_lock is busy. - */ +/* workqueue function to free objects. */ static void free_obj_work(struct work_struct *work) { - struct debug_obj *obj; - unsigned long flags; - HLIST_HEAD(tofree); + bool free =3D true; =20 WRITE_ONCE(obj_freeing, false); - if (!raw_spin_trylock_irqsave(&pool_lock, flags)) + + if (!pool_count(&pool_to_free)) return; =20 - if (pool_global.cnt >=3D pool_global.max_cnt) - goto free_objs; + for (unsigned int cnt =3D 0; cnt < ODEBUG_FREE_WORK_MAX; cnt++) { + HLIST_HEAD(tofree); =20 - /* - * The objs on the pool list might be allocated before the work is - * run, so recheck if pool list it full or not, if not fill pool - * list from the global free list. As it is likely that a workload - * may be gearing up to use more and more objects, don't free any - * of them until the next round. - */ - while (pool_to_free.cnt && pool_global.cnt < pool_global.max_cnt) { - obj =3D hlist_entry(pool_to_free.objects.first, typeof(*obj), node); - hlist_del(&obj->node); - hlist_add_head(&obj->node, &pool_global.objects); - WRITE_ONCE(pool_to_free.cnt, pool_to_free.cnt - 1); - WRITE_ONCE(pool_global.cnt, pool_global.cnt + 1); - } - raw_spin_unlock_irqrestore(&pool_lock, flags); - return; - -free_objs: - /* - * Pool list is already full and there are still objs on the free - * list. Move remaining free objs to a temporary list to free the - * memory outside the pool_lock held region. - */ - if (pool_to_free.cnt) { - hlist_move_list(&pool_to_free.objects, &tofree); - WRITE_ONCE(pool_to_free.cnt, 0); + /* Acquire and drop the lock for each batch */ + scoped_guard(raw_spinlock_irqsave, &pool_lock) { + if (!pool_to_free.cnt) + return; + + /* Refill the global pool if possible */ + if (pool_move_batch(&pool_global, &pool_to_free)) { + /* Don't free as there seems to be demand */ + free =3D false; + } else if (free) { + pool_pop_batch(&tofree, &pool_to_free); + } else { + return; + } + } + free_object_list(&tofree); } - raw_spin_unlock_irqrestore(&pool_lock, flags); - - free_object_list(&tofree); } =20 static void __free_object(struct debug_obj *obj) From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 91B7C1DE2B9 for ; Mon, 7 Oct 2024 16:50:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319819; cv=none; b=qu5whUGllekAa0qYiBnA+8xS0k+boaS+HlX8rtS5EPzWftF77aZrFj3aUOMk/tClaO30GNtvKCsfqCQN8zIKdegx7UNqfjdgXMhhmEzxeznVNqrsKC3LvvToYaOYxA6b9hCRAnTyD/orsnnhgSRVGKd8q/DVZCeMs9NchU9D6M8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319819; c=relaxed/simple; bh=5+EAf50dSxAoBX7Xv/AahGjoTi0OfZ9NeYeRmvt3lrU=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=Z11IN3UGweX0Y/GWia8bCfgrMR/nniu6Ou9/zJVzKkJ9AuV6p3psaaLcSi9eQVZaaffh3/IguaRVd9UH7hXBI9+kwuer8moNjRMiJraNSUEKKRdWca6YcSJq/Ndqs5DftzBj+oh6ee8oFhOms3w+ph1ohg2eyss3IBcR4hNJO4g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=OlzWtgHv; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=EeoWapdo; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="OlzWtgHv"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="EeoWapdo" Message-ID: <20241007164914.077247071@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319813; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=w2tTfBGbQWuv1aJS1y+ptC+wplD7DmV2MTe3q95US6c=; b=OlzWtgHviMXIEkcZLgx12tl5ZzGvXziLpD3MIOVBv9tII6EY/DiS5cKnBhlnF9fQ0MYpV0 P8UuV8z/Yy8n9jB1kFsfupGMpcKRcYF3RlEGCgDTeIWxyKFFmDvAeEo2zD10jmxxSveR5B tvNSB9Xk5YDOrI0iKQPIqRHXLS6TwQtGAh9lel7GfdZ0ZDff5nEGZ94TaCjWXmo7IUxoEJ ae+8982nJr5lAAwt3q8s6D9n9DqFWF0a5Fiua5sMpzX9a7KObZj4EzB0lxfD/kuTs0QPs3 u/mCfegoJiF6swUg8YqkSVFjwpQs9FAfgr1x0L9PzFM4u7rs5dYbjL7Af0RArg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319813; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=w2tTfBGbQWuv1aJS1y+ptC+wplD7DmV2MTe3q95US6c=; b=EeoWapdoqK9BQdsL+SBaj5TnFKE+w3qBRsMwez2CrudQ7g3vpb3eUp98/WOHHS+T3BiT3O JEKjVDas3dxJ06Bw== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 18/25] debugobjects: Use static key for boot pool selection References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:50:13 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Get rid of the conditional in the hot path. Signed-off-by: Thomas Gleixner Reviewed-by: Zhen Lei --- lib/debugobjects.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -7,16 +7,16 @@ =20 #define pr_fmt(fmt) "ODEBUG: " fmt =20 +#include #include -#include +#include +#include +#include #include #include #include -#include #include -#include -#include -#include +#include =20 #define ODEBUG_HASH_BITS 14 #define ODEBUG_HASH_SIZE (1 << ODEBUG_HASH_BITS) @@ -103,6 +103,8 @@ static int __data_racy debug_objects_fr static void free_obj_work(struct work_struct *work); static DECLARE_DELAYED_WORK(debug_obj_work, free_obj_work); =20 +static DEFINE_STATIC_KEY_FALSE(obj_cache_enabled); + static int __init enable_object_debug(char *str) { debug_objects_enabled =3D true; @@ -343,7 +345,7 @@ static struct debug_obj *alloc_object(vo { struct debug_obj *obj; =20 - if (likely(obj_cache)) + if (static_branch_likely(&obj_cache_enabled)) obj =3D pcpu_alloc(); else obj =3D __alloc_object(&pool_boot); @@ -393,7 +395,7 @@ static void free_obj_work(struct work_st static void __free_object(struct debug_obj *obj) { guard(irqsave)(); - if (likely(obj_cache)) + if (static_branch_likely(&obj_cache_enabled)) pcpu_free(obj); else hlist_add_head(&obj->node, &pool_boot); @@ -572,7 +574,7 @@ static struct debug_obj *lookup_object_o =20 static void debug_objects_fill_pool(void) { - if (unlikely(!obj_cache)) + if (!static_branch_likely(&obj_cache_enabled)) return; =20 if (likely(!pool_should_refill(&pool_global))) @@ -1378,6 +1380,7 @@ void __init debug_objects_mem_init(void) =20 /* Everything worked. Expose the cache */ obj_cache =3D cache; + static_branch_enable(&obj_cache_enabled); =20 #ifdef CONFIG_HOTPLUG_CPU cpuhp_setup_state_nocalls(CPUHP_DEBUG_OBJ_DEAD, "object:offline", NULL, From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B0C41DE2CA for ; Mon, 7 Oct 2024 16:50:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319817; cv=none; b=fxMIAJUKUu0NQJ20SKrIWq1RMkFN5UaYxhuP8S+bsw4CXKNhmIQx74bBIuUXvU4GqiP3ijNw2BEKmCWkmNJ2bUmm6eU3CmIdnhnpEqJaB8J5+uoYjePwawddF/Pj5M0UjcJeWRkPTY4fOl2xQroXKHb1FN6huR1y5rcM32cdilk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319817; c=relaxed/simple; bh=TfE6+8mBkTEIhywmT4Bzq2nS1tp8WDshlGWqbEmSS/4=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=GhUUSXRDSC84OF/OXgDPEjE9uB1hp6/0JCjadpzhnjrCg7LMkUjruwenZxAYZ1ezuvKNR5oEAJvWQ1uBWD60Ppv+759P6ZZVldHPHv1cSt6bUcw2QIG9g3lwHmMqmq8Id77KPgc8VXTfmMvxAdGoocfYbvgLpst6iyDAO9rXiw0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=tetk/qHh; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=3aTLhbrQ; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="tetk/qHh"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="3aTLhbrQ" Message-ID: <20241007164914.139204961@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319814; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=NrMB7HiuGiz9Z3PezgOl5F9+0LPyymDMhrwUaKfLbFc=; b=tetk/qHhY1GEajgrfYN0MUKKeUyY4TpjVHv0bUXujXnFZfGtoOxyX9DdhIp4dud49CVOnn Nzkd3Q09wNNp/OYDkMpDRQM90nokp3nkJxTO0tR3XmxDV1scv61G3L8LliOIWSSLKVoR7p 649RVFGuuSrzsy/r0hxOssbTq+iIUSKT7YbFh2N1BxuMTKxRZWBuGvYBM03G0iMod3o4k4 fs2pO2TT4UtCggZ+6w0tkUQwDCA1cqUaLDLSrqw00qdAyS2v6GJHAD1V7XYCeO99MuivaK ka+auO2EblgQR7QnXckSp77u0sAyZ5YAOSEMXaxqTZeGtKF9tyuWnx7hQdmPRQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319814; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=NrMB7HiuGiz9Z3PezgOl5F9+0LPyymDMhrwUaKfLbFc=; b=3aTLhbrQ5pwcqUtfL8+rR3VZPY1CQbFYjtJlbdlAMcbG0EfOLi8XwivYAb23HxQYcFfae+ AnhAfJ028AMBNtDg== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 19/25] debugobjects: Prepare for batching References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:50:14 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move the debug_obj::object pointer into a union and add a pointer to the last node in a batch. That allows to implement batch processing efficiently by utilizing the stack property of hlist: When the first object of a batch is added to the list, then the batch pointer is set to the hlist node of the object itself. Any subsequent add retrieves the pointer to the last node from the first object in the list and uses that for storing the last node pointer in the newly added object. Add the pointer to the data structure and ensure that all relevant pool sizes are strictly batch sized. The actual batching implementation follows in subsequent changes. Signed-off-by: Thomas Gleixner Reviewed-by: Zhen Lei --- include/linux/debugobjects.h | 12 ++++++++---- lib/debugobjects.c | 10 +++++++--- 2 files changed, 15 insertions(+), 7 deletions(-) --- a/include/linux/debugobjects.h +++ b/include/linux/debugobjects.h @@ -23,13 +23,17 @@ struct debug_obj_descr; * @state: tracked object state * @astate: current active state * @object: pointer to the real object + * @batch_last: pointer to the last hlist node in a batch * @descr: pointer to an object type specific debug description structure */ struct debug_obj { - struct hlist_node node; - enum debug_obj_state state; - unsigned int astate; - void *object; + struct hlist_node node; + enum debug_obj_state state; + unsigned int astate; + union { + void *object; + struct hlist_node *batch_last; + }; const struct debug_obj_descr *descr; }; =20 --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -21,11 +21,15 @@ #define ODEBUG_HASH_BITS 14 #define ODEBUG_HASH_SIZE (1 << ODEBUG_HASH_BITS) =20 -#define ODEBUG_POOL_SIZE 1024 -#define ODEBUG_POOL_MIN_LEVEL 256 -#define ODEBUG_POOL_PERCPU_SIZE 64 +/* Must be power of two */ #define ODEBUG_BATCH_SIZE 16 =20 +/* Initial values. Must all be a multiple of batch size */ +#define ODEBUG_POOL_SIZE (64 * ODEBUG_BATCH_SIZE) +#define ODEBUG_POOL_MIN_LEVEL (ODEBUG_POOL_SIZE / 4) + +#define ODEBUG_POOL_PERCPU_SIZE (4 * ODEBUG_BATCH_SIZE) + #define ODEBUG_CHUNK_SHIFT PAGE_SHIFT #define ODEBUG_CHUNK_SIZE (1 << ODEBUG_CHUNK_SHIFT) #define ODEBUG_CHUNK_MASK (~(ODEBUG_CHUNK_SIZE - 1)) From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 26EB51DE3AC for ; Mon, 7 Oct 2024 16:50:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319819; cv=none; b=uky4Tu+IcuwCbvKd6Fqj7O6BB9r0uu89I9RmO2ziaF8oKbPqYe8R6tKGxQadOmoAdxo93hp3EzE+wLxfv0QRA88OFjm7/3lm2pjJc7fDF2vUAN8bAkIVTQz85VglC4FuJNMLin017f8Sibn1PVQP8XN7BUISKhsKEvgEXyHxbbE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319819; c=relaxed/simple; bh=ChB9QCtPxHuyVuVC/mOtOnn3hRZjwhD+HY88NV1O/bI=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=JJMRPun2E8FdvGqeP9fgLtzhu8CIx/Zs8IueTITrxGz5/Ii3Id+d2kV2qdJSREY+G4iwMdctMogmCGLlMpKZXJ6Y+h0HybkatqCMZzSBY8TiwqFxx3DG21hIZEX42gvJW/SDLTQgVOszYuO+nuXskKbaglvK6qra4sdS/BHRc0c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=YU4OGWjp; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=2Ef+kZiF; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="YU4OGWjp"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="2Ef+kZiF" Message-ID: <20241007164914.198647184@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319816; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=aM/x+wJOhqKJ3/ssLuuYF0bn8qxcMJkxeyzQCufMmNY=; b=YU4OGWjpRiH9po725drvUrAzhB2Wx+rhH3+RSCIU5TrmBCR5hsqUW7oYLSVgNCUBMBMT41 LS9I3t4/G0hsX0juKeoXTuA62qz9C3Qr0Sq7PjuWixX+657ZEkixpXXSG5tmZZqGuyjmet vdmMKKIBBcoJBcvHS6nPBEJhjcIGOO72NpRp4GLJp+j/Cbi5sMWCP/d1JM5Nj+22yNnUqw ojftnWQ6Ciop76VZaHFlJJ5DFUFUsJLwoVAg8khOT+0FBYTiZPCvN90AIBpxPUbCqGTaob lFz70kv/A4yqVdcl3qre8ObFB7PSoIzM9m034w/4ItBG5fgPhwFqs6oMUlHdGA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319816; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=aM/x+wJOhqKJ3/ssLuuYF0bn8qxcMJkxeyzQCufMmNY=; b=2Ef+kZiFgMlYebt/9b20l+94FRQlsL3DG50QmIUIiU/no00oNdPvc1q2vK2Nlk0uamcQgk dDZhKEvamAUIm8BQ== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 20/25] debugobjects: Prepare kmem_cache allocations for batching References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:50:15 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Allocate a batch and then push it into the pool. Utilize the debug_obj::last_node pointer for keeping track of the batch boundary. Signed-off-by: Thomas Gleixner --- lib/debugobjects.c | 80 ++++++++++++++++++++++++++++++++----------------= ----- 1 file changed, 49 insertions(+), 31 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -164,6 +164,22 @@ static bool pool_move_batch(struct obj_p return true; } =20 +static bool pool_push_batch(struct obj_pool *dst, struct hlist_head *head) +{ + struct hlist_node *last; + struct debug_obj *obj; + + if (dst->cnt >=3D dst->max_cnt) + return false; + + obj =3D hlist_entry(head->first, typeof(*obj), node); + last =3D obj->batch_last; + + hlist_splice_init(head, last, &dst->objects); + WRITE_ONCE(dst->cnt, dst->cnt + ODEBUG_BATCH_SIZE); + return true; +} + static bool pool_pop_batch(struct hlist_head *head, struct obj_pool *src) { if (!src->cnt) @@ -288,6 +304,28 @@ static void fill_pool_from_freelist(void clear_bit(0, &state); } =20 +static bool kmem_alloc_batch(struct hlist_head *head, struct kmem_cache *c= ache, gfp_t gfp) +{ + struct hlist_node *last =3D NULL; + struct debug_obj *obj; + + for (int cnt =3D 0; cnt < ODEBUG_BATCH_SIZE; cnt++) { + obj =3D kmem_cache_zalloc(cache, gfp); + if (!obj) { + free_object_list(head); + return false; + } + debug_objects_allocated++; + + if (!last) + last =3D &obj->node; + obj->batch_last =3D last; + + hlist_add_head(&obj->node, head); + } + return true; +} + static void fill_pool(void) { static atomic_t cpus_allocating; @@ -302,25 +340,14 @@ static void fill_pool(void) =20 atomic_inc(&cpus_allocating); while (pool_should_refill(&pool_global)) { - struct debug_obj *new, *last =3D NULL; HLIST_HEAD(head); - int cnt; =20 - for (cnt =3D 0; cnt < ODEBUG_BATCH_SIZE; cnt++) { - new =3D kmem_cache_zalloc(obj_cache, __GFP_HIGH | __GFP_NOWARN); - if (!new) - break; - hlist_add_head(&new->node, &head); - if (!last) - last =3D new; - } - if (!cnt) + if (!kmem_alloc_batch(&head, obj_cache, __GFP_HIGH | __GFP_NOWARN)) break; =20 guard(raw_spinlock_irqsave)(&pool_lock); - hlist_splice_init(&head, &last->node, &pool_global.objects); - debug_objects_allocated +=3D cnt; - WRITE_ONCE(pool_global.cnt, pool_global.cnt + cnt); + if (!pool_push_batch(&pool_global, &head)) + pool_push_batch(&pool_to_free, &head); } atomic_dec(&cpus_allocating); } @@ -1302,26 +1329,18 @@ void __init debug_objects_early_init(voi static bool __init debug_objects_replace_static_objects(struct kmem_cache = *cache) { struct debug_bucket *db =3D obj_hash; - struct debug_obj *obj, *new; struct hlist_node *tmp; + struct debug_obj *obj; HLIST_HEAD(objects); int i; =20 - for (i =3D 0; i < ODEBUG_POOL_SIZE; i++) { - obj =3D kmem_cache_zalloc(cache, GFP_KERNEL); - if (!obj) + for (i =3D 0; i < ODEBUG_POOL_SIZE; i +=3D ODEBUG_BATCH_SIZE) { + if (!kmem_alloc_batch(&objects, cache, GFP_KERNEL)) goto free; - hlist_add_head(&obj->node, &objects); + pool_push_batch(&pool_global, &objects); } =20 - debug_objects_allocated =3D ODEBUG_POOL_SIZE; - pool_global.cnt =3D ODEBUG_POOL_SIZE; - - /* - * Move the allocated objects to the global pool and disconnect the - * boot pool. - */ - hlist_move_list(&objects, &pool_global.objects); + /* Disconnect the boot pool. */ pool_boot.first =3D NULL; =20 /* Replace the active object references */ @@ -1329,9 +1348,8 @@ static bool __init debug_objects_replace hlist_move_list(&db->list, &objects); =20 hlist_for_each_entry(obj, &objects, node) { - new =3D hlist_entry(pool_global.objects.first, typeof(*obj), node); - hlist_del(&new->node); - pool_global.cnt--; + struct debug_obj *new =3D pcpu_alloc(); + /* copy object data */ *new =3D *obj; hlist_add_head(&new->node, &db->list); @@ -1340,7 +1358,7 @@ static bool __init debug_objects_replace return true; free: /* Can't use free_object_list() as the cache is not populated yet */ - hlist_for_each_entry_safe(obj, tmp, &objects, node) { + hlist_for_each_entry_safe(obj, tmp, &pool_global.objects, node) { hlist_del(&obj->node); kmem_cache_free(cache, obj); } From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8226B1DC187 for ; Mon, 7 Oct 2024 16:50:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319820; cv=none; b=qFGIW9bJE7dCDUQnkboce0W6z9kSPo0F95CdCoKErj8mc067LSLFW3DvCX5FkWPMVYi+ViJ4x4HxE2IRc2H4i+xD9Izz6h0tLWRLjgKAYW7Ew+MmXApP8wI/zAPa/nZB8sCtqpockKjfKUuCGRUAeiLzz7kPQfV7Sk0lVyKFOB8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319820; c=relaxed/simple; bh=A/QoDAJ8UX7nh5/h6SAiKHXrp6KgHkiTwFui3vTEugs=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=vFeg/+hv0u8JuSIZ+La5L0a2BYaUhHbTXW6EGi3yik+iRouEe6zJ3xAZV29aoODLI0BHBxTyEculPzgxO8Rte5EQ6WEXUdxAg++J8TlQLE9Tojozv7VolY4SKBuPUL1RDgiknt457byBVXjIoOSyqXkHRtTq4xMFEbEXYbVBC2A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=3iy5XBcx; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=jWhpu0ZS; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="3iy5XBcx"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="jWhpu0ZS" Message-ID: <20241007164914.258995000@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319817; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=2KVZpIKee4ylgIKgtuYLEAYJPCkBNT2M6tYQt6vWCCo=; b=3iy5XBcxVzTRm1l4neosADf9zarTysfMYt91IJ1rRHiKjelEoQbss8/7SnF5ih+mZqQpKw 0Vgw6Aa7ijL8ey7rKvfh9/zvtQyby4y0sUh59u7B5YWzNjMhQNKZ9DcdMPWI1pk7NKx+I3 mRK5h4f8uYkXPJYzk8cQH3lRYm9Uymms+0JOpuahvM2Ixgc1pKqJZqAu0KY7oPqvaTTlVe tz6RKFMVHTzf4+c3jx0ZR0+NwNdYLn0uTzs/ZmaVjvAhMbVwJj5EgLQGSBu5ZD0P6VcMkP 5z9IyKSZ881wVIVurjEkHZauIKs01tHVq7laS9wtETyBCSnpsaD7ZUA/POph4Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319817; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=2KVZpIKee4ylgIKgtuYLEAYJPCkBNT2M6tYQt6vWCCo=; b=jWhpu0ZSHk/ECb2Rtw1Z1lokwnC71cRTVSUt/wD76qxZBcmN5WzU9bJUUt4S+yEl/VjRjT YJjRLGuUgC+9hgBw== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 21/25] debugobjects: Implement batch processing References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:50:17 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Adding and removing single objects in a loop is bad in terms of lock contention and cache line accesses. To implement batching, record the last object in a batch in the object itself. This is trivialy possible as hlists are strictly stacks. At a batch boundary, when the first object is added to the list the object stores a pointer to itself in debug_obj::batch_last. When the next object is added to the list then the batch_last pointer is retrieved from the first object in the list and stored in the to be added one. That means for batch processing the first object always has a pointer to the last object in a batch, which allows to move batches in a cache line efficient way and reduces the lock held time. Signed-off-by: Thomas Gleixner Reviewed-by: Zhen Lei --- lib/debugobjects.c | 61 +++++++++++++++++++++++++++++++++++++++---------= ----- 1 file changed, 46 insertions(+), 15 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -149,18 +149,31 @@ static __always_inline bool pool_must_re =20 static bool pool_move_batch(struct obj_pool *dst, struct obj_pool *src) { - if (dst->cnt + ODEBUG_BATCH_SIZE > dst->max_cnt || !src->cnt) + struct hlist_node *last, *next_batch, *first_batch; + struct debug_obj *obj; + + if (dst->cnt >=3D dst->max_cnt || !src->cnt) return false; =20 - for (int i =3D 0; i < ODEBUG_BATCH_SIZE && src->cnt; i++) { - struct hlist_node *node =3D src->objects.first; + first_batch =3D src->objects.first; + obj =3D hlist_entry(first_batch, typeof(*obj), node); + last =3D obj->batch_last; + next_batch =3D last->next; =20 - WRITE_ONCE(src->cnt, src->cnt - 1); - WRITE_ONCE(dst->cnt, dst->cnt + 1); + /* Move the next batch to the front of the source pool */ + src->objects.first =3D next_batch; + if (next_batch) + next_batch->pprev =3D &src->objects.first; + + /* Add the extracted batch to the destination pool */ + last->next =3D dst->objects.first; + if (last->next) + last->next->pprev =3D &last->next; + first_batch->pprev =3D &dst->objects.first; + dst->objects.first =3D first_batch; =20 - hlist_del(node); - hlist_add_head(node, &dst->objects); - } + WRITE_ONCE(src->cnt, src->cnt - ODEBUG_BATCH_SIZE); + WRITE_ONCE(dst->cnt, dst->cnt + ODEBUG_BATCH_SIZE); return true; } =20 @@ -182,16 +195,27 @@ static bool pool_push_batch(struct obj_p =20 static bool pool_pop_batch(struct hlist_head *head, struct obj_pool *src) { + struct hlist_node *last, *next; + struct debug_obj *obj; + if (!src->cnt) return false; =20 - for (int i =3D 0; src->cnt && i < ODEBUG_BATCH_SIZE; i++) { - struct hlist_node *node =3D src->objects.first; + /* Move the complete list to the head */ + hlist_move_list(&src->objects, head); =20 - WRITE_ONCE(src->cnt, src->cnt - 1); - hlist_del(node); - hlist_add_head(node, head); - } + obj =3D hlist_entry(head->first, typeof(*obj), node); + last =3D obj->batch_last; + next =3D last->next; + /* Disconnect the batch from the list */ + last->next =3D NULL; + + /* Move the node after last back to the source pool. */ + src->objects.first =3D next; + if (next) + next->pprev =3D &src->objects.first; + + WRITE_ONCE(src->cnt, src->cnt - ODEBUG_BATCH_SIZE); return true; } =20 @@ -226,7 +250,7 @@ static struct debug_obj *pcpu_alloc(void if (!pool_move_batch(pcp, &pool_global)) return NULL; } - obj_pool_used +=3D pcp->cnt; + obj_pool_used +=3D ODEBUG_BATCH_SIZE; =20 if (obj_pool_used > obj_pool_max_used) obj_pool_max_used =3D obj_pool_used; @@ -239,9 +263,16 @@ static struct debug_obj *pcpu_alloc(void static void pcpu_free(struct debug_obj *obj) { struct obj_pool *pcp =3D this_cpu_ptr(&pool_pcpu); + struct debug_obj *first; =20 lockdep_assert_irqs_disabled(); =20 + if (!(pcp->cnt % ODEBUG_BATCH_SIZE)) { + obj->batch_last =3D &obj->node; + } else { + first =3D hlist_entry(pcp->objects.first, typeof(*first), node); + obj->batch_last =3D first->batch_last; + } hlist_add_head(&obj->node, &pcp->objects); pcp->cnt++; From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BD4571DE3D0 for ; Mon, 7 Oct 2024 16:50:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319821; cv=none; b=mELHsjjaUbGv6OrZYs3gzPcb+DIzde+uKI26W23C4oQZE88Ss8A3Fd/OmeUfuelwF4YS76fnY9lD+KBHA9RggGcuFfNfZTxRqDbG+fORTrdORyc4RHqB/LvNAD3dr8IZFiV8X2gLrLAA7cziqM21rOgxLh8aU7Aqqwm4KqrQzvQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319821; c=relaxed/simple; bh=OTNHBzJjLjVr4pJvMT7It2p7L/n3O0GjawQZblft/aE=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=u1F6XfpLJiwMBKJ7j6ym2A6K/Er+lp1KP4ikRejlfO3dDVkK0vEaLQef55/6hnblDoyB13B3YQz2XWE0b0/o2HaqZRTkXMRNFNSxTCr9pAWvMQdz06W0gUhRqf8RrEJdAtJRGo4JylQtL7V2gzD0O6aKL2n9c2rrZMtPSQUEqlM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=l/c5K/zS; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=HFLObFmX; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="l/c5K/zS"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="HFLObFmX" Message-ID: <20241007164914.318776207@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319818; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=X6M/ZpzoVuNVXCR6ojlXr0UcScmjUOMKBbYbJO7HIQA=; b=l/c5K/zSdVTFrg60DwX2b/SEhl+ZaGNFOGEP1OCXDOlNqR+mWSXC0O7OWfGenhaj5e15Jk uWcD5So+2o5Bce1MQVBgXZRIKDoRp1CScR4U91DG2mfQmsHUmvQLY9PXHeMBe1MPD7HgiT 4dK9QLOE8dWMYeZ89M3YSf2uwaVIaXEz8ZfSMPvlsm5Loje1fKc9zkXFGGm0KabJO3n/d2 8MytTOC3DfLOZ9ZmFw+QJ521kdLSrUg5Py9qM7vpK8KOoKHDBH3mazjKps54L+PelewQHc NfkjFRjL1dnqcB3xlWjh5owyAwufikvN6f1eDCOlFXT5VcyJb+yWD9UgSsirzA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319818; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=X6M/ZpzoVuNVXCR6ojlXr0UcScmjUOMKBbYbJO7HIQA=; b=HFLObFmX8Bg9ZAvYgM1bVNa/JC8haRSuxvVxLZytJqMgat29s0iNGRKz2m7oM+ga69KfqH FReEJsL2h0vjSXCA== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 22/25] debugobjects: Move pool statistics into global_pool struct References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:50:18 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Keep it along with the pool as that's a hot cache line anyway and it makes the code more comprehensible. Signed-off-by: Thomas Gleixner Reviewed-by: Zhen Lei --- lib/debugobjects.c | 85 +++++++++++++++++++++++++++++++-----------------= ----- 1 file changed, 51 insertions(+), 34 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -47,11 +47,18 @@ struct debug_bucket { raw_spinlock_t lock; }; =20 +struct pool_stats { + unsigned int cur_used; + unsigned int max_used; + unsigned int min_fill; +}; + struct obj_pool { struct hlist_head objects; unsigned int cnt; unsigned int min_cnt; unsigned int max_cnt; + struct pool_stats stats; } ____cacheline_aligned; =20 =20 @@ -66,8 +73,11 @@ static struct debug_obj obj_static_pool static DEFINE_RAW_SPINLOCK(pool_lock); =20 static struct obj_pool pool_global =3D { - .min_cnt =3D ODEBUG_POOL_MIN_LEVEL, - .max_cnt =3D ODEBUG_POOL_SIZE, + .min_cnt =3D ODEBUG_POOL_MIN_LEVEL, + .max_cnt =3D ODEBUG_POOL_SIZE, + .stats =3D { + .min_fill =3D ODEBUG_POOL_SIZE, + }, }; =20 static struct obj_pool pool_to_free =3D { @@ -76,16 +86,6 @@ static struct obj_pool pool_to_free =3D { =20 static HLIST_HEAD(pool_boot); =20 -/* - * Because of the presence of percpu free pools, obj_pool_free will - * under-count those in the percpu free pools. Similarly, obj_pool_used - * will over-count those in the percpu free pools. Adjustments will be - * made at debug_stats_show(). Both obj_pool_min_free and obj_pool_max_used - * can be off. - */ -static int __data_racy obj_pool_min_free =3D ODEBUG_POOL_SIZE; -static int obj_pool_used; -static int __data_racy obj_pool_max_used; static bool obj_freeing; =20 static int __data_racy debug_objects_maxchain __read_mostly; @@ -231,6 +231,19 @@ static struct debug_obj *__alloc_object( return obj; } =20 +static void pcpu_refill_stats(void) +{ + struct pool_stats *stats =3D &pool_global.stats; + + WRITE_ONCE(stats->cur_used, stats->cur_used + ODEBUG_BATCH_SIZE); + + if (stats->cur_used > stats->max_used) + stats->max_used =3D stats->cur_used; + + if (pool_global.cnt < stats->min_fill) + stats->min_fill =3D pool_global.cnt; +} + static struct debug_obj *pcpu_alloc(void) { struct obj_pool *pcp =3D this_cpu_ptr(&pool_pcpu); @@ -250,13 +263,7 @@ static struct debug_obj *pcpu_alloc(void if (!pool_move_batch(pcp, &pool_global)) return NULL; } - obj_pool_used +=3D ODEBUG_BATCH_SIZE; - - if (obj_pool_used > obj_pool_max_used) - obj_pool_max_used =3D obj_pool_used; - - if (pool_global.cnt < obj_pool_min_free) - obj_pool_min_free =3D pool_global.cnt; + pcpu_refill_stats(); } } =20 @@ -285,7 +292,7 @@ static void pcpu_free(struct debug_obj * /* Try to fit the batch into the pool_global first */ if (!pool_move_batch(&pool_global, pcp)) pool_move_batch(&pool_to_free, pcp); - obj_pool_used -=3D ODEBUG_BATCH_SIZE; + WRITE_ONCE(pool_global.stats.cur_used, pool_global.stats.cur_used - ODEBU= G_BATCH_SIZE); } =20 static void free_object_list(struct hlist_head *head) @@ -1074,23 +1081,33 @@ void debug_check_no_obj_freed(const void =20 static int debug_stats_show(struct seq_file *m, void *v) { - int cpu, obj_percpu_free =3D 0; + unsigned int cpu, pool_used, pcp_free =3D 0; =20 + /* + * pool_global.stats.cur_used is the number of batches currently + * handed out to per CPU pools. Convert it to number of objects + * and subtract the number of free objects in the per CPU pools. + * As this is lockless the number is an estimate. + */ for_each_possible_cpu(cpu) - obj_percpu_free +=3D per_cpu(pool_pcpu.cnt, cpu); + pcp_free +=3D per_cpu(pool_pcpu.cnt, cpu); =20 - seq_printf(m, "max_chain :%d\n", debug_objects_maxchain); - seq_printf(m, "max_checked :%d\n", debug_objects_maxchecked); - seq_printf(m, "warnings :%d\n", debug_objects_warnings); - seq_printf(m, "fixups :%d\n", debug_objects_fixups); - seq_printf(m, "pool_free :%d\n", pool_count(&pool_global) + obj_percp= u_free); - seq_printf(m, "pool_pcp_free :%d\n", obj_percpu_free); - seq_printf(m, "pool_min_free :%d\n", obj_pool_min_free); - seq_printf(m, "pool_used :%d\n", obj_pool_used - obj_percpu_free); - seq_printf(m, "pool_max_used :%d\n", obj_pool_max_used); - seq_printf(m, "on_free_list :%d\n", pool_count(&pool_to_free)); - seq_printf(m, "objs_allocated:%d\n", debug_objects_allocated); - seq_printf(m, "objs_freed :%d\n", debug_objects_freed); + pool_used =3D data_race(pool_global.stats.cur_used); + pcp_free =3D min(pool_used, pcp_free); + pool_used -=3D pcp_free; + + seq_printf(m, "max_chain : %d\n", debug_objects_maxchain); + seq_printf(m, "max_checked : %d\n", debug_objects_maxchecked); + seq_printf(m, "warnings : %d\n", debug_objects_warnings); + seq_printf(m, "fixups : %d\n", debug_objects_fixups); + seq_printf(m, "pool_free : %u\n", pool_count(&pool_global) + pcp_free= ); + seq_printf(m, "pool_pcp_free : %u\n", pcp_free); + seq_printf(m, "pool_min_free : %u\n", data_race(pool_global.stats.min_fil= l)); + seq_printf(m, "pool_used : %u\n", pool_used); + seq_printf(m, "pool_max_used : %u\n", data_race(pool_global.stats.max_use= d)); + seq_printf(m, "on_free_list : %u\n", pool_count(&pool_to_free)); + seq_printf(m, "objs_allocated: %d\n", debug_objects_allocated); + seq_printf(m, "objs_freed : %d\n", debug_objects_freed); return 0; } DEFINE_SHOW_ATTRIBUTE(debug_stats); From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 475B81DE4E0 for ; Mon, 7 Oct 2024 16:50:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319822; cv=none; b=s8g0vxL2M/V0oVWQbbXjFBZhCxsVrboZ1h7/V0L7crh6CAPisZrAaO3hF1vwuxWwT8Zopd2BsSyigpZGtqVO6Zi1Bcd9TOUN+lGryD0ibKmfwT4dnu67q2whzziaQZJ9e0fKvYT8rHYnj3g6mNSS2nQoTK2C26JTyXkDYPmIF5w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319822; c=relaxed/simple; bh=ZexrWVUerzcxTJsbqug7YTAVb8B0maQyfX0COGGQBxM=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=JVjgWLtPHBEv17tHjWN7tzSlJGQvyWwEp87EpPB6AYEh1ZlQ6DdJorHtb40xRDLTXB/neiGvslTMsIwJYS+fb2Nc2a2n29MJ9zMU4NLk0VliF138Q7fI/cQNhJomoI7uV17ETHjPr9/d7s9hmLOQZ8m1z8843nc2UICMCkKsqzU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=pg/EqtzB; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=fQKdwfKM; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="pg/EqtzB"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="fQKdwfKM" Message-ID: <20241007164914.378676302@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319819; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=f6OqY96nNmXu7osNi/AN7bUMqCbv9Be5svEmpKd6MDE=; b=pg/EqtzB3GeUmPs6WR+LzHzhzh+MFY3VSWtjxKH80nqUaBdCtPicvtMCNAxDdKuoFfL1OX bPpyhj1ElgVa2XAJq39VNuJk9hGd1me5cznafXVCY7X752shxcNv9tYMUGyXJpVaazN2pP UvZgJAXdnLyLoSKbMfegvYX4+FTuOE6k3hysKL6Ylouq3xM9PJvh0H+5kDVWQ64Ly6EMsD E2kTy9t855WdzRbpcCvxGryNquSVMuSsXo46n4rUErLKCYoqJkfgynOiYufm/RMpZT/B2x sA2daFpHXA/jB8kWHiyT1eZIY3OWIQRX42XyTs/+AgoRXe0iZrxrDvTye4sZ6Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319819; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=f6OqY96nNmXu7osNi/AN7bUMqCbv9Be5svEmpKd6MDE=; b=fQKdwfKMets9HP8XKYrMlQx3ZuaiJwXBjjFglZ4PsG1q69LGPwa6HkGo7un4OciwpqH3hC Yr7aWd/0h6aeUCCw== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 23/25] debugobjects: Double the per CPU slots References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:50:19 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In situations where objects are rapidly allocated from the pool and handed back, the size of the per CPU pool turns out to be too small. Double the size of the per CPU pool. This reduces the kmem cache allocation and free operations during a kernel = compile: alloc free Baseline: 380k 330k Double size: 295k 245k Especially the reduction of allocations is important because that happens in the hot path when objects are initialized. The maximum increase in per CPU pool memory consumption is about 2.5K per online CPU, which is acceptable. Signed-off-by: Thomas Gleixner Reviewed-by: Zhen Lei --- lib/debugobjects.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -28,7 +28,7 @@ #define ODEBUG_POOL_SIZE (64 * ODEBUG_BATCH_SIZE) #define ODEBUG_POOL_MIN_LEVEL (ODEBUG_POOL_SIZE / 4) =20 -#define ODEBUG_POOL_PERCPU_SIZE (4 * ODEBUG_BATCH_SIZE) +#define ODEBUG_POOL_PERCPU_SIZE (8 * ODEBUG_BATCH_SIZE) =20 #define ODEBUG_CHUNK_SHIFT PAGE_SHIFT #define ODEBUG_CHUNK_SIZE (1 << ODEBUG_CHUNK_SHIFT) From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 340D61DE4F6 for ; Mon, 7 Oct 2024 16:50:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319823; cv=none; b=Sn9AeIPOvXkXxrT4TxP8uVdA9v2zqMcoTypWUii/Jp2WooRTg/qG540bcx+FljrWYldLqlG7Cbbr1mgBF2v4kkLdY2+GbWunBHvqm6nBy6r+6mwtXZ1g7HBq4imH3q2Wqz1r+nDd9sq9vGiaDm74hYIshYOgAbFkDz6gDov1Ei4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319823; c=relaxed/simple; bh=A9HbtYbk1zUjHwIbER7JB9z1psrSbF3hl4GInqdSUAs=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=owfMmPicY4SAKbSMNjbgkvlSYeMr3bK3YfjETTBzGRTxw7YADGLJ7ju22ik0NdKhslJupacjKJDvOlF/qLHs2ym/73RzuUVNGjXPnj671OYI/c095QeA+v5YVDYCM59pgEGvrYGIJDwpBOroHQnovUkbdvgkxnKQZWGS/0SVYw0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=Nw02v3u4; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=uMQ9fmZT; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="Nw02v3u4"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="uMQ9fmZT" Message-ID: <20241007164914.439053085@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319820; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=8EaFnUcTOTl6eBkUKro450FuKCraH5nZVOpj/jHvlNA=; b=Nw02v3u4NVdtFf4nltBHBprJcmq+KduMrejVKG9dzhWT44bDVu+KXGeLHdqx/GIErauzOd CMMxArpXKCWQMwu1RkQ8W+x7cIyEOSot7wzpnu1vu199VZK2D/2nToPw2RfHui4JP1kWog C+K7Rpg2ZqVUBJVtnskI8GA5u9wPgnqqddAG+amN6YVIVrpFqPhe+iRFLTtmDRO2bOl8Yr XBkD1MHuhXckU6BPbjZgMkZLJFKtbrVVOn9JqoVuy1ITudUSe2YCpndRss2OIiJE20ktA0 QtCq50VPUZtL+Ya6A3EudhJUjPhj8KYCcEWl77umzdIYBE81f8bk2JqIwh7EGw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319820; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=8EaFnUcTOTl6eBkUKro450FuKCraH5nZVOpj/jHvlNA=; b=uMQ9fmZTwuy1rsSiNCf/PXGDMTCbVvUfgRFc3eQNRTjR8/5BRrN4lGwDzlDm+OonB/MebC lF/ulf07ifKhvwCg== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 24/25] debugobjects: Refill per CPU pool more agressively References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:50:20 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Right now the per CPU pools are only refilled when they become empty. That's suboptimal especially when there are still non-freed objects in the to free list. Check whether an allocation from the per CPU pool emptied a batch and try to allocate from the free pool if that still has objects available. kmem_cache_alloc() kmem_cache_free() Baseline: 295k 245k Refill: 225k 173k Signed-off-by: Thomas Gleixner Reviewed-by: Zhen Lei --- lib/debugobjects.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -255,6 +255,24 @@ static struct debug_obj *pcpu_alloc(void =20 if (likely(obj)) { pcp->cnt--; + /* + * If this emptied a batch try to refill from the + * free pool. Don't do that if this was the top-most + * batch as pcpu_free() expects the per CPU pool + * to be less than ODEBUG_POOL_PERCPU_SIZE. + */ + if (unlikely(pcp->cnt < (ODEBUG_POOL_PERCPU_SIZE - ODEBUG_BATCH_SIZE) && + !(pcp->cnt % ODEBUG_BATCH_SIZE))) { + /* + * Don't try to allocate from the regular pool here + * to not exhaust it prematurely. + */ + if (pool_count(&pool_to_free)) { + guard(raw_spinlock)(&pool_lock); + pool_move_batch(pcp, &pool_to_free); + pcpu_refill_stats(); + } + } return obj; } From nobody Wed Nov 27 21:51:34 2024 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5A02D1DE896 for ; Mon, 7 Oct 2024 16:50:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319825; cv=none; b=SJN21N2PepUBzGXOfwJVYE5fKnuSHfAbmuIQ8AKshXU/RbPYSoau4sCkHvLSPORlpSiZN+cGdIHOzy/CHBrSzodK9Xix9XwANJa1P8i7aQFi9bQCSoFwJWvTdOQaUGKM50WS3eairDvmoAN7yzZr9BnOU0YbozLlKUjlPtobOZ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728319825; c=relaxed/simple; bh=279xa8mpg28AxbvwJ5VJI0fKKgnKuyWo7R6bGg0y0mo=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=IY+/ga2YBL/UqgpwXh80muqtk7uxFOZdFPDqtVqPINS4EHCbljujvEoX/I1pyAyKOC60UXgxOsmY1JKjWW8/0LzMoTcBJY+P14QaFCQwqoZJLaNyWC9eyYcYntpfias5Cw6GgL2U62/Ut3iUUmhSdzTxb2ihry7SMoXlw8W6abg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=ZLLF2hBO; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=pKRVkm8D; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="ZLLF2hBO"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="pKRVkm8D" Message-ID: <20241007164914.499184215@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1728319822; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=U+tLkJdJzX5oBPcMxYGnMu9J72CXO1tkPlg0+hp5+mc=; b=ZLLF2hBORDZNgqB6RGuan1NNhc9PPs1VCx2W8udEzRhxGg3bNFVKw6efNVpPaIEvWDItID TL7gHlj5ZeR/oWD3vAMA4y0xAO+nOMzv/uRZjWunyPn2td8xAv0zzU8wGpyqEZ2cOU59FW 0oniecanWP9fsdripRvouZqyTEiAi9VEhkH9yr2Sl+A6L9713cOirfQwq1ywCar/rZV3Rj GO3UWZfnnQkXjNbQlYPy2Y/6wvGTlGYkCoAjIKSOD9Md2PV7qlgx8fNTIrwOT5GJdN6cSw d4T+C09Bl5kqfvvYideBPHeoU1dG4N1TNkJtfO8JUbQqaWjeHcD8xgW0Ozn9bQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1728319822; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=U+tLkJdJzX5oBPcMxYGnMu9J72CXO1tkPlg0+hp5+mc=; b=pKRVkm8DBGXSlBWiDWmRwtEdt6hNdOmO+WR2gziTKtO5OjoNxHTO/e8uxPPH7DqrmpjdZx wZO5BVDoMCxvqkDQ== From: Thomas Gleixner To: LKML Cc: Zhen Lei , Waiman Long Subject: [patch 25/25] debugobjects: Track object usage to avoid premature freeing of objects References: <20241007163507.647617031@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Mon, 7 Oct 2024 18:50:21 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The freelist is freed at a constant rate independent of the actual usage requirements. That's bad in scenarios where usage comes in bursts. The end of a burst puts the object on the free list and freeing proceeds even when the next burst which requires objects started again. Keep track of the usage with a exponentially wheighted moving average and take that into account in the worker function which frees objects from the free list. This further reduces the kmem_cache allocation/free rate for a full kernel compile: kmem_cache_alloc() kmem_cache_free() Baseline: 225k 245k Usage: 170k 117k Signed-off-by: Thomas Gleixner Reviewed-by: Zhen Lei --- lib/debugobjects.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++++= ----- 1 file changed, 57 insertions(+), 5 deletions(-) --- a/lib/debugobjects.c +++ b/lib/debugobjects.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -86,6 +87,7 @@ static struct obj_pool pool_to_free =3D { =20 static HLIST_HEAD(pool_boot); =20 +static unsigned long avg_usage; static bool obj_freeing; =20 static int __data_racy debug_objects_maxchain __read_mostly; @@ -382,11 +384,28 @@ static bool kmem_alloc_batch(struct hlis return true; } =20 +static bool pool_can_fill(struct obj_pool *dst, struct obj_pool *src) +{ + unsigned int cnt =3D pool_count(dst); + + if (cnt >=3D dst->min_cnt) + return true; + + return READ_ONCE(src->cnt) >=3D cnt - dst->min_cnt; +} + static void fill_pool(void) { static atomic_t cpus_allocating; =20 /* + * If the free pool has enough capacity, then don't try allocating + * unless the global pool has reached the cricital level. + */ + if (!pool_must_refill(&pool_global) && pool_can_fill(&pool_global, &pool_= to_free)) + return; + + /* * Avoid allocation and lock contention when: * - One other CPU is already allocating * - the global pool has not reached the critical level yet @@ -427,11 +446,31 @@ static struct debug_obj *lookup_object(v return NULL; } =20 +static void calc_usage(void) +{ + static DEFINE_RAW_SPINLOCK(avg_lock); + static unsigned long avg_period; + unsigned long cur, now =3D jiffies; + + if (!time_after_eq(now, READ_ONCE(avg_period))) + return; + + if (!raw_spin_trylock(&avg_lock)) + return; + + WRITE_ONCE(avg_period, now + msecs_to_jiffies(10)); + cur =3D READ_ONCE(pool_global.stats.cur_used) * ODEBUG_FREE_WORK_MAX; + WRITE_ONCE(avg_usage, calc_load(avg_usage, EXP_5, cur)); + raw_spin_unlock(&avg_lock); +} + static struct debug_obj *alloc_object(void *addr, struct debug_bucket *b, const struct debug_obj_descr *descr) { struct debug_obj *obj; =20 + calc_usage(); + if (static_branch_likely(&obj_cache_enabled)) obj =3D pcpu_alloc(); else @@ -450,14 +489,26 @@ static struct debug_obj *alloc_object(vo /* workqueue function to free objects. */ static void free_obj_work(struct work_struct *work) { - bool free =3D true; + static unsigned long last_use_avg; + unsigned long cur_used, last_used, delta; + unsigned int max_free =3D 0; =20 WRITE_ONCE(obj_freeing, false); =20 + /* Rate limit freeing based on current use average */ + cur_used =3D READ_ONCE(avg_usage); + last_used =3D last_use_avg; + last_use_avg =3D cur_used; + if (!pool_count(&pool_to_free)) return; =20 - for (unsigned int cnt =3D 0; cnt < ODEBUG_FREE_WORK_MAX; cnt++) { + if (cur_used <=3D last_used) { + delta =3D (last_used - cur_used) / ODEBUG_FREE_WORK_MAX; + max_free =3D min(delta, ODEBUG_FREE_WORK_MAX); + } + + for (int cnt =3D 0; cnt < ODEBUG_FREE_WORK_MAX; cnt++) { HLIST_HEAD(tofree); =20 /* Acquire and drop the lock for each batch */ @@ -468,9 +519,10 @@ static void free_obj_work(struct work_st /* Refill the global pool if possible */ if (pool_move_batch(&pool_global, &pool_to_free)) { /* Don't free as there seems to be demand */ - free =3D false; - } else if (free) { + max_free =3D 0; + } else if (max_free) { pool_pop_batch(&tofree, &pool_to_free); + max_free--; } else { return; } @@ -1110,7 +1162,7 @@ static int debug_stats_show(struct seq_f for_each_possible_cpu(cpu) pcp_free +=3D per_cpu(pool_pcpu.cnt, cpu); =20 - pool_used =3D data_race(pool_global.stats.cur_used); + pool_used =3D READ_ONCE(pool_global.stats.cur_used); pcp_free =3D min(pool_used, pcp_free); pool_used -=3D pcp_free;