From nobody Wed Dec 17 19:18:24 2025 Received: from out-181.mta1.migadu.com (out-181.mta1.migadu.com [95.215.58.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 654931A5BA2 for ; Sun, 12 Oct 2025 21:24:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760304269; cv=none; b=ZzqKiSGChfRMoIIXFV9xtqU2GBsD6gVLLt1F162NgcHTSiCFhX87ksjq6Ly6CamZ8Sqf4eU98bX+1LYJB4l64RcoId2Jxb2GUqMgR5K8xOYdHDC6TG5odpEVzYMk9nUbQqmA6eisaw7aIq0qLGnJ3CnQQ9Pg7DPuqegl7cYec9E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760304269; c=relaxed/simple; bh=mX/hl98PUS0ab9cB7TcvhKPHENndP8MazQbLOij4Aow=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sy1e3Af45minY6Je7T5X6gmeuGbkzN9o195kPcxn36Rb6rBSDXHrxlxWsganZuC/9DAxUH7AQfh/mCSq3AIXRMuaWub7xg/afNgPWC69uT7Kh3TQQ3f5gYqRnwVukaDlqkb/BiaaC64OinR2Rht3wjL45gb3NDp8fpRhbEQc7Mo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=oSKmYwHf; arc=none smtp.client-ip=95.215.58.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="oSKmYwHf" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1760304264; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gNiDE729JY73QUQVdhCZ0AfYtP+HlTQfQJFdiqUEqdU=; b=oSKmYwHf4CBk5a2AVp34iRvshjTH6CccZBByednp0UQp9LfWB58QlaCoWIqOcbwlYLL3JV AaUs/WbAJj/VYxYW0sE/Tg1lYoz9VwuPn/iaw5IRO64vHW98ZUWSm24KS3dUkGbrW65kJk nhx2A4Y59mj7/K+Yle6/VZiKbM3CqSM= From: Kent Overstreet To: akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, Kent Overstreet Subject: [PATCH v2 1/8] closures: Improve closure_put_after_sub_checks Date: Sun, 12 Oct 2025 17:24:04 -0400 Message-ID: <20251012212414.3225948-2-kent.overstreet@linux.dev> In-Reply-To: <20251012212414.3225948-1-kent.overstreet@linux.dev> References: <20251012212414.3225948-1-kent.overstreet@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" Add a guard bit for CLOSURE_DESTRUCTOR underflow, print the full cl->remaining value, and rename variables for clarity. Signed-off-by: Kent Overstreet --- include/linux/closure.h | 2 +- lib/closure.c | 28 +++++++++++++++------------- 2 files changed, 16 insertions(+), 14 deletions(-) diff --git a/include/linux/closure.h b/include/linux/closure.h index 880fe85e35e9..a6fcc33fafce 100644 --- a/include/linux/closure.h +++ b/include/linux/closure.h @@ -135,7 +135,7 @@ enum closure_state { }; =20 #define CLOSURE_GUARD_MASK \ - ((CLOSURE_DESTRUCTOR|CLOSURE_WAITING|CLOSURE_RUNNING) << 1) + (((CLOSURE_DESTRUCTOR|CLOSURE_WAITING|CLOSURE_RUNNING) << 1)|(CLOSURE_BIT= S_START >> 1)) =20 #define CLOSURE_REMAINING_MASK (CLOSURE_BITS_START - 1) #define CLOSURE_REMAINING_INITIALIZER (1|CLOSURE_RUNNING) diff --git a/lib/closure.c b/lib/closure.c index 2bfe7d2a0048..5fafc8b0e99d 100644 --- a/lib/closure.c +++ b/lib/closure.c @@ -13,23 +13,25 @@ #include #include =20 -static inline void closure_put_after_sub_checks(int flags) +static void closure_val_checks(struct closure *cl, unsigned new) { - int r =3D flags & CLOSURE_REMAINING_MASK; - - if (WARN(flags & CLOSURE_GUARD_MASK, - "closure has guard bits set: %x (%u)", - flags & CLOSURE_GUARD_MASK, (unsigned) __fls(r))) - r &=3D ~CLOSURE_GUARD_MASK; - - WARN(!r && (flags & ~CLOSURE_DESTRUCTOR), - "closure ref hit 0 with incorrect flags set: %x (%u)", - flags & ~CLOSURE_DESTRUCTOR, (unsigned) __fls(flags)); + unsigned count =3D new & CLOSURE_REMAINING_MASK; + + if (WARN(new & CLOSURE_GUARD_MASK, + "closure %ps has guard bits set: %x (%u)", + cl->fn, + new, (unsigned) __fls(new & CLOSURE_GUARD_MASK))) + new &=3D ~CLOSURE_GUARD_MASK; + + WARN(!count && (new & ~CLOSURE_DESTRUCTOR), + "closure %ps ref hit 0 with incorrect flags set: %x (%u)", + cl->fn, + new, (unsigned) __fls(new)); } =20 static inline void closure_put_after_sub(struct closure *cl, int flags) { - closure_put_after_sub_checks(flags); + closure_val_checks(cl, flags); =20 if (!(flags & CLOSURE_REMAINING_MASK)) { smp_acquire__after_ctrl_dep(); @@ -167,7 +169,7 @@ void __sched closure_return_sync(struct closure *cl) unsigned flags =3D atomic_sub_return_release(1 + CLOSURE_RUNNING - CLOSUR= E_DESTRUCTOR, &cl->remaining); =20 - closure_put_after_sub_checks(flags); + closure_val_checks(cl, flags); =20 if (unlikely(flags & CLOSURE_REMAINING_MASK)) { while (1) { --=20 2.51.0 From nobody Wed Dec 17 19:18:24 2025 Received: from out-173.mta1.migadu.com (out-173.mta1.migadu.com [95.215.58.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6FBB622CBD9 for ; Sun, 12 Oct 2025 21:24:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760304270; cv=none; b=dSYPNQzagMsKZ5dkMevrW9OtXhiYu9OiudLNslP7k54+W2TIPpw3sV8SmFn0ufGbkccaSsAtn9g6j1eERNkVjX5bbIr+hQzFfAq+Zf5Fa19Q0wMCALIfsjE8xAbbzwVN0kwp6bgOqy8DEWC4ZPdTjXqLezUfqnbQxlvhBpdCpks= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760304270; c=relaxed/simple; bh=KoAm2saD7kfuW5yN0J71ngCyAbYwzOe7QtEeq809yes=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cYkQ1eR9hDRrjUP2i1GG+zh6kGt+VbhINO9Sbvi+2cbi3o/GqiPC6LOOuEFHyYZNTPRG7/WAKYAOFyAJ2nTZbb4p5unJR2Lx1ICXq15RRTayZjBnSUtl+oc6zDIeMngQm3nISVytZM6kL6HAw7GQmy7g6Czh7Ml2VQX+8gX3fPI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=JQn4VLWn; arc=none smtp.client-ip=95.215.58.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="JQn4VLWn" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1760304265; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zdLf9ksaW/ExPQ+mYz9EpqCEmK6bAhu602u9/qlr3zc=; b=JQn4VLWnVpFz8Xpp1qOSWwXycEZw+GAX15MLjbbKkpwigj3HfNYm4W1jCS0JHnAgliERU7 v83jobdJsektZEhZWTOidCDac5DLd/aXFlsFEXHeCykTw+tcZDCBs4lfkHOs4HqgeUxDE7 P/DSipg/bJldEfcDubpFVJpl7Asmt44= From: Kent Overstreet To: akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, Kent Overstreet Subject: [PATCH v2 2/8] closures: closure_sub() uses cmpxchg Date: Sun, 12 Oct 2025 17:24:05 -0400 Message-ID: <20251012212414.3225948-3-kent.overstreet@linux.dev> In-Reply-To: <20251012212414.3225948-1-kent.overstreet@linux.dev> References: <20251012212414.3225948-1-kent.overstreet@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" Part one of fixing a race in __closure_sync_timeout(). lib/closure.c was initially designed with the assumption that refs only hit 0 once, and when they hit 0 the thread that caused it to hit 0 now owns it - which avoids many races without the need for cmpxchg. But __closure_sync_timeout() broke this, because the timeout may elapse and the thread doing the sync may race with the final put. Switch to cmpxchg for manipulating cl->remaining; the next patch will fix some issues with the other thread's closure_put() accessing variables in struct closure after that final put. Signed-off-by: Kent Overstreet --- include/linux/closure.h | 9 ++++- lib/closure.c | 79 ++++++++++++++++++++++------------------- 2 files changed, 50 insertions(+), 38 deletions(-) diff --git a/include/linux/closure.h b/include/linux/closure.h index a6fcc33fafce..f626044d6ca2 100644 --- a/include/linux/closure.h +++ b/include/linux/closure.h @@ -169,11 +169,18 @@ struct closure { }; =20 void closure_sub(struct closure *cl, int v); -void closure_put(struct closure *cl); void __closure_wake_up(struct closure_waitlist *list); bool closure_wait(struct closure_waitlist *list, struct closure *cl); void __closure_sync(struct closure *cl); =20 +/* + * closure_put - decrement a closure's refcount + */ +static inline void closure_put(struct closure *cl) +{ + closure_sub(cl, 1); +} + static inline unsigned closure_nr_remaining(struct closure *cl) { return atomic_read(&cl->remaining) & CLOSURE_REMAINING_MASK; diff --git a/lib/closure.c b/lib/closure.c index 5fafc8b0e99d..21fadd12093c 100644 --- a/lib/closure.c +++ b/lib/closure.c @@ -13,14 +13,14 @@ #include #include =20 -static void closure_val_checks(struct closure *cl, unsigned new) +static void closure_val_checks(struct closure *cl, unsigned new, int d) { unsigned count =3D new & CLOSURE_REMAINING_MASK; =20 if (WARN(new & CLOSURE_GUARD_MASK, - "closure %ps has guard bits set: %x (%u)", + "closure %ps has guard bits set: %x (%u), delta %i", cl->fn, - new, (unsigned) __fls(new & CLOSURE_GUARD_MASK))) + new, (unsigned) __fls(new & CLOSURE_GUARD_MASK), d)) new &=3D ~CLOSURE_GUARD_MASK; =20 WARN(!count && (new & ~CLOSURE_DESTRUCTOR), @@ -29,49 +29,54 @@ static void closure_val_checks(struct closure *cl, unsi= gned new) new, (unsigned) __fls(new)); } =20 -static inline void closure_put_after_sub(struct closure *cl, int flags) -{ - closure_val_checks(cl, flags); +enum new_closure_state { + CLOSURE_normal_put, + CLOSURE_requeue, + CLOSURE_done, +}; =20 - if (!(flags & CLOSURE_REMAINING_MASK)) { - smp_acquire__after_ctrl_dep(); +/* For clearing flags with the same atomic op as a put */ +void closure_sub(struct closure *cl, int v) +{ + enum new_closure_state s; =20 - cl->closure_get_happened =3D false; + int old =3D atomic_read(&cl->remaining), new; + do { + new =3D old - v; =20 - if (cl->fn && !(flags & CLOSURE_DESTRUCTOR)) { - atomic_set(&cl->remaining, - CLOSURE_REMAINING_INITIALIZER); - closure_queue(cl); + if (new & CLOSURE_REMAINING_MASK) { + s =3D CLOSURE_normal_put; } else { - struct closure *parent =3D cl->parent; - closure_fn *destructor =3D cl->fn; + if (cl->fn && !(new & CLOSURE_DESTRUCTOR)) { + s =3D CLOSURE_requeue; + new +=3D CLOSURE_REMAINING_INITIALIZER; + } else + s =3D CLOSURE_done; + } =20 - closure_debug_destroy(cl); + closure_val_checks(cl, new, -v); + } while (!atomic_try_cmpxchg_release(&cl->remaining, &old, new)); =20 - if (destructor) - destructor(&cl->work); + if (s =3D=3D CLOSURE_normal_put) + return; =20 - if (parent) - closure_put(parent); - } - } -} + if (s =3D=3D CLOSURE_requeue) { + cl->closure_get_happened =3D false; + closure_queue(cl); + } else { + struct closure *parent =3D cl->parent; + closure_fn *destructor =3D cl->fn; =20 -/* For clearing flags with the same atomic op as a put */ -void closure_sub(struct closure *cl, int v) -{ - closure_put_after_sub(cl, atomic_sub_return_release(v, &cl->remaining)); -} -EXPORT_SYMBOL(closure_sub); + closure_debug_destroy(cl); =20 -/* - * closure_put - decrement a closure's refcount - */ -void closure_put(struct closure *cl) -{ - closure_put_after_sub(cl, atomic_dec_return_release(&cl->remaining)); + if (destructor) + destructor(&cl->work); + + if (parent) + closure_put(parent); + } } -EXPORT_SYMBOL(closure_put); +EXPORT_SYMBOL(closure_sub); =20 /* * closure_wake_up - wake up all closures on a wait list, without memory b= arrier @@ -169,7 +174,7 @@ void __sched closure_return_sync(struct closure *cl) unsigned flags =3D atomic_sub_return_release(1 + CLOSURE_RUNNING - CLOSUR= E_DESTRUCTOR, &cl->remaining); =20 - closure_val_checks(cl, flags); + closure_val_checks(cl, flags, 1 + CLOSURE_RUNNING - CLOSURE_DESTRUCTOR); =20 if (unlikely(flags & CLOSURE_REMAINING_MASK)) { while (1) { --=20 2.51.0 From nobody Wed Dec 17 19:18:24 2025 Received: from out-186.mta1.migadu.com (out-186.mta1.migadu.com [95.215.58.186]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 31FF6235041 for ; Sun, 12 Oct 2025 21:24:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.186 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760304270; cv=none; b=BKi1joPoBDHcaCqApNr7W0mFPY47TMTdL1hDfBd1yG7D3MqpWiYAQAQx85d7PNWwpfkGzDK1F1b7HCkEQMITmQEXREOHhmJZgsKFthdx7MH3saYh55UHlCWSpfI2EznvGGx/kFxRK94inXU/F7JNsmuqlZNaM4uLGhC8Z+irT/g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760304270; c=relaxed/simple; bh=3FHL8eEuvGz/X2WWUy2G3HLnzD6gSjidjDrX08OPteg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CVzZelN3kVtWJOmRMKjquSKwxvIyj3s/HoyxCTCzpfHMqIkXNsxLA8QmzeQe/+mRqjZm7RJUGDt2KijQJJXWNXq9UYGZcwdRTiTa8V5AzTWnKs7xMTk30qjoGSGH6DEHtEoCt0iqNzkIzT/PW92YeuMJsYTr4whNlBXHLyT6iXU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=jGF/byNR; arc=none smtp.client-ip=95.215.58.186 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="jGF/byNR" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1760304266; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ruc5ohiAFaPupk6sasT1+CEuUEGK2Z9wdZI/6H5//wk=; b=jGF/byNRiAB1RuuKPNtPa0YRSHYEjKHyOgcJ1lta5yPs1NjayGk60XF7RmsQrjAuRqVbgG KIbn+t6YqXlXmzrQzQdbL9iLcPIRUOvIqOafY37EVlzXe+ZzthcIY+QS1bJshZ4UTdLwKo Bkc+f5XtrRs5cfaMDOlV50tQ5BhXa2s= From: Kent Overstreet To: akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, Kent Overstreet Subject: [PATCH v2 3/8] closures: CLOSURE_SLEEPING Date: Sun, 12 Oct 2025 17:24:06 -0400 Message-ID: <20251012212414.3225948-4-kent.overstreet@linux.dev> In-Reply-To: <20251012212414.3225948-1-kent.overstreet@linux.dev> References: <20251012212414.3225948-1-kent.overstreet@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" Part two of fixing __closure_sync_timeout(). Add a bit in cl->remaining (protected by cmpxchg()) for indicating whether a closure has a sleeper that should be woken up after the final put; we can now read the pointer to the waiting task_struct before the final put This elimating accesses to another thread's closure after the final put, which are racy because it may be unwinding after the timeout elapses. Signed-off-by: Kent Overstreet --- include/linux/closure.h | 9 +-- lib/closure.c | 118 +++++++++++++++++++--------------------- 2 files changed, 62 insertions(+), 65 deletions(-) diff --git a/include/linux/closure.h b/include/linux/closure.h index f626044d6ca2..d0195570514a 100644 --- a/include/linux/closure.h +++ b/include/linux/closure.h @@ -128,14 +128,15 @@ enum closure_state { * annotate where references are being transferred. */ =20 - CLOSURE_BITS_START =3D (1U << 26), - CLOSURE_DESTRUCTOR =3D (1U << 26), + CLOSURE_BITS_START =3D (1U << 24), + CLOSURE_DESTRUCTOR =3D (1U << 24), + CLOSURE_SLEEPING =3D (1U << 26), CLOSURE_WAITING =3D (1U << 28), CLOSURE_RUNNING =3D (1U << 30), }; =20 #define CLOSURE_GUARD_MASK \ - (((CLOSURE_DESTRUCTOR|CLOSURE_WAITING|CLOSURE_RUNNING) << 1)|(CLOSURE_BIT= S_START >> 1)) + (((CLOSURE_DESTRUCTOR|CLOSURE_SLEEPING|CLOSURE_WAITING|CLOSURE_RUNNING) <= < 1)|(CLOSURE_BITS_START >> 1)) =20 #define CLOSURE_REMAINING_MASK (CLOSURE_BITS_START - 1) #define CLOSURE_REMAINING_INITIALIZER (1|CLOSURE_RUNNING) @@ -144,7 +145,7 @@ struct closure { union { struct { struct workqueue_struct *wq; - struct closure_syncer *s; + struct task_struct *sleeper; struct llist_node list; closure_fn *fn; }; diff --git a/lib/closure.c b/lib/closure.c index 21fadd12093c..c49d49916788 100644 --- a/lib/closure.c +++ b/lib/closure.c @@ -23,7 +23,7 @@ static void closure_val_checks(struct closure *cl, unsign= ed new, int d) new, (unsigned) __fls(new & CLOSURE_GUARD_MASK), d)) new &=3D ~CLOSURE_GUARD_MASK; =20 - WARN(!count && (new & ~CLOSURE_DESTRUCTOR), + WARN(!count && (new & ~(CLOSURE_DESTRUCTOR|CLOSURE_SLEEPING)), "closure %ps ref hit 0 with incorrect flags set: %x (%u)", cl->fn, new, (unsigned) __fls(new)); @@ -39,19 +39,27 @@ enum new_closure_state { void closure_sub(struct closure *cl, int v) { enum new_closure_state s; + struct task_struct *sleeper; =20 - int old =3D atomic_read(&cl->remaining), new; + /* rcu_read_lock, atomic_read_acquire() are both for cl->sleeper: */ + guard(rcu)(); + + int old =3D atomic_read_acquire(&cl->remaining), new; do { new =3D old - v; =20 if (new & CLOSURE_REMAINING_MASK) { s =3D CLOSURE_normal_put; } else { - if (cl->fn && !(new & CLOSURE_DESTRUCTOR)) { + if ((cl->fn || (new & CLOSURE_SLEEPING)) && + !(new & CLOSURE_DESTRUCTOR)) { s =3D CLOSURE_requeue; new +=3D CLOSURE_REMAINING_INITIALIZER; } else s =3D CLOSURE_done; + + sleeper =3D new & CLOSURE_SLEEPING ? cl->sleeper : NULL; + new &=3D ~CLOSURE_SLEEPING; } =20 closure_val_checks(cl, new, -v); @@ -60,6 +68,12 @@ void closure_sub(struct closure *cl, int v) if (s =3D=3D CLOSURE_normal_put) return; =20 + if (sleeper) { + smp_mb(); + wake_up_process(sleeper); + return; + } + if (s =3D=3D CLOSURE_requeue) { cl->closure_get_happened =3D false; closure_queue(cl); @@ -114,41 +128,25 @@ bool closure_wait(struct closure_waitlist *waitlist, = struct closure *cl) =20 cl->closure_get_happened =3D true; closure_set_waiting(cl, _RET_IP_); - atomic_add(CLOSURE_WAITING + 1, &cl->remaining); + unsigned r =3D atomic_add_return(CLOSURE_WAITING + 1, &cl->remaining); + closure_val_checks(cl, r, CLOSURE_WAITING + 1); + llist_add(&cl->list, &waitlist->list); =20 return true; } EXPORT_SYMBOL(closure_wait); =20 -struct closure_syncer { - struct task_struct *task; - int done; -}; - -static CLOSURE_CALLBACK(closure_sync_fn) -{ - struct closure *cl =3D container_of(ws, struct closure, work); - struct closure_syncer *s =3D cl->s; - struct task_struct *p; - - rcu_read_lock(); - p =3D READ_ONCE(s->task); - s->done =3D 1; - wake_up_process(p); - rcu_read_unlock(); -} - void __sched __closure_sync(struct closure *cl) { - struct closure_syncer s =3D { .task =3D current }; - - cl->s =3D &s; - continue_at(cl, closure_sync_fn, NULL); + cl->sleeper =3D current; + closure_sub(cl, + CLOSURE_REMAINING_INITIALIZER - + CLOSURE_SLEEPING); =20 while (1) { set_current_state(TASK_UNINTERRUPTIBLE); - if (s.done) + if (!(atomic_read(&cl->remaining) & CLOSURE_SLEEPING)) break; schedule(); } @@ -162,31 +160,25 @@ EXPORT_SYMBOL(__closure_sync); * for outstanding get()s to finish) and returning once closure refcount i= s 0. * * Unlike closure_sync() this doesn't reinit the ref to 1; subsequent - * closure_get_not_zero() calls waill fail. + * closure_get_not_zero() calls will fail. */ void __sched closure_return_sync(struct closure *cl) { - struct closure_syncer s =3D { .task =3D current }; - - cl->s =3D &s; - set_closure_fn(cl, closure_sync_fn, NULL); - - unsigned flags =3D atomic_sub_return_release(1 + CLOSURE_RUNNING - CLOSUR= E_DESTRUCTOR, - &cl->remaining); + cl->sleeper =3D current; + closure_sub(cl, + CLOSURE_REMAINING_INITIALIZER - + CLOSURE_DESTRUCTOR - + CLOSURE_SLEEPING); =20 - closure_val_checks(cl, flags, 1 + CLOSURE_RUNNING - CLOSURE_DESTRUCTOR); - - if (unlikely(flags & CLOSURE_REMAINING_MASK)) { - while (1) { - set_current_state(TASK_UNINTERRUPTIBLE); - if (s.done) - break; - schedule(); - } - - __set_current_state(TASK_RUNNING); + while (1) { + set_current_state(TASK_UNINTERRUPTIBLE); + if (!(atomic_read(&cl->remaining) & CLOSURE_SLEEPING)) + break; + schedule(); } =20 + __set_current_state(TASK_RUNNING); + if (cl->parent) closure_put(cl->parent); } @@ -194,31 +186,35 @@ EXPORT_SYMBOL(closure_return_sync); =20 int __sched __closure_sync_timeout(struct closure *cl, unsigned long timeo= ut) { - struct closure_syncer s =3D { .task =3D current }; int ret =3D 0; =20 - cl->s =3D &s; - continue_at(cl, closure_sync_fn, NULL); + cl->sleeper =3D current; + closure_sub(cl, + CLOSURE_REMAINING_INITIALIZER - + CLOSURE_SLEEPING); =20 while (1) { set_current_state(TASK_UNINTERRUPTIBLE); - if (s.done) - break; + /* + * Carefully undo the continue_at() - but only if it + * hasn't completed, i.e. the final closure_put() hasn't + * happened yet: + */ + unsigned old =3D atomic_read(&cl->remaining), new; + if (!(old & CLOSURE_SLEEPING)) + goto success; + if (!timeout) { - /* - * Carefully undo the continue_at() - but only if it - * hasn't completed, i.e. the final closure_put() hasn't - * happened yet: - */ - unsigned old, new, v =3D atomic_read(&cl->remaining); do { - old =3D v; - if (!old || (old & CLOSURE_RUNNING)) + if (!(old & CLOSURE_SLEEPING)) goto success; =20 - new =3D old + CLOSURE_REMAINING_INITIALIZER; - } while ((v =3D atomic_cmpxchg(&cl->remaining, old, new)) !=3D old); + new =3D old + CLOSURE_REMAINING_INITIALIZER - CLOSURE_SLEEPING; + closure_val_checks(cl, new, CLOSURE_REMAINING_INITIALIZER - CLOSURE_SL= EEPING); + } while (!atomic_try_cmpxchg(&cl->remaining, &old, new)); + ret =3D -ETIME; + break; } =20 timeout =3D schedule_timeout(timeout); --=20 2.51.0 From nobody Wed Dec 17 19:18:24 2025 Received: from out-183.mta1.migadu.com (out-183.mta1.migadu.com [95.215.58.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 01095235063 for ; Sun, 12 Oct 2025 21:24:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.183 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760304271; cv=none; b=niVHM8E8FEKp/6sMd8V6YPOOyCQYq2we11sEKwG+rYdTDDGNu0S/h/I5W8xF3QJFAAHvqlADNSFE8Ew255ETOrRA0tKbU4TNWLqmnweDU/JwkAkjVeQKGzJI345WcWNOlhS70fropPv5LT8t31HA12N8RxPecXK9oEYlp4U49Ic= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760304271; c=relaxed/simple; bh=5Aab4a69dI91R6PuKaJ9VuWaPXFyB6Ov8v4M3CyBg2c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Z8qVhqEEIiMp0Onz7ypX1BahjXwQtWhHP9qlTzsV53KqJTqV2UN1CqnMOkCfzBL03YeqgqK7wKMO5y+bTUISN94z2FEi59NqrRWXt3dqHR+I4wMT5E/KdJ8u68atzHivAB95314sLRCXx9Z1mZoBclGZj7ngPT7rXOiRMy8CJmA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=CLnPvS/x; arc=none smtp.client-ip=95.215.58.183 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="CLnPvS/x" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1760304267; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5l/Nb9kVvBLievHWD8SolIsSd40A68uARxnurr3QvOA=; b=CLnPvS/xiPfXZlMj7iW1BS00IcDFqkerG/04r0bRkQNl0VZ+VyygBiuK8IjYuD9rVEsJ9a wLKYlphKRxm9J+cIvG5FRNXUwZ4KPFpXJELRKbEa+uXh526OQrb+dixFhaIYHiES2J6aBx sGKaNqg5vSXRtTOOFhWTo7Wig3zoA9k= From: Kent Overstreet To: akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, Kent Overstreet Subject: [PATCH v2 4/8] closures: kill closure.closure_get_happened Date: Sun, 12 Oct 2025 17:24:07 -0400 Message-ID: <20251012212414.3225948-5-kent.overstreet@linux.dev> In-Reply-To: <20251012212414.3225948-1-kent.overstreet@linux.dev> References: <20251012212414.3225948-1-kent.overstreet@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" With closure_put() now using cmpxchg, this is no longer needed. Signed-off-by: Kent Overstreet --- fs/bcachefs/alloc_foreground.h | 2 +- fs/bcachefs/fs-io-direct.c | 1 - include/linux/closure.h | 15 ++------------- lib/closure.c | 2 -- 4 files changed, 3 insertions(+), 17 deletions(-) diff --git a/fs/bcachefs/alloc_foreground.h b/fs/bcachefs/alloc_foreground.h index 1b3fc8460096..5a6f0dfe3df5 100644 --- a/fs/bcachefs/alloc_foreground.h +++ b/fs/bcachefs/alloc_foreground.h @@ -311,7 +311,7 @@ void bch2_dev_alloc_debug_to_text(struct printbuf *, st= ruct bch_dev *); void __bch2_wait_on_allocator(struct bch_fs *, struct closure *); static inline void bch2_wait_on_allocator(struct bch_fs *c, struct closure= *cl) { - if (cl->closure_get_happened) + if (closure_nr_remaining(cl) > 1) __bch2_wait_on_allocator(c, cl); } =20 diff --git a/fs/bcachefs/fs-io-direct.c b/fs/bcachefs/fs-io-direct.c index 1f5154d9676b..e306eba734a1 100644 --- a/fs/bcachefs/fs-io-direct.c +++ b/fs/bcachefs/fs-io-direct.c @@ -117,7 +117,6 @@ static int bch2_direct_IO_read(struct kiocb *req, struc= t iov_iter *iter) } else { atomic_set(&dio->cl.remaining, CLOSURE_REMAINING_INITIALIZER + 1); - dio->cl.closure_get_happened =3D true; } =20 dio->req =3D req; diff --git a/include/linux/closure.h b/include/linux/closure.h index d0195570514a..83a0dde389bc 100644 --- a/include/linux/closure.h +++ b/include/linux/closure.h @@ -155,7 +155,6 @@ struct closure { struct closure *parent; =20 atomic_t remaining; - bool closure_get_happened; =20 #ifdef CONFIG_DEBUG_CLOSURES #define CLOSURE_MAGIC_DEAD 0xc054dead @@ -195,11 +194,7 @@ static inline unsigned closure_nr_remaining(struct clo= sure *cl) */ static inline void closure_sync(struct closure *cl) { -#ifdef CONFIG_DEBUG_CLOSURES - BUG_ON(closure_nr_remaining(cl) !=3D 1 && !cl->closure_get_happened); -#endif - - if (cl->closure_get_happened) + if (closure_nr_remaining(cl) > 1) __closure_sync(cl); } =20 @@ -207,10 +202,7 @@ int __closure_sync_timeout(struct closure *cl, unsigne= d long timeout); =20 static inline int closure_sync_timeout(struct closure *cl, unsigned long t= imeout) { -#ifdef CONFIG_DEBUG_CLOSURES - BUG_ON(closure_nr_remaining(cl) !=3D 1 && !cl->closure_get_happened); -#endif - return cl->closure_get_happened + return closure_nr_remaining(cl) > 1 ? __closure_sync_timeout(cl, timeout) : 0; } @@ -283,8 +275,6 @@ static inline void closure_queue(struct closure *cl) */ static inline void closure_get(struct closure *cl) { - cl->closure_get_happened =3D true; - #ifdef CONFIG_DEBUG_CLOSURES BUG_ON((atomic_inc_return(&cl->remaining) & CLOSURE_REMAINING_MASK) <=3D 1); @@ -322,7 +312,6 @@ static inline void closure_init(struct closure *cl, str= uct closure *parent) closure_get(parent); =20 atomic_set(&cl->remaining, CLOSURE_REMAINING_INITIALIZER); - cl->closure_get_happened =3D false; =20 closure_debug_create(cl); closure_set_ip(cl); diff --git a/lib/closure.c b/lib/closure.c index c49d49916788..f1b4a797c9db 100644 --- a/lib/closure.c +++ b/lib/closure.c @@ -75,7 +75,6 @@ void closure_sub(struct closure *cl, int v) } =20 if (s =3D=3D CLOSURE_requeue) { - cl->closure_get_happened =3D false; closure_queue(cl); } else { struct closure *parent =3D cl->parent; @@ -126,7 +125,6 @@ bool closure_wait(struct closure_waitlist *waitlist, st= ruct closure *cl) if (atomic_read(&cl->remaining) & CLOSURE_WAITING) return false; =20 - cl->closure_get_happened =3D true; closure_set_waiting(cl, _RET_IP_); unsigned r =3D atomic_add_return(CLOSURE_WAITING + 1, &cl->remaining); closure_val_checks(cl, r, CLOSURE_WAITING + 1); --=20 2.51.0 From nobody Wed Dec 17 19:18:24 2025 Received: from out-177.mta1.migadu.com (out-177.mta1.migadu.com [95.215.58.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 066A223D7D3 for ; Sun, 12 Oct 2025 21:24:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760304271; cv=none; b=sPR6iNpVT+7tpZhKNB52c88Q9cAaO3ANcpuAPC6sqgG52onXIlsP+UN5Q/YN/SLNjyaCVEkwAPKJIJmGyXqC3KlpLrB/BJhOQSgQownwxsI5y+/h+tCFiniJBkzwli2azUIneIwqyDGmmhzC/NtCQKDn1tbic4/q7+0jcAa8Z5c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760304271; c=relaxed/simple; bh=QhB19pcOpxyDzk2Ur3Y2O3RcyjRlLnMNVwMCOcElCIE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eaK8vLMHgPnKiTGXepnz/exPJT1MV2Fdw+rezI+ohZUPIcfgoNFShvot1rF4azIgtpjjFhFXvCRZcLHA/GY1fOLtoUnME33WR9A8Oe8OmkhQ7OnlpHdsXjfS7O4yLNO8dyLkSVJtzxh44l+zsQxaLNZXyhx1jCObWVIbgSOElHg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=pJmfdGec; arc=none smtp.client-ip=95.215.58.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="pJmfdGec" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1760304268; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=E41ygSrpXGOBskATUwM6ql0sSacMMv3vaLKjgxUJMAk=; b=pJmfdGecKZf6Wk7rf2ac1CQj8dkuW6CpRkWsfQTbeinpqtdHQDCCzkCzqFEwGF2lHJIdMl XTzNYdpF2r6gUyBkKEtTtoWINHju4v9woxd61UoGEN3haV/NK9E7853aW6lwdasQ+uqlmt zoj79j4NKAFYS93u76OHPH9oeLVlDG0= From: Kent Overstreet To: akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, Kent Overstreet , Eric Biggers Subject: [PATCH v2 5/8] lib: Give closures, min_heap config opts names Date: Sun, 12 Oct 2025 17:24:08 -0400 Message-ID: <20251012212414.3225948-6-kent.overstreet@linux.dev> In-Reply-To: <20251012212414.3225948-1-kent.overstreet@linux.dev> References: <20251012212414.3225948-1-kent.overstreet@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" Give these config options names so that they show up under the "Library routines" kernel configuration menu, and can be enabled by distributions. These are needed for bcachefs to be built out of tree. The closures and min_heap code originates from bcache and bcachefs. Cc: Andrew Morton Cc: Eric Biggers Signed-off-by: Kent Overstreet --- lib/Kconfig | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/Kconfig b/lib/Kconfig index c483951b624f..badcb5ca9efd 100644 --- a/lib/Kconfig +++ b/lib/Kconfig @@ -367,7 +367,7 @@ config ASSOCIATIVE_ARRAY for more information. =20 config CLOSURES - bool + bool "bcache/bcachefs async widgets" =20 config HAS_IOMEM bool @@ -646,4 +646,4 @@ config UNION_FIND bool =20 config MIN_HEAP - bool + bool "Generic minheap algorithm" --=20 2.51.0 From nobody Wed Dec 17 19:18:24 2025 Received: from out-188.mta1.migadu.com (out-188.mta1.migadu.com [95.215.58.188]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A57CE2417E0 for ; Sun, 12 Oct 2025 21:24:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.188 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760304272; cv=none; b=bSg8SiOXniywT/E/fTRFSBjuBI+mtPvEYpCJsc0CGjh6/F785Bs5eNuBW8gbblWk0xKIbi6faSVpYJfWyB1SbXXGbttg9xw04IB27alipuNvUpgWgNaTXNaYpgx9bTamS7DD1q825TlfHOgzC8G8PeOnMNfypFLWpvPuB1zD9MA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760304272; c=relaxed/simple; bh=E+NwvqNKQPDoP9IOTvsJFY6yXIqEEdE6aOeCoA+Zpx8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sfQLfasqHYgF//vB0NRgI3tWr+hdhLHf3feXwgrHDw2t/DVpEPOt8KIMGpGb4sd9j5iXwJ4iheM+Oo/xhQzPjDsPo0Q3pccldmEXpHMRtPYSusXC236z2Fz8Yu1sGNRK11FRbG4We4PEo9Xz2IttPEmSjSqyQeEwDjkkTzKV4WI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=LV+xpeWm; arc=none smtp.client-ip=95.215.58.188 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="LV+xpeWm" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1760304269; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=skKjuSA1gxdZITVYhcLcHqL1gW6SF0W1jTkGYAWJENk=; b=LV+xpeWmDI8jczSpK7VvTtMCj5UK0oURcm211GlIOCd4gmUL6nryJ+bloY5oUraNjMnaxS R1H4rm4XSxex1Fij2H8rv1tUBpi7i+bSup4e8LnOHpeDJV/P9Hq7xcfTBRqNpi/8+9MvRD 9gy0S2OojoxesRDXQww7GPyvSNDePWM= From: Kent Overstreet To: akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, Kent Overstreet , NeilBrown , Eric Biggers Subject: [PATCH v2 6/8] lib: Give XOR_BLOCKS, RAID6_PQ config opts names Date: Sun, 12 Oct 2025 17:24:09 -0400 Message-ID: <20251012212414.3225948-7-kent.overstreet@linux.dev> In-Reply-To: <20251012212414.3225948-1-kent.overstreet@linux.dev> References: <20251012212414.3225948-1-kent.overstreet@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" Give these config options names so that they show up under the "Library routines" kernel configuration menu, and can be enabled by distributions. These libraries are needed for bcachefs to be built out of tree. Per Neal - "the proposed change seems appropriate in that it makes our code more available and hence more useful. It has no apparent cost and I can (now) see no better way to achieve a comparable outcome." These libraries are both for RAID5/6. Reviewed-By: NeilBrown Cc: Andrew Morton Cc: Eric Biggers Signed-off-by: Kent Overstreet --- crypto/Kconfig | 2 +- lib/Kconfig | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/crypto/Kconfig b/crypto/Kconfig index 23bd98981ae8..da4f072abae0 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -3,7 +3,7 @@ # Generic algorithms support # config XOR_BLOCKS - tristate + tristate "Accelerated block xor algorithm" =20 # # async_tx api: hardware offloaded memory transfer/transform support diff --git a/lib/Kconfig b/lib/Kconfig index badcb5ca9efd..e831f4462453 100644 --- a/lib/Kconfig +++ b/lib/Kconfig @@ -9,7 +9,9 @@ config BINARY_PRINTF menu "Library routines" =20 config RAID6_PQ - tristate + tristate "Reed-solomon RAID5/6 algorithms" + help + Provides routines for block level reed-solomon, for RAID5/6 =20 config RAID6_PQ_BENCHMARK bool "Automatically choose fastest RAID6 PQ functions" --=20 2.51.0 From nobody Wed Dec 17 19:18:24 2025 Received: from out-181.mta1.migadu.com (out-181.mta1.migadu.com [95.215.58.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C1EBD2475CF for ; Sun, 12 Oct 2025 21:24:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760304273; cv=none; b=WBFYIsYjUHlkl6A8+ZVc+xWgCZYN+Q1EqUWiCHnpbVonwDYuzP4eyWafwN3JwSwlDYH9+p843tiA7YgONk0qPkQWVdOXuG0h1n6kdixq2SUzbdhScHCS139BD5IC4jVdhOQ+rHxyP4owmjrNkr/jXj6rt6jtgSHyNYAJ2Ddq4fI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760304273; c=relaxed/simple; bh=38ZWKBcbJoo+ZpVwpCcfo4cyVHQ8oMbojN7hMUXTONw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WZySWoWx2azKfpjTuEm1VkJpOJYG7RLEOZlyS81WDZuwXJ4cOutikHLUCoG86Ju0Lu+gIBY9auFaMKewlYq+u//Q78z+/7a18gZERCH8LIN4aOr3mZf8lTJ8DmcDg8ltGe3Dwr7KtKKPW3aydmQd8bHyGc0Ft1Gf9EZHXhPiluc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=UeBODTwZ; arc=none smtp.client-ip=95.215.58.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="UeBODTwZ" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1760304269; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pSG5sV4PEZzE+jj0CTiGJHwsg0H0GUSCrU0VHyAplac=; b=UeBODTwZVseezxGg25DGY2YUKxK8WkalrTH/gGDj1KB1k4gVX11hoadKMlVH1vKrWap21j YUWWaNPFB7o0JflX2EaymhNIzMKCUEeuJvwm0KcFXdxBBiIHgxaPhIl1at+Q20NyGchMI0 N7i1k9h9HWN0Tu4gEZJaeOwNcwvUwtk= From: Kent Overstreet To: akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, Kent Overstreet , Eric Biggers Subject: [PATCH v2 7/8] lib: Give compression, checksum, crypto config opts names Date: Sun, 12 Oct 2025 17:24:10 -0400 Message-ID: <20251012212414.3225948-8-kent.overstreet@linux.dev> In-Reply-To: <20251012212414.3225948-1-kent.overstreet@linux.dev> References: <20251012212414.3225948-1-kent.overstreet@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" Give these config options names so that they show up under the "Library routines" kernel configuration menu, and can be enabled by distributions. These are needed for bcachefs to be built out of tree. Most (but not all, notably CRC64) of these libraries have crypto API wrappers which are already selectable under the "Cryptographic API" configuration menu; however, Eric Biggers has been working to reduce usage of the cryptographic API in favor of much more lightweight direct APIs. Cc: Andrew Morton Cc: Eric Biggers Signed-off-by: Kent Overstreet --- lib/Kconfig | 16 ++++++++-------- lib/crc/Kconfig | 4 ++-- lib/crypto/Kconfig | 4 ++-- 3 files changed, 12 insertions(+), 12 deletions(-) diff --git a/lib/Kconfig b/lib/Kconfig index e831f4462453..eb3b6b96b303 100644 --- a/lib/Kconfig +++ b/lib/Kconfig @@ -142,7 +142,7 @@ source "lib/crc/Kconfig" source "lib/crypto/Kconfig" =20 config XXHASH - tristate + tristate "XXHASH checksum algorithm" =20 config AUDIT_GENERIC bool @@ -176,10 +176,10 @@ config 842_DECOMPRESS tristate =20 config ZLIB_INFLATE - tristate + tristate "Zlib decompression" =20 config ZLIB_DEFLATE - tristate + tristate "Zlib compression" select BITREVERSE =20 config ZLIB_DFLTCC @@ -196,13 +196,13 @@ config LZO_DECOMPRESS tristate =20 config LZ4_COMPRESS - tristate + tristate "LZ4 compression" =20 config LZ4HC_COMPRESS - tristate + tristate "LZ4HC compression" =20 config LZ4_DECOMPRESS - tristate + tristate "LZC decompression" =20 config ZSTD_COMMON select XXHASH @@ -210,11 +210,11 @@ config ZSTD_COMMON =20 config ZSTD_COMPRESS select ZSTD_COMMON - tristate + tristate "ZSTD compression" =20 config ZSTD_DECOMPRESS select ZSTD_COMMON - tristate + tristate "ZSTD decompression" =20 source "lib/xz/Kconfig" =20 diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig index 70e7a6016de3..3a3d6290211b 100644 --- a/lib/crc/Kconfig +++ b/lib/crc/Kconfig @@ -54,7 +54,7 @@ config CRC_T10DIF_ARCH default y if X86 =20 config CRC32 - tristate + tristate "CRC32 algorithm" select BITREVERSE help The CRC32 library functions. Select this if your module uses any of @@ -74,7 +74,7 @@ config CRC32_ARCH default y if X86 =20 config CRC64 - tristate + tristate "CRC64 algorithm" help The CRC64 library functions. Select this if your module uses any of the functions from . diff --git a/lib/crypto/Kconfig b/lib/crypto/Kconfig index 1e6b008f8fca..51e4e8d3b793 100644 --- a/lib/crypto/Kconfig +++ b/lib/crypto/Kconfig @@ -62,7 +62,7 @@ config CRYPTO_LIB_CHACHA_GENERIC enabled, this implementation serves the users of CRYPTO_LIB_CHACHA. =20 config CRYPTO_LIB_CHACHA - tristate + tristate "Chacha encryption algorithm" help Enable the ChaCha library interface. This interface may be fulfilled by either the generic implementation or an arch-specific one, if one @@ -125,7 +125,7 @@ config CRYPTO_LIB_POLY1305_GENERIC enabled, this implementation serves the users of CRYPTO_LIB_POLY1305. =20 config CRYPTO_LIB_POLY1305 - tristate + tristate "Poly1305 authentication algorithm" help Enable the Poly1305 library interface. This interface may be fulfilled by either the generic implementation or an arch-specific one, if one --=20 2.51.0 From nobody Wed Dec 17 19:18:24 2025 Received: from out-173.mta1.migadu.com (out-173.mta1.migadu.com [95.215.58.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4F34624E4D4 for ; Sun, 12 Oct 2025 21:24:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760304274; cv=none; b=YRk6kNYXEaBx+Xjo+WKO/tCbCSg9Z125saKfeANPSlvxE9XEq0gQ5ll3PHxTUbWfjgHxyIOvsEIxuNVyJhfJq2ewKyzCKMVQZORGUQmwTxRzpYVOiwqVF4t3W5H6ylvZV4Mm90TkdSvHngy8k0qXEI19DSXXtNCZn2hXiMycccM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760304274; c=relaxed/simple; bh=kZ2kxPL0DwQa6FaA47JtKYNl7J+Rv40MhInzKCJkN7c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=i3e0rR8Pfda40HHNLsXmnr9n4P1J3nvGHfb3Rc9ZAZnMPMVG1t6ocd3XP+azXAVFripcWQq/F3x50k8LXRfDfu4yAAD/q47CIfX7h8Dd4PRZ7s4qDF2RiT6ol+E9OE7XbUE/lTF41AWP7vr5MV9BntkPN8QOYcjYjnZfOIpVCF0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=v1T6rivl; arc=none smtp.client-ip=95.215.58.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="v1T6rivl" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1760304270; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JbmcSlWYwAKTQNHjQETLXgL9nJD2V6VdefvVxWJt/kY=; b=v1T6rivlNK6uKJv5b6AWznx0aqWs34w8q4MBH7gTxte1s5xikFXl4mVHQsC7bCibi4i1FO NXGBxiiL/GC2omdTp2ORO6btAk6fTDjXQ2G+SZdTzcYYBeM+fR+voZVzrz1FG4z1xnJFsw YjsPAJDyX2A1avXSyaVfO4y5lRght2c= From: Kent Overstreet To: akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, Kent Overstreet Subject: [PATCH v2 8/8] generix-radix-tree: Overflow checking Date: Sun, 12 Oct 2025 17:24:11 -0400 Message-ID: <20251012212414.3225948-9-kent.overstreet@linux.dev> In-Reply-To: <20251012212414.3225948-1-kent.overstreet@linux.dev> References: <20251012212414.3225948-1-kent.overstreet@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" Internally, genradixes index based on the byte offset into the radix tree - index * obj_size - which means accidental overflows are a real concern. This was noticed in bcachefs where we were using them to index inode numbers, for hardlink checking/detection. Signed-off-by: Kent Overstreet --- include/linux/generic-radix-tree.h | 110 +++++++++++++++++------------ 1 file changed, 65 insertions(+), 45 deletions(-) diff --git a/include/linux/generic-radix-tree.h b/include/linux/generic-rad= ix-tree.h index 5b51c3d582d6..52b012efe8f4 100644 --- a/include/linux/generic-radix-tree.h +++ b/include/linux/generic-radix-tree.h @@ -155,20 +155,27 @@ void __genradix_free(struct __genradix *); */ #define genradix_free(_radix) __genradix_free(&(_radix)->tree) =20 -static inline size_t __idx_to_offset(size_t idx, size_t obj_size) +static inline bool __idx_to_offset(size_t idx, size_t obj_size, size_t *of= fset) { + /* + * XXX: check for overflow, we have a bug in multiple places where we + * assume idx fits in 64 bits (because it's an inode number; already a + * bug on 32 bit) - but we then multiply by the object size and we + * overflow + */ if (__builtin_constant_p(obj_size)) BUILD_BUG_ON(obj_size > GENRADIX_NODE_SIZE); else BUG_ON(obj_size > GENRADIX_NODE_SIZE); =20 if (!is_power_of_2(obj_size)) { - size_t objs_per_page =3D GENRADIX_NODE_SIZE / obj_size; + size_t objs_per_page =3D GENRADIX_NODE_SIZE / obj_size, node, node_offse= t; =20 - return (idx / objs_per_page) * GENRADIX_NODE_SIZE + - (idx % objs_per_page) * obj_size; + return check_mul_overflow(idx / objs_per_page, GENRADIX_NODE_SIZE, &nod= e) ? true : + check_mul_overflow(idx % objs_per_page, obj_size, &node_offset) ? true : + check_add_overflow(node, node_offset, offset); } else { - return idx * obj_size; + return check_mul_overflow(idx, obj_size, offset); } } =20 @@ -179,8 +186,8 @@ static inline size_t __idx_to_offset(size_t idx, size_t= obj_size) #define __genradix_page_remainder(_radix) \ (GENRADIX_NODE_SIZE % sizeof((_radix)->type[0])) =20 -#define __genradix_idx_to_offset(_radix, _idx) \ - __idx_to_offset(_idx, __genradix_obj_size(_radix)) +#define __genradix_idx_to_offset(_radix, _idx, _offset) \ + __idx_to_offset(_idx, __genradix_obj_size(_radix), _offset) =20 static inline void *__genradix_ptr_inlined(struct __genradix *radix, size_= t offset) { @@ -202,9 +209,13 @@ static inline void *__genradix_ptr_inlined(struct __ge= nradix *radix, size_t offs } =20 #define genradix_ptr_inlined(_radix, _idx) \ - (__genradix_cast(_radix) \ - __genradix_ptr_inlined(&(_radix)->tree, \ - __genradix_idx_to_offset(_radix, _idx))) +({ \ + size_t _offset; \ + !__genradix_idx_to_offset(_radix, _idx, &_offset) \ + ? __genradix_cast(_radix) \ + __genradix_ptr_inlined(&(_radix)->tree, _offset) \ + : NULL; \ +}) =20 void *__genradix_ptr(struct __genradix *, size_t); =20 @@ -216,28 +227,39 @@ void *__genradix_ptr(struct __genradix *, size_t); * Returns a pointer to entry at @_idx, or NULL if that entry does not exi= st. */ #define genradix_ptr(_radix, _idx) \ - (__genradix_cast(_radix) \ - __genradix_ptr(&(_radix)->tree, \ - __genradix_idx_to_offset(_radix, _idx))) +({ \ + size_t _offset; \ + !__genradix_idx_to_offset(_radix, _idx, &_offset) \ + ? __genradix_cast(_radix) \ + __genradix_ptr(&(_radix)->tree, _offset) \ + : NULL; \ +}) =20 void *__genradix_ptr_alloc(struct __genradix *, size_t, struct genradix_node **, gfp_t); =20 -#define genradix_ptr_alloc_inlined(_radix, _idx, _gfp) \ - (__genradix_cast(_radix) \ - (__genradix_ptr_inlined(&(_radix)->tree, \ - __genradix_idx_to_offset(_radix, _idx)) ?: \ - __genradix_ptr_alloc(&(_radix)->tree, \ - __genradix_idx_to_offset(_radix, _idx), \ - NULL, _gfp))) - #define genradix_ptr_alloc_preallocated_inlined(_radix, _idx, _new_node, _= gfp)\ - (__genradix_cast(_radix) \ - (__genradix_ptr_inlined(&(_radix)->tree, \ - __genradix_idx_to_offset(_radix, _idx)) ?: \ - __genradix_ptr_alloc(&(_radix)->tree, \ - __genradix_idx_to_offset(_radix, _idx), \ - _new_node, _gfp))) +({ \ + size_t _offset; \ + !__genradix_idx_to_offset(_radix, _idx, &_offset) \ + ? __genradix_cast(_radix) \ + (__genradix_ptr_inlined(&(_radix)->tree, _offset) ?: \ + __genradix_ptr_alloc(&(_radix)->tree, _offset, \ + _new_node, _gfp)) \ + : NULL; \ +}) + +#define genradix_ptr_alloc_inlined(_radix, _idx, _gfp) \ + genradix_ptr_alloc_preallocated_inlined(_radix, _idx, NULL, _gfp) + +#define genradix_ptr_alloc_preallocated(_radix, _idx, _new_node, _gfp) \ +({ \ + size_t _offset; \ + !__genradix_idx_to_offset(_radix, _idx, &_offset) \ + ? __genradix_cast(_radix) \ + __genradix_ptr_alloc(&(_radix)->tree, _offset, _new_node, _gfp) \ + : NULL; \ +}) =20 /** * genradix_ptr_alloc - get a pointer to a genradix entry, allocating it @@ -248,17 +270,8 @@ void *__genradix_ptr_alloc(struct __genradix *, size_t, * * Returns a pointer to entry at @_idx, or NULL on allocation failure */ -#define genradix_ptr_alloc(_radix, _idx, _gfp) \ - (__genradix_cast(_radix) \ - __genradix_ptr_alloc(&(_radix)->tree, \ - __genradix_idx_to_offset(_radix, _idx), \ - NULL, _gfp)) - -#define genradix_ptr_alloc_preallocated(_radix, _idx, _new_node, _gfp)\ - (__genradix_cast(_radix) \ - __genradix_ptr_alloc(&(_radix)->tree, \ - __genradix_idx_to_offset(_radix, _idx), \ - _new_node, _gfp)) +#define genradix_ptr_alloc(_radix, _idx, _gfp) \ + genradix_ptr_alloc_preallocated(_radix, _idx, NULL, _gfp) =20 struct genradix_iter { size_t offset; @@ -271,10 +284,15 @@ struct genradix_iter { * @_idx: index to start iterating from */ #define genradix_iter_init(_radix, _idx) \ - ((struct genradix_iter) { \ +({ \ + size_t _offset; \ + __genradix_idx_to_offset(_radix, _idx, &_offset); \ + \ + (struct genradix_iter) { \ .pos =3D (_idx), \ - .offset =3D __genradix_idx_to_offset((_radix), (_idx)),\ - }) + .offset =3D _offset, \ + }; \ +}) =20 void *__genradix_iter_peek(struct genradix_iter *, struct __genradix *, si= ze_t); =20 @@ -394,9 +412,11 @@ int __genradix_prealloc(struct __genradix *, size_t, g= fp_t); * Returns 0 on success, -ENOMEM on failure */ #define genradix_prealloc(_radix, _nr, _gfp) \ - __genradix_prealloc(&(_radix)->tree, \ - __genradix_idx_to_offset(_radix, _nr + 1),\ - _gfp) - +({ \ + size_t _offset; \ + !__genradix_idx_to_offset(_radix, _nr + 1, &_offset) \ + ? __genradix_prealloc(&(_radix)->tree, _offset, _gfp) \ + : -ENOMEM; \ +}) =20 #endif /* _LINUX_GENERIC_RADIX_TREE_H */ --=20 2.51.0