From nobody Fri Apr 17 21:11:09 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A60FC433EF for ; Fri, 22 Jul 2022 09:44:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234985AbiGVJox (ORCPT ); Fri, 22 Jul 2022 05:44:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42024 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229839AbiGVJo2 (ORCPT ); Fri, 22 Jul 2022 05:44:28 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D6E97BB8C5 for ; Fri, 22 Jul 2022 02:38:50 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-31e63e48e49so35224497b3.5 for ; Fri, 22 Jul 2022 02:38:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=EdBwhDB1kTfOuaZ44XnXqAFiApYJfH6lT/hr3SDnJ/w=; b=gPVLoOuKJt62FkkLy8/2yUn70N9cMwMg0NqDjsY+Frsa95mUBpYG68nMSa0FlEBDLd iXKK/rDCH1qzi2geeyk73a+raqj9Q36BvBNH5xp9fCWSLHVCibioJcnlTZYIVxMxzXqq IeqUfW1fM+/ypF6f2quRJdn2sMscnROePSbn/Fy0lzhzJP/D2nE0ElTYzDXvx8W71FHg qCLcQ+YcJODw5bXSHcaC8KO0Yb8EsrlZpxc89qJizbABdwf05Qd39DbF6ASL7fb34Dxb n85N1RtO5VzPNQE/OrW1DFOTJPNg0KS8BiKXbnmtiM65ENnrtSb4791mDMmlMYgRYXMP 0jXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=EdBwhDB1kTfOuaZ44XnXqAFiApYJfH6lT/hr3SDnJ/w=; b=oYeW67a46D0pHcMMhLs2OuvKkq4PponEFp7PFog5rSdExKGzCD5aRALSoDRSAypyxn JVIVXHLFgXbeP7RqiiHFAA9onGtnwzBHvUpKUHuqs0fSxL3r58T5ufEb5Pb4YNPvLu1A DCZO54PFDo6lAYc8m96/GozlSVjf1aadAT8cxqmK9eMS15BqNpNoSX6QUAOlnCjTAYPc 5h8HC4s2oqcnOQqN85yynpwJHEmeE2iLJ85CrurKLuu3iYNwXQz+oHFdqoY3jHfvsHLY rd7L7hEHcs9shko2amzm7YedbgbmGvOaqpYZg8SqCCG/8H0AcIwOzFDhV0WZ1L5114fj Nqug== X-Gm-Message-State: AJIora9BM8xzmdkISKfbqCIVfz/iA1jm6raTA6YtumhxMSnAP5qaPOBj hY7JDDNaC2uZe6/l/P4byt2YTlP6I64afEMj7NDmNTZ+bDahwAl/bcJyUHIQSjB4rsmOTeq7b7q Fe7fJBq/vbASs6t79CXwjmKK5dsXLgE6F4Lv4DrkkJ5QmxywiHo3z1mZkBeDxC5VU9Qo= X-Google-Smtp-Source: AGRyM1v/0YsN7N1ZDNPpQQjFr5qD9ZaFHhp2KlBQW9DYS5Wzckg6JL2WrqE3KHHwbiqK6mFr34QSFCzL7g== X-Received: from nhuck.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:39cc]) (user=nhuck job=sendgmr) by 2002:a25:71d7:0:b0:670:403b:a74f with SMTP id m206-20020a2571d7000000b00670403ba74fmr2198395ybc.239.1658482729838; Fri, 22 Jul 2022 02:38:49 -0700 (PDT) Date: Fri, 22 Jul 2022 09:38:21 +0000 In-Reply-To: <20220722093823.4158756-1-nhuck@google.com> Message-Id: <20220722093823.4158756-2-nhuck@google.com> Mime-Version: 1.0 References: <20220722093823.4158756-1-nhuck@google.com> X-Mailer: git-send-email 2.37.1.359.gd136c6c3e2-goog Subject: [PATCH 1/3] dm-bufio: Add flags for dm_bufio_client_create From: Nathan Huckleberry To: linux-kernel@vger.kernel.org, dm-devel@redhat.com, Alasdair Kergon , Mike Snitzer Cc: Eric Biggers , Sami Tolvanen , Nathan Huckleberry Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a flags argument to dm_bufio_client_create and update all the callers. This is in preparation to add the DM_BUFIO_GET_CANT_SLEEP flag. Signed-off-by: Nathan Huckleberry --- drivers/md/dm-bufio.c | 3 ++- drivers/md/dm-ebs-target.c | 3 ++- drivers/md/dm-integrity.c | 2 +- drivers/md/dm-snap-persistent.c | 2 +- drivers/md/dm-verity-fec.c | 4 ++-- drivers/md/dm-verity-target.c | 2 +- drivers/md/persistent-data/dm-block-manager.c | 3 ++- include/linux/dm-bufio.h | 3 ++- 8 files changed, 13 insertions(+), 9 deletions(-) diff --git a/drivers/md/dm-bufio.c b/drivers/md/dm-bufio.c index 5ffa1dcf84cf..ad5603eb12e3 100644 --- a/drivers/md/dm-bufio.c +++ b/drivers/md/dm-bufio.c @@ -1717,7 +1717,8 @@ static unsigned long dm_bufio_shrink_count(struct shr= inker *shrink, struct shrin struct dm_bufio_client *dm_bufio_client_create(struct block_device *bdev, = unsigned block_size, unsigned reserved_buffers, unsigned aux_size, void (*alloc_callback)(struct dm_buffer *), - void (*write_callback)(struct dm_buffer *)) + void (*write_callback)(struct dm_buffer *), + unsigned int flags) { int r; struct dm_bufio_client *c; diff --git a/drivers/md/dm-ebs-target.c b/drivers/md/dm-ebs-target.c index 0221fa63f888..c90f9b9b1f02 100644 --- a/drivers/md/dm-ebs-target.c +++ b/drivers/md/dm-ebs-target.c @@ -312,7 +312,8 @@ static int ebs_ctr(struct dm_target *ti, unsigned int a= rgc, char **argv) goto bad; } =20 - ec->bufio =3D dm_bufio_client_create(ec->dev->bdev, to_bytes(ec->u_bs), 1= , 0, NULL, NULL); + ec->bufio =3D dm_bufio_client_create(ec->dev->bdev, to_bytes(ec->u_bs), 1, + 0, NULL, NULL, 0); if (IS_ERR(ec->bufio)) { ti->error =3D "Cannot create dm bufio client"; r =3D PTR_ERR(ec->bufio); diff --git a/drivers/md/dm-integrity.c b/drivers/md/dm-integrity.c index 3d5a0ce123c9..a508073d8414 100644 --- a/drivers/md/dm-integrity.c +++ b/drivers/md/dm-integrity.c @@ -4439,7 +4439,7 @@ static int dm_integrity_ctr(struct dm_target *ti, uns= igned argc, char **argv) } =20 ic->bufio =3D dm_bufio_client_create(ic->meta_dev ? ic->meta_dev->bdev : = ic->dev->bdev, - 1U << (SECTOR_SHIFT + ic->log2_buffer_sectors), 1, 0, NULL, NULL); + 1U << (SECTOR_SHIFT + ic->log2_buffer_sectors), 1, 0, NULL, NULL, 0); if (IS_ERR(ic->bufio)) { r =3D PTR_ERR(ic->bufio); ti->error =3D "Cannot initialize dm-bufio"; diff --git a/drivers/md/dm-snap-persistent.c b/drivers/md/dm-snap-persisten= t.c index 3bb5cff5d6fc..aaa699749c3b 100644 --- a/drivers/md/dm-snap-persistent.c +++ b/drivers/md/dm-snap-persistent.c @@ -494,7 +494,7 @@ static int read_exceptions(struct pstore *ps, =20 client =3D dm_bufio_client_create(dm_snap_cow(ps->store->snap)->bdev, ps->store->chunk_size << SECTOR_SHIFT, - 1, 0, NULL, NULL); + 1, 0, NULL, NULL, 0); =20 if (IS_ERR(client)) return PTR_ERR(client); diff --git a/drivers/md/dm-verity-fec.c b/drivers/md/dm-verity-fec.c index cea2b3789736..23cffce56403 100644 --- a/drivers/md/dm-verity-fec.c +++ b/drivers/md/dm-verity-fec.c @@ -749,7 +749,7 @@ int verity_fec_ctr(struct dm_verity *v) =20 f->bufio =3D dm_bufio_client_create(f->dev->bdev, f->io_size, - 1, 0, NULL, NULL); + 1, 0, NULL, NULL, 0); if (IS_ERR(f->bufio)) { ti->error =3D "Cannot initialize FEC bufio client"; return PTR_ERR(f->bufio); @@ -765,7 +765,7 @@ int verity_fec_ctr(struct dm_verity *v) =20 f->data_bufio =3D dm_bufio_client_create(v->data_dev->bdev, 1 << v->data_dev_block_bits, - 1, 0, NULL, NULL); + 1, 0, NULL, NULL, 0); if (IS_ERR(f->data_bufio)) { ti->error =3D "Cannot initialize FEC data bufio client"; return PTR_ERR(f->data_bufio); diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c index d6dbd47492a8..5d3fc58a3c34 100644 --- a/drivers/md/dm-verity-target.c +++ b/drivers/md/dm-verity-target.c @@ -1266,7 +1266,7 @@ static int verity_ctr(struct dm_target *ti, unsigned = argc, char **argv) =20 v->bufio =3D dm_bufio_client_create(v->hash_dev->bdev, 1 << v->hash_dev_block_bits, 1, sizeof(struct buffer_aux), - dm_bufio_alloc_callback, NULL); + dm_bufio_alloc_callback, NULL, 0); if (IS_ERR(v->bufio)) { ti->error =3D "Cannot initialize dm-bufio"; r =3D PTR_ERR(v->bufio); diff --git a/drivers/md/persistent-data/dm-block-manager.c b/drivers/md/per= sistent-data/dm-block-manager.c index 54c089a50b15..11935864f50f 100644 --- a/drivers/md/persistent-data/dm-block-manager.c +++ b/drivers/md/persistent-data/dm-block-manager.c @@ -391,7 +391,8 @@ struct dm_block_manager *dm_block_manager_create(struct= block_device *bdev, bm->bufio =3D dm_bufio_client_create(bdev, block_size, max_held_per_threa= d, sizeof(struct buffer_aux), dm_block_manager_alloc_callback, - dm_block_manager_write_callback); + dm_block_manager_write_callback, + 0); if (IS_ERR(bm->bufio)) { r =3D PTR_ERR(bm->bufio); kfree(bm); diff --git a/include/linux/dm-bufio.h b/include/linux/dm-bufio.h index 90bd558a17f5..e21480715255 100644 --- a/include/linux/dm-bufio.h +++ b/include/linux/dm-bufio.h @@ -24,7 +24,8 @@ struct dm_bufio_client * dm_bufio_client_create(struct block_device *bdev, unsigned block_size, unsigned reserved_buffers, unsigned aux_size, void (*alloc_callback)(struct dm_buffer *), - void (*write_callback)(struct dm_buffer *)); + void (*write_callback)(struct dm_buffer *), + unsigned int flags); =20 /* * Release a buffered IO cache. --=20 2.37.1.359.gd136c6c3e2-goog From nobody Fri Apr 17 21:11:09 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1AFDC433EF for ; Fri, 22 Jul 2022 09:45:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235092AbiGVJo7 (ORCPT ); Fri, 22 Jul 2022 05:44:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41974 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234643AbiGVJob (ORCPT ); Fri, 22 Jul 2022 05:44:31 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B4EAABB8E0 for ; Fri, 22 Jul 2022 02:38:52 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id m123-20020a253f81000000b0066ff6484995so3324897yba.22 for ; Fri, 22 Jul 2022 02:38:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=0HvoRxN2Vq0/Bi3xt58hq8m4jj9hpFn+i7fw2drMGQc=; b=rQP+Tru3fYyh2EUytXVRTQMhbH4x9vMaL4BCxtqbAMHLZeOSnAzZETqPWY1yuNQdt+ wBT5SXNEJZirRIcmB/75TQkdR8/ZDn7n0zIV8/wfsfQdlVwu3J9gq9UVC+DCEWZcHK9t 1UQb/rDV1iVdWFKxVVtTMLHp57Dg9AiKDYKElHq0R7xIduVsHkqNjZXJjc1iccHM4fJE jfMfmm9Te2uAVd+/y0Gh3MDgcVDs+iTiVxK5LeZU41d0NWfwdBAz5eDI23Mg9auSxrGg kLJCrc9E+m/2DFovGiEVLwZUfuBN+hUm6bqypCIroe0/K2DaG1yzoBAc0s4lWceU0IeI Jm+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=0HvoRxN2Vq0/Bi3xt58hq8m4jj9hpFn+i7fw2drMGQc=; b=UpvQ/9kM9oPFWP1otYU4Q3PT0y1PYapk75soNqSOywnnG+oBmueH9uHIK/TgIrquyR kRVseWaOQbznDusSMHRRXRN/gDtDrS6Cf6ZhAYjilOUC8kflGO+jcuvftk20lbBpLPLo y2jp26mQ1P3DMW/Ubzw25y4Oyti2+zYs1g9mxkMxvp8INYLNAvi5YLp94kPEwozlXenX FrVPnqMucD2M0ukLXiYWgsHukdWXRA49Yz7a70h3WFhXk+s0TDlof/Wdz0SJDDd/NoHW 0nIJ8IqwDCh9ABrizqcfkwb5dwtJMPyk6UEN0KJMxJC6CRyHZkCagCysOA63fUDftHl+ xmqA== X-Gm-Message-State: AJIora/e9y8P9p0ydiZEiA2lpFo9JG4hgtYCDpJpGGpI7VNj2t7lvuHQ pLffzbM+TosUOKGvLBcUKq1dj/Td5TBVyT7woHT9+5YGGv8vrUi9n9RnTrR6HpuBHKgA2FIytOh BgGUC/sPtDsMkVsSUyWwjAVNI7rNQT7sJfIOcrnx8efZ5GuMWBn+qMs0xfyzjZkN+QZg= X-Google-Smtp-Source: AGRyM1vrqyRWf0OgG6fTxyUyuG16olDyk74u0WLdD/aZC47eoaJHVQNRfVoKKDTJI4OkFZyLIEFXkpH0+Q== X-Received: from nhuck.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:39cc]) (user=nhuck job=sendgmr) by 2002:a25:fc21:0:b0:670:8160:a9f9 with SMTP id v33-20020a25fc21000000b006708160a9f9mr2134064ybd.513.1658482731701; Fri, 22 Jul 2022 02:38:51 -0700 (PDT) Date: Fri, 22 Jul 2022 09:38:22 +0000 In-Reply-To: <20220722093823.4158756-1-nhuck@google.com> Message-Id: <20220722093823.4158756-3-nhuck@google.com> Mime-Version: 1.0 References: <20220722093823.4158756-1-nhuck@google.com> X-Mailer: git-send-email 2.37.1.359.gd136c6c3e2-goog Subject: [PATCH 2/3] dm-bufio: Add DM_BUFIO_GET_CANT_SLEEP From: Nathan Huckleberry To: linux-kernel@vger.kernel.org, dm-devel@redhat.com, Alasdair Kergon , Mike Snitzer Cc: Eric Biggers , Sami Tolvanen , Nathan Huckleberry Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add an optional flag that ensures dm_bufio_get does not sleep. This allows the dm-bufio cache to be queried from interrupt context. To ensure that dm-bufio does not sleep, dm-bufio must use a spinlock instead of a mutex. Additionally, to avoid deadlocks, special care must be taken so that dm-bufio does not sleep while holding the spinlock. DM_BUFIO_GET_CANT_SLEEP is useful in some contexts, such as dm-verity, so that we can query the dm-bufio cache in a tasklet. If the required data is cached, processing can be handled immediately in the tasklet instead of waiting for a work-queue job to be scheduled. This can reduce latency when there is high CPU load and memory pressure. Signed-off-by: Nathan Huckleberry --- drivers/md/dm-bufio.c | 26 ++++++++++++++++++++++---- include/linux/dm-bufio.h | 5 +++++ 2 files changed, 27 insertions(+), 4 deletions(-) diff --git a/drivers/md/dm-bufio.c b/drivers/md/dm-bufio.c index ad5603eb12e3..3edeca7cfca6 100644 --- a/drivers/md/dm-bufio.c +++ b/drivers/md/dm-bufio.c @@ -81,6 +81,8 @@ */ struct dm_bufio_client { struct mutex lock; + spinlock_t spinlock; + unsigned long spinlock_flags; =20 struct list_head lru[LIST_SIZE]; unsigned long n_buffers[LIST_SIZE]; @@ -90,6 +92,7 @@ struct dm_bufio_client { s8 sectors_per_block_bits; void (*alloc_callback)(struct dm_buffer *); void (*write_callback)(struct dm_buffer *); + bool may_sleep; =20 struct kmem_cache *slab_buffer; struct kmem_cache *slab_cache; @@ -167,17 +170,26 @@ struct dm_buffer { =20 static void dm_bufio_lock(struct dm_bufio_client *c) { - mutex_lock_nested(&c->lock, dm_bufio_in_request()); + if (c->may_sleep) + mutex_lock_nested(&c->lock, dm_bufio_in_request()); + else + spin_lock_irqsave_nested(&c->spinlock, c->spinlock_flags, dm_bufio_in_re= quest()); } =20 static int dm_bufio_trylock(struct dm_bufio_client *c) { - return mutex_trylock(&c->lock); + if (c->may_sleep) + return mutex_trylock(&c->lock); + else + return spin_trylock_irqsave(&c->spinlock, c->spinlock_flags); } =20 static void dm_bufio_unlock(struct dm_bufio_client *c) { - mutex_unlock(&c->lock); + if (c->may_sleep) + mutex_unlock(&c->lock); + else + spin_unlock_irqrestore(&c->spinlock, c->spinlock_flags); } =20 /*----------------------------------------------------------------*/ @@ -878,7 +890,7 @@ static struct dm_buffer *__alloc_buffer_wait_no_callbac= k(struct dm_bufio_client * be allocated. */ while (1) { - if (dm_bufio_cache_size_latch !=3D 1) { + if (dm_bufio_cache_size_latch !=3D 1 && c->may_sleep) { b =3D alloc_buffer(c, GFP_NOWAIT | __GFP_NORETRY | __GFP_NOMEMALLOC | _= _GFP_NOWARN); if (b) return b; @@ -1041,6 +1053,7 @@ static struct dm_buffer *__bufio_new(struct dm_bufio_= client *c, sector_t block, if (nf =3D=3D NF_GET && unlikely(test_bit(B_READING, &b->state))) return NULL; =20 + b->hold_count++; __relink_lru(b, test_bit(B_DIRTY, &b->state) || test_bit(B_WRITING, &b->state)); @@ -1748,12 +1761,17 @@ struct dm_bufio_client *dm_bufio_client_create(stru= ct block_device *bdev, unsign c->alloc_callback =3D alloc_callback; c->write_callback =3D write_callback; =20 + c->may_sleep =3D true; + if (flags & DM_BUFIO_GET_CANT_SLEEP) + c->may_sleep =3D false; + for (i =3D 0; i < LIST_SIZE; i++) { INIT_LIST_HEAD(&c->lru[i]); c->n_buffers[i] =3D 0; } =20 mutex_init(&c->lock); + spin_lock_init(&c->spinlock); INIT_LIST_HEAD(&c->reserved_buffers); c->need_reserved_buffers =3D reserved_buffers; =20 diff --git a/include/linux/dm-bufio.h b/include/linux/dm-bufio.h index e21480715255..2a78f0cb8e71 100644 --- a/include/linux/dm-bufio.h +++ b/include/linux/dm-bufio.h @@ -17,6 +17,11 @@ struct dm_bufio_client; struct dm_buffer; =20 +/* + * Flags for dm_bufio_client_create + */ +#define DM_BUFIO_GET_CANT_SLEEP 0x1 + /* * Create a buffered IO cache on a given device */ --=20 2.37.1.359.gd136c6c3e2-goog From nobody Fri Apr 17 21:11:09 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21244C43334 for ; Fri, 22 Jul 2022 09:45:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229761AbiGVJpL (ORCPT ); Fri, 22 Jul 2022 05:45:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42072 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235286AbiGVJof (ORCPT ); Fri, 22 Jul 2022 05:44:35 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B90F0BB8F6 for ; Fri, 22 Jul 2022 02:38:54 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-31cdce3ed04so35620477b3.13 for ; Fri, 22 Jul 2022 02:38:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=+nB20Wi810hrWlIP5pvDiRZ1VMjjX1lRuMuiUP0CtOE=; b=NqWmN2shXmLgw7TB8EYBK+ZlvxKjvonEGl6fvKWYyNnGsNALfDJs3RwdTv9cpF7cQN Da/3mS9qAr/skonuFDha5s9UWpMwA7xR2h56VMClowaT7ormfIfhZJpDrYsGpYx4Bux4 eIuZ0gVc6/uUPk7vrSc/2fzYHqeogHDQPxgV9vDeEkKUyhkaIB91BK1Gozw922XbaBfY 6DeCuT6aXFU0xHdK2ytnGorB7A5oBYy3uQBqHCQdhSDlCui9c2W77Ob0BaNhVe9ig9+7 7a28rxquDYbLIKf1SDPYtIJpTqra0+bpHUAYjBh2x17QoF3843DUR7EfonW18iPQcVDY hKOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=+nB20Wi810hrWlIP5pvDiRZ1VMjjX1lRuMuiUP0CtOE=; b=m0RS7iNz/wX2qKrkRx9FIP1nIUSoyl6Au/Gipl20NeX+CTtGD+S8Hpufxp6JpeNS3T bc3Kikk68FRkSNZu7MYD8fE0MSmX198ngNueUUOX6cs2OgwGzXtsdTeEwLSMP37TYOaZ pQjKZ3AYiKr7V4z0vHqtm8Sl6wevekEj8JWTNoospRCUkiE3ZRPx83JoggPcqJ3ULV0q BnfnfHaHizkonjFy0SgwvcgjN8WCozLWYuHImoeEaN4azOvJ8fIj1K/1Q49TPjFDx7lj S8Bm/rgyc+sTtW0FRbuDRVX8P3LvQGjsSEhkXmUWmEArsqxSneSpltSkfJcCM9Oa0Hyv FJsw== X-Gm-Message-State: AJIora/m3Mvlby9IRpQRfp27Y0dc+KOsHQD2/WPQDf3xK4AsM3KEPBuG wfNz2JGcuo5d0GkGbNnCVsgNI5860UraCHLNpx4yxdCDl/adQcufdTqxts0FUa0y7LdyOBmDyhd IxRdcUeA6idQH37WQOxA95PrkGS6nfX00a/j+nkJUFqoc29kfSSClAxA+9b+KXxFDXSQ= X-Google-Smtp-Source: AGRyM1s26ZXo/6tQtihIsdDRm6L/HqasYPcozEjSaAtHWxveiAl9+jglTQqnpl6VN9mDS39c1CYZVGIwyg== X-Received: from nhuck.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:39cc]) (user=nhuck job=sendgmr) by 2002:a25:258:0:b0:670:a3b3:b941 with SMTP id 85-20020a250258000000b00670a3b3b941mr2140164ybc.533.1658482733913; Fri, 22 Jul 2022 02:38:53 -0700 (PDT) Date: Fri, 22 Jul 2022 09:38:23 +0000 In-Reply-To: <20220722093823.4158756-1-nhuck@google.com> Message-Id: <20220722093823.4158756-4-nhuck@google.com> Mime-Version: 1.0 References: <20220722093823.4158756-1-nhuck@google.com> X-Mailer: git-send-email 2.37.1.359.gd136c6c3e2-goog Subject: [PATCH 3/3] dm-verity: Add try_verify_in_tasklet From: Nathan Huckleberry To: linux-kernel@vger.kernel.org, dm-devel@redhat.com, Alasdair Kergon , Mike Snitzer Cc: Eric Biggers , Sami Tolvanen , Nathan Huckleberry Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Using tasklets for disk verification can reduce IO latency. When there are accelerated hash instructions it is often better to compute the hash immedi= ately using a tasklet rather than deferring verification to a work-queue. This reduces time spent waiting to schedule work-queue jobs, but requires spendi= ng slightly more time in interrupt context. If the dm-bufio cache does not have the required hashes we fallback to the work-queue implementation. The following shows a speed comparison of random reads on a dm-verity devic= e. The dm-verity device uses a 1G ramdisk for data and a 1G ramdisk for hashes. One test was run using tasklets and one test was run using the existing work-queue solution. Both tests were run when the dm-bufio cache was hot. = The tasklet implementation performs significantly better since there is no time waiting for work-queue jobs to be scheduled. # /data/fio /data/tasklet.fio | grep READ READ: bw=3D181MiB/s (190MB/s), 181MiB/s-181MiB/s (190MB/s-190MB/s), io= =3D512MiB (537MB), run=3D2827-2827msec # /data/fio /data/workqueue.fio | grep READ READ: bw=3D23.6MiB/s (24.8MB/s), 23.6MiB/s-23.6MiB/s (24.8MB/s-24.8MB/s), io=3D512MiB (537MB), run=3D21688-21688msec Signed-off-by: Nathan Huckleberry --- drivers/md/dm-bufio.c | 14 +++--- drivers/md/dm-verity-target.c | 87 ++++++++++++++++++++++++++++++----- drivers/md/dm-verity.h | 5 ++ 3 files changed, 87 insertions(+), 19 deletions(-) diff --git a/drivers/md/dm-bufio.c b/drivers/md/dm-bufio.c index 3edeca7cfca6..039f39c29594 100644 --- a/drivers/md/dm-bufio.c +++ b/drivers/md/dm-bufio.c @@ -92,7 +92,7 @@ struct dm_bufio_client { s8 sectors_per_block_bits; void (*alloc_callback)(struct dm_buffer *); void (*write_callback)(struct dm_buffer *); - bool may_sleep; + bool get_may_sleep; =20 struct kmem_cache *slab_buffer; struct kmem_cache *slab_cache; @@ -170,7 +170,7 @@ struct dm_buffer { =20 static void dm_bufio_lock(struct dm_bufio_client *c) { - if (c->may_sleep) + if (c->get_may_sleep) mutex_lock_nested(&c->lock, dm_bufio_in_request()); else spin_lock_irqsave_nested(&c->spinlock, c->spinlock_flags, dm_bufio_in_re= quest()); @@ -178,7 +178,7 @@ static void dm_bufio_lock(struct dm_bufio_client *c) =20 static int dm_bufio_trylock(struct dm_bufio_client *c) { - if (c->may_sleep) + if (c->get_may_sleep) return mutex_trylock(&c->lock); else return spin_trylock_irqsave(&c->spinlock, c->spinlock_flags); @@ -186,7 +186,7 @@ static int dm_bufio_trylock(struct dm_bufio_client *c) =20 static void dm_bufio_unlock(struct dm_bufio_client *c) { - if (c->may_sleep) + if (c->get_may_sleep) mutex_unlock(&c->lock); else spin_unlock_irqrestore(&c->spinlock, c->spinlock_flags); @@ -890,7 +890,7 @@ static struct dm_buffer *__alloc_buffer_wait_no_callbac= k(struct dm_bufio_client * be allocated. */ while (1) { - if (dm_bufio_cache_size_latch !=3D 1 && c->may_sleep) { + if (dm_bufio_cache_size_latch !=3D 1 && c->get_may_sleep) { b =3D alloc_buffer(c, GFP_NOWAIT | __GFP_NORETRY | __GFP_NOMEMALLOC | _= _GFP_NOWARN); if (b) return b; @@ -1761,9 +1761,9 @@ struct dm_bufio_client *dm_bufio_client_create(struct= block_device *bdev, unsign c->alloc_callback =3D alloc_callback; c->write_callback =3D write_callback; =20 - c->may_sleep =3D true; + c->get_may_sleep =3D true; if (flags & DM_BUFIO_GET_CANT_SLEEP) - c->may_sleep =3D false; + c->get_may_sleep =3D false; =20 for (i =3D 0; i < LIST_SIZE; i++) { INIT_LIST_HEAD(&c->lru[i]); diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c index 5d3fc58a3c34..e4d1b674a278 100644 --- a/drivers/md/dm-verity-target.c +++ b/drivers/md/dm-verity-target.c @@ -34,6 +34,7 @@ #define DM_VERITY_OPT_PANIC "panic_on_corruption" #define DM_VERITY_OPT_IGN_ZEROES "ignore_zero_blocks" #define DM_VERITY_OPT_AT_MOST_ONCE "check_at_most_once" +#define DM_VERITY_OPT_TASKLET_VERIFY "try_verify_in_tasklet" =20 #define DM_VERITY_OPTS_MAX (3 + DM_VERITY_OPTS_FEC + \ DM_VERITY_ROOT_HASH_VERIFICATION_OPTS) @@ -286,7 +287,20 @@ static int verity_verify_level(struct dm_verity *v, st= ruct dm_verity_io *io, =20 verity_hash_at_level(v, block, level, &hash_block, &offset); =20 - data =3D dm_bufio_read(v->bufio, hash_block, &buf); + if (io->in_tasklet) + data =3D dm_bufio_get(v->bufio, hash_block, &buf); + else + data =3D dm_bufio_read(v->bufio, hash_block, &buf); + + if (data =3D=3D NULL) { + /* + * We're running as a tasklet and the hash was not in the bufio + * cache. Return early and resume execution from a work-queue + * so that we can read the hash from disk. + */ + return -EAGAIN; + } + if (IS_ERR(data)) return PTR_ERR(data); =20 @@ -307,6 +321,14 @@ static int verity_verify_level(struct dm_verity *v, st= ruct dm_verity_io *io, if (likely(memcmp(verity_io_real_digest(v, io), want_digest, v->digest_size) =3D=3D 0)) aux->hash_verified =3D 1; + else if (io->in_tasklet) { + /* + * FEC code cannot be run in a tasklet since it may + * sleep. We need to resume execution in a work-queue + * to handle FEC. + */ + return -EAGAIN; + } else if (verity_fec_decode(v, io, DM_VERITY_BLOCK_TYPE_METADATA, hash_block, data, NULL) =3D=3D 0) @@ -474,13 +496,12 @@ static int verity_verify_io(struct dm_verity_io *io) bool is_zero; struct dm_verity *v =3D io->v; struct bvec_iter start; - unsigned b; struct crypto_wait wait; struct bio *bio =3D dm_bio_from_per_bio_data(io, v->ti->per_io_data_size); =20 - for (b =3D 0; b < io->n_blocks; b++) { + for (; io->block_idx < io->n_blocks; io->block_idx++) { int r; - sector_t cur_block =3D io->block + b; + sector_t cur_block =3D io->block + io->block_idx; struct ahash_request *req =3D verity_io_hash_req(v, io); =20 if (v->validated_blocks && @@ -527,6 +548,13 @@ static int verity_verify_io(struct dm_verity_io *io) if (v->validated_blocks) set_bit(cur_block, v->validated_blocks); continue; + } else if (io->in_tasklet) { + /* + * FEC code cannot be run in a tasklet since it may + * sleep. We need to resume execution in a work-queue + * to handle FEC. + */ + return -EAGAIN; } else if (verity_fec_decode(v, io, DM_VERITY_BLOCK_TYPE_DATA, cur_block, NULL, &start) =3D=3D 0) @@ -567,7 +595,8 @@ static void verity_finish_io(struct dm_verity_io *io, b= lk_status_t status) bio->bi_end_io =3D io->orig_bi_end_io; bio->bi_status =3D status; =20 - verity_fec_finish_io(io); + if (!io->in_tasklet) + verity_fec_finish_io(io); =20 bio_endio(bio); } @@ -575,8 +604,28 @@ static void verity_finish_io(struct dm_verity_io *io, = blk_status_t status) static void verity_work(struct work_struct *w) { struct dm_verity_io *io =3D container_of(w, struct dm_verity_io, work); + int err; + + io->in_tasklet =3D false; + err =3D verity_verify_io(io); =20 - verity_finish_io(io, errno_to_blk_status(verity_verify_io(io))); + verity_finish_io(io, errno_to_blk_status(err)); +} + +static void verity_tasklet(unsigned long data) +{ + struct dm_verity_io *io =3D (struct dm_verity_io *) data; + int err; + + io->in_tasklet =3D true; + err =3D verity_verify_io(io); + if (err) { + INIT_WORK(&io->work, verity_work); + queue_work(io->v->verify_wq, &io->work); + return; + } + + verity_finish_io(io, errno_to_blk_status(err)); } =20 static void verity_end_io(struct bio *bio) @@ -589,8 +638,14 @@ static void verity_end_io(struct bio *bio) return; } =20 - INIT_WORK(&io->work, verity_work); - queue_work(io->v->verify_wq, &io->work); + io->block_idx =3D 0; + if (io->v->use_tasklet) { + tasklet_init(&io->tasklet, verity_tasklet, (unsigned long)io); + tasklet_schedule(&io->tasklet); + } else { + INIT_WORK(&io->work, verity_work); + queue_work(io->v->verify_wq, &io->work); + } } =20 /* @@ -1012,7 +1067,12 @@ static int verity_parse_opt_args(struct dm_arg_set *= as, struct dm_verity *v, return r; continue; =20 - } else if (verity_is_fec_opt_arg(arg_name)) { + } else if (!strcasecmp(arg_name, DM_VERITY_OPT_TASKLET_VERIFY)) { + v->use_tasklet =3D true; + continue; + } + + else if (verity_is_fec_opt_arg(arg_name)) { r =3D verity_fec_parse_opt_args(as, v, &argc, arg_name); if (r) return r; @@ -1068,6 +1128,7 @@ static int verity_ctr(struct dm_target *ti, unsigned = argc, char **argv) } ti->private =3D v; v->ti =3D ti; + v->use_tasklet =3D false; =20 r =3D verity_fec_ctr_alloc(v); if (r) @@ -1156,7 +1217,7 @@ static int verity_ctr(struct dm_target *ti, unsigned = argc, char **argv) goto bad; } =20 - v->tfm =3D crypto_alloc_ahash(v->alg_name, 0, 0); + v->tfm =3D crypto_alloc_ahash(v->alg_name, 0, CRYPTO_ALG_ASYNC); if (IS_ERR(v->tfm)) { ti->error =3D "Cannot initialize hash function"; r =3D PTR_ERR(v->tfm); @@ -1266,7 +1327,8 @@ static int verity_ctr(struct dm_target *ti, unsigned = argc, char **argv) =20 v->bufio =3D dm_bufio_client_create(v->hash_dev->bdev, 1 << v->hash_dev_block_bits, 1, sizeof(struct buffer_aux), - dm_bufio_alloc_callback, NULL, 0); + dm_bufio_alloc_callback, NULL, v->use_tasklet ? + DM_BUFIO_GET_CANT_SLEEP : 0); if (IS_ERR(v->bufio)) { ti->error =3D "Cannot initialize dm-bufio"; r =3D PTR_ERR(v->bufio); @@ -1281,7 +1343,8 @@ static int verity_ctr(struct dm_target *ti, unsigned = argc, char **argv) } =20 /* WQ_UNBOUND greatly improves performance when running on ramdisk */ - v->verify_wq =3D alloc_workqueue("kverityd", WQ_CPU_INTENSIVE | WQ_MEM_RE= CLAIM | WQ_UNBOUND, num_online_cpus()); + v->verify_wq =3D alloc_workqueue("kverityd", WQ_HIGHPRI | WQ_MEM_RECLAIM | + WQ_UNBOUND, num_online_cpus()); if (!v->verify_wq) { ti->error =3D "Cannot allocate workqueue"; r =3D -ENOMEM; diff --git a/drivers/md/dm-verity.h b/drivers/md/dm-verity.h index 4e769d13473a..4e65f93bd8d6 100644 --- a/drivers/md/dm-verity.h +++ b/drivers/md/dm-verity.h @@ -13,6 +13,7 @@ =20 #include #include +#include #include =20 #define DM_VERITY_MAX_LEVELS 63 @@ -50,6 +51,7 @@ struct dm_verity { unsigned char hash_dev_block_bits; /* log2(hash blocksize) */ unsigned char hash_per_block_bits; /* log2(hashes in hash block) */ unsigned char levels; /* the number of tree levels */ + bool use_tasklet; /* try to verify in tasklet before work-queue */ unsigned char version; unsigned digest_size; /* digest size for the current hash algorithm */ unsigned int ahash_reqsize;/* the size of temporary space for crypto */ @@ -76,10 +78,13 @@ struct dm_verity_io { =20 sector_t block; unsigned n_blocks; + unsigned int block_idx; + bool in_tasklet; =20 struct bvec_iter iter; =20 struct work_struct work; + struct tasklet_struct tasklet; =20 /* * Three variably-size fields follow this struct: --=20 2.37.1.359.gd136c6c3e2-goog