From nobody Fri Dec 19 07:00:25 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CADA41F4173; Sun, 24 Mar 2024 22:59:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711321148; cv=none; b=MUeudeIMQgoTW/fgAMhX4RO7b2gXURtOaFKpgsdorb2h09hSAafIezFk+tGv/+R1sBCGHT2PhPDPKEHCvyiypDP2maU4V3IPsaTDQzQLpvmaUqdJ1GPJcgBUpke580Pu0V/x0AVm06DsHN4obZ1XxFDY4ZPLGkYhiDXdTdpFuZY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711321148; c=relaxed/simple; bh=U1UaOiNkNgzH2RSDym1dmZ72i3ioP1h7+0px3gN4/X0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PKLmzqtcV90PNY8S9GcX2pjEsRNPj+8BN/fIGMYK4gHNvdXFXqxcqsIzMt0vrtG9k7aSDjpkyiwxabkbHNmh9KMHUBBaoss9FPe2DGITgG7xYjTICk9IkQzJ2nc5DbmJzgvizVnIkswxHzsQVXIxHxsAwVGmwHNdpM+x0jNR78s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=MAJC3Ngs; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="MAJC3Ngs" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C2D0BC433C7; Sun, 24 Mar 2024 22:59:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1711321147; bh=U1UaOiNkNgzH2RSDym1dmZ72i3ioP1h7+0px3gN4/X0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MAJC3NgsLUl8/GHn80Iw2GUIEVCABzwNgqkSm0Spxc5zjjgKFIYhp5ZX5HeFgN18p haenoJ/3sPWGQmhjesw/FUOqMrjZ7cUszlYZZ5JX1m6Gi1chU9fMHSCy9D2ntNA4Hw iKC+jcLhHv/rJ9to2Axx/14DCCJMkiV+RY+spCLFRhgwoUrkGy95/qBicxPRH8CfRD bXnB77emN94XNxKsi+M5804QhE+1GEayaQLZIQozeCd/e9jBpsZn+tXVXvZrsyNqXG BZAS9J6pd2jpsL/RH2os6gEgbMlnQGi1VHcg2KnHZWYo5rdNb1p9F/3Sfk/AInY6lM 12dGo7DjPWJtQ== From: Sasha Levin To: linux-kernel@vger.kernel.org, stable@vger.kernel.org Cc: Hongyu Jin , Yibin Ding , Eric Biggers , Mikulas Patocka , Mike Snitzer , Sasha Levin Subject: [PATCH 6.7 709/713] dm io: Support IO priority Date: Sun, 24 Mar 2024 18:47:15 -0400 Message-ID: <20240324224720.1345309-710-sashal@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240324224720.1345309-1-sashal@kernel.org> References: <20240324224720.1345309-1-sashal@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-stable: review X-Patchwork-Hint: Ignore Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hongyu Jin [ Upstream commit 6e5f0f6383b4896c7e9b943d84b136149d0f45e9 ] Some IO will dispatch from kworker with different io_context settings than the submitting task, we may need to specify a priority to avoid losing priority. Add IO priority parameter to dm_io() and update all callers. Co-developed-by: Yibin Ding Signed-off-by: Yibin Ding Signed-off-by: Hongyu Jin Reviewed-by: Eric Biggers Reviewed-by: Mikulas Patocka Signed-off-by: Mike Snitzer Stable-dep-of: b4d78cfeb304 ("dm-integrity: align the outgoing bio in integ= rity_recheck") Signed-off-by: Sasha Levin --- drivers/md/dm-bufio.c | 6 +++--- drivers/md/dm-integrity.c | 12 ++++++------ drivers/md/dm-io.c | 23 +++++++++++++---------- drivers/md/dm-kcopyd.c | 4 ++-- drivers/md/dm-log.c | 4 ++-- drivers/md/dm-raid1.c | 6 +++--- drivers/md/dm-snap-persistent.c | 4 ++-- drivers/md/dm-verity-target.c | 2 +- drivers/md/dm-writecache.c | 8 ++++---- include/linux/dm-io.h | 3 ++- 10 files changed, 38 insertions(+), 34 deletions(-) diff --git a/drivers/md/dm-bufio.c b/drivers/md/dm-bufio.c index f03d7dba270c5..4f2808ef387f6 100644 --- a/drivers/md/dm-bufio.c +++ b/drivers/md/dm-bufio.c @@ -1315,7 +1315,7 @@ static void use_dmio(struct dm_buffer *b, enum req_op= op, sector_t sector, io_req.mem.ptr.vma =3D (char *)b->data + offset; } =20 - r =3D dm_io(&io_req, 1, ®ion, NULL); + r =3D dm_io(&io_req, 1, ®ion, NULL, IOPRIO_DEFAULT); if (unlikely(r)) b->end_io(b, errno_to_blk_status(r)); } @@ -2167,7 +2167,7 @@ int dm_bufio_issue_flush(struct dm_bufio_client *c) if (WARN_ON_ONCE(dm_bufio_in_request())) return -EINVAL; =20 - return dm_io(&io_req, 1, &io_reg, NULL); + return dm_io(&io_req, 1, &io_reg, NULL, IOPRIO_DEFAULT); } EXPORT_SYMBOL_GPL(dm_bufio_issue_flush); =20 @@ -2191,7 +2191,7 @@ int dm_bufio_issue_discard(struct dm_bufio_client *c,= sector_t block, sector_t c if (WARN_ON_ONCE(dm_bufio_in_request())) return -EINVAL; /* discards are optional */ =20 - return dm_io(&io_req, 1, &io_reg, NULL); + return dm_io(&io_req, 1, &io_reg, NULL, IOPRIO_DEFAULT); } EXPORT_SYMBOL_GPL(dm_bufio_issue_discard); =20 diff --git a/drivers/md/dm-integrity.c b/drivers/md/dm-integrity.c index cc834c8423bcf..ec8df735077fe 100644 --- a/drivers/md/dm-integrity.c +++ b/drivers/md/dm-integrity.c @@ -555,7 +555,7 @@ static int sync_rw_sb(struct dm_integrity_c *ic, blk_op= f_t opf) } } =20 - r =3D dm_io(&io_req, 1, &io_loc, NULL); + r =3D dm_io(&io_req, 1, &io_loc, NULL, IOPRIO_DEFAULT); if (unlikely(r)) return r; =20 @@ -1073,7 +1073,7 @@ static void rw_journal_sectors(struct dm_integrity_c = *ic, blk_opf_t opf, io_loc.sector =3D ic->start + SB_SECTORS + sector; io_loc.count =3D n_sectors; =20 - r =3D dm_io(&io_req, 1, &io_loc, NULL); + r =3D dm_io(&io_req, 1, &io_loc, NULL, IOPRIO_DEFAULT); if (unlikely(r)) { dm_integrity_io_error(ic, (opf & REQ_OP_MASK) =3D=3D REQ_OP_READ ? "reading journal" : "writing journal", r); @@ -1190,7 +1190,7 @@ static void copy_from_journal(struct dm_integrity_c *= ic, unsigned int section, u io_loc.sector =3D target; io_loc.count =3D n_sectors; =20 - r =3D dm_io(&io_req, 1, &io_loc, NULL); + r =3D dm_io(&io_req, 1, &io_loc, NULL, IOPRIO_DEFAULT); if (unlikely(r)) { WARN_ONCE(1, "asynchronous dm_io failed: %d", r); fn(-1UL, data); @@ -1519,7 +1519,7 @@ static void dm_integrity_flush_buffers(struct dm_inte= grity_c *ic, bool flush_dat fr.io_reg.count =3D 0, fr.ic =3D ic; init_completion(&fr.comp); - r =3D dm_io(&fr.io_req, 1, &fr.io_reg, NULL); + r =3D dm_io(&fr.io_req, 1, &fr.io_reg, NULL, IOPRIO_DEFAULT); BUG_ON(r); } =20 @@ -1727,7 +1727,7 @@ static noinline void integrity_recheck(struct dm_inte= grity_io *dio, char *checks io_loc.sector =3D sector; io_loc.count =3D ic->sectors_per_block; =20 - r =3D dm_io(&io_req, 1, &io_loc, NULL); + r =3D dm_io(&io_req, 1, &io_loc, NULL, IOPRIO_DEFAULT); if (unlikely(r)) { dio->bi_status =3D errno_to_blk_status(r); goto free_ret; @@ -2806,7 +2806,7 @@ static void integrity_recalc(struct work_struct *w) io_loc.sector =3D get_data_sector(ic, area, offset); io_loc.count =3D n_sectors; =20 - r =3D dm_io(&io_req, 1, &io_loc, NULL); + r =3D dm_io(&io_req, 1, &io_loc, NULL, IOPRIO_DEFAULT); if (unlikely(r)) { dm_integrity_io_error(ic, "reading data", r); goto err; diff --git a/drivers/md/dm-io.c b/drivers/md/dm-io.c index f053ce2458147..7409490259d1d 100644 --- a/drivers/md/dm-io.c +++ b/drivers/md/dm-io.c @@ -305,7 +305,7 @@ static void km_dp_init(struct dpages *dp, void *data) */ static void do_region(const blk_opf_t opf, unsigned int region, struct dm_io_region *where, struct dpages *dp, - struct io *io) + struct io *io, unsigned short ioprio) { struct bio *bio; struct page *page; @@ -354,6 +354,7 @@ static void do_region(const blk_opf_t opf, unsigned int= region, &io->client->bios); bio->bi_iter.bi_sector =3D where->sector + (where->count - remaining); bio->bi_end_io =3D endio; + bio->bi_ioprio =3D ioprio; store_io_and_region_in_bio(bio, io, region); =20 if (op =3D=3D REQ_OP_DISCARD || op =3D=3D REQ_OP_WRITE_ZEROES) { @@ -383,7 +384,7 @@ static void do_region(const blk_opf_t opf, unsigned int= region, =20 static void dispatch_io(blk_opf_t opf, unsigned int num_regions, struct dm_io_region *where, struct dpages *dp, - struct io *io, int sync) + struct io *io, int sync, unsigned short ioprio) { int i; struct dpages old_pages =3D *dp; @@ -400,7 +401,7 @@ static void dispatch_io(blk_opf_t opf, unsigned int num= _regions, for (i =3D 0; i < num_regions; i++) { *dp =3D old_pages; if (where[i].count || (opf & REQ_PREFLUSH)) - do_region(opf, i, where + i, dp, io); + do_region(opf, i, where + i, dp, io, ioprio); } =20 /* @@ -425,7 +426,7 @@ static void sync_io_complete(unsigned long error, void = *context) =20 static int sync_io(struct dm_io_client *client, unsigned int num_regions, struct dm_io_region *where, blk_opf_t opf, struct dpages *dp, - unsigned long *error_bits) + unsigned long *error_bits, unsigned short ioprio) { struct io *io; struct sync_io sio; @@ -447,7 +448,7 @@ static int sync_io(struct dm_io_client *client, unsigne= d int num_regions, io->vma_invalidate_address =3D dp->vma_invalidate_address; io->vma_invalidate_size =3D dp->vma_invalidate_size; =20 - dispatch_io(opf, num_regions, where, dp, io, 1); + dispatch_io(opf, num_regions, where, dp, io, 1, ioprio); =20 wait_for_completion_io(&sio.wait); =20 @@ -459,7 +460,8 @@ static int sync_io(struct dm_io_client *client, unsigne= d int num_regions, =20 static int async_io(struct dm_io_client *client, unsigned int num_regions, struct dm_io_region *where, blk_opf_t opf, - struct dpages *dp, io_notify_fn fn, void *context) + struct dpages *dp, io_notify_fn fn, void *context, + unsigned short ioprio) { struct io *io; =20 @@ -479,7 +481,7 @@ static int async_io(struct dm_io_client *client, unsign= ed int num_regions, io->vma_invalidate_address =3D dp->vma_invalidate_address; io->vma_invalidate_size =3D dp->vma_invalidate_size; =20 - dispatch_io(opf, num_regions, where, dp, io, 0); + dispatch_io(opf, num_regions, where, dp, io, 0, ioprio); return 0; } =20 @@ -521,7 +523,8 @@ static int dp_init(struct dm_io_request *io_req, struct= dpages *dp, } =20 int dm_io(struct dm_io_request *io_req, unsigned int num_regions, - struct dm_io_region *where, unsigned long *sync_error_bits) + struct dm_io_region *where, unsigned long *sync_error_bits, + unsigned short ioprio) { int r; struct dpages dp; @@ -532,11 +535,11 @@ int dm_io(struct dm_io_request *io_req, unsigned int = num_regions, =20 if (!io_req->notify.fn) return sync_io(io_req->client, num_regions, where, - io_req->bi_opf, &dp, sync_error_bits); + io_req->bi_opf, &dp, sync_error_bits, ioprio); =20 return async_io(io_req->client, num_regions, where, io_req->bi_opf, &dp, io_req->notify.fn, - io_req->notify.context); + io_req->notify.context, ioprio); } EXPORT_SYMBOL(dm_io); =20 diff --git a/drivers/md/dm-kcopyd.c b/drivers/md/dm-kcopyd.c index d01807c50f20b..79c65c9ad5fa8 100644 --- a/drivers/md/dm-kcopyd.c +++ b/drivers/md/dm-kcopyd.c @@ -578,9 +578,9 @@ static int run_io_job(struct kcopyd_job *job) io_job_start(job->kc->throttle); =20 if (job->op =3D=3D REQ_OP_READ) - r =3D dm_io(&io_req, 1, &job->source, NULL); + r =3D dm_io(&io_req, 1, &job->source, NULL, IOPRIO_DEFAULT); else - r =3D dm_io(&io_req, job->num_dests, job->dests, NULL); + r =3D dm_io(&io_req, job->num_dests, job->dests, NULL, IOPRIO_DEFAULT); =20 return r; } diff --git a/drivers/md/dm-log.c b/drivers/md/dm-log.c index f9f84236dfcd7..f7f9c2100937b 100644 --- a/drivers/md/dm-log.c +++ b/drivers/md/dm-log.c @@ -300,7 +300,7 @@ static int rw_header(struct log_c *lc, enum req_op op) { lc->io_req.bi_opf =3D op; =20 - return dm_io(&lc->io_req, 1, &lc->header_location, NULL); + return dm_io(&lc->io_req, 1, &lc->header_location, NULL, IOPRIO_DEFAULT); } =20 static int flush_header(struct log_c *lc) @@ -313,7 +313,7 @@ static int flush_header(struct log_c *lc) =20 lc->io_req.bi_opf =3D REQ_OP_WRITE | REQ_PREFLUSH; =20 - return dm_io(&lc->io_req, 1, &null_location, NULL); + return dm_io(&lc->io_req, 1, &null_location, NULL, IOPRIO_DEFAULT); } =20 static int read_header(struct log_c *log) diff --git a/drivers/md/dm-raid1.c b/drivers/md/dm-raid1.c index ddcb2bc4a6179..9511dae5b556a 100644 --- a/drivers/md/dm-raid1.c +++ b/drivers/md/dm-raid1.c @@ -278,7 +278,7 @@ static int mirror_flush(struct dm_target *ti) } =20 error_bits =3D -1; - dm_io(&io_req, ms->nr_mirrors, io, &error_bits); + dm_io(&io_req, ms->nr_mirrors, io, &error_bits, IOPRIO_DEFAULT); if (unlikely(error_bits !=3D 0)) { for (i =3D 0; i < ms->nr_mirrors; i++) if (test_bit(i, &error_bits)) @@ -554,7 +554,7 @@ static void read_async_bio(struct mirror *m, struct bio= *bio) =20 map_region(&io, m, bio); bio_set_m(bio, m); - BUG_ON(dm_io(&io_req, 1, &io, NULL)); + BUG_ON(dm_io(&io_req, 1, &io, NULL, IOPRIO_DEFAULT)); } =20 static inline int region_in_sync(struct mirror_set *ms, region_t region, @@ -681,7 +681,7 @@ static void do_write(struct mirror_set *ms, struct bio = *bio) */ bio_set_m(bio, get_default_mirror(ms)); =20 - BUG_ON(dm_io(&io_req, ms->nr_mirrors, io, NULL)); + BUG_ON(dm_io(&io_req, ms->nr_mirrors, io, NULL, IOPRIO_DEFAULT)); } =20 static void do_writes(struct mirror_set *ms, struct bio_list *writes) diff --git a/drivers/md/dm-snap-persistent.c b/drivers/md/dm-snap-persisten= t.c index 15649921f2a9b..568d10842b1f4 100644 --- a/drivers/md/dm-snap-persistent.c +++ b/drivers/md/dm-snap-persistent.c @@ -223,7 +223,7 @@ static void do_metadata(struct work_struct *work) { struct mdata_req *req =3D container_of(work, struct mdata_req, work); =20 - req->result =3D dm_io(req->io_req, 1, req->where, NULL); + req->result =3D dm_io(req->io_req, 1, req->where, NULL, IOPRIO_DEFAULT); } =20 /* @@ -247,7 +247,7 @@ static int chunk_io(struct pstore *ps, void *area, chun= k_t chunk, blk_opf_t opf, struct mdata_req req; =20 if (!metadata) - return dm_io(&io_req, 1, &where, NULL); + return dm_io(&io_req, 1, &where, NULL, IOPRIO_DEFAULT); =20 req.where =3D &where; req.io_req =3D &io_req; diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c index 7b620b187da90..49e4a35d70196 100644 --- a/drivers/md/dm-verity-target.c +++ b/drivers/md/dm-verity-target.c @@ -511,7 +511,7 @@ static noinline int verity_recheck(struct dm_verity *v,= struct dm_verity_io *io, io_loc.bdev =3D v->data_dev->bdev; io_loc.sector =3D cur_block << (v->data_dev_block_bits - SECTOR_SHIFT); io_loc.count =3D 1 << (v->data_dev_block_bits - SECTOR_SHIFT); - r =3D dm_io(&io_req, 1, &io_loc, NULL); + r =3D dm_io(&io_req, 1, &io_loc, NULL, IOPRIO_DEFAULT); if (unlikely(r)) goto free_ret; =20 diff --git a/drivers/md/dm-writecache.c b/drivers/md/dm-writecache.c index 074cb785eafc1..6a4279bfb1e77 100644 --- a/drivers/md/dm-writecache.c +++ b/drivers/md/dm-writecache.c @@ -531,7 +531,7 @@ static void ssd_commit_flushed(struct dm_writecache *wc= , bool wait_for_ios) req.notify.context =3D &endio; =20 /* writing via async dm-io (implied by notify.fn above) won't return an = error */ - (void) dm_io(&req, 1, ®ion, NULL); + (void) dm_io(&req, 1, ®ion, NULL, IOPRIO_DEFAULT); i =3D j; } =20 @@ -568,7 +568,7 @@ static void ssd_commit_superblock(struct dm_writecache = *wc) req.notify.fn =3D NULL; req.notify.context =3D NULL; =20 - r =3D dm_io(&req, 1, ®ion, NULL); + r =3D dm_io(&req, 1, ®ion, NULL, IOPRIO_DEFAULT); if (unlikely(r)) writecache_error(wc, r, "error writing superblock"); } @@ -596,7 +596,7 @@ static void writecache_disk_flush(struct dm_writecache = *wc, struct dm_dev *dev) req.client =3D wc->dm_io; req.notify.fn =3D NULL; =20 - r =3D dm_io(&req, 1, ®ion, NULL); + r =3D dm_io(&req, 1, ®ion, NULL, IOPRIO_DEFAULT); if (unlikely(r)) writecache_error(wc, r, "error flushing metadata: %d", r); } @@ -990,7 +990,7 @@ static int writecache_read_metadata(struct dm_writecach= e *wc, sector_t n_sectors req.client =3D wc->dm_io; req.notify.fn =3D NULL; =20 - return dm_io(&req, 1, ®ion, NULL); + return dm_io(&req, 1, ®ion, NULL, IOPRIO_DEFAULT); } =20 static void writecache_resume(struct dm_target *ti) diff --git a/include/linux/dm-io.h b/include/linux/dm-io.h index 7595142f3fc57..7b2968612b7e6 100644 --- a/include/linux/dm-io.h +++ b/include/linux/dm-io.h @@ -80,7 +80,8 @@ void dm_io_client_destroy(struct dm_io_client *client); * error occurred doing io to the corresponding region. */ int dm_io(struct dm_io_request *io_req, unsigned int num_regions, - struct dm_io_region *region, unsigned int long *sync_error_bits); + struct dm_io_region *region, unsigned int long *sync_error_bits, + unsigned short ioprio); =20 #endif /* __KERNEL__ */ #endif /* _LINUX_DM_IO_H */ --=20 2.43.0