From nobody Fri Dec 19 14:35:34 2025 Received: from mail-wr1-f103.google.com (mail-wr1-f103.google.com [209.85.221.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 322C2333452 for ; Wed, 5 Nov 2025 20:28:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.103 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374517; cv=none; b=cXhiOY8fr6v2899iimGzlvw4ihyOm85GCxWWv0DacnSQZQxj8xH9VSuCWjRdySIvtMGARhFsgj8jxCxxqPTZ1qYzPnmAsC5C+dZ/aJyV17paT/df4y4UPEWbYyUhW6KhcX2ypANtQCYHNYioxMAYiOnk4L7JEaNUNwmH0bIoAkM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374517; c=relaxed/simple; bh=y7cph3TN+LG3pXaIQbG7EKhMI9WouvmH01KqS9mjU2M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hBQObX+K+XIPtSOScAqKFz24sdPNE69zDZ+efw78C3j/TflRejW0xu2swRvzzwNHuu5yU3IHsJiAjJL2Z6C1wLHAaXOGWJ56npsoJQnR6kdz62V5f3ICMf83RsbjKBYW3J1UGTnq9XiBw4sS1QW2HhPue8k+TEDJs2QVBESShp0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com; spf=fail smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=TFoQ0lgY; arc=none smtp.client-ip=209.85.221.103 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="TFoQ0lgY" Received: by mail-wr1-f103.google.com with SMTP id ffacd0b85a97d-4271234b49cso40514f8f.3 for ; Wed, 05 Nov 2025 12:28:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1762374512; x=1762979312; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jvk9JvTk/zngD9H7iDECFvgJx6bX0wcaTFOZurY55Sw=; b=TFoQ0lgYJykNJISZ9sT4UT90UfWRTplZwC3cnwhTeJWkauqvO+NLuQoDIdCtfIP9/p BkpEm9aGa3z8MSRmDVeF+zcvPvV+iVBfRFOp0rdkiItA4jV3p3A4suX4BQ/CkJzv6Ddl CnY3OgJkIrKugsryQEEBkBbBI+5N684MiO6A1egdUmeaGTE24HBCZlJU22HmIr1zAqow 22qUCchQwyqnc1p0QJ2dt4DpujpbXqEP8QM3t7VUBLNrWDYjjIzukMLF4ITLU5TbFDXj CFap2CUFMO/RRuYm8k/JVCMXWbDsJPxfT062YW6Jr/0320BrrfyRHlbNpdkdoO0Fiqp3 OF3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762374512; x=1762979312; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jvk9JvTk/zngD9H7iDECFvgJx6bX0wcaTFOZurY55Sw=; b=V9MYV2aKjs/kFW7Mlkj5YgeAhiIHYlO9X06TID10+gOO79JVrRpTF4NRlYMwpuhf4j Nv9aDu10idqaXc1wnOAfM+46mLImXTbNp7u/eQKTBu0HyycmYPGcTmECHAC7dZ7o7h7P Usczqhg4uzRliamcy7UP7cewqZwr6F2S53fYY+dQZfqkKCOtSGXb+rJssquK4ThuQNQn kQ8rn1dcG4rAr1kpqAZIvPgBmiRs5etYAjiUzkZp9ZTvp1msl8xjSIp/EIv1Tb1tLGi0 8doVCdZT8l/YwYxvWUJmC1rc4RhkfeHtZYViQgTNhNAMHogYKbRSJZ4IBJio5Lc8tmYT Rhkg== X-Forwarded-Encrypted: i=1; AJvYcCUi2daSxSdq01cKAlh0vY4PKPxBl7V6vg2I4nufcpHHbebxNpf/C38XY9xttnjhR3p8cL9GVLVQL4ENbDY=@vger.kernel.org X-Gm-Message-State: AOJu0YzH30MmDnjDx5MTaJ3cenykWXMm8larY3va1Ynng8BgQJkW0ihu qtIaXNJiO/GoAdPZbwSSzStQlDMpG5Tb6puRejpxLs+kH+gclgztESI5ODZgeRxi8YtmDqZ2IVb kvqqGFE4Hi0EV/WgJkmd/bXntDffJkYae5VgO2WPDcQqCLNeCX4MJ X-Gm-Gg: ASbGncvAF3NybKHMKmRN0stwp5FXhnG2ek8XrCuAagHGw+up6WImFXGX2tZfueWnHd2 k35L6TN7gM8enOKLGFogD+UNYUlR0/bfk2co3xpYkcYDu9J259lsxOSDgV2FtWg0amk79cKTWnP BHAoFVMMu1eEw6v8+LMCsRP66dJSaSY/DtyctBiNcurEFrXXdLqsuK3NJTMXiMuMCR17udZRM/1 wX917bHtmKayWeT6dsTNBWHZisitbbZacbooDhCg7jy2My0Hnwiy+KP/HQZx8M97X7ffNH8F1Zo brzQ6LpLCYoX4yirEPsABID1GTf9i2K4EpigSGesCxcBL4pK9UmrXooSulNF3rBNWoUTseou3cm eOI+Ynu1VcvoIoXj7 X-Google-Smtp-Source: AGHT+IE4XWhmeoD7+ZfhKtJxniWegM5tVl5BSUxfYAckBqExLj8aQDOpGX57QEZndaxwC4e8NdZt+dAGIElM X-Received: by 2002:a05:600c:358d:b0:477:598d:65e7 with SMTP id 5b1f17b1804b1-4775ce1e88emr25079535e9.4.1762374512322; Wed, 05 Nov 2025 12:28:32 -0800 (PST) Received: from c7-smtp-2023.dev.purestorage.com ([2620:125:9017:12:36:3:5:0]) by smtp-relay.gmail.com with ESMTPS id ffacd0b85a97d-429eb435628sm24245f8f.21.2025.11.05.12.28.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Nov 2025 12:28:32 -0800 (PST) X-Relaying-Domain: purestorage.com Received: from dev-csander.dev.purestorage.com (unknown [IPv6:2620:125:9007:640:ffff::1199]) by c7-smtp-2023.dev.purestorage.com (Postfix) with ESMTP id 5F136340505; Wed, 5 Nov 2025 13:28:30 -0700 (MST) Received: by dev-csander.dev.purestorage.com (Postfix, from userid 1557716354) id 7F1B1E413BB; Wed, 5 Nov 2025 13:28:30 -0700 (MST) From: Caleb Sander Mateos To: Ming Lei , Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Caleb Sander Mateos Subject: [PATCH v2 1/2] ublk: use copy_{to,from}_iter() for user copy Date: Wed, 5 Nov 2025 13:28:22 -0700 Message-ID: <20251105202823.2198194-2-csander@purestorage.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20251105202823.2198194-1-csander@purestorage.com> References: <20251105202823.2198194-1-csander@purestorage.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" ublk_copy_user_pages()/ublk_copy_io_pages() currently uses iov_iter_get_pages2() to extract the pages from the iov_iter and memcpy()s between the bvec_iter and the iov_iter's pages one at a time. Switch to using copy_to_iter()/copy_from_iter() instead. This avoids the user page reference count increments and decrements and needing to split the memcpy() at user page boundaries. It also simplifies the code considerably. Ming reports a 40% throughput improvement when issuing I/O to the selftests null ublk server with zero-copy disabled. Signed-off-by: Caleb Sander Mateos Reviewed-by: Ming Lei --- drivers/block/ublk_drv.c | 62 +++++++++------------------------------- 1 file changed, 14 insertions(+), 48 deletions(-) diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c index 97cc4bc0a6ce..40eee3e15a4c 100644 --- a/drivers/block/ublk_drv.c +++ b/drivers/block/ublk_drv.c @@ -911,58 +911,47 @@ static const struct block_device_operations ub_fops = =3D { .open =3D ublk_open, .free_disk =3D ublk_free_disk, .report_zones =3D ublk_report_zones, }; =20 -#define UBLK_MAX_PIN_PAGES 32 - struct ublk_io_iter { - struct page *pages[UBLK_MAX_PIN_PAGES]; struct bio *bio; struct bvec_iter iter; }; =20 -/* return how many pages are copied */ -static void ublk_copy_io_pages(struct ublk_io_iter *data, - size_t total, size_t pg_off, int dir) +/* return how many bytes are copied */ +static size_t ublk_copy_io_pages(struct ublk_io_iter *data, + struct iov_iter *uiter, int dir) { - unsigned done =3D 0; - unsigned pg_idx =3D 0; + size_t done =3D 0; =20 - while (done < total) { + for (;;) { struct bio_vec bv =3D bio_iter_iovec(data->bio, data->iter); - unsigned int bytes =3D min3(bv.bv_len, (unsigned)total - done, - (unsigned)(PAGE_SIZE - pg_off)); void *bv_buf =3D bvec_kmap_local(&bv); - void *pg_buf =3D kmap_local_page(data->pages[pg_idx]); + size_t copied; =20 if (dir =3D=3D ITER_DEST) - memcpy(pg_buf + pg_off, bv_buf, bytes); + copied =3D copy_to_iter(bv_buf, bv.bv_len, uiter); else - memcpy(bv_buf, pg_buf + pg_off, bytes); + copied =3D copy_from_iter(bv_buf, bv.bv_len, uiter); =20 - kunmap_local(pg_buf); kunmap_local(bv_buf); =20 - /* advance page array */ - pg_off +=3D bytes; - if (pg_off =3D=3D PAGE_SIZE) { - pg_idx +=3D 1; - pg_off =3D 0; - } - - done +=3D bytes; + done +=3D copied; + if (copied < bv.bv_len) + break; =20 /* advance bio */ - bio_advance_iter_single(data->bio, &data->iter, bytes); + bio_advance_iter_single(data->bio, &data->iter, copied); if (!data->iter.bi_size) { data->bio =3D data->bio->bi_next; if (data->bio =3D=3D NULL) break; data->iter =3D data->bio->bi_iter; } } + return done; } =20 static bool ublk_advance_io_iter(const struct request *req, struct ublk_io_iter *iter, unsigned int offset) { @@ -986,38 +975,15 @@ static bool ublk_advance_io_iter(const struct request= *req, */ static size_t ublk_copy_user_pages(const struct request *req, unsigned offset, struct iov_iter *uiter, int dir) { struct ublk_io_iter iter; - size_t done =3D 0; =20 if (!ublk_advance_io_iter(req, &iter, offset)) return 0; =20 - while (iov_iter_count(uiter) && iter.bio) { - unsigned nr_pages; - ssize_t len; - size_t off; - int i; - - len =3D iov_iter_get_pages2(uiter, iter.pages, - iov_iter_count(uiter), - UBLK_MAX_PIN_PAGES, &off); - if (len <=3D 0) - return done; - - ublk_copy_io_pages(&iter, len, off, dir); - nr_pages =3D DIV_ROUND_UP(len + off, PAGE_SIZE); - for (i =3D 0; i < nr_pages; i++) { - if (dir =3D=3D ITER_DEST) - set_page_dirty(iter.pages[i]); - put_page(iter.pages[i]); - } - done +=3D len; - } - - return done; + return ublk_copy_io_pages(&iter, uiter, dir); } =20 static inline bool ublk_need_map_req(const struct request *req) { return ublk_rq_has_data(req) && req_op(req) =3D=3D REQ_OP_WRITE; --=20 2.45.2 From nobody Fri Dec 19 14:35:34 2025 Received: from mail-wr1-f102.google.com (mail-wr1-f102.google.com [209.85.221.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 97835346FBA for ; Wed, 5 Nov 2025 20:28:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.102 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374517; cv=none; b=p2E5P1+ntyM80NM0aFkm/AaqgPDdAfk40CaR3pKGBU5/3qAEiQt7lJVqDbM86VdiRdvMaRCTGcez7NrQKRRopNF5uFcBlx1M+LWJzKIrQGD4mrHGwd+gfr9rMzE3UkfOTgvAsqpObg5ccF9qbl6s4FvaPBi9lVYuOsV78X/yz5Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374517; c=relaxed/simple; bh=ZNQBWF5E/kCd66vnCIEh1pwOhtP6knbpleihD38HzT8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=svXMZwblX64BQryIYJumB+tCQ+5iW3DjkVfV3W5+CI/HBqhfMRoKRBMBdIob6NZbSxLkWoBiKpcmv9MD5F1xzmzwShja8Hl1SBZ2L0+nOCC2Uq3L9DUKDbogAYwW0JNusB1SiBQ3PlOb39goGDtmWgUlB06e++DssRklVYg+510= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com; spf=fail smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=EIOjqxkG; arc=none smtp.client-ip=209.85.221.102 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="EIOjqxkG" Received: by mail-wr1-f102.google.com with SMTP id ffacd0b85a97d-429bfa1d3f8so27904f8f.2 for ; Wed, 05 Nov 2025 12:28:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1762374514; x=1762979314; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=piuezMo/+nJvsS63m8MZriLez8iEvJ7btVwHROt2ogA=; b=EIOjqxkGKrivEbXDb9Hzqr+XKgCZu9uLXAO52ajIC1d5bU5D+wKp8UPN/eTqZhUoh2 jVwjznhDeZgD5NtVc75BcA68M1csx86CihXWniV9cwT4Zx7k4tN7B2zR/CFQRavzOOt+ +vAn2TExCUJcCbb1b7egXbmLVa+oC+nAeuiDXnh71N16FtrltrvQugt1BkU+IgHOJxGf ydDxP14jYMripXPKDke84T4JHvEIbjiqurMgNCsIH/wdhTdgtvKLBrxl2Gq+EuSqaQIz TyRbpv7WBYXsUOQUr9CnJ5Q12AXnGS/VmgCyMaEw4KI3tXLRRw5YVdVZwRP5ILHGt4jU KJnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762374514; x=1762979314; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=piuezMo/+nJvsS63m8MZriLez8iEvJ7btVwHROt2ogA=; b=k3j/u6spKpkUbZRZtSTgNVEP1LYnFU+mNGUZUOBXtJBbcIMbgC43mIMu7zjlFKsGJE q+3hq8oEHSh0svS1THhCMLmr9cnXYNzz21gzzOe+bGi293gAnbo5rP966cbF5A+iR8vc j7sopN/1m8EmOFZd6+sURw7AJvEr4rksaW9W+PcrCcf3dlaSUBymrDQMgqfGf9qdrX8w Nyhwo0AWJjJOkfjAwtagf7BjhPw+pYLbTEkXytH9Bs7KXE5OziXITIjIdKrfOvYeC1n0 9SFCjW8KB278UrF4/yxCsuVivwheKtBJk/oOFjh+rHa/kdY1Wjl1dwEOwZgqQw+e1rGl vBlQ== X-Forwarded-Encrypted: i=1; AJvYcCWHLJEdu2F7cji2j3I8s31NEE9VLKrplec36JncP8ILzWYQ8dmEiO0NVkpj59s5cGnJvGYEr9SBdbKGkR4=@vger.kernel.org X-Gm-Message-State: AOJu0YwB1+Z/XECSfPurqofEOmNAlFY+nl6s8IdLDsETn5C07vRvrqeS GVNANbQOB7rnGyrrQYOJp1LrpcRIhs+zX5M9XGTkRlNn/sTA59vJ3NN9Ug0ly2upOSYTk8EBksq GY6yMO7y2lGaULfR0YcQ42Z8PLfFhkfEyvWzW X-Gm-Gg: ASbGncuMXPWmimyrLWYQNyP153Cl8S7zvwp4WM+QmrYizgOu7xrIMyz3O5Ami+0rRlD r5MLam0BKoOQ9zCDo/x71pOST/u9Qwdo8rZDJOE6uXVvNSs68U2ByKljDnL6F6jEaDQ3tBICvAq A4sEGxYBC6v+bGlpmKxTrqXVROjBgUpQ2yA8Fob24Qvk+MPeJ9U1jDPcg9iqH1n0a8cTjT0G8w+ jhc6UZygih/KkPyBzWZpmK7nH1grx+lFkDhYtk4wgrjNPwrC0cRwNI5LLjod5CWMJZ9D4wXB9ZF WbZJUtW6cL6lLHao5ousBGA+zReCaaTSHyhfBYKYyiVspS7GyD/RqYBKV6f+Xno5p+j2KqbDC+/ Q/hthZGQr5obg2uDnhwy3tRwiLW0rfiQ= X-Google-Smtp-Source: AGHT+IF3HJ9NqzHMzyJZYtlBawCznKJ3NSlqmqxl8ss4jfLNnF2OITyYlaRDgdmSnz2QY8Aul6AHUCVP6eiM X-Received: by 2002:a5d:64c9:0:b0:429:b697:1fa with SMTP id ffacd0b85a97d-429e32c53fcmr2205430f8f.2.1762374513766; Wed, 05 Nov 2025 12:28:33 -0800 (PST) Received: from c7-smtp-2023.dev.purestorage.com ([208.88.159.128]) by smtp-relay.gmail.com with ESMTPS id ffacd0b85a97d-429eb3d499bsm24593f8f.19.2025.11.05.12.28.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Nov 2025 12:28:33 -0800 (PST) X-Relaying-Domain: purestorage.com Received: from dev-csander.dev.purestorage.com (dev-csander.dev.purestorage.com [10.7.70.37]) by c7-smtp-2023.dev.purestorage.com (Postfix) with ESMTP id E416934057A; Wed, 5 Nov 2025 13:28:30 -0700 (MST) Received: by dev-csander.dev.purestorage.com (Postfix, from userid 1557716354) id 10591E413BB; Wed, 5 Nov 2025 13:28:31 -0700 (MST) From: Caleb Sander Mateos To: Ming Lei , Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Caleb Sander Mateos Subject: [PATCH v2 2/2] ublk: use rq_for_each_bvec() for user copy Date: Wed, 5 Nov 2025 13:28:23 -0700 Message-ID: <20251105202823.2198194-3-csander@purestorage.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20251105202823.2198194-1-csander@purestorage.com> References: <20251105202823.2198194-1-csander@purestorage.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" ublk_advance_io_iter() and ublk_copy_io_pages() currently open-code the iteration over request's bvecs. Switch to the rq_for_each_bvec() macro provided by blk-mq to avoid reaching into the bio internals and simplify the code. Unlike bio_iter_iovec(), rq_for_each_bvec() can return multi-page bvecs. So switch from copy_{to,from}_iter() to copy_page_{to,from}_iter() to map and copy each page in the bvec. Suggested-by: Ming Lei Signed-off-by: Caleb Sander Mateos --- drivers/block/ublk_drv.c | 78 ++++++++++++---------------------------- 1 file changed, 23 insertions(+), 55 deletions(-) diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c index 40eee3e15a4c..929d40fe0250 100644 --- a/drivers/block/ublk_drv.c +++ b/drivers/block/ublk_drv.c @@ -911,81 +911,49 @@ static const struct block_device_operations ub_fops = =3D { .open =3D ublk_open, .free_disk =3D ublk_free_disk, .report_zones =3D ublk_report_zones, }; =20 -struct ublk_io_iter { - struct bio *bio; - struct bvec_iter iter; -}; - -/* return how many bytes are copied */ -static size_t ublk_copy_io_pages(struct ublk_io_iter *data, - struct iov_iter *uiter, int dir) +/* + * Copy data between request pages and io_iter, and 'offset' + * is the start point of linear offset of request. + */ +static size_t ublk_copy_user_pages(const struct request *req, + unsigned offset, struct iov_iter *uiter, int dir) { + struct req_iterator iter; + struct bio_vec bv; size_t done =3D 0; =20 - for (;;) { - struct bio_vec bv =3D bio_iter_iovec(data->bio, data->iter); - void *bv_buf =3D bvec_kmap_local(&bv); + rq_for_each_bvec(bv, req, iter) { size_t copied; =20 + if (offset >=3D bv.bv_len) { + offset -=3D bv.bv_len; + continue; + } + + bv.bv_offset +=3D offset; + bv.bv_len -=3D offset; + bv.bv_page +=3D bv.bv_offset / PAGE_SIZE; + bv.bv_offset %=3D PAGE_SIZE; if (dir =3D=3D ITER_DEST) - copied =3D copy_to_iter(bv_buf, bv.bv_len, uiter); + copied =3D copy_page_to_iter( + bv.bv_page, bv.bv_offset, bv.bv_len, uiter); else - copied =3D copy_from_iter(bv_buf, bv.bv_len, uiter); - - kunmap_local(bv_buf); + copied =3D copy_page_from_iter( + bv.bv_page, bv.bv_offset, bv.bv_len, uiter); =20 done +=3D copied; if (copied < bv.bv_len) break; =20 - /* advance bio */ - bio_advance_iter_single(data->bio, &data->iter, copied); - if (!data->iter.bi_size) { - data->bio =3D data->bio->bi_next; - if (data->bio =3D=3D NULL) - break; - data->iter =3D data->bio->bi_iter; - } + offset =3D 0; } return done; } =20 -static bool ublk_advance_io_iter(const struct request *req, - struct ublk_io_iter *iter, unsigned int offset) -{ - struct bio *bio =3D req->bio; - - for_each_bio(bio) { - if (bio->bi_iter.bi_size > offset) { - iter->bio =3D bio; - iter->iter =3D bio->bi_iter; - bio_advance_iter(iter->bio, &iter->iter, offset); - return true; - } - offset -=3D bio->bi_iter.bi_size; - } - return false; -} - -/* - * Copy data between request pages and io_iter, and 'offset' - * is the start point of linear offset of request. - */ -static size_t ublk_copy_user_pages(const struct request *req, - unsigned offset, struct iov_iter *uiter, int dir) -{ - struct ublk_io_iter iter; - - if (!ublk_advance_io_iter(req, &iter, offset)) - return 0; - - return ublk_copy_io_pages(&iter, uiter, dir); -} - static inline bool ublk_need_map_req(const struct request *req) { return ublk_rq_has_data(req) && req_op(req) =3D=3D REQ_OP_WRITE; } =20 --=20 2.45.2