From nobody Wed Dec 17 20:57:40 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54E34C04FE1 for ; Wed, 16 Aug 2023 12:09:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244973AbjHPMJF (ORCPT ); Wed, 16 Aug 2023 08:09:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49406 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244958AbjHPMIl (ORCPT ); Wed, 16 Aug 2023 08:08:41 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E3392135 for ; Wed, 16 Aug 2023 05:08:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1692187679; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+ROhK//9C7npYS4YCKPkIYJubo9kN7OrTey+LSWwnT4=; b=dvblIB8YLF+4vHCugyj5QLcNkYrPB0onxNfq1KxFI7HaLTTRzT1klfmTKr6GSE7YA2Vn8k k9uDSGpmvy43U0YNWRac2nXgL9trE7trnVX+8sqFBLwlDXvm6D33UzBptFkew/KagtWkep MiS9R9Vr7iM8iBeTiEZ6RBpf9abKuAg= Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-474-TlHIBK7IPzKzraEWQSsN4A-1; Wed, 16 Aug 2023 08:07:56 -0400 X-MC-Unique: TlHIBK7IPzKzraEWQSsN4A-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 942513801BCE; Wed, 16 Aug 2023 12:07:55 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.13]) by smtp.corp.redhat.com (Postfix) with ESMTP id D18C3C15BAD; Wed, 16 Aug 2023 12:07:52 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Hellwig Subject: [PATCH v3 1/2] iov_iter: Convert iterate*() to inline funcs Date: Wed, 16 Aug 2023 13:07:40 +0100 Message-ID: <20230816120741.534415-2-dhowells@redhat.com> In-Reply-To: <20230816120741.534415-1-dhowells@redhat.com> References: <20230816120741.534415-1-dhowells@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Convert the iov_iter iteration macros to inline functions to make the code easier to follow. The functions are marked __always_inline as we don't want to end up with indirect calls in the code. This, however, leaves dealing with ->copy_mc in an awkard situation since the step function (memcpy_from_iter_mc()) needs to test the flag in the iterator, but isn't passed the iterator. This will be dealt with in a follow-up patch. The variable names in the per-type iterator functions have been harmonised as much as possible and made clearer as to the variable purpose. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig , cc: Christian Brauner , cc: Matthew Wilcox , cc: Linus Torvalds cc: David Laight cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org Link: https://lore.kernel.org/r/3710261.1691764329@warthog.procyon.org.uk/ = # v1 Link: https://lore.kernel.org/r/855.1692047347@warthog.procyon.org.uk/ # v2 --- Notes: Changes =3D=3D=3D=3D=3D=3D=3D ver #3) - Use min_t(size_t,) not min() to avoid a warning on Hexagon. - Inline all the step functions. =20 ver #2) - Rebased on top of Willy's changes in linux-next. - Change the checksum argument to the iteration functions to be a gene= ral void* and use it to pass iter->copy_mc flag to memcpy_from_iter_mc()= to avoid using a function pointer. - Arrange the end of the iterate_*() functions to look the same to give the optimiser the best chance. - Make iterate_and_advance() a wrapper around iterate_and_advance2(). - Adjust iterate_and_advance2() to use if-else-if-else-if-else rather = than switch(), to put ITER_BVEC before KVEC and to mark UBUF and IOVEC as likely(). - Move "iter->count +=3D progress" into iterate_and_advance2() from the iterate functions. - Mark a number of the iterator helpers with __always_inline. - Fix _copy_from_iter_flushcache() to use memcpy_from_iter_flushcache() not memcpy_from_iter(). lib/iov_iter.c | 612 ++++++++++++++++++++++++++++++------------------- 1 file changed, 371 insertions(+), 241 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 424737045b97..378da0cb3069 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -14,188 +14,264 @@ #include #include =20 -/* covers ubuf and kbuf alike */ -#define iterate_buf(i, n, base, len, off, __p, STEP) { \ - size_t __maybe_unused off =3D 0; \ - len =3D n; \ - base =3D __p + i->iov_offset; \ - len -=3D (STEP); \ - i->iov_offset +=3D len; \ - n =3D len; \ -} - -/* covers iovec and kvec alike */ -#define iterate_iovec(i, n, base, len, off, __p, STEP) { \ - size_t off =3D 0; \ - size_t skip =3D i->iov_offset; \ - do { \ - len =3D min(n, __p->iov_len - skip); \ - if (likely(len)) { \ - base =3D __p->iov_base + skip; \ - len -=3D (STEP); \ - off +=3D len; \ - skip +=3D len; \ - n -=3D len; \ - if (skip < __p->iov_len) \ - break; \ - } \ - __p++; \ - skip =3D 0; \ - } while (n); \ - i->iov_offset =3D skip; \ - n =3D off; \ -} - -#define iterate_bvec(i, n, base, len, off, p, STEP) { \ - size_t off =3D 0; \ - unsigned skip =3D i->iov_offset; \ - while (n) { \ - unsigned offset =3D p->bv_offset + skip; \ - unsigned left; \ - void *kaddr =3D kmap_local_page(p->bv_page + \ - offset / PAGE_SIZE); \ - base =3D kaddr + offset % PAGE_SIZE; \ - len =3D min(min(n, (size_t)(p->bv_len - skip)), \ - (size_t)(PAGE_SIZE - offset % PAGE_SIZE)); \ - left =3D (STEP); \ - kunmap_local(kaddr); \ - len -=3D left; \ - off +=3D len; \ - skip +=3D len; \ - if (skip =3D=3D p->bv_len) { \ - skip =3D 0; \ - p++; \ - } \ - n -=3D len; \ - if (left) \ - break; \ - } \ - i->iov_offset =3D skip; \ - n =3D off; \ -} - -#define iterate_xarray(i, n, base, len, __off, STEP) { \ - __label__ __out; \ - size_t __off =3D 0; \ - struct folio *folio; \ - loff_t start =3D i->xarray_start + i->iov_offset; \ - pgoff_t index =3D start / PAGE_SIZE; \ - XA_STATE(xas, i->xarray, index); \ - \ - len =3D PAGE_SIZE - offset_in_page(start); \ - rcu_read_lock(); \ - xas_for_each(&xas, folio, ULONG_MAX) { \ - unsigned left; \ - size_t offset; \ - if (xas_retry(&xas, folio)) \ - continue; \ - if (WARN_ON(xa_is_value(folio))) \ - break; \ - if (WARN_ON(folio_test_hugetlb(folio))) \ - break; \ - offset =3D offset_in_folio(folio, start + __off); \ - while (offset < folio_size(folio)) { \ - base =3D kmap_local_folio(folio, offset); \ - len =3D min(n, len); \ - left =3D (STEP); \ - kunmap_local(base); \ - len -=3D left; \ - __off +=3D len; \ - n -=3D len; \ - if (left || n =3D=3D 0) \ - goto __out; \ - offset +=3D len; \ - len =3D PAGE_SIZE; \ - } \ - } \ -__out: \ - rcu_read_unlock(); \ - i->iov_offset +=3D __off; \ - n =3D __off; \ -} - -#define __iterate_and_advance(i, n, base, len, off, I, K) { \ - if (unlikely(i->count < n)) \ - n =3D i->count; \ - if (likely(n)) { \ - if (likely(iter_is_ubuf(i))) { \ - void __user *base; \ - size_t len; \ - iterate_buf(i, n, base, len, off, \ - i->ubuf, (I)) \ - } else if (likely(iter_is_iovec(i))) { \ - const struct iovec *iov =3D iter_iov(i); \ - void __user *base; \ - size_t len; \ - iterate_iovec(i, n, base, len, off, \ - iov, (I)) \ - i->nr_segs -=3D iov - iter_iov(i); \ - i->__iov =3D iov; \ - } else if (iov_iter_is_bvec(i)) { \ - const struct bio_vec *bvec =3D i->bvec; \ - void *base; \ - size_t len; \ - iterate_bvec(i, n, base, len, off, \ - bvec, (K)) \ - i->nr_segs -=3D bvec - i->bvec; \ - i->bvec =3D bvec; \ - } else if (iov_iter_is_kvec(i)) { \ - const struct kvec *kvec =3D i->kvec; \ - void *base; \ - size_t len; \ - iterate_iovec(i, n, base, len, off, \ - kvec, (K)) \ - i->nr_segs -=3D kvec - i->kvec; \ - i->kvec =3D kvec; \ - } else if (iov_iter_is_xarray(i)) { \ - void *base; \ - size_t len; \ - iterate_xarray(i, n, base, len, off, \ - (K)) \ - } \ - i->count -=3D n; \ - } \ -} -#define iterate_and_advance(i, n, base, len, off, I, K) \ - __iterate_and_advance(i, n, base, len, off, I, ((void)(K),0)) - -static int copyout(void __user *to, const void *from, size_t n) +typedef size_t (*iov_step_f)(void *iter_base, size_t progress, size_t len, + void *priv, void *priv2); +typedef size_t (*iov_ustep_f)(void __user *iter_base, size_t progress, siz= e_t len, + void *priv, void *priv2); + +static __always_inline +size_t iterate_ubuf(struct iov_iter *iter, size_t len, void *priv, void *p= riv2, + iov_ustep_f step) { - if (should_fail_usercopy()) - return n; - if (access_ok(to, n)) { - instrument_copy_to_user(to, from, n); - n =3D raw_copy_to_user(to, from, n); + void __user *base =3D iter->ubuf; + size_t progress =3D 0, remain; + + remain =3D step(base + iter->iov_offset, 0, len, priv, priv2); + progress =3D len - remain; + iter->iov_offset +=3D progress; + return progress; +} + +static __always_inline +size_t iterate_iovec(struct iov_iter *iter, size_t len, void *priv, void *= priv2, + iov_ustep_f step) +{ + const struct iovec *p =3D iter->__iov; + size_t progress =3D 0, skip =3D iter->iov_offset; + + do { + size_t remain, consumed; + size_t part =3D min(len, p->iov_len - skip); + + if (likely(part)) { + remain =3D step(p->iov_base + skip, progress, part, priv, priv2); + consumed =3D part - remain; + progress +=3D consumed; + skip +=3D consumed; + len -=3D consumed; + if (skip < p->iov_len) + break; + } + p++; + skip =3D 0; + } while (len); + + iter->__iov =3D p; + iter->nr_segs -=3D p - iter->__iov; + iter->iov_offset =3D skip; + return progress; +} + +static __always_inline +size_t iterate_kvec(struct iov_iter *iter, size_t len, void *priv, void *p= riv2, + iov_step_f step) +{ + const struct kvec *p =3D iter->kvec; + size_t progress =3D 0, skip =3D iter->iov_offset; + + do { + size_t remain, consumed; + size_t part =3D min(len, p->iov_len - skip); + + if (likely(part)) { + remain =3D step(p->iov_base + skip, progress, part, priv, priv2); + consumed =3D part - remain; + progress +=3D consumed; + skip +=3D consumed; + len -=3D consumed; + if (skip < p->iov_len) + break; + } + p++; + skip =3D 0; + } while (len); + + iter->kvec =3D p; + iter->nr_segs -=3D p - iter->kvec; + iter->iov_offset =3D skip; + return progress; +} + +static __always_inline +size_t iterate_bvec(struct iov_iter *iter, size_t len, void *priv, void *p= riv2, + iov_step_f step) +{ + const struct bio_vec *p =3D iter->bvec; + size_t progress =3D 0, skip =3D iter->iov_offset; + + do { + size_t remain, consumed; + size_t offset =3D p->bv_offset + skip, part; + void *kaddr =3D kmap_local_page(p->bv_page + offset / PAGE_SIZE); + + part =3D min3(len, + (size_t)(p->bv_len - skip), + (size_t)(PAGE_SIZE - offset % PAGE_SIZE)); + remain =3D step(kaddr + offset % PAGE_SIZE, progress, part, priv, priv2); + kunmap_local(kaddr); + consumed =3D part - remain; + len -=3D consumed; + progress +=3D consumed; + skip +=3D consumed; + if (skip >=3D p->bv_len) { + skip =3D 0; + p++; + } + if (remain) + break; + } while (len); + + iter->bvec =3D p; + iter->nr_segs -=3D p - iter->bvec; + iter->iov_offset =3D skip; + return progress; +} + +static __always_inline +size_t iterate_xarray(struct iov_iter *iter, size_t len, void *priv, void = *priv2, + iov_step_f step) +{ + struct folio *folio; + size_t progress =3D 0; + loff_t start =3D iter->xarray_start + iter->iov_offset; + pgoff_t index =3D start / PAGE_SIZE; + XA_STATE(xas, iter->xarray, index); + + rcu_read_lock(); + xas_for_each(&xas, folio, ULONG_MAX) { + size_t remain, consumed, offset, part, flen; + + if (xas_retry(&xas, folio)) + continue; + if (WARN_ON(xa_is_value(folio))) + break; + if (WARN_ON(folio_test_hugetlb(folio))) + break; + + offset =3D offset_in_folio(folio, start); + flen =3D min(folio_size(folio) - offset, len); + start +=3D flen; + + while (flen) { + void *base =3D kmap_local_folio(folio, offset); + + part =3D min_t(size_t, flen, + PAGE_SIZE - offset_in_page(offset)); + remain =3D step(base, progress, part, priv, priv2); + kunmap_local(base); + + consumed =3D part - remain; + progress +=3D consumed; + len -=3D consumed; + + if (remain || len =3D=3D 0) + goto out; + flen -=3D consumed; + offset +=3D consumed; + } } - return n; + +out: + rcu_read_unlock(); + iter->iov_offset +=3D progress; + return progress; } =20 -static int copyout_nofault(void __user *to, const void *from, size_t n) +static __always_inline +size_t iterate_and_advance2(struct iov_iter *iter, size_t len, void *priv, + void *priv2, iov_ustep_f ustep, iov_step_f step) { - long res; + size_t progress; =20 + if (unlikely(iter->count < len)) + len =3D iter->count; + if (unlikely(!len)) + return 0; + + if (likely(iter_is_ubuf(iter))) + progress =3D iterate_ubuf(iter, len, priv, priv2, ustep); + else if (likely(iter_is_iovec(iter))) + progress =3D iterate_iovec(iter, len, priv, priv2, ustep); + else if (iov_iter_is_bvec(iter)) + progress =3D iterate_bvec(iter, len, priv, priv2, step); + else if (iov_iter_is_kvec(iter)) + progress =3D iterate_kvec(iter, len, priv, priv2, step); + else if (iov_iter_is_xarray(iter)) + progress =3D iterate_xarray(iter, len, priv, priv2, step); + else + progress =3D len; + iter->count -=3D progress; + return progress; +} + +static __always_inline +size_t iterate_and_advance(struct iov_iter *iter, size_t len, void *priv, + iov_ustep_f ustep, iov_step_f step) +{ + return iterate_and_advance2(iter, len, priv, NULL, ustep, step); +} + +static __always_inline +size_t copy_to_user_iter(void __user *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ if (should_fail_usercopy()) - return n; + return len; + if (access_ok(iter_to, len)) { + from +=3D progress; + instrument_copy_to_user(iter_to, from, len); + len =3D raw_copy_to_user(iter_to, from, len); + } + return len; +} =20 - res =3D copy_to_user_nofault(to, from, n); +static __always_inline +size_t copy_to_user_iter_nofault(void __user *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + ssize_t res; =20 - return res < 0 ? n : res; + if (should_fail_usercopy()) + return len; + + from +=3D progress; + res =3D copy_to_user_nofault(iter_to, from, len); + return res < 0 ? len : res; } =20 -static int copyin(void *to, const void __user *from, size_t n) +static __always_inline +size_t copy_from_user_iter(void __user *iter_from, size_t progress, + size_t len, void *to, void *priv2) { - size_t res =3D n; + size_t res =3D len; =20 if (should_fail_usercopy()) - return n; - if (access_ok(from, n)) { - instrument_copy_from_user_before(to, from, n); - res =3D raw_copy_from_user(to, from, n); - instrument_copy_from_user_after(to, from, n, res); + return len; + if (access_ok(iter_from, len)) { + to +=3D progress; + instrument_copy_from_user_before(to, iter_from, len); + res =3D raw_copy_from_user(to, iter_from, len); + instrument_copy_from_user_after(to, iter_from, len, res); } return res; } =20 +static __always_inline +size_t memcpy_to_iter(void *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + memcpy(iter_to, from + progress, len); + return 0; +} + +static __always_inline +size_t memcpy_from_iter(void *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + memcpy(to + progress, iter_from, len); + return 0; +} + /* * fault_in_iov_iter_readable - fault in iov iterator for reading * @i: iterator @@ -313,23 +389,29 @@ size_t _copy_to_iter(const void *addr, size_t bytes, = struct iov_iter *i) return 0; if (user_backed_iter(i)) might_fault(); - iterate_and_advance(i, bytes, base, len, off, - copyout(base, addr + off, len), - memcpy(base, addr + off, len) - ) - - return bytes; + return iterate_and_advance(i, bytes, (void *)addr, + copy_to_user_iter, memcpy_to_iter); } EXPORT_SYMBOL(_copy_to_iter); =20 #ifdef CONFIG_ARCH_HAS_COPY_MC -static int copyout_mc(void __user *to, const void *from, size_t n) -{ - if (access_ok(to, n)) { - instrument_copy_to_user(to, from, n); - n =3D copy_mc_to_user((__force void *) to, from, n); +static __always_inline +size_t copy_to_user_iter_mc(void __user *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + if (access_ok(iter_to, len)) { + from +=3D progress; + instrument_copy_to_user(iter_to, from, len); + len =3D copy_mc_to_user(iter_to, from, len); } - return n; + return len; +} + +static __always_inline +size_t memcpy_to_iter_mc(void *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + return copy_mc_to_kernel(iter_to, from + progress, len); } =20 /** @@ -362,22 +444,20 @@ size_t _copy_mc_to_iter(const void *addr, size_t byte= s, struct iov_iter *i) return 0; if (user_backed_iter(i)) might_fault(); - __iterate_and_advance(i, bytes, base, len, off, - copyout_mc(base, addr + off, len), - copy_mc_to_kernel(base, addr + off, len) - ) - - return bytes; + return iterate_and_advance(i, bytes, (void *)addr, + copy_to_user_iter_mc, memcpy_to_iter_mc); } EXPORT_SYMBOL_GPL(_copy_mc_to_iter); #endif /* CONFIG_ARCH_HAS_COPY_MC */ =20 -static void *memcpy_from_iter(struct iov_iter *i, void *to, const void *fr= om, - size_t size) +static size_t memcpy_from_iter_mc(void *iter_from, size_t progress, + size_t len, void *to, void *priv2) { - if (iov_iter_is_copy_mc(i)) - return (void *)copy_mc_to_kernel(to, from, size); - return memcpy(to, from, size); + struct iov_iter *iter =3D priv2; + + if (iov_iter_is_copy_mc(iter)) + return copy_mc_to_kernel(to + progress, iter_from, len); + return memcpy_from_iter(iter_from, progress, len, to, priv2); } =20 size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) @@ -387,30 +467,46 @@ size_t _copy_from_iter(void *addr, size_t bytes, stru= ct iov_iter *i) =20 if (user_backed_iter(i)) might_fault(); - iterate_and_advance(i, bytes, base, len, off, - copyin(addr + off, base, len), - memcpy_from_iter(i, addr + off, base, len) - ) - - return bytes; + return iterate_and_advance2(i, bytes, addr, i, + copy_from_user_iter, + memcpy_from_iter_mc); } EXPORT_SYMBOL(_copy_from_iter); =20 +static __always_inline +size_t copy_from_user_iter_nocache(void __user *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + return __copy_from_user_inatomic_nocache(to + progress, iter_from, len); +} + size_t _copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *= i) { if (WARN_ON_ONCE(!i->data_source)) return 0; =20 - iterate_and_advance(i, bytes, base, len, off, - __copy_from_user_inatomic_nocache(addr + off, base, len), - memcpy(addr + off, base, len) - ) - - return bytes; + return iterate_and_advance(i, bytes, addr, + copy_from_user_iter_nocache, + memcpy_from_iter); } EXPORT_SYMBOL(_copy_from_iter_nocache); =20 #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE +static __always_inline +size_t copy_from_user_iter_flushcache(void __user *iter_from, size_t progr= ess, + size_t len, void *to, void *priv2) +{ + return __copy_from_user_flushcache(to + progress, iter_from, len); +} + +static __always_inline +size_t memcpy_from_iter_flushcache(void *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + memcpy_flushcache(to + progress, iter_from, len); + return 0; +} + /** * _copy_from_iter_flushcache - write destination through cpu cache * @addr: destination kernel address @@ -432,12 +528,9 @@ size_t _copy_from_iter_flushcache(void *addr, size_t b= ytes, struct iov_iter *i) if (WARN_ON_ONCE(!i->data_source)) return 0; =20 - iterate_and_advance(i, bytes, base, len, off, - __copy_from_user_flushcache(addr + off, base, len), - memcpy_flushcache(addr + off, base, len) - ) - - return bytes; + return iterate_and_advance(i, bytes, addr, + copy_from_user_iter_flushcache, + memcpy_from_iter_flushcache); } EXPORT_SYMBOL_GPL(_copy_from_iter_flushcache); #endif @@ -509,10 +602,9 @@ size_t copy_page_to_iter_nofault(struct page *page, un= signed offset, size_t byte void *kaddr =3D kmap_local_page(page); size_t n =3D min(bytes, (size_t)PAGE_SIZE - offset); =20 - iterate_and_advance(i, n, base, len, off, - copyout_nofault(base, kaddr + offset + off, len), - memcpy(base, kaddr + offset + off, len) - ) + n =3D iterate_and_advance(i, bytes, kaddr, + copy_to_user_iter_nofault, + memcpy_to_iter); kunmap_local(kaddr); res +=3D n; bytes -=3D n; @@ -555,14 +647,25 @@ size_t copy_page_from_iter(struct page *page, size_t = offset, size_t bytes, } EXPORT_SYMBOL(copy_page_from_iter); =20 -size_t iov_iter_zero(size_t bytes, struct iov_iter *i) +static __always_inline +size_t zero_to_user_iter(void __user *iter_to, size_t progress, + size_t len, void *priv, void *priv2) { - iterate_and_advance(i, bytes, base, len, count, - clear_user(base, len), - memset(base, 0, len) - ) + return clear_user(iter_to, len); +} =20 - return bytes; +static __always_inline +size_t zero_to_iter(void *iter_to, size_t progress, + size_t len, void *priv, void *priv2) +{ + memset(iter_to, 0, len); + return 0; +} + +size_t iov_iter_zero(size_t bytes, struct iov_iter *i) +{ + return iterate_and_advance(i, bytes, NULL, + zero_to_user_iter, zero_to_iter); } EXPORT_SYMBOL(iov_iter_zero); =20 @@ -587,10 +690,9 @@ size_t copy_page_from_iter_atomic(struct page *page, s= ize_t offset, } =20 p =3D kmap_atomic(page) + offset; - iterate_and_advance(i, n, base, len, off, - copyin(p + off, base, len), - memcpy_from_iter(i, p + off, base, len) - ) + n =3D iterate_and_advance2(i, n, p, i, + copy_from_user_iter, + memcpy_from_iter_mc); kunmap_atomic(p); copied +=3D n; offset +=3D n; @@ -1181,32 +1283,64 @@ ssize_t iov_iter_get_pages_alloc2(struct iov_iter *= i, } EXPORT_SYMBOL(iov_iter_get_pages_alloc2); =20 +static __always_inline +size_t copy_from_user_iter_csum(void __user *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + __wsum next, *csum =3D priv2; + + next =3D csum_and_copy_from_user(iter_from, to + progress, len); + *csum =3D csum_block_add(*csum, next, progress); + return next ? 0 : len; +} + +static __always_inline +size_t memcpy_from_iter_csum(void *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + __wsum *csum =3D priv2; + + *csum =3D csum_and_memcpy(to + progress, iter_from, len, *csum, progress); + return 0; +} + size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i) { - __wsum sum, next; - sum =3D *csum; if (WARN_ON_ONCE(!i->data_source)) return 0; - - iterate_and_advance(i, bytes, base, len, off, ({ - next =3D csum_and_copy_from_user(base, addr + off, len); - sum =3D csum_block_add(sum, next, off); - next ? 0 : len; - }), ({ - sum =3D csum_and_memcpy(addr + off, base, len, sum, off); - }) - ) - *csum =3D sum; - return bytes; + return iterate_and_advance2(i, bytes, addr, csum, + copy_from_user_iter_csum, + memcpy_from_iter_csum); } EXPORT_SYMBOL(csum_and_copy_from_iter); =20 +static __always_inline +size_t copy_to_user_iter_csum(void __user *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + __wsum next, *csum =3D priv2; + + next =3D csum_and_copy_to_user(from + progress, iter_to, len); + *csum =3D csum_block_add(*csum, next, progress); + return next ? 0 : len; +} + +static __always_inline +size_t memcpy_to_iter_csum(void *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + __wsum *csum =3D priv2; + + *csum =3D csum_and_memcpy(iter_to, from + progress, len, *csum, progress); + return 0; +} + size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *_csstat= e, struct iov_iter *i) { struct csum_state *csstate =3D _csstate; - __wsum sum, next; + __wsum sum; =20 if (WARN_ON_ONCE(i->data_source)) return 0; @@ -1220,14 +1354,10 @@ size_t csum_and_copy_to_iter(const void *addr, size= _t bytes, void *_csstate, } =20 sum =3D csum_shift(csstate->csum, csstate->off); - iterate_and_advance(i, bytes, base, len, off, ({ - next =3D csum_and_copy_to_user(addr + off, base, len); - sum =3D csum_block_add(sum, next, off); - next ? 0 : len; - }), ({ - sum =3D csum_and_memcpy(base, addr + off, len, sum, off); - }) - ) +=09 + bytes =3D iterate_and_advance2(i, bytes, (void *)addr, &sum, + copy_to_user_iter_csum, + memcpy_to_iter_csum); csstate->csum =3D csum_shift(sum, csstate->off); csstate->off +=3D bytes; return bytes; From nobody Wed Dec 17 20:57:40 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64C4CC05052 for ; Wed, 16 Aug 2023 12:09:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244979AbjHPMJG (ORCPT ); Wed, 16 Aug 2023 08:09:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49444 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244956AbjHPMIl (ORCPT ); Wed, 16 Aug 2023 08:08:41 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 71751268F for ; Wed, 16 Aug 2023 05:08:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1692187682; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wO3T+Kgn8IqvgBuNhCcF+dgBv+VoPvzdg3BdkPf9/3I=; b=S+6bCzf0g5DmHq4V6IkpR/MQoVOzs1RhjEqNZ+6FMTZnhGUbB/q78D0ofezlR5YmoCBZ89 PukHRsSVwtzzj39pAfTSugfJS05+pEJkAUpC/suX0wejSIRrGOuZvMvxCp2Lb6v+JQfNjk y5Y8MJfGyBDyfvCwKzzbq+emsmIZZ1w= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-517-ow2g6_iFP9a7UhFz3Anncw-1; Wed, 16 Aug 2023 08:07:59 -0400 X-MC-Unique: ow2g6_iFP9a7UhFz3Anncw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AF1CC101A53C; Wed, 16 Aug 2023 12:07:58 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.13]) by smtp.corp.redhat.com (Postfix) with ESMTP id D03D91121314; Wed, 16 Aug 2023 12:07:56 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 2/2] iov_iter: Don't deal with iter->copy_mc in memcpy_from_iter_mc() Date: Wed, 16 Aug 2023 13:07:41 +0100 Message-ID: <20230816120741.534415-3-dhowells@redhat.com> In-Reply-To: <20230816120741.534415-1-dhowells@redhat.com> References: <20230816120741.534415-1-dhowells@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" iter->copy_mc is only used with a bvec iterator and only by dump_emit_page() in fs/coredump.c so rather than handle this in memcpy_from_iter_mc() where it is checked repeatedly by _copy_from_iter() a= nd copy_page_from_iter_atomic(), --- lib/iov_iter.c | 39 +++++++++++++++++++++++++++------------ 1 file changed, 27 insertions(+), 12 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 378da0cb3069..33febccadd9d 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -450,14 +450,33 @@ size_t _copy_mc_to_iter(const void *addr, size_t byte= s, struct iov_iter *i) EXPORT_SYMBOL_GPL(_copy_mc_to_iter); #endif /* CONFIG_ARCH_HAS_COPY_MC */ =20 -static size_t memcpy_from_iter_mc(void *iter_from, size_t progress, - size_t len, void *to, void *priv2) +static __always_inline +size_t memcpy_from_iter_mc(void *iter_from, size_t progress, + size_t len, void *to, void *priv2) { - struct iov_iter *iter =3D priv2; + return copy_mc_to_kernel(to + progress, iter_from, len); +} =20 - if (iov_iter_is_copy_mc(iter)) - return copy_mc_to_kernel(to + progress, iter_from, len); - return memcpy_from_iter(iter_from, progress, len, to, priv2); +static size_t __copy_from_iter_mc(void *addr, size_t bytes, struct iov_ite= r *i) +{ + size_t progress; + + if (unlikely(i->count < bytes)) + bytes =3D i->count; + if (unlikely(!bytes)) + return 0; + progress =3D iterate_bvec(i, bytes, addr, NULL, memcpy_from_iter_mc); + i->count -=3D progress; + return progress; +} + +static __always_inline +size_t __copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) +{ + if (unlikely(iov_iter_is_copy_mc(i))) + return __copy_from_iter_mc(addr, bytes, i); + return iterate_and_advance(i, bytes, addr, + copy_from_user_iter, memcpy_from_iter); } =20 size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) @@ -467,9 +486,7 @@ size_t _copy_from_iter(void *addr, size_t bytes, struct= iov_iter *i) =20 if (user_backed_iter(i)) might_fault(); - return iterate_and_advance2(i, bytes, addr, i, - copy_from_user_iter, - memcpy_from_iter_mc); + return __copy_from_iter(addr, bytes, i); } EXPORT_SYMBOL(_copy_from_iter); =20 @@ -690,9 +707,7 @@ size_t copy_page_from_iter_atomic(struct page *page, si= ze_t offset, } =20 p =3D kmap_atomic(page) + offset; - n =3D iterate_and_advance2(i, n, p, i, - copy_from_user_iter, - memcpy_from_iter_mc); + __copy_from_iter(p, n, i); kunmap_atomic(p); copied +=3D n; offset +=3D n;