When copying data from buffers that are mapped to user space, or from
buffers that are used for dma, it is impossible to guarantee absence of
concurrent memory operations on those buffers. Copying data to/from `Page`
from/to these buffers would be undefined behavior if regular memcpy
operations are used.

The operation can be made well defined, if the buffers that potentially
observe racy operations can be said to exist outside of any Rust
allocation. For this to be true, the kernel must only interact with the
buffers using raw volatile reads and writes.

Add methods on `Page` to read and write the contents using volatile
operations.

Also improve clarity by specifying additional requirements on
`read_raw`/`write_raw` methods regarding concurrent operations on involved
buffers.

Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
---
 rust/kernel/page.rs | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)

diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs
index 432fc0297d4a8..6568a0d3b3baa 100644
--- a/rust/kernel/page.rs
+++ b/rust/kernel/page.rs
@@ -7,6 +7,7 @@
     bindings,
     error::code::*,
     error::Result,
+    ffi::c_void,
     uaccess::UserSliceReader,
 };
 use core::{
@@ -260,6 +261,8 @@ fn with_pointer_into_page<T>(
     /// # Safety
     ///
     /// * Callers must ensure that `dst` is valid for writing `len` bytes.
+    /// * Callers must ensure that there are no other concurrent reads or writes to/from the
+    ///   destination memory region.
     /// * Callers must ensure that this call does not race with a write to the same page that
     ///   overlaps with this read.
     pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
@@ -274,6 +277,30 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result
         })
     }
 
+    /// Maps the page and reads from it into the given IO memory region using volatile memory
+    /// operations.
+    ///
+    /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
+    /// outside of the page, then this call returns [`EINVAL`].
+    ///
+    /// # Safety
+    /// Callers must ensure that:
+    ///
+    /// * The destination memory region is outside of any Rust memory allocation.
+    /// * The destination memory region is writable.
+    /// * This call does not race with a write to the same source page that overlaps with this read.
+    pub unsafe fn read_raw_toio(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
+        self.with_pointer_into_page(offset, len, move |src| {
+            // SAFETY: If `with_pointer_into_page` calls into this closure, then
+            // it has performed a bounds check and guarantees that `src` is
+            // valid for `len` bytes.
+            //
+            // There caller guarantees that there is no data race at the source.
+            unsafe { bindings::memcpy_toio(dst.cast::<c_void>(), src.cast::<c_void>(), len) };
+            Ok(())
+        })
+    }
+
     /// Maps the page and writes into it from the given buffer.
     ///
     /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
@@ -282,6 +309,7 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result
     /// # Safety
     ///
     /// * Callers must ensure that `src` is valid for reading `len` bytes.
+    /// * Callers must ensure that there are no concurrent writes to the source memory region.
     /// * Callers must ensure that this call does not race with a read or write to the same page
     ///   that overlaps with this write.
     pub unsafe fn write_raw(&self, src: *const u8, offset: usize, len: usize) -> Result {
@@ -295,6 +323,31 @@ pub unsafe fn write_raw(&self, src: *const u8, offset: usize, len: usize) -> Res
         })
     }
 
+    /// Maps the page and writes into it from the given IO memory region using volatile memory
+    /// operations.
+    ///
+    /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
+    /// outside of the page, then this call returns [`EINVAL`].
+    ///
+    /// # Safety
+    ///
+    /// Callers must ensure that:
+    ///
+    /// * The source memory region is outside of any Rust memory allocation.
+    /// * The source memory region is readable.
+    /// * This call does not race with a read or write to the same destination page that overlaps
+    ///   with this write.
+    pub unsafe fn write_raw_fromio(&self, src: *const u8, offset: usize, len: usize) -> Result {
+        self.with_pointer_into_page(offset, len, move |dst| {
+            // SAFETY: If `with_pointer_into_page` calls into this closure, then it has performed a
+            // bounds check and guarantees that `dst` is valid for `len` bytes.
+            //
+            // There caller guarantees that there is no data race at the destination.
+            unsafe { bindings::memcpy_fromio(dst.cast::<c_void>(), src.cast::<c_void>(), len) };
+            Ok(())
+        })
+    }
+
     /// Maps the page and zeroes the given slice.
     ///
     /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes

---
base-commit: 63804fed149a6750ffd28610c5c1c98cce6bd377
change-id: 20260130-page-volatile-io-05ff595507d3

Best regards,
-- 
Andreas Hindborg <a.hindborg@kernel.org>

On Fri Jan 30, 2026 at 12:33 PM GMT, Andreas Hindborg wrote:
> When copying data from buffers that are mapped to user space, or from
> buffers that are used for dma, it is impossible to guarantee absence of
> concurrent memory operations on those buffers. Copying data to/from `Page`
> from/to these buffers would be undefined behavior if regular memcpy
> operations are used.
>
> The operation can be made well defined, if the buffers that potentially
> observe racy operations can be said to exist outside of any Rust
> allocation. For this to be true, the kernel must only interact with the
> buffers using raw volatile reads and writes.
>
> Add methods on `Page` to read and write the contents using volatile
> operations.
>
> Also improve clarity by specifying additional requirements on
> `read_raw`/`write_raw` methods regarding concurrent operations on involved
> buffers.
>
> Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
> ---
>  rust/kernel/page.rs | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 53 insertions(+)
>
> diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs
> index 432fc0297d4a8..6568a0d3b3baa 100644
> --- a/rust/kernel/page.rs
> +++ b/rust/kernel/page.rs
> @@ -7,6 +7,7 @@
>      bindings,
>      error::code::*,
>      error::Result,
> +    ffi::c_void,
>      uaccess::UserSliceReader,
>  };
>  use core::{
> @@ -260,6 +261,8 @@ fn with_pointer_into_page<T>(
>      /// # Safety
>      ///
>      /// * Callers must ensure that `dst` is valid for writing `len` bytes.
> +    /// * Callers must ensure that there are no other concurrent reads or writes to/from the
> +    ///   destination memory region.
>      /// * Callers must ensure that this call does not race with a write to the same page that
>      ///   overlaps with this read.
>      pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
> @@ -274,6 +277,30 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result
>          })
>      }
>  
> +    /// Maps the page and reads from it into the given IO memory region using volatile memory
> +    /// operations.
> +    ///
> +    /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
> +    /// outside of the page, then this call returns [`EINVAL`].
> +    ///
> +    /// # Safety
> +    /// Callers must ensure that:
> +    ///
> +    /// * The destination memory region is outside of any Rust memory allocation.
> +    /// * The destination memory region is writable.
> +    /// * This call does not race with a write to the same source page that overlaps with this read.
> +    pub unsafe fn read_raw_toio(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
> +        self.with_pointer_into_page(offset, len, move |src| {
> +            // SAFETY: If `with_pointer_into_page` calls into this closure, then
> +            // it has performed a bounds check and guarantees that `src` is
> +            // valid for `len` bytes.
> +            //
> +            // There caller guarantees that there is no data race at the source.
> +            unsafe { bindings::memcpy_toio(dst.cast::<c_void>(), src.cast::<c_void>(), len) };

I feel that this should be a generic utility that integrates with our IO infra
that allows you to copy/from IO to a slice.

Best,
Gary

> +            Ok(())
> +        })
> +    }
> +
>      /// Maps the page and writes into it from the given buffer.
>      ///
>      /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
> @@ -282,6 +309,7 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result
>      /// # Safety
>      ///
>      /// * Callers must ensure that `src` is valid for reading `len` bytes.
> +    /// * Callers must ensure that there are no concurrent writes to the source memory region.
>      /// * Callers must ensure that this call does not race with a read or write to the same page
>      ///   that overlaps with this write.
>      pub unsafe fn write_raw(&self, src: *const u8, offset: usize, len: usize) -> Result {
> @@ -295,6 +323,31 @@ pub unsafe fn write_raw(&self, src: *const u8, offset: usize, len: usize) -> Res
>          })
>      }
>  
> +    /// Maps the page and writes into it from the given IO memory region using volatile memory
> +    /// operations.
> +    ///
> +    /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
> +    /// outside of the page, then this call returns [`EINVAL`].
> +    ///
> +    /// # Safety
> +    ///
> +    /// Callers must ensure that:
> +    ///
> +    /// * The source memory region is outside of any Rust memory allocation.
> +    /// * The source memory region is readable.
> +    /// * This call does not race with a read or write to the same destination page that overlaps
> +    ///   with this write.
> +    pub unsafe fn write_raw_fromio(&self, src: *const u8, offset: usize, len: usize) -> Result {
> +        self.with_pointer_into_page(offset, len, move |dst| {
> +            // SAFETY: If `with_pointer_into_page` calls into this closure, then it has performed a
> +            // bounds check and guarantees that `dst` is valid for `len` bytes.
> +            //
> +            // There caller guarantees that there is no data race at the destination.
> +            unsafe { bindings::memcpy_fromio(dst.cast::<c_void>(), src.cast::<c_void>(), len) };
> +            Ok(())
> +        })
> +    }
> +
>      /// Maps the page and zeroes the given slice.
>      ///
>      /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
>
> ---
> base-commit: 63804fed149a6750ffd28610c5c1c98cce6bd377
> change-id: 20260130-page-volatile-io-05ff595507d3
>
> Best regards,

"Gary Guo" <gary@garyguo.net> writes:

> On Fri Jan 30, 2026 at 12:33 PM GMT, Andreas Hindborg wrote:
>> When copying data from buffers that are mapped to user space, or from
>> buffers that are used for dma, it is impossible to guarantee absence of
>> concurrent memory operations on those buffers. Copying data to/from `Page`
>> from/to these buffers would be undefined behavior if regular memcpy
>> operations are used.
>>
>> The operation can be made well defined, if the buffers that potentially
>> observe racy operations can be said to exist outside of any Rust
>> allocation. For this to be true, the kernel must only interact with the
>> buffers using raw volatile reads and writes.
>>
>> Add methods on `Page` to read and write the contents using volatile
>> operations.
>>
>> Also improve clarity by specifying additional requirements on
>> `read_raw`/`write_raw` methods regarding concurrent operations on involved
>> buffers.
>>
>> Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
>> ---
>>  rust/kernel/page.rs | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 53 insertions(+)
>>
>> diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs
>> index 432fc0297d4a8..6568a0d3b3baa 100644
>> --- a/rust/kernel/page.rs
>> +++ b/rust/kernel/page.rs
>> @@ -7,6 +7,7 @@
>>      bindings,
>>      error::code::*,
>>      error::Result,
>> +    ffi::c_void,
>>      uaccess::UserSliceReader,
>>  };
>>  use core::{
>> @@ -260,6 +261,8 @@ fn with_pointer_into_page<T>(
>>      /// # Safety
>>      ///
>>      /// * Callers must ensure that `dst` is valid for writing `len` bytes.
>> +    /// * Callers must ensure that there are no other concurrent reads or writes to/from the
>> +    ///   destination memory region.
>>      /// * Callers must ensure that this call does not race with a write to the same page that
>>      ///   overlaps with this read.
>>      pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
>> @@ -274,6 +277,30 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result
>>          })
>>      }
>>
>> +    /// Maps the page and reads from it into the given IO memory region using volatile memory
>> +    /// operations.
>> +    ///
>> +    /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
>> +    /// outside of the page, then this call returns [`EINVAL`].
>> +    ///
>> +    /// # Safety
>> +    /// Callers must ensure that:
>> +    ///
>> +    /// * The destination memory region is outside of any Rust memory allocation.
>> +    /// * The destination memory region is writable.
>> +    /// * This call does not race with a write to the same source page that overlaps with this read.
>> +    pub unsafe fn read_raw_toio(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
>> +        self.with_pointer_into_page(offset, len, move |src| {
>> +            // SAFETY: If `with_pointer_into_page` calls into this closure, then
>> +            // it has performed a bounds check and guarantees that `src` is
>> +            // valid for `len` bytes.
>> +            //
>> +            // There caller guarantees that there is no data race at the source.
>> +            unsafe { bindings::memcpy_toio(dst.cast::<c_void>(), src.cast::<c_void>(), len) };
>
> I feel that this should be a generic utility that integrates with our IO infra
> that allows you to copy/from IO to a slice.

While that might also be useful, for my particular use case I am copying
between two pages. One is mapped from user space, the other one is
allocated by a driver. No slices involved. Pasting for reference [1]:


    /// Copy data to the current page of this segment from `src_page`.
    ///
    /// Copies  `PAGE_SIZE - (self.offset() % PAGE_SIZE` bytes of data from `src_page` to this
    /// segment starting at `self.offset()` from offset `self.offset() % PAGE_SIZE`. This call
    /// will advance offset and reduce length of `self`.
    ///
    /// Returns the number of bytes copied.
    pub fn copy_from_page(&mut self, src_page: &Page, src_offset: usize) -> usize {
        let dst_offset = self.offset() % PAGE_SIZE;
        debug_assert!(src_offset <= PAGE_SIZE);
        let length = (PAGE_SIZE - dst_offset)
            .min(self.len() as usize)
            .min(PAGE_SIZE - src_offset);
        let page_idx = self.offset() / PAGE_SIZE;

        // SAFETY: self.bio_vec is valid and thus bv_page must be a valid
        // pointer to a `struct page`.
        let dst_page = unsafe { Page::from_raw(self.bio_vec.bv_page.add(page_idx)) };

        dst_page
            .with_pointer_into_page(dst_offset, length, |dst| {
                // SAFETY:
                // - If `with_pointer_into_page` calls this closure, then it has performed bounds
                //   checks and guarantees that `dst` is valid for `length` bytes.
                // - Since we have a shared reference to `src_page`, the read cannot race with any
                //   writes to `src_page`.
                unsafe { src_page.read_raw_toio(dst, src_offset, length) }
            })
            .expect("Assertion failure, bounds check failed.");

        self.advance(length as u32)
            .expect("Assertion failure, bounds check failed.");

        length



Best regards,
Andreas Hindborg

[1] https://github.com/metaspace/linux/blob/3e46e95f0707fa71259b1d241f689144ad61cc62/rust/kernel/block/bio/vec.rs#L142

On Fri Jan 30, 2026 at 1:48 PM GMT, Andreas Hindborg wrote:
> "Gary Guo" <gary@garyguo.net> writes:
>
>> On Fri Jan 30, 2026 at 12:33 PM GMT, Andreas Hindborg wrote:
>>> When copying data from buffers that are mapped to user space, or from
>>> buffers that are used for dma, it is impossible to guarantee absence of
>>> concurrent memory operations on those buffers. Copying data to/from `Page`
>>> from/to these buffers would be undefined behavior if regular memcpy
>>> operations are used.
>>>
>>> The operation can be made well defined, if the buffers that potentially
>>> observe racy operations can be said to exist outside of any Rust
>>> allocation. For this to be true, the kernel must only interact with the
>>> buffers using raw volatile reads and writes.
>>>
>>> Add methods on `Page` to read and write the contents using volatile
>>> operations.
>>>
>>> Also improve clarity by specifying additional requirements on
>>> `read_raw`/`write_raw` methods regarding concurrent operations on involved
>>> buffers.
>>>
>>> Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
>>> ---
>>>  rust/kernel/page.rs | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>  1 file changed, 53 insertions(+)
>>>
>>> diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs
>>> index 432fc0297d4a8..6568a0d3b3baa 100644
>>> --- a/rust/kernel/page.rs
>>> +++ b/rust/kernel/page.rs
>>> @@ -7,6 +7,7 @@
>>>      bindings,
>>>      error::code::*,
>>>      error::Result,
>>> +    ffi::c_void,
>>>      uaccess::UserSliceReader,
>>>  };
>>>  use core::{
>>> @@ -260,6 +261,8 @@ fn with_pointer_into_page<T>(
>>>      /// # Safety
>>>      ///
>>>      /// * Callers must ensure that `dst` is valid for writing `len` bytes.
>>> +    /// * Callers must ensure that there are no other concurrent reads or writes to/from the
>>> +    ///   destination memory region.
>>>      /// * Callers must ensure that this call does not race with a write to the same page that
>>>      ///   overlaps with this read.
>>>      pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
>>> @@ -274,6 +277,30 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result
>>>          })
>>>      }
>>>
>>> +    /// Maps the page and reads from it into the given IO memory region using volatile memory
>>> +    /// operations.
>>> +    ///
>>> +    /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
>>> +    /// outside of the page, then this call returns [`EINVAL`].
>>> +    ///
>>> +    /// # Safety
>>> +    /// Callers must ensure that:
>>> +    ///
>>> +    /// * The destination memory region is outside of any Rust memory allocation.
>>> +    /// * The destination memory region is writable.
>>> +    /// * This call does not race with a write to the same source page that overlaps with this read.
>>> +    pub unsafe fn read_raw_toio(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
>>> +        self.with_pointer_into_page(offset, len, move |src| {
>>> +            // SAFETY: If `with_pointer_into_page` calls into this closure, then
>>> +            // it has performed a bounds check and guarantees that `src` is
>>> +            // valid for `len` bytes.
>>> +            //
>>> +            // There caller guarantees that there is no data race at the source.
>>> +            unsafe { bindings::memcpy_toio(dst.cast::<c_void>(), src.cast::<c_void>(), len) };
>>
>> I feel that this should be a generic utility that integrates with our IO infra
>> that allows you to copy/from IO to a slice.
>
> While that might also be useful, for my particular use case I am copying
> between two pages. One is mapped from user space, the other one is
> allocated by a driver. No slices involved. Pasting for reference [1]:

Then what you need is a byte-wise atomic memcpy, not memcpy_{from,to}io.

Best,
Gary

>
>
>     /// Copy data to the current page of this segment from `src_page`.
>     ///
>     /// Copies  `PAGE_SIZE - (self.offset() % PAGE_SIZE` bytes of data from `src_page` to this
>     /// segment starting at `self.offset()` from offset `self.offset() % PAGE_SIZE`. This call
>     /// will advance offset and reduce length of `self`.
>     ///
>     /// Returns the number of bytes copied.
>     pub fn copy_from_page(&mut self, src_page: &Page, src_offset: usize) -> usize {
>         let dst_offset = self.offset() % PAGE_SIZE;
>         debug_assert!(src_offset <= PAGE_SIZE);
>         let length = (PAGE_SIZE - dst_offset)
>             .min(self.len() as usize)
>             .min(PAGE_SIZE - src_offset);
>         let page_idx = self.offset() / PAGE_SIZE;
>
>         // SAFETY: self.bio_vec is valid and thus bv_page must be a valid
>         // pointer to a `struct page`.
>         let dst_page = unsafe { Page::from_raw(self.bio_vec.bv_page.add(page_idx)) };
>
>         dst_page
>             .with_pointer_into_page(dst_offset, length, |dst| {
>                 // SAFETY:
>                 // - If `with_pointer_into_page` calls this closure, then it has performed bounds
>                 //   checks and guarantees that `dst` is valid for `length` bytes.
>                 // - Since we have a shared reference to `src_page`, the read cannot race with any
>                 //   writes to `src_page`.
>                 unsafe { src_page.read_raw_toio(dst, src_offset, length) }
>             })
>             .expect("Assertion failure, bounds check failed.");
>
>         self.advance(length as u32)
>             .expect("Assertion failure, bounds check failed.");
>
>         length
>
>
>
> Best regards,
> Andreas Hindborg
>
> [1] https://github.com/metaspace/linux/blob/3e46e95f0707fa71259b1d241f689144ad61cc62/rust/kernel/block/bio/vec.rs#L142

"Gary Guo" <gary@garyguo.net> writes:

> On Fri Jan 30, 2026 at 1:48 PM GMT, Andreas Hindborg wrote:
>> "Gary Guo" <gary@garyguo.net> writes:
>>
>>> On Fri Jan 30, 2026 at 12:33 PM GMT, Andreas Hindborg wrote:
>>>> When copying data from buffers that are mapped to user space, or from
>>>> buffers that are used for dma, it is impossible to guarantee absence of
>>>> concurrent memory operations on those buffers. Copying data to/from `Page`
>>>> from/to these buffers would be undefined behavior if regular memcpy
>>>> operations are used.
>>>>
>>>> The operation can be made well defined, if the buffers that potentially
>>>> observe racy operations can be said to exist outside of any Rust
>>>> allocation. For this to be true, the kernel must only interact with the
>>>> buffers using raw volatile reads and writes.
>>>>
>>>> Add methods on `Page` to read and write the contents using volatile
>>>> operations.
>>>>
>>>> Also improve clarity by specifying additional requirements on
>>>> `read_raw`/`write_raw` methods regarding concurrent operations on involved
>>>> buffers.
>>>>
>>>> Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
>>>> ---
>>>>  rust/kernel/page.rs | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>  1 file changed, 53 insertions(+)
>>>>
>>>> diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs
>>>> index 432fc0297d4a8..6568a0d3b3baa 100644
>>>> --- a/rust/kernel/page.rs
>>>> +++ b/rust/kernel/page.rs
>>>> @@ -7,6 +7,7 @@
>>>>      bindings,
>>>>      error::code::*,
>>>>      error::Result,
>>>> +    ffi::c_void,
>>>>      uaccess::UserSliceReader,
>>>>  };
>>>>  use core::{
>>>> @@ -260,6 +261,8 @@ fn with_pointer_into_page<T>(
>>>>      /// # Safety
>>>>      ///
>>>>      /// * Callers must ensure that `dst` is valid for writing `len` bytes.
>>>> +    /// * Callers must ensure that there are no other concurrent reads or writes to/from the
>>>> +    ///   destination memory region.
>>>>      /// * Callers must ensure that this call does not race with a write to the same page that
>>>>      ///   overlaps with this read.
>>>>      pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
>>>> @@ -274,6 +277,30 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result
>>>>          })
>>>>      }
>>>>
>>>> +    /// Maps the page and reads from it into the given IO memory region using volatile memory
>>>> +    /// operations.
>>>> +    ///
>>>> +    /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
>>>> +    /// outside of the page, then this call returns [`EINVAL`].
>>>> +    ///
>>>> +    /// # Safety
>>>> +    /// Callers must ensure that:
>>>> +    ///
>>>> +    /// * The destination memory region is outside of any Rust memory allocation.
>>>> +    /// * The destination memory region is writable.
>>>> +    /// * This call does not race with a write to the same source page that overlaps with this read.
>>>> +    pub unsafe fn read_raw_toio(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
>>>> +        self.with_pointer_into_page(offset, len, move |src| {
>>>> +            // SAFETY: If `with_pointer_into_page` calls into this closure, then
>>>> +            // it has performed a bounds check and guarantees that `src` is
>>>> +            // valid for `len` bytes.
>>>> +            //
>>>> +            // There caller guarantees that there is no data race at the source.
>>>> +            unsafe { bindings::memcpy_toio(dst.cast::<c_void>(), src.cast::<c_void>(), len) };
>>>
>>> I feel that this should be a generic utility that integrates with our IO infra
>>> that allows you to copy/from IO to a slice.
>>
>> While that might also be useful, for my particular use case I am copying
>> between two pages. One is mapped from user space, the other one is
>> allocated by a driver. No slices involved. Pasting for reference [1]:
>
> Then what you need is a byte-wise atomic memcpy, not memcpy_{from,to}io.

Can you elaborate on how you get to this requirement?


Best regards,
Andreas Hindborg

On Fri Jan 30, 2026 at 2:42 PM GMT, Andreas Hindborg wrote:
> "Gary Guo" <gary@garyguo.net> writes:
>
>> On Fri Jan 30, 2026 at 1:48 PM GMT, Andreas Hindborg wrote:
>>> "Gary Guo" <gary@garyguo.net> writes:
>>>
>>>> On Fri Jan 30, 2026 at 12:33 PM GMT, Andreas Hindborg wrote:
>>>>> When copying data from buffers that are mapped to user space, or from
>>>>> buffers that are used for dma, it is impossible to guarantee absence of
>>>>> concurrent memory operations on those buffers. Copying data to/from `Page`
>>>>> from/to these buffers would be undefined behavior if regular memcpy
>>>>> operations are used.
>>>>>
>>>>> The operation can be made well defined, if the buffers that potentially
>>>>> observe racy operations can be said to exist outside of any Rust
>>>>> allocation. For this to be true, the kernel must only interact with the
>>>>> buffers using raw volatile reads and writes.
>>>>>
>>>>> Add methods on `Page` to read and write the contents using volatile
>>>>> operations.
>>>>>
>>>>> Also improve clarity by specifying additional requirements on
>>>>> `read_raw`/`write_raw` methods regarding concurrent operations on involved
>>>>> buffers.
>>>>>
>>>>> Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
>>>>> ---
>>>>>  rust/kernel/page.rs | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>  1 file changed, 53 insertions(+)
>>>>>
>>>>> diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs
>>>>> index 432fc0297d4a8..6568a0d3b3baa 100644
>>>>> --- a/rust/kernel/page.rs
>>>>> +++ b/rust/kernel/page.rs
>>>>> @@ -7,6 +7,7 @@
>>>>>      bindings,
>>>>>      error::code::*,
>>>>>      error::Result,
>>>>> +    ffi::c_void,
>>>>>      uaccess::UserSliceReader,
>>>>>  };
>>>>>  use core::{
>>>>> @@ -260,6 +261,8 @@ fn with_pointer_into_page<T>(
>>>>>      /// # Safety
>>>>>      ///
>>>>>      /// * Callers must ensure that `dst` is valid for writing `len` bytes.
>>>>> +    /// * Callers must ensure that there are no other concurrent reads or writes to/from the
>>>>> +    ///   destination memory region.
>>>>>      /// * Callers must ensure that this call does not race with a write to the same page that
>>>>>      ///   overlaps with this read.
>>>>>      pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
>>>>> @@ -274,6 +277,30 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result
>>>>>          })
>>>>>      }
>>>>>
>>>>> +    /// Maps the page and reads from it into the given IO memory region using volatile memory
>>>>> +    /// operations.
>>>>> +    ///
>>>>> +    /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
>>>>> +    /// outside of the page, then this call returns [`EINVAL`].
>>>>> +    ///
>>>>> +    /// # Safety
>>>>> +    /// Callers must ensure that:
>>>>> +    ///
>>>>> +    /// * The destination memory region is outside of any Rust memory allocation.
>>>>> +    /// * The destination memory region is writable.
>>>>> +    /// * This call does not race with a write to the same source page that overlaps with this read.
>>>>> +    pub unsafe fn read_raw_toio(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
>>>>> +        self.with_pointer_into_page(offset, len, move |src| {
>>>>> +            // SAFETY: If `with_pointer_into_page` calls into this closure, then
>>>>> +            // it has performed a bounds check and guarantees that `src` is
>>>>> +            // valid for `len` bytes.
>>>>> +            //
>>>>> +            // There caller guarantees that there is no data race at the source.
>>>>> +            unsafe { bindings::memcpy_toio(dst.cast::<c_void>(), src.cast::<c_void>(), len) };
>>>>
>>>> I feel that this should be a generic utility that integrates with our IO infra
>>>> that allows you to copy/from IO to a slice.
>>>
>>> While that might also be useful, for my particular use case I am copying
>>> between two pages. One is mapped from user space, the other one is
>>> allocated by a driver. No slices involved. Pasting for reference [1]:
>>
>> Then what you need is a byte-wise atomic memcpy, not memcpy_{from,to}io.
>
> Can you elaborate on how you get to this requirement?

Memory that is possibly mapped into userspace is still normal memory, it is not
I/O. I/O accessors (and IO memcpy) are specifically used for MMIO, and you
should not be using them for userspace memory.

For memory that can be mutated from userspace you can just treat them as a
potentially concurrent accessor hence all accesses should be using atomic. When
tearing is acceptable, byte-wise atomic is sufficient.

Best,
Gary

>
>
> Best regards,
> Andreas Hindborg

"Gary Guo" <gary@garyguo.net> writes:

> On Fri Jan 30, 2026 at 2:42 PM GMT, Andreas Hindborg wrote:
>> "Gary Guo" <gary@garyguo.net> writes:
>>
>>> On Fri Jan 30, 2026 at 1:48 PM GMT, Andreas Hindborg wrote:
>>>> "Gary Guo" <gary@garyguo.net> writes:
>>>>
>>>>> On Fri Jan 30, 2026 at 12:33 PM GMT, Andreas Hindborg wrote:
>>>>>> When copying data from buffers that are mapped to user space, or from
>>>>>> buffers that are used for dma, it is impossible to guarantee absence of
>>>>>> concurrent memory operations on those buffers. Copying data to/from `Page`
>>>>>> from/to these buffers would be undefined behavior if regular memcpy
>>>>>> operations are used.
>>>>>>
>>>>>> The operation can be made well defined, if the buffers that potentially
>>>>>> observe racy operations can be said to exist outside of any Rust
>>>>>> allocation. For this to be true, the kernel must only interact with the
>>>>>> buffers using raw volatile reads and writes.
>>>>>>
>>>>>> Add methods on `Page` to read and write the contents using volatile
>>>>>> operations.
>>>>>>
>>>>>> Also improve clarity by specifying additional requirements on
>>>>>> `read_raw`/`write_raw` methods regarding concurrent operations on involved
>>>>>> buffers.
>>>>>>
>>>>>> Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
>>>>>> ---
>>>>>>  rust/kernel/page.rs | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>  1 file changed, 53 insertions(+)
>>>>>>
>>>>>> diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs
>>>>>> index 432fc0297d4a8..6568a0d3b3baa 100644
>>>>>> --- a/rust/kernel/page.rs
>>>>>> +++ b/rust/kernel/page.rs
>>>>>> @@ -7,6 +7,7 @@
>>>>>>      bindings,
>>>>>>      error::code::*,
>>>>>>      error::Result,
>>>>>> +    ffi::c_void,
>>>>>>      uaccess::UserSliceReader,
>>>>>>  };
>>>>>>  use core::{
>>>>>> @@ -260,6 +261,8 @@ fn with_pointer_into_page<T>(
>>>>>>      /// # Safety
>>>>>>      ///
>>>>>>      /// * Callers must ensure that `dst` is valid for writing `len` bytes.
>>>>>> +    /// * Callers must ensure that there are no other concurrent reads or writes to/from the
>>>>>> +    ///   destination memory region.
>>>>>>      /// * Callers must ensure that this call does not race with a write to the same page that
>>>>>>      ///   overlaps with this read.
>>>>>>      pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
>>>>>> @@ -274,6 +277,30 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result
>>>>>>          })
>>>>>>      }
>>>>>>
>>>>>> +    /// Maps the page and reads from it into the given IO memory region using volatile memory
>>>>>> +    /// operations.
>>>>>> +    ///
>>>>>> +    /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
>>>>>> +    /// outside of the page, then this call returns [`EINVAL`].
>>>>>> +    ///
>>>>>> +    /// # Safety
>>>>>> +    /// Callers must ensure that:
>>>>>> +    ///
>>>>>> +    /// * The destination memory region is outside of any Rust memory allocation.
>>>>>> +    /// * The destination memory region is writable.
>>>>>> +    /// * This call does not race with a write to the same source page that overlaps with this read.
>>>>>> +    pub unsafe fn read_raw_toio(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
>>>>>> +        self.with_pointer_into_page(offset, len, move |src| {
>>>>>> +            // SAFETY: If `with_pointer_into_page` calls into this closure, then
>>>>>> +            // it has performed a bounds check and guarantees that `src` is
>>>>>> +            // valid for `len` bytes.
>>>>>> +            //
>>>>>> +            // There caller guarantees that there is no data race at the source.
>>>>>> +            unsafe { bindings::memcpy_toio(dst.cast::<c_void>(), src.cast::<c_void>(), len) };
>>>>>
>>>>> I feel that this should be a generic utility that integrates with our IO infra
>>>>> that allows you to copy/from IO to a slice.
>>>>
>>>> While that might also be useful, for my particular use case I am copying
>>>> between two pages. One is mapped from user space, the other one is
>>>> allocated by a driver. No slices involved. Pasting for reference [1]:
>>>
>>> Then what you need is a byte-wise atomic memcpy, not memcpy_{from,to}io.
>>
>> Can you elaborate on how you get to this requirement?
>
> Memory that is possibly mapped into userspace is still normal memory, it is not
> I/O. I/O accessors (and IO memcpy) are specifically used for MMIO, and you
> should not be using them for userspace memory.
>
> For memory that can be mutated from userspace you can just treat them as a
> potentially concurrent accessor hence all accesses should be using atomic. When
> tearing is acceptable, byte-wise atomic is sufficient.

I would treat them the same as DMA regions and MMIO regions. As these
regions are outside of any Rust allocation, if we never make references
to them and if we only operate on them with volatile operations,
behavior of the copy operations like these are defined, as far as I
understand.

In the last discussions we had on this, the conclusion was to use
`volatile_copy_memory` whenever that is available, or write a volatile
copy function in assembly.

Using memcpy_{from,to}io is the latter solution. These functions are
simply volatile memcpy implemented in assembly.

There is nothing special about MMIO. These functions are name as they
are because they are useful for MMIO.


Best regards,
Andreas Hindborg

On Fri Jan 30, 2026 at 3:23 PM GMT, Andreas Hindborg wrote:
> "Gary Guo" <gary@garyguo.net> writes:
>
>> On Fri Jan 30, 2026 at 2:42 PM GMT, Andreas Hindborg wrote:
>>> "Gary Guo" <gary@garyguo.net> writes:
>>>
>>>> On Fri Jan 30, 2026 at 1:48 PM GMT, Andreas Hindborg wrote:
>>>>> "Gary Guo" <gary@garyguo.net> writes:
>>>>>
>>>>>> On Fri Jan 30, 2026 at 12:33 PM GMT, Andreas Hindborg wrote:
>>>>>>> When copying data from buffers that are mapped to user space, or from
>>>>>>> buffers that are used for dma, it is impossible to guarantee absence of
>>>>>>> concurrent memory operations on those buffers. Copying data to/from `Page`
>>>>>>> from/to these buffers would be undefined behavior if regular memcpy
>>>>>>> operations are used.
>>>>>>>
>>>>>>> The operation can be made well defined, if the buffers that potentially
>>>>>>> observe racy operations can be said to exist outside of any Rust
>>>>>>> allocation. For this to be true, the kernel must only interact with the
>>>>>>> buffers using raw volatile reads and writes.
>>>>>>>
>>>>>>> Add methods on `Page` to read and write the contents using volatile
>>>>>>> operations.
>>>>>>>
>>>>>>> Also improve clarity by specifying additional requirements on
>>>>>>> `read_raw`/`write_raw` methods regarding concurrent operations on involved
>>>>>>> buffers.
>>>>>>>
>>>>>>> Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
>>>>>>> ---
>>>>>>>  rust/kernel/page.rs | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>  1 file changed, 53 insertions(+)
>>>>>>>
>>>>>>> diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs
>>>>>>> index 432fc0297d4a8..6568a0d3b3baa 100644
>>>>>>> --- a/rust/kernel/page.rs
>>>>>>> +++ b/rust/kernel/page.rs
>>>>>>> @@ -7,6 +7,7 @@
>>>>>>>      bindings,
>>>>>>>      error::code::*,
>>>>>>>      error::Result,
>>>>>>> +    ffi::c_void,
>>>>>>>      uaccess::UserSliceReader,
>>>>>>>  };
>>>>>>>  use core::{
>>>>>>> @@ -260,6 +261,8 @@ fn with_pointer_into_page<T>(
>>>>>>>      /// # Safety
>>>>>>>      ///
>>>>>>>      /// * Callers must ensure that `dst` is valid for writing `len` bytes.
>>>>>>> +    /// * Callers must ensure that there are no other concurrent reads or writes to/from the
>>>>>>> +    ///   destination memory region.
>>>>>>>      /// * Callers must ensure that this call does not race with a write to the same page that
>>>>>>>      ///   overlaps with this read.
>>>>>>>      pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
>>>>>>> @@ -274,6 +277,30 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result
>>>>>>>          })
>>>>>>>      }
>>>>>>>
>>>>>>> +    /// Maps the page and reads from it into the given IO memory region using volatile memory
>>>>>>> +    /// operations.
>>>>>>> +    ///
>>>>>>> +    /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
>>>>>>> +    /// outside of the page, then this call returns [`EINVAL`].
>>>>>>> +    ///
>>>>>>> +    /// # Safety
>>>>>>> +    /// Callers must ensure that:
>>>>>>> +    ///
>>>>>>> +    /// * The destination memory region is outside of any Rust memory allocation.
>>>>>>> +    /// * The destination memory region is writable.
>>>>>>> +    /// * This call does not race with a write to the same source page that overlaps with this read.
>>>>>>> +    pub unsafe fn read_raw_toio(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
>>>>>>> +        self.with_pointer_into_page(offset, len, move |src| {
>>>>>>> +            // SAFETY: If `with_pointer_into_page` calls into this closure, then
>>>>>>> +            // it has performed a bounds check and guarantees that `src` is
>>>>>>> +            // valid for `len` bytes.
>>>>>>> +            //
>>>>>>> +            // There caller guarantees that there is no data race at the source.
>>>>>>> +            unsafe { bindings::memcpy_toio(dst.cast::<c_void>(), src.cast::<c_void>(), len) };
>>>>>>
>>>>>> I feel that this should be a generic utility that integrates with our IO infra
>>>>>> that allows you to copy/from IO to a slice.
>>>>>
>>>>> While that might also be useful, for my particular use case I am copying
>>>>> between two pages. One is mapped from user space, the other one is
>>>>> allocated by a driver. No slices involved. Pasting for reference [1]:
>>>>
>>>> Then what you need is a byte-wise atomic memcpy, not memcpy_{from,to}io.
>>>
>>> Can you elaborate on how you get to this requirement?
>>
>> Memory that is possibly mapped into userspace is still normal memory, it is not
>> I/O. I/O accessors (and IO memcpy) are specifically used for MMIO, and you
>> should not be using them for userspace memory.
>>
>> For memory that can be mutated from userspace you can just treat them as a
>> potentially concurrent accessor hence all accesses should be using atomic. When
>> tearing is acceptable, byte-wise atomic is sufficient.
>
> I would treat them the same as DMA regions and MMIO regions. As these
> regions are outside of any Rust allocation, if we never make references
> to them and if we only operate on them with volatile operations,
> behavior of the copy operations like these are defined, as far as I
> understand.

I don't find the argument about these being outside Rust allocation very useful.
Apart from MMIO, I view all other types of memory still within purview of the
abstract machine.

>
> In the last discussions we had on this, the conclusion was to use
> `volatile_copy_memory` whenever that is available, or write a volatile
> copy function in assembly.
>
> Using memcpy_{from,to}io is the latter solution. These functions are
> simply volatile memcpy implemented in assembly.
>
> There is nothing special about MMIO. These functions are name as they
> are because they are useful for MMIO.

No. MMIO are really special. A few architectures require them to be accessed
completely differently compared to normal memory. We also have things like
INDIRECT_IOMEM. memory_{from,to}io are special as they use MMIO accessor such as
readb to perform access on the __iomem pointer. They should not be mixed with
normal memory. They must be treated as if they're from a completely separate
address space.

Normal memory vs DMA vs MMIO are all distinct, and this is demonstrated by the
different types of barriers needed to order things correctly for each type of
memory region.

Userspace-mapped memory (that is also mapped in the kernel space, not __user) is
the least special one out of these. They could practically share all atomic infra
available for the kernel, hence the suggestion of using byte-wise atomic memcpy.

Best,
Gary

>
>
> Best regards,
> Andreas Hindborg

"Gary Guo" <gary@garyguo.net> writes:

> On Fri Jan 30, 2026 at 3:23 PM GMT, Andreas Hindborg wrote:
>> "Gary Guo" <gary@garyguo.net> writes:
>>
>>> On Fri Jan 30, 2026 at 2:42 PM GMT, Andreas Hindborg wrote:
>>>> "Gary Guo" <gary@garyguo.net> writes:
>>>>
>>>>> On Fri Jan 30, 2026 at 1:48 PM GMT, Andreas Hindborg wrote:
>>>>>> "Gary Guo" <gary@garyguo.net> writes:
>>>>>>
>>>>>>> On Fri Jan 30, 2026 at 12:33 PM GMT, Andreas Hindborg wrote:
>>>>>>>> When copying data from buffers that are mapped to user space, or from
>>>>>>>> buffers that are used for dma, it is impossible to guarantee absence of
>>>>>>>> concurrent memory operations on those buffers. Copying data to/from `Page`
>>>>>>>> from/to these buffers would be undefined behavior if regular memcpy
>>>>>>>> operations are used.
>>>>>>>>
>>>>>>>> The operation can be made well defined, if the buffers that potentially
>>>>>>>> observe racy operations can be said to exist outside of any Rust
>>>>>>>> allocation. For this to be true, the kernel must only interact with the
>>>>>>>> buffers using raw volatile reads and writes.
>>>>>>>>
>>>>>>>> Add methods on `Page` to read and write the contents using volatile
>>>>>>>> operations.
>>>>>>>>
>>>>>>>> Also improve clarity by specifying additional requirements on
>>>>>>>> `read_raw`/`write_raw` methods regarding concurrent operations on involved
>>>>>>>> buffers.
>>>>>>>>
>>>>>>>> Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
>>>>>>>> ---
>>>>>>>>  rust/kernel/page.rs | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>  1 file changed, 53 insertions(+)
>>>>>>>>
>>>>>>>> diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs
>>>>>>>> index 432fc0297d4a8..6568a0d3b3baa 100644
>>>>>>>> --- a/rust/kernel/page.rs
>>>>>>>> +++ b/rust/kernel/page.rs
>>>>>>>> @@ -7,6 +7,7 @@
>>>>>>>>      bindings,
>>>>>>>>      error::code::*,
>>>>>>>>      error::Result,
>>>>>>>> +    ffi::c_void,
>>>>>>>>      uaccess::UserSliceReader,
>>>>>>>>  };
>>>>>>>>  use core::{
>>>>>>>> @@ -260,6 +261,8 @@ fn with_pointer_into_page<T>(
>>>>>>>>      /// # Safety
>>>>>>>>      ///
>>>>>>>>      /// * Callers must ensure that `dst` is valid for writing `len` bytes.
>>>>>>>> +    /// * Callers must ensure that there are no other concurrent reads or writes to/from the
>>>>>>>> +    ///   destination memory region.
>>>>>>>>      /// * Callers must ensure that this call does not race with a write to the same page that
>>>>>>>>      ///   overlaps with this read.
>>>>>>>>      pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
>>>>>>>> @@ -274,6 +277,30 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result
>>>>>>>>          })
>>>>>>>>      }
>>>>>>>>
>>>>>>>> +    /// Maps the page and reads from it into the given IO memory region using volatile memory
>>>>>>>> +    /// operations.
>>>>>>>> +    ///
>>>>>>>> +    /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
>>>>>>>> +    /// outside of the page, then this call returns [`EINVAL`].
>>>>>>>> +    ///
>>>>>>>> +    /// # Safety
>>>>>>>> +    /// Callers must ensure that:
>>>>>>>> +    ///
>>>>>>>> +    /// * The destination memory region is outside of any Rust memory allocation.
>>>>>>>> +    /// * The destination memory region is writable.
>>>>>>>> +    /// * This call does not race with a write to the same source page that overlaps with this read.
>>>>>>>> +    pub unsafe fn read_raw_toio(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
>>>>>>>> +        self.with_pointer_into_page(offset, len, move |src| {
>>>>>>>> +            // SAFETY: If `with_pointer_into_page` calls into this closure, then
>>>>>>>> +            // it has performed a bounds check and guarantees that `src` is
>>>>>>>> +            // valid for `len` bytes.
>>>>>>>> +            //
>>>>>>>> +            // There caller guarantees that there is no data race at the source.
>>>>>>>> +            unsafe { bindings::memcpy_toio(dst.cast::<c_void>(), src.cast::<c_void>(), len) };
>>>>>>>
>>>>>>> I feel that this should be a generic utility that integrates with our IO infra
>>>>>>> that allows you to copy/from IO to a slice.
>>>>>>
>>>>>> While that might also be useful, for my particular use case I am copying
>>>>>> between two pages. One is mapped from user space, the other one is
>>>>>> allocated by a driver. No slices involved. Pasting for reference [1]:
>>>>>
>>>>> Then what you need is a byte-wise atomic memcpy, not memcpy_{from,to}io.
>>>>
>>>> Can you elaborate on how you get to this requirement?
>>>
>>> Memory that is possibly mapped into userspace is still normal memory, it is not
>>> I/O. I/O accessors (and IO memcpy) are specifically used for MMIO, and you
>>> should not be using them for userspace memory.
>>>
>>> For memory that can be mutated from userspace you can just treat them as a
>>> potentially concurrent accessor hence all accesses should be using atomic. When
>>> tearing is acceptable, byte-wise atomic is sufficient.
>>
>> I would treat them the same as DMA regions and MMIO regions. As these
>> regions are outside of any Rust allocation, if we never make references
>> to them and if we only operate on them with volatile operations,
>> behavior of the copy operations like these are defined, as far as I
>> understand.
>
> I don't find the argument about these being outside Rust allocation very useful.
> Apart from MMIO, I view all other types of memory still within purview of the
> abstract machine.

From the discussions we had on reading memory that may incur racy
writes, what I picked up is that these are OK under the conditions that
the regions are outside of any Rust allocation (whatever that means),
and they are only ever accessed by volatile operations. That is why I
think it is useful to consider these allocations outside of any Rust
allocation.

A similar argument to the one I am trying to make here was made in
`UserSliceReader::read_raw`, which delegates to a C function that (after
mapping and setup) ends up in an assembly implemented memcpy.

>
>>
>> In the last discussions we had on this, the conclusion was to use
>> `volatile_copy_memory` whenever that is available, or write a volatile
>> copy function in assembly.
>>
>> Using memcpy_{from,to}io is the latter solution. These functions are
>> simply volatile memcpy implemented in assembly.
>>
>> There is nothing special about MMIO. These functions are name as they
>> are because they are useful for MMIO.
>
> No. MMIO are really special. A few architectures require them to be accessed
> completely differently compared to normal memory. We also have things like
> INDIRECT_IOMEM. memory_{from,to}io are special as they use MMIO accessor such as
> readb to perform access on the __iomem pointer. They should not be mixed with
> normal memory. They must be treated as if they're from a completely separate
> address space.
>
> Normal memory vs DMA vs MMIO are all distinct, and this is demonstrated by the
> different types of barriers needed to order things correctly for each type of
> memory region.
>
> Userspace-mapped memory (that is also mapped in the kernel space, not __user) is
> the least special one out of these. They could practically share all atomic infra
> available for the kernel, hence the suggestion of using byte-wise atomic memcpy.

I see. I did not consider this.

At any rate, I still don't understand why I need an atomic copy function, or why I
need a byte-wise copy function. A volatile copy function should be fine, no?

And what is the exact problem in using memcpy_{from,to}io. Looking at
it, I would end up writing something similar if I wrote a copy function
myself.

If it is the wrong function to use, can you point at a fitting funciton?


Best regards,
Andreas Hindborg

On Fri, Jan 30, 2026 at 05:20:11PM +0100, Andreas Hindborg wrote:
[...]
> >> In the last discussions we had on this, the conclusion was to use
> >> `volatile_copy_memory` whenever that is available, or write a volatile
> >> copy function in assembly.
> >>
> >> Using memcpy_{from,to}io is the latter solution. These functions are
> >> simply volatile memcpy implemented in assembly.
> >>
> >> There is nothing special about MMIO. These functions are name as they
> >> are because they are useful for MMIO.
> >
> > No. MMIO are really special. A few architectures require them to be accessed
> > completely differently compared to normal memory. We also have things like
> > INDIRECT_IOMEM. memory_{from,to}io are special as they use MMIO accessor such as
> > readb to perform access on the __iomem pointer. They should not be mixed with
> > normal memory. They must be treated as if they're from a completely separate
> > address space.
> >
> > Normal memory vs DMA vs MMIO are all distinct, and this is demonstrated by the
> > different types of barriers needed to order things correctly for each type of
> > memory region.
> >
> > Userspace-mapped memory (that is also mapped in the kernel space, not __user) is
> > the least special one out of these. They could practically share all atomic infra
> > available for the kernel, hence the suggestion of using byte-wise atomic memcpy.
> 
> I see. I did not consider this.
> 
> At any rate, I still don't understand why I need an atomic copy function, or why I
> need a byte-wise copy function. A volatile copy function should be fine, no?
> 

but memcpy_{from,to}io() are not just volatile copy functions, they have
additional side effects for MMIO ;-)

> And what is the exact problem in using memcpy_{from,to}io. Looking at
> it, I would end up writing something similar if I wrote a copy function
> myself.
> 
> If it is the wrong function to use, can you point at a fitting funciton?
> 

I *think* for your use cases, a `user_page.read_volatile()` should
suffice if the only potential concurrent writer is in the userspace
(outside the Rust AM). The reason/rule I'm using is: a volatile
operation may race with an access that compiler can know about (i.e.
from Rust and C code), but it will not race with an external access.

However, byte-wise atomic memcpy will be more defined without paying any
extra penalty.

Regards,
Boqun

> 
> Best regards,
> Andreas Hindborg
> 
>

"Boqun Feng" <boqun@kernel.org> writes:

> On Fri, Jan 30, 2026 at 05:20:11PM +0100, Andreas Hindborg wrote:
> [...]
>> >> In the last discussions we had on this, the conclusion was to use
>> >> `volatile_copy_memory` whenever that is available, or write a volatile
>> >> copy function in assembly.
>> >>
>> >> Using memcpy_{from,to}io is the latter solution. These functions are
>> >> simply volatile memcpy implemented in assembly.
>> >>
>> >> There is nothing special about MMIO. These functions are name as they
>> >> are because they are useful for MMIO.
>> >
>> > No. MMIO are really special. A few architectures require them to be accessed
>> > completely differently compared to normal memory. We also have things like
>> > INDIRECT_IOMEM. memory_{from,to}io are special as they use MMIO accessor such as
>> > readb to perform access on the __iomem pointer. They should not be mixed with
>> > normal memory. They must be treated as if they're from a completely separate
>> > address space.
>> >
>> > Normal memory vs DMA vs MMIO are all distinct, and this is demonstrated by the
>> > different types of barriers needed to order things correctly for each type of
>> > memory region.
>> >
>> > Userspace-mapped memory (that is also mapped in the kernel space, not __user) is
>> > the least special one out of these. They could practically share all atomic infra
>> > available for the kernel, hence the suggestion of using byte-wise atomic memcpy.
>>
>> I see. I did not consider this.
>>
>> At any rate, I still don't understand why I need an atomic copy function, or why I
>> need a byte-wise copy function. A volatile copy function should be fine, no?
>>
>
> but memcpy_{from,to}io() are not just volatile copy functions, they have
> additional side effects for MMIO ;-)

Alright. For the sake of my curiosity, could you explain these
additional side effects and the way thy are handled in the
implementation of these functions?

>
>> And what is the exact problem in using memcpy_{from,to}io. Looking at
>> it, I would end up writing something similar if I wrote a copy function
>> myself.
>>
>> If it is the wrong function to use, can you point at a fitting funciton?
>>
>
> I *think* for your use cases, a `user_page.read_volatile()` should
> suffice if the only potential concurrent writer is in the userspace
> (outside the Rust AM). The reason/rule I'm using is: a volatile
> operation may race with an access that compiler can know about (i.e.
> from Rust and C code), but it will not race with an external access.

That is my reasoning as well.

>
> However, byte-wise atomic memcpy will be more defined without paying any
> extra penalty.

Could you explain the additional penalty of `core::ptr::read_volatile`
vs `kernel::sync::atomic::Atomic::load` with  relaxed ordering?


Best regards,
Andreas Hindborg

On Sat, Jan 31, 2026 at 02:19:05PM +0100, Andreas Hindborg wrote:
[..]
> >
> > However, byte-wise atomic memcpy will be more defined without paying any
> > extra penalty.
> 
> Could you explain the additional penalty of `core::ptr::read_volatile`
> vs `kernel::sync::atomic::Atomic::load` with  relaxed ordering?
> 

I don't understand your question, so allow me to explain what I meant:
for the sake of discussion, let's assume we have both

	fn volatile_copy_memory(src: *mut u8, dst: *mut u8, count: usize)

and

	fn volatile_byte_wise_atomic_copy_memory(<same signature>, ordering: Ordering)

implemented. What I meant was to the best of my knowledge, when ordering
= Relaxed, these two would generate the exact same code because all the
architectures that I'm aware of have byte wise atomicity in the
load/store instructions. And compared to volatile_copy_memory(),
volatile_byte_wise_atomic_copy_memory() can bear the race with another
volatile_byte_wise_atomic_copy_memory() or any other atomic access
(meaning that's not a UB). So I'd prefer using that if we have it.

Regards,
Boqun

> 
> Best regards,
> Andreas Hindborg
> 
>

"Boqun Feng" <boqun@kernel.org> writes:

> On Sat, Jan 31, 2026 at 02:19:05PM +0100, Andreas Hindborg wrote:
> [..]
>> >
>> > However, byte-wise atomic memcpy will be more defined without paying any
>> > extra penalty.
>>
>> Could you explain the additional penalty of `core::ptr::read_volatile`
>> vs `kernel::sync::atomic::Atomic::load` with  relaxed ordering?
>>
>
> I don't understand your question, so allow me to explain what I meant:
> for the sake of discussion, let's assume we have both
>
> 	fn volatile_copy_memory(src: *mut u8, dst: *mut u8, count: usize)
>
> and
>
> 	fn volatile_byte_wise_atomic_copy_memory(<same signature>, ordering: Ordering)
>
> implemented. What I meant was to the best of my knowledge, when ordering
> = Relaxed, these two would generate the exact same code because all the
> architectures that I'm aware of have byte wise atomicity in the
> load/store instructions. And compared to volatile_copy_memory(),
> volatile_byte_wise_atomic_copy_memory() can bear the race with another
> volatile_byte_wise_atomic_copy_memory() or any other atomic access
> (meaning that's not a UB). So I'd prefer using that if we have it.

Ok, thanks for clarifying. I assumed you were referring to the other
functions I mentioned, because they exist in `kernel` or `core`.
`volatile_copy_memory` is unstable in `core`, and as far as I know
`volatile_byte_wise_atomic_copy_memory` does not exist.

When you wrote `read_volatile`, I assumed you meant
`core::ptr::read_volatile`, and the atomics we have are
`kernel::sync::atomic::*`.

So now I am a bit confused as to what method you think is usable here.
Is it something we need to implement?

Best regards,
Andreas Hindborg

On Sat, Jan 31, 2026 at 08:10:21PM +0100, Andreas Hindborg wrote:
> "Boqun Feng" <boqun@kernel.org> writes:
> 
> > On Sat, Jan 31, 2026 at 02:19:05PM +0100, Andreas Hindborg wrote:
> > [..]
> >> >
> >> > However, byte-wise atomic memcpy will be more defined without paying any
> >> > extra penalty.
> >>
> >> Could you explain the additional penalty of `core::ptr::read_volatile`
> >> vs `kernel::sync::atomic::Atomic::load` with  relaxed ordering?
> >>
> >
> > I don't understand your question, so allow me to explain what I meant:
> > for the sake of discussion, let's assume we have both
> >
> > 	fn volatile_copy_memory(src: *mut u8, dst: *mut u8, count: usize)
> >
> > and
> >
> > 	fn volatile_byte_wise_atomic_copy_memory(<same signature>, ordering: Ordering)
> >
> > implemented. What I meant was to the best of my knowledge, when ordering
> > = Relaxed, these two would generate the exact same code because all the
> > architectures that I'm aware of have byte wise atomicity in the
> > load/store instructions. And compared to volatile_copy_memory(),
> > volatile_byte_wise_atomic_copy_memory() can bear the race with another
> > volatile_byte_wise_atomic_copy_memory() or any other atomic access
> > (meaning that's not a UB). So I'd prefer using that if we have it.
> 
> Ok, thanks for clarifying. I assumed you were referring to the other
> functions I mentioned, because they exist in `kernel` or `core`.
> `volatile_copy_memory` is unstable in `core`, and as far as I know
> `volatile_byte_wise_atomic_copy_memory` does not exist.

I was using volatile_byte_wise_atomic_copy_memory() to represent the
concept that we have a volatile byte-wise atomic memcpy. I was trying to
discuss the performance difference (which is 0) between a "volatile
memory copy" and "a volatile byte-wise atomic memory copy" based on
these concepts to answer your question about the "penalty" part of my
previous reply.

> 
> When you wrote `read_volatile`, I assumed you meant
> `core::ptr::read_volatile`, and the atomics we have are
> `kernel::sync::atomic::*`.

It was the curse of knowledge, when I referred to "byte-wise atomic
memcpy", I meant the concept of this [1], i.e. a memcpy that provides
atomicity of each byte.

[1]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2022/p1478r7.html

> 
> So now I am a bit confused as to what method you think is usable here.
> Is it something we need to implement?
> 

First, since the length of the copy is not fixed, we will need something
like `volatile_copy_memcpy()` to handle that. So I need to take back my
previous suggestion about using `read_volatile()`, not because it would
cause UB, but because it doesn't handle variable lengths.

But if there could be a concurrent writer to the page we are copying
from, we need a `volatile_byte_wise_atomic_copy_memory()` that we need
either implement on our own or ask Rust to provide one.

Does this help?

Regards,
Boqun

> Best regards,
> Andreas Hindborg
> 
> 
>

"Boqun Feng" <boqun@kernel.org> writes:

> On Sat, Jan 31, 2026 at 08:10:21PM +0100, Andreas Hindborg wrote:
>> "Boqun Feng" <boqun@kernel.org> writes:
>>
>> > On Sat, Jan 31, 2026 at 02:19:05PM +0100, Andreas Hindborg wrote:
>> > [..]
>> >> >
>> >> > However, byte-wise atomic memcpy will be more defined without paying any
>> >> > extra penalty.
>> >>
>> >> Could you explain the additional penalty of `core::ptr::read_volatile`
>> >> vs `kernel::sync::atomic::Atomic::load` with  relaxed ordering?
>> >>
>> >
>> > I don't understand your question, so allow me to explain what I meant:
>> > for the sake of discussion, let's assume we have both
>> >
>> > 	fn volatile_copy_memory(src: *mut u8, dst: *mut u8, count: usize)
>> >
>> > and
>> >
>> > 	fn volatile_byte_wise_atomic_copy_memory(<same signature>, ordering: Ordering)
>> >
>> > implemented. What I meant was to the best of my knowledge, when ordering
>> > = Relaxed, these two would generate the exact same code because all the
>> > architectures that I'm aware of have byte wise atomicity in the
>> > load/store instructions. And compared to volatile_copy_memory(),
>> > volatile_byte_wise_atomic_copy_memory() can bear the race with another
>> > volatile_byte_wise_atomic_copy_memory() or any other atomic access
>> > (meaning that's not a UB). So I'd prefer using that if we have it.
>>
>> Ok, thanks for clarifying. I assumed you were referring to the other
>> functions I mentioned, because they exist in `kernel` or `core`.
>> `volatile_copy_memory` is unstable in `core`, and as far as I know
>> `volatile_byte_wise_atomic_copy_memory` does not exist.
>
> I was using volatile_byte_wise_atomic_copy_memory() to represent the
> concept that we have a volatile byte-wise atomic memcpy. I was trying to
> discuss the performance difference (which is 0) between a "volatile
> memory copy" and "a volatile byte-wise atomic memory copy" based on
> these concepts to answer your question about the "penalty" part of my
> previous reply.
>
>>
>> When you wrote `read_volatile`, I assumed you meant
>> `core::ptr::read_volatile`, and the atomics we have are
>> `kernel::sync::atomic::*`.
>
> It was the curse of knowledge, when I referred to "byte-wise atomic
> memcpy", I meant the concept of this [1], i.e. a memcpy that provides
> atomicity of each byte.
>
> [1]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2022/p1478r7.html
>
>>
>> So now I am a bit confused as to what method you think is usable here.
>> Is it something we need to implement?
>>
>
> First, since the length of the copy is not fixed, we will need something
> like `volatile_copy_memcpy()` to handle that. So I need to take back my
> previous suggestion about using `read_volatile()`, not because it would
> cause UB, but because it doesn't handle variable lengths.

We could call it in a loop? Would that be inefficient?

>
> But if there could be a concurrent writer to the page we are copying
> from, we need a `volatile_byte_wise_atomic_copy_memory()` that we need
> either implement on our own or ask Rust to provide one.
>
> Does this help?

Yes, this is all super helpful and much appreciated. Thanks!


Best regards,
Andreas Hindborg

On Fri, Jan 30, 2026 at 01:41:05PM -0800, Boqun Feng wrote:
> On Fri, Jan 30, 2026 at 05:20:11PM +0100, Andreas Hindborg wrote:
> [...]
> > >> In the last discussions we had on this, the conclusion was to use
> > >> `volatile_copy_memory` whenever that is available, or write a volatile
> > >> copy function in assembly.
> > >>
> > >> Using memcpy_{from,to}io is the latter solution. These functions are
> > >> simply volatile memcpy implemented in assembly.
> > >>
> > >> There is nothing special about MMIO. These functions are name as they
> > >> are because they are useful for MMIO.
> > >
> > > No. MMIO are really special. A few architectures require them to be accessed
> > > completely differently compared to normal memory. We also have things like
> > > INDIRECT_IOMEM. memory_{from,to}io are special as they use MMIO accessor such as
> > > readb to perform access on the __iomem pointer. They should not be mixed with
> > > normal memory. They must be treated as if they're from a completely separate
> > > address space.
> > >
> > > Normal memory vs DMA vs MMIO are all distinct, and this is demonstrated by the
> > > different types of barriers needed to order things correctly for each type of
> > > memory region.
> > >
> > > Userspace-mapped memory (that is also mapped in the kernel space, not __user) is
> > > the least special one out of these. They could practically share all atomic infra
> > > available for the kernel, hence the suggestion of using byte-wise atomic memcpy.
> > 
> > I see. I did not consider this.
> > 
> > At any rate, I still don't understand why I need an atomic copy function, or why I
> > need a byte-wise copy function. A volatile copy function should be fine, no?
> > 
> 
> but memcpy_{from,to}io() are not just volatile copy functions, they have
> additional side effects for MMIO ;-)
> 

For example, powerpc's memcpy_fromio() has eioio() in it, which we don't
need for normal (user -> kernel) memory copy.

> > And what is the exact problem in using memcpy_{from,to}io. Looking at

I think the main problem of using memcpy_{from,to}io here is not that
they are not volatile memcpy (they might be), but it's because we
wouldn't use them for the same thing in C, because they are designed for
memory copying between MMIO and kernel memory (RAM).

For MMIO, as Gary mentioned, because they are different than the normal
memory, special instructions or extra barriers are needed.

For DMA memory, it can be almost treated as external normal memory,
however, different archictures/systems/platforms may have different
requirement regarding cache coherent between CPU and devices, specially
mapping or special instructions may be needed.

For __user memory, because kernel is only given a userspace address, and
userspace can lie or unmap the address while kernel accessing it,
copy_{from,to}_user() is needed to handle page faults.

Your use case (copying between userspace-mapped memory and kernel
memory) is, as Gary said, the least special here. So using
memcpy_{from,to}io() would be overkill and probably misleading. I
suggest we use `{read,write}_volatile()` (unless I'm missing something
subtle of course), however `{read,write}_volatile()` only works on Sized
types, so we may have to use `bindings::memcpy()` or
core::intrinsics::volatile_copy_memory() [1] (or suggest Rust to
stablize something) if we want to avoid implementing something by
ourselves.

[1]: https://doc.rust-lang.org/std/intrinsics/fn.volatile_copy_memory.html

Regards,
Boqun

> > it, I would end up writing something similar if I wrote a copy function
> > myself.
> > 
> > If it is the wrong function to use, can you point at a fitting funciton?
> > 
> 
> I *think* for your use cases, a `user_page.read_volatile()` should
> suffice if the only potential concurrent writer is in the userspace
> (outside the Rust AM). The reason/rule I'm using is: a volatile
> operation may race with an access that compiler can know about (i.e.
> from Rust and C code), but it will not race with an external access.
> 
> However, byte-wise atomic memcpy will be more defined without paying any
> extra penalty.
> 
> Regards,
> Boqun
> 
> > 
> > Best regards,
> > Andreas Hindborg
> > 
> >

"Boqun Feng" <boqun@kernel.org> writes:

> On Fri, Jan 30, 2026 at 01:41:05PM -0800, Boqun Feng wrote:
>> On Fri, Jan 30, 2026 at 05:20:11PM +0100, Andreas Hindborg wrote:
>> [...]
>> > >> In the last discussions we had on this, the conclusion was to use
>> > >> `volatile_copy_memory` whenever that is available, or write a volatile
>> > >> copy function in assembly.
>> > >>
>> > >> Using memcpy_{from,to}io is the latter solution. These functions are
>> > >> simply volatile memcpy implemented in assembly.
>> > >>
>> > >> There is nothing special about MMIO. These functions are name as they
>> > >> are because they are useful for MMIO.
>> > >
>> > > No. MMIO are really special. A few architectures require them to be accessed
>> > > completely differently compared to normal memory. We also have things like
>> > > INDIRECT_IOMEM. memory_{from,to}io are special as they use MMIO accessor such as
>> > > readb to perform access on the __iomem pointer. They should not be mixed with
>> > > normal memory. They must be treated as if they're from a completely separate
>> > > address space.
>> > >
>> > > Normal memory vs DMA vs MMIO are all distinct, and this is demonstrated by the
>> > > different types of barriers needed to order things correctly for each type of
>> > > memory region.
>> > >
>> > > Userspace-mapped memory (that is also mapped in the kernel space, not __user) is
>> > > the least special one out of these. They could practically share all atomic infra
>> > > available for the kernel, hence the suggestion of using byte-wise atomic memcpy.
>> >
>> > I see. I did not consider this.
>> >
>> > At any rate, I still don't understand why I need an atomic copy function, or why I
>> > need a byte-wise copy function. A volatile copy function should be fine, no?
>> >
>>
>> but memcpy_{from,to}io() are not just volatile copy functions, they have
>> additional side effects for MMIO ;-)
>>
>
> For example, powerpc's memcpy_fromio() has eioio() in it, which we don't
> need for normal (user -> kernel) memory copy.

Ok, I see. Thanks for explaining. I was only looking at the x86
implementation, which is of course not enough.

>
>> > And what is the exact problem in using memcpy_{from,to}io. Looking at
>
> I think the main problem of using memcpy_{from,to}io here is not that
> they are not volatile memcpy (they might be), but it's because we
> wouldn't use them for the same thing in C, because they are designed for
> memory copying between MMIO and kernel memory (RAM).
>
> For MMIO, as Gary mentioned, because they are different than the normal
> memory, special instructions or extra barriers are needed.

I see, I was not aware.

>
> For DMA memory, it can be almost treated as external normal memory,
> however, different archictures/systems/platforms may have different
> requirement regarding cache coherent between CPU and devices, specially
> mapping or special instructions may be needed.

Cache flushing and barriers, got it.

>
> For __user memory, because kernel is only given a userspace address, and
> userspace can lie or unmap the address while kernel accessing it,
> copy_{from,to}_user() is needed to handle page faults.

Just to clarify, for my use case, the page is already mapped to kernel
space, and it is guaranteed to be mapped for the duration of the call
where I do the copy. Also, it _may_ be a user page, but it might not
always be the case.

>
> Your use case (copying between userspace-mapped memory and kernel
> memory) is, as Gary said, the least special here. So using
> memcpy_{from,to}io() would be overkill and probably misleading.

Ok, I understand.

> I
> suggest we use `{read,write}_volatile()` (unless I'm missing something
> subtle of course), however `{read,write}_volatile()` only works on Sized
> types,

We can copy as u8? Or would it be more efficient to copy as a larger size?

You suggested atomic in the other email, did you abandon that idea?

> so we may have to use `bindings::memcpy()` or
> core::intrinsics::volatile_copy_memory() [1]

I was looking at this one, but it is unstable behind `core_intrinsics`.
I was uncertain about pulling in additional unstable features. This is
why I was looking for something in the C kernel to use.

I think `bindings::memcpy` is not guaranteed to be implemented as inline
assembly, so it may not have volatile semantics?


Best regards,
Andreas Hindborg

On Sat, Jan 31, 2026 at 02:34:02PM +0100, Andreas Hindborg wrote:
[..]
> >
> > For DMA memory, it can be almost treated as external normal memory,
> > however, different archictures/systems/platforms may have different
> > requirement regarding cache coherent between CPU and devices, specially
> > mapping or special instructions may be needed.
> 
> Cache flushing and barriers, got it.
> 

For completeness, I think for some architectures/platforms, cache
coherence between CPU and devices can be achieved by hardware, in that
case, DMA memory access can be just a normal memory access.

> >
> > For __user memory, because kernel is only given a userspace address, and
> > userspace can lie or unmap the address while kernel accessing it,
> > copy_{from,to}_user() is needed to handle page faults.
> 
> Just to clarify, for my use case, the page is already mapped to kernel
> space, and it is guaranteed to be mapped for the duration of the call
> where I do the copy. Also, it _may_ be a user page, but it might not
> always be the case.
> 

Ok, if it's not a page mapped to userspace, would there be any other
access from kernel while copying the page? If there is other kernel
thread or interrupt could write to source page, the write needs to be
atomic in some level (byte-wise for example), so does the read part of
the copy.

> >
> > Your use case (copying between userspace-mapped memory and kernel
> > memory) is, as Gary said, the least special here. So using
> > memcpy_{from,to}io() would be overkill and probably misleading.
> 
> Ok, I understand.
> 
> > I
> > suggest we use `{read,write}_volatile()` (unless I'm missing something
> > subtle of course), however `{read,write}_volatile()` only works on Sized
> > types,
> 
> We can copy as u8? Or would it be more efficient to copy as a larger size?
> 

Copying as a larger size is more efficient: less instructions for the
same amount of data to copy.

> You suggested atomic in the other email, did you abandon that idea?
> 

No, if we have byte-wise atomic copy, I'd still use that, but that is
not something already implemented in Rust. (my reply had a "if we want
to avoid implementing something by ourselves" at last)

> > so we may have to use `bindings::memcpy()` or
> > core::intrinsics::volatile_copy_memory() [1]
> 
> I was looking at this one, but it is unstable behind `core_intrinsics`.
> I was uncertain about pulling in additional unstable features. This is

That's also why I said "(or suggest Rust to stabilize something).

> why I was looking for something in the C kernel to use.
> 
> I think `bindings::memcpy` is not guaranteed to be implemented as inline
> assembly, so it may not have volatile semantics?
> 

Well, it's used in C as if it's volatile, for example, it's used in the
similar case in bio_copy_data_iter() (hopefully you can confirm that's
indeed a similar case). And I'm suggesting we use it forever, just use
it while waiting for volatile_copy_memory() or something.

Regards,
Boqun

> 
> Best regards,
> Andreas Hindborg
> 
> 
>

"Boqun Feng" <boqun@kernel.org> writes:

> On Sat, Jan 31, 2026 at 02:34:02PM +0100, Andreas Hindborg wrote:
> [..]
>> >
>> > For DMA memory, it can be almost treated as external normal memory,
>> > however, different archictures/systems/platforms may have different
>> > requirement regarding cache coherent between CPU and devices, specially
>> > mapping or special instructions may be needed.
>>
>> Cache flushing and barriers, got it.
>>
>
> For completeness, I think for some architectures/platforms, cache
> coherence between CPU and devices can be achieved by hardware, in that
> case, DMA memory access can be just a normal memory access.
>
>> >
>> > For __user memory, because kernel is only given a userspace address, and
>> > userspace can lie or unmap the address while kernel accessing it,
>> > copy_{from,to}_user() is needed to handle page faults.
>>
>> Just to clarify, for my use case, the page is already mapped to kernel
>> space, and it is guaranteed to be mapped for the duration of the call
>> where I do the copy. Also, it _may_ be a user page, but it might not
>> always be the case.
>>
>
> Ok, if it's not a page mapped to userspace, would there be any other
> access from kernel while copying the page? If there is other kernel
> thread or interrupt could write to source page, the write needs to be
> atomic in some level (byte-wise for example), so does the read part of
> the copy.

No matter if it is a page from user space or a page only mapped in the
kernel, there should be no concurrent access by kernel threads.

These pages are the IO buffers for block device IO. For direct user IO
they would be user space pages mapped to the kernel during IO. For
regular IO, they are managed by the page cache. I am pretty sure proper
locking is in place so that the pages are not mutated or read while an
IO request is outstanding from the page cache.

Maybe encryption could be a case where data is written by kernel threads
(in case of no hw acceleration) to IO buffers mapped in kernel space.
But again, these operations would not overlap with the IO request, and
proper synchronization should be in place.

>
>> >
>> > Your use case (copying between userspace-mapped memory and kernel
>> > memory) is, as Gary said, the least special here. So using
>> > memcpy_{from,to}io() would be overkill and probably misleading.
>>
>> Ok, I understand.
>>
>> > I
>> > suggest we use `{read,write}_volatile()` (unless I'm missing something
>> > subtle of course), however `{read,write}_volatile()` only works on Sized
>> > types,
>>
>> We can copy as u8? Or would it be more efficient to copy as a larger size?
>>
>
> Copying as a larger size is more efficient: less instructions for the
> same amount of data to copy.
>
>> You suggested atomic in the other email, did you abandon that idea?
>>
>
> No, if we have byte-wise atomic copy, I'd still use that, but that is
> not something already implemented in Rust. (my reply had a "if we want
> to avoid implementing something by ourselves" at last)

Got it.

>
>> > so we may have to use `bindings::memcpy()` or
>> > core::intrinsics::volatile_copy_memory() [1]
>>
>> I was looking at this one, but it is unstable behind `core_intrinsics`.
>> I was uncertain about pulling in additional unstable features. This is
>
> That's also why I said "(or suggest Rust to stabilize something).
>
>> why I was looking for something in the C kernel to use.
>>
>> I think `bindings::memcpy` is not guaranteed to be implemented as inline
>> assembly, so it may not have volatile semantics?
>>
>
> Well, it's used in C as if it's volatile, for example, it's used in the
> similar case in bio_copy_data_iter() (hopefully you can confirm that's
> indeed a similar case).

Yes, this is the exact same situation. It is used higher in the stack,
but same constraints.

> And I'm suggesting we use it forever, just use
> it while waiting for volatile_copy_memory() or something.

Ok, I'm fine with that. We can use this one and add a TODO note.


Best regards,
Andreas Hindborg

On Sat Jan 31, 2026 at 1:34 PM GMT, Andreas Hindborg wrote:
> "Boqun Feng" <boqun@kernel.org> writes:
>
>> On Fri, Jan 30, 2026 at 01:41:05PM -0800, Boqun Feng wrote:
>>> On Fri, Jan 30, 2026 at 05:20:11PM +0100, Andreas Hindborg wrote:
>>> [...]
>>> > >> In the last discussions we had on this, the conclusion was to use
>>> > >> `volatile_copy_memory` whenever that is available, or write a volatile
>>> > >> copy function in assembly.
>>> > >>
>>> > >> Using memcpy_{from,to}io is the latter solution. These functions are
>>> > >> simply volatile memcpy implemented in assembly.
>>> > >>
>>> > >> There is nothing special about MMIO. These functions are name as they
>>> > >> are because they are useful for MMIO.
>>> > >
>>> > > No. MMIO are really special. A few architectures require them to be accessed
>>> > > completely differently compared to normal memory. We also have things like
>>> > > INDIRECT_IOMEM. memory_{from,to}io are special as they use MMIO accessor such as
>>> > > readb to perform access on the __iomem pointer. They should not be mixed with
>>> > > normal memory. They must be treated as if they're from a completely separate
>>> > > address space.
>>> > >
>>> > > Normal memory vs DMA vs MMIO are all distinct, and this is demonstrated by the
>>> > > different types of barriers needed to order things correctly for each type of
>>> > > memory region.
>>> > >
>>> > > Userspace-mapped memory (that is also mapped in the kernel space, not __user) is
>>> > > the least special one out of these. They could practically share all atomic infra
>>> > > available for the kernel, hence the suggestion of using byte-wise atomic memcpy.
>>> >
>>> > I see. I did not consider this.
>>> >
>>> > At any rate, I still don't understand why I need an atomic copy function, or why I
>>> > need a byte-wise copy function. A volatile copy function should be fine, no?
>>> >
>>>
>>> but memcpy_{from,to}io() are not just volatile copy functions, they have
>>> additional side effects for MMIO ;-)
>>>
>>
>> For example, powerpc's memcpy_fromio() has eioio() in it, which we don't
>> need for normal (user -> kernel) memory copy.
>
> Ok, I see. Thanks for explaining. I was only looking at the x86
> implementation, which is of course not enough.
>
>>
>>> > And what is the exact problem in using memcpy_{from,to}io. Looking at
>>
>> I think the main problem of using memcpy_{from,to}io here is not that
>> they are not volatile memcpy (they might be), but it's because we
>> wouldn't use them for the same thing in C, because they are designed for
>> memory copying between MMIO and kernel memory (RAM).
>>
>> For MMIO, as Gary mentioned, because they are different than the normal
>> memory, special instructions or extra barriers are needed.
>
> I see, I was not aware.
>
>>
>> For DMA memory, it can be almost treated as external normal memory,
>> however, different archictures/systems/platforms may have different
>> requirement regarding cache coherent between CPU and devices, specially
>> mapping or special instructions may be needed.
>
> Cache flushing and barriers, got it.
>
>>
>> For __user memory, because kernel is only given a userspace address, and
>> userspace can lie or unmap the address while kernel accessing it,
>> copy_{from,to}_user() is needed to handle page faults.
>
> Just to clarify, for my use case, the page is already mapped to kernel
> space, and it is guaranteed to be mapped for the duration of the call
> where I do the copy. Also, it _may_ be a user page, but it might not
> always be the case.

In that case you should also assume there might be other kernel-space users.
Byte-wise atomic memcpy would be best tool.

>
>>
>> Your use case (copying between userspace-mapped memory and kernel
>> memory) is, as Gary said, the least special here. So using
>> memcpy_{from,to}io() would be overkill and probably misleading.
>
> Ok, I understand.
>
>> I
>> suggest we use `{read,write}_volatile()` (unless I'm missing something
>> subtle of course), however `{read,write}_volatile()` only works on Sized
>> types,
>
> We can copy as u8? Or would it be more efficient to copy as a larger size?

Byte-wise atomic means that the atomicity is restricted to byte level (hence
it's okay to say if you read a u32 with it and does not observe an atomic
update). It does not mean that the access needs to be byte-wise, so it's
perfectly fine to do a 32-bit load and it'll still be byte-wise atomic.

>
> You suggested atomic in the other email, did you abandon that idea?

The semantics we want is byte-wise atomic, although as a impl detail, using
volatile for now is all that we need.

>
>> so we may have to use `bindings::memcpy()` or
>> core::intrinsics::volatile_copy_memory() [1]
>
> I was looking at this one, but it is unstable behind `core_intrinsics`.
> I was uncertain about pulling in additional unstable features. This is
> why I was looking for something in the C kernel to use.
>
> I think `bindings::memcpy` is not guaranteed to be implemented as inline
> assembly, so it may not have volatile semantics?

In absence of full language LTO as we have today, it'll be fine (in practice).
Unlike C, if you reference a symbol called "memcpy", it won't be treated as
special and get turned into non-volatile memcpy.

If the volatile memcpy intrinsics is stable, then we can switch to use that.

Best,
Gary

>
>
> Best regards,
> Andreas Hindborg

"Gary Guo" <gary@garyguo.net> writes:

> On Sat Jan 31, 2026 at 1:34 PM GMT, Andreas Hindborg wrote:
>> "Boqun Feng" <boqun@kernel.org> writes:
>>
>>> On Fri, Jan 30, 2026 at 01:41:05PM -0800, Boqun Feng wrote:
>>>> On Fri, Jan 30, 2026 at 05:20:11PM +0100, Andreas Hindborg wrote:
>>>> [...]
>>>> > >> In the last discussions we had on this, the conclusion was to use
>>>> > >> `volatile_copy_memory` whenever that is available, or write a volatile
>>>> > >> copy function in assembly.
>>>> > >>
>>>> > >> Using memcpy_{from,to}io is the latter solution. These functions are
>>>> > >> simply volatile memcpy implemented in assembly.
>>>> > >>
>>>> > >> There is nothing special about MMIO. These functions are name as they
>>>> > >> are because they are useful for MMIO.
>>>> > >
>>>> > > No. MMIO are really special. A few architectures require them to be accessed
>>>> > > completely differently compared to normal memory. We also have things like
>>>> > > INDIRECT_IOMEM. memory_{from,to}io are special as they use MMIO accessor such as
>>>> > > readb to perform access on the __iomem pointer. They should not be mixed with
>>>> > > normal memory. They must be treated as if they're from a completely separate
>>>> > > address space.
>>>> > >
>>>> > > Normal memory vs DMA vs MMIO are all distinct, and this is demonstrated by the
>>>> > > different types of barriers needed to order things correctly for each type of
>>>> > > memory region.
>>>> > >
>>>> > > Userspace-mapped memory (that is also mapped in the kernel space, not __user) is
>>>> > > the least special one out of these. They could practically share all atomic infra
>>>> > > available for the kernel, hence the suggestion of using byte-wise atomic memcpy.
>>>> >
>>>> > I see. I did not consider this.
>>>> >
>>>> > At any rate, I still don't understand why I need an atomic copy function, or why I
>>>> > need a byte-wise copy function. A volatile copy function should be fine, no?
>>>> >
>>>>
>>>> but memcpy_{from,to}io() are not just volatile copy functions, they have
>>>> additional side effects for MMIO ;-)
>>>>
>>>
>>> For example, powerpc's memcpy_fromio() has eioio() in it, which we don't
>>> need for normal (user -> kernel) memory copy.
>>
>> Ok, I see. Thanks for explaining. I was only looking at the x86
>> implementation, which is of course not enough.
>>
>>>
>>>> > And what is the exact problem in using memcpy_{from,to}io. Looking at
>>>
>>> I think the main problem of using memcpy_{from,to}io here is not that
>>> they are not volatile memcpy (they might be), but it's because we
>>> wouldn't use them for the same thing in C, because they are designed for
>>> memory copying between MMIO and kernel memory (RAM).
>>>
>>> For MMIO, as Gary mentioned, because they are different than the normal
>>> memory, special instructions or extra barriers are needed.
>>
>> I see, I was not aware.
>>
>>>
>>> For DMA memory, it can be almost treated as external normal memory,
>>> however, different archictures/systems/platforms may have different
>>> requirement regarding cache coherent between CPU and devices, specially
>>> mapping or special instructions may be needed.
>>
>> Cache flushing and barriers, got it.
>>
>>>
>>> For __user memory, because kernel is only given a userspace address, and
>>> userspace can lie or unmap the address while kernel accessing it,
>>> copy_{from,to}_user() is needed to handle page faults.
>>
>> Just to clarify, for my use case, the page is already mapped to kernel
>> space, and it is guaranteed to be mapped for the duration of the call
>> where I do the copy. Also, it _may_ be a user page, but it might not
>> always be the case.
>
> In that case you should also assume there might be other kernel-space users.
> Byte-wise atomic memcpy would be best tool.

Other concurrent kernel readers/writers would be a kernel bug in my use
case. We could add this to the safety requirements.

>
>>
>>>
>>> Your use case (copying between userspace-mapped memory and kernel
>>> memory) is, as Gary said, the least special here. So using
>>> memcpy_{from,to}io() would be overkill and probably misleading.
>>
>> Ok, I understand.
>>
>>> I
>>> suggest we use `{read,write}_volatile()` (unless I'm missing something
>>> subtle of course), however `{read,write}_volatile()` only works on Sized
>>> types,
>>
>> We can copy as u8? Or would it be more efficient to copy as a larger size?
>
> Byte-wise atomic means that the atomicity is restricted to byte level (hence
> it's okay to say if you read a u32 with it and does not observe an atomic
> update). It does not mean that the access needs to be byte-wise, so it's
> perfectly fine to do a 32-bit load and it'll still be byte-wise atomic.

Ah.

>
>>
>> You suggested atomic in the other email, did you abandon that idea?
>
> The semantics we want is byte-wise atomic, although as a impl detail, using
> volatile for now is all that we need.
>
>>
>>> so we may have to use `bindings::memcpy()` or
>>> core::intrinsics::volatile_copy_memory() [1]
>>
>> I was looking at this one, but it is unstable behind `core_intrinsics`.
>> I was uncertain about pulling in additional unstable features. This is
>> why I was looking for something in the C kernel to use.
>>
>> I think `bindings::memcpy` is not guaranteed to be implemented as inline
>> assembly, so it may not have volatile semantics?
>
> In absence of full language LTO as we have today, it'll be fine (in practice).
> Unlike C, if you reference a symbol called "memcpy", it won't be treated as
> special and get turned into non-volatile memcpy.
>
> If the volatile memcpy intrinsics is stable, then we can switch to use that.

Got it, this aligns with what Boqun is writing. Let's go for that.

It also looks like memcpy is implemented in assembly for arm, arm32,
x86_64. Which would exempt it from LTO. Not sure about 32bit x86 though.
It defers to `__memcpy`. I could not figure out what that resolves to.
Is it from the compiler?


Best regards,
Andreas Hindborg

Andreas Hindborg <a.hindborg@kernel.org> writes:

> "Gary Guo" <gary@garyguo.net> writes:
>
>> On Sat Jan 31, 2026 at 1:34 PM GMT, Andreas Hindborg wrote:
>>> "Boqun Feng" <boqun@kernel.org> writes:
>>>
>>>> On Fri, Jan 30, 2026 at 01:41:05PM -0800, Boqun Feng wrote:
>>>>> On Fri, Jan 30, 2026 at 05:20:11PM +0100, Andreas Hindborg wrote:
>>>>> [...]
>>>>> > >> In the last discussions we had on this, the conclusion was to use
>>>>> > >> `volatile_copy_memory` whenever that is available, or write a volatile
>>>>> > >> copy function in assembly.
>>>>> > >>
>>>>> > >> Using memcpy_{from,to}io is the latter solution. These functions are
>>>>> > >> simply volatile memcpy implemented in assembly.
>>>>> > >>
>>>>> > >> There is nothing special about MMIO. These functions are name as they
>>>>> > >> are because they are useful for MMIO.
>>>>> > >
>>>>> > > No. MMIO are really special. A few architectures require them to be accessed
>>>>> > > completely differently compared to normal memory. We also have things like
>>>>> > > INDIRECT_IOMEM. memory_{from,to}io are special as they use MMIO accessor such as
>>>>> > > readb to perform access on the __iomem pointer. They should not be mixed with
>>>>> > > normal memory. They must be treated as if they're from a completely separate
>>>>> > > address space.
>>>>> > >
>>>>> > > Normal memory vs DMA vs MMIO are all distinct, and this is demonstrated by the
>>>>> > > different types of barriers needed to order things correctly for each type of
>>>>> > > memory region.
>>>>> > >
>>>>> > > Userspace-mapped memory (that is also mapped in the kernel space, not __user) is
>>>>> > > the least special one out of these. They could practically share all atomic infra
>>>>> > > available for the kernel, hence the suggestion of using byte-wise atomic memcpy.
>>>>> >
>>>>> > I see. I did not consider this.
>>>>> >
>>>>> > At any rate, I still don't understand why I need an atomic copy function, or why I
>>>>> > need a byte-wise copy function. A volatile copy function should be fine, no?
>>>>> >
>>>>>
>>>>> but memcpy_{from,to}io() are not just volatile copy functions, they have
>>>>> additional side effects for MMIO ;-)
>>>>>
>>>>
>>>> For example, powerpc's memcpy_fromio() has eioio() in it, which we don't
>>>> need for normal (user -> kernel) memory copy.
>>>
>>> Ok, I see. Thanks for explaining. I was only looking at the x86
>>> implementation, which is of course not enough.
>>>
>>>>
>>>>> > And what is the exact problem in using memcpy_{from,to}io. Looking at
>>>>
>>>> I think the main problem of using memcpy_{from,to}io here is not that
>>>> they are not volatile memcpy (they might be), but it's because we
>>>> wouldn't use them for the same thing in C, because they are designed for
>>>> memory copying between MMIO and kernel memory (RAM).
>>>>
>>>> For MMIO, as Gary mentioned, because they are different than the normal
>>>> memory, special instructions or extra barriers are needed.
>>>
>>> I see, I was not aware.
>>>
>>>>
>>>> For DMA memory, it can be almost treated as external normal memory,
>>>> however, different archictures/systems/platforms may have different
>>>> requirement regarding cache coherent between CPU and devices, specially
>>>> mapping or special instructions may be needed.
>>>
>>> Cache flushing and barriers, got it.
>>>
>>>>
>>>> For __user memory, because kernel is only given a userspace address, and
>>>> userspace can lie or unmap the address while kernel accessing it,
>>>> copy_{from,to}_user() is needed to handle page faults.
>>>
>>> Just to clarify, for my use case, the page is already mapped to kernel
>>> space, and it is guaranteed to be mapped for the duration of the call
>>> where I do the copy. Also, it _may_ be a user page, but it might not
>>> always be the case.
>>
>> In that case you should also assume there might be other kernel-space users.
>> Byte-wise atomic memcpy would be best tool.
>
> Other concurrent kernel readers/writers would be a kernel bug in my use
> case. We could add this to the safety requirements.
>

Actually, one case just crossed my mind. I think nothing will prevent a
user space process from concurrently submitting multiple reads to the
same user page. It would not make sense, but it can be done.

If the reads are issued to different null block devices, the null block
driver might concurrently write the user page when servicing each IO
request concurrently.

The same situation would happen in real block device drivers, except the
writes would be done by dma engines rather than kernel threads.


Best regards,
Andreas Hindborg

On Sat, Jan 31, 2026 at 10:31:13PM +0100, Andreas Hindborg wrote:
[...]
> >>>>
> >>>> For __user memory, because kernel is only given a userspace address, and
> >>>> userspace can lie or unmap the address while kernel accessing it,
> >>>> copy_{from,to}_user() is needed to handle page faults.
> >>>
> >>> Just to clarify, for my use case, the page is already mapped to kernel
> >>> space, and it is guaranteed to be mapped for the duration of the call
> >>> where I do the copy. Also, it _may_ be a user page, but it might not
> >>> always be the case.
> >>
> >> In that case you should also assume there might be other kernel-space users.
> >> Byte-wise atomic memcpy would be best tool.
> >
> > Other concurrent kernel readers/writers would be a kernel bug in my use
> > case. We could add this to the safety requirements.
> >
> 
> Actually, one case just crossed my mind. I think nothing will prevent a
> user space process from concurrently submitting multiple reads to the
> same user page. It would not make sense, but it can be done.
> 
> If the reads are issued to different null block devices, the null block
> driver might concurrently write the user page when servicing each IO
> request concurrently.
> 
> The same situation would happen in real block device drivers, except the
> writes would be done by dma engines rather than kernel threads.
> 

Then we better use byte-wise atomic memcpy, and I think for all the
architectures that Linux kernel support, memcpy() is in fact byte-wise
atomic if it's volatile. Because down the actual instructions, either a
byte-size read/write is used, or a larger-size read/write is used but
they are guaranteed to be byte-wise atomic even for unaligned read or
write. So "volatile memcpy" and "volatile byte-wise atomic memcpy" have
the same implementation.

(The C++ paper [1] also says: "In fact, we expect that existing assembly
memcpy implementations will suffice when suffixed with the required
fence.")

So to make thing move forward, do you mind to introduce a
`atomic_per_byte_memcpy()` in rust::sync::atomic based on
bindings::memcpy(), and cc linux-arch and all the archs that support
Rust for some confirmation? Thanks!

[1]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1478r5.html

Regards,
Boqun

> 
> Best regards,
> Andreas Hindborg
> 
>

Boqun Feng <boqun@kernel.org> writes:

> On Sat, Jan 31, 2026 at 10:31:13PM +0100, Andreas Hindborg wrote:
> [...]
>> >>>>
>> >>>> For __user memory, because kernel is only given a userspace address, and
>> >>>> userspace can lie or unmap the address while kernel accessing it,
>> >>>> copy_{from,to}_user() is needed to handle page faults.
>> >>>
>> >>> Just to clarify, for my use case, the page is already mapped to kernel
>> >>> space, and it is guaranteed to be mapped for the duration of the call
>> >>> where I do the copy. Also, it _may_ be a user page, but it might not
>> >>> always be the case.
>> >>
>> >> In that case you should also assume there might be other kernel-space users.
>> >> Byte-wise atomic memcpy would be best tool.
>> >
>> > Other concurrent kernel readers/writers would be a kernel bug in my use
>> > case. We could add this to the safety requirements.
>> >
>> 
>> Actually, one case just crossed my mind. I think nothing will prevent a
>> user space process from concurrently submitting multiple reads to the
>> same user page. It would not make sense, but it can be done.
>> 
>> If the reads are issued to different null block devices, the null block
>> driver might concurrently write the user page when servicing each IO
>> request concurrently.
>> 
>> The same situation would happen in real block device drivers, except the
>> writes would be done by dma engines rather than kernel threads.
>> 
>
> Then we better use byte-wise atomic memcpy, and I think for all the
> architectures that Linux kernel support, memcpy() is in fact byte-wise
> atomic if it's volatile. Because down the actual instructions, either a
> byte-size read/write is used, or a larger-size read/write is used but
> they are guaranteed to be byte-wise atomic even for unaligned read or
> write. So "volatile memcpy" and "volatile byte-wise atomic memcpy" have
> the same implementation.
>
> (The C++ paper [1] also says: "In fact, we expect that existing assembly
> memcpy implementations will suffice when suffixed with the required
> fence.")
>
> So to make thing move forward, do you mind to introduce a
> `atomic_per_byte_memcpy()` in rust::sync::atomic based on
> bindings::memcpy(), and cc linux-arch and all the archs that support
> Rust for some confirmation? Thanks!

There is a few things I do not fully understand:

 - Does the operation need to be both atomic and volatile, or is atomic enough on its
   own (why)?
 - The article you reference has separate `atomic_load_per_byte_memcpy`
   and `atomic_store_per_byte_memcpy` that allows inserting an acquire
   fence before the load and a release fence after the store. Do we not
   need that?
 - It is unclear to me how to formulate the safety requirements for
   `atomic_per_byte_memcpy`. In this series, one end of the operation is
   the potential racy area. For `atomic_per_byte_memcpy` it could be
   either end (or both?). Do we even mention an area being "outside the
   Rust AM"?

First attempt below. I am quite uncertain about this. I feel like we
have two things going on: Potential races with other kernel threads,
which we solve by saying all accesses are byte-wise atomic, and reaces
with user space processes, which we solve with volatile semantics?

Should the functin name be `volatile_atomic_per_byte_memcpy`?

/// Copy `len` bytes from `src` to `dst` using byte-wise atomic operations.
///
/// This copy operation is volatile.
///
/// # Safety
///
/// Callers must ensure that:
///
/// * The source memory region is readable and reading from the region will not trap.
/// * The destination memory region is writable and writing to the region will not trap.
/// * No references exist to the source or destination regions.
/// * If the source or destination region is within the Rust AM, any concurrent reads or writes to
///   source or destination memory regions by the Rust AM must use byte-wise atomic operations.
pub unsafe fn atomic_per_byte_memcpy(src: *const u8, dst: *mut u8, len: usize) {
    // SAFETY: By the safety requirements of this function, the following operation will not:
    //  - Trap.
    //  - Invalidate any reference invariants.
    //  - Race with any operation by the Rust AM, as `bindings::memcpy` is a byte-wise atomic
    //    operation and all operations by the Rust AM use byte-wise atomic semantics.
    //
    //  Further, as `bindings::memcpy` is a volatile operation, the operation will not race with any
    //  read or write operation to the source or destination area if the area can be considered to
    //  be outside the Rust AM.
    unsafe { bindings::memcpy(dst.cast::<kernel::ffi::c_void>(), src.cast::<kernel::ffi::c_void>(), len) };
}


Best regards,
Andreas Hindborg

On Wed Feb 4, 2026 at 1:16 PM GMT, Andreas Hindborg wrote:
> Boqun Feng <boqun@kernel.org> writes:
>
>> On Sat, Jan 31, 2026 at 10:31:13PM +0100, Andreas Hindborg wrote:
>> [...]
>>> >>>>
>>> >>>> For __user memory, because kernel is only given a userspace address, and
>>> >>>> userspace can lie or unmap the address while kernel accessing it,
>>> >>>> copy_{from,to}_user() is needed to handle page faults.
>>> >>>
>>> >>> Just to clarify, for my use case, the page is already mapped to kernel
>>> >>> space, and it is guaranteed to be mapped for the duration of the call
>>> >>> where I do the copy. Also, it _may_ be a user page, but it might not
>>> >>> always be the case.
>>> >>
>>> >> In that case you should also assume there might be other kernel-space users.
>>> >> Byte-wise atomic memcpy would be best tool.
>>> >
>>> > Other concurrent kernel readers/writers would be a kernel bug in my use
>>> > case. We could add this to the safety requirements.
>>> >
>>> 
>>> Actually, one case just crossed my mind. I think nothing will prevent a
>>> user space process from concurrently submitting multiple reads to the
>>> same user page. It would not make sense, but it can be done.
>>> 
>>> If the reads are issued to different null block devices, the null block
>>> driver might concurrently write the user page when servicing each IO
>>> request concurrently.
>>> 
>>> The same situation would happen in real block device drivers, except the
>>> writes would be done by dma engines rather than kernel threads.
>>> 
>>
>> Then we better use byte-wise atomic memcpy, and I think for all the
>> architectures that Linux kernel support, memcpy() is in fact byte-wise
>> atomic if it's volatile. Because down the actual instructions, either a
>> byte-size read/write is used, or a larger-size read/write is used but
>> they are guaranteed to be byte-wise atomic even for unaligned read or
>> write. So "volatile memcpy" and "volatile byte-wise atomic memcpy" have
>> the same implementation.
>>
>> (The C++ paper [1] also says: "In fact, we expect that existing assembly
>> memcpy implementations will suffice when suffixed with the required
>> fence.")
>>
>> So to make thing move forward, do you mind to introduce a
>> `atomic_per_byte_memcpy()` in rust::sync::atomic based on
>> bindings::memcpy(), and cc linux-arch and all the archs that support
>> Rust for some confirmation? Thanks!
>
> There is a few things I do not fully understand:
>
>  - Does the operation need to be both atomic and volatile, or is atomic enough on its
>    own (why)?

In theory, C11 atomic (without using volatile keyword) and Rust atomic are not
volatile, so compiler can optimize them, e.g. coalesce two relaxed read of the
same address into one. In practice, no compiler is doing this. LKMM atomics are
always volatile.

>  - The article you reference has separate `atomic_load_per_byte_memcpy`
>    and `atomic_store_per_byte_memcpy` that allows inserting an acquire
>    fence before the load and a release fence after the store. Do we not
>    need that?

It's distinct so that the semantics on the ordering is clear, as the "acquire"
or "release" order is for the atomic argument, and there's no ordering for the
other argument.

Another thing is that without two methods, you need an extra conversion to
convert a slice to non-atomic slice, which is not generally sound. (I.e. you
cannot turn &[u8] to &[Atomic<u8>], as doing so would give you the ability to
write to immutable memory.

>  - It is unclear to me how to formulate the safety requirements for
>    `atomic_per_byte_memcpy`. In this series, one end of the operation is
>    the potential racy area. For `atomic_per_byte_memcpy` it could be
>    either end (or both?). Do we even mention an area being "outside the
>    Rust AM"?

No, atomics are inside the AM. A piece of memory is either in AM or outside. For
a page that both kernel and userspace access, we should just treat it as other
memory and treat userspace as an always-atomic user.

>
> First attempt below. I am quite uncertain about this. I feel like we
> have two things going on: Potential races with other kernel threads,
> which we solve by saying all accesses are byte-wise atomic, and reaces
> with user space processes, which we solve with volatile semantics?
>
> Should the functin name be `volatile_atomic_per_byte_memcpy`?
>
> /// Copy `len` bytes from `src` to `dst` using byte-wise atomic operations.
> ///
> /// This copy operation is volatile.
> ///
> /// # Safety
> ///
> /// Callers must ensure that:
> ///
> /// * The source memory region is readable and reading from the region will not trap.

We should just use standard terminology here, similar to Atomic::from_ptr.

> /// * The destination memory region is writable and writing to the region will not trap.
> /// * No references exist to the source or destination regions.
> /// * If the source or destination region is within the Rust AM, any concurrent reads or writes to
> ///   source or destination memory regions by the Rust AM must use byte-wise atomic operations.

This should be dropped.

> pub unsafe fn atomic_per_byte_memcpy(src: *const u8, dst: *mut u8, len: usize) {
>     // SAFETY: By the safety requirements of this function, the following operation will not:
>     //  - Trap.
>     //  - Invalidate any reference invariants.
>     //  - Race with any operation by the Rust AM, as `bindings::memcpy` is a byte-wise atomic
>     //    operation and all operations by the Rust AM use byte-wise atomic semantics.
>     //
>     //  Further, as `bindings::memcpy` is a volatile operation, the operation will not race with any
>     //  read or write operation to the source or destination area if the area can be considered to
>     //  be outside the Rust AM.
>     unsafe { bindings::memcpy(dst.cast::<kernel::ffi::c_void>(), src.cast::<kernel::ffi::c_void>(), len) };

The `cast()` don't need explicit types I think?

Best,
Gary

> }
>
>
> Best regards,
> Andreas Hindborg

On Wed, Feb 04, 2026 at 02:16:37PM +0100, Andreas Hindborg wrote:
> Boqun Feng <boqun@kernel.org> writes:
> 
> > On Sat, Jan 31, 2026 at 10:31:13PM +0100, Andreas Hindborg wrote:
> > [...]
> >> >>>>
> >> >>>> For __user memory, because kernel is only given a userspace address, and
> >> >>>> userspace can lie or unmap the address while kernel accessing it,
> >> >>>> copy_{from,to}_user() is needed to handle page faults.
> >> >>>
> >> >>> Just to clarify, for my use case, the page is already mapped to kernel
> >> >>> space, and it is guaranteed to be mapped for the duration of the call
> >> >>> where I do the copy. Also, it _may_ be a user page, but it might not
> >> >>> always be the case.
> >> >>
> >> >> In that case you should also assume there might be other kernel-space users.
> >> >> Byte-wise atomic memcpy would be best tool.
> >> >
> >> > Other concurrent kernel readers/writers would be a kernel bug in my use
> >> > case. We could add this to the safety requirements.
> >> >
> >> 
> >> Actually, one case just crossed my mind. I think nothing will prevent a
> >> user space process from concurrently submitting multiple reads to the
> >> same user page. It would not make sense, but it can be done.
> >> 
> >> If the reads are issued to different null block devices, the null block
> >> driver might concurrently write the user page when servicing each IO
> >> request concurrently.
> >> 
> >> The same situation would happen in real block device drivers, except the
> >> writes would be done by dma engines rather than kernel threads.
> >> 
> >
> > Then we better use byte-wise atomic memcpy, and I think for all the
> > architectures that Linux kernel support, memcpy() is in fact byte-wise
> > atomic if it's volatile. Because down the actual instructions, either a
> > byte-size read/write is used, or a larger-size read/write is used but
> > they are guaranteed to be byte-wise atomic even for unaligned read or
> > write. So "volatile memcpy" and "volatile byte-wise atomic memcpy" have
> > the same implementation.
> >
> > (The C++ paper [1] also says: "In fact, we expect that existing assembly
> > memcpy implementations will suffice when suffixed with the required
> > fence.")
> >
> > So to make thing move forward, do you mind to introduce a
> > `atomic_per_byte_memcpy()` in rust::sync::atomic based on
> > bindings::memcpy(), and cc linux-arch and all the archs that support
> > Rust for some confirmation? Thanks!
> 
> There is a few things I do not fully understand:
> 
>  - Does the operation need to be both atomic and volatile, or is atomic enough on its
>    own (why)?
>  - The article you reference has separate `atomic_load_per_byte_memcpy`
>    and `atomic_store_per_byte_memcpy` that allows inserting an acquire
>    fence before the load and a release fence after the store. Do we not
>    need that?

We can just make both src and dst into per-byte atomics. We don't really
lose anything from it. Technically we're performing unnecessary atomic
ops on one side, but who cares?

>  - It is unclear to me how to formulate the safety requirements for
>    `atomic_per_byte_memcpy`. In this series, one end of the operation is
>    the potential racy area. For `atomic_per_byte_memcpy` it could be
>    either end (or both?). Do we even mention an area being "outside the
>    Rust AM"?
> 
> First attempt below. I am quite uncertain about this. I feel like we
> have two things going on: Potential races with other kernel threads,
> which we solve by saying all accesses are byte-wise atomic, and reaces
> with user space processes, which we solve with volatile semantics?
> 
> Should the functin name be `volatile_atomic_per_byte_memcpy`?
> 
> /// Copy `len` bytes from `src` to `dst` using byte-wise atomic operations.
> ///
> /// This copy operation is volatile.
> ///
> /// # Safety
> ///
> /// Callers must ensure that:
> ///
> /// * The source memory region is readable and reading from the region will not trap.
> /// * The destination memory region is writable and writing to the region will not trap.

Ok.

> /// * No references exist to the source or destination regions.

You can omit this requirement. Creating references have safety
requirements, and if such references exist, you're also violating the
safety requirements of creating a reference, so you do not need to
repeat it here.

> /// * If the source or destination region is within the Rust AM, any concurrent reads or writes to
> ///   source or destination memory regions by the Rust AM must use byte-wise atomic operations.

Unless you need to support memory outside the Rust AM, we can drop this.

Alice

"Alice Ryhl" <aliceryhl@google.com> writes:

> On Wed, Feb 04, 2026 at 02:16:37PM +0100, Andreas Hindborg wrote:
>> Boqun Feng <boqun@kernel.org> writes:
>>
>> > On Sat, Jan 31, 2026 at 10:31:13PM +0100, Andreas Hindborg wrote:
>> > [...]
>> >> >>>>
>> >> >>>> For __user memory, because kernel is only given a userspace address, and
>> >> >>>> userspace can lie or unmap the address while kernel accessing it,
>> >> >>>> copy_{from,to}_user() is needed to handle page faults.
>> >> >>>
>> >> >>> Just to clarify, for my use case, the page is already mapped to kernel
>> >> >>> space, and it is guaranteed to be mapped for the duration of the call
>> >> >>> where I do the copy. Also, it _may_ be a user page, but it might not
>> >> >>> always be the case.
>> >> >>
>> >> >> In that case you should also assume there might be other kernel-space users.
>> >> >> Byte-wise atomic memcpy would be best tool.
>> >> >
>> >> > Other concurrent kernel readers/writers would be a kernel bug in my use
>> >> > case. We could add this to the safety requirements.
>> >> >
>> >>
>> >> Actually, one case just crossed my mind. I think nothing will prevent a
>> >> user space process from concurrently submitting multiple reads to the
>> >> same user page. It would not make sense, but it can be done.
>> >>
>> >> If the reads are issued to different null block devices, the null block
>> >> driver might concurrently write the user page when servicing each IO
>> >> request concurrently.
>> >>
>> >> The same situation would happen in real block device drivers, except the
>> >> writes would be done by dma engines rather than kernel threads.
>> >>
>> >
>> > Then we better use byte-wise atomic memcpy, and I think for all the
>> > architectures that Linux kernel support, memcpy() is in fact byte-wise
>> > atomic if it's volatile. Because down the actual instructions, either a
>> > byte-size read/write is used, or a larger-size read/write is used but
>> > they are guaranteed to be byte-wise atomic even for unaligned read or
>> > write. So "volatile memcpy" and "volatile byte-wise atomic memcpy" have
>> > the same implementation.
>> >
>> > (The C++ paper [1] also says: "In fact, we expect that existing assembly
>> > memcpy implementations will suffice when suffixed with the required
>> > fence.")
>> >
>> > So to make thing move forward, do you mind to introduce a
>> > `atomic_per_byte_memcpy()` in rust::sync::atomic based on
>> > bindings::memcpy(), and cc linux-arch and all the archs that support
>> > Rust for some confirmation? Thanks!
>>
>> There is a few things I do not fully understand:
>>
>>  - Does the operation need to be both atomic and volatile, or is atomic enough on its
>>    own (why)?
>>  - The article you reference has separate `atomic_load_per_byte_memcpy`
>>    and `atomic_store_per_byte_memcpy` that allows inserting an acquire
>>    fence before the load and a release fence after the store. Do we not
>>    need that?
>
> We can just make both src and dst into per-byte atomics. We don't really
> lose anything from it. Technically we're performing unnecessary atomic
> ops on one side, but who cares?

OK.

>
>>  - It is unclear to me how to formulate the safety requirements for
>>    `atomic_per_byte_memcpy`. In this series, one end of the operation is
>>    the potential racy area. For `atomic_per_byte_memcpy` it could be
>>    either end (or both?). Do we even mention an area being "outside the
>>    Rust AM"?
>>
>> First attempt below. I am quite uncertain about this. I feel like we
>> have two things going on: Potential races with other kernel threads,
>> which we solve by saying all accesses are byte-wise atomic, and reaces
>> with user space processes, which we solve with volatile semantics?
>>
>> Should the functin name be `volatile_atomic_per_byte_memcpy`?
>>
>> /// Copy `len` bytes from `src` to `dst` using byte-wise atomic operations.
>> ///
>> /// This copy operation is volatile.
>> ///
>> /// # Safety
>> ///
>> /// Callers must ensure that:
>> ///
>> /// * The source memory region is readable and reading from the region will not trap.
>> /// * The destination memory region is writable and writing to the region will not trap.
>
> Ok.
>
>> /// * No references exist to the source or destination regions.
>
> You can omit this requirement. Creating references have safety
> requirements, and if such references exist, you're also violating the
> safety requirements of creating a reference, so you do not need to
> repeat it here.

Cool.

>
>> /// * If the source or destination region is within the Rust AM, any concurrent reads or writes to
>> ///   source or destination memory regions by the Rust AM must use byte-wise atomic operations.
>
> Unless you need to support memory outside the Rust AM, we can drop this.

I need to support pages that are concurrently mapped to user processes.
I think we decided these pages are outside the Rust AM if we only do
non-reference volatile IO operations on them from kernel space.

They are different than pages that are never mapped to user space, in
the sense that they can incur concurrent reads/writes from the user
space process, and we cannot require any kind of atomicity for these
reads/writes.


Best regards,
Andreas Hindborg

On Sat Jan 31, 2026 at 8:30 PM GMT, Andreas Hindborg wrote:
> "Gary Guo" <gary@garyguo.net> writes:
>
>> On Sat Jan 31, 2026 at 1:34 PM GMT, Andreas Hindborg wrote:
>>> "Boqun Feng" <boqun@kernel.org> writes:
>>>
>>>> On Fri, Jan 30, 2026 at 01:41:05PM -0800, Boqun Feng wrote:
>>>>> On Fri, Jan 30, 2026 at 05:20:11PM +0100, Andreas Hindborg wrote:
>>>>> [...]
>>>>> > >> In the last discussions we had on this, the conclusion was to use
>>>>> > >> `volatile_copy_memory` whenever that is available, or write a volatile
>>>>> > >> copy function in assembly.
>>>>> > >>
>>>>> > >> Using memcpy_{from,to}io is the latter solution. These functions are
>>>>> > >> simply volatile memcpy implemented in assembly.
>>>>> > >>
>>>>> > >> There is nothing special about MMIO. These functions are name as they
>>>>> > >> are because they are useful for MMIO.
>>>>> > >
>>>>> > > No. MMIO are really special. A few architectures require them to be accessed
>>>>> > > completely differently compared to normal memory. We also have things like
>>>>> > > INDIRECT_IOMEM. memory_{from,to}io are special as they use MMIO accessor such as
>>>>> > > readb to perform access on the __iomem pointer. They should not be mixed with
>>>>> > > normal memory. They must be treated as if they're from a completely separate
>>>>> > > address space.
>>>>> > >
>>>>> > > Normal memory vs DMA vs MMIO are all distinct, and this is demonstrated by the
>>>>> > > different types of barriers needed to order things correctly for each type of
>>>>> > > memory region.
>>>>> > >
>>>>> > > Userspace-mapped memory (that is also mapped in the kernel space, not __user) is
>>>>> > > the least special one out of these. They could practically share all atomic infra
>>>>> > > available for the kernel, hence the suggestion of using byte-wise atomic memcpy.
>>>>> >
>>>>> > I see. I did not consider this.
>>>>> >
>>>>> > At any rate, I still don't understand why I need an atomic copy function, or why I
>>>>> > need a byte-wise copy function. A volatile copy function should be fine, no?
>>>>> >
>>>>>
>>>>> but memcpy_{from,to}io() are not just volatile copy functions, they have
>>>>> additional side effects for MMIO ;-)
>>>>>
>>>>
>>>> For example, powerpc's memcpy_fromio() has eioio() in it, which we don't
>>>> need for normal (user -> kernel) memory copy.
>>>
>>> Ok, I see. Thanks for explaining. I was only looking at the x86
>>> implementation, which is of course not enough.
>>>
>>>>
>>>>> > And what is the exact problem in using memcpy_{from,to}io. Looking at
>>>>
>>>> I think the main problem of using memcpy_{from,to}io here is not that
>>>> they are not volatile memcpy (they might be), but it's because we
>>>> wouldn't use them for the same thing in C, because they are designed for
>>>> memory copying between MMIO and kernel memory (RAM).
>>>>
>>>> For MMIO, as Gary mentioned, because they are different than the normal
>>>> memory, special instructions or extra barriers are needed.
>>>
>>> I see, I was not aware.
>>>
>>>>
>>>> For DMA memory, it can be almost treated as external normal memory,
>>>> however, different archictures/systems/platforms may have different
>>>> requirement regarding cache coherent between CPU and devices, specially
>>>> mapping or special instructions may be needed.
>>>
>>> Cache flushing and barriers, got it.
>>>
>>>>
>>>> For __user memory, because kernel is only given a userspace address, and
>>>> userspace can lie or unmap the address while kernel accessing it,
>>>> copy_{from,to}_user() is needed to handle page faults.
>>>
>>> Just to clarify, for my use case, the page is already mapped to kernel
>>> space, and it is guaranteed to be mapped for the duration of the call
>>> where I do the copy. Also, it _may_ be a user page, but it might not
>>> always be the case.
>>
>> In that case you should also assume there might be other kernel-space users.
>> Byte-wise atomic memcpy would be best tool.
>
> Other concurrent kernel readers/writers would be a kernel bug in my use
> case. We could add this to the safety requirements.
>
>>
>>>
>>>>
>>>> Your use case (copying between userspace-mapped memory and kernel
>>>> memory) is, as Gary said, the least special here. So using
>>>> memcpy_{from,to}io() would be overkill and probably misleading.
>>>
>>> Ok, I understand.
>>>
>>>> I
>>>> suggest we use `{read,write}_volatile()` (unless I'm missing something
>>>> subtle of course), however `{read,write}_volatile()` only works on Sized
>>>> types,
>>>
>>> We can copy as u8? Or would it be more efficient to copy as a larger size?
>>
>> Byte-wise atomic means that the atomicity is restricted to byte level (hence
>> it's okay to say if you read a u32 with it and does not observe an atomic
>> update). It does not mean that the access needs to be byte-wise, so it's
>> perfectly fine to do a 32-bit load and it'll still be byte-wise atomic.
>
> Ah.
>
>>
>>>
>>> You suggested atomic in the other email, did you abandon that idea?
>>
>> The semantics we want is byte-wise atomic, although as a impl detail, using
>> volatile for now is all that we need.
>>
>>>
>>>> so we may have to use `bindings::memcpy()` or
>>>> core::intrinsics::volatile_copy_memory() [1]
>>>
>>> I was looking at this one, but it is unstable behind `core_intrinsics`.
>>> I was uncertain about pulling in additional unstable features. This is
>>> why I was looking for something in the C kernel to use.
>>>
>>> I think `bindings::memcpy` is not guaranteed to be implemented as inline
>>> assembly, so it may not have volatile semantics?
>>
>> In absence of full language LTO as we have today, it'll be fine (in practice).
>> Unlike C, if you reference a symbol called "memcpy", it won't be treated as
>> special and get turned into non-volatile memcpy.
>>
>> If the volatile memcpy intrinsics is stable, then we can switch to use that.
>
> Got it, this aligns with what Boqun is writing. Let's go for that.
>
> It also looks like memcpy is implemented in assembly for arm, arm32,
> x86_64. Which would exempt it from LTO. Not sure about 32bit x86 though.
> It defers to `__memcpy`. I could not figure out what that resolves to.
> Is it from the compiler?

I think it's the one in arch/x86/include/asm/string_32.h? That is also inline
assembly.

There's no need to worry about if things can be optimized wrongly. I haven't
looked at the current defence against LTO when the code is implemented in C, but
As Boqun pointed out, the `memcpy` and `memmove` symbols are assumed to have
volatile semantics anyway. So the issue is not unique to Rust (also, we're
immune at the moment as there's no linker-plugin LTO support for Rust).

Ultimately, `volatile_copy_nonoverlapping_memory` is translated to `memcpy`
(similarly, `volatile_copy_memory` is `memmove`). The benefit of the intrinsics
is that if the size is fixed, it can be optimized a single volatile load/store
by LLVM.

Best,
Gary

>
>
> Best regards,
> Andreas Hindborg