All ram device regions was turned to be indirectly accessible by commit
4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads
to a hanged guest where a NVidia GH100 GPU is passed from host. The memory
in its PCI BAR#4 can be allocated as DMA target buffer. qemu has to take
DMA bounce buffer in address_space_map() to cover the DMA request. However,
the bounce buffer size is 4096 bytes and we're overrunning it easily when
the guest has significant disk activities on compiling 'cuda-samples'.
The full log and problem description can be found from PATCH[1/2]'s commit
log.

Try to fix the issue handled in commit 4a2e242bbb by replacing memcopy()/
memmove() with newly added helpers qemu_ram_{copy, move}() that works on
top of __builtin_{memcpy, memmove} or unaligned access friendly memory
movement in the accessors to the ram device regions. With this, we can
basically revert that commit to make ram device region directly accessible
again and bypass the bounce buffer in address_space_map() where the guest
hang is caused.

PATCH[1] uses qemu_ram_{copy, move}() in ram device region accessors
PATCH[2] makes ram device region directly accessible again

Changelog
=========
v2 -> v3:
  * https://lore.kernel.org/qemu-arm/20260615100200.266968-1-gshan@redhat.com/
  * Documentation for qemu_ram_{copy, move}           (Peter/Michael)
  * Support qemu_ram_move() for overlapped src/dest   (Richard)
  * Use {memcpy, memmove} if step is 16-bytes or more (Michael)
  * Code improvements                                 (Richard/Michael)
v1 -> v2:
  * https://lore.kernel.org/qemu-arm/20260612110307.1264798-1-gshan@redhat.com/
  * Rename address_space_{memcpy, memmove}() to qemu_ram_{copy, move}()
    and move them to physmem.c and memory.h   (Philippe)
  * Use memcpy() and memmove() in qemu_ram_{copy, move}() for the variable
    length case                               (Miachel)
  * Handle unaligned access in qemu_ram_{copy, move}() for all archs
    except i386 and x86_64                    (Richard/Michael)
RFCv1 -> v1:
  * https://lists.nongnu.org/archive/html/qemu-arm/2026-06/msg00307.html
  * Reworked solution based on suggestions from Peter Xu, Peter Maydell
    and Michael S. Tsirkin

Gavin Shan (2):
  system/memory: Use qemu_ram_{copy, move}() in ram device region
    accessors
  system/memory: Make ram device region directly accessible

 hw/remote/vfio-user-obj.c |   4 +-
 include/system/memory.h   |  43 ++++++---
 system/memory.c           |  41 +--------
 system/physmem.c          | 178 +++++++++++++++++++++++++++++++++++++-
 system/trace-events       |   2 -
 5 files changed, 210 insertions(+), 58 deletions(-)

-- 
2.54.0

On 6/16/26 3:25 PM, Gavin Shan wrote:
> All ram device regions was turned to be indirectly accessible by commit
> 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads
> to a hanged guest where a NVidia GH100 GPU is passed from host. The memory
> in its PCI BAR#4 can be allocated as DMA target buffer. qemu has to take
> DMA bounce buffer in address_space_map() to cover the DMA request. However,
> the bounce buffer size is 4096 bytes and we're overrunning it easily when
> the guest has significant disk activities on compiling 'cuda-samples'.
> The full log and problem description can be found from PATCH[1/2]'s commit
> log.
> 
> Try to fix the issue handled in commit 4a2e242bbb by replacing memcopy()/
> memmove() with newly added helpers qemu_ram_{copy, move}() that works on
> top of __builtin_{memcpy, memmove} or unaligned access friendly memory
> movement in the accessors to the ram device regions. With this, we can
> basically revert that commit to make ram device region directly accessible
> again and bypass the bounce buffer in address_space_map() where the guest
> hang is caused.
> 
> PATCH[1] uses qemu_ram_{copy, move}() in ram device region accessors
> PATCH[2] makes ram device region directly accessible again
> 
Michael asked to include below context in the cover letter in v3, but I
didn't noticed that before I sent v3 series, appended with them.

----

The issues listed by Michael:

1. On x86, memcpy is different from __builtin_memcpy if one uses old 1.0
    force-headers from 2019. Likely no longer relevant.

2. variable length memcpy can translate 2,4,8 byte guest access into
    multiple byte accesses. doing this for mmio is guaranteed to break devices.

3. (theoretical concern) also on x86, unaligned accesses are possible on guest
    and host, so converting an unaligned access to a series of aligned ones can
    in theory break devices.

4. also on x86, vector instructions for large (>16 byte) writes into
    pgprot_noncached memory are safe and faster than multiple 8 byte ones.

5. also on x86 it so happens that if you write a fixed-size memcpy this gets
    optimized to a single store/load and it works for aligned and unaligned
    addresses on that architecture. How to ensure this keeps being correct
    is left as an excerise for the reader. But qemu already relies on this
    and did for years.

6. on non-x86 both unaligned accesses and vector instructions for accessing
    UC memory are illegal.

7. standard vfio gives KVM VM_ALLOW_ANY_UNCACHED, so even on non x86 guest can
    map the memory as as pgprot_noncached/ioremap or pgprot_writecombine/ioremap_uc.
    If it does the second then it can use unaligned or vector for access.
    This is why normal passthrough tends to work - it never traps to qemu at
    all. But for qemu, vfio uses  pgprot_noncached unconditionally so qemu
    can't use unaligned or vector instructions on non-x86.


8. But for nvgrace RAM, vfio has a driver that uses pgprot_writecombine/ioremap_uc.
    so qemu could safely use unaligned/vector instructioons even on non-x86.

9. Except sadly, vfio currently does not tell qemu how it maps
    the memory, so qemu can not know what is safe on non-x86.


Now, what is to be done?


A. on x86, we must avoid converting 2,4,8 byte accesses into byte accesses.
At least for aligned, perferably for unaligned accesses too.
Fixed width memcpy seems to work for this. Whether we should bother with
__builtin to work around broken old fortify headers, I donnu.
I do not have any answer how to check that compiler does this correctly.
If anyone is motivated enough, adding a GCC builtin could be possible.
Given qemu did this for years, I think we can leave solving this for
another day.

B. Also on x86, I do not see why we should not use memcpy for large
accesses if we can. Better perf.

C. on non-x86, we currently must not memcpy since we do not know if it
is pgprot_noncached. yes, performance will be bad for DMA into device RAM.

D. It goes without saying that casting an unaligned address to unint32_t
(be it for qatomic_set or whatever) is undefined behaviour in C
and so a bad idea on any architecture.

E. also for non-x86, we really should teach vfio to tell qemu whether
it maps device pgprot_noncached or pgprot_writecombine.
we will then be able to use memcpy for >8 accesses.

Anyone, correct me if I'm wrong? Maybe I should start a new thread with
this summary?

Thanks,
Gavin

On Tue, Jun 16, 2026 at 03:40:34PM +1000, Gavin Shan wrote:
> On 6/16/26 3:25 PM, Gavin Shan wrote:
> > All ram device regions was turned to be indirectly accessible by commit
> > 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads
> > to a hanged guest where a NVidia GH100 GPU is passed from host. The memory
> > in its PCI BAR#4 can be allocated as DMA target buffer. qemu has to take
> > DMA bounce buffer in address_space_map() to cover the DMA request. However,
> > the bounce buffer size is 4096 bytes and we're overrunning it easily when
> > the guest has significant disk activities on compiling 'cuda-samples'.
> > The full log and problem description can be found from PATCH[1/2]'s commit
> > log.
> > 
> > Try to fix the issue handled in commit 4a2e242bbb by replacing memcopy()/
> > memmove() with newly added helpers qemu_ram_{copy, move}() that works on
> > top of __builtin_{memcpy, memmove} or unaligned access friendly memory
> > movement in the accessors to the ram device regions. With this, we can
> > basically revert that commit to make ram device region directly accessible
> > again and bypass the bounce buffer in address_space_map() where the guest
> > hang is caused.
> > 
> > PATCH[1] uses qemu_ram_{copy, move}() in ram device region accessors
> > PATCH[2] makes ram device region directly accessible again
> > 
> Michael asked to include below context in the cover letter in v3, but I
> didn't noticed that before I sent v3 series, appended with them.
> 
> ----
> 
> The issues listed by Michael:
> 
> 1. On x86, memcpy is different from __builtin_memcpy if one uses old 1.0
>    force-headers from 2019. Likely no longer relevant.
> 
> 2. variable length memcpy can translate 2,4,8 byte guest access into
>    multiple byte accesses. doing this for mmio is guaranteed to break devices.
> 
> 3. (theoretical concern) also on x86, unaligned accesses are possible on guest
>    and host, so converting an unaligned access to a series of aligned ones can
>    in theory break devices.
> 
> 4. also on x86, vector instructions for large (>16 byte) writes into
>    pgprot_noncached memory are safe and faster than multiple 8 byte ones.
> 
> 5. also on x86 it so happens that if you write a fixed-size memcpy this gets
>    optimized to a single store/load and it works for aligned and unaligned
>    addresses on that architecture. How to ensure this keeps being correct
>    is left as an excerise for the reader. But qemu already relies on this
>    and did for years.
> 
> 6. on non-x86 both unaligned accesses and vector instructions for accessing
>    UC memory are illegal.
> 
> 7. standard vfio gives KVM VM_ALLOW_ANY_UNCACHED, so even on non x86 guest can
>    map the memory as as pgprot_noncached/ioremap or pgprot_writecombine/ioremap_uc.
>    If it does the second then it can use unaligned or vector for access.
>    This is why normal passthrough tends to work - it never traps to qemu at
>    all. But for qemu, vfio uses  pgprot_noncached unconditionally so qemu
>    can't use unaligned or vector instructions on non-x86.
> 
> 
> 8. But for nvgrace RAM, vfio has a driver that uses pgprot_writecombine/ioremap_uc.
>    so qemu could safely use unaligned/vector instructioons even on non-x86.
> 
> 9. Except sadly, vfio currently does not tell qemu how it maps
>    the memory, so qemu can not know what is safe on non-x86.
> 

And more:

10. on x86 memcpy will sometimes do multiple overlapping stores when
size is not a power of 2. for example, a 15 byte write is done with
2 8-byte stores. This is theoretically an issue
if guest does something super clever with ordering,
but does not seem to be in practice.

10. on non-x86 memcpy will do multiple overlapping stores even
for single byte writes. E.g. it does it to avoid extra branches.
This is causing issues in practice.





> Now, what is to be done?
> 
> 
> A. on x86, we must avoid converting 2,4,8 byte accesses into byte accesses.
> At least for aligned, perferably for unaligned accesses too.
> Fixed width memcpy seems to work for this. Whether we should bother with
> __builtin to work around broken old fortify headers, I donnu.
> I do not have any answer how to check that compiler does this correctly.
> If anyone is motivated enough, adding a GCC builtin could be possible.
> Given qemu did this for years, I think we can leave solving this for
> another day.
> 
> B. Also on x86, I do not see why we should not use memcpy for large
> accesses if we can. Better perf.
> 
> C. on non-x86, we currently must not memcpy since we do not know if it
> is pgprot_noncached. yes, performance will be bad for DMA into device RAM.
> 
> D. It goes without saying that casting an unaligned address to unint32_t
> (be it for qatomic_set or whatever) is undefined behaviour in C
> and so a bad idea on any architecture.
> 
> E. also for non-x86, we really should teach vfio to tell qemu whether
> it maps device pgprot_noncached or pgprot_writecombine.
> we will then be able to use memcpy for >8 accesses.
> 
> Anyone, correct me if I'm wrong? Maybe I should start a new thread with
> this summary?
> 
> Thanks,
> Gavin

On 6/16/26 3:44 PM, Michael S. Tsirkin wrote:
> On Tue, Jun 16, 2026 at 03:40:34PM +1000, Gavin Shan wrote:
>> On 6/16/26 3:25 PM, Gavin Shan wrote:
>>> All ram device regions was turned to be indirectly accessible by commit
>>> 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads
>>> to a hanged guest where a NVidia GH100 GPU is passed from host. The memory
>>> in its PCI BAR#4 can be allocated as DMA target buffer. qemu has to take
>>> DMA bounce buffer in address_space_map() to cover the DMA request. However,
>>> the bounce buffer size is 4096 bytes and we're overrunning it easily when
>>> the guest has significant disk activities on compiling 'cuda-samples'.
>>> The full log and problem description can be found from PATCH[1/2]'s commit
>>> log.
>>>
>>> Try to fix the issue handled in commit 4a2e242bbb by replacing memcopy()/
>>> memmove() with newly added helpers qemu_ram_{copy, move}() that works on
>>> top of __builtin_{memcpy, memmove} or unaligned access friendly memory
>>> movement in the accessors to the ram device regions. With this, we can
>>> basically revert that commit to make ram device region directly accessible
>>> again and bypass the bounce buffer in address_space_map() where the guest
>>> hang is caused.
>>>
>>> PATCH[1] uses qemu_ram_{copy, move}() in ram device region accessors
>>> PATCH[2] makes ram device region directly accessible again
>>>
>> Michael asked to include below context in the cover letter in v3, but I
>> didn't noticed that before I sent v3 series, appended with them.
>>

Looking at the list of issues (questions) raised by Michael, I don't understand
every one before I'm able to put more time to dig, but I feel this series has
too ambitious goal to cover accesses to all the directly accessible regions
with the newly introduced qemu_ram_{copy, move}. It causes too many behavior
changes and concerns, making this series impossible to land.

I would suggest to break down the goal and step back to apply the newly introduced
qemu_ram_{copy, move} to the ram device regions only? It's actually something
proposed by Peter Xu in the earlier replies. Taking address_space_write() as an
example, the indirectly accessible regions are covered by memory_region_dispatch_write()
in (1), the ram device region is covered by qemu_ram_move() in (2), and all other
directly accessible regions are covered by memmove() in (3).

   address_space_write
     flatview_write
       flatview_write_continue
         flatview_write_continue_step
           memory_access_size             // (1) indirectly accessible region
           memory_region_dispatch_write
             access_with_adjusted_size
               memory_region_write_accessor
                 mr->ops->write
           qemu_ram_move                  // (2) ram device region
           memmove                        // (3) all other directly accessible regions

With the limitation, only the ram device regions in (2) are affected. We're
basically moving the accesses to the ram device region from (1) to (2). No
changes introduced to other types of regions. The goal is to make the ram device
region accessible so that the bounce buffer can be bypassed in DMA path.

>> ----
>>
>> The issues listed by Michael:
>>
>> 1. On x86, memcpy is different from __builtin_memcpy if one uses old 1.0
>>     force-headers from 2019. Likely no longer relevant.
>>
>> 2. variable length memcpy can translate 2,4,8 byte guest access into
>>     multiple byte accesses. doing this for mmio is guaranteed to break devices.
>>
>> 3. (theoretical concern) also on x86, unaligned accesses are possible on guest
>>     and host, so converting an unaligned access to a series of aligned ones can
>>     in theory break devices.
>>
>> 4. also on x86, vector instructions for large (>16 byte) writes into
>>     pgprot_noncached memory are safe and faster than multiple 8 byte ones.
>>
>> 5. also on x86 it so happens that if you write a fixed-size memcpy this gets
>>     optimized to a single store/load and it works for aligned and unaligned
>>     addresses on that architecture. How to ensure this keeps being correct
>>     is left as an excerise for the reader. But qemu already relies on this
>>     and did for years.
>>
>> 6. on non-x86 both unaligned accesses and vector instructions for accessing
>>     UC memory are illegal.
>>
>> 7. standard vfio gives KVM VM_ALLOW_ANY_UNCACHED, so even on non x86 guest can
>>     map the memory as as pgprot_noncached/ioremap or pgprot_writecombine/ioremap_uc.
>>     If it does the second then it can use unaligned or vector for access.
>>     This is why normal passthrough tends to work - it never traps to qemu at
>>     all. But for qemu, vfio uses  pgprot_noncached unconditionally so qemu
>>     can't use unaligned or vector instructions on non-x86.
>>
>>
>> 8. But for nvgrace RAM, vfio has a driver that uses pgprot_writecombine/ioremap_uc.
>>     so qemu could safely use unaligned/vector instructioons even on non-x86.
>>
>> 9. Except sadly, vfio currently does not tell qemu how it maps
>>     the memory, so qemu can not know what is safe on non-x86.
>>
> 
> And more:
> 
> 10. on x86 memcpy will sometimes do multiple overlapping stores when
> size is not a power of 2. for example, a 15 byte write is done with
> 2 8-byte stores. This is theoretically an issue
> if guest does something super clever with ordering,
> but does not seem to be in practice.
> 
> 10. on non-x86 memcpy will do multiple overlapping stores even
> for single byte writes. E.g. it does it to avoid extra branches.
> This is causing issues in practice.
> 
> 
> 
> 
> 
>> Now, what is to be done?
>>
>>
>> A. on x86, we must avoid converting 2,4,8 byte accesses into byte accesses.
>> At least for aligned, perferably for unaligned accesses too.
>> Fixed width memcpy seems to work for this. Whether we should bother with
>> __builtin to work around broken old fortify headers, I donnu.
>> I do not have any answer how to check that compiler does this correctly.
>> If anyone is motivated enough, adding a GCC builtin could be possible.
>> Given qemu did this for years, I think we can leave solving this for
>> another day.
>>
>> B. Also on x86, I do not see why we should not use memcpy for large
>> accesses if we can. Better perf.
>>
>> C. on non-x86, we currently must not memcpy since we do not know if it
>> is pgprot_noncached. yes, performance will be bad for DMA into device RAM.
>>
>> D. It goes without saying that casting an unaligned address to unint32_t
>> (be it for qatomic_set or whatever) is undefined behaviour in C
>> and so a bad idea on any architecture.
>>
>> E. also for non-x86, we really should teach vfio to tell qemu whether
>> it maps device pgprot_noncached or pgprot_writecombine.
>> we will then be able to use memcpy for >8 accesses.
>>
>> Anyone, correct me if I'm wrong? Maybe I should start a new thread with
>> this summary?
>>

Thanks,
Gavin

On Wed, Jun 17, 2026 at 12:35:00PM +1000, Gavin Shan wrote:
> On 6/16/26 3:44 PM, Michael S. Tsirkin wrote:
> > On Tue, Jun 16, 2026 at 03:40:34PM +1000, Gavin Shan wrote:
> > > On 6/16/26 3:25 PM, Gavin Shan wrote:
> > > > All ram device regions was turned to be indirectly accessible by commit
> > > > 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads
> > > > to a hanged guest where a NVidia GH100 GPU is passed from host. The memory
> > > > in its PCI BAR#4 can be allocated as DMA target buffer. qemu has to take
> > > > DMA bounce buffer in address_space_map() to cover the DMA request. However,
> > > > the bounce buffer size is 4096 bytes and we're overrunning it easily when
> > > > the guest has significant disk activities on compiling 'cuda-samples'.
> > > > The full log and problem description can be found from PATCH[1/2]'s commit
> > > > log.
> > > > 
> > > > Try to fix the issue handled in commit 4a2e242bbb by replacing memcopy()/
> > > > memmove() with newly added helpers qemu_ram_{copy, move}() that works on
> > > > top of __builtin_{memcpy, memmove} or unaligned access friendly memory
> > > > movement in the accessors to the ram device regions. With this, we can
> > > > basically revert that commit to make ram device region directly accessible
> > > > again and bypass the bounce buffer in address_space_map() where the guest
> > > > hang is caused.
> > > > 
> > > > PATCH[1] uses qemu_ram_{copy, move}() in ram device region accessors
> > > > PATCH[2] makes ram device region directly accessible again
> > > > 
> > > Michael asked to include below context in the cover letter in v3, but I
> > > didn't noticed that before I sent v3 series, appended with them.
> > > 
> 
> Looking at the list of issues (questions) raised by Michael, I don't understand
> every one

Gavin, I doubt one should make memory.c changes without understanding the issues
it is trying to address.

What is unclear? Ask away.


> before I'm able to put more time to dig, but I feel this series has
> too ambitious goal to cover accesses to all the directly accessible regions
> with the newly introduced qemu_ram_{copy, move}. It causes too many behavior
> changes and concerns, making this series impossible to land.
> 
> I would suggest to break down the goal and step back to apply the newly introduced
> qemu_ram_{copy, move} to the ram device regions only? It's actually something
> proposed by Peter Xu in the earlier replies. Taking address_space_write() as an
> example, the indirectly accessible regions are covered by memory_region_dispatch_write()
> in (1), the ram device region is covered by qemu_ram_move() in (2), and all other
> directly accessible regions are covered by memmove() in (3).
> 
>   address_space_write
>     flatview_write
>       flatview_write_continue
>         flatview_write_continue_step
>           memory_access_size             // (1) indirectly accessible region
>           memory_region_dispatch_write
>             access_with_adjusted_size
>               memory_region_write_accessor
>                 mr->ops->write
>           qemu_ram_move                  // (2) ram device region
>           memmove                        // (3) all other directly accessible regions
> 
> With the limitation, only the ram device regions in (2) are affected. We're
> basically moving the accesses to the ram device region from (1) to (2). No
> changes introduced to other types of regions. The goal is to make the ram device
> region accessible so that the bounce buffer can be bypassed in DMA path.

Esthetics aside - ram device regions have all the same issues.

Maybe you can limit the scope of the changes,
but I doubt you can get out understanding)


> > > ----
> > > 
> > > The issues listed by Michael:
> > > 
> > > 1. On x86, memcpy is different from __builtin_memcpy if one uses old 1.0
> > >     force-headers from 2019. Likely no longer relevant.
> > > 
> > > 2. variable length memcpy can translate 2,4,8 byte guest access into
> > >     multiple byte accesses. doing this for mmio is guaranteed to break devices.
> > > 
> > > 3. (theoretical concern) also on x86, unaligned accesses are possible on guest
> > >     and host, so converting an unaligned access to a series of aligned ones can
> > >     in theory break devices.
> > > 
> > > 4. also on x86, vector instructions for large (>16 byte) writes into
> > >     pgprot_noncached memory are safe and faster than multiple 8 byte ones.
> > > 
> > > 5. also on x86 it so happens that if you write a fixed-size memcpy this gets
> > >     optimized to a single store/load and it works for aligned and unaligned
> > >     addresses on that architecture. How to ensure this keeps being correct
> > >     is left as an excerise for the reader. But qemu already relies on this
> > >     and did for years.
> > > 
> > > 6. on non-x86 both unaligned accesses and vector instructions for accessing
> > >     UC memory are illegal.
> > > 
> > > 7. standard vfio gives KVM VM_ALLOW_ANY_UNCACHED, so even on non x86 guest can
> > >     map the memory as as pgprot_noncached/ioremap or pgprot_writecombine/ioremap_uc.
> > >     If it does the second then it can use unaligned or vector for access.
> > >     This is why normal passthrough tends to work - it never traps to qemu at
> > >     all. But for qemu, vfio uses  pgprot_noncached unconditionally so qemu
> > >     can't use unaligned or vector instructions on non-x86.
> > > 
> > > 
> > > 8. But for nvgrace RAM, vfio has a driver that uses pgprot_writecombine/ioremap_uc.
> > >     so qemu could safely use unaligned/vector instructioons even on non-x86.
> > > 
> > > 9. Except sadly, vfio currently does not tell qemu how it maps
> > >     the memory, so qemu can not know what is safe on non-x86.
> > > 
> > 
> > And more:
> > 
> > 10. on x86 memcpy will sometimes do multiple overlapping stores when
> > size is not a power of 2. for example, a 15 byte write is done with
> > 2 8-byte stores. This is theoretically an issue
> > if guest does something super clever with ordering,
> > but does not seem to be in practice.
> > 
> > 10. on non-x86 memcpy will do multiple overlapping stores even
> > for single byte writes. E.g. it does it to avoid extra branches.
> > This is causing issues in practice.
> > 
> > 
> > 
> > 
> > 
> > > Now, what is to be done?
> > > 
> > > 
> > > A. on x86, we must avoid converting 2,4,8 byte accesses into byte accesses.
> > > At least for aligned, perferably for unaligned accesses too.
> > > Fixed width memcpy seems to work for this. Whether we should bother with
> > > __builtin to work around broken old fortify headers, I donnu.
> > > I do not have any answer how to check that compiler does this correctly.
> > > If anyone is motivated enough, adding a GCC builtin could be possible.
> > > Given qemu did this for years, I think we can leave solving this for
> > > another day.
> > > 
> > > B. Also on x86, I do not see why we should not use memcpy for large
> > > accesses if we can. Better perf.
> > > 
> > > C. on non-x86, we currently must not memcpy since we do not know if it
> > > is pgprot_noncached. yes, performance will be bad for DMA into device RAM.
> > > 
> > > D. It goes without saying that casting an unaligned address to unint32_t
> > > (be it for qatomic_set or whatever) is undefined behaviour in C
> > > and so a bad idea on any architecture.
> > > 
> > > E. also for non-x86, we really should teach vfio to tell qemu whether
> > > it maps device pgprot_noncached or pgprot_writecombine.
> > > we will then be able to use memcpy for >8 accesses.
> > > 
> > > Anyone, correct me if I'm wrong? Maybe I should start a new thread with
> > > this summary?
> > > 
> 
> Thanks,
> Gavin

Hi Michael,

On 6/17/26 3:52 PM, Michael S. Tsirkin wrote:
> On Wed, Jun 17, 2026 at 12:35:00PM +1000, Gavin Shan wrote:
>> On 6/16/26 3:44 PM, Michael S. Tsirkin wrote:
>>> On Tue, Jun 16, 2026 at 03:40:34PM +1000, Gavin Shan wrote:
>>>> On 6/16/26 3:25 PM, Gavin Shan wrote:
>>>>> All ram device regions was turned to be indirectly accessible by commit
>>>>> 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads
>>>>> to a hanged guest where a NVidia GH100 GPU is passed from host. The memory
>>>>> in its PCI BAR#4 can be allocated as DMA target buffer. qemu has to take
>>>>> DMA bounce buffer in address_space_map() to cover the DMA request. However,
>>>>> the bounce buffer size is 4096 bytes and we're overrunning it easily when
>>>>> the guest has significant disk activities on compiling 'cuda-samples'.
>>>>> The full log and problem description can be found from PATCH[1/2]'s commit
>>>>> log.
>>>>>
>>>>> Try to fix the issue handled in commit 4a2e242bbb by replacing memcopy()/
>>>>> memmove() with newly added helpers qemu_ram_{copy, move}() that works on
>>>>> top of __builtin_{memcpy, memmove} or unaligned access friendly memory
>>>>> movement in the accessors to the ram device regions. With this, we can
>>>>> basically revert that commit to make ram device region directly accessible
>>>>> again and bypass the bounce buffer in address_space_map() where the guest
>>>>> hang is caused.
>>>>>
>>>>> PATCH[1] uses qemu_ram_{copy, move}() in ram device region accessors
>>>>> PATCH[2] makes ram device region directly accessible again
>>>>>
>>>> Michael asked to include below context in the cover letter in v3, but I
>>>> didn't noticed that before I sent v3 series, appended with them.
>>>>
>>
>> Looking at the list of issues (questions) raised by Michael, I don't understand
>> every one
> 
> Gavin, I doubt one should make memory.c changes without understanding the issues
> it is trying to address.
> 
> What is unclear? Ask away.
> 

Yeah, absolutely. I need some time to understand all questions or suggestions
by digging the code a bit, before I'm able to come back to you, but not very
soon though :-)

> 
>> before I'm able to put more time to dig, but I feel this series has
>> too ambitious goal to cover accesses to all the directly accessible regions
>> with the newly introduced qemu_ram_{copy, move}. It causes too many behavior
>> changes and concerns, making this series impossible to land.
>>
>> I would suggest to break down the goal and step back to apply the newly introduced
>> qemu_ram_{copy, move} to the ram device regions only? It's actually something
>> proposed by Peter Xu in the earlier replies. Taking address_space_write() as an
>> example, the indirectly accessible regions are covered by memory_region_dispatch_write()
>> in (1), the ram device region is covered by qemu_ram_move() in (2), and all other
>> directly accessible regions are covered by memmove() in (3).
>>
>>    address_space_write
>>      flatview_write
>>        flatview_write_continue
>>          flatview_write_continue_step
>>            memory_access_size             // (1) indirectly accessible region
>>            memory_region_dispatch_write
>>              access_with_adjusted_size
>>                memory_region_write_accessor
>>                  mr->ops->write
>>            qemu_ram_move                  // (2) ram device region
>>            memmove                        // (3) all other directly accessible regions
>>
>> With the limitation, only the ram device regions in (2) are affected. We're
>> basically moving the accesses to the ram device region from (1) to (2). No
>> changes introduced to other types of regions. The goal is to make the ram device
>> region accessible so that the bounce buffer can be bypassed in DMA path.
> 
> Esthetics aside - ram device regions have all the same issues.
> 
> Maybe you can limit the scope of the changes,
> but I doubt you can get out understanding)
> 

Yes. Limited to the scope of VFIO and PCI BARs, it depends on how the PCI BARs
are mapped, pgprot_noncached or pgprot_writecombine. The listed problems and
concerns are existing on the ram device region exposed with pgprot_noncached.
However, everything should be just fine if the region is exposed with pgprot_writecombine.
Am I understanding this correctly?

[...]

Thanks,
Gavin

On Wed, Jun 17, 2026 at 05:00:45PM +1000, Gavin Shan wrote:
> Hi Michael,
> 
> On 6/17/26 3:52 PM, Michael S. Tsirkin wrote:
> > On Wed, Jun 17, 2026 at 12:35:00PM +1000, Gavin Shan wrote:
> > > On 6/16/26 3:44 PM, Michael S. Tsirkin wrote:
> > > > On Tue, Jun 16, 2026 at 03:40:34PM +1000, Gavin Shan wrote:
> > > > > On 6/16/26 3:25 PM, Gavin Shan wrote:
> > > > > > All ram device regions was turned to be indirectly accessible by commit
> > > > > > 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads
> > > > > > to a hanged guest where a NVidia GH100 GPU is passed from host. The memory
> > > > > > in its PCI BAR#4 can be allocated as DMA target buffer. qemu has to take
> > > > > > DMA bounce buffer in address_space_map() to cover the DMA request. However,
> > > > > > the bounce buffer size is 4096 bytes and we're overrunning it easily when
> > > > > > the guest has significant disk activities on compiling 'cuda-samples'.
> > > > > > The full log and problem description can be found from PATCH[1/2]'s commit
> > > > > > log.
> > > > > > 
> > > > > > Try to fix the issue handled in commit 4a2e242bbb by replacing memcopy()/
> > > > > > memmove() with newly added helpers qemu_ram_{copy, move}() that works on
> > > > > > top of __builtin_{memcpy, memmove} or unaligned access friendly memory
> > > > > > movement in the accessors to the ram device regions. With this, we can
> > > > > > basically revert that commit to make ram device region directly accessible
> > > > > > again and bypass the bounce buffer in address_space_map() where the guest
> > > > > > hang is caused.
> > > > > > 
> > > > > > PATCH[1] uses qemu_ram_{copy, move}() in ram device region accessors
> > > > > > PATCH[2] makes ram device region directly accessible again
> > > > > > 
> > > > > Michael asked to include below context in the cover letter in v3, but I
> > > > > didn't noticed that before I sent v3 series, appended with them.
> > > > > 
> > > 
> > > Looking at the list of issues (questions) raised by Michael, I don't understand
> > > every one
> > 
> > Gavin, I doubt one should make memory.c changes without understanding the issues
> > it is trying to address.
> > 
> > What is unclear? Ask away.
> > 
> 
> Yeah, absolutely. I need some time to understand all questions or suggestions
> by digging the code a bit, before I'm able to come back to you, but not very
> soon though :-)
> 
> > 
> > > before I'm able to put more time to dig, but I feel this series has
> > > too ambitious goal to cover accesses to all the directly accessible regions
> > > with the newly introduced qemu_ram_{copy, move}. It causes too many behavior
> > > changes and concerns, making this series impossible to land.
> > > 
> > > I would suggest to break down the goal and step back to apply the newly introduced
> > > qemu_ram_{copy, move} to the ram device regions only? It's actually something
> > > proposed by Peter Xu in the earlier replies. Taking address_space_write() as an
> > > example, the indirectly accessible regions are covered by memory_region_dispatch_write()
> > > in (1), the ram device region is covered by qemu_ram_move() in (2), and all other
> > > directly accessible regions are covered by memmove() in (3).
> > > 
> > >    address_space_write
> > >      flatview_write
> > >        flatview_write_continue
> > >          flatview_write_continue_step
> > >            memory_access_size             // (1) indirectly accessible region
> > >            memory_region_dispatch_write
> > >              access_with_adjusted_size
> > >                memory_region_write_accessor
> > >                  mr->ops->write
> > >            qemu_ram_move                  // (2) ram device region
> > >            memmove                        // (3) all other directly accessible regions
> > > 
> > > With the limitation, only the ram device regions in (2) are affected. We're
> > > basically moving the accesses to the ram device region from (1) to (2). No
> > > changes introduced to other types of regions. The goal is to make the ram device
> > > region accessible so that the bounce buffer can be bypassed in DMA path.
> > 
> > Esthetics aside - ram device regions have all the same issues.
> > 
> > Maybe you can limit the scope of the changes,
> > but I doubt you can get out understanding)
> > 
> 
> Yes. Limited to the scope of VFIO and PCI BARs, it depends on how the PCI BARs
> are mapped, pgprot_noncached or pgprot_writecombine. The listed problems and
> concerns are existing on the ram device region exposed with pgprot_noncached.
> However, everything should be just fine if the region is exposed with pgprot_writecombine.
> Am I understanding this correctly?

Not everything)

We have issues around guest RAM access, too.

But yes you can then mostly access device RAM as guest RAM on
arm and x86. I am not sure about power. for power pgprot_writecombine
is cache inhibited and I am not sure e.g. vector instructions
with misaligned addresses behave the same.



> [...]
> 
> Thanks,
> Gavin

On Tue, Jun 16, 2026 at 03:25:50PM +1000, Gavin Shan wrote:
> All ram device regions was turned to be indirectly accessible by commit
> 4a2e242bbb ("memory: Don't use memcpy for ram_device regions").


What is missing here is the list of issues around
direct ram access and how we are solving them.

Thanks,
-- 
MST

On 6/16/26 3:36 PM, Michael S. Tsirkin wrote:
> On Tue, Jun 16, 2026 at 03:25:50PM +1000, Gavin Shan wrote:
>> All ram device regions was turned to be indirectly accessible by commit
>> 4a2e242bbb ("memory: Don't use memcpy for ram_device regions").
> 
> 
> What is missing here is the list of issues around
> direct ram access and how we are solving them.
> 

Sorry that I saw your request for the inclusion after v3 series was
posted. They have been appended to the thread as a reply.

Thanks,
Gavin

> Thanks,