v2: (Thanks to Peter Xu for reviewing!)
- New patch 1 to rename addr1 to mr_addr in the interests of meaningful naming.
- Take advantage of a cached address space only allow for a single MR to simplify
  the new code.
- Various cleanups of indentation etc.
- Cover letter and some patch descriptions updated to reflect changes.
- Changes all called out in specific patches.

Issue seen testing virtio-blk-pci with CXL emulated interleave memory.
Tests were done on arm64, but the issue isn't architecture specific.
Note that some additional fixes are needed to TCG to be able to run far
enough to hit this on arm64 or x86. Most of these are now upstream
with exception of:

target/i386: Enable page walking from MMIO memory
https://lore.kernel.org/qemu-devel/20240219173153.12114-3-Jonathan.Cameron@huawei.com/

The address_space_read_cached_slow() and address_space_write_cached_slow()
functions query the MemoryRegion for the cached address space correctly
using address_space_translate_cached() but then call into
flatview_read_continue() / flatview_write_continue().
If the access is to a MMIO MemoryRegion and is bigger than the MemoryRegion
supports, the loop will query the MemoryRegion for the next access to use.
That query uses flatview_translate() but the address passed is suitable
for the cache, not the flatview. On my test setup that mean the second
8 bytes and onwards of the virtio descriptor was read from flash memory
at the beginning of the system address map, not the CXL emulated memory
where the descriptor was found.  Result happened to be all fs so easy to
spot.

Changes these calls to assume that the MemoryRegion does not change
as multiple acceses are perfomed to the MemoryRegionCache.
The first patch renames the addr1 parameter to the hopefully more
informative mr_addr.

To avoid duplicating most of the code, the next 2 patches factor out
the common parts of flatview_read_continue() and flatview_write_continue()
so they can be reused.

Write path has not been tested but it so similar to the read path I've
included it here.

Jonathan Cameron (4):
  physmem: Rename addr1 to more informative mr_addr in
    flatview_read/write() and similar
  physmem: Reduce local variable scope in flatview_read/write_continue()
  physmem: Factor out body of flatview_read/write_continue() loop
  physmem: Fix wrong address in large
    address_space_read/write_cached_slow()

 system/physmem.c | 260 +++++++++++++++++++++++++++++++----------------
 1 file changed, 170 insertions(+), 90 deletions(-)

-- 
2.39.2

On Thu, Mar 07, 2024 at 03:37:06PM +0000, Jonathan Cameron wrote:
> v2: (Thanks to Peter Xu for reviewing!)
> - New patch 1 to rename addr1 to mr_addr in the interests of meaningful naming.
> - Take advantage of a cached address space only allow for a single MR to simplify
>   the new code.
> - Various cleanups of indentation etc.
> - Cover letter and some patch descriptions updated to reflect changes.
> - Changes all called out in specific patches.

All look good to me, thanks.  Having the new functions' first argument as
MemTxAttrs is slightly weird to me, but that's not a big deal and we can
clean it up later if wanted.  I guess it's good to fix this in 9.0 first as
it's a real bug even if not trivial to hit.

I queued it in my migration tree (with my "memory API" hat..).

I won't send a pull until next Monday.  Please shoot if there's any objections!

-- 
Peter Xu

On 8/3/24 08:36, Peter Xu wrote:

> I queued it in my migration tree (with my "memory API" hat..).
> 
> I won't send a pull until next Monday.  Please shoot if there's any objections!

Thanks!

On 08.03.24 08:36, Peter Xu wrote:
> On Thu, Mar 07, 2024 at 03:37:06PM +0000, Jonathan Cameron wrote:
>> v2: (Thanks to Peter Xu for reviewing!)
>> - New patch 1 to rename addr1 to mr_addr in the interests of meaningful naming.
>> - Take advantage of a cached address space only allow for a single MR to simplify
>>    the new code.
>> - Various cleanups of indentation etc.
>> - Cover letter and some patch descriptions updated to reflect changes.
>> - Changes all called out in specific patches.
> 
> All look good to me, thanks.  Having the new functions' first argument as
> MemTxAttrs is slightly weird to me, but that's not a big deal and we can
> clean it up later if wanted.  I guess it's good to fix this in 9.0 first as
> it's a real bug even if not trivial to hit.
> 
> I queued it in my migration tree (with my "memory API" hat..).
> 
> I won't send a pull until next Monday.  Please shoot if there's any objections!
> 

Works for me, thanks!

-- 
Cheers,

David / dhildenb