From nobody Thu Oct 30 15:31:12 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1525817857738437.03970147294854; Tue, 8 May 2018 15:17:37 -0700 (PDT) Received: from localhost ([::1]:53517 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fGAvQ-0004XD-Mc for importer@patchew.org; Tue, 08 May 2018 18:17:36 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41249) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fGAt0-0002ym-H9 for qemu-devel@nongnu.org; Tue, 08 May 2018 18:15:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fGAsy-0001N7-7c for qemu-devel@nongnu.org; Tue, 08 May 2018 18:15:06 -0400 Received: from mail-wm0-x242.google.com ([2a00:1450:400c:c09::242]:33816) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fGAsx-0001MT-TB for qemu-devel@nongnu.org; Tue, 08 May 2018 18:15:04 -0400 Received: by mail-wm0-x242.google.com with SMTP id a137-v6so20817189wme.1 for ; Tue, 08 May 2018 15:15:03 -0700 (PDT) Received: from 640k.lan (dynamic-adsl-78-12-189-60.clienti.tiscali.it. [78.12.189.60]) by smtp.gmail.com with ESMTPSA id c15-v6sm14020129edr.78.2018.05.08.15.15.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 May 2018 15:15:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:subject:date:message-id:in-reply-to:references; bh=LpNXSD8S3Rn5TFLxgUO30t+836FzgJu0APTnSrUKwHo=; b=DjCkNohenr4Ik1t5X2VFTpRaq3K/H7Ui8U1CPVdBFSjHcs0/7Z5E1G7TWVY1nZjkLE 7LIyQvcz5KeQjwvCqIajYHpejLQKWVH2cbhj4JIfVjUbv5wfJ6wHv60xedRVt1xOPKNY Vf1YFUkoiuz4nYDIcnb7TI3gfqVESCI226Gj89Q6NYL62r9Wy3IhGzN7q1D2EWViiDuK zPq2Dw+U7+7zOhAZouLvha2mQ3uWeB3u2xU8UBDDCbf5XiTxrf82hXFPmDvuQGKNfls4 GQXzVq2Hr+UqOQkMvI4O9txx7UqW/c6oh/GzELu78smabxy+EfcqaMXBo/WX5S9xTpOG jjuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:subject:date:message-id :in-reply-to:references; bh=LpNXSD8S3Rn5TFLxgUO30t+836FzgJu0APTnSrUKwHo=; b=dvGzc/cKcVRV2mqF0x8wr/hp138imAmHTYNvaaIn26pUr1dckwEn2WGBak/3ERsXuk r2VjMmyss1j02iMoXIruja682H0TYsjEyZ4BMiF3nntGhXBhw5LCD/LlM70aaDwhjCvR AGJOrvNA71blO/fJ07gIKkKsfOGvwI66Btzcf/MRsd31YgkHrAtmL6Xj9gM/+gy1TcIf Kt8mS+W2wszZkcYLckgdLpRx8x4PFP7Et75c0IV2txP7z20zKRDyVCDEQRctMALVwDUl F304KIc78EhrO/n7nsBKIECS/lcYw99sYs9U5QchdWJi/NHahkTglulY5wlzqACMssDS HwiA== X-Gm-Message-State: ALQs6tCScMExF7ts/mscmzr39zn8ejuUVPtyunPK78xuhz1ttOqAjHcG C26GSlNfrWgX7bfvtdRD1jZ8GPsi X-Google-Smtp-Source: AB8JxZrQF8Nkrpl6QhnE+ZD/rAIJ94Shlx6hoLBOLXOwDI8jrN432KSJwxdHeFjA2Xajm+A8AIPoKw== X-Received: by 2002:a50:aca4:: with SMTP id x33-v6mr57006646edc.270.1525817702572; Tue, 08 May 2018 15:15:02 -0700 (PDT) From: Paolo Bonzini To: qemu-devel@nongnu.org Date: Wed, 9 May 2018 00:14:27 +0200 Message-Id: <1525817687-34620-11-git-send-email-pbonzini@redhat.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1525817687-34620-1-git-send-email-pbonzini@redhat.com> References: <1525817687-34620-1-git-send-email-pbonzini@redhat.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::242 Subject: [Qemu-devel] [PULL 10/30] exec: reintroduce MemoryRegion caching X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" MemoryRegionCache was reverted to "normal" address_space_* operations for 2.9, due to lack of support for IOMMUs. Reinstate the optimizations, caching only the IOMMU translation at address_cache_init but not the IOMMU lookup and target AddressSpace translation are not cached; now that MemoryRegionCache supports IOMMUs, it becomes more widely applicable too. The inlined fast path is defined in memory_ldst_cached.inc.h, while the slow path uses memory_ldst.inc.c as before. The smaller fast path causes a little code size reduction in MemoryRegionCache users: hw/virtio/virtio.o text size before: 32373 hw/virtio/virtio.o text size after: 31941 Signed-off-by: Paolo Bonzini --- exec.c | 121 ++++++++++++++++++++++++++++++= ---- include/exec/cpu-all.h | 6 +- include/exec/memory-internal.h | 3 + include/exec/memory.h | 58 ++++++++++++++-- include/exec/memory_ldst_cached.inc.h | 108 ++++++++++++++++++++++++++++++ memory.c | 4 +- 6 files changed, 280 insertions(+), 20 deletions(-) create mode 100644 include/exec/memory_ldst_cached.inc.h diff --git a/exec.c b/exec.c index 5f98106..ffa1099 100644 --- a/exec.c +++ b/exec.c @@ -3641,33 +3641,130 @@ int64_t address_space_cache_init(MemoryRegionCache= *cache, hwaddr len, bool is_write) { - cache->len =3D len; - cache->as =3D as; - cache->xlat =3D addr; - return len; + AddressSpaceDispatch *d; + hwaddr l; + MemoryRegion *mr; + + assert(len > 0); + + l =3D len; + cache->fv =3D address_space_get_flatview(as); + d =3D flatview_to_dispatch(cache->fv); + cache->mrs =3D *address_space_translate_internal(d, addr, &cache->xlat= , &l, true); + + mr =3D cache->mrs.mr; + memory_region_ref(mr); + if (memory_access_is_direct(mr, is_write)) { + l =3D flatview_extend_translation(cache->fv, addr, len, mr, + cache->xlat, l, is_write); + cache->ptr =3D qemu_ram_ptr_length(mr->ram_block, cache->xlat, &l,= true); + } else { + cache->ptr =3D NULL; + } + + cache->len =3D l; + cache->is_write =3D is_write; + return l; } =20 void address_space_cache_invalidate(MemoryRegionCache *cache, hwaddr addr, hwaddr access_len) { + assert(cache->is_write); + if (likely(cache->ptr)) { + invalidate_and_set_dirty(cache->mrs.mr, addr + cache->xlat, access= _len); + } } =20 void address_space_cache_destroy(MemoryRegionCache *cache) { - cache->as =3D NULL; + if (!cache->mrs.mr) { + return; + } + + if (xen_enabled()) { + xen_invalidate_map_cache_entry(cache->ptr); + } + memory_region_unref(cache->mrs.mr); + flatview_unref(cache->fv); + cache->mrs.mr =3D NULL; + cache->fv =3D NULL; +} + +/* Called from RCU critical section. This function has the same + * semantics as address_space_translate, but it only works on a + * predefined range of a MemoryRegion that was mapped with + * address_space_cache_init. + */ +static inline MemoryRegion *address_space_translate_cached( + MemoryRegionCache *cache, hwaddr addr, hwaddr *xlat, + hwaddr *plen, bool is_write) +{ + MemoryRegionSection section; + MemoryRegion *mr; + IOMMUMemoryRegion *iommu_mr; + AddressSpace *target_as; + + assert(!cache->ptr); + *xlat =3D addr + cache->xlat; + + mr =3D cache->mrs.mr; + iommu_mr =3D memory_region_get_iommu(mr); + if (!iommu_mr) { + /* MMIO region. */ + return mr; + } + + section =3D address_space_translate_iommu(iommu_mr, xlat, plen, + NULL, is_write, true, + &target_as); + return section.mr; +} + +/* Called from RCU critical section. address_space_read_cached uses this + * out of line function when the target is an MMIO or IOMMU region. + */ +void +address_space_read_cached_slow(MemoryRegionCache *cache, hwaddr addr, + void *buf, int len) +{ + hwaddr addr1, l; + MemoryRegion *mr; + + l =3D len; + mr =3D address_space_translate_cached(cache, addr, &addr1, &l, false); + flatview_read_continue(cache->fv, + addr, MEMTXATTRS_UNSPECIFIED, buf, len, + addr1, l, mr); +} + +/* Called from RCU critical section. address_space_write_cached uses this + * out of line function when the target is an MMIO or IOMMU region. + */ +void +address_space_write_cached_slow(MemoryRegionCache *cache, hwaddr addr, + const void *buf, int len) +{ + hwaddr addr1, l; + MemoryRegion *mr; + + l =3D len; + mr =3D address_space_translate_cached(cache, addr, &addr1, &l, true); + flatview_write_continue(cache->fv, + addr, MEMTXATTRS_UNSPECIFIED, buf, len, + addr1, l, mr); } =20 #define ARG1_DECL MemoryRegionCache *cache #define ARG1 cache -#define SUFFIX _cached -#define TRANSLATE(addr, ...) \ - address_space_translate(cache->as, cache->xlat + (addr), __VA_ARGS__) -#define IS_DIRECT(mr, is_write) true -#define MAP_RAM(mr, ofs) qemu_map_ram_ptr((mr)->ram_block, ofs) +#define SUFFIX _cached_slow +#define TRANSLATE(...) address_space_translate_cached(cache, __V= A_ARGS__) +#define IS_DIRECT(mr, is_write) memory_access_is_direct(mr, is_write) +#define MAP_RAM(mr, ofs) (cache->ptr + (ofs - cache->xlat)) #define INVALIDATE(mr, ofs, len) invalidate_and_set_dirty(mr, ofs, len) -#define RCU_READ_LOCK() rcu_read_lock() -#define RCU_READ_UNLOCK() rcu_read_unlock() +#define RCU_READ_LOCK() ((void)0) +#define RCU_READ_UNLOCK() ((void)0) #include "memory_ldst.inc.c" =20 /* virtual memory access for debug (includes writing to ROM) */ diff --git a/include/exec/cpu-all.h b/include/exec/cpu-all.h index 173edd1..a635f53 100644 --- a/include/exec/cpu-all.h +++ b/include/exec/cpu-all.h @@ -175,7 +175,7 @@ extern unsigned long reserved_va; #define TARGET_ENDIANNESS #include "exec/memory_ldst.inc.h" =20 -#define SUFFIX _cached +#define SUFFIX _cached_slow #define ARG1 cache #define ARG1_DECL MemoryRegionCache *cache #define TARGET_ENDIANNESS @@ -193,6 +193,10 @@ static inline void stl_phys_notdirty(AddressSpace *as,= hwaddr addr, uint32_t val #define TARGET_ENDIANNESS #include "exec/memory_ldst_phys.inc.h" =20 +/* Inline fast path for direct RAM access. */ +#define ENDIANNESS +#include "exec/memory_ldst_cached.inc.h" + #define SUFFIX _cached #define ARG1 cache #define ARG1_DECL MemoryRegionCache *cache diff --git a/include/exec/memory-internal.h b/include/exec/memory-internal.h index 6a5ee42..58399b9 100644 --- a/include/exec/memory-internal.h +++ b/include/exec/memory-internal.h @@ -31,6 +31,9 @@ static inline AddressSpaceDispatch *address_space_to_disp= atch(AddressSpace *as) return flatview_to_dispatch(address_space_to_flatview(as)); } =20 +FlatView *address_space_get_flatview(AddressSpace *as); +void flatview_unref(FlatView *view); + extern const MemoryRegionOps unassigned_mem_ops; =20 bool memory_region_access_valid(MemoryRegion *mr, hwaddr addr, diff --git a/include/exec/memory.h b/include/exec/memory.h index ca361bc..525619a 100644 --- a/include/exec/memory.h +++ b/include/exec/memory.h @@ -1688,12 +1688,16 @@ MemTxResult address_space_write(AddressSpace *as, h= waddr addr, #include "exec/memory_ldst_phys.inc.h" =20 struct MemoryRegionCache { + void *ptr; hwaddr xlat; hwaddr len; - AddressSpace *as; + FlatView *fv; + MemoryRegionSection mrs; + bool is_write; }; =20 -#define MEMORY_REGION_CACHE_INVALID ((MemoryRegionCache) { .as =3D NULL }) +#define MEMORY_REGION_CACHE_INVALID ((MemoryRegionCache) { .mrs.mr =3D NUL= L }) + =20 /* address_space_ld*_cached: load from a cached #MemoryRegion * address_space_st*_cached: store into a cached #MemoryRegion @@ -1719,11 +1723,40 @@ struct MemoryRegionCache { * if NULL, this information is discarded */ =20 -#define SUFFIX _cached +#define SUFFIX _cached_slow #define ARG1 cache #define ARG1_DECL MemoryRegionCache *cache #include "exec/memory_ldst.inc.h" =20 +/* Inline fast path for direct RAM access. */ +static inline uint8_t address_space_ldub_cached(MemoryRegionCache *cache, + hwaddr addr, MemTxAttrs attrs, MemTxResult *result) +{ + assert(addr < cache->len); + if (likely(cache->ptr)) { + return ldub_p(cache->ptr + addr); + } else { + return address_space_ldub_cached_slow(cache, addr, attrs, result); + } +} + +static inline void address_space_stb_cached(MemoryRegionCache *cache, + hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result) +{ + assert(addr < cache->len); + if (likely(cache->ptr)) { + stb_p(cache->ptr + addr, val); + } else { + address_space_stb_cached_slow(cache, addr, val, attrs, result); + } +} + +#define ENDIANNESS _le +#include "exec/memory_ldst_cached.inc.h" + +#define ENDIANNESS _be +#include "exec/memory_ldst_cached.inc.h" + #define SUFFIX _cached #define ARG1 cache #define ARG1_DECL MemoryRegionCache *cache @@ -1860,6 +1893,13 @@ MemTxResult flatview_read_continue(FlatView *fv, hwa= ddr addr, MemoryRegion *mr); void *qemu_map_ram_ptr(RAMBlock *ram_block, ram_addr_t addr); =20 +/* Internal functions, part of the implementation of address_space_read_ca= ched + * and address_space_write_cached. */ +void address_space_read_cached_slow(MemoryRegionCache *cache, + hwaddr addr, void *buf, int len); +void address_space_write_cached_slow(MemoryRegionCache *cache, + hwaddr addr, const void *buf, int len= ); + static inline bool memory_access_is_direct(MemoryRegion *mr, bool is_write) { if (is_write) { @@ -1928,7 +1968,11 @@ address_space_read_cached(MemoryRegionCache *cache, = hwaddr addr, void *buf, int len) { assert(addr < cache->len && len <=3D cache->len - addr); - address_space_read(cache->as, cache->xlat + addr, MEMTXATTRS_UNSPECIFI= ED, buf, len); + if (likely(cache->ptr)) { + memcpy(buf, cache->ptr + addr, len); + } else { + address_space_read_cached_slow(cache, addr, buf, len); + } } =20 /** @@ -1944,7 +1988,11 @@ address_space_write_cached(MemoryRegionCache *cache,= hwaddr addr, void *buf, int len) { assert(addr < cache->len && len <=3D cache->len - addr); - address_space_write(cache->as, cache->xlat + addr, MEMTXATTRS_UNSPECIF= IED, buf, len); + if (likely(cache->ptr)) { + memcpy(cache->ptr + addr, buf, len); + } else { + address_space_write_cached_slow(cache, addr, buf, len); + } } =20 #endif diff --git a/include/exec/memory_ldst_cached.inc.h b/include/exec/memory_ld= st_cached.inc.h new file mode 100644 index 0000000..fd4bbb4 --- /dev/null +++ b/include/exec/memory_ldst_cached.inc.h @@ -0,0 +1,108 @@ +/* + * Memory access templates for MemoryRegionCache + * + * Copyright (c) 2018 Red Hat, Inc. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, see . + */ + +#define ADDRESS_SPACE_LD_CACHED(size) \ + glue(glue(address_space_ld, size), glue(ENDIANNESS, _cached)) +#define ADDRESS_SPACE_LD_CACHED_SLOW(size) \ + glue(glue(address_space_ld, size), glue(ENDIANNESS, _cached_slow)) +#define LD_P(size) \ + glue(glue(ld, size), glue(ENDIANNESS, _p)) + +static inline uint32_t ADDRESS_SPACE_LD_CACHED(l)(MemoryRegionCache *cache, + hwaddr addr, MemTxAttrs attrs, MemTxResult *result) +{ + assert(addr < cache->len && 4 <=3D cache->len - addr); + if (likely(cache->ptr)) { + return LD_P(l)(cache->ptr + addr); + } else { + return ADDRESS_SPACE_LD_CACHED_SLOW(l)(cache, addr, attrs, result); + } +} + +static inline uint64_t ADDRESS_SPACE_LD_CACHED(q)(MemoryRegionCache *cache, + hwaddr addr, MemTxAttrs attrs, MemTxResult *result) +{ + assert(addr < cache->len && 8 <=3D cache->len - addr); + if (likely(cache->ptr)) { + return LD_P(q)(cache->ptr + addr); + } else { + return ADDRESS_SPACE_LD_CACHED_SLOW(q)(cache, addr, attrs, result); + } +} + +static inline uint32_t ADDRESS_SPACE_LD_CACHED(uw)(MemoryRegionCache *cach= e, + hwaddr addr, MemTxAttrs attrs, MemTxResult *result) +{ + assert(addr < cache->len && 2 <=3D cache->len - addr); + if (likely(cache->ptr)) { + return LD_P(uw)(cache->ptr + addr); + } else { + return ADDRESS_SPACE_LD_CACHED_SLOW(uw)(cache, addr, attrs, result= ); + } +} + +#undef ADDRESS_SPACE_LD_CACHED +#undef ADDRESS_SPACE_LD_CACHED_SLOW +#undef LD_P + +#define ADDRESS_SPACE_ST_CACHED(size) \ + glue(glue(address_space_st, size), glue(ENDIANNESS, _cached)) +#define ADDRESS_SPACE_ST_CACHED_SLOW(size) \ + glue(glue(address_space_st, size), glue(ENDIANNESS, _cached_slow)) +#define ST_P(size) \ + glue(glue(st, size), glue(ENDIANNESS, _p)) + +static inline void ADDRESS_SPACE_ST_CACHED(l)(MemoryRegionCache *cache, + hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result) +{ + assert(addr < cache->len && 4 <=3D cache->len - addr); + if (likely(cache->ptr)) { + ST_P(l)(cache->ptr + addr, val); + } else { + ADDRESS_SPACE_ST_CACHED_SLOW(l)(cache, addr, val, attrs, result); + } +} + +static inline void ADDRESS_SPACE_ST_CACHED(w)(MemoryRegionCache *cache, + hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result) +{ + assert(addr < cache->len && 2 <=3D cache->len - addr); + if (likely(cache->ptr)) { + ST_P(w)(cache->ptr + addr, val); + } else { + ADDRESS_SPACE_ST_CACHED_SLOW(w)(cache, addr, val, attrs, result); + } +} + +static inline void ADDRESS_SPACE_ST_CACHED(q)(MemoryRegionCache *cache, + hwaddr addr, uint64_t val, MemTxAttrs attrs, MemTxResult *result) +{ + assert(addr < cache->len && 8 <=3D cache->len - addr); + if (likely(cache->ptr)) { + ST_P(q)(cache->ptr + addr, val); + } else { + ADDRESS_SPACE_ST_CACHED_SLOW(q)(cache, addr, val, attrs, result); + } +} + +#undef ADDRESS_SPACE_ST_CACHED +#undef ADDRESS_SPACE_ST_CACHED_SLOW +#undef ST_P + +#undef ENDIANNESS diff --git a/memory.c b/memory.c index e70b64b..fc7f9b7 100644 --- a/memory.c +++ b/memory.c @@ -298,7 +298,7 @@ static bool flatview_ref(FlatView *view) return atomic_fetch_inc_nonzero(&view->ref) > 0; } =20 -static void flatview_unref(FlatView *view) +void flatview_unref(FlatView *view) { if (atomic_fetch_dec(&view->ref) =3D=3D 1) { trace_flatview_destroy_rcu(view, view->root); @@ -822,7 +822,7 @@ static void address_space_add_del_ioeventfds(AddressSpa= ce *as, } } =20 -static FlatView *address_space_get_flatview(AddressSpace *as) +FlatView *address_space_get_flatview(AddressSpace *as) { FlatView *view; =20 --=20 1.8.3.1