From nobody Fri May 3 15:18:57 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 152397434406784.81645497999205; Tue, 17 Apr 2018 07:12:24 -0700 (PDT) Received: from localhost ([::1]:41078 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f8RLH-0003KO-AY for importer@patchew.org; Tue, 17 Apr 2018 10:12:19 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36677) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f8RHJ-0000NX-Vu for qemu-devel@nongnu.org; Tue, 17 Apr 2018 10:08:17 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f8RHB-0003Fh-SX for qemu-devel@nongnu.org; Tue, 17 Apr 2018 10:08:13 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:39202 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1f8RHB-0003FM-Ke for qemu-devel@nongnu.org; Tue, 17 Apr 2018 10:08:05 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 474284070208 for ; Tue, 17 Apr 2018 14:08:05 +0000 (UTC) Received: from donizetti.redhat.com (ovpn-116-72.ams2.redhat.com [10.36.116.72]) by smtp.corp.redhat.com (Postfix) with ESMTP id 504C62017DD3 for ; Tue, 17 Apr 2018 14:08:04 +0000 (UTC) From: Paolo Bonzini To: qemu-devel@nongnu.org Date: Tue, 17 Apr 2018 16:07:59 +0200 Message-Id: <20180417140802.16711-2-pbonzini@redhat.com> In-Reply-To: <20180417140802.16711-1-pbonzini@redhat.com> References: <20180417140802.16711-1-pbonzini@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Tue, 17 Apr 2018 14:08:05 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Tue, 17 Apr 2018 14:08:05 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'pbonzini@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 1/4] exec: move memory access declarations to a common header, inline *_phys functions X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" For now, this reduces the text size very slightly due to the newly-added inlining: text size before: 9301965 text size after: 9300645 Later, however, the declarations in include/exec/memory_ldst.inc.h will be reused for the MemoryRegionCache slow path functions. Signed-off-by: Paolo Bonzini --- include/exec/cpu-all.h | 75 ++++++-------- include/exec/memory.h | 153 ++++++++-------------------- include/exec/memory_ldst.inc.h | 71 +++++++++++++ include/exec/memory_ldst_phys.inc.h | 147 ++++++++++++++++++++++++++ memory_ldst.inc.c | 126 ----------------------- 5 files changed, 292 insertions(+), 280 deletions(-) create mode 100644 include/exec/memory_ldst.inc.h create mode 100644 include/exec/memory_ldst_phys.inc.h diff --git a/include/exec/cpu-all.h b/include/exec/cpu-all.h index f4fa94e966..173edd1fb4 100644 --- a/include/exec/cpu-all.h +++ b/include/exec/cpu-all.h @@ -168,51 +168,36 @@ extern unsigned long reserved_va; #else =20 #include "exec/hwaddr.h" -uint32_t lduw_phys(AddressSpace *as, hwaddr addr); -uint32_t ldl_phys(AddressSpace *as, hwaddr addr); -uint64_t ldq_phys(AddressSpace *as, hwaddr addr); -void stl_phys_notdirty(AddressSpace *as, hwaddr addr, uint32_t val); -void stw_phys(AddressSpace *as, hwaddr addr, uint32_t val); -void stl_phys(AddressSpace *as, hwaddr addr, uint32_t val); -void stq_phys(AddressSpace *as, hwaddr addr, uint64_t val); - -uint32_t address_space_lduw(AddressSpace *as, hwaddr addr, - MemTxAttrs attrs, MemTxResult *result); -uint32_t address_space_ldl(AddressSpace *as, hwaddr addr, - MemTxAttrs attrs, MemTxResult *result); -uint64_t address_space_ldq(AddressSpace *as, hwaddr addr, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stl_notdirty(AddressSpace *as, hwaddr addr, uint32_t va= l, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stw(AddressSpace *as, hwaddr addr, uint32_t val, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stl(AddressSpace *as, hwaddr addr, uint32_t val, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stq(AddressSpace *as, hwaddr addr, uint64_t val, - MemTxAttrs attrs, MemTxResult *result); - -uint32_t lduw_phys_cached(MemoryRegionCache *cache, hwaddr addr); -uint32_t ldl_phys_cached(MemoryRegionCache *cache, hwaddr addr); -uint64_t ldq_phys_cached(MemoryRegionCache *cache, hwaddr addr); -void stl_phys_notdirty_cached(MemoryRegionCache *cache, hwaddr addr, uint3= 2_t val); -void stw_phys_cached(MemoryRegionCache *cache, hwaddr addr, uint32_t val); -void stl_phys_cached(MemoryRegionCache *cache, hwaddr addr, uint32_t val); -void stq_phys_cached(MemoryRegionCache *cache, hwaddr addr, uint64_t val); - -uint32_t address_space_lduw_cached(MemoryRegionCache *cache, hwaddr addr, - MemTxAttrs attrs, MemTxResult *result); -uint32_t address_space_ldl_cached(MemoryRegionCache *cache, hwaddr addr, - MemTxAttrs attrs, MemTxResult *result); -uint64_t address_space_ldq_cached(MemoryRegionCache *cache, hwaddr addr, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stl_notdirty_cached(MemoryRegionCache *cache, hwaddr ad= dr, - uint32_t val, MemTxAttrs attrs, MemTxResult *r= esult); -void address_space_stw_cached(MemoryRegionCache *cache, hwaddr addr, uint3= 2_t val, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stl_cached(MemoryRegionCache *cache, hwaddr addr, uint3= 2_t val, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stq_cached(MemoryRegionCache *cache, hwaddr addr, uint6= 4_t val, - MemTxAttrs attrs, MemTxResult *result); + +#define SUFFIX +#define ARG1 as +#define ARG1_DECL AddressSpace *as +#define TARGET_ENDIANNESS +#include "exec/memory_ldst.inc.h" + +#define SUFFIX _cached +#define ARG1 cache +#define ARG1_DECL MemoryRegionCache *cache +#define TARGET_ENDIANNESS +#include "exec/memory_ldst.inc.h" + +static inline void stl_phys_notdirty(AddressSpace *as, hwaddr addr, uint32= _t val) +{ + address_space_stl_notdirty(as, addr, val, + MEMTXATTRS_UNSPECIFIED, NULL); +} + +#define SUFFIX +#define ARG1 as +#define ARG1_DECL AddressSpace *as +#define TARGET_ENDIANNESS +#include "exec/memory_ldst_phys.inc.h" + +#define SUFFIX _cached +#define ARG1 cache +#define ARG1_DECL MemoryRegionCache *cache +#define TARGET_ENDIANNESS +#include "exec/memory_ldst_phys.inc.h" #endif =20 /* page related stuff */ diff --git a/include/exec/memory.h b/include/exec/memory.h index 31eae0a640..ca361bc409 100644 --- a/include/exec/memory.h +++ b/include/exec/memory.h @@ -1676,49 +1676,16 @@ MemTxResult address_space_write(AddressSpace *as, h= waddr addr, * @result: location to write the success/failure of the transaction; * if NULL, this information is discarded */ -uint32_t address_space_ldub(AddressSpace *as, hwaddr addr, - MemTxAttrs attrs, MemTxResult *result); -uint32_t address_space_lduw_le(AddressSpace *as, hwaddr addr, - MemTxAttrs attrs, MemTxResult *result); -uint32_t address_space_lduw_be(AddressSpace *as, hwaddr addr, - MemTxAttrs attrs, MemTxResult *result); -uint32_t address_space_ldl_le(AddressSpace *as, hwaddr addr, - MemTxAttrs attrs, MemTxResult *result); -uint32_t address_space_ldl_be(AddressSpace *as, hwaddr addr, - MemTxAttrs attrs, MemTxResult *result); -uint64_t address_space_ldq_le(AddressSpace *as, hwaddr addr, - MemTxAttrs attrs, MemTxResult *result); -uint64_t address_space_ldq_be(AddressSpace *as, hwaddr addr, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stb(AddressSpace *as, hwaddr addr, uint32_t val, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stw_le(AddressSpace *as, hwaddr addr, uint32_t val, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stw_be(AddressSpace *as, hwaddr addr, uint32_t val, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stl_le(AddressSpace *as, hwaddr addr, uint32_t val, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stl_be(AddressSpace *as, hwaddr addr, uint32_t val, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stq_le(AddressSpace *as, hwaddr addr, uint64_t val, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stq_be(AddressSpace *as, hwaddr addr, uint64_t val, - MemTxAttrs attrs, MemTxResult *result); - -uint32_t ldub_phys(AddressSpace *as, hwaddr addr); -uint32_t lduw_le_phys(AddressSpace *as, hwaddr addr); -uint32_t lduw_be_phys(AddressSpace *as, hwaddr addr); -uint32_t ldl_le_phys(AddressSpace *as, hwaddr addr); -uint32_t ldl_be_phys(AddressSpace *as, hwaddr addr); -uint64_t ldq_le_phys(AddressSpace *as, hwaddr addr); -uint64_t ldq_be_phys(AddressSpace *as, hwaddr addr); -void stb_phys(AddressSpace *as, hwaddr addr, uint32_t val); -void stw_le_phys(AddressSpace *as, hwaddr addr, uint32_t val); -void stw_be_phys(AddressSpace *as, hwaddr addr, uint32_t val); -void stl_le_phys(AddressSpace *as, hwaddr addr, uint32_t val); -void stl_be_phys(AddressSpace *as, hwaddr addr, uint32_t val); -void stq_le_phys(AddressSpace *as, hwaddr addr, uint64_t val); -void stq_be_phys(AddressSpace *as, hwaddr addr, uint64_t val); + +#define SUFFIX +#define ARG1 as +#define ARG1_DECL AddressSpace *as +#include "exec/memory_ldst.inc.h" + +#define SUFFIX +#define ARG1 as +#define ARG1_DECL AddressSpace *as +#include "exec/memory_ldst_phys.inc.h" =20 struct MemoryRegionCache { hwaddr xlat; @@ -1728,6 +1695,40 @@ struct MemoryRegionCache { =20 #define MEMORY_REGION_CACHE_INVALID ((MemoryRegionCache) { .as =3D NULL }) =20 +/* address_space_ld*_cached: load from a cached #MemoryRegion + * address_space_st*_cached: store into a cached #MemoryRegion + * + * These functions perform a load or store of the byte, word, + * longword or quad to the specified address. The address is + * a physical address in the AddressSpace, but it must lie within + * a #MemoryRegion that was mapped with address_space_cache_init. + * + * The _le suffixed functions treat the data as little endian; + * _be indicates big endian; no suffix indicates "same endianness + * as guest CPU". + * + * The "guest CPU endianness" accessors are deprecated for use outside + * target-* code; devices should be CPU-agnostic and use either the LE + * or the BE accessors. + * + * @cache: previously initialized #MemoryRegionCache to be accessed + * @addr: address within the address space + * @val: data value, for stores + * @attrs: memory transaction attributes + * @result: location to write the success/failure of the transaction; + * if NULL, this information is discarded + */ + +#define SUFFIX _cached +#define ARG1 cache +#define ARG1_DECL MemoryRegionCache *cache +#include "exec/memory_ldst.inc.h" + +#define SUFFIX _cached +#define ARG1 cache +#define ARG1_DECL MemoryRegionCache *cache +#include "exec/memory_ldst_phys.inc.h" + /* address_space_cache_init: prepare for repeated access to a physical * memory region * @@ -1772,72 +1773,6 @@ void address_space_cache_invalidate(MemoryRegionCach= e *cache, */ void address_space_cache_destroy(MemoryRegionCache *cache); =20 -/* address_space_ld*_cached: load from a cached #MemoryRegion - * address_space_st*_cached: store into a cached #MemoryRegion - * - * These functions perform a load or store of the byte, word, - * longword or quad to the specified address. The address is - * a physical address in the AddressSpace, but it must lie within - * a #MemoryRegion that was mapped with address_space_cache_init. - * - * The _le suffixed functions treat the data as little endian; - * _be indicates big endian; no suffix indicates "same endianness - * as guest CPU". - * - * The "guest CPU endianness" accessors are deprecated for use outside - * target-* code; devices should be CPU-agnostic and use either the LE - * or the BE accessors. - * - * @cache: previously initialized #MemoryRegionCache to be accessed - * @addr: address within the address space - * @val: data value, for stores - * @attrs: memory transaction attributes - * @result: location to write the success/failure of the transaction; - * if NULL, this information is discarded - */ -uint32_t address_space_ldub_cached(MemoryRegionCache *cache, hwaddr addr, - MemTxAttrs attrs, MemTxResult *result); -uint32_t address_space_lduw_le_cached(MemoryRegionCache *cache, hwaddr add= r, - MemTxAttrs attrs, MemTxResult *result); -uint32_t address_space_lduw_be_cached(MemoryRegionCache *cache, hwaddr add= r, - MemTxAttrs attrs, MemTxResult *result); -uint32_t address_space_ldl_le_cached(MemoryRegionCache *cache, hwaddr addr, - MemTxAttrs attrs, MemTxResult *result); -uint32_t address_space_ldl_be_cached(MemoryRegionCache *cache, hwaddr addr, - MemTxAttrs attrs, MemTxResult *result); -uint64_t address_space_ldq_le_cached(MemoryRegionCache *cache, hwaddr addr, - MemTxAttrs attrs, MemTxResult *result); -uint64_t address_space_ldq_be_cached(MemoryRegionCache *cache, hwaddr addr, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stb_cached(MemoryRegionCache *cache, hwaddr addr, uint3= 2_t val, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stw_le_cached(MemoryRegionCache *cache, hwaddr addr, ui= nt32_t val, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stw_be_cached(MemoryRegionCache *cache, hwaddr addr, ui= nt32_t val, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stl_le_cached(MemoryRegionCache *cache, hwaddr addr, ui= nt32_t val, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stl_be_cached(MemoryRegionCache *cache, hwaddr addr, ui= nt32_t val, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stq_le_cached(MemoryRegionCache *cache, hwaddr addr, ui= nt64_t val, - MemTxAttrs attrs, MemTxResult *result); -void address_space_stq_be_cached(MemoryRegionCache *cache, hwaddr addr, ui= nt64_t val, - MemTxAttrs attrs, MemTxResult *result); - -uint32_t ldub_phys_cached(MemoryRegionCache *cache, hwaddr addr); -uint32_t lduw_le_phys_cached(MemoryRegionCache *cache, hwaddr addr); -uint32_t lduw_be_phys_cached(MemoryRegionCache *cache, hwaddr addr); -uint32_t ldl_le_phys_cached(MemoryRegionCache *cache, hwaddr addr); -uint32_t ldl_be_phys_cached(MemoryRegionCache *cache, hwaddr addr); -uint64_t ldq_le_phys_cached(MemoryRegionCache *cache, hwaddr addr); -uint64_t ldq_be_phys_cached(MemoryRegionCache *cache, hwaddr addr); -void stb_phys_cached(MemoryRegionCache *cache, hwaddr addr, uint32_t val); -void stw_le_phys_cached(MemoryRegionCache *cache, hwaddr addr, uint32_t va= l); -void stw_be_phys_cached(MemoryRegionCache *cache, hwaddr addr, uint32_t va= l); -void stl_le_phys_cached(MemoryRegionCache *cache, hwaddr addr, uint32_t va= l); -void stl_be_phys_cached(MemoryRegionCache *cache, hwaddr addr, uint32_t va= l); -void stq_le_phys_cached(MemoryRegionCache *cache, hwaddr addr, uint64_t va= l); -void stq_be_phys_cached(MemoryRegionCache *cache, hwaddr addr, uint64_t va= l); /* address_space_get_iotlb_entry: translate an address into an IOTLB * entry. Should be called from an RCU critical section. */ diff --git a/include/exec/memory_ldst.inc.h b/include/exec/memory_ldst.inc.h new file mode 100644 index 0000000000..272c20f02e --- /dev/null +++ b/include/exec/memory_ldst.inc.h @@ -0,0 +1,71 @@ +/* + * Physical memory access templates + * + * Copyright (c) 2003 Fabrice Bellard + * Copyright (c) 2015 Linaro, Inc. + * Copyright (c) 2018 Red Hat, Inc. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, see . + */ + +#ifdef TARGET_ENDIANNESS +extern uint32_t glue(address_space_lduw, SUFFIX)(ARG1_DECL, + hwaddr addr, MemTxAttrs attrs, MemTxResult *result); +extern uint32_t glue(address_space_ldl, SUFFIX)(ARG1_DECL, + hwaddr addr, MemTxAttrs attrs, MemTxResult *result); +extern uint64_t glue(address_space_ldq, SUFFIX)(ARG1_DECL, + hwaddr addr, MemTxAttrs attrs, MemTxResult *result); +extern void glue(address_space_stl_notdirty, SUFFIX)(ARG1_DECL, + hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result); +extern void glue(address_space_stw, SUFFIX)(ARG1_DECL, + hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result); +extern void glue(address_space_stl, SUFFIX)(ARG1_DECL, + hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result); +extern void glue(address_space_stq, SUFFIX)(ARG1_DECL, + hwaddr addr, uint64_t val, MemTxAttrs attrs, MemTxResult *result); +#else +extern uint32_t glue(address_space_ldub, SUFFIX)(ARG1_DECL, + hwaddr addr, MemTxAttrs attrs, MemTxResult *result); +extern uint32_t glue(address_space_lduw_le, SUFFIX)(ARG1_DECL, + hwaddr addr, MemTxAttrs attrs, MemTxResult *result); +extern uint32_t glue(address_space_lduw_be, SUFFIX)(ARG1_DECL, + hwaddr addr, MemTxAttrs attrs, MemTxResult *result); +extern uint32_t glue(address_space_ldl_le, SUFFIX)(ARG1_DECL, + hwaddr addr, MemTxAttrs attrs, MemTxResult *result); +extern uint32_t glue(address_space_ldl_be, SUFFIX)(ARG1_DECL, + hwaddr addr, MemTxAttrs attrs, MemTxResult *result); +extern uint64_t glue(address_space_ldq_le, SUFFIX)(ARG1_DECL, + hwaddr addr, MemTxAttrs attrs, MemTxResult *result); +extern uint64_t glue(address_space_ldq_be, SUFFIX)(ARG1_DECL, + hwaddr addr, MemTxAttrs attrs, MemTxResult *result); +extern void glue(address_space_stb, SUFFIX)(ARG1_DECL, + hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result); +extern void glue(address_space_stw_le, SUFFIX)(ARG1_DECL, + hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result); +extern void glue(address_space_stw_be, SUFFIX)(ARG1_DECL, + hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result); +extern void glue(address_space_stl_le, SUFFIX)(ARG1_DECL, + hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result); +extern void glue(address_space_stl_be, SUFFIX)(ARG1_DECL, + hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result); +extern void glue(address_space_stq_le, SUFFIX)(ARG1_DECL, + hwaddr addr, uint64_t val, MemTxAttrs attrs, MemTxResult *result); +extern void glue(address_space_stq_be, SUFFIX)(ARG1_DECL, + hwaddr addr, uint64_t val, MemTxAttrs attrs, MemTxResult *result); +#endif + +#undef ARG1_DECL +#undef ARG1 +#undef SUFFIX +#undef TARGET_ENDIANNESS diff --git a/include/exec/memory_ldst_phys.inc.h b/include/exec/memory_ldst= _phys.inc.h new file mode 100644 index 0000000000..91f72973cb --- /dev/null +++ b/include/exec/memory_ldst_phys.inc.h @@ -0,0 +1,147 @@ +/* + * Physical memory access templates + * + * Copyright (c) 2003 Fabrice Bellard + * Copyright (c) 2015 Linaro, Inc. + * Copyright (c) 2018 Red Hat, Inc. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, see . + */ + +#ifdef TARGET_ENDIANNESS +static inline uint32_t glue(ldl_phys, SUFFIX)(ARG1_DECL, hwaddr addr) +{ + return glue(address_space_ldl, SUFFIX)(ARG1, addr, + MEMTXATTRS_UNSPECIFIED, NULL); +} + +static inline uint64_t glue(ldq_phys, SUFFIX)(ARG1_DECL, hwaddr addr) +{ + return glue(address_space_ldq, SUFFIX)(ARG1, addr, + MEMTXATTRS_UNSPECIFIED, NULL); +} + +static inline uint32_t glue(lduw_phys, SUFFIX)(ARG1_DECL, hwaddr addr) +{ + return glue(address_space_lduw, SUFFIX)(ARG1, addr, + MEMTXATTRS_UNSPECIFIED, NULL); +} + +static inline void glue(stl_phys, SUFFIX)(ARG1_DECL, hwaddr addr, uint32_t= val) +{ + glue(address_space_stl, SUFFIX)(ARG1, addr, val, + MEMTXATTRS_UNSPECIFIED, NULL); +} + +static inline void glue(stw_phys, SUFFIX)(ARG1_DECL, hwaddr addr, uint32_t= val) +{ + glue(address_space_stw, SUFFIX)(ARG1, addr, val, + MEMTXATTRS_UNSPECIFIED, NULL); +} + +static inline void glue(stq_phys, SUFFIX)(ARG1_DECL, hwaddr addr, uint64_t= val) +{ + glue(address_space_stq, SUFFIX)(ARG1, addr, val, + MEMTXATTRS_UNSPECIFIED, NULL); +} +#else +static inline uint32_t glue(ldl_le_phys, SUFFIX)(ARG1_DECL, hwaddr addr) +{ + return glue(address_space_ldl_le, SUFFIX)(ARG1, addr, + MEMTXATTRS_UNSPECIFIED, NULL= ); +} + +static inline uint32_t glue(ldl_be_phys, SUFFIX)(ARG1_DECL, hwaddr addr) +{ + return glue(address_space_ldl_be, SUFFIX)(ARG1, addr, + MEMTXATTRS_UNSPECIFIED, NULL= ); +} + +static inline uint64_t glue(ldq_le_phys, SUFFIX)(ARG1_DECL, hwaddr addr) +{ + return glue(address_space_ldq_le, SUFFIX)(ARG1, addr, + MEMTXATTRS_UNSPECIFIED, NULL= ); +} + +static inline uint64_t glue(ldq_be_phys, SUFFIX)(ARG1_DECL, hwaddr addr) +{ + return glue(address_space_ldq_be, SUFFIX)(ARG1, addr, + MEMTXATTRS_UNSPECIFIED, NULL= ); +} + +static inline uint32_t glue(ldub_phys, SUFFIX)(ARG1_DECL, hwaddr addr) +{ + return glue(address_space_ldub, SUFFIX)(ARG1, addr, + MEMTXATTRS_UNSPECIFIED, NULL); +} + +static inline uint32_t glue(lduw_le_phys, SUFFIX)(ARG1_DECL, hwaddr addr) +{ + return glue(address_space_lduw_le, SUFFIX)(ARG1, addr, + MEMTXATTRS_UNSPECIFIED, NUL= L); +} + +static inline uint32_t glue(lduw_be_phys, SUFFIX)(ARG1_DECL, hwaddr addr) +{ + return glue(address_space_lduw_be, SUFFIX)(ARG1, addr, + MEMTXATTRS_UNSPECIFIED, NUL= L); +} + +static inline void glue(stl_le_phys, SUFFIX)(ARG1_DECL, hwaddr addr, uint3= 2_t val) +{ + glue(address_space_stl_le, SUFFIX)(ARG1, addr, val, + MEMTXATTRS_UNSPECIFIED, NULL); +} + +static inline void glue(stl_be_phys, SUFFIX)(ARG1_DECL, hwaddr addr, uint3= 2_t val) +{ + glue(address_space_stl_be, SUFFIX)(ARG1, addr, val, + MEMTXATTRS_UNSPECIFIED, NULL); +} + +static inline void glue(stb_phys, SUFFIX)(ARG1_DECL, hwaddr addr, uint32_t= val) +{ + glue(address_space_stb, SUFFIX)(ARG1, addr, val, + MEMTXATTRS_UNSPECIFIED, NULL); +} + +static inline void glue(stw_le_phys, SUFFIX)(ARG1_DECL, hwaddr addr, uint3= 2_t val) +{ + glue(address_space_stw_le, SUFFIX)(ARG1, addr, val, + MEMTXATTRS_UNSPECIFIED, NULL); +} + +static inline void glue(stw_be_phys, SUFFIX)(ARG1_DECL, hwaddr addr, uint3= 2_t val) +{ + glue(address_space_stw_be, SUFFIX)(ARG1, addr, val, + MEMTXATTRS_UNSPECIFIED, NULL); +} + +static inline void glue(stq_le_phys, SUFFIX)(ARG1_DECL, hwaddr addr, uint6= 4_t val) +{ + glue(address_space_stq_le, SUFFIX)(ARG1, addr, val, + MEMTXATTRS_UNSPECIFIED, NULL); +} + +static inline void glue(stq_be_phys, SUFFIX)(ARG1_DECL, hwaddr addr, uint6= 4_t val) +{ + glue(address_space_stq_be, SUFFIX)(ARG1, addr, val, + MEMTXATTRS_UNSPECIFIED, NULL); +} +#endif + +#undef ARG1_DECL +#undef ARG1 +#undef SUFFIX +#undef TARGET_ENDIANNESS diff --git a/memory_ldst.inc.c b/memory_ldst.inc.c index 5dbff9cef8..25d6125747 100644 --- a/memory_ldst.inc.c +++ b/memory_ldst.inc.c @@ -95,24 +95,6 @@ uint32_t glue(address_space_ldl_be, SUFFIX)(ARG1_DECL, DEVICE_BIG_ENDIAN); } =20 -uint32_t glue(ldl_phys, SUFFIX)(ARG1_DECL, hwaddr addr) -{ - return glue(address_space_ldl, SUFFIX)(ARG1, addr, - MEMTXATTRS_UNSPECIFIED, NULL); -} - -uint32_t glue(ldl_le_phys, SUFFIX)(ARG1_DECL, hwaddr addr) -{ - return glue(address_space_ldl_le, SUFFIX)(ARG1, addr, - MEMTXATTRS_UNSPECIFIED, NULL= ); -} - -uint32_t glue(ldl_be_phys, SUFFIX)(ARG1_DECL, hwaddr addr) -{ - return glue(address_space_ldl_be, SUFFIX)(ARG1, addr, - MEMTXATTRS_UNSPECIFIED, NULL= ); -} - /* warning: addr must be aligned */ static inline uint64_t glue(address_space_ldq_internal, SUFFIX)(ARG1_DECL, hwaddr addr, MemTxAttrs attrs, MemTxResult *result, @@ -189,24 +171,6 @@ uint64_t glue(address_space_ldq_be, SUFFIX)(ARG1_DECL, DEVICE_BIG_ENDIAN); } =20 -uint64_t glue(ldq_phys, SUFFIX)(ARG1_DECL, hwaddr addr) -{ - return glue(address_space_ldq, SUFFIX)(ARG1, addr, - MEMTXATTRS_UNSPECIFIED, NULL); -} - -uint64_t glue(ldq_le_phys, SUFFIX)(ARG1_DECL, hwaddr addr) -{ - return glue(address_space_ldq_le, SUFFIX)(ARG1, addr, - MEMTXATTRS_UNSPECIFIED, NULL= ); -} - -uint64_t glue(ldq_be_phys, SUFFIX)(ARG1_DECL, hwaddr addr) -{ - return glue(address_space_ldq_be, SUFFIX)(ARG1, addr, - MEMTXATTRS_UNSPECIFIED, NULL= ); -} - uint32_t glue(address_space_ldub, SUFFIX)(ARG1_DECL, hwaddr addr, MemTxAttrs attrs, MemTxResult *result) { @@ -241,12 +205,6 @@ uint32_t glue(address_space_ldub, SUFFIX)(ARG1_DECL, return val; } =20 -uint32_t glue(ldub_phys, SUFFIX)(ARG1_DECL, hwaddr addr) -{ - return glue(address_space_ldub, SUFFIX)(ARG1, addr, - MEMTXATTRS_UNSPECIFIED, NULL); -} - /* warning: addr must be aligned */ static inline uint32_t glue(address_space_lduw_internal, SUFFIX)(ARG1_DECL, hwaddr addr, MemTxAttrs attrs, MemTxResult *result, @@ -323,24 +281,6 @@ uint32_t glue(address_space_lduw_be, SUFFIX)(ARG1_DECL, DEVICE_BIG_ENDIAN); } =20 -uint32_t glue(lduw_phys, SUFFIX)(ARG1_DECL, hwaddr addr) -{ - return glue(address_space_lduw, SUFFIX)(ARG1, addr, - MEMTXATTRS_UNSPECIFIED, NULL); -} - -uint32_t glue(lduw_le_phys, SUFFIX)(ARG1_DECL, hwaddr addr) -{ - return glue(address_space_lduw_le, SUFFIX)(ARG1, addr, - MEMTXATTRS_UNSPECIFIED, NUL= L); -} - -uint32_t glue(lduw_be_phys, SUFFIX)(ARG1_DECL, hwaddr addr) -{ - return glue(address_space_lduw_be, SUFFIX)(ARG1, addr, - MEMTXATTRS_UNSPECIFIED, NUL= L); -} - /* warning: addr must be aligned. The ram page is not masked as dirty and the code inside is not invalidated. It is useful if the dirty bits are used to track modified PTEs */ @@ -380,12 +320,6 @@ void glue(address_space_stl_notdirty, SUFFIX)(ARG1_DEC= L, RCU_READ_UNLOCK(); } =20 -void glue(stl_phys_notdirty, SUFFIX)(ARG1_DECL, hwaddr addr, uint32_t val) -{ - glue(address_space_stl_notdirty, SUFFIX)(ARG1, addr, val, - MEMTXATTRS_UNSPECIFIED, NULL); -} - /* warning: addr must be aligned */ static inline void glue(address_space_stl_internal, SUFFIX)(ARG1_DECL, hwaddr addr, uint32_t val, MemTxAttrs attrs, @@ -460,24 +394,6 @@ void glue(address_space_stl_be, SUFFIX)(ARG1_DECL, result, DEVICE_BIG_ENDIAN); } =20 -void glue(stl_phys, SUFFIX)(ARG1_DECL, hwaddr addr, uint32_t val) -{ - glue(address_space_stl, SUFFIX)(ARG1, addr, val, - MEMTXATTRS_UNSPECIFIED, NULL); -} - -void glue(stl_le_phys, SUFFIX)(ARG1_DECL, hwaddr addr, uint32_t val) -{ - glue(address_space_stl_le, SUFFIX)(ARG1, addr, val, - MEMTXATTRS_UNSPECIFIED, NULL); -} - -void glue(stl_be_phys, SUFFIX)(ARG1_DECL, hwaddr addr, uint32_t val) -{ - glue(address_space_stl_be, SUFFIX)(ARG1, addr, val, - MEMTXATTRS_UNSPECIFIED, NULL); -} - void glue(address_space_stb, SUFFIX)(ARG1_DECL, hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result) { @@ -509,12 +425,6 @@ void glue(address_space_stb, SUFFIX)(ARG1_DECL, RCU_READ_UNLOCK(); } =20 -void glue(stb_phys, SUFFIX)(ARG1_DECL, hwaddr addr, uint32_t val) -{ - glue(address_space_stb, SUFFIX)(ARG1, addr, val, - MEMTXATTRS_UNSPECIFIED, NULL); -} - /* warning: addr must be aligned */ static inline void glue(address_space_stw_internal, SUFFIX)(ARG1_DECL, hwaddr addr, uint32_t val, MemTxAttrs attrs, @@ -589,24 +499,6 @@ void glue(address_space_stw_be, SUFFIX)(ARG1_DECL, DEVICE_BIG_ENDIAN); } =20 -void glue(stw_phys, SUFFIX)(ARG1_DECL, hwaddr addr, uint32_t val) -{ - glue(address_space_stw, SUFFIX)(ARG1, addr, val, - MEMTXATTRS_UNSPECIFIED, NULL); -} - -void glue(stw_le_phys, SUFFIX)(ARG1_DECL, hwaddr addr, uint32_t val) -{ - glue(address_space_stw_le, SUFFIX)(ARG1, addr, val, - MEMTXATTRS_UNSPECIFIED, NULL); -} - -void glue(stw_be_phys, SUFFIX)(ARG1_DECL, hwaddr addr, uint32_t val) -{ - glue(address_space_stw_be, SUFFIX)(ARG1, addr, val, - MEMTXATTRS_UNSPECIFIED, NULL); -} - static void glue(address_space_stq_internal, SUFFIX)(ARG1_DECL, hwaddr addr, uint64_t val, MemTxAttrs attrs, MemTxResult *result, enum device_endian endian) @@ -680,24 +572,6 @@ void glue(address_space_stq_be, SUFFIX)(ARG1_DECL, DEVICE_BIG_ENDIAN); } =20 -void glue(stq_phys, SUFFIX)(ARG1_DECL, hwaddr addr, uint64_t val) -{ - glue(address_space_stq, SUFFIX)(ARG1, addr, val, - MEMTXATTRS_UNSPECIFIED, NULL); -} - -void glue(stq_le_phys, SUFFIX)(ARG1_DECL, hwaddr addr, uint64_t val) -{ - glue(address_space_stq_le, SUFFIX)(ARG1, addr, val, - MEMTXATTRS_UNSPECIFIED, NULL); -} - -void glue(stq_be_phys, SUFFIX)(ARG1_DECL, hwaddr addr, uint64_t val) -{ - glue(address_space_stq_be, SUFFIX)(ARG1, addr, val, - MEMTXATTRS_UNSPECIFIED, NULL); -} - #undef ARG1_DECL #undef ARG1 #undef SUFFIX --=20 2.17.0 From nobody Fri May 3 15:18:57 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1523974278606949.6993019064889; Tue, 17 Apr 2018 07:11:18 -0700 (PDT) Received: from localhost ([::1]:41052 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f8RKH-0002fK-Ez for importer@patchew.org; Tue, 17 Apr 2018 10:11:17 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36628) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f8RHE-0000IO-8y for qemu-devel@nongnu.org; Tue, 17 Apr 2018 10:08:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f8RHC-0003G9-Gm for qemu-devel@nongnu.org; Tue, 17 Apr 2018 10:08:07 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:35548 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1f8RHC-0003Fn-AZ for qemu-devel@nongnu.org; Tue, 17 Apr 2018 10:08:06 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 027EB406C510 for ; Tue, 17 Apr 2018 14:08:06 +0000 (UTC) Received: from donizetti.redhat.com (ovpn-116-72.ams2.redhat.com [10.36.116.72]) by smtp.corp.redhat.com (Postfix) with ESMTP id 887B0200BCB0 for ; Tue, 17 Apr 2018 14:08:05 +0000 (UTC) From: Paolo Bonzini To: qemu-devel@nongnu.org Date: Tue, 17 Apr 2018 16:08:00 +0200 Message-Id: <20180417140802.16711-3-pbonzini@redhat.com> In-Reply-To: <20180417140802.16711-1-pbonzini@redhat.com> References: <20180417140802.16711-1-pbonzini@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Tue, 17 Apr 2018 14:08:06 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Tue, 17 Apr 2018 14:08:06 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'pbonzini@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 2/4] exec: small changes to flatview_do_translate X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Prepare for extracting the IOMMU part to a separate function. Mostly cosmetic; the only semantic change is that, if there is more than one cascaded IOMMU and the second one fails to translate, *plen_out is now adjusted according to the page mask of the first IOMMU. Signed-off-by: Paolo Bonzini Reviewed-by: Peter Xu --- exec.c | 33 ++++++++++++++------------------- 1 file changed, 14 insertions(+), 19 deletions(-) diff --git a/exec.c b/exec.c index 02b1efebb7..1dea3e9c95 100644 --- a/exec.c +++ b/exec.c @@ -476,6 +476,7 @@ address_space_translate_internal(AddressSpaceDispatch *= d, hwaddr addr, hwaddr *x * would tell. It can be @NULL if we don't care about it. * @is_write: whether the translation operation is for write * @is_mmio: whether this can be MMIO, set true if it can + * @target_as: the address space targeted by the IOMMU * * This function is called from RCU critical section */ @@ -495,14 +496,14 @@ static MemoryRegionSection flatview_do_translate(Flat= View *fv, hwaddr page_mask =3D (hwaddr)(-1); hwaddr plen =3D (hwaddr)(-1); =20 - if (plen_out) { - plen =3D *plen_out; + if (!plen_out) { + plen_out =3D &plen; } =20 for (;;) { section =3D address_space_translate_internal( - flatview_to_dispatch(fv), addr, &addr, - &plen, is_mmio); + flatview_to_dispatch(fv), addr, xlat, + plen_out, is_mmio); =20 iommu_mr =3D memory_region_get_iommu(section->mr); if (!iommu_mr) { @@ -510,35 +511,29 @@ static MemoryRegionSection flatview_do_translate(Flat= View *fv, } imrc =3D memory_region_get_iommu_class_nocheck(iommu_mr); =20 + addr =3D *xlat; iotlb =3D imrc->translate(iommu_mr, addr, is_write ? IOMMU_WO : IOMMU_RO); - addr =3D ((iotlb.translated_addr & ~iotlb.addr_mask) - | (addr & iotlb.addr_mask)); - page_mask &=3D iotlb.addr_mask; - plen =3D MIN(plen, (addr | iotlb.addr_mask) - addr + 1); if (!(iotlb.perm & (1 << is_write))) { goto translate_fail; } =20 + addr =3D ((iotlb.translated_addr & ~iotlb.addr_mask) + | (addr & iotlb.addr_mask)); + page_mask &=3D iotlb.addr_mask; + *plen_out =3D MIN(*plen_out, (addr | iotlb.addr_mask) - addr + 1); fv =3D address_space_to_flatview(iotlb.target_as); *target_as =3D iotlb.target_as; } =20 - *xlat =3D addr; - - if (page_mask =3D=3D (hwaddr)(-1)) { - /* Not behind an IOMMU, use default page size. */ - page_mask =3D ~TARGET_PAGE_MASK; - } - if (page_mask_out) { + if (page_mask =3D=3D (hwaddr)(-1)) { + /* Not behind an IOMMU, use default page size. */ + page_mask =3D ~TARGET_PAGE_MASK; + } *page_mask_out =3D page_mask; } =20 - if (plen_out) { - *plen_out =3D plen; - } - return *section; =20 translate_fail: --=20 2.17.0 From nobody Fri May 3 15:18:57 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1523974196034512.6087323597259; Tue, 17 Apr 2018 07:09:56 -0700 (PDT) Received: from localhost ([::1]:40973 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f8RIx-0001MJ-6i for importer@patchew.org; Tue, 17 Apr 2018 10:09:55 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36637) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f8RHE-0000IQ-AB for qemu-devel@nongnu.org; Tue, 17 Apr 2018 10:08:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f8RHD-0003GT-8G for qemu-devel@nongnu.org; Tue, 17 Apr 2018 10:08:08 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51538 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1f8RHD-0003GN-26 for qemu-devel@nongnu.org; Tue, 17 Apr 2018 10:08:07 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B45B2406E97E for ; Tue, 17 Apr 2018 14:08:06 +0000 (UTC) Received: from donizetti.redhat.com (ovpn-116-72.ams2.redhat.com [10.36.116.72]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4411C202660D for ; Tue, 17 Apr 2018 14:08:06 +0000 (UTC) From: Paolo Bonzini To: qemu-devel@nongnu.org Date: Tue, 17 Apr 2018 16:08:01 +0200 Message-Id: <20180417140802.16711-4-pbonzini@redhat.com> In-Reply-To: <20180417140802.16711-1-pbonzini@redhat.com> References: <20180417140802.16711-1-pbonzini@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Tue, 17 Apr 2018 14:08:06 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Tue, 17 Apr 2018 14:08:06 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'pbonzini@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 3/4] exec: extract address_space_translate_iommu, fix page_mask corner case X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This will be used to process IOMMUs in a MemoryRegionCache. This includes a small bugfix, in that the returned page_mask is now correctly -1 if the IOMMU memory region maps the entire address space directly. Previously, address_space_get_iotlb_entry would return ~TARGET_PAGE_MASK. Signed-off-by: Paolo Bonzini Reviewed-by: Peter Xu --- exec.c | 113 +++++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 78 insertions(+), 35 deletions(-) diff --git a/exec.c b/exec.c index 1dea3e9c95..868f89c345 100644 --- a/exec.c +++ b/exec.c @@ -461,6 +461,72 @@ address_space_translate_internal(AddressSpaceDispatch = *d, hwaddr addr, hwaddr *x return section; } =20 +/** + * address_space_translate_iommu - translate an address through an IOMMU + * memory region and then through the target address space. + * + * @iommu_mr: the IOMMU memory region that we start the translation from + * @addr: the address to be translated through the MMU + * @xlat: the translated address offset within the destination memory regi= on. + * It cannot be %NULL. + * @plen_out: valid read/write length of the translated address. It + * cannot be %NULL. + * @page_mask_out: page mask for the translated address. This + * should only be meaningful for IOMMU translated + * addresses, since there may be huge pages that this bit + * would tell. It can be %NULL if we don't care about it. + * @is_write: whether the translation operation is for write + * @is_mmio: whether this can be MMIO, set true if it can + * @target_as: the address space targeted by the IOMMU + * + * This function is called from RCU critical section. + */ +static MemoryRegionSection address_space_translate_iommu(IOMMUMemoryRegion= *iommu_mr, + hwaddr *xlat, + hwaddr *plen_out, + hwaddr *page_mask= _out, + bool is_write, + bool is_mmio, + AddressSpace **ta= rget_as) +{ + MemoryRegionSection *section; + hwaddr page_mask =3D (hwaddr)-1; + + do { + hwaddr addr =3D *xlat; + IOMMUMemoryRegionClass *imrc =3D memory_region_get_iommu_class_noc= heck(iommu_mr); + IOMMUTLBEntry iotlb =3D imrc->translate(iommu_mr, addr, is_write ? + IOMMU_WO : IOMMU_RO); + + if (!(iotlb.perm & (1 << is_write))) { + goto unassigned; + } + + addr =3D ((iotlb.translated_addr & ~iotlb.addr_mask) + | (addr & iotlb.addr_mask)); + page_mask &=3D iotlb.addr_mask; + *plen_out =3D MIN(*plen_out, (addr | iotlb.addr_mask) - addr + 1); + *target_as =3D iotlb.target_as; + + section =3D address_space_translate_internal( + address_space_to_dispatch(iotlb.target_as), addr, xlat, + plen_out, is_mmio); + if (!section) { + goto unassigned; + } + + iommu_mr =3D memory_region_get_iommu(section->mr); + } while (iommu_mr); + + if (page_mask_out) { + *page_mask_out =3D page_mask; + } + return *section; + +unassigned: + return (MemoryRegionSection) { .mr =3D &io_mem_unassigned }; +} + /** * flatview_do_translate - translate an address in FlatView * @@ -489,55 +556,31 @@ static MemoryRegionSection flatview_do_translate(Flat= View *fv, bool is_mmio, AddressSpace **target_as) { - IOMMUTLBEntry iotlb; MemoryRegionSection *section; IOMMUMemoryRegion *iommu_mr; - IOMMUMemoryRegionClass *imrc; - hwaddr page_mask =3D (hwaddr)(-1); hwaddr plen =3D (hwaddr)(-1); =20 if (!plen_out) { plen_out =3D &plen; } =20 - for (;;) { - section =3D address_space_translate_internal( - flatview_to_dispatch(fv), addr, xlat, - plen_out, is_mmio); + section =3D address_space_translate_internal( + flatview_to_dispatch(fv), addr, xlat, + plen_out, is_mmio); =20 - iommu_mr =3D memory_region_get_iommu(section->mr); - if (!iommu_mr) { - break; - } - imrc =3D memory_region_get_iommu_class_nocheck(iommu_mr); - - addr =3D *xlat; - iotlb =3D imrc->translate(iommu_mr, addr, is_write ? - IOMMU_WO : IOMMU_RO); - if (!(iotlb.perm & (1 << is_write))) { - goto translate_fail; - } - - addr =3D ((iotlb.translated_addr & ~iotlb.addr_mask) - | (addr & iotlb.addr_mask)); - page_mask &=3D iotlb.addr_mask; - *plen_out =3D MIN(*plen_out, (addr | iotlb.addr_mask) - addr + 1); - fv =3D address_space_to_flatview(iotlb.target_as); - *target_as =3D iotlb.target_as; + iommu_mr =3D memory_region_get_iommu(section->mr); + if (unlikely(iommu_mr)) { + return address_space_translate_iommu(iommu_mr, xlat, + plen_out, page_mask_out, + is_write, is_mmio, + target_as); } - if (page_mask_out) { - if (page_mask =3D=3D (hwaddr)(-1)) { - /* Not behind an IOMMU, use default page size. */ - page_mask =3D ~TARGET_PAGE_MASK; - } - *page_mask_out =3D page_mask; + /* Not behind an IOMMU, use default page size. */ + *page_mask_out =3D ~TARGET_PAGE_MASK; } =20 return *section; - -translate_fail: - return (MemoryRegionSection) { .mr =3D &io_mem_unassigned }; } =20 /* Called from RCU critical section */ --=20 2.17.0 From nobody Fri May 3 15:18:57 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1523974428247319.9817137327385; Tue, 17 Apr 2018 07:13:48 -0700 (PDT) Received: from localhost ([::1]:41152 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f8RMh-0004XA-Ef for importer@patchew.org; Tue, 17 Apr 2018 10:13:47 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36682) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f8RHK-0000O6-FY for qemu-devel@nongnu.org; Tue, 17 Apr 2018 10:08:17 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f8RHE-0003HF-Ar for qemu-devel@nongnu.org; Tue, 17 Apr 2018 10:08:14 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:39208 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1f8RHE-0003Gm-5X for qemu-devel@nongnu.org; Tue, 17 Apr 2018 10:08:08 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D29F1406AE2C for ; Tue, 17 Apr 2018 14:08:07 +0000 (UTC) Received: from donizetti.redhat.com (ovpn-116-72.ams2.redhat.com [10.36.116.72]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0279C200BCB0 for ; Tue, 17 Apr 2018 14:08:06 +0000 (UTC) From: Paolo Bonzini To: qemu-devel@nongnu.org Date: Tue, 17 Apr 2018 16:08:02 +0200 Message-Id: <20180417140802.16711-5-pbonzini@redhat.com> In-Reply-To: <20180417140802.16711-1-pbonzini@redhat.com> References: <20180417140802.16711-1-pbonzini@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Tue, 17 Apr 2018 14:08:07 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Tue, 17 Apr 2018 14:08:07 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'pbonzini@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 4/4] exec: reintroduce MemoryRegion caching X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" MemoryRegionCache was reverted to "normal" address_space_* operations for 2.9, due to lack of support for IOMMUs. Reinstate the optimizations, caching only the IOMMU translation at address_cache_init but not the IOMMU lookup and target AddressSpace translation are not cached; now that MemoryRegionCache supports IOMMUs, it becomes more widely applicable too. The inlined fast path is defined in memory_ldst_cached.inc.h, while the slow path uses memory_ldst.inc.c as before. The smaller fast path causes a little code size reduction in MemoryRegionCache users: hw/virtio/virtio.o text size before: 32373 hw/virtio/virtio.o text size after: 31941 Signed-off-by: Paolo Bonzini --- exec.c | 121 +++++++++++++++++++++++--- include/exec/cpu-all.h | 6 +- include/exec/memory-internal.h | 3 + include/exec/memory.h | 58 ++++++++++-- include/exec/memory_ldst_cached.inc.h | 108 +++++++++++++++++++++++ memory.c | 4 +- 6 files changed, 280 insertions(+), 20 deletions(-) create mode 100644 include/exec/memory_ldst_cached.inc.h diff --git a/exec.c b/exec.c index 868f89c345..8c01f01e98 100644 --- a/exec.c +++ b/exec.c @@ -3654,33 +3654,130 @@ int64_t address_space_cache_init(MemoryRegionCache= *cache, hwaddr len, bool is_write) { - cache->len =3D len; - cache->as =3D as; - cache->xlat =3D addr; - return len; + AddressSpaceDispatch *d; + hwaddr l; + MemoryRegion *mr; + + assert(len > 0); + + l =3D len; + cache->fv =3D address_space_get_flatview(as); + d =3D flatview_to_dispatch(cache->fv); + cache->mrs =3D *address_space_translate_internal(d, addr, &cache->xlat= , &l, true); + + mr =3D cache->mrs.mr; + memory_region_ref(mr); + if (memory_access_is_direct(mr, is_write)) { + l =3D flatview_extend_translation(cache->fv, addr, len, mr, + cache->xlat, l, is_write); + cache->ptr =3D qemu_ram_ptr_length(mr->ram_block, cache->xlat, &l,= true); + } else { + cache->ptr =3D NULL; + } + + cache->len =3D l; + cache->is_write =3D is_write; + return l; } =20 void address_space_cache_invalidate(MemoryRegionCache *cache, hwaddr addr, hwaddr access_len) { + assert(cache->is_write); + if (likely(cache->ptr)) { + invalidate_and_set_dirty(cache->mrs.mr, addr + cache->xlat, access= _len); + } } =20 void address_space_cache_destroy(MemoryRegionCache *cache) { - cache->as =3D NULL; + if (!cache->mrs.mr) { + return; + } + + if (xen_enabled()) { + xen_invalidate_map_cache_entry(cache->ptr); + } + memory_region_unref(cache->mrs.mr); + flatview_unref(cache->fv); + cache->mrs.mr =3D NULL; + cache->fv =3D NULL; +} + +/* Called from RCU critical section. This function has the same + * semantics as address_space_translate, but it only works on a + * predefined range of a MemoryRegion that was mapped with + * address_space_cache_init. + */ +static inline MemoryRegion *address_space_translate_cached( + MemoryRegionCache *cache, hwaddr addr, hwaddr *xlat, + hwaddr *plen, bool is_write) +{ + MemoryRegionSection section; + MemoryRegion *mr; + IOMMUMemoryRegion *iommu_mr; + AddressSpace *target_as; + + assert(!cache->ptr); + *xlat =3D addr + cache->xlat; + + mr =3D cache->mrs.mr; + iommu_mr =3D memory_region_get_iommu(mr); + if (!iommu_mr) { + /* MMIO region. */ + return mr; + } + + section =3D address_space_translate_iommu(iommu_mr, xlat, plen, + NULL, is_write, true, + &target_as); + return section.mr; +} + +/* Called from RCU critical section. address_space_read_cached uses this + * out of line function when the target is an MMIO or IOMMU region. + */ +void +address_space_read_cached_slow(MemoryRegionCache *cache, hwaddr addr, + void *buf, int len) +{ + hwaddr addr1, l; + MemoryRegion *mr; + + l =3D len; + mr =3D address_space_translate_cached(cache, addr, &addr1, &l, false); + flatview_read_continue(cache->fv, + addr, MEMTXATTRS_UNSPECIFIED, buf, len, + addr1, l, mr); +} + +/* Called from RCU critical section. address_space_write_cached uses this + * out of line function when the target is an MMIO or IOMMU region. + */ +void +address_space_write_cached_slow(MemoryRegionCache *cache, hwaddr addr, + const void *buf, int len) +{ + hwaddr addr1, l; + MemoryRegion *mr; + + l =3D len; + mr =3D address_space_translate_cached(cache, addr, &addr1, &l, true); + flatview_write_continue(cache->fv, + addr, MEMTXATTRS_UNSPECIFIED, buf, len, + addr1, l, mr); } =20 #define ARG1_DECL MemoryRegionCache *cache #define ARG1 cache -#define SUFFIX _cached -#define TRANSLATE(addr, ...) \ - address_space_translate(cache->as, cache->xlat + (addr), __VA_ARGS__) -#define IS_DIRECT(mr, is_write) true -#define MAP_RAM(mr, ofs) qemu_map_ram_ptr((mr)->ram_block, ofs) +#define SUFFIX _cached_slow +#define TRANSLATE(...) address_space_translate_cached(cache, __V= A_ARGS__) +#define IS_DIRECT(mr, is_write) memory_access_is_direct(mr, is_write) +#define MAP_RAM(mr, ofs) (cache->ptr + (ofs - cache->xlat)) #define INVALIDATE(mr, ofs, len) invalidate_and_set_dirty(mr, ofs, len) -#define RCU_READ_LOCK() rcu_read_lock() -#define RCU_READ_UNLOCK() rcu_read_unlock() +#define RCU_READ_LOCK() ((void)0) +#define RCU_READ_UNLOCK() ((void)0) #include "memory_ldst.inc.c" =20 /* virtual memory access for debug (includes writing to ROM) */ diff --git a/include/exec/cpu-all.h b/include/exec/cpu-all.h index 173edd1fb4..a635f532f9 100644 --- a/include/exec/cpu-all.h +++ b/include/exec/cpu-all.h @@ -175,7 +175,7 @@ extern unsigned long reserved_va; #define TARGET_ENDIANNESS #include "exec/memory_ldst.inc.h" =20 -#define SUFFIX _cached +#define SUFFIX _cached_slow #define ARG1 cache #define ARG1_DECL MemoryRegionCache *cache #define TARGET_ENDIANNESS @@ -193,6 +193,10 @@ static inline void stl_phys_notdirty(AddressSpace *as,= hwaddr addr, uint32_t val #define TARGET_ENDIANNESS #include "exec/memory_ldst_phys.inc.h" =20 +/* Inline fast path for direct RAM access. */ +#define ENDIANNESS +#include "exec/memory_ldst_cached.inc.h" + #define SUFFIX _cached #define ARG1 cache #define ARG1_DECL MemoryRegionCache *cache diff --git a/include/exec/memory-internal.h b/include/exec/memory-internal.h index 6a5ee42d36..58399b9318 100644 --- a/include/exec/memory-internal.h +++ b/include/exec/memory-internal.h @@ -31,6 +31,9 @@ static inline AddressSpaceDispatch *address_space_to_disp= atch(AddressSpace *as) return flatview_to_dispatch(address_space_to_flatview(as)); } =20 +FlatView *address_space_get_flatview(AddressSpace *as); +void flatview_unref(FlatView *view); + extern const MemoryRegionOps unassigned_mem_ops; =20 bool memory_region_access_valid(MemoryRegion *mr, hwaddr addr, diff --git a/include/exec/memory.h b/include/exec/memory.h index ca361bc409..525619a5f4 100644 --- a/include/exec/memory.h +++ b/include/exec/memory.h @@ -1688,12 +1688,16 @@ MemTxResult address_space_write(AddressSpace *as, h= waddr addr, #include "exec/memory_ldst_phys.inc.h" =20 struct MemoryRegionCache { + void *ptr; hwaddr xlat; hwaddr len; - AddressSpace *as; + FlatView *fv; + MemoryRegionSection mrs; + bool is_write; }; =20 -#define MEMORY_REGION_CACHE_INVALID ((MemoryRegionCache) { .as =3D NULL }) +#define MEMORY_REGION_CACHE_INVALID ((MemoryRegionCache) { .mrs.mr =3D NUL= L }) + =20 /* address_space_ld*_cached: load from a cached #MemoryRegion * address_space_st*_cached: store into a cached #MemoryRegion @@ -1719,11 +1723,40 @@ struct MemoryRegionCache { * if NULL, this information is discarded */ =20 -#define SUFFIX _cached +#define SUFFIX _cached_slow #define ARG1 cache #define ARG1_DECL MemoryRegionCache *cache #include "exec/memory_ldst.inc.h" =20 +/* Inline fast path for direct RAM access. */ +static inline uint8_t address_space_ldub_cached(MemoryRegionCache *cache, + hwaddr addr, MemTxAttrs attrs, MemTxResult *result) +{ + assert(addr < cache->len); + if (likely(cache->ptr)) { + return ldub_p(cache->ptr + addr); + } else { + return address_space_ldub_cached_slow(cache, addr, attrs, result); + } +} + +static inline void address_space_stb_cached(MemoryRegionCache *cache, + hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result) +{ + assert(addr < cache->len); + if (likely(cache->ptr)) { + stb_p(cache->ptr + addr, val); + } else { + address_space_stb_cached_slow(cache, addr, val, attrs, result); + } +} + +#define ENDIANNESS _le +#include "exec/memory_ldst_cached.inc.h" + +#define ENDIANNESS _be +#include "exec/memory_ldst_cached.inc.h" + #define SUFFIX _cached #define ARG1 cache #define ARG1_DECL MemoryRegionCache *cache @@ -1860,6 +1893,13 @@ MemTxResult flatview_read_continue(FlatView *fv, hwa= ddr addr, MemoryRegion *mr); void *qemu_map_ram_ptr(RAMBlock *ram_block, ram_addr_t addr); =20 +/* Internal functions, part of the implementation of address_space_read_ca= ched + * and address_space_write_cached. */ +void address_space_read_cached_slow(MemoryRegionCache *cache, + hwaddr addr, void *buf, int len); +void address_space_write_cached_slow(MemoryRegionCache *cache, + hwaddr addr, const void *buf, int len= ); + static inline bool memory_access_is_direct(MemoryRegion *mr, bool is_write) { if (is_write) { @@ -1928,7 +1968,11 @@ address_space_read_cached(MemoryRegionCache *cache, = hwaddr addr, void *buf, int len) { assert(addr < cache->len && len <=3D cache->len - addr); - address_space_read(cache->as, cache->xlat + addr, MEMTXATTRS_UNSPECIFI= ED, buf, len); + if (likely(cache->ptr)) { + memcpy(buf, cache->ptr + addr, len); + } else { + address_space_read_cached_slow(cache, addr, buf, len); + } } =20 /** @@ -1944,7 +1988,11 @@ address_space_write_cached(MemoryRegionCache *cache,= hwaddr addr, void *buf, int len) { assert(addr < cache->len && len <=3D cache->len - addr); - address_space_write(cache->as, cache->xlat + addr, MEMTXATTRS_UNSPECIF= IED, buf, len); + if (likely(cache->ptr)) { + memcpy(cache->ptr + addr, buf, len); + } else { + address_space_write_cached_slow(cache, addr, buf, len); + } } =20 #endif diff --git a/include/exec/memory_ldst_cached.inc.h b/include/exec/memory_ld= st_cached.inc.h new file mode 100644 index 0000000000..fd4bbb40e7 --- /dev/null +++ b/include/exec/memory_ldst_cached.inc.h @@ -0,0 +1,108 @@ +/* + * Memory access templates for MemoryRegionCache + * + * Copyright (c) 2018 Red Hat, Inc. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, see . + */ + +#define ADDRESS_SPACE_LD_CACHED(size) \ + glue(glue(address_space_ld, size), glue(ENDIANNESS, _cached)) +#define ADDRESS_SPACE_LD_CACHED_SLOW(size) \ + glue(glue(address_space_ld, size), glue(ENDIANNESS, _cached_slow)) +#define LD_P(size) \ + glue(glue(ld, size), glue(ENDIANNESS, _p)) + +static inline uint32_t ADDRESS_SPACE_LD_CACHED(l)(MemoryRegionCache *cache, + hwaddr addr, MemTxAttrs attrs, MemTxResult *result) +{ + assert(addr < cache->len && 4 <=3D cache->len - addr); + if (likely(cache->ptr)) { + return LD_P(l)(cache->ptr + addr); + } else { + return ADDRESS_SPACE_LD_CACHED_SLOW(l)(cache, addr, attrs, result); + } +} + +static inline uint64_t ADDRESS_SPACE_LD_CACHED(q)(MemoryRegionCache *cache, + hwaddr addr, MemTxAttrs attrs, MemTxResult *result) +{ + assert(addr < cache->len && 8 <=3D cache->len - addr); + if (likely(cache->ptr)) { + return LD_P(q)(cache->ptr + addr); + } else { + return ADDRESS_SPACE_LD_CACHED_SLOW(q)(cache, addr, attrs, result); + } +} + +static inline uint32_t ADDRESS_SPACE_LD_CACHED(uw)(MemoryRegionCache *cach= e, + hwaddr addr, MemTxAttrs attrs, MemTxResult *result) +{ + assert(addr < cache->len && 2 <=3D cache->len - addr); + if (likely(cache->ptr)) { + return LD_P(uw)(cache->ptr + addr); + } else { + return ADDRESS_SPACE_LD_CACHED_SLOW(uw)(cache, addr, attrs, result= ); + } +} + +#undef ADDRESS_SPACE_LD_CACHED +#undef ADDRESS_SPACE_LD_CACHED_SLOW +#undef LD_P + +#define ADDRESS_SPACE_ST_CACHED(size) \ + glue(glue(address_space_st, size), glue(ENDIANNESS, _cached)) +#define ADDRESS_SPACE_ST_CACHED_SLOW(size) \ + glue(glue(address_space_st, size), glue(ENDIANNESS, _cached_slow)) +#define ST_P(size) \ + glue(glue(st, size), glue(ENDIANNESS, _p)) + +static inline void ADDRESS_SPACE_ST_CACHED(l)(MemoryRegionCache *cache, + hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result) +{ + assert(addr < cache->len && 4 <=3D cache->len - addr); + if (likely(cache->ptr)) { + ST_P(l)(cache->ptr + addr, val); + } else { + ADDRESS_SPACE_ST_CACHED_SLOW(l)(cache, addr, val, attrs, result); + } +} + +static inline void ADDRESS_SPACE_ST_CACHED(w)(MemoryRegionCache *cache, + hwaddr addr, uint32_t val, MemTxAttrs attrs, MemTxResult *result) +{ + assert(addr < cache->len && 2 <=3D cache->len - addr); + if (likely(cache->ptr)) { + ST_P(w)(cache->ptr + addr, val); + } else { + ADDRESS_SPACE_ST_CACHED_SLOW(w)(cache, addr, val, attrs, result); + } +} + +static inline void ADDRESS_SPACE_ST_CACHED(q)(MemoryRegionCache *cache, + hwaddr addr, uint64_t val, MemTxAttrs attrs, MemTxResult *result) +{ + assert(addr < cache->len && 8 <=3D cache->len - addr); + if (likely(cache->ptr)) { + ST_P(q)(cache->ptr + addr, val); + } else { + ADDRESS_SPACE_ST_CACHED_SLOW(q)(cache, addr, val, attrs, result); + } +} + +#undef ADDRESS_SPACE_ST_CACHED +#undef ADDRESS_SPACE_ST_CACHED_SLOW +#undef ST_P + +#undef ENDIANNESS diff --git a/memory.c b/memory.c index e70b64b8b9..fc7f9b782b 100644 --- a/memory.c +++ b/memory.c @@ -298,7 +298,7 @@ static bool flatview_ref(FlatView *view) return atomic_fetch_inc_nonzero(&view->ref) > 0; } =20 -static void flatview_unref(FlatView *view) +void flatview_unref(FlatView *view) { if (atomic_fetch_dec(&view->ref) =3D=3D 1) { trace_flatview_destroy_rcu(view, view->root); @@ -822,7 +822,7 @@ static void address_space_add_del_ioeventfds(AddressSpa= ce *as, } } =20 -static FlatView *address_space_get_flatview(AddressSpace *as) +FlatView *address_space_get_flatview(AddressSpace *as) { FlatView *view; =20 --=20 2.17.0