From nobody Wed Nov 5 13:17:22 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1534388357145546.8477223459454; Wed, 15 Aug 2018 19:59:17 -0700 (PDT) Received: from localhost ([::1]:53245 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fq8VI-0005sR-0u for importer@patchew.org; Wed, 15 Aug 2018 22:59:16 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50451) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fq8RB-0002IW-5e for qemu-devel@nongnu.org; Wed, 15 Aug 2018 22:55:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fq8R7-0006i2-RW for qemu-devel@nongnu.org; Wed, 15 Aug 2018 22:55:01 -0400 Received: from mail-pf1-x443.google.com ([2607:f8b0:4864:20::443]:40105) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fq8R7-0006hX-HQ for qemu-devel@nongnu.org; Wed, 15 Aug 2018 22:54:57 -0400 Received: by mail-pf1-x443.google.com with SMTP id e13-v6so1330792pff.7 for ; Wed, 15 Aug 2018 19:54:57 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id k64-v6sm45231497pfc.160.2018.08.15.19.54.54 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 15 Aug 2018 19:54:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=utTT1tFGHMA9lVa/XS5scEPRMxbQt548UTW7BU+eN4w=; b=hvhc9e9538htRKrHfM1sVVkZotBhDBOlpcOurxUfOGmRHJUTb6IumejUpQrQUX7Bg9 6vXj1BJGcugSMDPpjHpdyEFjHv2DXSjeBrUoaaY6rhVDafL/sWxZgAmPp56yeDC/fBk4 I+4KJvlmN6S2/3je4S1/NHPcbzonB/N/BcoZc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=utTT1tFGHMA9lVa/XS5scEPRMxbQt548UTW7BU+eN4w=; b=fJG6txc1yCbvuGEIwLMkWeR8SVnZQCHtp8lN6CEE3mVyorM7+YGASYIJPne+c/TW4o tS7WImJbpGsUXZ2UbmBz/kD/bd1Oqfy2mibBKwDfRctkii6jQO37Lfeooj0cgMSa4Ll8 FMLIq1YpnNfB8q1xM+t+rDhm1C9lg3nHrclP0QmpwVwOo6P5dRUjHv9hdjCtXIUuTSp+ gdf5dankAs+7U4+LdniPpeoPp/t3anvqbdkTVeubM+6yG8PDJM6wn511W75SJkNRML7y A8/8Rh5YxtGJiB5yvHvhZugYlUwngGiFxHGE6E2smiXMKWnMgJTMqwLYczUBGUwfk7VG zWBg== X-Gm-Message-State: AOUpUlFP1wXjHeBO0vWnlThFcpxq5zamQRqTripV79GsaiUHmvRzwNCa 5IMpThUwSksQUBqC4vYMpwlBUf0abYY= X-Google-Smtp-Source: AA+uWPyosPWXgwDh7F5v3QcFaXw1c7PC4bLucIB6Ltw+PwlajWJKlxJSz0VIL6rCyfNzL+jVqNwfiw== X-Received: by 2002:a62:1a8f:: with SMTP id a137-v6mr30442676pfa.190.1534388096003; Wed, 15 Aug 2018 19:54:56 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 15 Aug 2018 19:54:48 -0700 Message-Id: <20180816025452.21358-2-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180816025452.21358-1-richard.henderson@linaro.org> References: <20180816025452.21358-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::443 Subject: [Qemu-devel] [PATCH 1/5] tcg: Split CONFIG_ATOMIC128 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: pbonzini@redhat.com, cota@braap.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDMRC_1 RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" GCC7+ will no longer advertise support for 16-byte __atomic operations if only cmpxchg is supported, as for x86_64. Fortunately, x86_64 still has support for __sync_compare_and_swap_16 and we can make use of that. AArch64 does not have, nor ever has had such support, so open-code it. Signed-off-by: Richard Henderson --- accel/tcg/atomic_template.h | 22 +++-- include/qemu/atomic128.h | 162 ++++++++++++++++++++++++++++++++++++ tcg/tcg.h | 29 ++++--- accel/tcg/cputlb.c | 3 +- accel/tcg/user-exec.c | 5 +- configure | 19 +++++ 6 files changed, 219 insertions(+), 21 deletions(-) create mode 100644 include/qemu/atomic128.h diff --git a/accel/tcg/atomic_template.h b/accel/tcg/atomic_template.h index d751bcba48..a056b7b408 100644 --- a/accel/tcg/atomic_template.h +++ b/accel/tcg/atomic_template.h @@ -100,19 +100,24 @@ ABI_TYPE ATOMIC_NAME(cmpxchg)(CPUArchState *env, targ= et_ulong addr, DATA_TYPE ret; =20 ATOMIC_TRACE_RMW; - ret =3D atomic_cmpxchg__nocheck(haddr, cmpv, newv); +#if DATA_SIZE =3D=3D 16 + ret =3D atomic16_cmpxchg(haddr, cmpv, newv); +#else + ret =3D atomic_cmpxchg(haddr, cmpv, newv); +#endif ATOMIC_MMU_CLEANUP; return ret; } =20 #if DATA_SIZE >=3D 16 +#if HAVE_ATOMIC128 ABI_TYPE ATOMIC_NAME(ld)(CPUArchState *env, target_ulong addr EXTRA_ARGS) { ATOMIC_MMU_DECLS; DATA_TYPE val, *haddr =3D ATOMIC_MMU_LOOKUP; =20 ATOMIC_TRACE_LD; - __atomic_load(haddr, &val, __ATOMIC_RELAXED); + val =3D atomic16_read(haddr); ATOMIC_MMU_CLEANUP; return val; } @@ -124,9 +129,10 @@ void ATOMIC_NAME(st)(CPUArchState *env, target_ulong a= ddr, DATA_TYPE *haddr =3D ATOMIC_MMU_LOOKUP; =20 ATOMIC_TRACE_ST; - __atomic_store(haddr, &val, __ATOMIC_RELAXED); + atomic16_set(haddr, val); ATOMIC_MMU_CLEANUP; } +#endif #else ABI_TYPE ATOMIC_NAME(xchg)(CPUArchState *env, target_ulong addr, ABI_TYPE val EXTRA_ARGS) @@ -228,19 +234,24 @@ ABI_TYPE ATOMIC_NAME(cmpxchg)(CPUArchState *env, targ= et_ulong addr, DATA_TYPE ret; =20 ATOMIC_TRACE_RMW; +#if DATA_SIZE =3D=3D 16 + ret =3D atomic16_cmpxchg(haddr, BSWAP(cmpv), BSWAP(newv)); +#else ret =3D atomic_cmpxchg__nocheck(haddr, BSWAP(cmpv), BSWAP(newv)); +#endif ATOMIC_MMU_CLEANUP; return BSWAP(ret); } =20 #if DATA_SIZE >=3D 16 +#if HAVE_ATOMIC128 ABI_TYPE ATOMIC_NAME(ld)(CPUArchState *env, target_ulong addr EXTRA_ARGS) { ATOMIC_MMU_DECLS; DATA_TYPE val, *haddr =3D ATOMIC_MMU_LOOKUP; =20 ATOMIC_TRACE_LD; - __atomic_load(haddr, &val, __ATOMIC_RELAXED); + val =3D atomic16_read(haddr); ATOMIC_MMU_CLEANUP; return BSWAP(val); } @@ -253,9 +264,10 @@ void ATOMIC_NAME(st)(CPUArchState *env, target_ulong a= ddr, =20 ATOMIC_TRACE_ST; val =3D BSWAP(val); - __atomic_store(haddr, &val, __ATOMIC_RELAXED); + atomic16_set(haddr, val); ATOMIC_MMU_CLEANUP; } +#endif #else ABI_TYPE ATOMIC_NAME(xchg)(CPUArchState *env, target_ulong addr, ABI_TYPE val EXTRA_ARGS) diff --git a/include/qemu/atomic128.h b/include/qemu/atomic128.h new file mode 100644 index 0000000000..2613ebd352 --- /dev/null +++ b/include/qemu/atomic128.h @@ -0,0 +1,162 @@ +/* + * Simple interface for 128-bit atomic operations. + * + * Copyright (C) 2018 Linaro, Ltd. + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + * + * See docs/devel/atomics.txt for discussion about the guarantees each + * atomic primitive is meant to provide. + */ + +#ifndef QEMU_ATOMIC128_H +#define QEMU_ATOMIC128_H + +/* + * GCC is a house divided about supporting large atomic operations. + * + * For hosts that only have large compare-and-swap, a legalistic reading + * of the C++ standard means that one cannot implement __atomic_read on + * read-only memory, and thus all atomic operations must synchronize + * through libatomic. + * + * See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D80878 + * + * This interpretation is not especially helpful for QEMU. + * For softmmu, all RAM is always read/write from the hypervisor. + * For user-only, if the guest doesn't implement such an __atomic_read + * then the host need not worry about it either. + * + * Moreover, using libatomic is not an option, because its interface is + * built for std::atomic, and requires that *all* accesses to such an + * object go through the library. In our case we do not have an object + * in the C/C++ sense, but a view of memory as seen by the guest. + * The guest may issue a large atomic operation and then access those + * pieces using word-sized accesses. From the hypervisor, we have no + * way to connect those two actions. + * + * Therefore, special case each platform. + */ + +#if defined(CONFIG_ATOMIC128) +static inline Int128 atomic16_cmpxchg(Int128 *ptr, Int128 cmp, Int128 new) +{ + return atomic_cmpxchg__nocheck(ptr, cmp, new); +} +# define HAVE_CMPXCHG128 1 +#elif defined(CONFIG_CMPXCHG128) +static inline Int128 atomic16_cmpxchg(Int128 *ptr, Int128 cmp, Int128 new) +{ + return __sync_val_compare_and_swap_16(ptr, cmp, new); +} +# define HAVE_CMPXCHG128 1 +#elif defined(__aarch64__) +/* Through gcc 8, aarch64 has no support for 128-bit at all. */ +static inline Int128 atomic16_cmpxchg(Int128 *ptr, Int128 cmp, Int128 new) +{ + uint64_t cmpl =3D cmp, cmph =3D cmp >> 64; + uint64_t newl =3D new, newh =3D new >> 64; + uint64_t oldl, oldh; + uint32_t tmp; + + asm("0: ldaxp %[oldl], %[oldh], %[mem]\n\t" + "cmp %[oldl], %[cmpl]\n\t" + "ccmp %[oldh], %[cmph], #0, eq\n\t" + "b.ne 1f\n\t" + "stlxp %w[tmp], %[newl], %[newh], %[mem]\n\t" + "cbz %w[tmp], 0b\n" + "1:" + : [mem] "+m"(*ptr), [tmp] "=3D&r"(tmp), + [oldl] "=3D&r"(oldl), [oldh] "=3Dr"(oldh) + : [cmpl] "r"(cmpl), [cmph] "r"(cmph), + [newl] "r"(newl), [newh] "r"(newh) + : "memory", "cc"); + + return int128_make128(oldl, oldh); +} +# define HAVE_CMPXCHG128 1 +#endif /* Some definition for HAVE_CMPXCHG128 */ + + +#if defined(CONFIG_ATOMIC128) +static inline Int128 atomic16_read(Int128 *ptr) +{ + return atomic_read__nocheck(ptr); +} + +static inline void atomic16_set(Int128 *ptr, Int128 val) +{ + atomic_set__nocheck(ptr, val); +} + +# define HAVE_ATOMIC128 1 +#elif !defined(CONFIG_USER_ONLY) +# ifdef __aarch64__ +/* We can do better than cmpxchg for AArch64. */ +static inline Int128 atomic16_read(Int128 *ptr) +{ + uint64_t l, h; + uint32_t tmp; + + /* The load must be paired with the store to guarantee not tearing. */ + asm("0: ldxp %[l], %[h], %[mem]\n\t" + "stxp %w[tmp], %[l], %[h], %[mem]\n\t" + "cbz %w[tmp], 0b" + : [mem] "+m"(*ptr), [tmp] "=3Dr"(tmp), [l] "=3Dr"(l), [h] "=3Dr"(h= )); + + return int128_make128(l, h); +} + +static inline void atomic16_set(Int128 *ptr, Int128 val) +{ + uint64_t l =3D val, h =3D val >> 64, t1, t2; + + /* Load into temporaries to acquire the exclusive access lock. */ + asm("0: ldxp %[t1], %[t2], %[mem]\n\t" + "stxp %w[t1], %[l], %[h], %[mem]\n\t" + "cbz %w[t1], 0b" + : [mem] "+m"(*ptr), [t1] "=3D&r"(t1), [t2] "=3D&r"(t2) + : [l] "r"(l), [h] "r"(h)); +} + +# define HAVE_ATOMIC128 1 +# elif HAVE_CMPXCHG128 +static inline Int128 atomic16_read(Int128 *ptr) +{ + /* Maybe replace 0 with 0, returning the old value. */ + return atomic16_cmpxchg(ptr, 0, 0); +} + +static inline void atomic16_set(Int128 *ptr, Int128 val) +{ + Int128 old =3D *ptr, cmp; + do { + cmp =3D old; + old =3D atomic16_cmpxchg(ptr, cmp, val); + } while (old !=3D cmp); +} + +# define HAVE_ATOMIC128 1 +# endif +#endif + +/* + * Fallback definitions that must be optimized away, or error. + */ + +#ifndef HAVE_CMPXCHG128 +Int128 __attribute__((error("unsupported cmpxchg"))) + atomic16_cmpxchg(Int128 *ptr, Int128 cmp, Int128 new); +# define HAVE_CMPXCHG128 0 +#endif + +#ifndef HAVE_ATOMIC128 +Int128 __attribute__((error("unsupported atomic16_read"))) + atomic16_read(Int128 *ptr, Int128 cmp, Int128 new); +Int128 __attribute__((error("unsupported atomic16_set"))) + atomic16_set(Int128 *ptr, Int128 cmp, Int128 new); +# define HAVE_ATOMIC128 0 +#endif + +#endif /* QEMU_ATOMIC128_H */ diff --git a/tcg/tcg.h b/tcg/tcg.h index f9f12378e9..297b3f06ee 100644 --- a/tcg/tcg.h +++ b/tcg/tcg.h @@ -32,6 +32,8 @@ #include "qemu/queue.h" #include "tcg-mo.h" #include "tcg-target.h" +#include "qemu/atomic.h" +#include "qemu/int128.h" =20 /* XXX: make safe guess about sizes */ #define MAX_OP_PER_INSTR 266 @@ -1454,27 +1456,28 @@ GEN_ATOMIC_HELPER_ALL(xchg) #undef GEN_ATOMIC_HELPER #endif /* CONFIG_SOFTMMU */ =20 -#ifdef CONFIG_ATOMIC128 -#include "qemu/int128.h" - -/* These aren't really a "proper" helpers because TCG cannot manage Int128. - However, use the same format as the others, for use by the backends. */ +/* + * These aren't really a "proper" helpers because TCG cannot manage Int128. + * However, use the same format as the others, for use by the backends. + * + * The cmpxchg functions are only defined if HAVE_CMPXCHG128; + * the ld/st functions are only defined if HAVE_ATOMIC128, + * as defined by . + */ Int128 helper_atomic_cmpxchgo_le_mmu(CPUArchState *env, target_ulong addr, Int128 cmpv, Int128 newv, - TCGMemOpIdx oi, uintptr_t retaddr); + TCGMemOpIdx oi, uintptr_t ra); Int128 helper_atomic_cmpxchgo_be_mmu(CPUArchState *env, target_ulong addr, Int128 cmpv, Int128 newv, - TCGMemOpIdx oi, uintptr_t retaddr); + TCGMemOpIdx oi, uintptr_t ra); =20 Int128 helper_atomic_ldo_le_mmu(CPUArchState *env, target_ulong addr, - TCGMemOpIdx oi, uintptr_t retaddr); + TCGMemOpIdx oi, uintptr_t ra); Int128 helper_atomic_ldo_be_mmu(CPUArchState *env, target_ulong addr, - TCGMemOpIdx oi, uintptr_t retaddr); + TCGMemOpIdx oi, uintptr_t ra); void helper_atomic_sto_le_mmu(CPUArchState *env, target_ulong addr, Int128= val, - TCGMemOpIdx oi, uintptr_t retaddr); + TCGMemOpIdx oi, uintptr_t ra); void helper_atomic_sto_be_mmu(CPUArchState *env, target_ulong addr, Int128= val, - TCGMemOpIdx oi, uintptr_t retaddr); - -#endif /* CONFIG_ATOMIC128 */ + TCGMemOpIdx oi, uintptr_t ra); =20 #endif /* TCG_H */ diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index 563fa30117..a7ea35a9e9 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -32,6 +32,7 @@ #include "exec/log.h" #include "exec/helper-proto.h" #include "qemu/atomic.h" +#include "qemu/atomic128.h" =20 /* DEBUG defines, enable DEBUG_TLB_LOG to log to the CPU_LOG_MMU target */ /* #define DEBUG_TLB */ @@ -1188,7 +1189,7 @@ static void *atomic_mmu_lookup(CPUArchState *env, tar= get_ulong addr, #include "atomic_template.h" #endif =20 -#ifdef CONFIG_ATOMIC128 +#if HAVE_CMPXCHG128 || HAVE_ATOMIC128 #define DATA_SIZE 16 #include "atomic_template.h" #endif diff --git a/accel/tcg/user-exec.c b/accel/tcg/user-exec.c index 26a3ffbba1..cd75829cf2 100644 --- a/accel/tcg/user-exec.c +++ b/accel/tcg/user-exec.c @@ -25,6 +25,7 @@ #include "exec/cpu_ldst.h" #include "translate-all.h" #include "exec/helper-proto.h" +#include "qemu/atomic128.h" =20 #undef EAX #undef ECX @@ -615,7 +616,7 @@ static void *atomic_mmu_lookup(CPUArchState *env, targe= t_ulong addr, /* The following is only callable from other helpers, and matches up with the softmmu version. */ =20 -#ifdef CONFIG_ATOMIC128 +#if HAVE_ATOMIC128 || HAVE_CMPXCHG128 =20 #undef EXTRA_ARGS #undef ATOMIC_NAME @@ -628,4 +629,4 @@ static void *atomic_mmu_lookup(CPUArchState *env, targe= t_ulong addr, =20 #define DATA_SIZE 16 #include "atomic_template.h" -#endif /* CONFIG_ATOMIC128 */ +#endif diff --git a/configure b/configure index 2a7796ea80..7e24d89843 100755 --- a/configure +++ b/configure @@ -5117,6 +5117,21 @@ EOF fi fi =20 +cmpxchg128=3Dno +if test "$int128" =3D yes -a "$atomic128" =3D no; then + cat > $TMPC << EOF +int main(void) +{ + unsigned __int128 x =3D 0, y =3D 0; + __sync_val_compare_and_swap_16(&x, y, x); + return 0; +} +EOF + if compile_prog "" "" ; then + cmpxchg128=3Dyes + fi +fi + ######################################### # See if 64-bit atomic operations are supported. # Note that without __atomic builtins, we can only @@ -6610,6 +6625,10 @@ if test "$atomic128" =3D "yes" ; then echo "CONFIG_ATOMIC128=3Dy" >> $config_host_mak fi =20 +if test "$cmpxchg128" =3D "yes" ; then + echo "CONFIG_CMPXCHG128=3Dy" >> $config_host_mak +fi + if test "$atomic64" =3D "yes" ; then echo "CONFIG_ATOMIC64=3Dy" >> $config_host_mak fi --=20 2.17.1