From nobody Thu Apr 25 12:44:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1568166418; cv=none; d=zoho.com; s=zohoarc; b=D9iAA9RzIsiq5/Tn7diSJIZfXXO7rBCcClu2fJGGT0sEE3gkC8HGchwZ4+L7PHdtNcFKPLx6iSvwNRJAObYaxrdS/8QWZP5n+bPXqgE8/hRLnu3zTgJ1pLrv/+xx2jlX3T1Y7mmTK70qkAX7cAsf8lAC9TLex1PhMZYPrcoZHv4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1568166418; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=v4itJ8e/ejo0YBL9s61Q35bPcHrTTy3X5DE6Cxbfjzc=; b=ftAYyCSzVdIc1A/p93b/jRzPST0jZAz4N0MC9H/lXHY3q6txpdX2BgYi6l7s41YGiMxzzsBXeWBE0r2HN/MtJcmS9auprsfNJuXFMjdLbe6pQr+ypc3+o0IDsdx57vAHBjRse4H2+6r7pmBK0qb4mI+8h7wt3q7HE0NUY+U/lpk= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (209.51.188.17 [209.51.188.17]) by mx.zohomail.com with SMTPS id 1568166417956838.626386973906; Tue, 10 Sep 2019 18:46:57 -0700 (PDT) Received: from localhost ([::1]:45912 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i7ria-0002vr-85 for importer@patchew.org; Tue, 10 Sep 2019 21:46:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53856) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i7rfr-0001gQ-6O for qemu-devel@nongnu.org; Tue, 10 Sep 2019 21:44:00 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1i7rfq-0000a5-5V for qemu-devel@nongnu.org; Tue, 10 Sep 2019 21:43:59 -0400 Received: from mail-qt1-x841.google.com ([2607:f8b0:4864:20::841]:42272) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1i7rfq-0000a1-1w for qemu-devel@nongnu.org; Tue, 10 Sep 2019 21:43:58 -0400 Received: by mail-qt1-x841.google.com with SMTP id c17so3922050qtv.9 for ; Tue, 10 Sep 2019 18:43:58 -0700 (PDT) Received: from localhost.localdomain (otwaon236nw-grc-01-64-229-69-35.dsl.bell.ca. [64.229.69.35]) by smtp.gmail.com with ESMTPSA id a14sm10074676qkg.59.2019.09.10.18.43.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Sep 2019 18:43:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=v4itJ8e/ejo0YBL9s61Q35bPcHrTTy3X5DE6Cxbfjzc=; b=F63EEwzd/KNZhyEFNFQi/9f3Oa5pV2Q1a+Encvx19d5Dw4IxbcIObKJdryto7d9nMX jLlx5gkK452Evir4XS3AT3a+lX+/Izf+HW7nvodE/BtJNaifNhg0fo4aNMEVJ3HuRcYH Xvh3kxDTT+9otZc3mhYLmBlvevGQb/1HMHPMiVm9vasbqRhuE67GqrixiWoX2vrTgsRt +Xw6GM2zMFVq++Eejuu9vTTxvsPokCxnzABvsLVfMhdlfPpLDNqBVDIynVrJ1Ynqo32T xjxFAyJgR5hux8cT0MGSYYPv81AEBKLKxECBnC/POra6h0iYyc7n/mxxFfoTW/saEkrw GX4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=v4itJ8e/ejo0YBL9s61Q35bPcHrTTy3X5DE6Cxbfjzc=; b=YunNDTf0MuCQiVnBADBqaT24A6nPB8RnlGKDOM11dsOucHT22nU7C9m0S4wuA/LSl0 PSWUEowUMzccWQf+3NA8aEHHmN5S+ZgF1nnHZkn6hGseq3gSWmTwNuRyeEgk8tTkdutL JsYrWwjq3BifDX8fgdNDlkZf2V8KMvo4ximCL5+KROSU781D2A15wkTCdMZfHiY0QsJo aVfMUBeyudMspDmSX5VYsCaE3LWuXjqDoVVudSGeRrdTcUysjPNrwc0wB0ZbbDAmUUIb 8MfiM1XbV7+9ndkIGbWbkGEi9sCTv6Hl3pabZsvNDsCXcpM8opyharhAZrtRYeaBkm2r /iMA== X-Gm-Message-State: APjAAAXUvbqbzfLT2j7SwYWc4kBNSrf1SCb1vhofpWl0vB62A/AQH4+W +H/MNjoNKNVsrv17y6zBo0+mZ6X5Zbem3A== X-Google-Smtp-Source: APXvYqx4d0FBPXakZ3mJD+T2vNArsY0pwZpMHtvB/STCaeiGilyI0fGZ0ecNqJp6/zYOAal54sWD8Q== X-Received: by 2002:ac8:2d2c:: with SMTP id n41mr32519611qta.335.1568166237205; Tue, 10 Sep 2019 18:43:57 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 10 Sep 2019 21:43:51 -0400 Message-Id: <20190911014353.5926-2-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190911014353.5926-1-richard.henderson@linaro.org> References: <20190911014353.5926-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::841 Subject: [Qemu-devel] [PATCH 1/3] cputlb: Disable __always_inline__ without optimization X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, mark.cave-ayland@ilande.co.uk, tony.nguyen@bt.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This forced inlining can result in missing symbols, which makes a debugging build harder to follow. Reported-by: Peter Maydell Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- accel/tcg/cputlb.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index abae79650c..909f01ebcc 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -1269,6 +1269,18 @@ static void *atomic_mmu_lookup(CPUArchState *env, ta= rget_ulong addr, cpu_loop_exit_atomic(env_cpu(env), retaddr); } =20 +/* + * In order for the expected constant folding to happen, + * we require that some functions be inlined. + * However, this inlining can make debugging harder for a + * non-optimizing build. + */ +#ifdef __OPTIMIZE__ +#define ALWAYS_INLINE __attribute__((always_inline)) +#else +#define ALWAYS_INLINE +#endif + /* * Load Helpers * @@ -1281,7 +1293,7 @@ static void *atomic_mmu_lookup(CPUArchState *env, tar= get_ulong addr, typedef uint64_t FullLoadHelper(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr); =20 -static inline uint64_t __attribute__((always_inline)) +static inline uint64_t ALWAYS_INLINE load_helper(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr, MemOp op, bool code_read, FullLoadHelper *full_load) @@ -1530,7 +1542,7 @@ tcg_target_ulong helper_be_ldsl_mmu(CPUArchState *env= , target_ulong addr, * Store Helpers */ =20 -static inline void __attribute__((always_inline)) +static inline void ALWAYS_INLINE store_helper(CPUArchState *env, target_ulong addr, uint64_t val, TCGMemOpIdx oi, uintptr_t retaddr, MemOp op) { --=20 2.17.1 From nobody Thu Apr 25 12:44:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1568166418; cv=none; d=zoho.com; s=zohoarc; b=lW5L99VxLhuNI27ox7WDWjEViJaUzLnuJpCQxsRveF9QeWdsk+4fIVUrFKkXLRfFKOqPM3NwZb1lBw/yZLTh5Avi2Zr8F+C428+O1MgInAdm53bC/6HbY6FAHbcb/d5CtEVuziQXb/MNuxHps3v23WHJevPApO6NW6UcYdYXozM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1568166418; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=AczcJRqBhC9vS3T7GY2+tfCrph33Cek/6CQ4pFa7LNE=; b=gvB35Y+PfUpQpsYZXdc/ra+xPk6rRLfx4yw4pcCisMu+hs7gPNPw7bEde4kfOAo+RC57W0wK2WUpoI69dus5ZY4cX7lXj8XG5+zorGs1E+uvEQbxc6aDHbuWM7ZvgNfZGSUJC9L84qxBCykLaTgvIpzi3cUJB6XAvr4u4CKlRGQ= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (209.51.188.17 [209.51.188.17]) by mx.zohomail.com with SMTPS id 1568166418039332.4802771672331; Tue, 10 Sep 2019 18:46:58 -0700 (PDT) Received: from localhost ([::1]:45916 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i7rih-000314-7h for importer@patchew.org; Tue, 10 Sep 2019 21:46:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53876) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i7rft-0001ge-BH for qemu-devel@nongnu.org; Tue, 10 Sep 2019 21:44:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1i7rfr-0000aV-MF for qemu-devel@nongnu.org; Tue, 10 Sep 2019 21:44:01 -0400 Received: from mail-qk1-x742.google.com ([2607:f8b0:4864:20::742]:44053) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1i7rfr-0000aO-Hf for qemu-devel@nongnu.org; Tue, 10 Sep 2019 21:43:59 -0400 Received: by mail-qk1-x742.google.com with SMTP id i78so19150355qke.11 for ; Tue, 10 Sep 2019 18:43:59 -0700 (PDT) Received: from localhost.localdomain (otwaon236nw-grc-01-64-229-69-35.dsl.bell.ca. [64.229.69.35]) by smtp.gmail.com with ESMTPSA id a14sm10074676qkg.59.2019.09.10.18.43.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Sep 2019 18:43:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=AczcJRqBhC9vS3T7GY2+tfCrph33Cek/6CQ4pFa7LNE=; b=AM2oLPzgBVP/F3VK5PreEaT1RH0Q5Iilv3uwuwiFzwRi+sI7mH0Y8Z3GD/YsDaOvtO jgHtuA+vYjXoBXXsoUZ7srJkR9/5CUYPizDX7I+x3rMF8qObvaNvr5900KxUrgPY4crs +PP4eB3AROkctmQAmsbRHI+biZkAT9s1Qpuc7SiWLUXDXcSNsMN7cyra3BBOF0jb/fNU RYdp2gSB2fy9pFOeuNchUsZY4142Nv0fRqBBykAyBkBNiYFrBvC3x2D5QaKr1gBBq5A/ ugNVYGUfC7CpM0ee5xHPoVTaV5P66KUBZZYSKag0ud/08/GsVAHfNxxtnHKv2eDQhbYY 4OHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=AczcJRqBhC9vS3T7GY2+tfCrph33Cek/6CQ4pFa7LNE=; b=b0WSdvXrwrEgqmlSkxHmsJ9JLpShxX+2WrXw7lmXkEX7a1wg5Lj5IZ1FBzGSCtNj/n cJgr0Yuug31kp4LlYPIcA+T6Byy9SLFPd8l4gMZeka8Hq/PRTBnj+7XyjwNr9OGniGbD IhmQ7S5NqT5+qwWmI/OcbWpMRgqbjq2tIGgvHTWHY1N4qKEZuH3BNpv/+9PEaD6/nWt8 Qy8JsxVTOmE4swSHd/6vwcMnwQUCxIRE5CI2hyLDBahO82qwaFfua2jODPJtY19/SH8C KuRsuTbZuvOg2t2MNDdRZ9gT1rGiT50KV8izLQ9DqyNUsUltLi5cLMqw8kRUBWS4kA9D 493A== X-Gm-Message-State: APjAAAVi/1Crjk8V7qHG0v4br72rZt6jqR1JgBA6uMmw3AmzaMTowBvB 6h5RJ1jDSnmhjgwN2VX6HNEohimdRIkbqw== X-Google-Smtp-Source: APXvYqwJSbjmkUzyszq1BNXFCqSMnAtiSAezFObJYfERwgpvX1Ip52P3lbrk0yqP1yAs7h3dn+N1zw== X-Received: by 2002:a05:620a:530:: with SMTP id h16mr33670016qkh.396.1568166238573; Tue, 10 Sep 2019 18:43:58 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 10 Sep 2019 21:43:52 -0400 Message-Id: <20190911014353.5926-3-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190911014353.5926-1-richard.henderson@linaro.org> References: <20190911014353.5926-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::742 Subject: [Qemu-devel] [PATCH 2/3] cputlb: Replace switches in load/store_helper with callback X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, mark.cave-ayland@ilande.co.uk, tony.nguyen@bt.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Add a function parameter to perform the actual load/store to ram. With optimization, this results in identical code. Signed-off-by: Richard Henderson Reviewed-by: Tony Nguyen --- accel/tcg/cputlb.c | 157 +++++++++++++++++++++++---------------------- 1 file changed, 81 insertions(+), 76 deletions(-) diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index 909f01ebcc..e6229d100a 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -1292,11 +1292,37 @@ static void *atomic_mmu_lookup(CPUArchState *env, t= arget_ulong addr, =20 typedef uint64_t FullLoadHelper(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr); +typedef uint64_t DirectLoadHelper(const void *); + +static inline uint64_t direct_ldub(const void *haddr) +{ + return *(uint8_t *)haddr; +} + +static inline uint64_t direct_lduw_be(const void *haddr) +{ + return lduw_be_p(haddr); +} + +static inline uint64_t direct_lduw_le(const void *haddr) +{ + return lduw_le_p(haddr); +} + +static inline uint64_t direct_ldul_be(const void *haddr) +{ + return (uint32_t)ldl_be_p(haddr); +} + +static inline uint64_t direct_ldul_le(const void *haddr) +{ + return (uint32_t)ldl_le_p(haddr); +} =20 static inline uint64_t ALWAYS_INLINE load_helper(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr, MemOp op, bool code_read, - FullLoadHelper *full_load) + FullLoadHelper *full_load, DirectLoadHelper *direct) { uintptr_t mmu_idx =3D get_mmuidx(oi); uintptr_t index =3D tlb_index(env, mmu_idx, addr); @@ -1385,33 +1411,7 @@ load_helper(CPUArchState *env, target_ulong addr, TC= GMemOpIdx oi, =20 do_aligned_access: haddr =3D (void *)((uintptr_t)addr + entry->addend); - switch (op) { - case MO_UB: - res =3D ldub_p(haddr); - break; - case MO_BEUW: - res =3D lduw_be_p(haddr); - break; - case MO_LEUW: - res =3D lduw_le_p(haddr); - break; - case MO_BEUL: - res =3D (uint32_t)ldl_be_p(haddr); - break; - case MO_LEUL: - res =3D (uint32_t)ldl_le_p(haddr); - break; - case MO_BEQ: - res =3D ldq_be_p(haddr); - break; - case MO_LEQ: - res =3D ldq_le_p(haddr); - break; - default: - g_assert_not_reached(); - } - - return res; + return direct(haddr); } =20 /* @@ -1427,7 +1427,8 @@ load_helper(CPUArchState *env, target_ulong addr, TCG= MemOpIdx oi, static uint64_t full_ldub_mmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { - return load_helper(env, addr, oi, retaddr, MO_UB, false, full_ldub_mmu= ); + return load_helper(env, addr, oi, retaddr, MO_UB, false, + full_ldub_mmu, direct_ldub); } =20 tcg_target_ulong helper_ret_ldub_mmu(CPUArchState *env, target_ulong addr, @@ -1440,7 +1441,7 @@ static uint64_t full_le_lduw_mmu(CPUArchState *env, t= arget_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEUW, false, - full_le_lduw_mmu); + full_le_lduw_mmu, direct_lduw_le); } =20 tcg_target_ulong helper_le_lduw_mmu(CPUArchState *env, target_ulong addr, @@ -1453,7 +1454,7 @@ static uint64_t full_be_lduw_mmu(CPUArchState *env, t= arget_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEUW, false, - full_be_lduw_mmu); + full_be_lduw_mmu, direct_lduw_be); } =20 tcg_target_ulong helper_be_lduw_mmu(CPUArchState *env, target_ulong addr, @@ -1466,7 +1467,7 @@ static uint64_t full_le_ldul_mmu(CPUArchState *env, t= arget_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEUL, false, - full_le_ldul_mmu); + full_le_ldul_mmu, direct_ldul_le); } =20 tcg_target_ulong helper_le_ldul_mmu(CPUArchState *env, target_ulong addr, @@ -1479,7 +1480,7 @@ static uint64_t full_be_ldul_mmu(CPUArchState *env, t= arget_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEUL, false, - full_be_ldul_mmu); + full_be_ldul_mmu, direct_ldul_be); } =20 tcg_target_ulong helper_be_ldul_mmu(CPUArchState *env, target_ulong addr, @@ -1492,14 +1493,14 @@ uint64_t helper_le_ldq_mmu(CPUArchState *env, targe= t_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEQ, false, - helper_le_ldq_mmu); + helper_le_ldq_mmu, ldq_le_p); } =20 uint64_t helper_be_ldq_mmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEQ, false, - helper_be_ldq_mmu); + helper_be_ldq_mmu, ldq_be_p); } =20 /* @@ -1542,9 +1543,37 @@ tcg_target_ulong helper_be_ldsl_mmu(CPUArchState *en= v, target_ulong addr, * Store Helpers */ =20 +typedef void DirectStoreHelper(void *, uint64_t); + +static inline void direct_stb(void *haddr, uint64_t val) +{ + *(uint8_t *)haddr =3D val; +} + +static inline void direct_stw_be(void *haddr, uint64_t val) +{ + stw_be_p(haddr, val); +} + +static inline void direct_stw_le(void *haddr, uint64_t val) +{ + stw_le_p(haddr, val); +} + +static inline void direct_stl_be(void *haddr, uint64_t val) +{ + stl_be_p(haddr, val); +} + +static inline void direct_stl_le(void *haddr, uint64_t val) +{ + stl_le_p(haddr, val); +} + static inline void ALWAYS_INLINE store_helper(CPUArchState *env, target_ulong addr, uint64_t val, - TCGMemOpIdx oi, uintptr_t retaddr, MemOp op) + TCGMemOpIdx oi, uintptr_t retaddr, MemOp op, + DirectStoreHelper *direct) { uintptr_t mmu_idx =3D get_mmuidx(oi); uintptr_t index =3D tlb_index(env, mmu_idx, addr); @@ -1669,74 +1698,49 @@ store_helper(CPUArchState *env, target_ulong addr, = uint64_t val, =20 do_aligned_access: haddr =3D (void *)((uintptr_t)addr + entry->addend); - switch (op) { - case MO_UB: - stb_p(haddr, val); - break; - case MO_BEUW: - stw_be_p(haddr, val); - break; - case MO_LEUW: - stw_le_p(haddr, val); - break; - case MO_BEUL: - stl_be_p(haddr, val); - break; - case MO_LEUL: - stl_le_p(haddr, val); - break; - case MO_BEQ: - stq_be_p(haddr, val); - break; - case MO_LEQ: - stq_le_p(haddr, val); - break; - default: - g_assert_not_reached(); - break; - } + direct(haddr, val); } =20 void helper_ret_stb_mmu(CPUArchState *env, target_ulong addr, uint8_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_UB); + store_helper(env, addr, val, oi, retaddr, MO_UB, direct_stb); } =20 void helper_le_stw_mmu(CPUArchState *env, target_ulong addr, uint16_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_LEUW); + store_helper(env, addr, val, oi, retaddr, MO_LEUW, direct_stw_le); } =20 void helper_be_stw_mmu(CPUArchState *env, target_ulong addr, uint16_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_BEUW); + store_helper(env, addr, val, oi, retaddr, MO_BEUW, direct_stw_be); } =20 void helper_le_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_LEUL); + store_helper(env, addr, val, oi, retaddr, MO_LEUL, direct_stl_le); } =20 void helper_be_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_BEUL); + store_helper(env, addr, val, oi, retaddr, MO_BEUL, direct_stl_be); } =20 void helper_le_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_LEQ); + store_helper(env, addr, val, oi, retaddr, MO_LEQ, stq_le_p); } =20 void helper_be_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_BEQ); + store_helper(env, addr, val, oi, retaddr, MO_BEQ, stq_be_p); } =20 /* First set of helpers allows passing in of OI and RETADDR. This makes @@ -1801,7 +1805,8 @@ void helper_be_stq_mmu(CPUArchState *env, target_ulon= g addr, uint64_t val, static uint64_t full_ldub_cmmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { - return load_helper(env, addr, oi, retaddr, MO_8, true, full_ldub_cmmu); + return load_helper(env, addr, oi, retaddr, MO_8, true, + full_ldub_cmmu, direct_ldub); } =20 uint8_t helper_ret_ldb_cmmu(CPUArchState *env, target_ulong addr, @@ -1814,7 +1819,7 @@ static uint64_t full_le_lduw_cmmu(CPUArchState *env, = target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEUW, true, - full_le_lduw_cmmu); + full_le_lduw_cmmu, direct_lduw_le); } =20 uint16_t helper_le_ldw_cmmu(CPUArchState *env, target_ulong addr, @@ -1827,7 +1832,7 @@ static uint64_t full_be_lduw_cmmu(CPUArchState *env, = target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEUW, true, - full_be_lduw_cmmu); + full_be_lduw_cmmu, direct_lduw_be); } =20 uint16_t helper_be_ldw_cmmu(CPUArchState *env, target_ulong addr, @@ -1840,7 +1845,7 @@ static uint64_t full_le_ldul_cmmu(CPUArchState *env, = target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEUL, true, - full_le_ldul_cmmu); + full_le_ldul_cmmu, direct_ldul_le); } =20 uint32_t helper_le_ldl_cmmu(CPUArchState *env, target_ulong addr, @@ -1853,7 +1858,7 @@ static uint64_t full_be_ldul_cmmu(CPUArchState *env, = target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEUL, true, - full_be_ldul_cmmu); + full_be_ldul_cmmu, direct_ldul_be); } =20 uint32_t helper_be_ldl_cmmu(CPUArchState *env, target_ulong addr, @@ -1866,12 +1871,12 @@ uint64_t helper_le_ldq_cmmu(CPUArchState *env, targ= et_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEQ, true, - helper_le_ldq_cmmu); + helper_le_ldq_cmmu, ldq_le_p); } =20 uint64_t helper_be_ldq_cmmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEQ, true, - helper_be_ldq_cmmu); + helper_be_ldq_cmmu, ldq_be_p); } --=20 2.17.1 From nobody Thu Apr 25 12:44:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1568166506; cv=none; d=zoho.com; s=zohoarc; b=AU72VpMsdL1RDfX+kM2yJ3ONoU/winewBXCIgwnkV6r1Yby2bY56RalUWHirBKe0sWpnAgKCuaL+Yc//x0MGzm5tHnRCqZE/9ffv42jpv8Y17Tp54Jjkh9l+7Ae+APYKDaWLvcV5eQWQ5lQ1HyxnWBYDVVSlV7QSsPzwQY2cIqA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1568166506; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=qkRCAeXLj1zdLXAh0jGTAmz2ec7MpixYWyKmhPy98CA=; b=lSwdT3aVW/89yf1N5oG3+RYi+dc+F6lhqtK/whnTNkz66NVM48DGtoph8cg1tuVXwK6SDRWtXemt2nrzUEmLGEz8Q7+EMC80rI/6UsFww5K8CdtY0zBK+1SRBZdkUaiIkVOTb/bPZROz2X33dJoz22Be5TfKLl324OW2xA3rGEk= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1568166506235713.3391312569704; Tue, 10 Sep 2019 18:48:26 -0700 (PDT) Received: from localhost ([::1]:45948 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i7rk9-0005JM-3Z for importer@patchew.org; Tue, 10 Sep 2019 21:48:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53893) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i7rfu-0001hy-SW for qemu-devel@nongnu.org; Tue, 10 Sep 2019 21:44:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1i7rft-0000an-0q for qemu-devel@nongnu.org; Tue, 10 Sep 2019 21:44:02 -0400 Received: from mail-qt1-x82a.google.com ([2607:f8b0:4864:20::82a]:33845) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1i7rfs-0000ah-Sz for qemu-devel@nongnu.org; Tue, 10 Sep 2019 21:44:00 -0400 Received: by mail-qt1-x82a.google.com with SMTP id j1so10539268qth.1 for ; Tue, 10 Sep 2019 18:44:00 -0700 (PDT) Received: from localhost.localdomain (otwaon236nw-grc-01-64-229-69-35.dsl.bell.ca. [64.229.69.35]) by smtp.gmail.com with ESMTPSA id a14sm10074676qkg.59.2019.09.10.18.43.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Sep 2019 18:43:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=qkRCAeXLj1zdLXAh0jGTAmz2ec7MpixYWyKmhPy98CA=; b=S+B1gpXJcX9THwqSInhb8HygS2+r9ISU9zOMVNsCe0AZXjxun8igYsL2jyI2T++QAX sP7+HS0vDEcNotHcrfjkBqYnkVByDtEkipngiRjuvZFu3rn5x6Vh2qyABnC/Sb5N4rmz 3jaj3Tgm9FohZFwAjLvfFPZjVeSGEKOLUcuNAimzMatwJWpoO0ZmDZnPTCIy+dx19wKt bwYm7gz5HdQNvo5QH4TkVRFm2wu0L/hGqP9SptEzTyWUIQymsm6rAqBKVnRyPUgd4VJX OJnrapiZQbBlctV1PZfByBEWeZqk7ISBtlcMV0pEGegDwBmShEk7PysDodDgETxyXjo1 8hzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=qkRCAeXLj1zdLXAh0jGTAmz2ec7MpixYWyKmhPy98CA=; b=lj1RgD9C2gPBOrzQuzhDSXewP6tkK/XbpHfl7GqXvQbC8fxn0IncXu8HEPIDwRDhp5 sriDnHvoktbsJ68k6fBk0QUASZKXu6u7RmjN2O25f2hyAI5pkQhNkoa5bZcF2QTHRLn9 mM1PCAH4+eveeYv9DjSc89qgTOlYDN3mN07aWPo4OwHjpNCH1J2YYx4yTCHdADCOyU01 VoGgY8zj0QZcGQJ7EV/4mpFWPxc/6LFVzmIz5T+G5LeIiyZkm94eokiC/GVXjhTsFfuB 7oHIICWIXP6+wHJ47IfzzIVX0bumHu+zRVvzvdQhjPGb64UU/+Agq+u/h6uCn6Cu2u0a hU5Q== X-Gm-Message-State: APjAAAWP2/qbrcfaWi0Amv6Ttsc03/RVbVNsxySB57g6towGMremNLoR BauOw6vtyas8yFywPltuQ1FVnhgfPrExUA== X-Google-Smtp-Source: APXvYqyGJ0r6MpZ+nXRkEB3NXu1nHf/R7PfQxYj6ZAPXIzvW4bLEKMQX/MpmWK5hPs4gIRzfW42ooA== X-Received: by 2002:a0c:9352:: with SMTP id e18mr19909925qve.33.1568166239889; Tue, 10 Sep 2019 18:43:59 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 10 Sep 2019 21:43:53 -0400 Message-Id: <20190911014353.5926-4-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190911014353.5926-1-richard.henderson@linaro.org> References: <20190911014353.5926-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::82a Subject: [Qemu-devel] [PATCH 3/3] cputlb: Introduce TLB_BSWAP X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, mark.cave-ayland@ilande.co.uk, tony.nguyen@bt.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Handle bswap on ram directly in load/store_helper. This fixes a bug with the previous implementation in that one cannot use the I/O path for RAM. Fixes: a26fc6f5152b47f1 Signed-off-by: Richard Henderson --- include/exec/cpu-all.h | 2 + accel/tcg/cputlb.c | 118 ++++++++++++++++++++--------------------- 2 files changed, 59 insertions(+), 61 deletions(-) diff --git a/include/exec/cpu-all.h b/include/exec/cpu-all.h index d2d443c4f9..3928edab9a 100644 --- a/include/exec/cpu-all.h +++ b/include/exec/cpu-all.h @@ -331,6 +331,8 @@ CPUArchState *cpu_copy(CPUArchState *env); #define TLB_MMIO (1 << (TARGET_PAGE_BITS - 3)) /* Set if TLB entry contains a watchpoint. */ #define TLB_WATCHPOINT (1 << (TARGET_PAGE_BITS - 4)) +/* Set if TLB entry requires byte swap. */ +#define TLB_BSWAP (1 << (TARGET_PAGE_BITS - 5)) =20 /* Use this mask to check interception with an alignment mask * in a TCG backend. diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index e6229d100a..eace6c82e3 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -737,8 +737,7 @@ void tlb_set_page_with_attrs(CPUState *cpu, target_ulon= g vaddr, address |=3D TLB_INVALID_MASK; } if (attrs.byte_swap) { - /* Force the access through the I/O slow path. */ - address |=3D TLB_MMIO; + address |=3D TLB_BSWAP; } if (!memory_region_is_ram(section->mr) && !memory_region_is_romd(section->mr)) { @@ -901,10 +900,6 @@ static uint64_t io_readx(CPUArchState *env, CPUIOTLBEn= try *iotlbentry, bool locked =3D false; MemTxResult r; =20 - if (iotlbentry->attrs.byte_swap) { - op ^=3D MO_BSWAP; - } - section =3D iotlb_to_section(cpu, iotlbentry->addr, iotlbentry->attrs); mr =3D section->mr; mr_offset =3D (iotlbentry->addr & TARGET_PAGE_MASK) + addr; @@ -947,10 +942,6 @@ static void io_writex(CPUArchState *env, CPUIOTLBEntry= *iotlbentry, bool locked =3D false; MemTxResult r; =20 - if (iotlbentry->attrs.byte_swap) { - op ^=3D MO_BSWAP; - } - section =3D iotlb_to_section(cpu, iotlbentry->addr, iotlbentry->attrs); mr =3D section->mr; mr_offset =3D (iotlbentry->addr & TARGET_PAGE_MASK) + addr; @@ -1322,7 +1313,8 @@ static inline uint64_t direct_ldul_le(const void *had= dr) static inline uint64_t ALWAYS_INLINE load_helper(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr, MemOp op, bool code_read, - FullLoadHelper *full_load, DirectLoadHelper *direct) + FullLoadHelper *full_load, DirectLoadHelper *direct, + DirectLoadHelper *direct_swap) { uintptr_t mmu_idx =3D get_mmuidx(oi); uintptr_t index =3D tlb_index(env, mmu_idx, addr); @@ -1372,26 +1364,27 @@ load_helper(CPUArchState *env, target_ulong addr, T= CGMemOpIdx oi, /* On watchpoint hit, this will longjmp out. */ cpu_check_watchpoint(env_cpu(env), addr, size, iotlbentry->attrs, BP_MEM_READ, retaddr); - - /* The backing page may or may not require I/O. */ - tlb_addr &=3D ~TLB_WATCHPOINT; - if ((tlb_addr & ~TARGET_PAGE_MASK) =3D=3D 0) { - goto do_aligned_access; - } } =20 /* Handle I/O access. */ - return io_readx(env, iotlbentry, mmu_idx, addr, - retaddr, access_type, op); - } + if (likely(tlb_addr & TLB_MMIO)) { + return io_readx(env, iotlbentry, mmu_idx, addr, + retaddr, access_type, + op ^ (tlb_addr & TLB_BSWAP ? MO_BSWAP : 0)); + } =20 - /* Handle slow unaligned access (it spans two pages or IO). */ - if (size > 1 - && unlikely((addr & ~TARGET_PAGE_MASK) + size - 1 - >=3D TARGET_PAGE_SIZE)) { + if (unlikely(tlb_addr & TLB_BSWAP)) { + haddr =3D (void *)((uintptr_t)addr + entry->addend); + return direct_swap(haddr); + } + } else if (size > 1 + && unlikely((addr & ~TARGET_PAGE_MASK) + size - 1 + >=3D TARGET_PAGE_SIZE)) { + /* Handle slow unaligned access (it spans two pages or IO). */ target_ulong addr1, addr2; uint64_t r1, r2; unsigned shift; + do_unaligned_access: addr1 =3D addr & ~((target_ulong)size - 1); addr2 =3D addr1 + size; @@ -1409,7 +1402,6 @@ load_helper(CPUArchState *env, target_ulong addr, TCG= MemOpIdx oi, return res & MAKE_64BIT_MASK(0, size * 8); } =20 - do_aligned_access: haddr =3D (void *)((uintptr_t)addr + entry->addend); return direct(haddr); } @@ -1428,7 +1420,7 @@ static uint64_t full_ldub_mmu(CPUArchState *env, targ= et_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_UB, false, - full_ldub_mmu, direct_ldub); + full_ldub_mmu, direct_ldub, direct_ldub); } =20 tcg_target_ulong helper_ret_ldub_mmu(CPUArchState *env, target_ulong addr, @@ -1441,7 +1433,7 @@ static uint64_t full_le_lduw_mmu(CPUArchState *env, t= arget_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEUW, false, - full_le_lduw_mmu, direct_lduw_le); + full_le_lduw_mmu, direct_lduw_le, direct_lduw_be); } =20 tcg_target_ulong helper_le_lduw_mmu(CPUArchState *env, target_ulong addr, @@ -1454,7 +1446,7 @@ static uint64_t full_be_lduw_mmu(CPUArchState *env, t= arget_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEUW, false, - full_be_lduw_mmu, direct_lduw_be); + full_be_lduw_mmu, direct_lduw_be, direct_lduw_le); } =20 tcg_target_ulong helper_be_lduw_mmu(CPUArchState *env, target_ulong addr, @@ -1467,7 +1459,7 @@ static uint64_t full_le_ldul_mmu(CPUArchState *env, t= arget_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEUL, false, - full_le_ldul_mmu, direct_ldul_le); + full_le_ldul_mmu, direct_ldul_le, direct_ldul_be); } =20 tcg_target_ulong helper_le_ldul_mmu(CPUArchState *env, target_ulong addr, @@ -1480,7 +1472,7 @@ static uint64_t full_be_ldul_mmu(CPUArchState *env, t= arget_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEUL, false, - full_be_ldul_mmu, direct_ldul_be); + full_be_ldul_mmu, direct_ldul_be, direct_ldul_le); } =20 tcg_target_ulong helper_be_ldul_mmu(CPUArchState *env, target_ulong addr, @@ -1493,14 +1485,14 @@ uint64_t helper_le_ldq_mmu(CPUArchState *env, targe= t_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEQ, false, - helper_le_ldq_mmu, ldq_le_p); + helper_le_ldq_mmu, ldq_le_p, ldq_be_p); } =20 uint64_t helper_be_ldq_mmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEQ, false, - helper_be_ldq_mmu, ldq_be_p); + helper_be_ldq_mmu, ldq_be_p, ldq_le_p); } =20 /* @@ -1573,7 +1565,7 @@ static inline void direct_stl_le(void *haddr, uint64_= t val) static inline void ALWAYS_INLINE store_helper(CPUArchState *env, target_ulong addr, uint64_t val, TCGMemOpIdx oi, uintptr_t retaddr, MemOp op, - DirectStoreHelper *direct) + DirectStoreHelper *direct, DirectStoreHelper *direct_swap) { uintptr_t mmu_idx =3D get_mmuidx(oi); uintptr_t index =3D tlb_index(env, mmu_idx, addr); @@ -1618,23 +1610,24 @@ store_helper(CPUArchState *env, target_ulong addr, = uint64_t val, /* On watchpoint hit, this will longjmp out. */ cpu_check_watchpoint(env_cpu(env), addr, size, iotlbentry->attrs, BP_MEM_WRITE, retaddr); - - /* The backing page may or may not require I/O. */ - tlb_addr &=3D ~TLB_WATCHPOINT; - if ((tlb_addr & ~TARGET_PAGE_MASK) =3D=3D 0) { - goto do_aligned_access; - } } =20 /* Handle I/O access. */ - io_writex(env, iotlbentry, mmu_idx, val, addr, retaddr, op); - return; - } + if (likely(tlb_addr & TLB_MMIO)) { + io_writex(env, iotlbentry, mmu_idx, val, addr, retaddr, + op ^ (tlb_addr & TLB_BSWAP ? MO_BSWAP : 0)); + return; + } =20 - /* Handle slow unaligned access (it spans two pages or IO). */ - if (size > 1 - && unlikely((addr & ~TARGET_PAGE_MASK) + size - 1 - >=3D TARGET_PAGE_SIZE)) { + if (unlikely(tlb_addr & TLB_BSWAP)) { + haddr =3D (void *)((uintptr_t)addr + entry->addend); + direct_swap(haddr, val); + return; + } + } else if (size > 1 + && unlikely((addr & ~TARGET_PAGE_MASK) + size - 1 + >=3D TARGET_PAGE_SIZE)) { + /* Handle slow unaligned access (it spans two pages or IO). */ int i; uintptr_t index2; CPUTLBEntry *entry2; @@ -1696,7 +1689,6 @@ store_helper(CPUArchState *env, target_ulong addr, ui= nt64_t val, return; } =20 - do_aligned_access: haddr =3D (void *)((uintptr_t)addr + entry->addend); direct(haddr, val); } @@ -1704,43 +1696,47 @@ store_helper(CPUArchState *env, target_ulong addr, = uint64_t val, void helper_ret_stb_mmu(CPUArchState *env, target_ulong addr, uint8_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_UB, direct_stb); + store_helper(env, addr, val, oi, retaddr, MO_UB, direct_stb, direct_st= b); } =20 void helper_le_stw_mmu(CPUArchState *env, target_ulong addr, uint16_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_LEUW, direct_stw_le); + store_helper(env, addr, val, oi, retaddr, MO_LEUW, + direct_stw_le, direct_stw_be); } =20 void helper_be_stw_mmu(CPUArchState *env, target_ulong addr, uint16_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_BEUW, direct_stw_be); + store_helper(env, addr, val, oi, retaddr, MO_BEUW, + direct_stw_be, direct_stw_le); } =20 void helper_le_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_LEUL, direct_stl_le); + store_helper(env, addr, val, oi, retaddr, MO_LEUL, + direct_stl_le, direct_stl_be); } =20 void helper_be_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_BEUL, direct_stl_be); + store_helper(env, addr, val, oi, retaddr, MO_BEUL, + direct_stl_be, direct_stl_le); } =20 void helper_le_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_LEQ, stq_le_p); + store_helper(env, addr, val, oi, retaddr, MO_LEQ, stq_le_p, stq_be_p); } =20 void helper_be_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_BEQ, stq_be_p); + store_helper(env, addr, val, oi, retaddr, MO_BEQ, stq_be_p, stq_le_p); } =20 /* First set of helpers allows passing in of OI and RETADDR. This makes @@ -1806,7 +1802,7 @@ static uint64_t full_ldub_cmmu(CPUArchState *env, tar= get_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_8, true, - full_ldub_cmmu, direct_ldub); + full_ldub_cmmu, direct_ldub, direct_ldub); } =20 uint8_t helper_ret_ldb_cmmu(CPUArchState *env, target_ulong addr, @@ -1819,7 +1815,7 @@ static uint64_t full_le_lduw_cmmu(CPUArchState *env, = target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEUW, true, - full_le_lduw_cmmu, direct_lduw_le); + full_le_lduw_cmmu, direct_lduw_le, direct_lduw_be); } =20 uint16_t helper_le_ldw_cmmu(CPUArchState *env, target_ulong addr, @@ -1832,7 +1828,7 @@ static uint64_t full_be_lduw_cmmu(CPUArchState *env, = target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEUW, true, - full_be_lduw_cmmu, direct_lduw_be); + full_be_lduw_cmmu, direct_lduw_be, direct_lduw_le); } =20 uint16_t helper_be_ldw_cmmu(CPUArchState *env, target_ulong addr, @@ -1845,7 +1841,7 @@ static uint64_t full_le_ldul_cmmu(CPUArchState *env, = target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEUL, true, - full_le_ldul_cmmu, direct_ldul_le); + full_le_ldul_cmmu, direct_ldul_le, direct_ldul_be); } =20 uint32_t helper_le_ldl_cmmu(CPUArchState *env, target_ulong addr, @@ -1858,7 +1854,7 @@ static uint64_t full_be_ldul_cmmu(CPUArchState *env, = target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEUL, true, - full_be_ldul_cmmu, direct_ldul_be); + full_be_ldul_cmmu, direct_ldul_be, direct_ldul_le); } =20 uint32_t helper_be_ldl_cmmu(CPUArchState *env, target_ulong addr, @@ -1871,12 +1867,12 @@ uint64_t helper_le_ldq_cmmu(CPUArchState *env, targ= et_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEQ, true, - helper_le_ldq_cmmu, ldq_le_p); + helper_le_ldq_cmmu, ldq_le_p, ldq_be_p); } =20 uint64_t helper_be_ldq_cmmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEQ, true, - helper_be_ldq_cmmu, ldq_be_p); + helper_be_ldq_cmmu, ldq_be_p, ldq_le_p); } --=20 2.17.1