From nobody Mon Feb 9 18:46:37 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1571839179; cv=none; d=zoho.com; s=zohoarc; b=mIrPcQYERz46c4Pqs9HwNK562S3D3DLn0Hu55W4BJ6DAw1IsUu7DKu6NZXQN977dYeVMqYId8J8blkABDq2weXkEKIgb9TRpfD/QfcASw/2jDVzYY1h22vOtbKEjIlADkxyd24EHTsaTjC8Ahga1DgXbZmKsWD7AwzvvcsR8ELY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1571839179; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=sO+GuGSHENU6NdSBDwSuY/MWi1bieT2h6nm3/Hcnz14=; b=GTyZK6Gz0U0I2E6iR5Dempr2ikM0/63o4FlCXOpbJF2tr+/gLpQYKbpI1yhLx/Y60Aubr2uJI1KixuF1LR1Hc6oU/lPrCd/6K6L9d8G9adL0ErdNjbLNUBPgeKeGL1FOP9NY2PrzITWglZTm+PAuZ0BvxaPfzkyTG0W5WYEKOBk= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1571839179239771.5858011872488; Wed, 23 Oct 2019 06:59:39 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iNH9o-0004nH-FV; Wed, 23 Oct 2019 13:58:36 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iNH9m-0004mJ-Np for xen-devel@lists.xenproject.org; Wed, 23 Oct 2019 13:58:34 +0000 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 2ba9895a-f59d-11e9-947f-12813bfff9fa; Wed, 23 Oct 2019 13:58:20 +0000 (UTC) X-Inumbo-ID: 2ba9895a-f59d-11e9-947f-12813bfff9fa DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1571839100; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=HKJkSbXcmX/kEnPedYeC0yfTtOaX516ggrC6KOT5BGE=; b=NGRw5Emq08+0nPDZEZ9FR2me7wvU56xvzN4NBxexyJWBpy1wU4+JA6lG SwQXp1V3j3jyvkkAWkzrC4UW86I5Wrd85jZCZcN+hlpJSD9P8mikT2CEC vCLmdg+P2ci9drLOBtdIK51Tx548bMRmfS+SgWU7wlHM7lf7gcdauFaC4 Q=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=andrew.cooper3@citrix.com; spf=Pass smtp.mailfrom=Andrew.Cooper3@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of andrew.cooper3@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="andrew.cooper3@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa2.hc3370-68.iphmx.com: domain of Andrew.Cooper3@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="Andrew.Cooper3@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: CQlkvqemll8CyTNLaBa5YY25hrha+pcVxYWr7FuzBrS4yROwbYWSrtay5l2JX2yUKRK/qGs0CD K4xv2sdWHJasN2a5elDBf8o9HPAbZNE0ut46r8oA+hi9bPv7jiCCWbXzrNvlzPxDzPjVCPKsvm nSCVxTQJXfvVqsIBNewM0vY44EMhg7TQ08BR1nHV97ttspR+E+VAED45ysRGd7w26ntj1R27Rg FP3n+z+A23lTt+Db1QkKQrpYxR+6F+jbuKOTknWRyIDT+RAU2dNE3Atnk0/ZoAnXn87yd8URA8 rvI= X-SBRS: 2.7 X-MesageID: 7322263 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.68,221,1569297600"; d="scan'208";a="7322263" From: Andrew Cooper To: Xen-devel Date: Wed, 23 Oct 2019 14:58:12 +0100 Message-ID: <20191023135812.21348-8-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20191023135812.21348-1-andrew.cooper3@citrix.com> References: <20191023135812.21348-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v3 7/7] x86/nospec: Optimise array_index_mask_nospec() for power-of-2 arrays X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Juergen Gross , Andrew Cooper , Wei Liu , Jan Beulich , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) When the compiler can determine that an array bound is a power of two, the array index can be bounded even under speculation with a single and instruction. Respecify array_index_mask_nospec() to allow for masks other than ~0 and 0, and introduce an IS_POWER_OF_2() helper. Signed-off-by: Andrew Cooper --- CC: Jan Beulich CC: Wei Liu CC: Roger Pau Monn=C3=A9 CC: Juergen Gross This optimisation is not safe on ARM, because some CPUs do data value speculation, which is why the CSDB barrer was introduced. --- xen/include/asm-x86/nospec.h | 25 +++++++++++++++++++------ xen/include/xen/config.h | 1 + xen/include/xen/nospec.h | 3 ++- 3 files changed, 22 insertions(+), 7 deletions(-) diff --git a/xen/include/asm-x86/nospec.h b/xen/include/asm-x86/nospec.h index 0039cd2713..4f36069eac 100644 --- a/xen/include/asm-x86/nospec.h +++ b/xen/include/asm-x86/nospec.h @@ -7,13 +7,20 @@ #include =20 /** - * array_index_mask_nospec() - generate a mask that is ~0UL when the - * bounds check succeeds and 0 otherwise + * array_index_mask_nospec() - generate a mask to bound an array index + * which is safe even under adverse speculation. * @index: array element index * @size: number of elements in array * - * Returns: + * In general, returns: * 0 - (index < size) + * + * This yeild ~0UL in within-bounds case, and 0 in the out-of-bounds + * case. + * + * When the compiler can determine that the array is a power of two, a + * lower overhead option is to mask the index with a single and + * instruction. */ #define array_index_mask_nospec array_index_mask_nospec static inline unsigned long array_index_mask_nospec(unsigned long index, @@ -21,9 +28,15 @@ static inline unsigned long array_index_mask_nospec(unsi= gned long index, { unsigned long mask; =20 - asm volatile ( "cmp %[size], %[index]; sbb %[mask], %[mask];" - : [mask] "=3Dr" (mask) - : [size] "g" (size), [index] "r" (index) ); + if ( __builtin_constant_p(size) && IS_POWER_OF_2(size) ) + { + mask =3D size - 1; + OPTIMIZER_HIDE_VAR(mask); + } + else + asm volatile ( "cmp %[size], %[index]; sbb %[mask], %[mask];" + : [mask] "=3Dr" (mask) + : [size] "g" (size), [index] "r" (index) ); =20 return mask; } diff --git a/xen/include/xen/config.h b/xen/include/xen/config.h index a106380a23..21c763617c 100644 --- a/xen/include/xen/config.h +++ b/xen/include/xen/config.h @@ -75,6 +75,7 @@ #define GB(_gb) (_AC(_gb, ULL) << 30) =20 #define IS_ALIGNED(val, align) (((val) & ((align) - 1)) =3D=3D 0) +#define IS_POWER_OF_2(val) ((val) && IS_ALIGNED(val, val)) =20 #define __STR(...) #__VA_ARGS__ #define STR(...) __STR(__VA_ARGS__) diff --git a/xen/include/xen/nospec.h b/xen/include/xen/nospec.h index 7578210f16..cfc31f11b7 100644 --- a/xen/include/xen/nospec.h +++ b/xen/include/xen/nospec.h @@ -12,7 +12,8 @@ #include =20 /** - * array_index_mask_nospec() - generate a ~0 mask when index < size, 0 oth= erwise + * array_index_mask_nospec() - generate a mask to bound an array index + * which is safe even under adverse speculation. * @index: array element index * @size: number of elements in array * --=20 2.11.0 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel