From nobody Tue Feb 10 04:17:18 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1605878712; cv=none; d=zohomail.com; s=zohoarc; b=EGAfRqAU4mL7H/FaWhs80Lt/Czi9LoCUn1s2C4asRJIlXaUejfk+3AcrrYACXZ058VlG19a9UePx18GNgNlBehi2+N6RCghN+PxZXLWQZv7e2FfPmDvf2XjhAvyhQjBwD1zlilAWtdELJK4kE/TCZHcUcO5hXRXt4iscc1ZAuU4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1605878712; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=q4puytQMj2zWKl5eZW6uhYT5yUYlYoMz5vBb3O91yY0=; b=M56q+i/BFxZWGGkkPS549K8y5VSFZrR1BJh4PTLQkKeCyO8RPQgtKrBG3td9f5y2jnznktkDANZgkTZXcwMhT/A4qP363de6CLFyCkt3BMJTlKUXeCeAweS0twkDrtK1JhGvB9dLlZroxGG5woK5nLWKcfA9cBUl6b0UmS1hFWQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1605878712555804.018432918309; Fri, 20 Nov 2020 05:25:12 -0800 (PST) Received: from list by lists.xenproject.org with outflank-mailman.32186.63156 (Exim 4.92) (envelope-from ) id 1kg6P9-0001ik-Uo; Fri, 20 Nov 2020 13:24:47 +0000 Received: by outflank-mailman (output) from mailman id 32186.63156; Fri, 20 Nov 2020 13:24:47 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kg6P9-0001ib-Q8; Fri, 20 Nov 2020 13:24:47 +0000 Received: by outflank-mailman (input) for mailman id 32186; Fri, 20 Nov 2020 13:24:47 +0000 Received: from mail.xenproject.org ([104.130.215.37]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kg6P9-0001i2-00 for xen-devel@lists.xenproject.org; Fri, 20 Nov 2020 13:24:47 +0000 Received: from xenbits.xenproject.org ([104.239.192.120]) by mail.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kg6P8-0007Cw-Mn; Fri, 20 Nov 2020 13:24:46 +0000 Received: from host109-146-187-185.range109-146.btcentralplus.com ([109.146.187.185] helo=u2f063a87eabd5f.home) by xenbits.xenproject.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kg6P8-00028m-EY; Fri, 20 Nov 2020 13:24:46 +0000 Received: from mail.xenproject.org ([104.130.215.37]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kg6P9-0001i2-00 for xen-devel@lists.xenproject.org; Fri, 20 Nov 2020 13:24:47 +0000 Received: from xenbits.xenproject.org ([104.239.192.120]) by mail.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kg6P8-0007Cw-Mn; Fri, 20 Nov 2020 13:24:46 +0000 Received: from host109-146-187-185.range109-146.btcentralplus.com ([109.146.187.185] helo=u2f063a87eabd5f.home) by xenbits.xenproject.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kg6P8-00028m-EY; Fri, 20 Nov 2020 13:24:46 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=xen.org; s=20200302mail; h=Content-Transfer-Encoding:MIME-Version:References: In-Reply-To:Message-Id:Date:Subject:Cc:To:From; bh=q4puytQMj2zWKl5eZW6uhYT5yUYlYoMz5vBb3O91yY0=; b=AGCVfjdLJ32ta+FrdVF/jsfQ80 n0cbiXX/1ANWP+O9n+W9M1JP/q98cHvBiMtMJ9o4+FrxVGJsDcQVxeErSpRKZl9Flf8U7yk+rnFST EYnJ4YvsJmuQ1BRj5EZe/UZ1snVMXkHePS/Y3J0wgUo1W1WiLyDAz31h5kAOtfsvAGqc=; From: Paul Durrant To: xen-devel@lists.xenproject.org Cc: Paul Durrant , Jan Beulich , Julien Grall , Wei Liu , Andrew Cooper , George Dunlap , Ian Jackson , Stefano Stabellini Subject: [PATCH v10 2/7] common/grant_table: batch flush I/O TLB Date: Fri, 20 Nov 2020 13:24:35 +0000 Message-Id: <20201120132440.1141-3-paul@xen.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20201120132440.1141-1-paul@xen.org> References: <20201120132440.1141-1-paul@xen.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @xen.org) Content-Type: text/plain; charset="utf-8" From: Paul Durrant This patch avoids calling iommu_iotlb_flush() for each individual GNTTABOP = and instead calls iommu_iotlb_flush_all() at the end of a batch. This should me= an non-singleton map/unmap operations perform better. NOTE: A batch is the number of operations done before a pre-emption check a= nd, in the case of unmap, a TLB flush. Suggested-by: Jan Beulich Signed-off-by: Paul Durrant Reviewed-by: Jan Beulich Acked-by: Julien Grall Reviewed-by: Wei Liu --- Cc: Andrew Cooper Cc: George Dunlap Cc: Ian Jackson Cc: Stefano Stabellini v6: - Fix spelling of 'preemption' - Drop unneeded 'currd' stack variable v5: - Add batching to gnttab_map_grant_ref() to handle flushing before pre- emption check - Maintain per-op flushing in the case of singletons v3: - New in v3 --- xen/common/grant_table.c | 199 ++++++++++++++++++++++++++------------- 1 file changed, 133 insertions(+), 66 deletions(-) diff --git a/xen/common/grant_table.c b/xen/common/grant_table.c index beb6b2d40d68..1e3d7a2d33cb 100644 --- a/xen/common/grant_table.c +++ b/xen/common/grant_table.c @@ -241,7 +241,13 @@ struct gnttab_unmap_common { grant_ref_t ref; }; =20 -/* Number of unmap operations that are done between each tlb flush */ +/* Number of map operations that are done between each preemption check */ +#define GNTTAB_MAP_BATCH_SIZE 32 + +/* + * Number of unmap operations that are done between each tlb flush and + * preemption check. + */ #define GNTTAB_UNMAP_BATCH_SIZE 32 =20 =20 @@ -979,7 +985,7 @@ static unsigned int mapkind( =20 static void map_grant_ref( - struct gnttab_map_grant_ref *op) + struct gnttab_map_grant_ref *op, bool do_flush, unsigned int *flush_fl= ags) { struct domain *ld, *rd, *owner =3D NULL; struct grant_table *lgt, *rgt; @@ -1229,12 +1235,11 @@ map_grant_ref( if ( kind ) { dfn_t dfn =3D _dfn(mfn_x(mfn)); - unsigned int flush_flags =3D 0; int err; =20 - err =3D iommu_map(ld, dfn, mfn, 1, kind, &flush_flags); - if ( !err ) - err =3D iommu_iotlb_flush(ld, dfn, 1, flush_flags); + err =3D iommu_map(ld, dfn, mfn, 1, kind, flush_flags); + if ( do_flush && !err ) + err =3D iommu_iotlb_flush(ld, dfn, 1, *flush_flags); if ( err ) { double_gt_unlock(lgt, rgt); @@ -1319,29 +1324,59 @@ static long gnttab_map_grant_ref( XEN_GUEST_HANDLE_PARAM(gnttab_map_grant_ref_t) uop, unsigned int count) { - int i; - struct gnttab_map_grant_ref op; + unsigned int done =3D 0; + int rc =3D 0; =20 - for ( i =3D 0; i < count; i++ ) + while ( count ) { - if ( i && hypercall_preempt_check() ) - return i; + unsigned int i, c =3D min_t(unsigned int, count, GNTTAB_MAP_BATCH_= SIZE); + unsigned int flush_flags =3D 0; =20 - if ( unlikely(__copy_from_guest_offset(&op, uop, i, 1)) ) - return -EFAULT; + for ( i =3D 0; i < c; i++ ) + { + struct gnttab_map_grant_ref op; =20 - map_grant_ref(&op); + if ( unlikely(__copy_from_guest(&op, uop, 1)) ) + { + rc =3D -EFAULT; + break; + } =20 - if ( unlikely(__copy_to_guest_offset(uop, i, &op, 1)) ) - return -EFAULT; + map_grant_ref(&op, c =3D=3D 1, &flush_flags); + + if ( unlikely(__copy_to_guest(uop, &op, 1)) ) + { + rc =3D -EFAULT; + break; + } + + guest_handle_add_offset(uop, 1); + } + + if ( c > 1 ) + { + int err =3D iommu_iotlb_flush_all(current->domain, flush_flags= ); + + if ( !rc ) + rc =3D err; + } + + if ( rc ) + break; + + count -=3D c; + done +=3D c; + + if ( count && hypercall_preempt_check() ) + return done; } =20 - return 0; + return rc; } =20 static void unmap_common( - struct gnttab_unmap_common *op) + struct gnttab_unmap_common *op, bool do_flush, unsigned int *flush_fla= gs) { domid_t dom; struct domain *ld, *rd; @@ -1485,22 +1520,20 @@ unmap_common( { unsigned int kind; dfn_t dfn =3D _dfn(mfn_x(op->mfn)); - unsigned int flush_flags =3D 0; int err =3D 0; =20 double_gt_lock(lgt, rgt); =20 kind =3D mapkind(lgt, rd, op->mfn); if ( !kind ) - err =3D iommu_unmap(ld, dfn, 1, &flush_flags); + err =3D iommu_unmap(ld, dfn, 1, flush_flags); else if ( !(kind & MAPKIND_WRITE) ) - err =3D iommu_map(ld, dfn, op->mfn, 1, IOMMUF_readable, - &flush_flags); + err =3D iommu_map(ld, dfn, op->mfn, 1, IOMMUF_readable, flush_= flags); =20 double_gt_unlock(lgt, rgt); =20 - if ( !err ) - err =3D iommu_iotlb_flush(ld, dfn, 1, flush_flags); + if ( do_flush && !err ) + err =3D iommu_iotlb_flush(ld, dfn, 1, *flush_flags); if ( err ) rc =3D GNTST_general_error; } @@ -1599,8 +1632,8 @@ unmap_common_complete(struct gnttab_unmap_common *op) =20 static void unmap_grant_ref( - struct gnttab_unmap_grant_ref *op, - struct gnttab_unmap_common *common) + struct gnttab_unmap_grant_ref *op, struct gnttab_unmap_common *common, + bool do_flush, unsigned int *flush_flags) { common->host_addr =3D op->host_addr; common->dev_bus_addr =3D op->dev_bus_addr; @@ -1612,7 +1645,7 @@ unmap_grant_ref( common->rd =3D NULL; common->mfn =3D INVALID_MFN; =20 - unmap_common(common); + unmap_common(common, do_flush, flush_flags); op->status =3D common->status; } =20 @@ -1621,31 +1654,55 @@ static long gnttab_unmap_grant_ref( XEN_GUEST_HANDLE_PARAM(gnttab_unmap_grant_ref_t) uop, unsigned int cou= nt) { - int i, c, partial_done, done =3D 0; - struct gnttab_unmap_grant_ref op; - struct gnttab_unmap_common common[GNTTAB_UNMAP_BATCH_SIZE]; + struct domain *currd =3D current->domain; + unsigned int done =3D 0; + int rc =3D 0; =20 - while ( count !=3D 0 ) + while ( count ) { - c =3D min(count, (unsigned int)GNTTAB_UNMAP_BATCH_SIZE); - partial_done =3D 0; + struct gnttab_unmap_common common[GNTTAB_UNMAP_BATCH_SIZE]; + unsigned int i, c, partial_done =3D 0; + unsigned int flush_flags =3D 0; + + c =3D min_t(unsigned int, count, GNTTAB_UNMAP_BATCH_SIZE); =20 for ( i =3D 0; i < c; i++ ) { + struct gnttab_unmap_grant_ref op; + if ( unlikely(__copy_from_guest(&op, uop, 1)) ) - goto fault; - unmap_grant_ref(&op, &common[i]); + { + rc =3D -EFAULT; + break; + } + + unmap_grant_ref(&op, &common[i], c =3D=3D 1, &flush_flags); ++partial_done; + if ( unlikely(__copy_field_to_guest(uop, &op, status)) ) - goto fault; + { + rc =3D -EFAULT; + break; + } + guest_handle_add_offset(uop, 1); } =20 - gnttab_flush_tlb(current->domain); + gnttab_flush_tlb(currd); + if ( c > 1 ) + { + int err =3D iommu_iotlb_flush_all(currd, flush_flags); + + if ( !rc ) + rc =3D err; + } =20 for ( i =3D 0; i < partial_done; i++ ) unmap_common_complete(&common[i]); =20 + if ( rc ) + break; + count -=3D c; done +=3D c; =20 @@ -1653,20 +1710,13 @@ gnttab_unmap_grant_ref( return done; } =20 - return 0; - -fault: - gnttab_flush_tlb(current->domain); - - for ( i =3D 0; i < partial_done; i++ ) - unmap_common_complete(&common[i]); - return -EFAULT; + return rc; } =20 static void unmap_and_replace( - struct gnttab_unmap_and_replace *op, - struct gnttab_unmap_common *common) + struct gnttab_unmap_and_replace *op, struct gnttab_unmap_common *commo= n, + bool do_flush, unsigned int *flush_flags) { common->host_addr =3D op->host_addr; common->new_addr =3D op->new_addr; @@ -1678,7 +1728,7 @@ unmap_and_replace( common->rd =3D NULL; common->mfn =3D INVALID_MFN; =20 - unmap_common(common); + unmap_common(common, do_flush, flush_flags); op->status =3D common->status; } =20 @@ -1686,31 +1736,55 @@ static long gnttab_unmap_and_replace( XEN_GUEST_HANDLE_PARAM(gnttab_unmap_and_replace_t) uop, unsigned int c= ount) { - int i, c, partial_done, done =3D 0; - struct gnttab_unmap_and_replace op; - struct gnttab_unmap_common common[GNTTAB_UNMAP_BATCH_SIZE]; + struct domain *currd =3D current->domain; + unsigned int done =3D 0; + int rc =3D 0; =20 - while ( count !=3D 0 ) + while ( count ) { - c =3D min(count, (unsigned int)GNTTAB_UNMAP_BATCH_SIZE); - partial_done =3D 0; + struct gnttab_unmap_common common[GNTTAB_UNMAP_BATCH_SIZE]; + unsigned int i, c, partial_done =3D 0; + unsigned int flush_flags =3D 0; + + c =3D min_t(unsigned int, count, GNTTAB_UNMAP_BATCH_SIZE); =20 for ( i =3D 0; i < c; i++ ) { + struct gnttab_unmap_and_replace op; + if ( unlikely(__copy_from_guest(&op, uop, 1)) ) - goto fault; - unmap_and_replace(&op, &common[i]); + { + rc =3D -EFAULT; + break; + } + + unmap_and_replace(&op, &common[i], c =3D=3D 1, &flush_flags); ++partial_done; + if ( unlikely(__copy_field_to_guest(uop, &op, status)) ) - goto fault; + { + rc =3D -EFAULT; + break; + } + guest_handle_add_offset(uop, 1); } =20 - gnttab_flush_tlb(current->domain); + gnttab_flush_tlb(currd); + if ( c > 1 ) + { + int err =3D iommu_iotlb_flush_all(currd, flush_flags); + + if ( !rc ) + rc =3D err; + } =20 for ( i =3D 0; i < partial_done; i++ ) unmap_common_complete(&common[i]); =20 + if ( rc ) + break; + count -=3D c; done +=3D c; =20 @@ -1718,14 +1792,7 @@ gnttab_unmap_and_replace( return done; } =20 - return 0; - -fault: - gnttab_flush_tlb(current->domain); - - for ( i =3D 0; i < partial_done; i++ ) - unmap_common_complete(&common[i]); - return -EFAULT; + return rc; } =20 static int --=20 2.20.1