From nobody Tue Oct 7 13:49:04 2025 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8E4A228E604 for ; Thu, 10 Jul 2025 08:54:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752137653; cv=none; b=GXEx16EQgwD6B2mRIe8sO9fccZPfUq8+yjN9VwD8x58TtrFbwnUBNdFyy59EE7xQnvo9muYmETpgI+P7qrKidAZInxJFFtuwVODNEIkIqdadoD5w/X5hEO33BvfgxAuhd6j4lFpMRWimDHqSK/7gwGtoYGCTLtJqHJNN0rLRp7M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752137653; c=relaxed/simple; bh=0PVCuB/OvGXxkc33B1dLuwyIQOpnfJD+l7vZ5f9Ss+k=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Xu+Pvx5bR9z0w+cCHaaQ5da94IgchDdd5+mZ9dSUZRm5leAp4mySBoN29p0/MDTnuTnmi2IFxoPbTu80ixrW34zy52/FYzuHOz8RqOBTvJF2Tq8NkWctZvKbOaEvvzZ+rmkF8XqhJecreZi9H8rnf2x2kF9Hc4surwEH35ClgX8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=C1gasASQ; arc=none smtp.client-ip=209.85.215.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="C1gasASQ" Received: by mail-pg1-f171.google.com with SMTP id 41be03b00d2f7-b0b2d0b2843so673663a12.2 for ; Thu, 10 Jul 2025 01:54:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1752137651; x=1752742451; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WUKujTie/xgtNbVO7AZBjb1RhL5nzywDx5C8kGbmtCs=; b=C1gasASQIwtHVg5001ryHxZtam1La1Qi0Fwv2hJ8ahZdrQT7JeoOs9M499n4+dFHZa Tchs5oEjcMmas9dXo+CLs5LK2Gh0rOoatm81SvZDcFlXAKPOeMK8Ro24iAGlxsrH2WiI 1LzaelgY2zkQrKgTCpXzb14L66RlArspoOGa15ZHXs25mRmNM8SbO09fP9Y7HAmNoLAT 6RD3qxNwXEsj1Cr27fIgXzE8++Aijj0WDz64Au3UOU79a4PxImMK8NI4+S4f+FXIgnE8 9AEBpV23CwsyKz46cIZnDWS3YcTN+0FdQH6BVEN0GQR3dvvbdpSL0O+Ys/6PsgpyS1Zn E27A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752137651; x=1752742451; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WUKujTie/xgtNbVO7AZBjb1RhL5nzywDx5C8kGbmtCs=; b=Sqw4seVb8ldxEIORNDQSJebMDFs4cHKg6ynODcFguLHLEzWlZ/Qp8SjI9htfAnui2/ umlSezeTXDnBf3rI35c2KQP+XVJBGqC2odiEEHsr853exgLg3lENMJcPultCYxiAhcsg v7eiE/5fwnsRCxCgULr1MiztokOxy73EhijPWu1Zzsi6fObE3a1CC63aceJUxlDZ2+0f TOtd7valsy1xxsl4X5Qje2EVcq090a6PCTpDu8L1OrNvzaaOwSBi7K6Z44NT/Psk8Z9x R73s1Z1ivzutQn82H16xB5VM7MYaPU1RqHoqfZF9+g9tFg/8s0dgmKEIadOh5rV9b5Pc vGRQ== X-Forwarded-Encrypted: i=1; AJvYcCUJxtxvYpwbb2wfQaS/U+0hA6RdViN7aPQ6bXeqTb8ZlC9RLaDjVicJn95GGBVPB6KUxvfaJR6TyWRghwU=@vger.kernel.org X-Gm-Message-State: AOJu0YxpbDhooALwWSRnvhNFYNmO9lksF7PcmuxI4ovc2bNBIw89+VrZ 5TLbHfTU/SNrXm/R7hYmA1LyYRvpGyXOyU433KmEeQq6ufmFuYt80OE7ii+RPBvWZDo= X-Gm-Gg: ASbGncsfWDvxjm88fN43wndtUyIacvGE/SyVH5uwUs8e8fOd7JyA1iDcYavsVNbcaeR tFEIL9hMq71f47jVBylJ4UABKU3KD7hH7JYji8x+O4SqExNj8u1kYgsC0Co8EBgjx3mlkNoFQ+e p27X+o8ECuDa2pGyrhnKYQSLrDCWp5UB4bnbg65lJ7hpplOzjtU1eDW0ASm1G5xivWhR3+VCJD+ s8Ga+BIkzi9antLA6BgD8cJgM6xSKJOiX9ll33cLfh+f5L3C+SEa5Ea3N2iqxI9horOcJHtnlFh Yr+DXd0e3B+gcVWqHaZ7rbqCBdudUnffb3+JFO64p6W4sQgDsZc3Vwzrn+FlVPgPpN/FvFID3sD +4aQ7ZL7HeDiCSA== X-Google-Smtp-Source: AGHT+IFBl6XWSS3SCRPprWFOA1sj7pbVIrvseX/rWxTSbGo3aaOmi/r+s3MsC/OYlpN25M32JL4H2g== X-Received: by 2002:a17:90b:57e5:b0:311:b3e7:fb3c with SMTP id 98e67ed59e1d1-31c2fe0ee60mr9144636a91.31.1752137650816; Thu, 10 Jul 2025 01:54:10 -0700 (PDT) Received: from localhost.localdomain ([203.208.189.12]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-31c3e975d41sm1650228a91.13.2025.07.10.01.54.07 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 10 Jul 2025 01:54:10 -0700 (PDT) From: lizhe.67@bytedance.com To: alex.williamson@redhat.com, akpm@linux-foundation.org, david@redhat.com, jgg@ziepe.ca, peterx@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lizhe.67@bytedance.com Subject: [PATCH v4 1/5] mm: introduce num_pages_contiguous() Date: Thu, 10 Jul 2025 16:53:51 +0800 Message-ID: <20250710085355.54208-2-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250710085355.54208-1-lizhe.67@bytedance.com> References: <20250710085355.54208-1-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Li Zhe Function num_pages_contiguous() determine the number of contiguous pages starting from the first page in the given array of page pointers. VFIO will utilize this interface to accelerate the VFIO DMA map process. Suggested-by: David Hildenbrand Suggested-by: Jason Gunthorpe Signed-off-by: Li Zhe Acked-by: David Hildenbrand --- include/linux/mm.h | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0ef2ba0c667a..fae82df6d7d7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1761,6 +1761,29 @@ static inline unsigned long page_to_section(const st= ruct page *page) } #endif =20 +/* + * num_pages_contiguous() - determine the number of contiguous pages + * starting from the first page. + * + * Pages are contiguous if they represent contiguous PFNs. Depending on + * the memory model, this can mean that the addresses of the "struct page"s + * are not contiguous. + * + * @pages: an array of page pointers + * @nr_pages: length of the array + */ +static inline unsigned long num_pages_contiguous(struct page **pages, + size_t nr_pages) +{ + size_t i; + + for (i =3D 1; i < nr_pages; i++) + if (pages[i] !=3D nth_page(pages[0], i)) + break; + + return i; +} + /** * folio_pfn - Return the Page Frame Number of a folio. * @folio: The folio. --=20 2.20.1 From nobody Tue Oct 7 13:49:04 2025 Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8BC9628FABA for ; Thu, 10 Jul 2025 08:54:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752137658; cv=none; b=I1FRZDSOFEqIGi5pq3bg1+aR3iH7dNjbkUtf6baQyfQYLzLp3Koin02twlJbidIj+9oVxO57GRCU6GDfBAllZLUMP12iiDjj7z99vsoIK2R61HVA5xi3IARE73T8no5hnmjl9RSBH6EDIX7/YCSDKJpTMGemy3XO0AoqsDY7Q5w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752137658; c=relaxed/simple; bh=bvjnKLbum3GXpvkII6kS2kG30yi+AHK7jxW0XhSUXi4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RVQ+LdBEg+8hEP0JKOtkHwmGPRYMB1PHHQgTZs1N8m4VdUDDQ+HKaWaWzXahtfo+zWpLm0Idi28snz4ZJ99ldFgPqxFifbmrPk6l02QrTpWR9Mwd6/5I8Fs4WNvJloJfCv9+uVHLhd4LCpqvma+BDzWLlqosAdFXs955BIBLJ6E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=JF0cEiYx; arc=none smtp.client-ip=209.85.216.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="JF0cEiYx" Received: by mail-pj1-f42.google.com with SMTP id 98e67ed59e1d1-31393526d0dso641109a91.0 for ; Thu, 10 Jul 2025 01:54:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1752137656; x=1752742456; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3M40RlGNwuu5qNc+KyVxXFsF2gHBPU6g2HcwSX76Yw4=; b=JF0cEiYxUGsx3I/UUBPoX/xBRJ3xb9GNkQk4FqIXrvq9ksNv0Pmqfjrnhv8VMhkZNu W2STjcKCKN0RYvXgYK7IDZqAAt/N8rlNZGdro5eNdnYBwoaMu+VSLOkVDpY2bIZH8gf9 eWqXAY2LW7T5HSZDCGsyn73eSgJ9ZdAhzbsYiOrsHfVzyPMxQ83AgVlJIHWzppm4DWwX Ib2AT8xQMVFqVrZ0PpUEds6Xr1k+GJPf28ILR1r0dZiXzpxngsBJBBpECMNE5TTnOyuM 6HjXW/q555Gg3YFoZ8J3IQWSjx4RmVHdKbRnM13QI0rRUGR4fDWm2nstWtl2s1dBIVaP 6R7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752137656; x=1752742456; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3M40RlGNwuu5qNc+KyVxXFsF2gHBPU6g2HcwSX76Yw4=; b=os0jxYzI5uEm7We51r3bnfpFTpl3+H157tSSvxQkPNHi8j7qI3AUj00R1TooOw1RDq B1cZguf5IvNTLbwsf9P8bXQ29pEDKYOhg/svw4tChoS69nFYXWBF159ZqQIVdbaW4vJB G1ZG6HQKN/I9DU9m/6JqVwBXHh0l40GEbmjx628HgvJ/pjIN/5QAqpaAspFZM+wB5xW1 C087mrZP0CYpdaC5t8qssT46YSU0AXQgKN8fdchCZbwNwRYKt0ouQsJz6w6/YwbOqcEJ a1ThtWk9NzZbGBiuBxKscnC8SOB60W0dkmEcAQj1sDvk+hZBhEQyt7KfAgYTiM0MNvEe c/hQ== X-Forwarded-Encrypted: i=1; AJvYcCXt98GX7VRAqeS1dX8Lx8IF50FZpK8o1X3HooUIQNN4mTamWGGMakLVo9spz3aX9d95/ScpEXr2vzB68AQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yxvi+cN6NBdw50b8n/WObkdWDLwOXNK9X0UntdRlOtHJityuJMr OnPg0QQzU6eVsnh/ZIt5BbAo/pLpxVdUlyhsj6k3MVO2P2eXB2pkuJm+fDR+tjvwwtQ= X-Gm-Gg: ASbGncs99oSt3UcNLtT01l3hrI+brogrHhQrYKvTiO0LIWo0GH0ck3XppaMUT+uEv+L 2I4lzEBmaBCqAzXp6rWes9Z8WgAYB9aMjApUCYwgqWDRgyxPX01+uzqtKltQQGTFE/KBsZUNFQD u3BXf3ckHqDfXxCTsgVKoB1V/cs3T+zkIGybkN6LXgXnKVkFzVlNzCO/w8jp2zVc/NxNSd8hC0M bUX/6IAfE5a6KtNg8gMu8OtHLm1zcv7UDJT+f20LPDW9qDR8rMeDvdE+nswihRJ3Vasfl/bjxhY eqSnxyyGhVFaqwQldjsis2ZGMEewI0xZ3SWfBDhKeKz5JTHPypqxmkXSXocZ5X2bMTp57R8IWIv BMizt1YBjlsAI/A== X-Google-Smtp-Source: AGHT+IGdJiBeouQW4gmVvNkRjswMRfLMAfXS8gTDRoMqK+yKXRLmxVryApvFY7wzc49IWY06qUyFWQ== X-Received: by 2002:a17:90b:28c8:b0:312:e49b:c972 with SMTP id 98e67ed59e1d1-31c3c2a215cmr5805568a91.15.1752137655717; Thu, 10 Jul 2025 01:54:15 -0700 (PDT) Received: from localhost.localdomain ([203.208.189.12]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-31c3e975d41sm1650228a91.13.2025.07.10.01.54.12 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 10 Jul 2025 01:54:15 -0700 (PDT) From: lizhe.67@bytedance.com To: alex.williamson@redhat.com, akpm@linux-foundation.org, david@redhat.com, jgg@ziepe.ca, peterx@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lizhe.67@bytedance.com Subject: [PATCH v4 2/5] vfio/type1: optimize vfio_pin_pages_remote() Date: Thu, 10 Jul 2025 16:53:52 +0800 Message-ID: <20250710085355.54208-3-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250710085355.54208-1-lizhe.67@bytedance.com> References: <20250710085355.54208-1-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Li Zhe When vfio_pin_pages_remote() is called with a range of addresses that includes large folios, the function currently performs individual statistics counting operations for each page. This can lead to significant performance overheads, especially when dealing with large ranges of pages. Batch processing of statistical counting operations can effectively enhance performance. In addition, the pages obtained through longterm GUP are neither invalid nor reserved. Therefore, we can reduce the overhead associated with some calls to function is_invalid_reserved_pfn(). The performance test results for completing the 16G VFIO IOMMU DMA mapping are as follows. Base(v6.16-rc4): Acked-by: David Hildenbrand Tested-by: Eric Farman ------- AVERAGE (MADV_HUGEPAGE) -------- VFIO MAP DMA in 0.047 s (340.2 GB/s) ------- AVERAGE (MAP_POPULATE) -------- VFIO MAP DMA in 0.280 s (57.2 GB/s) ------- AVERAGE (HUGETLBFS) -------- VFIO MAP DMA in 0.052 s (310.5 GB/s) With this patch: ------- AVERAGE (MADV_HUGEPAGE) -------- VFIO MAP DMA in 0.027 s (602.1 GB/s) ------- AVERAGE (MAP_POPULATE) -------- VFIO MAP DMA in 0.257 s (62.4 GB/s) ------- AVERAGE (HUGETLBFS) -------- VFIO MAP DMA in 0.031 s (517.4 GB/s) For large folio, we achieve an over 40% performance improvement. For small folios, the performance test results indicate a slight improvement. Signed-off-by: Li Zhe Co-developed-by: Alex Williamson Signed-off-by: Alex Williamson Acked-by: David Hildenbrand --- drivers/vfio/vfio_iommu_type1.c | 83 ++++++++++++++++++++++++++++----- 1 file changed, 71 insertions(+), 12 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type= 1.c index 1136d7ac6b59..6909275e46c2 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -318,7 +318,13 @@ static void vfio_dma_bitmap_free_all(struct vfio_iommu= *iommu) /* * Helper Functions for host iova-pfn list */ -static struct vfio_pfn *vfio_find_vpfn(struct vfio_dma *dma, dma_addr_t io= va) + +/* + * Find the highest vfio_pfn that overlapping the range + * [iova_start, iova_end) in rb tree. + */ +static struct vfio_pfn *vfio_find_vpfn_range(struct vfio_dma *dma, + dma_addr_t iova_start, dma_addr_t iova_end) { struct vfio_pfn *vpfn; struct rb_node *node =3D dma->pfn_list.rb_node; @@ -326,9 +332,9 @@ static struct vfio_pfn *vfio_find_vpfn(struct vfio_dma = *dma, dma_addr_t iova) while (node) { vpfn =3D rb_entry(node, struct vfio_pfn, node); =20 - if (iova < vpfn->iova) + if (iova_end <=3D vpfn->iova) node =3D node->rb_left; - else if (iova > vpfn->iova) + else if (iova_start > vpfn->iova) node =3D node->rb_right; else return vpfn; @@ -336,6 +342,11 @@ static struct vfio_pfn *vfio_find_vpfn(struct vfio_dma= *dma, dma_addr_t iova) return NULL; } =20 +static inline struct vfio_pfn *vfio_find_vpfn(struct vfio_dma *dma, dma_ad= dr_t iova) +{ + return vfio_find_vpfn_range(dma, iova, iova + PAGE_SIZE); +} + static void vfio_link_pfn(struct vfio_dma *dma, struct vfio_pfn *new) { @@ -614,6 +625,39 @@ static long vaddr_get_pfns(struct mm_struct *mm, unsig= ned long vaddr, return ret; } =20 + +static long vpfn_pages(struct vfio_dma *dma, + dma_addr_t iova_start, long nr_pages) +{ + dma_addr_t iova_end =3D iova_start + (nr_pages << PAGE_SHIFT); + struct vfio_pfn *top =3D vfio_find_vpfn_range(dma, iova_start, iova_end); + long ret =3D 1; + struct vfio_pfn *vpfn; + struct rb_node *prev; + struct rb_node *next; + + if (likely(!top)) + return 0; + + prev =3D next =3D &top->node; + + while ((prev =3D rb_prev(prev))) { + vpfn =3D rb_entry(prev, struct vfio_pfn, node); + if (vpfn->iova < iova_start) + break; + ret++; + } + + while ((next =3D rb_next(next))) { + vpfn =3D rb_entry(next, struct vfio_pfn, node); + if (vpfn->iova >=3D iova_end) + break; + ret++; + } + + return ret; +} + /* * Attempt to pin pages. We really don't want to track all the pfns and * the iommu can only map chunks of consecutive pfns anyway, so get the @@ -680,32 +724,47 @@ static long vfio_pin_pages_remote(struct vfio_dma *dm= a, unsigned long vaddr, * and rsvd here, and therefore continues to use the batch. */ while (true) { + long nr_pages, acct_pages =3D 0; + if (pfn !=3D *pfn_base + pinned || rsvd !=3D is_invalid_reserved_pfn(pfn)) goto out; =20 + /* + * Using GUP with the FOLL_LONGTERM in + * vaddr_get_pfns() will not return invalid + * or reserved pages. + */ + nr_pages =3D num_pages_contiguous( + &batch->pages[batch->offset], + batch->size); + if (!rsvd) { + acct_pages =3D nr_pages; + acct_pages -=3D vpfn_pages(dma, iova, nr_pages); + } + /* * Reserved pages aren't counted against the user, * externally pinned pages are already counted against * the user. */ - if (!rsvd && !vfio_find_vpfn(dma, iova)) { + if (acct_pages) { if (!dma->lock_cap && - mm->locked_vm + lock_acct + 1 > limit) { + mm->locked_vm + lock_acct + acct_pages > limit) { pr_warn("%s: RLIMIT_MEMLOCK (%ld) exceeded\n", __func__, limit << PAGE_SHIFT); ret =3D -ENOMEM; goto unpin_out; } - lock_acct++; + lock_acct +=3D acct_pages; } =20 - pinned++; - npage--; - vaddr +=3D PAGE_SIZE; - iova +=3D PAGE_SIZE; - batch->offset++; - batch->size--; + pinned +=3D nr_pages; + npage -=3D nr_pages; + vaddr +=3D PAGE_SIZE * nr_pages; + iova +=3D PAGE_SIZE * nr_pages; + batch->offset +=3D nr_pages; + batch->size -=3D nr_pages; =20 if (!batch->size) break; --=20 2.20.1 From nobody Tue Oct 7 13:49:04 2025 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2DB852900A0 for ; Thu, 10 Jul 2025 08:54:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752137662; cv=none; b=TEcBRuXx3P7ccWi0DQQXq/PVbR1p9LL9gxUPeTDEU2mtLzHferLE+LsjfMaBFHtClc6SQW7QegNFXB5PolvylNvrcQGKb8iBbH6p4sR3D+DzEbcF7W6Tva3i2Oc5jLEbtoIklLpWmff5LOFaibGEbQnxxZ2INGCyMP33TblAqKA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752137662; c=relaxed/simple; bh=4N5I2w5e2jHfjeqUUs8UP8e7d+l4qfu2b+4DBY6K2Qc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hiLAbUjotb49Jh87Mt2dYJBlRy6MNwCp2Ya3MHsg8C6tcUNs6K94FOkVfhBdRLiX+f3d8c/jUTLdyvg+e7nSgQJHU/3v5vOhHS6F815YcszQ4GtpM891qwlJeFqe46LkMmY2iei25EE3tMwvv9eRVNwTNVEs1LsuTq85Xvv7kVg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=RBbP7lt5; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="RBbP7lt5" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-234f17910d8so7865155ad.3 for ; Thu, 10 Jul 2025 01:54:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1752137660; x=1752742460; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Nx6tn515BmvVRbPQz9Ke+EyJMh4nGWIC6oD4cxq1tFQ=; b=RBbP7lt5E2/ic+C0dUkuLmZYndw7LSu9MUGrM5BB8v1z+2l8K8zJMnWsjYN+7ilOiH TQGinApXthUPYA4B0dLDGJDpsa5qn25lcRzzqg7QMaVqyk3wdyIG7DCorP6eVjTqfHip i2ZDWrtt7ZnIi/35+nMXuKmO8z00lXQ3giisi3oYU8Kf5cw4W5ayCgGRXL8CPB5hdZgc 1TyEmDzLuDGTddmhcxggefBzBd6ZEd+YFi+ciiDwsAejhb1Q8i4YC3JIumT11l4A7zmZ x66Tl+o7OlQu7SDZcAAv9sVnzsleTr5SPHqcajHML6KvDiZ21Q2N1ElwMFYkd571G0GY kDFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752137660; x=1752742460; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Nx6tn515BmvVRbPQz9Ke+EyJMh4nGWIC6oD4cxq1tFQ=; b=oDWwe/6k3rlNW/uz617nsfm9qWPdfLwn1hwwA8pQ8xFrFunh/MD86LXeQ0hvOwwXYE bpGzhvST+fLbpd7y9J0Mxv3z3psTzuaGEa3/wR/1DaQhhSTCbCPWQ1FjDST7rMLidFVs FhE64k2YdkH+/0Ib7/CYlyfn0J+p4I8oEz49JW0xoN95r64J+nqgw+OSoHc6Cz2oPjvv ZZMUhbtCalk+WMheN4Bztjw2XJPdhN4+q46Lup2jN2+ErbubfaoONLpOKlF7UdbK7H2a jNn4fS+GHcYiqDtGakPkhtxBNuQOr/3WvqYjKLDHCvUgPWfErPnUCQdBBUa2Sp9R3Wuz 0g1Q== X-Forwarded-Encrypted: i=1; AJvYcCXcmDFq2IfkQgQh4X/fQzryIxiSBpUXzOqK/uxfqYPRz4xzuWWDCvZg2jy8aw4SMetGG4gvi0zH75zQuqE=@vger.kernel.org X-Gm-Message-State: AOJu0YzsqUrDf8wvCmUkSRXCl2Kvx0z7jghmCidP58b9lQBzx6Cmg1ae XZhbEo0oCGhs1JnwTLNoZgb8YT/5rwRJBfF4f5iZaOnCUuYMIfc0HmsRiJ8WYWL1TJ8= X-Gm-Gg: ASbGncub8TfDyIekTRRCMHX/3vVVwaAYe1SzvEecWa182Pijx81CpbLNrdlpaETrl47 R4CmtWlVadcvb4r1Q+xq08XpIKkNLC7N7Qb7gIcSL4dhBEp6nL1kTwtS7QmPuLAd4eU5p/K4gWm ZOPIsghUpMbmhY4Vh/D9oKE0uVHDRNnaEn+LNnafhG75rYy+XpjrbiwuJZBdLR9SCCxdXm17Xpa JBrqs3qRS0CQ3gR39sizQZAaRreG6xl4WW1H5/Z6Hl3HiAbniuOCKE8y2JvOhmufB9W7xa/KpSL +UAmFEhWle17cUx4U1/SD6VodDySnB5qMh4P+C3A7xJ6racbad1X+m9fi1lFcw71KXDI6JON8l0 06o2AJDRZVf/5lQW64Xuyr8Od X-Google-Smtp-Source: AGHT+IHinmUo1Q8JyIR8mVroxGH87B3iRX98m8gWf3ZeXBpz/jGd0mYLjHRrHOq8wekDWW+AqoXtuQ== X-Received: by 2002:a17:902:db0b:b0:235:225d:3087 with SMTP id d9443c01a7336-23de24d9865mr39979545ad.30.1752137660449; Thu, 10 Jul 2025 01:54:20 -0700 (PDT) Received: from localhost.localdomain ([203.208.189.12]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-31c3e975d41sm1650228a91.13.2025.07.10.01.54.16 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 10 Jul 2025 01:54:20 -0700 (PDT) From: lizhe.67@bytedance.com To: alex.williamson@redhat.com, akpm@linux-foundation.org, david@redhat.com, jgg@ziepe.ca, peterx@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lizhe.67@bytedance.com Subject: [PATCH v4 3/5] vfio/type1: batch vfio_find_vpfn() in function vfio_unpin_pages_remote() Date: Thu, 10 Jul 2025 16:53:53 +0800 Message-ID: <20250710085355.54208-4-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250710085355.54208-1-lizhe.67@bytedance.com> References: <20250710085355.54208-1-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Li Zhe The function vpfn_pages() can help us determine the number of vpfn nodes on the vpfn rb tree within a specified range. This allows us to avoid searching for each vpfn individually in the function vfio_unpin_pages_remote(). This patch batches the vfio_find_vpfn() calls in function vfio_unpin_pages_remote(). Signed-off-by: Li Zhe --- drivers/vfio/vfio_iommu_type1.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type= 1.c index 6909275e46c2..cade7ac25eea 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -794,16 +794,12 @@ static long vfio_unpin_pages_remote(struct vfio_dma *= dma, dma_addr_t iova, unsigned long pfn, unsigned long npage, bool do_accounting) { - long unlocked =3D 0, locked =3D 0; + long unlocked =3D 0, locked =3D vpfn_pages(dma, iova, npage); long i; =20 - for (i =3D 0; i < npage; i++, iova +=3D PAGE_SIZE) { - if (put_pfn(pfn++, dma->prot)) { + for (i =3D 0; i < npage; i++) + if (put_pfn(pfn++, dma->prot)) unlocked++; - if (vfio_find_vpfn(dma, iova)) - locked++; - } - } =20 if (do_accounting) vfio_lock_acct(dma, locked - unlocked, true); --=20 2.20.1 From nobody Tue Oct 7 13:49:04 2025 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D7FC292B5B for ; Thu, 10 Jul 2025 08:54:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752137667; cv=none; b=A/qrFmIyNAg1RfHiVyey5n6EMvHwSs36C8FJ7O1eHITXjDN41CJTY/mKmzPp+v2uGGUo+o+avDQEHaA1cTNP0bcWinLrETIrATAiEAqYA6KNaJgZRp9jSRWNTm4l8E7Mtrbt1CyW8DBlaBlHs7LNbI4bpsvi2+s/doDJTJ2GEU0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752137667; c=relaxed/simple; bh=kuMO4alh4taqq6qqK8Ezmq1eFdMQaf8rRvbNnP7x1ZQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ewaXxvDcA08GdKd5CkLUFVZsK0eqRuS2DlgxFOZRg6uXOsEfKh/xgw6A3XZe1LaRihrHgGXzm2dDVRsHO4nSvhA/dJ9yaX2DmixquUYiOvim5pLVHTGYgr/FuxDrDW2zWsUFxVRmJqyNJCgzz2iGD7kyzI3fhem6QlIXbqk4BAw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=LENEC6da; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="LENEC6da" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-237311f5a54so6588755ad.2 for ; Thu, 10 Jul 2025 01:54:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1752137666; x=1752742466; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=euNz/l/CHGPv/Gat71a8nUVi70h05zHrvuBm3oNz29E=; b=LENEC6dajcKktGlvKMstisUT7+YxOsw6i+54mmKubHTWy8oH2js8w+1etQkArkfdPd Z5BO9cwZc7LFDTpSt7kwHcQzOUEnB1YTVUpYfrP5nbq7aa92r9sKiscMtsgyuPZgcr6N /dW78f71s/1bf9ZO+eEqyiLT+9cx+0W+uMy4m5FyzjnwUrT6OnfPyS9mscJzp5nNv2JC ARkEtXIEeOSWNL3kLpmLoTULpVELNluih7qd6turjp0UEbtDPYTJu3FMm1wyI1zJkeIT OvAItdgXmNSCXrQuAcg8Y/xg9affIgB33aLOHHlzGgHZF+opOtuXP1pui/lPMrF8AYhd hS5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752137666; x=1752742466; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=euNz/l/CHGPv/Gat71a8nUVi70h05zHrvuBm3oNz29E=; b=iHe2qqX6jqTu/EAoUG6flOMv3qLG4n/mJs7QTvbguBQ30b+cOga1VYO01d10oQak7O qxsKCdUQwGdW3fJfQS3qgNjRHLMHZgAwnVRVNisW7ngAulPUgsyOXqNpwFbwhvLJ+YeZ IZXy1TyUqj/D8Uzl5G7t4YJPqA/kMifBqqdX1ypnPdKWlekTL462gPOq/DS0BOKWJEIN RGEPOoRExQIw8+3NFm5oco/r5jkSDcW3wlOX7O9I3tgEEEwCN89xl9Kvkf6OCXfHQ8wp YapHHPjXXWPqqko6BQd4r/xmPUumCQ0rsoh5XTXdYiWrk3EgXRxgHoN0sWaqQqaPzSB1 SdwQ== X-Forwarded-Encrypted: i=1; AJvYcCUzIiJFNLNEx3m4zO9Vln8mH/BsXhmaCY9uS7Q/sN4vHWc0K39OPONAiZJWEKoDLgManamUhagIjo7E94Y=@vger.kernel.org X-Gm-Message-State: AOJu0YzxI4A6SpX1gTrlehfiUTpODp/B2G+XYuJQh8kwBG9K7IBu81zn qlPZ3HZBrumpQ5/1DkvED+/Y57uLYRhLMBzlWA9K6Fq8mbn3dGbMLflaxqeaPnsPNehygjGemAj mf2Xx X-Gm-Gg: ASbGncsyHSRyI+V6sn6ivcb6WmGDjanig70+RADi+MBuywr8ZZelru3fWO1LTfil8jp ftvv5qpIy48C9mBnWH52QghQ6WR+aHNJBpgIQpjyMvjs3dH7zT1SXYD80bfmh9U9bV/0oe+loMY kLo7aaIwRh/EoznCKbFcXl+uTYFLUDdYlZB9i02+h38ZHaxrN9aqmPNJzKG3FJh1SVbD39nCazg 6VQFHwOh7Ji23+IKmnO8HYLkc5Lb6LYqjV6wlulSb0AbEMIkCYcyg5rr+5UzoRy2na+dvQvIE3T 5994XEuRc8jLI5Ml7F+Q/6wwSXzygKEqjXodyHic1VfnyhA2yKUcaGqh9sf1IpNjcTiwkcNYuKz +54FpuvFJiQAe7Q== X-Google-Smtp-Source: AGHT+IEt7OZwGubuhbBz8Z/1KB4UN6EU83aIIreg1b4VW2wSiqDfLzbhNwjVrNvPGleM5LD1QDFIHw== X-Received: by 2002:a17:903:a8f:b0:235:1962:1bf4 with SMTP id d9443c01a7336-23de4803965mr31979305ad.14.1752137665596; Thu, 10 Jul 2025 01:54:25 -0700 (PDT) Received: from localhost.localdomain ([203.208.189.12]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-31c3e975d41sm1650228a91.13.2025.07.10.01.54.21 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 10 Jul 2025 01:54:25 -0700 (PDT) From: lizhe.67@bytedance.com To: alex.williamson@redhat.com, akpm@linux-foundation.org, david@redhat.com, jgg@ziepe.ca, peterx@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lizhe.67@bytedance.com Subject: [PATCH v4 4/5] vfio/type1: introduce a new member has_rsvd for struct vfio_dma Date: Thu, 10 Jul 2025 16:53:54 +0800 Message-ID: <20250710085355.54208-5-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250710085355.54208-1-lizhe.67@bytedance.com> References: <20250710085355.54208-1-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Li Zhe Introduce a new member has_rsvd for struct vfio_dma. This member is used to indicate whether there are any reserved or invalid pfns in the region represented by this vfio_dma. If it is true, it indicates that there is at least one pfn in this region that is either reserved or invalid. Signed-off-by: Li Zhe Reviewed-by: David Hildenbrand --- drivers/vfio/vfio_iommu_type1.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type= 1.c index cade7ac25eea..61455d725412 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -92,6 +92,7 @@ struct vfio_dma { bool iommu_mapped; bool lock_cap; /* capable(CAP_IPC_LOCK) */ bool vaddr_invalid; + bool has_rsvd; /* has 1 or more rsvd pfns */ struct task_struct *task; struct rb_root pfn_list; /* Ex-user pinned pfn list */ unsigned long *bitmap; @@ -774,6 +775,7 @@ static long vfio_pin_pages_remote(struct vfio_dma *dma,= unsigned long vaddr, } =20 out: + dma->has_rsvd |=3D rsvd; ret =3D vfio_lock_acct(dma, lock_acct, false); =20 unpin_out: --=20 2.20.1 From nobody Tue Oct 7 13:49:04 2025 Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 52EB728F527 for ; Thu, 10 Jul 2025 08:54:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752137673; cv=none; b=p5YWw8x0jrYIy1N9BLuwKm3XETYafYWcIgojZo8GORkEMjCP/PjtdK4eH78UnLGOpJyy2EaBLeY5viJI15IsseCWEmtbBF1QIjx7HOCxqDwcL9AsOnSHiUw9gHRqGH9ishJkdpTAV4cMJmVv2oe7iv1mrUtOcz+LcuHG9K7g0OY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752137673; c=relaxed/simple; bh=XzjoEAh+cvHyRje+0Tjj6rqGbTKwrsSfOvDNlMabzbU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=W6jpiBCsreOzKDuyo9X3zRGDaco8aF31d1UggziUHNYJ7dxW35c0dIcYiF4kMIR1VC+BcfSEfb1f8esWZecyaFz8+AHzTgzappKkdbspI/5KVMwmWHCwEIh88GZVLa3tY7BaTrhUUlT8E9zD9JMGr26mBqwiixjTeOYGx77xfso= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=GvPDJlwY; arc=none smtp.client-ip=209.85.215.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="GvPDJlwY" Received: by mail-pg1-f176.google.com with SMTP id 41be03b00d2f7-b31c978688dso520156a12.1 for ; Thu, 10 Jul 2025 01:54:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1752137672; x=1752742472; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=i4mO6QS2sYfv7EymtBvZKW+/VtzFT/ogV5HmARdE954=; b=GvPDJlwYCa0EBotRO1G6jKvgrURnBT/0qOFF844NJuPh5mxNgqDfu1uX2wJ8zJP8ra EfVxne82R6oR1oQXLSZguqop+c/hrZ4bq0jInTgt5v+zW4IRAmrsmuVXR/kDMsVSWu1I Vsccg+SLrdhwBjO2sXzvRGfwkCDGKKeKBn62SwqfO0z/XUjfPBwF/CKRFyOu+81xslUJ My+ASvGz68uRYFE9FT2SHjqpMQQFCJxQs/sRXvG0dt1mffn0yzMr8Roc57r5fQI6rk1f jKROmQJt8/7IOB4X42ro+P9uTkjZ66IsYA3Q58sUN+EN5hi5j+LVHkXrWHE+ovJvw24B Tvqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752137672; x=1752742472; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=i4mO6QS2sYfv7EymtBvZKW+/VtzFT/ogV5HmARdE954=; b=gHpwtUQWyeuC4RXElPOyeZ+6szknmTYcnzBmFbmxVegKN11aA/TrYTrMXzPdFbquFD 1031WQ8lWy2vurcgfcIaWJ2mfpel4/XZ44dxOZtf5uZdGXMoUPyeo8/FpmO0BWhThx4t gV+A6mPMY8Af+f3gfFFsI5LaPmJTlxeVTQb16ETCILTz91eAxvQH5EXbLO6f+idzwBQg 842YZ4ra7zIzMfbtrKkoBuwbxtW86enUSAwDV1miYr2Oi2aQFm3yWKRQIybnTwQe31T8 50I526BjTaQjcTVoo7HUF7fkV0iHxL/6UIPpB1ZF5uNZ8mdE5ERp569o5kZi3+qNtGm2 Yf4g== X-Forwarded-Encrypted: i=1; AJvYcCXBjmjASmRwtg9fZy80xdhX/vELy6gZf+nYiT8uVyk+49mXECSGP+WhmB57giZ6V1In7/X/9wyFpLFB/BA=@vger.kernel.org X-Gm-Message-State: AOJu0YzsEken0AOE0ohYJxLRu6ppola9h+oBC1zrl7VZQbvPfoAcgSuu 0UzKApeb6RoK+vvuohoUpTgf03eOVIYgZBoUwquiK2J4+gpNzqkRxSgN23LtBWexCkE= X-Gm-Gg: ASbGncv28Et9m1LKipQKn0JGA7mwobIZFQz+Zr6E1SHnlyja1ugS9ayqHWnUMma46mo HvkkI3mgx37KbYn4HUjKUVietxMR2iubItGj0CT1NIqUpmki7EzsXWCR761SVejjWvFgLmBBbwc 4ha+nyB7aMOCJYzjXpcuMviR8BDbvw7eF5YfLjDlzTK3AJfLH0XszN/Ag0B7IqGnZ+AgDqhjFVQ qj64JjwRG/Nue2YhNfFysg+pYIuappOIin2gRRI21hXsFHxjGID2epi1dd12ohLGSeUXw2A5vJP yxOTu/050eRSC+w4iKtOrqtzp1DV7TUHba44jE9G2KPl4ntTlJ9ZPrLE7Jx3LwgnNJzji4f45KV KEnsChPyffna/xg== X-Google-Smtp-Source: AGHT+IHpGnM8/RSgCKrl528Fh5n6c4YlUKSZpBOagB5O4dzJXeVEsMY3FEHcb5l/h+FzLkNngxE9NQ== X-Received: by 2002:a17:90b:4a4a:b0:311:b005:93d4 with SMTP id 98e67ed59e1d1-31c2fdd1479mr9203981a91.25.1752137671529; Thu, 10 Jul 2025 01:54:31 -0700 (PDT) Received: from localhost.localdomain ([203.208.189.12]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-31c3e975d41sm1650228a91.13.2025.07.10.01.54.27 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 10 Jul 2025 01:54:31 -0700 (PDT) From: lizhe.67@bytedance.com To: alex.williamson@redhat.com, akpm@linux-foundation.org, david@redhat.com, jgg@ziepe.ca, peterx@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lizhe.67@bytedance.com Subject: [PATCH v4 5/5] vfio/type1: optimize vfio_unpin_pages_remote() Date: Thu, 10 Jul 2025 16:53:55 +0800 Message-ID: <20250710085355.54208-6-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250710085355.54208-1-lizhe.67@bytedance.com> References: <20250710085355.54208-1-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Li Zhe When vfio_unpin_pages_remote() is called with a range of addresses that includes large folios, the function currently performs individual put_pfn() operations for each page. This can lead to significant performance overheads, especially when dealing with large ranges of pages. It would be very rare for reserved PFNs and non reserved will to be mixed within the same range. So this patch utilizes the has_rsvd variable introduced in the previous patch to determine whether batch put_pfn() operations can be performed. Moreover, compared to put_pfn(), unpin_user_page_range_dirty_lock() is capable of handling large folio scenarios more efficiently. The performance test results for completing the 16G VFIO IOMMU DMA unmapping are as follows. Base(v6.16-rc4): ./vfio-pci-mem-dma-map 0000:03:00.0 16 Acked-by: David Hildenbrand Reviewed-by: David Hildenbrand Suggested-by: Jason Gunthorpe ------- AVERAGE (MADV_HUGEPAGE) -------- VFIO UNMAP DMA in 0.135 s (118.6 GB/s) ------- AVERAGE (MAP_POPULATE) -------- VFIO UNMAP DMA in 0.312 s (51.3 GB/s) ------- AVERAGE (HUGETLBFS) -------- VFIO UNMAP DMA in 0.136 s (117.3 GB/s) With this patchset: ------- AVERAGE (MADV_HUGEPAGE) -------- VFIO UNMAP DMA in 0.045 s (357.0 GB/s) ------- AVERAGE (MAP_POPULATE) -------- VFIO UNMAP DMA in 0.288 s (55.6 GB/s) ------- AVERAGE (HUGETLBFS) -------- VFIO UNMAP DMA in 0.045 s (353.9 GB/s) For large folio, we achieve an over 66% performance improvement in the VFIO UNMAP DMA item. For small folios, the performance test results appear to show a slight improvement. Suggested-by: Jason Gunthorpe Signed-off-by: Li Zhe Reviewed-by: David Hildenbrand Acked-by: David Hildenbrand --- drivers/vfio/vfio_iommu_type1.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type= 1.c index 61455d725412..133f284ae168 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -792,17 +792,29 @@ static long vfio_pin_pages_remote(struct vfio_dma *dm= a, unsigned long vaddr, return pinned; } =20 +static inline void put_valid_unreserved_pfns(unsigned long start_pfn, + unsigned long npage, int prot) +{ + unpin_user_page_range_dirty_lock(pfn_to_page(start_pfn), npage, + prot & IOMMU_WRITE); +} + static long vfio_unpin_pages_remote(struct vfio_dma *dma, dma_addr_t iova, unsigned long pfn, unsigned long npage, bool do_accounting) { long unlocked =3D 0, locked =3D vpfn_pages(dma, iova, npage); - long i; =20 - for (i =3D 0; i < npage; i++) - if (put_pfn(pfn++, dma->prot)) - unlocked++; + if (dma->has_rsvd) { + unsigned long i; =20 + for (i =3D 0; i < npage; i++) + if (put_pfn(pfn++, dma->prot)) + unlocked++; + } else { + put_valid_unreserved_pfns(pfn, npage, dma->prot); + unlocked =3D npage; + } if (do_accounting) vfio_lock_acct(dma, locked - unlocked, true); =20 --=20 2.20.1