From nobody Tue Oct 7 19:55:41 2025 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EED5B27B500 for ; Mon, 7 Jul 2025 06:50:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751871013; cv=none; b=bHKNabmRSwEbMIvBUHq4SIlV7SBiUo8olgbSrd1hwyo0QMRjblO6cyxngfMZRK+s0fNXc4MPJEUMOOXBypm0qjqvP4+f9yU6HRfYcjHCMet5JQPsO0lmRElpR+oDAqeFN9dEh94MXiwz/Ky0OBZkUy77W8a1dBxFDvmknO/3N6E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751871013; c=relaxed/simple; bh=yySkyAzKw9ZyNy7KrfMmwyacolQBHXtTfXNgEj0CYxg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gd+B5f9axih8V16EYcu/Wsh3J6WdbgDIJiX7dxHjNCZq66uIVW7IlxJfslQqOsqsc/+MBX5bZ7i5UDBgAwTRG738c4A8gKIU9ftMEWulrMp+Eoy2REwVkL6jT44yMMfHNM4lDhN2/h7e7BUW1WjiH8dkXpM/VioDyf7zzl58PLo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=QeN9lzcF; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="QeN9lzcF" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-2366e5e4dbaso27737125ad.1 for ; Sun, 06 Jul 2025 23:50:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1751871011; x=1752475811; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=e83zaooyctJVGMAyfqno/5MLE5GO1gHM6Xp57jXORKE=; b=QeN9lzcFeolJxlBX+zujzzwobxK2Gg6SiwflUy05DPo4ug3gvEMhonmN5vdSkicxeR pHEMI6SwATKY9gYMbgZgiXstVFINEMrB24meOeTUtl6P4RHKC8hLf9bVMOrmu5Gsyn7b j1K3z+yxVMsGNtDL8aRntfd6d8qM6NMFRgJ48U1uioPY4XFVlKYYbTLpETQaCA7SAeDU tw6KtNxVhHVzp8nIgUCLbDYHsDPpuDW7Qvk+rSvnHtnTO7Jtix5qcWcC8ikzAsA6E91D wJez2vn7Ise6yJlGUmbNuja6Kodpmd83lT+X2Akw4YGa0VWqdEDTYI8KL4J5DY8x60IQ b7hw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751871011; x=1752475811; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=e83zaooyctJVGMAyfqno/5MLE5GO1gHM6Xp57jXORKE=; b=o08yILyfCCA8asaoQV17BSXUUqe2rtWfEqaGHHhVPEcZn08cgRNOUjXbkFR/6iMiiO t7cgb54UBw5+r3rgogA+hEQwhOUtAnbT3XIgPU//9EsJzf3E2QW2mN+t1cbMw+tnRItZ egnh1W0e1CfAd/EoyHcCLuBlMrtSUVgEJ5yzlLPeaH2GdWsqACN1QwqlC7AMT8Q3R0tD lnZ/Sbgp7QnKMOruric0kBzDS3XIR5V1/58oed2EoH8il1JOLSaCEZWtZ261eJ1t8RrV HpbIhl1EjdzSVVSR8bcuEHTyI/VRkzm2a0F8v6Sz4O9JAH+ASLzbG/6tA0xw0MJy+k9s tuPw== X-Forwarded-Encrypted: i=1; AJvYcCVBSOzENswBEkW9i9WH7WXTKmd3ZEG0aVPKngBm1Qkl7FRQsPlKjZTZsdRBVdB3ZLY/T/QuMROYOswWzGM=@vger.kernel.org X-Gm-Message-State: AOJu0YxMnGPHKhWPJ+kbt3vdTPoMG97/vGZCs0VD5bHRE3cKkuSVbZON oKNN6DIWcXr1qTI4FbKPgIaqVIcJ2puDxaRn++SpEkkoasgtrk79ObWX0iI+p8sn42Q= X-Gm-Gg: ASbGncsApjbZ9hDXtnG7mNbNdBfWKb250rSmLPwS/vln0r5W3JWXXLTqsD6emaIP1wL ggEBZVIaF//WP2Yy8b1iFRHROgtt4Wgy6LFpPR4+i8Ed/7/GThanevP2gqsUSlP+jeZirJlq7YT N4UsXvBMkpxKSlfeWBkHlJSkSkhWGQHCjZSdwQR40KGPSMcVhZm2NPD5hJTGap/xy2XdbDpqgAy ggMucXrfoYw4ROyELRbNXfeCmTA5EwN92jBa7Y0CE+xTzHSpR2qLmxXQ7E9NW/ANRhA2qjOamEQ loPWCzkY7EVd+ywPTdgj5XbdFW9i5ci3L6rSui/zDHCBUNx1YUpLPtohNxaOHJOQgnSOvv28cX1 lZKTSTgehmpTi X-Google-Smtp-Source: AGHT+IHZIatv+CFtPNahfbucN1jWL70KTS2wbiORevgf6e2vHgPOO4LbLUKtKdQFdYuPICyB+irnmw== X-Received: by 2002:a17:903:1ce:b0:237:e753:1808 with SMTP id d9443c01a7336-23c84b9a123mr186936395ad.20.1751871011205; Sun, 06 Jul 2025 23:50:11 -0700 (PDT) Received: from localhost.localdomain ([203.208.189.9]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-23c8431a1aasm77377635ad.15.2025.07.06.23.50.07 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sun, 06 Jul 2025 23:50:10 -0700 (PDT) From: lizhe.67@bytedance.com To: alex.williamson@redhat.com, akpm@linux-foundation.org, david@redhat.com, jgg@ziepe.ca, peterx@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lizhe.67@bytedance.com Subject: [PATCH v3 1/5] mm: introduce num_pages_contiguous() Date: Mon, 7 Jul 2025 14:49:46 +0800 Message-ID: <20250707064950.72048-2-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250707064950.72048-1-lizhe.67@bytedance.com> References: <20250707064950.72048-1-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Li Zhe Function num_pages_contiguous() determine the number of contiguous pages starting from the first page in the given array of page pointers. VFIO will utilize this interface to accelerate the VFIO DMA map process. Suggested-by: David Hildenbrand Suggested-by: Jason Gunthorpe Signed-off-by: Li Zhe Acked-by: David Hildenbrand --- include/linux/mm.h | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0ef2ba0c667a..fae82df6d7d7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1761,6 +1761,29 @@ static inline unsigned long page_to_section(const st= ruct page *page) } #endif =20 +/* + * num_pages_contiguous() - determine the number of contiguous pages + * starting from the first page. + * + * Pages are contiguous if they represent contiguous PFNs. Depending on + * the memory model, this can mean that the addresses of the "struct page"s + * are not contiguous. + * + * @pages: an array of page pointers + * @nr_pages: length of the array + */ +static inline unsigned long num_pages_contiguous(struct page **pages, + size_t nr_pages) +{ + size_t i; + + for (i =3D 1; i < nr_pages; i++) + if (pages[i] !=3D nth_page(pages[0], i)) + break; + + return i; +} + /** * folio_pfn - Return the Page Frame Number of a folio. * @folio: The folio. --=20 2.20.1 From nobody Tue Oct 7 19:55:41 2025 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A803627A451 for ; Mon, 7 Jul 2025 06:50:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751871018; cv=none; b=awjvG+9DOqdcTZCqYjunqA25LmHsF5Y9vqw+o2SUGuttHY/VXyjv9bk0i/U6HJAnCSsRXn38R0DZfVeIdhiAZjroBB/SJ+uJd+qJRrfoMiq4p7vU+KZI6wVuPXcbbmUmfPHvgIFW0nNC8R2TOItF6n50wKq/GHvOVgA0oCXnH2w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751871018; c=relaxed/simple; bh=fb6co8jrIb0ye1DMUwvR0cGT/E/Yrsdll1Jah3tS+yU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EoGBpVbc6z9uQbyEul7FzbZygKvVc1RLe013wU8TKjZ2hQh4K9wwpFlzGBMjvXjT80811wOsOTBKdrpnMn4HraF4ndusECT/Z8bH41H3GJQkvENnShr3iNXIzDr1NXejQS0SvuqKkgEQGbU+2tAKAI8dsw3kZmUFuaoVn3uOCJI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=UBR1UjVl; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="UBR1UjVl" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-23c703c471dso31210285ad.0 for ; Sun, 06 Jul 2025 23:50:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1751871016; x=1752475816; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=e8Cadi1skh4P1ApGM+PqU48O5jgE1uMvu2URGbuzsOs=; b=UBR1UjVluBNaTGe5tQTWF3yhhXj1kxwWFOueTxtjq5IlWPGaCB6p6FcIjAYi6bS+2L n8UIgzaGXxuD48j8Du69t0ZYr6U2cgdsKMPs1J3TsLi5qkFdI0A33ecuR+td0oqiSH7Y 16DKoCjRC7k+6GwGPu5CLiwsC3J175GECCE0hYYEwzCNtAW5gCeKR3Rlc+iDKH0OMuyu fHDGjkutpbQnMuNAMhuNN8E4gjowc17jTAW12d5IFKELwP/jGcq1w733F86b9z6mfqiz PC+wt+w6ITpXBw9hktwLiM19yAF+3khA4aSmWN3RmsSK0tlnF94W2GLD6IUtawJ+EXNK 6RcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751871016; x=1752475816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=e8Cadi1skh4P1ApGM+PqU48O5jgE1uMvu2URGbuzsOs=; b=oNgaYJJEflJzPP4RCOB80K7KIeyoGh/iJGjP0UGTJbMWtCon5MiVg7wMdBH48lIo1X +WTkI2wTYHnjNnGImBC54gPmxfSBEiKahqdRnOzBZCgefQ5yCOWriwqZH0GRDXdl/cHs Yj553JgvdWpjBpo9PZc5t0/wRr2UjorXvxeH7kvRycJNWsRnlNXQf6na0Gms2L6PNwIl dt+XKbKs3VZE+bwNBDf2woeORygu3GxIRpAJdNs+5qvkhymoZt1KA0pjjbMmDgIEdS5+ jzl+E8y6cjU+j2qCw9r7wiqCx05U06B63EUzcIyf0699Zx3CCN16NPpMKxNrbdxo8WlJ 4+FA== X-Forwarded-Encrypted: i=1; AJvYcCWd+h42EfWR5e62+6reVMiUMNFtU+I5GINbbj5toXKZOpOMAJuO2ChXQ28uXV3mcv/AWOjhdlEPIkmXoVw=@vger.kernel.org X-Gm-Message-State: AOJu0YyY8gIdTEjq+VGNlar8qUEcOvQoVUBGTqryei3D0TIs7HoemKbL jYAvzvwSz+YRXBkMT7ROrcbjfKpzezII1f6fHWS2kWo8v2E/FpGbyBr4Kns8szN7Yn0= X-Gm-Gg: ASbGncuQwshUQ6eqdVqffbeTr2tUdKhrJlTGsuOrodY9VwxLf2LWb8gNk23OZ7w+tT+ RxIftEUj0+CFEBhdX0tqKevKoEvTac9f3Y0xgN2ERJmFhlLHlgliFESvPXJCv5x0HNkslj1v/Nr 1QEKkdkfWrSVL1sZ56gzWTIN9RUg71J8CkX/z7NhWGKc+3NryBErygfs96DjYlWwemQTXY93YF3 Cio6boVk+w1O8SrGXKVtQvFUbsp4MHQLdjM5Tw8Nk50LSiwWv3JqAh2MyGT+Vsfw0RfNXYoBVD7 fh3AYl8nXcQAP1EeAVRLeurkcsQMZQnkf+bdRp1urN0ubK3ebTum8eDr3YzpR9epO2/2+ITsZtY 4d9Mtz/75B1XK X-Google-Smtp-Source: AGHT+IFDMC5zNe5FLEop89MsvruvqnioIQVRr64gsK9+ppHn/U4Xbh1NwxhORd/zYO8Y0kwaArR9Dg== X-Received: by 2002:a17:902:e80f:b0:236:6f43:7053 with SMTP id d9443c01a7336-23c85886bb4mr158575025ad.2.1751871015977; Sun, 06 Jul 2025 23:50:15 -0700 (PDT) Received: from localhost.localdomain ([203.208.189.9]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-23c8431a1aasm77377635ad.15.2025.07.06.23.50.12 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sun, 06 Jul 2025 23:50:15 -0700 (PDT) From: lizhe.67@bytedance.com To: alex.williamson@redhat.com, akpm@linux-foundation.org, david@redhat.com, jgg@ziepe.ca, peterx@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lizhe.67@bytedance.com Subject: [PATCH v3 2/5] vfio/type1: optimize vfio_pin_pages_remote() Date: Mon, 7 Jul 2025 14:49:47 +0800 Message-ID: <20250707064950.72048-3-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250707064950.72048-1-lizhe.67@bytedance.com> References: <20250707064950.72048-1-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Li Zhe When vfio_pin_pages_remote() is called with a range of addresses that includes large folios, the function currently performs individual statistics counting operations for each page. This can lead to significant performance overheads, especially when dealing with large ranges of pages. Batch processing of statistical counting operations can effectively enhance performance. In addition, the pages obtained through longterm GUP are neither invalid nor reserved. Therefore, we can reduce the overhead associated with some calls to function is_invalid_reserved_pfn(). The performance test results for completing the 16G VFIO IOMMU DMA mapping are as follows. Base(v6.16-rc4): Acked-by: David Hildenbrand ------- AVERAGE (MADV_HUGEPAGE) -------- VFIO MAP DMA in 0.047 s (340.2 GB/s) ------- AVERAGE (MAP_POPULATE) -------- VFIO MAP DMA in 0.280 s (57.2 GB/s) ------- AVERAGE (HUGETLBFS) -------- VFIO MAP DMA in 0.052 s (310.5 GB/s) With this patch: ------- AVERAGE (MADV_HUGEPAGE) -------- VFIO MAP DMA in 0.027 s (602.1 GB/s) ------- AVERAGE (MAP_POPULATE) -------- VFIO MAP DMA in 0.257 s (62.4 GB/s) ------- AVERAGE (HUGETLBFS) -------- VFIO MAP DMA in 0.031 s (517.4 GB/s) For large folio, we achieve an over 40% performance improvement. For small folios, the performance test results indicate a slight improvement. Signed-off-by: Li Zhe Co-developed-by: Alex Williamson Signed-off-by: Alex Williamson --- drivers/vfio/vfio_iommu_type1.c | 83 ++++++++++++++++++++++++++++----- 1 file changed, 71 insertions(+), 12 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type= 1.c index 1136d7ac6b59..03fce54e1372 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -318,7 +318,13 @@ static void vfio_dma_bitmap_free_all(struct vfio_iommu= *iommu) /* * Helper Functions for host iova-pfn list */ -static struct vfio_pfn *vfio_find_vpfn(struct vfio_dma *dma, dma_addr_t io= va) + +/* + * Find the highest vfio_pfn that overlapping the range + * [iova_start, iova_end) in rb tree. + */ +static struct vfio_pfn *vfio_find_vpfn_range(struct vfio_dma *dma, + dma_addr_t iova_start, dma_addr_t iova_end) { struct vfio_pfn *vpfn; struct rb_node *node =3D dma->pfn_list.rb_node; @@ -326,9 +332,9 @@ static struct vfio_pfn *vfio_find_vpfn(struct vfio_dma = *dma, dma_addr_t iova) while (node) { vpfn =3D rb_entry(node, struct vfio_pfn, node); =20 - if (iova < vpfn->iova) + if (iova_end <=3D vpfn->iova) node =3D node->rb_left; - else if (iova > vpfn->iova) + else if (iova_start > vpfn->iova) node =3D node->rb_right; else return vpfn; @@ -336,6 +342,11 @@ static struct vfio_pfn *vfio_find_vpfn(struct vfio_dma= *dma, dma_addr_t iova) return NULL; } =20 +static inline struct vfio_pfn *vfio_find_vpfn(struct vfio_dma *dma, dma_ad= dr_t iova) +{ + return vfio_find_vpfn_range(dma, iova, iova + PAGE_SIZE); +} + static void vfio_link_pfn(struct vfio_dma *dma, struct vfio_pfn *new) { @@ -614,6 +625,39 @@ static long vaddr_get_pfns(struct mm_struct *mm, unsig= ned long vaddr, return ret; } =20 + +static long vpfn_pages(struct vfio_dma *dma, + dma_addr_t iova_start, long nr_pages) +{ + dma_addr_t iova_end =3D iova_start + (nr_pages << PAGE_SHIFT); + struct vfio_pfn *top =3D vfio_find_vpfn_range(dma, iova_start, iova_end); + long ret =3D 1; + struct vfio_pfn *vpfn; + struct rb_node *prev; + struct rb_node *next; + + if (likely(!top)) + return 0; + + prev =3D next =3D &top->node; + + while ((prev =3D rb_prev(prev))) { + vpfn =3D rb_entry(prev, struct vfio_pfn, node); + if (vpfn->iova < iova_start) + break; + ret++; + } + + while ((next =3D rb_next(next))) { + vpfn =3D rb_entry(next, struct vfio_pfn, node); + if (vpfn->iova >=3D iova_end) + break; + ret++; + } + + return ret; +} + /* * Attempt to pin pages. We really don't want to track all the pfns and * the iommu can only map chunks of consecutive pfns anyway, so get the @@ -680,32 +724,47 @@ static long vfio_pin_pages_remote(struct vfio_dma *dm= a, unsigned long vaddr, * and rsvd here, and therefore continues to use the batch. */ while (true) { + long nr_pages, acct_pages =3D 0; + if (pfn !=3D *pfn_base + pinned || rsvd !=3D is_invalid_reserved_pfn(pfn)) goto out; =20 + /* + * Using GUP with the FOLL_LONGTERM in + * vaddr_get_pfns() will not return invalid + * or reserved pages. + */ + nr_pages =3D num_pages_contiguous( + &batch->pages[batch->offset], + batch->size); + if (!rsvd) { + acct_pages =3D nr_pages; + acct_pages -=3D vpfn_pages(dma, iova, nr_pages); + } + /* * Reserved pages aren't counted against the user, * externally pinned pages are already counted against * the user. */ - if (!rsvd && !vfio_find_vpfn(dma, iova)) { + if (acct_pages) { if (!dma->lock_cap && - mm->locked_vm + lock_acct + 1 > limit) { + mm->locked_vm + lock_acct + acct_pages > limit) { pr_warn("%s: RLIMIT_MEMLOCK (%ld) exceeded\n", __func__, limit << PAGE_SHIFT); ret =3D -ENOMEM; goto unpin_out; } - lock_acct++; + lock_acct +=3D acct_pages; } =20 - pinned++; - npage--; - vaddr +=3D PAGE_SIZE; - iova +=3D PAGE_SIZE; - batch->offset++; - batch->size--; + pinned +=3D nr_pages; + npage -=3D nr_pages; + vaddr +=3D PAGE_SIZE * nr_pages; + iova +=3D PAGE_SIZE * nr_pages; + batch->offset +=3D nr_pages; + batch->size -=3D nr_pages; =20 if (!batch->size) break; --=20 2.20.1 From nobody Tue Oct 7 19:55:41 2025 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 294CA27A451 for ; Mon, 7 Jul 2025 06:50:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751871025; cv=none; b=jSuqEnHEu0eQcnr9WDQ9D68wgSXdUTCXq0RTnqfj2ptKwQfaku+Yv9ltiLgrak8RlrxagQMR0/7PhhXCtSNr60QhT+yKNjC6gR9ZniaqJKFB7HgV8H5NkhIq2DJPCw8XcFFzRjS7b7Bbw/kU9GrrMtmo3THPZNoXs4ijQBBKodA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751871025; c=relaxed/simple; bh=2Ck7da429bj6tPACUdRedQc7kvHlhF/GdsKaHEkLWBk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aZsBWMwBdnHl6rg439e7vQU7HIahmrbEdQVpvWQFSHHg7DIopj5JC0uM4EZcjZHCxsAhX2x9sx6YoiInyIYWcRaifgRjkQaIPVHFWT4kJ17eu5x/2ZQqV06TFpveH5vdaIOHmaDGUyort9sclGvqCrlmyYD0+lCmgIfMlJ6oDoc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=OepFZ09r; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="OepFZ09r" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-234fcadde3eso40772865ad.0 for ; Sun, 06 Jul 2025 23:50:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1751871024; x=1752475824; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QiHfvwOVI0/DzO8i0IezA1Z19+xb67OoONslKakPCGo=; b=OepFZ09ry0Y45kaLbmmVB9g2ZxOIGKThy4FQfyDq2cijzMnT7B/OupDrRWOOK+E/pV znqaUrRUGwksQFKcDkPduIMHfHZzpMWNSiwiIrTfy+vVXsY5ID1rj75AiLkr4UhCUOg6 SRyt4THFnpg/vDLT2QOBm3p4FPyTV9D2gmoS8OeUmR/REj9kjxQXUPm3N0UhJ5h30Kfs QeHZMv6LWkHO370X1h5a4uHq9M3tuQj1o9526MAXzjL+5hLpQoG0g27klR81CWrd8Gvk 9fzM3KnxI+VaWeM7SbiTydNOtHswL+S2wmP0058b9cmQI/ezElMcTb8lzqP+A6gWQDZL MLtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751871024; x=1752475824; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QiHfvwOVI0/DzO8i0IezA1Z19+xb67OoONslKakPCGo=; b=G+BAJr92ZYUMZr8u5UOJ1slk9yymyA6BE/mbDU2t6pNvPpGeqBs3OWH4UESz5l1k1s F7wcy7bvhry1iP9Vap1HseG6/9WDr/+TWxH8/gY2ySyq/4OVDoY6iVrdC/deAOAeiReO g9VLCpZX/BMal3TRjJlw8tY/RLDtnM9EfLdFmCIsHjPTBwZ1YkpQXVsLcT3iFT4YfhGX wm6j21S3Hd9oqAkAKHV300WUwcZDGaak697+4CxriBmbhs7UHTCHEt5fiir+0YxpOFNi R3ozHCMxxEZZ3IX8i1iUk7HxZFKwzXxsZZBim5a6Df1JmWTcdLJ5JDeqkToeX8pAeHcs 9HXg== X-Forwarded-Encrypted: i=1; AJvYcCWNFrb+uoE05EKZe16mwoAlSIQHtAzjKuI5r72rijYNMi1bp8W2l8Qh/PTu60gmtlLWQcLtJRphcFdaXOA=@vger.kernel.org X-Gm-Message-State: AOJu0Yzz3J5nGjKJJNc0cnINnCoLe1RO8zeOHIvCbCk3xupu0t3HArGL gY2CDqfwz9brL7k/UPuOrXnXb1X+mn0e0VOkU/P423aAH+RBeCFHh/EUPZKPqauxdHQ= X-Gm-Gg: ASbGncsIPikx5fwEXmXkWFV2hhUNPvNMReKbVbDHTGI+RmNmBzkxJyfyKPU5IDg9C7f GkV8uOjkJS5yD3h3t+kvSOOWKUb4iNTFZfqkkdLUK23vdGhzRsB4LJVm15Jzq6o/PbSI74A1eDs d9yaYInBOZleYT7KtpnAbc0PVkO1sKDtZxd8PkP7JdUt05CfTBhsDfm27gmG2tgbEOseut/gsRo J+7d4L+jBdiOI4P7FFFiKWN70dUtNtVKjeVfqjrtyHtQJXwNGS3CxrkmBb9QvC38ymJxylwRCFX FJKcMwIUxD0PuAAmSDtR7tniZdjUpQ9xvc3fwfxKy3d1OjmemQPzqpBCFPVBy0FRyklFPUyujyr uGGHTjZUMfuBqGbXjqT0QjFY= X-Google-Smtp-Source: AGHT+IFMBY0822v4rgg5psHQsiI89RRrF9TLtvLgFYQSnw9DJEUz57tWFMIxOBTqvqWWyVw9h1H5CA== X-Received: by 2002:a17:903:1b4d:b0:23a:bc47:381f with SMTP id d9443c01a7336-23c8759dbb3mr181052285ad.36.1751871023683; Sun, 06 Jul 2025 23:50:23 -0700 (PDT) Received: from localhost.localdomain ([203.208.189.9]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-23c8431a1aasm77377635ad.15.2025.07.06.23.50.20 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sun, 06 Jul 2025 23:50:23 -0700 (PDT) From: lizhe.67@bytedance.com To: alex.williamson@redhat.com, akpm@linux-foundation.org, david@redhat.com, jgg@ziepe.ca, peterx@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lizhe.67@bytedance.com Subject: [PATCH v3 3/5] vfio/type1: batch vfio_find_vpfn() in function vfio_unpin_pages_remote() Date: Mon, 7 Jul 2025 14:49:48 +0800 Message-ID: <20250707064950.72048-4-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250707064950.72048-1-lizhe.67@bytedance.com> References: <20250707064950.72048-1-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Li Zhe The function vpfn_pages() can help us determine the number of vpfn nodes on the vpfn rb tree within a specified range. This allows us to avoid searching for each vpfn individually in the function vfio_unpin_pages_remote(). This patch batches the vfio_find_vpfn() calls in function vfio_unpin_pages_remote(). Signed-off-by: Li Zhe --- drivers/vfio/vfio_iommu_type1.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type= 1.c index 03fce54e1372..b770eb1c4e07 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -794,16 +794,12 @@ static long vfio_unpin_pages_remote(struct vfio_dma *= dma, dma_addr_t iova, unsigned long pfn, unsigned long npage, bool do_accounting) { - long unlocked =3D 0, locked =3D 0; + long unlocked =3D 0, locked =3D vpfn_pages(dma, iova, npage); long i; =20 - for (i =3D 0; i < npage; i++, iova +=3D PAGE_SIZE) { - if (put_pfn(pfn++, dma->prot)) { + for (i =3D 0; i < npage; i++) + if (put_pfn(pfn++, dma->prot)) unlocked++; - if (vfio_find_vpfn(dma, iova)) - locked++; - } - } =20 if (do_accounting) vfio_lock_acct(dma, locked - unlocked, true); --=20 2.20.1 From nobody Tue Oct 7 19:55:41 2025 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 53CFC286430 for ; Mon, 7 Jul 2025 06:50:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751871031; cv=none; b=Ef/Xq2FBTJirlXQd5C/SSHIIXQo6/Su0y5s55ABcNtrFlpDzpv4hwFTyXon86vuFklaSbmFOV+R1UgNCneSghPrSXqbUTymrdtU1cKXvsxTie1pkuh6vrif04WCmrRn3UnC3qTvozc1eWg7GActrhWrwXM4CjcHa0LE7r4EdFUg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751871031; c=relaxed/simple; bh=/B2Cmebs2wx0RR0cS/Zre9jsbWIH/bpLwhd4AwWN4Zs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cjYEzXmEtuhsE6o/NZi1jaQ2SoANlEcpCztZOkZgyzx2oRd72MHgfMjqqEnGjq23nZkNwPu//tBGEIsgUJX2/i+eGAnl5OfRVnUlNxl27f3/KobhAkTRc56dAH/u7uNscqIy7v0+8dW2oLYoAx/Ktf5kPBx1off3lPFF71flU30= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=ZyRrB7mX; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="ZyRrB7mX" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-236377f00easo34413035ad.1 for ; Sun, 06 Jul 2025 23:50:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1751871029; x=1752475829; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=90En1W3zs0tALEbbRQaodK/vejigNx50ns8gS3LDvBw=; b=ZyRrB7mXt4+uVJYdLr7VzAMd8qE6RFZp1yalP7Zp5KCelemeFiUsUzcea0CHlZyA2z y/np/lMT6W8Xywhht8pfKxEz0n4EBT3pG6IKynHHjy/wRdEhOwAwU8o1awIRg6rIOjNA eOOFlKR8pqhaVO/ntznKeNXNDEB4+pcX6albrPxw0U9xYCdqDJ/5AuYkeSgAPKCaFtzm 1u4KqnKgypID2+ER0d2PeR5T2R2zSzy8p0mXT18GQ+q0TcEW9og4Dqs96syz3swbJSE+ PgHjrk2vzSmWNIzbiKKBheJ9ZBJe/j96kAd7uq0WJ+jjIw7qJFOTsRt9cMeVgpeKr91M Lchw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751871029; x=1752475829; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=90En1W3zs0tALEbbRQaodK/vejigNx50ns8gS3LDvBw=; b=E2w916tH+X0KFbEI7CPquQ4UtXSZt9M99gACXVLtho20pPuraYq7H0FpNjZVJX+YRb uKdb4pC/rwOGc0Gd8mHXTd73+RGofxU8kTt2Zok4NegC++u2jy1R5EoWeTU2ZgMN5sTX 0x9TjmV5/G935ngZwV7pK4wh4H4TQiD70YK8d3rgKYgbAEwdWGMd8B+Z8g9iR3azzOXS /R+8TwAZCm3De/3b43QevznKa10Gm3ft0D7WeJGPwjkgTs+WW8Qj85saOVCaE5rruCp4 Jn5Auuy/TjJ5vnIuPnEdFCEm3XWDyvHvRLMq/Qm80wRyl65l7XlfzJxQGW8iA9+4K9X+ Z7BQ== X-Forwarded-Encrypted: i=1; AJvYcCXaXC9ZBg39FpZHyD4Hn0Hu1bORWv1gA97371xtMmDmB/ndgi6veUBFjKAC+O1arzjR2Yr6oeh6gWD1j0c=@vger.kernel.org X-Gm-Message-State: AOJu0YxdTQU4sANKheuUlLt5Dx9bFm5GszEfopvbstuOT+KBF0ahLawa VJbvBQ/4qnzekVRtiea/3aF5SYeDf1ZeM8fQYdztYM5M7CqKwWH+jb6Hpn5ff5BOg1EnD6doXQz gBizI X-Gm-Gg: ASbGnct0GcI9qbCuOhbIVNDsMjuerud1qn1t8Krq662YmCkHHDkvbNzBZbS+MEGJYy7 nzUbyhZsPAygtwsJ++Xum0BWku28cAwGSM2Xv33nOcSVRlLpDpqHnVEB8cqlM4EP/bXuVa8jJQf XdbAlh4veDe1vlc0uhukv2JYqX6pgSMkKg4G0irycYMs9QwPZEneGswCQzB91HwVSjx7dyKkWXm qO60pMPiIe9yrxlS7g4ujk8jSFhGt1YJc44SeSr1tj+WOJWxlO/jOGBe7z24B2XW/uKcp8oMphw Iix6O6vvlFNeDLODNY5OfXw6gTTVR036Z6NlEClJ0ssNC9y1pA1xZP7iWEsOckiUtnU0pmgY3KC T9mZfpa+XS0JJ X-Google-Smtp-Source: AGHT+IGk089Voe8sYujhmiUPCIl62YdQ2hxYB5OqRb9ETbkQLoZK53+o1hrm29Rn78YQ81hdAXergg== X-Received: by 2002:a17:903:1446:b0:235:e942:cb9c with SMTP id d9443c01a7336-23c85d9f014mr192498945ad.5.1751871029513; Sun, 06 Jul 2025 23:50:29 -0700 (PDT) Received: from localhost.localdomain ([203.208.189.9]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-23c8431a1aasm77377635ad.15.2025.07.06.23.50.25 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sun, 06 Jul 2025 23:50:29 -0700 (PDT) From: lizhe.67@bytedance.com To: alex.williamson@redhat.com, akpm@linux-foundation.org, david@redhat.com, jgg@ziepe.ca, peterx@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lizhe.67@bytedance.com Subject: [PATCH v3 4/5] vfio/type1: introduce a new member has_rsvd for struct vfio_dma Date: Mon, 7 Jul 2025 14:49:49 +0800 Message-ID: <20250707064950.72048-5-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250707064950.72048-1-lizhe.67@bytedance.com> References: <20250707064950.72048-1-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Li Zhe Introduce a new member has_rsvd for struct vfio_dma. This member is used to indicate whether there are any reserved or invalid pfns in the region represented by this vfio_dma. If it is true, it indicates that there is at least one pfn in this region that is either reserved or invalid. Reviewed-by: David Hildenbrand Signed-off-by: Li Zhe --- drivers/vfio/vfio_iommu_type1.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type= 1.c index b770eb1c4e07..13c5667d431c 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -92,6 +92,7 @@ struct vfio_dma { bool iommu_mapped; bool lock_cap; /* capable(CAP_IPC_LOCK) */ bool vaddr_invalid; + bool has_rsvd; /* has 1 or more rsvd pfns */ struct task_struct *task; struct rb_root pfn_list; /* Ex-user pinned pfn list */ unsigned long *bitmap; @@ -774,6 +775,7 @@ static long vfio_pin_pages_remote(struct vfio_dma *dma,= unsigned long vaddr, } =20 out: + dma->has_rsvd |=3D rsvd; ret =3D vfio_lock_acct(dma, lock_acct, false); =20 unpin_out: --=20 2.20.1 From nobody Tue Oct 7 19:55:41 2025 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 293F9286430 for ; Mon, 7 Jul 2025 06:50:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751871036; cv=none; b=m3El5RExg/q3Je1kBAuBAtNMCIvHlLhDKgdB6h2kk8RLZoerc/HMlzkchHdS3cUNRBGkZX5M0VQ6TERTdXV5CfsyAT7ihrcM6FPW/NInKhbU3N423g9bEmQ3oWuXZMtI2e1qshvvmmbCGljWmKagm0eKjXrIHTL2jVhVNyiMHKw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751871036; c=relaxed/simple; bh=ls52+XrvwO/234TukpAvew2O69meLJpK2+6xI0hEEfk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=io/9C7opFxhRzp+bDwWoZ2TPSbc9Bbez4FPgTHSL4CWg27KlZFPtMdGXT2dmLbB++afLp/mQMVUqj6biAAcFl4a656vrE3jLfmF5qpe6+RgKYRqnTw7WljCz9epwZvwvbQKNaz3ckZzQkr6+0fBA4AmsX9bSCe0M7ycClmzFtsw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=b+5Q91nP; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="b+5Q91nP" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-23636167afeso24619335ad.3 for ; Sun, 06 Jul 2025 23:50:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1751871034; x=1752475834; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DJCrtQpscf7JigPaIY2b7xTm+tWVRMcaJVrpHVo23Lg=; b=b+5Q91nPBQWTJvIg/RM8snOHNnHpph0Inl1kCuZRedVF3VT2VC+Ufkk73pHkvshwNn KmEHWH3XafI6BHoD222KeD/9bbw55wZEVOKZCYI+zcGoACktQK9+o8snVlrOCBI0Vj80 NcgPTt0clkEB5+mwNhxojCrwxKCKDx9p+k/L2tyVmqXc3Ry2m6FHwe+8FzLvqmsrc/IX AroBlwFX2zTwm9j2C4VY2BEhk9OfRqb+7HAJaIcVnKV2lyJgPBI2DJwlrdNnwqDBlbvb sHZhBa2hGDpD6O1YTpKNRGL/kPaJgpTJz0+Q8bDb+4OE2GpesTeN+LFtBWauNoDgZ4Ly v4Qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751871034; x=1752475834; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DJCrtQpscf7JigPaIY2b7xTm+tWVRMcaJVrpHVo23Lg=; b=TMKf21jEf6B4PLmgU+80jncJ8LzMjDflUZcJgl2N6yRpJrK7j4ivwtU3Pa8ba6LoeQ RQS4aKIWz58TgEAMWrot9YunkLDMt3T2bKt9TSMKCUgbQ4aznUDglv/bEUO0rD885HUf lqNHxThwyf86OD0qoMs6hmeCJh6EI1iS1xcBdp/wyTKeWK0KjpIUF8h5WAr7FpQdn810 jmIijtwwLFlzdggqoBbZMHnA8oEwCWTe1ZAvefSkgIJ/eJDt91nkTGKjonMnvs0IuuGr Sn1hBULIwd8RHctiAjPAS1H3NbN0FfhYESklY29s5z4fiDxNg6zOP5+YvoV+oU/l3ijN do2g== X-Forwarded-Encrypted: i=1; AJvYcCVjjZzGUoMIu+2V9emOVC0Qcy7xj+zOUIWK44G5a5t5SDKffRGe1QdMvQwiJq8f/bskg0+pF9E8yckvLx8=@vger.kernel.org X-Gm-Message-State: AOJu0YwvyP+5g7UspPQztEVkD7ut/2jCzjv25zPcK/H3vjFtnOMEjh7/ cC1Y6Br6sFks20Dna9ZbedWE/y9YDSruLEIVxuv1i7Y1VS7UOEeYuUnfrc6UBd6sQwQ= X-Gm-Gg: ASbGncuITrYroM38mo9Z7GdOPA2YT/9nhuiGJPkPWxeflw4YZHIOYaAlE9RlRrRW2ge e1uIz0cPEofEvkNNA03fGkaMFbFop1kgSLOZCBwhsjaDBIe/DQf3QHJbpKTp/5Cf16JbWPlU7Hz OUl/zG78ndB0UE/gfVD3MWLrH3OGNEsMbdE6CYEAYw4sGX6EhLX9DxYvfVMQpgvTuZWsi7zHK0q ddPukKoXQh79Vb+GDnyveyKNjeM+OqPL6oPfkpm2zTyALgGAlCD5iRogL/uV1eAyqDmRFLk5IeR 6FaNLAtQypfnzQIDDOEQ8SKrfD7yzMCTVniXsdN+xBNsVmE1jsTxp7X7zBDuev1P9WGh/Ahlp64 aO8rD/14m1THXQjTKcU2hZQw= X-Google-Smtp-Source: AGHT+IGh2/KBCNysHt7plUCFAgV1uGLkxyUZAwvcA0Vu+vwTRDqonN+b71HZNp4xw4YPISfBYqMzdA== X-Received: by 2002:a17:902:e5d0:b0:234:c8ec:51b5 with SMTP id d9443c01a7336-23c875e4d89mr159552995ad.53.1751871034310; Sun, 06 Jul 2025 23:50:34 -0700 (PDT) Received: from localhost.localdomain ([203.208.189.9]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-23c8431a1aasm77377635ad.15.2025.07.06.23.50.30 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sun, 06 Jul 2025 23:50:34 -0700 (PDT) From: lizhe.67@bytedance.com To: alex.williamson@redhat.com, akpm@linux-foundation.org, david@redhat.com, jgg@ziepe.ca, peterx@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lizhe.67@bytedance.com Subject: [PATCH v3 5/5] vfio/type1: optimize vfio_unpin_pages_remote() Date: Mon, 7 Jul 2025 14:49:50 +0800 Message-ID: <20250707064950.72048-6-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250707064950.72048-1-lizhe.67@bytedance.com> References: <20250707064950.72048-1-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Li Zhe When vfio_unpin_pages_remote() is called with a range of addresses that includes large folios, the function currently performs individual put_pfn() operations for each page. This can lead to significant performance overheads, especially when dealing with large ranges of pages. It would be very rare for reserved PFNs and non reserved will to be mixed within the same range. So this patch utilizes the has_rsvd variable introduced in the previous patch to determine whether batch put_pfn() operations can be performed. Moreover, compared to put_pfn(), unpin_user_page_range_dirty_lock() is capable of handling large folio scenarios more efficiently. The performance test results for completing the 16G VFIO IOMMU DMA unmapping are as follows. Base(v6.16-rc4): ./vfio-pci-mem-dma-map 0000:03:00.0 16 Acked-by: David Hildenbrand Suggested-by: Jason Gunthorpe ------- AVERAGE (MADV_HUGEPAGE) -------- VFIO UNMAP DMA in 0.135 s (118.6 GB/s) ------- AVERAGE (MAP_POPULATE) -------- VFIO UNMAP DMA in 0.312 s (51.3 GB/s) ------- AVERAGE (HUGETLBFS) -------- VFIO UNMAP DMA in 0.136 s (117.3 GB/s) With this patchset: ------- AVERAGE (MADV_HUGEPAGE) -------- VFIO UNMAP DMA in 0.045 s (357.0 GB/s) ------- AVERAGE (MAP_POPULATE) -------- VFIO UNMAP DMA in 0.288 s (55.6 GB/s) ------- AVERAGE (HUGETLBFS) -------- VFIO UNMAP DMA in 0.045 s (353.9 GB/s) For large folio, we achieve an over 66% performance improvement in the VFIO UNMAP DMA item. For small folios, the performance test results appear to show a slight improvement. Suggested-by: Jason Gunthorpe Signed-off-by: Li Zhe --- drivers/vfio/vfio_iommu_type1.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type= 1.c index 13c5667d431c..208576bd5ac3 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -792,17 +792,29 @@ static long vfio_pin_pages_remote(struct vfio_dma *dm= a, unsigned long vaddr, return pinned; } =20 +static inline void put_valid_unreserved_pfns(unsigned long start_pfn, + unsigned long npage, int prot) +{ + unpin_user_page_range_dirty_lock(pfn_to_page(start_pfn), npage, + prot & IOMMU_WRITE); +} + static long vfio_unpin_pages_remote(struct vfio_dma *dma, dma_addr_t iova, unsigned long pfn, unsigned long npage, bool do_accounting) { long unlocked =3D 0, locked =3D vpfn_pages(dma, iova, npage); - long i; =20 - for (i =3D 0; i < npage; i++) - if (put_pfn(pfn++, dma->prot)) - unlocked++; + if (dma->has_rsvd) { + unsigned long i; =20 + for (i =3D 0; i < npage; i++) + if (put_pfn(pfn++, dma->prot)) + unlocked++; + } else { + put_valid_unreserved_pfns(pfn, npage, dma->prot); + unlocked =3D npage; + } if (do_accounting) vfio_lock_acct(dma, locked - unlocked, true); =20 --=20 2.20.1