From nobody Tue Oct 7 23:15:02 2025 Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 303922459D7 for ; Fri, 4 Jul 2025 06:26:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751610401; cv=none; b=aqvuUbMjAN0ZpkU4tKO5kDeNpAa0YfrHIZ43oPddVPpkxZEfVZ9fkQWTvyQn7VL7vw2PF3gUXxaM1g0pQYFiMd7bHtsaJG6o7By9HjYb7ubJdfIsfPHzSQE1l9MBNffDDuZW7DJ3TrAxwgiMlclKtwFhye6YJSxFzjZ34CCtFbM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751610401; c=relaxed/simple; bh=zFqecOrZpi0qTmPninPaSLMJPvbB/zlF0VDVWE7x3aQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BXEuTSlUQK5WYs08NIIUoxVSTqXEM2wW1gag7Lso5tLx0zgf6SLwWe9e8yC/cKhu39Dg8Cd34GzeflBmEPrnJCyl53Dp6SxWD8TTJCCXt68rrhNf22wqyWeCYhzWGARHbPQ3Yjv+TqyulfEkBxG9O2j6TZarkavsFeJaOLLXl/A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=LtQWwl0s; arc=none smtp.client-ip=209.85.215.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="LtQWwl0s" Received: by mail-pg1-f178.google.com with SMTP id 41be03b00d2f7-b3182c6d03bso702869a12.0 for ; Thu, 03 Jul 2025 23:26:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1751610397; x=1752215197; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2g/dwRr6sLGaceuHHFqckJ3jttaQlwjxhPJ/ULf0C3w=; b=LtQWwl0si2ZHFo394eqK410kyFb8tsW3HTyufWP7iaT2Q51DMxCanf+MfmA3R3MGoV iM3i5VNNKAErYgX2N15Fy0JEytL44CxGlOR54bjZHS+iDtCMcoDeFNLREgSUM7r3PoF3 Mu9ovwUR2BoxSBATKgU0aJpA+K/RyLkfxYRMSnUEfUyfknnBlGNA49oY94FI2aYG3H6I b42JcHnmyrhdm0HYreMo3T8ShAzqDqD6mqZZOoYVegfBpVEDeYkJNwqbRCZXwG5Hu2tY dSDXCygU7t4Rynl0VgzzZ3kcAG3x/x48Z2hrJO6XpR/6dPLdZTgOLvXXvcc2b3zNYT08 Lqqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751610397; x=1752215197; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2g/dwRr6sLGaceuHHFqckJ3jttaQlwjxhPJ/ULf0C3w=; b=vpXXchm1c5Pu3waLeSavlaALp0cPKBe9PW9evT0u202nGFki1Z0WLcn6TVa/buzpp+ huUL0Rkws7twSrGvkqYxz7GZXPe/FFQe8npL4BIimr5hLdb59CLLpzK7BToOxjKNj17Z YgL5j5N8YOvkS5KVmAXnjgWLKz8ygfFr7oTJLNfwLg7UVlPPgINXf3jMvNzvHLphYF4x HDVvrzBQ5XIPTrUatbOT/DSDT5zqoFZfbGAGQzQEQ3qo0KxtaOnIw4vYIHXd1oED30GL O4tgr8FdK80uW5GaF/pUoOPknnaij8Fn97pwB7YBCbdFlG7R2stiCXnC+sWBQ8/wutUW kkHg== X-Forwarded-Encrypted: i=1; AJvYcCULSn/AB/ZcptCZ2y2tR24aLTBcAe0BkDbRHj1Lf7pQH07hDkDeMxTJp9m2ch3NFeKa4TYmcTowgMZ2M6w=@vger.kernel.org X-Gm-Message-State: AOJu0YwBtja3XB37UzSijzNFw3KkikLLp5Cp61zil5+Obp9/T1qqcAmj N5syDSYmHS2sEojEL80EyyjfutxtA6VDAiMYUlimcdKCrgf0CmLnBzLZA8tploi0PPvqrDXdN3d msL2O X-Gm-Gg: ASbGncvFoiN8zm7/s29WfqaGbCkaxl6PCvEeqmxxLkPX1JWEi0LwT6hX/lQOod85b9b BNO2SGU+rdlAK+N0SjyEuY64mpgnjOReCB6BxGwvkx0oN1kSxwvjVKLpNNoAvqJdngARvwmtWrH /FWkxsNLLYw818dOwtjj9O3n2MMJRNXkdBDa3bR+0e5lKWOPqYfkw2oYtrlOOb45E6JHn3YUUoD lqKyI6HG7kOFZ4sGIrTpp0PKauSTL1FDA1HGeuTrMcwn9Y8Bb+TSj6l94C4Ja0bjqKpjtUYuw/5 OelXj8ACkAA+19OkJwt2KpBqCC5b9B9eh77yZQLU7VGVtSVlI/GpWIAmfSVtdWVmV79ugotCe9m EDV59sh8AI6yt4kaYnkxWDIQ= X-Google-Smtp-Source: AGHT+IFgq2ivg/0MglwTFQ4WJrnXsHzXC1MwrFb4H0GFB2MQTzaMHFKRHwI7ymeiA/dv4aTbTUdETg== X-Received: by 2002:a05:6a20:9195:b0:203:bb65:995a with SMTP id adf61e73a8af0-225c08e0ffamr2890704637.30.1751610397524; Thu, 03 Jul 2025 23:26:37 -0700 (PDT) Received: from localhost.localdomain ([203.208.189.8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-b38ee5f643dsm1183240a12.37.2025.07.03.23.26.33 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 03 Jul 2025 23:26:37 -0700 (PDT) From: lizhe.67@bytedance.com To: alex.williamson@redhat.com, akpm@linux-foundation.org, david@redhat.com, peterx@redhat.com, jgg@ziepe.ca Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lizhe.67@bytedance.com Subject: [PATCH v2 1/5] mm: introduce num_pages_contiguous() Date: Fri, 4 Jul 2025 14:25:58 +0800 Message-ID: <20250704062602.33500-2-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250704062602.33500-1-lizhe.67@bytedance.com> References: <20250704062602.33500-1-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Li Zhe Function num_pages_contiguous() determine the number of contiguous pages starting from the first page in the given array of page pointers. VFIO will utilize this interface to accelerate the VFIO DMA map process. Suggested-by: David Hildenbrand Signed-off-by: Li Zhe Suggested-by: Jason Gunthorpe --- include/linux/mm.h | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0ef2ba0c667a..1d26203d1ced 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -205,6 +205,26 @@ extern unsigned long sysctl_admin_reserve_kbytes; #define folio_page_idx(folio, p) ((p) - &(folio)->page) #endif =20 +/* + * num_pages_contiguous() - determine the number of contiguous pages + * starting from the first page. + * + * @pages: an array of page pointers + * @nr_pages: length of the array + */ +static inline unsigned long num_pages_contiguous(struct page **pages, + unsigned long nr_pages) +{ + struct page *first_page =3D pages[0]; + unsigned long i; + + for (i =3D 1; i < nr_pages; i++) + if (pages[i] !=3D nth_page(first_page, i)) + break; + + return i; +} + /* to align the pointer to the (next) page boundary */ #define PAGE_ALIGN(addr) ALIGN(addr, PAGE_SIZE) =20 --=20 2.20.1 From nobody Tue Oct 7 23:15:02 2025 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 91DD324500A for ; Fri, 4 Jul 2025 06:26:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751610418; cv=none; b=DZxhtbpIE0nMLVQ89FSrAOxr549BRFR/nQvqkQBhHQrsN4WUUS/w2r8C5ZPXW1gaeGRQYu5I6A/qcxXOWZwCqLrXwl3M1H7jbdHqDnGcVgpMPZhN+oTtn3He2qV030acWCt/+kcVQoRK8z3psmemMtvfex/0IM9h55d2DZ6Pffg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751610418; c=relaxed/simple; bh=fb6co8jrIb0ye1DMUwvR0cGT/E/Yrsdll1Jah3tS+yU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sp82SqZ6H3jpuRGYCskmMDp6muy0EvzcgRAzRAs6wsSZz+Jym0Uy6Re1NQeHjomkOCKZX4xxUw8T1CbJBjtiLtawrncHi2Lv86soWfgnwZ5AAchC3yrmFocdaullCA1yybg4nRBrk1yb+kk3DmdFmulXtxUc0CZF3XN99IlemB4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=hDWAGzAD; arc=none smtp.client-ip=209.85.215.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="hDWAGzAD" Received: by mail-pg1-f171.google.com with SMTP id 41be03b00d2f7-af51596da56so515844a12.0 for ; Thu, 03 Jul 2025 23:26:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1751610415; x=1752215215; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=e8Cadi1skh4P1ApGM+PqU48O5jgE1uMvu2URGbuzsOs=; b=hDWAGzADIs12fKOuKp1YQk8/cmpW8Wy8feJUMGY8L1ky4VsHkAAh+Ur/+Xfbhy8EdQ QxjWqPi7Ft1sxr/dssAI5S2b6wKqq0Q457kkv19utarqRUpFljfvNrD8yy9ppSNHCoyq EDw80HoHfDo2SVwU8m5PcVD+BsHtmd3mSy4Sr/hlzHC/DpfAUsgtHgtqQ0/knfgNqFlD Jlczyw4gTrSf+ihu51OfJhideDzLmPl6xq2BLLdSMj26op3j+lwk4OJkk7QeMz5cLex4 qsnlCrGb1u2hTirvVzvIiVhbIw8xq7/xrZd1UJ6+xOABpfH93aCA0dZmfMHS8kro5cfP LpEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751610415; x=1752215215; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=e8Cadi1skh4P1ApGM+PqU48O5jgE1uMvu2URGbuzsOs=; b=Upj/shecACZL+2Oa/ViVbp2OwuPZFSxzeFKuCkqJDB//9cdMvGvOrGmVcuPzL58cyk PZlCvk5KJrBHIh0SWzTLN9oR8by2i4uxAlFrPftqJ0ILgDj05ol3SCCzWaSwdTHdA0Mb pZm8FfZAraaeN9fh9gJrP8TJkX4No8Dnh9uB3GhjoCKFg91/zexwQ6b39kiD6WAnipRi no0elsb0NJ+eCP3LjWhjmzWjixQBAJwzElcOlxgXOTCjDqI/pYagZEfwGyQf+rMTjZOI SnZ150bykDwXcBtfs1He26nWH7JXz9PoGcf+zW25VoSu0B3ec7NYWSxY01bH1LS957Ft a+rg== X-Forwarded-Encrypted: i=1; AJvYcCXkHQwz8lmHnYWVnaAjFgBom3joV3djBa4E0INeP2mIEsqIFtAZY/Iz4sajkNA2SZj1UOQBUhMyiyA8d5Y=@vger.kernel.org X-Gm-Message-State: AOJu0YzACWDiYZ57YTVmnAIAEkipNhNTjdIueLyjntItxSTKisODNOkr qqT5XkscittHFwc+aGT3CYGVkvYFmgeyORH8S8TfaDsibpKY2LvFPu8Nyj4/TZE/8LU= X-Gm-Gg: ASbGncu5MUOxHBoHokQsgovbj6QpSpLBTkDlwIhgaOvsnWQJ3D25IKoDP/07E/PScsQ xIH+7iwYfm7gZ9AlBhrxgs8+7f9lztfFJQoSlOxoQjvuOyHiDar/+xNcl6C/YgZ6lSV/DmW8WX6 4SCmFgSwWG/8LAxiO8zfOBLi+9V8phbJs1cRe4gMKqqm0n9nnTHVn4Sgjgub3fQKiaxPKnfaQr/ gratw8gJj39/Yt8+orG5CXdR1svFnAjmRXI3acH38T1XuaIOTASKxSs9WjxhFNY0PVWWA5C0k3w dCeLwbozQ/ChwfWYXj4JBRWHSnfM3eJx3PRAv99XVAA4EdlnAAPDLKN0uSc0GTw/FwqdQwFI+I3 ka6Z2RPUDM0Ne X-Google-Smtp-Source: AGHT+IHFaMJdkIhohSafckGbE8EHuJ/2ezNmEPpqd4C13bAJDIW+YKMzKbMEQKLN3/tjGWY/2GuvOw== X-Received: by 2002:a05:6a21:3a85:b0:204:4573:d855 with SMTP id adf61e73a8af0-226092b0ea7mr1255567637.9.1751610414723; Thu, 03 Jul 2025 23:26:54 -0700 (PDT) Received: from localhost.localdomain ([203.208.189.8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-b38ee5f643dsm1183240a12.37.2025.07.03.23.26.51 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 03 Jul 2025 23:26:54 -0700 (PDT) From: lizhe.67@bytedance.com To: alex.williamson@redhat.com, akpm@linux-foundation.org, david@redhat.com, peterx@redhat.com, jgg@ziepe.ca Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lizhe.67@bytedance.com Subject: [PATCH v2 2/5] vfio/type1: optimize vfio_pin_pages_remote() Date: Fri, 4 Jul 2025 14:25:59 +0800 Message-ID: <20250704062602.33500-3-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250704062602.33500-1-lizhe.67@bytedance.com> References: <20250704062602.33500-1-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Li Zhe When vfio_pin_pages_remote() is called with a range of addresses that includes large folios, the function currently performs individual statistics counting operations for each page. This can lead to significant performance overheads, especially when dealing with large ranges of pages. Batch processing of statistical counting operations can effectively enhance performance. In addition, the pages obtained through longterm GUP are neither invalid nor reserved. Therefore, we can reduce the overhead associated with some calls to function is_invalid_reserved_pfn(). The performance test results for completing the 16G VFIO IOMMU DMA mapping are as follows. Base(v6.16-rc4): ------- AVERAGE (MADV_HUGEPAGE) -------- VFIO MAP DMA in 0.047 s (340.2 GB/s) ------- AVERAGE (MAP_POPULATE) -------- VFIO MAP DMA in 0.280 s (57.2 GB/s) ------- AVERAGE (HUGETLBFS) -------- VFIO MAP DMA in 0.052 s (310.5 GB/s) With this patch: ------- AVERAGE (MADV_HUGEPAGE) -------- VFIO MAP DMA in 0.027 s (602.1 GB/s) ------- AVERAGE (MAP_POPULATE) -------- VFIO MAP DMA in 0.257 s (62.4 GB/s) ------- AVERAGE (HUGETLBFS) -------- VFIO MAP DMA in 0.031 s (517.4 GB/s) For large folio, we achieve an over 40% performance improvement. For small folios, the performance test results indicate a slight improvement. Signed-off-by: Li Zhe Co-developed-by: Alex Williamson Signed-off-by: Alex Williamson --- drivers/vfio/vfio_iommu_type1.c | 83 ++++++++++++++++++++++++++++----- 1 file changed, 71 insertions(+), 12 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type= 1.c index 1136d7ac6b59..03fce54e1372 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -318,7 +318,13 @@ static void vfio_dma_bitmap_free_all(struct vfio_iommu= *iommu) /* * Helper Functions for host iova-pfn list */ -static struct vfio_pfn *vfio_find_vpfn(struct vfio_dma *dma, dma_addr_t io= va) + +/* + * Find the highest vfio_pfn that overlapping the range + * [iova_start, iova_end) in rb tree. + */ +static struct vfio_pfn *vfio_find_vpfn_range(struct vfio_dma *dma, + dma_addr_t iova_start, dma_addr_t iova_end) { struct vfio_pfn *vpfn; struct rb_node *node =3D dma->pfn_list.rb_node; @@ -326,9 +332,9 @@ static struct vfio_pfn *vfio_find_vpfn(struct vfio_dma = *dma, dma_addr_t iova) while (node) { vpfn =3D rb_entry(node, struct vfio_pfn, node); =20 - if (iova < vpfn->iova) + if (iova_end <=3D vpfn->iova) node =3D node->rb_left; - else if (iova > vpfn->iova) + else if (iova_start > vpfn->iova) node =3D node->rb_right; else return vpfn; @@ -336,6 +342,11 @@ static struct vfio_pfn *vfio_find_vpfn(struct vfio_dma= *dma, dma_addr_t iova) return NULL; } =20 +static inline struct vfio_pfn *vfio_find_vpfn(struct vfio_dma *dma, dma_ad= dr_t iova) +{ + return vfio_find_vpfn_range(dma, iova, iova + PAGE_SIZE); +} + static void vfio_link_pfn(struct vfio_dma *dma, struct vfio_pfn *new) { @@ -614,6 +625,39 @@ static long vaddr_get_pfns(struct mm_struct *mm, unsig= ned long vaddr, return ret; } =20 + +static long vpfn_pages(struct vfio_dma *dma, + dma_addr_t iova_start, long nr_pages) +{ + dma_addr_t iova_end =3D iova_start + (nr_pages << PAGE_SHIFT); + struct vfio_pfn *top =3D vfio_find_vpfn_range(dma, iova_start, iova_end); + long ret =3D 1; + struct vfio_pfn *vpfn; + struct rb_node *prev; + struct rb_node *next; + + if (likely(!top)) + return 0; + + prev =3D next =3D &top->node; + + while ((prev =3D rb_prev(prev))) { + vpfn =3D rb_entry(prev, struct vfio_pfn, node); + if (vpfn->iova < iova_start) + break; + ret++; + } + + while ((next =3D rb_next(next))) { + vpfn =3D rb_entry(next, struct vfio_pfn, node); + if (vpfn->iova >=3D iova_end) + break; + ret++; + } + + return ret; +} + /* * Attempt to pin pages. We really don't want to track all the pfns and * the iommu can only map chunks of consecutive pfns anyway, so get the @@ -680,32 +724,47 @@ static long vfio_pin_pages_remote(struct vfio_dma *dm= a, unsigned long vaddr, * and rsvd here, and therefore continues to use the batch. */ while (true) { + long nr_pages, acct_pages =3D 0; + if (pfn !=3D *pfn_base + pinned || rsvd !=3D is_invalid_reserved_pfn(pfn)) goto out; =20 + /* + * Using GUP with the FOLL_LONGTERM in + * vaddr_get_pfns() will not return invalid + * or reserved pages. + */ + nr_pages =3D num_pages_contiguous( + &batch->pages[batch->offset], + batch->size); + if (!rsvd) { + acct_pages =3D nr_pages; + acct_pages -=3D vpfn_pages(dma, iova, nr_pages); + } + /* * Reserved pages aren't counted against the user, * externally pinned pages are already counted against * the user. */ - if (!rsvd && !vfio_find_vpfn(dma, iova)) { + if (acct_pages) { if (!dma->lock_cap && - mm->locked_vm + lock_acct + 1 > limit) { + mm->locked_vm + lock_acct + acct_pages > limit) { pr_warn("%s: RLIMIT_MEMLOCK (%ld) exceeded\n", __func__, limit << PAGE_SHIFT); ret =3D -ENOMEM; goto unpin_out; } - lock_acct++; + lock_acct +=3D acct_pages; } =20 - pinned++; - npage--; - vaddr +=3D PAGE_SIZE; - iova +=3D PAGE_SIZE; - batch->offset++; - batch->size--; + pinned +=3D nr_pages; + npage -=3D nr_pages; + vaddr +=3D PAGE_SIZE * nr_pages; + iova +=3D PAGE_SIZE * nr_pages; + batch->offset +=3D nr_pages; + batch->size -=3D nr_pages; =20 if (!batch->size) break; --=20 2.20.1 From nobody Tue Oct 7 23:15:02 2025 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DEC57248F73 for ; Fri, 4 Jul 2025 06:27:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751610422; cv=none; b=G7wG0l+OZvG0jcMI2DQG//b4gFiUpa+i39XW2qE9/u2T6JP3sMFAFoZXbEtIm1fuWkjxHt8RLHm0eZJqNfLPgvhwVNnGe5R8JICMbk/9bKO8xc4NuLtSY+hlXziTR74BibUw87OuQhhfJUMbuxndz/HP9z+vfpvoxmFNl6XV8u8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751610422; c=relaxed/simple; bh=2Ck7da429bj6tPACUdRedQc7kvHlhF/GdsKaHEkLWBk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cgNso0m7pzU5DSIABqML0kcMftvXDUzmPoOr2vjZo97fhiqmEKHQlvknDzwRkX5lPfJHvWx2ZKQtZPhPn2d61LAjjDAn5Mu6er4NXj2vevyQ/ROmYsqGdg3J89tS+NRgk6Jk2eNzLsr1fCacpj27cj37GV6frBYWR8CTwCOzWvo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=arY4hajF; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="arY4hajF" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-235d6de331fso8989495ad.3 for ; Thu, 03 Jul 2025 23:27:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1751610420; x=1752215220; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QiHfvwOVI0/DzO8i0IezA1Z19+xb67OoONslKakPCGo=; b=arY4hajFESXaQEwzEyNLZeMh6o5+kMlYtLy3wdGrRrm1gmPg1PoSDJ1WhCGA2hDxAV 9U+n4+TgNX/alA3fZHaKUFoXUMsuFsf5ZzVJ29gx7eMueUUOxaFWqHCF35rZDWe20Jfr z5ZMcfQ6oz8fnuUphcxMRB7ePKzByXvU/y1u4hlreEEt++u+VTdXkwmEifbl2xOUUV2T FTGNZZnK7vvUtFXuJZDgaDSmsazcNbKv+xN0AeaY927S4GrDI6VNXzNP1wc3n6rjMxcL 3dnen0UP7QNA6Xa4pu+CvGZpCRgSrH1ArPImhhMwQpVoPSwIWI/BweL8jnn0Q8a5FGa/ 3lbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751610420; x=1752215220; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QiHfvwOVI0/DzO8i0IezA1Z19+xb67OoONslKakPCGo=; b=DRVkJUn7KhbGzvQqDYtej3PSo3V3GWqCDdjztytsufN9Tbk72t8hZ3IerJGtVg6QAQ ARIRQZupMsk3pTPj/mLLltj5ju74CrjvcLBItZ+aSI3h9YOth4cfX7I+AzkOCW5sQsyE 6bY1iv+Lh5/63zNSmsBRcX9iRYaYh3Yz+ANpsNt+Zm6NoUt7q4SiK5+1/PlfB5mVLfiF MOKQqj9uh7E/VokIYDLUvIHRqY0KZbNwbr39g5JZ1eP/HAGkbRDweDzgo3PMW4CwPW+e nPG7YOPe6218XJXGhNck7O/ZY+2l7r9q6xNWBCVTJtMfZOWBSrMNYhcsM8qlvvLtsVej ECsA== X-Forwarded-Encrypted: i=1; AJvYcCULzWz2MXGwFxxxXclKymcM5ZtWylvSmSTO9Ejdzcf5Zoj6AauJ1Ro+51Wrpyh/pN4trJr6bpo+1TiScJE=@vger.kernel.org X-Gm-Message-State: AOJu0YyIymkoxO8saJlPECGCql6RwiPSNKrqfXG6e+brS779gWr1RuRp 7VVifnXQD8bTtPSoEMg3++lYODbWCpe0CqXsRGgP+QoRq+pCU2pcyrvXf4njtz3O3rE= X-Gm-Gg: ASbGncuvFsOEoNtvaXfq74fAPs6KtDYxD1tOkOjb0Tgs+EJuw11kWEp95yYk+vGnH6I m4XL79kEvYOKc63KbPM3GMhuPTkZfUBTLSqpavFn3s/Hr4AdxspoJrk39w1oEcgpkansMA5A5Kd pSUo0rtg6p6U39OIevUjO40+fwQchLMdGF0GwAvNLUwfz2ULWrYFngU0jjrW/kE4w836wiMNwqu IZRijcI5A2FYGgKZOAT3ipvVYb/nrkBScZzM89aPYQzu8z7Y5F3D9txVtAo4g3+SpYAgTa0MJqS eCfVV71IG0YVTrGp313ilZIfjYn+ZbYgPdWw7MmOBwLS04jiS9QWhVR1N7TFeRNK1OiLR37Swi4 C3m4ueB+iB0Am X-Google-Smtp-Source: AGHT+IFXOrpM+FeRqzRx01wW+29lMpHFGQ2PU6Rtc5xxE9IkYr1rATfhqAIwp0ljrPlZY65KFR/JyQ== X-Received: by 2002:a17:902:d592:b0:235:799:eca5 with SMTP id d9443c01a7336-23c85eb051dmr27110365ad.44.1751610420130; Thu, 03 Jul 2025 23:27:00 -0700 (PDT) Received: from localhost.localdomain ([203.208.189.8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-b38ee5f643dsm1183240a12.37.2025.07.03.23.26.56 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 03 Jul 2025 23:26:59 -0700 (PDT) From: lizhe.67@bytedance.com To: alex.williamson@redhat.com, akpm@linux-foundation.org, david@redhat.com, peterx@redhat.com, jgg@ziepe.ca Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lizhe.67@bytedance.com Subject: [PATCH v2 3/5] vfio/type1: batch vfio_find_vpfn() in function vfio_unpin_pages_remote() Date: Fri, 4 Jul 2025 14:26:00 +0800 Message-ID: <20250704062602.33500-4-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250704062602.33500-1-lizhe.67@bytedance.com> References: <20250704062602.33500-1-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Li Zhe The function vpfn_pages() can help us determine the number of vpfn nodes on the vpfn rb tree within a specified range. This allows us to avoid searching for each vpfn individually in the function vfio_unpin_pages_remote(). This patch batches the vfio_find_vpfn() calls in function vfio_unpin_pages_remote(). Signed-off-by: Li Zhe --- drivers/vfio/vfio_iommu_type1.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type= 1.c index 03fce54e1372..b770eb1c4e07 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -794,16 +794,12 @@ static long vfio_unpin_pages_remote(struct vfio_dma *= dma, dma_addr_t iova, unsigned long pfn, unsigned long npage, bool do_accounting) { - long unlocked =3D 0, locked =3D 0; + long unlocked =3D 0, locked =3D vpfn_pages(dma, iova, npage); long i; =20 - for (i =3D 0; i < npage; i++, iova +=3D PAGE_SIZE) { - if (put_pfn(pfn++, dma->prot)) { + for (i =3D 0; i < npage; i++) + if (put_pfn(pfn++, dma->prot)) unlocked++; - if (vfio_find_vpfn(dma, iova)) - locked++; - } - } =20 if (do_accounting) vfio_lock_acct(dma, locked - unlocked, true); --=20 2.20.1 From nobody Tue Oct 7 23:15:02 2025 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A7B5324678F for ; Fri, 4 Jul 2025 06:27:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751610427; cv=none; b=WGrrZAOGlgbOn5kbEN1NW0gflggOjthi9r4Fy7McR0elWUYeohKLWuE/HnNW0dXaMTOVOktt6c5lvJgIx0nJXVU1aFXPF4o01/q15KM9fCU73urKwVZgWEmMIVCj00QGQ6CivElzQa2mbQ1i85QIU3Bt/ioYmriHSFXyMaxd7vU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751610427; c=relaxed/simple; bh=eqnxbsj14Y9W2BV2T0VsO5bpIYVtD7sCrCgDaNmvUzY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=T3WW+PJVTwVZJXbuyvIcXi9bsMvw4Jy9CnsDUcdPq46gn2aOk1kLtKnlTIQ/niX3mmtyo0ePxvZi3XU6M03jQzUFFWsn7F5wAQPzU7IByEcNREJpqssxo9jz0aNIL5b4G6Xox50zahZINWQmyrAn5n9FVFZjeFZFKyhRJAYN9yk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=aShmA0qo; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="aShmA0qo" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-7481600130eso970889b3a.3 for ; Thu, 03 Jul 2025 23:27:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1751610425; x=1752215225; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2/oMFootAUhR4O6zt87edsWksnJ6wDOqtWn4lJQpVn0=; b=aShmA0qoMQEmPCIoWDrBKAxKBFI8EzaOP8HY/6JTQ9LjLWLJzCRWHnHC+hDpVpItV1 AADMK6hbmeFQNmkgx0SBJ7XblU+TcZlZsYRiKIM5XLpxmOFSt4g4rkVThkUyh4XrMNca owfQY+pJ+Sgvb7zoKf0TSKZschT/U7pD8uIUNRbQqqrV+SuCkLlfSzkDc6zfSCoAa8s0 LhTnHg+2jkvQbkEAe6FZP1qxNlG4WzfdPHD8CyMmhTpSKmAUL4aZE9yVUsaTp/gp7xtA 40jWUe+RFBnyqaGk0hi5rCAmPtVRZ3T6WQenaTHzzycFsB6P33guaM5hGLgNHcoLPvrJ holw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751610425; x=1752215225; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2/oMFootAUhR4O6zt87edsWksnJ6wDOqtWn4lJQpVn0=; b=aM73scjuyHthp7CiB3qR9bJHMgMGId4WfX0L6s3Y5v2xbAQDsNSyhOIbvzGjesNC0K IQaDre/65fS2F+FKS5S6QH9kLnnq4wXoJvWVMEN7o5I0zg0bAtj7VPj3eQsh/cZAjOJm njZWP3dCOrCqPS5sARkmHp2e6uisHYf7toLlFSKCOKDsnj7HoWK/LUk/fGNEn4gTcJC1 kuv/03xdtxGTr84j5cOZI8btBuDi5IMUwR7Z2+T0G0zkngpmmEqsQ69vS4n+K/S2mdZu 46J4Cbf7uuACz6hzMegq3rWIejfLcXxTK9bjCOb7MSnX6pTQeCvHxDg0eUqWpfO2iGTt 7GqA== X-Forwarded-Encrypted: i=1; AJvYcCWSVWK5Qodaw1FrZ0OaFw19ncrkV0Z20Xul1lydImqtJLD7ze0XaXx1HjdxJZ8HAJmfIgvY7bxpUvuvYPE=@vger.kernel.org X-Gm-Message-State: AOJu0Yzpyt7qvAqVOTTd6Ebh4A6ffxk2cUC31Ge2QPr+SPdiHNzw7jNm Dbxedfk6TgzioT3d5lV8yjwC92sm+vcajH8xJdjRRSXK9QAV6dD5Es7JaD3svH2wcYo= X-Gm-Gg: ASbGncvJ95P0InmBuvkxhhDtwHVL5Qf67dTULnZWR+NgU79fHtdmSDXciTrlWQlGQAX 5Q/LGP9wqsf8GecrvRNMhInEzWmvnOqSuB1ZyuQ4yQw/yDD9YN3GtHmt9s8aGbuUeg4krPA7z1H b9Gif8+ucma1lNGw7LjNSOQgE7bGcWd84NGsuUKtDHHtCIBUDMXn2XkDVZBdqBdq6H8oNkWbMSV BwZThBTL1XEiKsgxt7xYY8pWHMu7JIW5NcaPqrX4GQX36/unaLhTd6QpEuzb0u47zU3au5oZStZ 12CMkxSEW3W1SsyiqWOtLelapHUA0mMsLkeQk+3+zVdii//JbA/JQ+PIIs5XfgBrqDpVcoMtHvk 5Ue3I5NIp65CS X-Google-Smtp-Source: AGHT+IG7T3GkhpHyXokGGwnbLUIOIE+bDXBFA0sb55px9wI3AAa5v6JbBWaMLj5Gi0BMN4In5dKGIw== X-Received: by 2002:a05:6a21:6b02:b0:215:d611:5d9b with SMTP id adf61e73a8af0-225b85f3f18mr2349868637.12.1751610425169; Thu, 03 Jul 2025 23:27:05 -0700 (PDT) Received: from localhost.localdomain ([203.208.189.8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-b38ee5f643dsm1183240a12.37.2025.07.03.23.27.01 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 03 Jul 2025 23:27:04 -0700 (PDT) From: lizhe.67@bytedance.com To: alex.williamson@redhat.com, akpm@linux-foundation.org, david@redhat.com, peterx@redhat.com, jgg@ziepe.ca Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lizhe.67@bytedance.com Subject: [PATCH v2 4/5] vfio/type1: introduce a new member has_rsvd for struct vfio_dma Date: Fri, 4 Jul 2025 14:26:01 +0800 Message-ID: <20250704062602.33500-5-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250704062602.33500-1-lizhe.67@bytedance.com> References: <20250704062602.33500-1-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Li Zhe Introduce a new member has_rsvd for struct vfio_dma. This member is used to indicate whether there are any reserved or invalid pfns in the region represented by this vfio_dma. If it is true, it indicates that there is at least one pfn in this region that is either reserved or invalid. Signed-off-by: Li Zhe Reviewed-by: David Hildenbrand --- drivers/vfio/vfio_iommu_type1.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type= 1.c index b770eb1c4e07..13c5667d431c 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -92,6 +92,7 @@ struct vfio_dma { bool iommu_mapped; bool lock_cap; /* capable(CAP_IPC_LOCK) */ bool vaddr_invalid; + bool has_rsvd; /* has 1 or more rsvd pfns */ struct task_struct *task; struct rb_root pfn_list; /* Ex-user pinned pfn list */ unsigned long *bitmap; @@ -774,6 +775,7 @@ static long vfio_pin_pages_remote(struct vfio_dma *dma,= unsigned long vaddr, } =20 out: + dma->has_rsvd |=3D rsvd; ret =3D vfio_lock_acct(dma, lock_acct, false); =20 unpin_out: --=20 2.20.1 From nobody Tue Oct 7 23:15:02 2025 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9CACF255E30 for ; Fri, 4 Jul 2025 06:27:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751610432; cv=none; b=XO1TBZhuivbFZukFF/k1aG47zYn1/DYDaTHrjCxCaB/wHMm5YTVsxX1Ai7LD6OM+w3/SVlXaLot/kAiWHDLekmbOKoFwgxnurFxJjG8N4UjpAS59H7lQHFqTqdYXZd0rpNZS0KrOaknBm4WTtxKSrYuN+KZ9bFYz+ZbiZRNxpYk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751610432; c=relaxed/simple; bh=rnNdiZd6S9Ifs9xRwSIe2HZJaDOtznG3EYvdYdrT1TU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oLwX8Be9YSstHTgjzhKgarLOztC7aPID//F1qdc4OZGHDDOqiVAxRM3RcJR/wUmalTQM/u7owWt+xuSfFI6zDoK+d4XLtGwPWo8a/50NOtoeDRD6Y9pG4NYzzOy/G+ShxvqGKOsNC3CvkeX0p7yBunQXl6MP1aqcu24hZHcQvfQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Vbis5qIC; arc=none smtp.client-ip=209.85.210.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Vbis5qIC" Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-73c17c770a7so906698b3a.2 for ; Thu, 03 Jul 2025 23:27:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1751610430; x=1752215230; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CssPMht2RNCT6oHZnq2lcog7NRiakEoMHIsMI/+GOsw=; b=Vbis5qICrohdxM74gq2+DLjmDRR4SJnJIcdTuGavTqeAVqfNIpmCGDWXEk9j1V50WD JyIVQoCjC/zM3486i/nLrG+7EzOTnnpQ239lZ35NHi1Ka1yqhWeFxYIPUyOzc5YMAHF6 D5Jiv7NefMK2GJpUH5Yvdlq0mW3NjQdQpSEYCef4eXQvB4nOrwi8Nxi4xJgnYyr8zOAs qbp7lloOhPBaUYJME+eCK8idlGP1hE/A+i0MumykXwQTR2c4MCjv3UxLz/0gf44S/eWf z7boAqjS2iUu27K62JguBxgnsmp1QnI02f+JN0lCgr3+cZbgyE8fUnXsFRgV0Jh/+giI Lezg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751610430; x=1752215230; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CssPMht2RNCT6oHZnq2lcog7NRiakEoMHIsMI/+GOsw=; b=S8IhhB+LUqytWxTmZoWsyWsLhh42VGwLi7Vcgtc/dbhTI0H3iZql43gIzO0oeIShwO PI//Vrc2gdPT/hyaWFcN3/O6Oc+8eMiIpmZ5P6HWVvk7Mrj5ZymM/xV+VDL85ntVvAAQ zejmksgAbgQRo6T1gZtexLufn0Zm6+kHUgrUtN150GH4dTCTtK+3d+5bzaZ1L/Mz000y CVwOuF8YJFt2K0Paqfnf3TPpwjkHdeYg3f1LgdoTjaqe+d4F5ZQm8S/gSfaV8BKNKi8O Qn4SAjf87pC+hJoS9Z6EUPqLZrbenNSOpb4Nstov+yzmOMNyIl99nWMqYYkt2oHKDQ3S CdKQ== X-Forwarded-Encrypted: i=1; AJvYcCWuTxWYex5iKfZZ5ziIseqtyyz4JpABBck/NH74W/BsH14MPzqA1+xv6B8z8OMiPpuNlpf+G+mm8X4g5xM=@vger.kernel.org X-Gm-Message-State: AOJu0YxnHa36dQzVYrT9V0I7UPzaXMKIwF/4ktaR3VSf3DmUazAtokyI FFsxzLwYmTyJRH22FgqLnRTgEmdyv4/JkoXg5Io9SY/QZeLeffMNAfGlJ0iKBMHFYno= X-Gm-Gg: ASbGnctW2wcwQ2Wy5T7cLvaehS291hOgSPt2Q3GqwStCcAvHFEgToPGnXHDdCEKUDfQ hNmAONPN1b1KNT4K/Mqv7/G/lzC1kXpIQK3YQnLOzdETLVhlGYTxjBX1P6gx664j/WCcGGX4VdZ Eax54aaLjLjV7k00CmhGOcQNDZlHS308nn1Cd5G/gukDp+ZutJFJpOQSYupzL8hCEVKZHK1ndXz f2BdyrIbsn2LQPwBO3TU6UKnuMvdvJVms6t19aXzDBIpraGVgG7DjfaumajX5ZtR3QffksDTg3F ENu98+Qu8I99QU6pYnye0KawWF7N0y90U3AHiUxXMq83b5+YjQHdqbquEXmz0L5j77wORVd7oI/ 6muJ5K9uhakQZ X-Google-Smtp-Source: AGHT+IHAZZUJlbwPG8pSg0jZDLcpYtT864xhpoOT0kytf9EjZhBe/0eDYndqrOdf/jc7iu5owyQo+A== X-Received: by 2002:a05:6a20:3ca6:b0:225:c286:5907 with SMTP id adf61e73a8af0-2260a0a362bmr1393632637.3.1751610429935; Thu, 03 Jul 2025 23:27:09 -0700 (PDT) Received: from localhost.localdomain ([203.208.189.8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-b38ee5f643dsm1183240a12.37.2025.07.03.23.27.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 03 Jul 2025 23:27:09 -0700 (PDT) From: lizhe.67@bytedance.com To: alex.williamson@redhat.com, akpm@linux-foundation.org, david@redhat.com, peterx@redhat.com, jgg@ziepe.ca Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lizhe.67@bytedance.com Subject: [PATCH v2 5/5] vfio/type1: optimize vfio_unpin_pages_remote() Date: Fri, 4 Jul 2025 14:26:02 +0800 Message-ID: <20250704062602.33500-6-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250704062602.33500-1-lizhe.67@bytedance.com> References: <20250704062602.33500-1-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Li Zhe When vfio_unpin_pages_remote() is called with a range of addresses that includes large folios, the function currently performs individual put_pfn() operations for each page. This can lead to significant performance overheads, especially when dealing with large ranges of pages. It would be very rare for reserved PFNs and non reserved will to be mixed within the same range. So this patch utilizes the has_rsvd variable introduced in the previous patch to determine whether batch put_pfn() operations can be performed. Moreover, compared to put_pfn(), unpin_user_page_range_dirty_lock() is capable of handling large folio scenarios more efficiently. The performance test results for completing the 16G VFIO IOMMU DMA unmapping are as follows. Base(v6.16-rc4): ./vfio-pci-mem-dma-map 0000:03:00.0 16 Reviewed-by: David Hildenbrand Suggested-by: Jason Gunthorpe ------- AVERAGE (MADV_HUGEPAGE) -------- VFIO UNMAP DMA in 0.135 s (118.6 GB/s) ------- AVERAGE (MAP_POPULATE) -------- VFIO UNMAP DMA in 0.312 s (51.3 GB/s) ------- AVERAGE (HUGETLBFS) -------- VFIO UNMAP DMA in 0.136 s (117.3 GB/s) With this patchset: ------- AVERAGE (MADV_HUGEPAGE) -------- VFIO UNMAP DMA in 0.045 s (357.0 GB/s) ------- AVERAGE (MAP_POPULATE) -------- VFIO UNMAP DMA in 0.288 s (55.6 GB/s) ------- AVERAGE (HUGETLBFS) -------- VFIO UNMAP DMA in 0.045 s (353.9 GB/s) For large folio, we achieve an over 66% performance improvement in the VFIO UNMAP DMA item. For small folios, the performance test results appear to show a slight improvement. Suggested-by: Jason Gunthorpe Signed-off-by: Li Zhe --- drivers/vfio/vfio_iommu_type1.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type= 1.c index 13c5667d431c..3971539b0d67 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -792,17 +792,29 @@ static long vfio_pin_pages_remote(struct vfio_dma *dm= a, unsigned long vaddr, return pinned; } =20 +static inline void put_valid_unreserved_pfns(unsigned long start_pfn, + unsigned long npage, int prot) +{ + unpin_user_page_range_dirty_lock(pfn_to_page(start_pfn), npage, + prot & IOMMU_WRITE); +} + static long vfio_unpin_pages_remote(struct vfio_dma *dma, dma_addr_t iova, unsigned long pfn, unsigned long npage, bool do_accounting) { long unlocked =3D 0, locked =3D vpfn_pages(dma, iova, npage); - long i; =20 - for (i =3D 0; i < npage; i++) - if (put_pfn(pfn++, dma->prot)) - unlocked++; + if (dma->has_rsvd) { + long i; =20 + for (i =3D 0; i < npage; i++) + if (put_pfn(pfn++, dma->prot)) + unlocked++; + } else { + put_valid_unreserved_pfns(pfn, npage, dma->prot); + unlocked =3D npage; + } if (do_accounting) vfio_lock_acct(dma, locked - unlocked, true); =20 --=20 2.20.1