From nobody Fri Dec 19 13:45:04 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 57011199EA1 for ; Mon, 26 Aug 2024 20:44:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705043; cv=none; b=QAaTmq4oaCRqN/namg4B1VAS4qH7aYJDpw0mvz/trtHYsaTDNneRSJPINBCSZ6QTLV9NLGC1+BiOq3PIm4XuNpO+Weha0G7HZyLZhXaZEOQMsqj+kxu3uGzMhuG3+ywRdmYfARQo5NplWwFb2syUcWV3vcGLD/JcO/uVbY9X3QY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705043; c=relaxed/simple; bh=bqt9KW78i0VdjWfmtysLtEN3zBnedLCNShRvlblkAoU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iszhFdfJrp86GDQo6vnyLtbGxXimY988Lm6tzslYc6vJZi9CJ8OveIsUYRNT7dPKDFaM8m+SdY5HuI+ROktwjzOFf/iBROs/zBwgIaOgJPxdKd5fG8TlOPewloAzHfO/ZwviLMwQ0+E2ji0h1Lfz/a77aSHI1CNo2URxSO1BDUI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Ar+zbt5o; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Ar+zbt5o" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705041; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2iIRbDNVyj1lHjlop0tZMw2JlHFFftRtxMxE0Q8z7r8=; b=Ar+zbt5oCah6Qf33SxCiUAvIabyqP86l1dGq0t/zeL1oUHWtfPc5NI/xI5MX6KdMHIXtK8 RXYscjgwNh4EXTAyPiwdnCqtZDP4KMT9lutKDJI/GABRFNGxoFjMHSbmLPKo/8dpFnlsyz Q787vIi9PCkC3nrbB5Watux6GGg2VbE= Received: from mail-oo1-f72.google.com (mail-oo1-f72.google.com [209.85.161.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-660-hsTtSPi_N_mF8Kg-yWxfIQ-1; Mon, 26 Aug 2024 16:43:59 -0400 X-MC-Unique: hsTtSPi_N_mF8Kg-yWxfIQ-1 Received: by mail-oo1-f72.google.com with SMTP id 006d021491bc7-5d5ba96af29so6521257eaf.0 for ; Mon, 26 Aug 2024 13:43:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705038; x=1725309838; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2iIRbDNVyj1lHjlop0tZMw2JlHFFftRtxMxE0Q8z7r8=; b=UirOOojUR0qxhflKONyvS+1+O+bXOc9wSE1zHcPWBGFI3vRQsEGBR9jYnrpJ9kvwnk mTBzK5Ee7Na0qH7zYtJKk4qiirDDTmq4l3ikyamr+tbRgliWgxtVLgszc23Ntd6JIO2h gYw/uTpuDniJnTai4rN/o5IBnfz6baz9PwYAU3S55nVLeTCwT3idZpEZl+cp4d+n3PpG 2r1cml6WN42OJaD3pXJsUPW8KPnVpLPZwqqrugtaFznu6xe+IsatUtbiYpWJ/0LIIsdx tummxG46kAcHhxnqGfAhS4jJW5WglmM+OOrijUlhfnf9REx83RR8IYRDrgOaNHh9oOtq SxiQ== X-Gm-Message-State: AOJu0Yy/Vz80In756j3uWzuKQgVUWCEEeYHxUuCSWk5A0yteiYP/r6q/ +YNhAPvxXJKFTuv0141qykfl3cHG88uYAON8wNWTFILWHtQ4WltChNYn3yV0pcEqffPiCxj2dsY vaFBMLJTlsrauq6jDR2dSzpk2U2XBpT7oXVV1drhruPTuDSC7XcSgXzXyMYVQLShB2uLbUfjG1W AMRcHJFgPiqcRnTtqVTf3HSWIAn7XAA/fwZqWdgnIpmlI= X-Received: by 2002:a05:6358:52ca:b0:1b5:a032:9994 with SMTP id e5c5f4694b2df-1b5ebe5b1b3mr113537455d.0.1724705038510; Mon, 26 Aug 2024 13:43:58 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHkrFRhmUpfOOmJ2YwtqClR5K8A29/GQ8sXabHeZ52sv0b77XENUJCQ9LJLwxTjS7HZUYg0sQ== X-Received: by 2002:a05:6358:52ca:b0:1b5:a032:9994 with SMTP id e5c5f4694b2df-1b5ebe5b1b3mr113532855d.0.1724705038035; Mon, 26 Aug 2024 13:43:58 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.43.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:43:57 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 01/19] mm: Introduce ARCH_SUPPORTS_HUGE_PFNMAP and special bits to pmd/pud Date: Mon, 26 Aug 2024 16:43:35 -0400 Message-ID: <20240826204353.2228736-2-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This patch introduces the option to introduce special pte bit into pmd/puds. Archs can start to define pmd_special / pud_special when supported by selecting the new option. Per-arch support will be added later. Before that, create fallbacks for these helpers so that they are always available. Signed-off-by: Peter Xu --- include/linux/mm.h | 24 ++++++++++++++++++++++++ mm/Kconfig | 13 +++++++++++++ 2 files changed, 37 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index b0ff06d18c71..d900f15b7650 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2643,6 +2643,30 @@ static inline pte_t pte_mkspecial(pte_t pte) } #endif =20 +#ifndef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +static inline bool pmd_special(pmd_t pmd) +{ + return false; +} + +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return pmd; +} +#endif /* CONFIG_ARCH_SUPPORTS_PMD_PFNMAP */ + +#ifndef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +static inline bool pud_special(pud_t pud) +{ + return false; +} + +static inline pud_t pud_mkspecial(pud_t pud) +{ + return pud; +} +#endif /* CONFIG_ARCH_SUPPORTS_PUD_PFNMAP */ + #ifndef CONFIG_ARCH_HAS_PTE_DEVMAP static inline int pte_devmap(pte_t pte) { diff --git a/mm/Kconfig b/mm/Kconfig index 8078a4b3c509..b23913d4e47e 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -875,6 +875,19 @@ endif # TRANSPARENT_HUGEPAGE config PGTABLE_HAS_HUGE_LEAVES def_bool TRANSPARENT_HUGEPAGE || HUGETLB_PAGE =20 +# TODO: Allow to be enabled without THP +config ARCH_SUPPORTS_HUGE_PFNMAP + def_bool n + depends on TRANSPARENT_HUGEPAGE + +config ARCH_SUPPORTS_PMD_PFNMAP + def_bool y + depends on ARCH_SUPPORTS_HUGE_PFNMAP && HAVE_ARCH_TRANSPARENT_HUGEPAGE + +config ARCH_SUPPORTS_PUD_PFNMAP + def_bool y + depends on ARCH_SUPPORTS_HUGE_PFNMAP && HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD + # # UP and nommu archs use km based percpu allocator # --=20 2.45.0 From nobody Fri Dec 19 13:45:04 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F01119AD7E for ; Mon, 26 Aug 2024 20:44:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705045; cv=none; b=jPJ8GaI5wCBqO7th/TXw+JICmylW0N3EAWy8JmXSDmYLMD1eF2x8Th+bVPEwWSCZRKupfJeG05J00fQ+O963Uyty4DeCh1B+zwdoq5nY1RFWkSUW3nu17kzV1idT5qTHoay6qQYSRqS6IVVmDGckYP9VizFp2SruQ6WYp8zHySk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705045; c=relaxed/simple; bh=34MqcYAeedS1HMCphQeI6ozj3aHeCly2AzsezqBdon4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GucAZB7W2lC5v7fJL+6YsagsI8Ba//jUQPiOYRHyRQ+bi9NjxbWlhFhGK6NCjRCttWXJmLYmQ7hVroGtX2sceN7Qk6iHINyua+Xad0EOStND024+r8t8wuO5SxSnkEfAtDqIz6PnbEnfvgvuboc+oCOCB+zBP7hBimRst7OImm4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=jBoBoFEr; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="jBoBoFEr" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705043; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rCV2q6t08l8ndW0pGxUU3F/GDhRRI1SX3G16E23iQDg=; b=jBoBoFEr0jb36+d9AWMwiUzDB1QmjCvw8b41LINAueOmv0Ve5Ce3jQphOq0oMQPNTZ3Ryw l4OCLP2nHjw3eZ0VCK3LfGQVkxfIHoBBOTuV8x5AY/EaOKV6WKf04RmmO/5ZJs8DdNaVYW wP8QHhV4K2UY+8b7ymLfVmzvfL1VUMc= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-379-1HBWbHlSPBmy2Tvgr0nz-w-1; Mon, 26 Aug 2024 16:44:01 -0400 X-MC-Unique: 1HBWbHlSPBmy2Tvgr0nz-w-1 Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-7a66b52c944so520440285a.1 for ; Mon, 26 Aug 2024 13:44:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705040; x=1725309840; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rCV2q6t08l8ndW0pGxUU3F/GDhRRI1SX3G16E23iQDg=; b=IpEZi+CyuC3SOqLLrTqhWn9VFSxBhXo862xmGD9TnignRtJ6fh5f4SzEPxzYc9rvW1 RpMBJIFWnePDgqeTd5F/eYbskWry9FsJ4JbxIFcJ+1y3tOHkTtauZM9yQwKdcwqErieR XSpUUmQZHssAxkre1b6X1ObMgkcSBL0IHObXBDki+oAPgHGbh/JsGmksFJca/DXio+jq HkudBuAKuTz89AFCN92jfG0S4WzpWQ16PQB0DanuQjbFaMQmlC4yAXpCg2FmEynpxNnf eazw7yHYayNr1nofTzXqCdNfN5/N+TYfwJgEOKAQew7Rk5Kt2wcg1qEUsXppcrkhcQJ7 oqkg== X-Gm-Message-State: AOJu0Yz+fPAD3+xH+JO1frZJSvbP8GvUaIRdLuWy+rwxxJdD+cugXj8A t9vn2zDIzp3qFPUlRH7toZOCnV8gLdMNHqwwRMnaFtQqZaIYAN+P4o3JQ8l6eQLSLzCU79bJ5Yy W5Z6Vo0eBd1A23kqpsIfKhH9jmFVuMqnVZlQ2tFeyno1LEyXeGZyl0Aznubzm1uda++6Z0ceVwV TZzSXIwH36uy3QENQ60wJOjHjfrtJ/jyWI372UBH/QH3U= X-Received: by 2002:a05:620a:28c8:b0:7a1:62ad:9d89 with SMTP id af79cd13be357-7a7e4e6d3c1mr92956985a.64.1724705040577; Mon, 26 Aug 2024 13:44:00 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEW5yVxsMo5HJEI7stPZogQTAiTuzeFOof5FGrO8bEsBYthlHw7SRF31ZNT2saoeMAKCiiqzA== X-Received: by 2002:a05:620a:28c8:b0:7a1:62ad:9d89 with SMTP id af79cd13be357-7a7e4e6d3c1mr92952485a.64.1724705040096; Mon, 26 Aug 2024 13:44:00 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.43.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:43:59 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson , Matthew Wilcox , "Aneesh Kumar K . V" Subject: [PATCH v2 02/19] mm: Drop is_huge_zero_pud() Date: Mon, 26 Aug 2024 16:43:36 -0400 Message-ID: <20240826204353.2228736-3-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" It constantly returns false since 2017. One assertion is added in 2019 but it should never have triggered, IOW it means what is checked should be asserted instead. If it didn't exist for 7 years maybe it's good idea to remove it and only add it when it comes. Cc: Matthew Wilcox Cc: Aneesh Kumar K.V Acked-by: David Hildenbrand Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- include/linux/huge_mm.h | 10 ---------- mm/huge_memory.c | 13 +------------ 2 files changed, 1 insertion(+), 22 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 4902e2f7e896..b550b5a248bb 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -433,11 +433,6 @@ static inline bool is_huge_zero_pmd(pmd_t pmd) return pmd_present(pmd) && READ_ONCE(huge_zero_pfn) =3D=3D pmd_pfn(pmd); } =20 -static inline bool is_huge_zero_pud(pud_t pud) -{ - return false; -} - struct folio *mm_get_huge_zero_folio(struct mm_struct *mm); void mm_put_huge_zero_folio(struct mm_struct *mm); =20 @@ -578,11 +573,6 @@ static inline bool is_huge_zero_pmd(pmd_t pmd) return false; } =20 -static inline bool is_huge_zero_pud(pud_t pud) -{ - return false; -} - static inline void mm_put_huge_zero_folio(struct mm_struct *mm) { return; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a81eab98d6b8..3f74b09ada38 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1429,10 +1429,8 @@ static void insert_pfn_pud(struct vm_area_struct *vm= a, unsigned long addr, ptl =3D pud_lock(mm, pud); if (!pud_none(*pud)) { if (write) { - if (pud_pfn(*pud) !=3D pfn_t_to_pfn(pfn)) { - WARN_ON_ONCE(!is_huge_zero_pud(*pud)); + if (WARN_ON_ONCE(pud_pfn(*pud) !=3D pfn_t_to_pfn(pfn))) goto out_unlock; - } entry =3D pud_mkyoung(*pud); entry =3D maybe_pud_mkwrite(pud_mkdirty(entry), vma); if (pudp_set_access_flags(vma, addr, pud, entry, 1)) @@ -1680,15 +1678,6 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct m= m_struct *src_mm, if (unlikely(!pud_trans_huge(pud) && !pud_devmap(pud))) goto out_unlock; =20 - /* - * When page table lock is held, the huge zero pud should not be - * under splitting since we don't split the page itself, only pud to - * a page table. - */ - if (is_huge_zero_pud(pud)) { - /* No huge zero pud yet */ - } - /* * TODO: once we support anonymous pages, use * folio_try_dup_anon_rmap_*() and split if duplicating fails. --=20 2.45.0 From nobody Fri Dec 19 13:45:04 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BD26019D07A for ; Mon, 26 Aug 2024 20:44:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705050; cv=none; b=VrCZNx+u0+Tf6Ce6M1Fz556EdBbL2vldbwU1oJZDZw0hlllTI8MRNEGLVI3hHipw/Aq9d9yahfpyxaOjkaZIRShP600Q1Pp6iL8in+KYzQanvsKNgaZEkzTdvaHdJxH8ADUzrNyfvKUwuLnMoNDm0l7UczaK5DMJ/ibUtfAzje8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705050; c=relaxed/simple; bh=cBR1CRPKBp17TA49iECdr5UwUokaTW4WPHX9nl57T+I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OkRMMhf9LCbI5moRIWWXJzpojF8hiPhtEtnR4Z6RfxMnI9x4FsVKmWfKVUZNXmsY5cY731hyJyxi4psFv2aTphU2y0PvrSBui+hsppT6OZ9Lynxy18wqxfw0uTv9RKhaLUYJcxPs+E0YDoWbp9uebdjpKltid+xcL4O/ND6WXpc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=GSwD8eln; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="GSwD8eln" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705047; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gD9ggS8OhpZf/NsZoH3fZjUIikns33eAnWwzmGyUsEI=; b=GSwD8elnSkUU0ri/z1c0FtxzSRtc8JfP+apWnD3UZDViRP5kShi4KjQ/I97emBPnEaLyuk Q6nZF/TVC8Nsf/aeEfSlj7IFv/CQ63o++L+HG+L4Nzsz5hkHbV7ExIwUMEsIHW7bD27KYQ cRhXBI+2yfogAGTver3Y/nwmA9fvIe8= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-615--J0mzlJZMc6ryLmEliJ0aQ-1; Mon, 26 Aug 2024 16:44:03 -0400 X-MC-Unique: -J0mzlJZMc6ryLmEliJ0aQ-1 Received: by mail-qk1-f200.google.com with SMTP id af79cd13be357-7a1d06f8f03so562435485a.1 for ; Mon, 26 Aug 2024 13:44:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705043; x=1725309843; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gD9ggS8OhpZf/NsZoH3fZjUIikns33eAnWwzmGyUsEI=; b=Xsa6In5q1W4zWBOOb5XBZ5S5whXugxMRD6mprFdy4uZ/jZDbqcWPYQsNPC+zLcM4Ws 6VzbfpL6RrnUZUA4iY6llQ+oQ3PnoyEVY+mZ0a7ydVEAnF/BdOrfWbWexM8JRdTuUxjA x8Bb90J5oQSd/NC7OVW8oV3nla9pGvYxq8gim6VWMEhe6DmDRUunkfSpQYul0L9ohd07 19m6f730nZm8UtOmaS+gb49o9CiJCsZsl0wL76v76pdJ4gCfbRS4GD9VHSkzW77xUlmu uWCjKFnQ4pj8A9D0L5q7Pr/Z3RV7Q4stoOK8znlAw/8hGtHlB1pcW9wfBq0IHjv6x9nF qXDw== X-Gm-Message-State: AOJu0YyL/llOxfMY6WnSonWrvCum5Mhs6LS5qJxPdzZo+YLsuopjXP1T 3ZigEm3Dd5xF1QXfhMnXY+dp0zKEfGZaG5vZSLUzSWv7LrfGvZh2e8c9Y5EhQ6H3j2eyDrTli0+ uWZZKmfXbJ/BVaVuG8NQY05n6cE+aMzkrrwoACAk+fPdeGqF0KKhJUCfZXgytLjv30yrBSSsfAR ca+6kInFT/+EJm5CyLu+9Jn1SJ4HoXpE0SCWuLS3yu22I= X-Received: by 2002:a05:620a:24d5:b0:7a3:6dd9:ef9d with SMTP id af79cd13be357-7a6897c6e76mr1277669085a.62.1724705042739; Mon, 26 Aug 2024 13:44:02 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHIqrjqvJMs9Ym5fhhSHDCZDk2wQ6104PL5R9Vk0hTAFm1CUlIqFGQmzGdyDHGQhWdDyxioXg== X-Received: by 2002:a05:620a:24d5:b0:7a3:6dd9:ef9d with SMTP id af79cd13be357-7a6897c6e76mr1277664185a.62.1724705042003; Mon, 26 Aug 2024 13:44:02 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:01 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 03/19] mm: Mark special bits for huge pfn mappings when inject Date: Mon, 26 Aug 2024 16:43:37 -0400 Message-ID: <20240826204353.2228736-4-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We need these special bits to be around on pfnmaps. Mark properly for !devmap case, reflecting that there's no page struct backing the entry. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu Acked-by: David Hildenbrand --- mm/huge_memory.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 3f74b09ada38..dec17d09390f 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1346,6 +1346,8 @@ static void insert_pfn_pmd(struct vm_area_struct *vma= , unsigned long addr, entry =3D pmd_mkhuge(pfn_t_pmd(pfn, prot)); if (pfn_t_devmap(pfn)) entry =3D pmd_mkdevmap(entry); + else + entry =3D pmd_mkspecial(entry); if (write) { entry =3D pmd_mkyoung(pmd_mkdirty(entry)); entry =3D maybe_pmd_mkwrite(entry, vma); @@ -1442,6 +1444,8 @@ static void insert_pfn_pud(struct vm_area_struct *vma= , unsigned long addr, entry =3D pud_mkhuge(pfn_t_pud(pfn, prot)); if (pfn_t_devmap(pfn)) entry =3D pud_mkdevmap(entry); + else + entry =3D pud_mkspecial(entry); if (write) { entry =3D pud_mkyoung(pud_mkdirty(entry)); entry =3D maybe_pud_mkwrite(entry, vma); --=20 2.45.0 From nobody Fri Dec 19 13:45:04 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 80AE419D075 for ; Mon, 26 Aug 2024 20:44:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705050; cv=none; b=okp+3SLCPvaJVGVc2W6+HqGNhS2rnFPziJ5e0SnfFbK78gBgc6so0I6WLZI4yLVxXCndjr+JVvaRapd/Ic1d2uaLll4eBIRoAJEfFgDtj/ky2PjEC3dENWhiN2Zv7/yq20kWwJ2Lni/lorotn32NrCSdJNO8qFsH6yqYsWwUVtQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705050; c=relaxed/simple; bh=aAVG1dQkK4CK+bGKDFmGiwLRBBmSB50x/ZhpxKIy1+Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MgCUhjn/hNOXOEuzujiQvHqIemIogDXCluorzFQD0I0K5JaV5+Yg32XcB3xdj7k+5x8BbzekfnpoW70wfNnTiKQatSbv54xwey6Q5KQr43Ue8cxMXsOV4GELn/tNyPDkH0g00hul03wtFsFWK6/FLaPuDZU0k/8w3WtP4Bq6PTg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=U/IvSmSi; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="U/IvSmSi" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705047; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=p41Ko1t0SYR+aQ+NnkPoKcQjWVRbF5iAoso+I7QNClw=; b=U/IvSmSirmaWE7Ef8vFO/UCoqdpHgwXgcB21EQ0xYAHCKX4ww5M7QJgkYHrqhIRaZVKKDt oQsRiF5dePjh3ynn/KKuf9gP/RuOtqkhCP+sSewgmtLnNcouhljEO/itrzqKrSM+9bj9kx oM5DnUZpkoWk8wulDD2hQSsA9asi9Sk= Received: from mail-oa1-f69.google.com (mail-oa1-f69.google.com [209.85.160.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-145-IbEQQn_8Pt2o-fsa8U2Uww-1; Mon, 26 Aug 2024 16:44:06 -0400 X-MC-Unique: IbEQQn_8Pt2o-fsa8U2Uww-1 Received: by mail-oa1-f69.google.com with SMTP id 586e51a60fabf-26823e7e16eso10889817fac.3 for ; Mon, 26 Aug 2024 13:44:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705044; x=1725309844; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=p41Ko1t0SYR+aQ+NnkPoKcQjWVRbF5iAoso+I7QNClw=; b=Jp6ydELLv1ConluF+gkkSt8H/rPj+QxFIczg5d3ReahH8T4GdB7B5dgOO3zD0wyWL9 IBJBR1O/PvSbTnbeWuVqrz8SW1EChL5Cw4jyWWSZs5jS0YrayAWguPCrJvbLQZYCXJUK 0tQR/6EF+BoBwEDFUCzAdKQ77r5MO5zoG1BVJPC/MoHrOgDzWzeu+7nYrqb09k4Wi2SP 6B8/dmOwD2kptao0hg05bfXABujbGVkpMMtqbT6qevVIHae9LHt8kH8YDw0gPvXYZ6EL iNGl4Oq0cWvkxMk1SqAlUsv0a7KL8DAjNDNb7gDdHR+godayoXocTPI8R2NaedlVcZet bs0g== X-Gm-Message-State: AOJu0Yx2D3hvslrM/h5vNozkJ5swFAGA5liHIlhwsjf9eBt5JiXf/OCo hu6NORgzVVVGgXKbSkuh9plSHtCQt974In91OlXrefcB7ZH0AnbrsaIAHDf5vbud6IQK7RiFFzG I+B5bpKl4tFkskKD2F2jgpKKwRFTZ/NK3Q20PGZF951Zou0nWykg/f7m44fZ38vH0nSblCkXPiM UATH8uupe746lDPSZHRhO+msv0ukC8t5OF2D06Q6Eb+bU= X-Received: by 2002:a05:6358:ca5:b0:1b5:ecce:b760 with SMTP id e5c5f4694b2df-1b5ecceb805mr68657855d.21.1724705044596; Mon, 26 Aug 2024 13:44:04 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGqSprkmyPyKIDES0Mxu2D+0b60kqL8Leav+5Mf5w5kuR6qgHz2l36Zs0Y1NhYRXD/bnn90Fg== X-Received: by 2002:a05:6358:ca5:b0:1b5:ecce:b760 with SMTP id e5c5f4694b2df-1b5ecceb805mr68653955d.21.1724705044223; Mon, 26 Aug 2024 13:44:04 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:03 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson , Matthew Wilcox , Ryan Roberts Subject: [PATCH v2 04/19] mm: Allow THP orders for PFNMAPs Date: Mon, 26 Aug 2024 16:43:38 -0400 Message-ID: <20240826204353.2228736-5-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This enables PFNMAPs to be mapped at either pmd/pud layers. Generalize the dax case into vma_is_special_huge() so as to cover both. Meanwhile, rename the macro to THP_ORDERS_ALL_SPECIAL. Cc: Matthew Wilcox Cc: Gavin Shan Cc: Ryan Roberts Cc: David Hildenbrand Cc: Zi Yan Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu Acked-by: David Hildenbrand --- include/linux/huge_mm.h | 6 +++--- mm/huge_memory.c | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index b550b5a248bb..4da102b74a8c 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -76,9 +76,9 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr; /* * Mask of all large folio orders supported for file THP. Folios in a DAX * file is never split and the MAX_PAGECACHE_ORDER limit does not apply to - * it. + * it. Same to PFNMAPs where there's neither page* nor pagecache. */ -#define THP_ORDERS_ALL_FILE_DAX \ +#define THP_ORDERS_ALL_SPECIAL \ (BIT(PMD_ORDER) | BIT(PUD_ORDER)) #define THP_ORDERS_ALL_FILE_DEFAULT \ ((BIT(MAX_PAGECACHE_ORDER + 1) - 1) & ~BIT(0)) @@ -87,7 +87,7 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr; * Mask of all large folio orders supported for THP. */ #define THP_ORDERS_ALL \ - (THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_FILE_DAX | THP_ORDERS_ALL_FILE_DEFA= ULT) + (THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_SPECIAL | THP_ORDERS_ALL_FILE_DEFAU= LT) =20 #define TVA_SMAPS (1 << 0) /* Will be used for procfs */ #define TVA_IN_PF (1 << 1) /* Page fault handler */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index dec17d09390f..e2c314f631f3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -96,8 +96,8 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_s= truct *vma, /* Check the intersection of requested and supported orders. */ if (vma_is_anonymous(vma)) supported_orders =3D THP_ORDERS_ALL_ANON; - else if (vma_is_dax(vma)) - supported_orders =3D THP_ORDERS_ALL_FILE_DAX; + else if (vma_is_special_huge(vma)) + supported_orders =3D THP_ORDERS_ALL_SPECIAL; else supported_orders =3D THP_ORDERS_ALL_FILE_DEFAULT; =20 --=20 2.45.0 From nobody Fri Dec 19 13:45:04 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DDCF319A282 for ; Mon, 26 Aug 2024 20:44:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705051; cv=none; b=O1LWY0/jCLYbMUnlRvI8XOe1JMRuv7gDhYIQuLR0/Q0P8rTB5BHOeGyaPhIka7ZeC98eWnmipATvtPykSwzQjPW+p/UXd0cNQx3xZGabNm9O48kJuixhc+8GQUEQmPbOzLNJDb/tWpFOR7fpE4CIQwrK/D4+aKsODbMDEUpmt1E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705051; c=relaxed/simple; bh=B4CjEPEBNhW/7ay+iPhYCq//7mfGSaIpgiOs2kdAYJU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=H71KcEso55ilbzznE3kSKc7wI92YoBoA6jbhirUbuI1vSS+IOxQ2k8izrlB0IuCGXwC0WCdzsxQkOcWYyfo0pUMG5v9RLgs4t7qnwaH1rFj6gWuvcQO9Q4C9LhqhNpfaV9AhX6rW4x8CtjP7x0JlqBCjF8bLQLpwg3Z59/QHZCc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=LEWHxxWV; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="LEWHxxWV" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705049; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SFHgAMnpeJHqDMa0ADr1pyvSduy44GuxF04BOe5aqcw=; b=LEWHxxWV8B/d9KIasPxZRdl7+97lJzSjELsdoiljqRqKCSpLXavw+Msj8Swo2aUJnCRRxw Ti12aQE7K4xioUm6jHUoQFu2JYBiu7UuPQmuIYbo2dhube5JAQc4ClI0HqgOJlB3PqQQbJ +Z2b4E7bPyiSt8U2C6I7VorRLRfOtho= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-552-VDlJFQtUOGa2JtpI0OZahw-1; Mon, 26 Aug 2024 16:44:07 -0400 X-MC-Unique: VDlJFQtUOGa2JtpI0OZahw-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7a1de8a2adbso643966485a.1 for ; Mon, 26 Aug 2024 13:44:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705047; x=1725309847; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SFHgAMnpeJHqDMa0ADr1pyvSduy44GuxF04BOe5aqcw=; b=Ef0s8DatuycrXPNZWzmlhqfJrr8rH4c16M8uDRsGTNhMst80YiDpOfDza/0qMB1ZQh EsBTXZ78DFq4Zf0PPjUxtq/0f31ryUR24Aq4xBXafSBiJNVwwHUo3rXOobbSmQ4KBI/E VoCA6M+OlB+td9OR2OZFwhHAJAjoOiq3YQp7jrzq5+rgsAwAUgkdvneYLSpcCExowRkH 8LFF6Up43SmQ7rjQGM9Jt1lN4S6E1bqstb28S8eNfnCbKkba9Q8nl2+n4GRFkntRZpju dE0nYW6/bykQX4N64b+m74TCFFGju87f2PzxA7/ZMOvNnauRbYrGvogDjtg4FF4qQWKa NXPg== X-Gm-Message-State: AOJu0YxvgolvSOk/++JPcbfhDPIFgEMVp5neL/mplfNcTKTOLTHuxI/v 28msSh2OqFHoIB+hkSToWyxoK49xih47+NwSdAUyjp7wjM/oPXQIRtPlJQtZRa1FYEGOOdqzk8H AuKf6e/3zUN9Dvkal5HN1yC7QhcBgA53q4vBhg1BSfOj1nBtmrV+rtH00eWCm6bdcijoMv636mN bTLhwfE9MJHj1t8F/zkNA3ta3giXKUKjeHhHdaa63zePo= X-Received: by 2002:a05:620a:2a14:b0:7a2:ccd:9672 with SMTP id af79cd13be357-7a7e4e6144fmr67569885a.67.1724705046930; Mon, 26 Aug 2024 13:44:06 -0700 (PDT) X-Google-Smtp-Source: AGHT+IG++rZwRsHPi3LEX2hBpbUb+Bhnwx97nv6F7Fy5QJm5RgQYxXJ7nF8aD8ykhSm+rXvWrvbUzQ== X-Received: by 2002:a05:620a:2a14:b0:7a2:ccd:9672 with SMTP id af79cd13be357-7a7e4e6144fmr67565885a.67.1724705046455; Mon, 26 Aug 2024 13:44:06 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:05 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 05/19] mm/gup: Detect huge pfnmap entries in gup-fast Date: Mon, 26 Aug 2024 16:43:39 -0400 Message-ID: <20240826204353.2228736-6-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Since gup-fast doesn't have the vma reference, teach it to detect such huge pfnmaps by checking the special bit for pmd/pud too, just like ptes. Acked-by: David Hildenbrand Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- mm/gup.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index d19884e097fd..a49f67a512ee 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -3038,6 +3038,9 @@ static int gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp,= unsigned long addr, if (!pmd_access_permitted(orig, flags & FOLL_WRITE)) return 0; =20 + if (pmd_special(orig)) + return 0; + if (pmd_devmap(orig)) { if (unlikely(flags & FOLL_LONGTERM)) return 0; @@ -3082,6 +3085,9 @@ static int gup_fast_pud_leaf(pud_t orig, pud_t *pudp,= unsigned long addr, if (!pud_access_permitted(orig, flags & FOLL_WRITE)) return 0; =20 + if (pud_special(orig)) + return 0; + if (pud_devmap(orig)) { if (unlikely(flags & FOLL_LONGTERM)) return 0; --=20 2.45.0 From nobody Fri Dec 19 13:45:04 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EB02F19D89C for ; Mon, 26 Aug 2024 20:44:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705053; cv=none; b=F0UtvGhXay7SiNcZuiauu56MIm0AAjQummIxQgqR8ywmUdkz5q3kit9DOImyYXTWv1UkP4PXijrEveAXmnHB1tXQOCZYyu7gJKwo9AISmE8zIyhf/L2/H1Q89traGsFNpX7pT4IzBJvUw+u8lL3Ewnx9KZqC/RSUStO3MbgJiQs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705053; c=relaxed/simple; bh=0JOvh9d6Ck9MlI1IecLNRwn4z9Son10cjOiVCsL6GIo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UAXVa5WNFnJgD3NoEMuuVgeQpn+cYQTpmhpT3kKaAIolnxglOjnqkf9M1Sh6IvpyQPKFjbchLGC/tthJAYrkYY1L20Qu0TLb7QsqskEZshiGU1J3R4KBJjLU/CTJdY5O4fCmRmZnnv7vnP3R51p4Efr5uTyDaECS53JUQTN4Kdw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Ok+baKrA; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Ok+baKrA" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705051; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UaLXo2idJup/Oy4OwRPy2aK364ayUQ63MrUET5wBpvQ=; b=Ok+baKrA060QFvLET3CXcxfjMa9U4bQXV/+WOBg70a6njCWetklHda8WaK+bVEfm45SVhf rwPME4bnkyNv1Xhsj04LSlUqOc2OOQ5pmMR8LWlvrwmCx/7DRPMTeozMJdjGH1fWvJcp66 wTuosEhrDM5NpQJxSDwj9W2Tlhylp+Q= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-435-8Zm0eE6MPQCIMelgK2W-2w-1; Mon, 26 Aug 2024 16:44:09 -0400 X-MC-Unique: 8Zm0eE6MPQCIMelgK2W-2w-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7a1d06f8e78so523628485a.3 for ; Mon, 26 Aug 2024 13:44:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705049; x=1725309849; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UaLXo2idJup/Oy4OwRPy2aK364ayUQ63MrUET5wBpvQ=; b=Oc6wxvhdxM+v7MWwBZrhXeAetn42nyQYt2G7fSC+lbP6313xk1y+hz2SM7TNjf+BZy +svklbJNu/J6r8fiymYdLPvBhfJW/cklc6m4goKUuDWeTEzev6OZ/7UzQpO/kznbDTw8 YA35IFUwL1uoz3th3JmT1Rg3IQ1ksosJF4qg1XZEKqvvOcNaV5QZVVCvfsuwxkuu0vbd aT4k4BlGvTBuyNEhbIldG7edz9vSsdc8xcs6sgprZ7gFgrMZpKL2g6+8J89EJeXukooe byDi7FVVtMwVfDPxmAhiAD4lArb0DYBxPL4DVt5LSDGuJdhrVECAQQB7KOfHUnEckT7j W0DA== X-Gm-Message-State: AOJu0YwcHoLVX8p4bXZYdbwEnEj7f4k4KoUCJnKsntVZA7nsmF7Z5+IE +boDuABDalH9vP4GnG1ntsYc5oUHxlPcQC12JQlmMvkdYDPxAWSpu+DatQpIlUEnKkzNYqAUTDm atKaraPMYCdbxluLWN6u5r9zRCfVpu0978u4yK3hmOVakTVXSXGUyjSafLHek7ZqhHYArV18XWP r2Ms6FQ3QaZJZa8rc1hxER6MLhw9AcAE09Nl/OEiGJa7A= X-Received: by 2002:a05:620a:4112:b0:7a3:49dc:e6e3 with SMTP id af79cd13be357-7a6897a6f65mr1244453385a.53.1724705048971; Mon, 26 Aug 2024 13:44:08 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFAEqST5igJBEO0jhxDT4OAwN3xGv5rzMyL4N+pVFoujxFDEyZ/hbdQR2zB1kD+DQ4JLF652A== X-Received: by 2002:a05:620a:4112:b0:7a3:49dc:e6e3 with SMTP id af79cd13be357-7a6897a6f65mr1244449385a.53.1724705048557; Mon, 26 Aug 2024 13:44:08 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:07 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 06/19] mm/pagewalk: Check pfnmap for folio_walk_start() Date: Mon, 26 Aug 2024 16:43:40 -0400 Message-ID: <20240826204353.2228736-7-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Teach folio_walk_start() to recognize special pmd/pud mappings, and fail them properly as it means there's no folio backing them. Cc: David Hildenbrand Signed-off-by: Peter Xu --- mm/pagewalk.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/pagewalk.c b/mm/pagewalk.c index cd79fb3b89e5..12be5222d70e 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -753,7 +753,7 @@ struct folio *folio_walk_start(struct folio_walk *fw, fw->pudp =3D pudp; fw->pud =3D pud; =20 - if (!pud_present(pud) || pud_devmap(pud)) { + if (!pud_present(pud) || pud_devmap(pud) || pud_special(pud)) { spin_unlock(ptl); goto not_found; } else if (!pud_leaf(pud)) { @@ -783,7 +783,7 @@ struct folio *folio_walk_start(struct folio_walk *fw, fw->pmdp =3D pmdp; fw->pmd =3D pmd; =20 - if (pmd_none(pmd)) { + if (pmd_none(pmd) || pmd_special(pmd)) { spin_unlock(ptl); goto not_found; } else if (!pmd_leaf(pmd)) { --=20 2.45.0 From nobody Fri Dec 19 13:45:04 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 105F819DF63 for ; Mon, 26 Aug 2024 20:44:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705055; cv=none; b=RSJIS9mep3eLT4pMNXGk4/mc6OjwSaiQhVAWg6sPTaElQ8NAH7cIDPgzs7LM8v2883xVkYwRWkQV9n81MGdT7zswjVsfz3QGEJ2s8j4GULHHw+5IBXROfHd/Zu7Cg86/Ys7eXFfoLkj0vOisnbdYK1dKQohcZUJQjVtjDPksV4E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705055; c=relaxed/simple; bh=2o5q081kpM9gqW6xM+rZ8YlKEO3D1caJMlb8NRd33tM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=K+oJvauFTbROxYbviheuE3QuzzWXK3oQH5Gw7p5PqbvYi26yWsZ73EAyI4sJSNDg2YCatRZIsry6DwC5szYwn6I+B+Yli8h3Bl/LFqdPRgtKtGkIXjrJZdWBD/Su9ujUvHsDJVGOiSRkCY4y9mk/wGp1QsihIA4uBPd/skCIizM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Qy1+1mTn; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Qy1+1mTn" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705053; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xat90K+7laktWlKWrEYj2NKDd4TQak32OHmU5MtaKmE=; b=Qy1+1mTnSQYinUC9F85qYc1/PEjUphn6hb+QD4bflomoRRA7oJlFCVudTO4KVrOmWsF5+u J0+rgyjha/jVo9QBkRHtAU7uxEraq0hKtzVtDBLEHpCPSTA3Zj4/P/h3i5Z6U7cxpFhYNf gyjSs/6A6BTXBbcntD2h8Wlif+h39+U= Received: from mail-yw1-f197.google.com (mail-yw1-f197.google.com [209.85.128.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-227-QXd0yMaDNFyWmn_51Lu7dg-1; Mon, 26 Aug 2024 16:44:11 -0400 X-MC-Unique: QXd0yMaDNFyWmn_51Lu7dg-1 Received: by mail-yw1-f197.google.com with SMTP id 00721157ae682-6b41e02c293so94039777b3.0 for ; Mon, 26 Aug 2024 13:44:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705051; x=1725309851; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xat90K+7laktWlKWrEYj2NKDd4TQak32OHmU5MtaKmE=; b=kYKo+zy+3qhqq9+/PPkTdrUjeqCPvrTP3uMRdnOdcvGDCN7n6ol+z/KcM0UeOqwewo 1dCqJsEAFgtvCc1+WswX7t23kf2xTRFrPrW7KWiPig112T8uSZkyB4f7eWCS77bHxjKJ ceIR7+vHlxed1FyLtuWE4baIhusRa9Tb1tox8lV0bITeRan7ohR5u6FZ89egcMgrJzjJ wG+a9WJxvplYB1yGMt3Zvn/1NFot7css1jglgYyQlJvpsIswyuf7UCeNCpn75PbZHn4z Af2wR70ymMjXt0PqnlHBTC6tg17BbK5NHXw/3f2S6CWBTt9piVFxNTqygRZeNToLNOUj htqg== X-Gm-Message-State: AOJu0YwopCi9H+teXa/7NQcaq1ak005oXnJQ9Ult71Ju77Lha/Y/j4qx y4gxhjoTPbkv3Sl8iWK2ggt25B/EtAQbfysfcj6he0MlEmdWWFpq+1D5iBsxVJBZzc1Eu0W4Zlw qOwfBzlRXNQICDbc1978f25tadM3yaSZTZxww9I2jnb3DBR9aZ/bIbYg7zntv8Jc+GxVvBExSgw egTEeabG/chOHyFs7sf2xhGmtN+IIhelecu7nCu7nnOy0= X-Received: by 2002:a05:690c:380b:b0:6c1:2b6d:1964 with SMTP id 00721157ae682-6c62906557cmr136660897b3.38.1724705050785; Mon, 26 Aug 2024 13:44:10 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGVQdSmnkARvKpk+I4g97hETTq/PaNGcO/OaJoau0YQSMQm37evWRXkCU2N5l4bob/h1UXyLw== X-Received: by 2002:a05:690c:380b:b0:6c1:2b6d:1964 with SMTP id 00721157ae682-6c62906557cmr136660497b3.38.1724705050419; Mon, 26 Aug 2024 13:44:10 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:09 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 07/19] mm/fork: Accept huge pfnmap entries Date: Mon, 26 Aug 2024 16:43:41 -0400 Message-ID: <20240826204353.2228736-8-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Teach the fork code to properly copy pfnmaps for pmd/pud levels. Pud is much easier, the write bit needs to be persisted though for writable and shared pud mappings like PFNMAP ones, otherwise a follow up write in either parent or child process will trigger a write fault. Do the same for pmd level. Signed-off-by: Peter Xu --- mm/huge_memory.c | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e2c314f631f3..15418ffdd377 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1559,6 +1559,24 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct m= m_struct *src_mm, pgtable_t pgtable =3D NULL; int ret =3D -ENOMEM; =20 + pmd =3D pmdp_get_lockless(src_pmd); + if (unlikely(pmd_special(pmd))) { + dst_ptl =3D pmd_lock(dst_mm, dst_pmd); + src_ptl =3D pmd_lockptr(src_mm, src_pmd); + spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); + /* + * No need to recheck the pmd, it can't change with write + * mmap lock held here. + * + * Meanwhile, making sure it's not a CoW VMA with writable + * mapping, otherwise it means either the anon page wrongly + * applied special bit, or we made the PRIVATE mapping be + * able to wrongly write to the backend MMIO. + */ + VM_WARN_ON_ONCE(is_cow_mapping(src_vma->vm_flags) && pmd_write(pmd)); + goto set_pmd; + } + /* Skip if can be re-fill on fault */ if (!vma_is_anonymous(dst_vma)) return 0; @@ -1640,7 +1658,9 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm= _struct *src_mm, pmdp_set_wrprotect(src_mm, addr, src_pmd); if (!userfaultfd_wp(dst_vma)) pmd =3D pmd_clear_uffd_wp(pmd); - pmd =3D pmd_mkold(pmd_wrprotect(pmd)); + pmd =3D pmd_wrprotect(pmd); +set_pmd: + pmd =3D pmd_mkold(pmd); set_pmd_at(dst_mm, addr, dst_pmd, pmd); =20 ret =3D 0; @@ -1686,8 +1706,11 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct m= m_struct *src_mm, * TODO: once we support anonymous pages, use * folio_try_dup_anon_rmap_*() and split if duplicating fails. */ - pudp_set_wrprotect(src_mm, addr, src_pud); - pud =3D pud_mkold(pud_wrprotect(pud)); + if (is_cow_mapping(vma->vm_flags) && pud_write(pud)) { + pudp_set_wrprotect(src_mm, addr, src_pud); + pud =3D pud_wrprotect(pud); + } + pud =3D pud_mkold(pud); set_pud_at(dst_mm, addr, dst_pud, pud); =20 ret =3D 0; --=20 2.45.0 From nobody Fri Dec 19 13:45:04 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1721419DFA7 for ; Mon, 26 Aug 2024 20:44:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705057; cv=none; b=dmmlf0P7Ng/O31zaCSzVPw+aof/t2y3+G2DvhkVb/p26ELKCX99ypUR+BBBZCsGp/kKdkSueYYztidSwoC3owVBJbgvxEPiBdCGLvbqw/9c0juTPXO//LZXxnEMTZtVCDNXI0hkBWc8N9tNiaoLGc4v2gY191XNnqlAAqGC/uHg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705057; c=relaxed/simple; bh=5m+a/sFeWuT8yBU8zTVl6UkOZQKWMf5wpgIpcmHb57A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bkPLZWwX6YE2PPZ8ng7ClwiEXQrekkixGOfkOdXPyUk7xwbheg45FDvD40hL1mzDQzzoNWQdYSrW/8Q2/g5dvnJn0peKMOPJAhbI06WdtHVoxCAE91gxhTUBvGWpWr6D9nb10NKWvNMMkRFnOFowAoD1t6mfUxQp1zMCsnRWlCQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=cqn1o6Yk; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="cqn1o6Yk" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705055; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vK1PMhUcFVpQCvjQRaUqQOYBcILrsdr8yUlCuFWve7Q=; b=cqn1o6YkXtY2Da0jJ6CZFzfGjc/aMwAX/KqsM+od6aT9Llk2e2iSMpXjqrBOzYoQwwI6OZ RcM3dXMW0h38Nfa/pRkhQNGOrmtPudjG3Hij1X7bDT460Sl53PfjCOuyrVRCYyrsCuq3rF MekJBDmgUhW5JSeGQDqtbjMZ4jCjDcs= Received: from mail-yw1-f197.google.com (mail-yw1-f197.google.com [209.85.128.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-651-V5SrlegaOz-ZUS8_EKPqJQ-1; Mon, 26 Aug 2024 16:44:13 -0400 X-MC-Unique: V5SrlegaOz-ZUS8_EKPqJQ-1 Received: by mail-yw1-f197.google.com with SMTP id 00721157ae682-6c3982a0c65so90371247b3.1 for ; Mon, 26 Aug 2024 13:44:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705053; x=1725309853; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vK1PMhUcFVpQCvjQRaUqQOYBcILrsdr8yUlCuFWve7Q=; b=vfHj2smUXXhIf0eioFCx1MgHhJRxMkeLPyuddAq4DGxIZiHjymVPFKVo5sNn/VnBwS ew+I/nc/TZe7059F3osu+ZmIsSzeq7aQYZQn1gSz7QX686BbKKbqNM09LkYSEb5tnGxE O4RTUpUPT/T4NE4ORG1gL6GkynHezrlOhpY24Tlv3UBy0e6ZxFx4DEP093ZzBRT7JHAI ff0JxYp4JYJcEkKgyoYOSBgtwUJ/NI/Hlp7/7/aMqWLTB3ZaHg2K7bKc8xbQpM4xytSw g06gR0CqgZ1A5bitooklflf/uJkod6VKNPk/Vz4Id3xEUTSmF0MdfkOQcHOY4aKB1Z1V bUZQ== X-Gm-Message-State: AOJu0Yyq23hEpaqgvaADmMAgtssYz7CXkQGbICKwCAZ+Y5TwJEe7nzX6 /b99BKso6YP1oaNK6t9FCJ9V1JTfzob75681CklDGEQzvAul0bzJAbCTUokDW5GjwNtggUuhc20 ZVd7tmirEqsBFm9hrcPihYUerXjRLJBLoPtscawKQZSwdwtOU9ERBVHUAl20A0HeCOAi7ZV8hzY 8StYhtw1ehOAxatL9oPwjlnQUaKoqZ4UB0O7FD66Dr/Rw= X-Received: by 2002:a05:690c:60c2:b0:6bd:fdd3:e170 with SMTP id 00721157ae682-6c624dc7042mr142339047b3.10.1724705052712; Mon, 26 Aug 2024 13:44:12 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEDLUzP2hp4LBST6IRuRTdz5MRV8z70t30cnU5hPcRwmZ021NQU+FxAT+ZUF1I58Qs4qkGJLg== X-Received: by 2002:a05:690c:60c2:b0:6bd:fdd3:e170 with SMTP id 00721157ae682-6c624dc7042mr142338617b3.10.1724705052349; Mon, 26 Aug 2024 13:44:12 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:11 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 08/19] mm: Always define pxx_pgprot() Date: Mon, 26 Aug 2024 16:43:42 -0400 Message-ID: <20240826204353.2228736-9-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" There're: - 8 archs (arc, arm64, include, mips, powerpc, s390, sh, x86) that support pte_pgprot(). - 2 archs (x86, sparc) that support pmd_pgprot(). - 1 arch (x86) that support pud_pgprot(). Always define them to be used in generic code, and then we don't need to fiddle with "#ifdef"s when doing so. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- arch/arm64/include/asm/pgtable.h | 1 + arch/powerpc/include/asm/pgtable.h | 1 + arch/s390/include/asm/pgtable.h | 1 + arch/sparc/include/asm/pgtable_64.h | 1 + include/linux/pgtable.h | 12 ++++++++++++ 5 files changed, 16 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index 7a4f5604be3f..b78cc4a6758b 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -384,6 +384,7 @@ static inline void __sync_cache_and_tags(pte_t pte, uns= igned int nr_pages) /* * Select all bits except the pfn */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pfn =3D pte_pfn(pte); diff --git a/arch/powerpc/include/asm/pgtable.h b/arch/powerpc/include/asm/= pgtable.h index 264a6c09517a..2f72ad885332 100644 --- a/arch/powerpc/include/asm/pgtable.h +++ b/arch/powerpc/include/asm/pgtable.h @@ -65,6 +65,7 @@ static inline unsigned long pte_pfn(pte_t pte) /* * Select all bits except the pfn */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pte_flags; diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtabl= e.h index 3fa280d0672a..0ffbaf741955 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -955,6 +955,7 @@ static inline int pte_unused(pte_t pte) * young/old accounting is not supported, i.e _PAGE_PROTECT and _PAGE_INVA= LID * must not be set. */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pte_flags =3D pte_val(pte) & _PAGE_CHG_MASK; diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/p= gtable_64.h index 3fe429d73a65..2b7f358762c1 100644 --- a/arch/sparc/include/asm/pgtable_64.h +++ b/arch/sparc/include/asm/pgtable_64.h @@ -783,6 +783,7 @@ static inline pmd_t pmd_mkwrite_novma(pmd_t pmd) return __pmd(pte_val(pte)); } =20 +#define pmd_pgprot pmd_pgprot static inline pgprot_t pmd_pgprot(pmd_t entry) { unsigned long val =3D pmd_val(entry); diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 780f3b439d98..e8b2ac6bd2ae 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1956,6 +1956,18 @@ typedef unsigned int pgtbl_mod_mask; #define MAX_PTRS_PER_P4D PTRS_PER_P4D #endif =20 +#ifndef pte_pgprot +#define pte_pgprot(x) ((pgprot_t) {0}) +#endif + +#ifndef pmd_pgprot +#define pmd_pgprot(x) ((pgprot_t) {0}) +#endif + +#ifndef pud_pgprot +#define pud_pgprot(x) ((pgprot_t) {0}) +#endif + /* description of effects of mapping type and prot in current implementati= on. * this is due to the limited x86 page protection hardware. The expected * behavior is in parens: --=20 2.45.0 From nobody Fri Dec 19 13:45:04 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C6A5819E7E0 for ; Mon, 26 Aug 2024 20:44:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705059; cv=none; b=CPGll/ULOpavn59xxGNbzjOsLvQ05ZWcFFBKzKd5OwHvovlZBHztnuZVb8XVixtNLg0cuGo96T0BZtQC6scIwuLFKjkdjOm65OKH/FBJ3IT1fmzWY6W2Jfqma3LQR5kptY/ROkDpxvVGG0i/24uvBcO3orSvfwj1PIeJqqO4xhM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705059; c=relaxed/simple; bh=OG1HOirCXbGLRvA8SpEinM1gGzP9lRsOJkqbcyKglh8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ORDZ+8Swn9V+y3K9Ll5Qfv8/JpbgqKAggp6A8pdDrRuN2VjJ83aHLrH8borVP4GRnVM8xGm7R2xGuF2B+K4v5+Qvy/u9fWOSDqM0tuzygDewhhoMCrAsCkDO9V7ksAFxVqLUrGQW9IZTmZwY+FHVclhOA3NJuxLtkXEi/QD6krw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Y7DgqhFi; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Y7DgqhFi" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705057; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XzWn17lHZnGD5Tr66iu6y8/zqhTRy72agTNrtdvxzwE=; b=Y7DgqhFiU9py8NcaEACQZfKOvi9JTLUlm3mwv9T3grgx/EUGBfkjCso6/IKeYYRt1HtKXL XiVkbLQvNckgR3M9urfRaDThtsuXbTmImGekdIXQxtsDR6+9/byPkPiubIq5wSKrBM6Wgv K7xmMZQl073267MLHpqeFDd22o5i3n0= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-205-4sx6jgSaOAelwQ_DDnMeIw-1; Mon, 26 Aug 2024 16:44:15 -0400 X-MC-Unique: 4sx6jgSaOAelwQ_DDnMeIw-1 Received: by mail-qk1-f200.google.com with SMTP id af79cd13be357-7a20ce27a6fso629051985a.0 for ; Mon, 26 Aug 2024 13:44:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705055; x=1725309855; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XzWn17lHZnGD5Tr66iu6y8/zqhTRy72agTNrtdvxzwE=; b=SeZvw41D3cX87l8mMLy/ETJFqvZkDxhmBDs8UXMRP+Zm/J5+Kdr+1b2DpX8sWpvysB ixGW0OCxSOYGDbAy47rlDy5zjfdBjN2BOpSS52bja3HRt0ZEpUBywOlM7lBzwrq5Fi/i D40Q5QkCGsYu0ynyMDL+BNlom2aZIb6fS5UbBd1KVXQfM44T6OII6QAqTIfpVy/5JJNm Md+bU41tjCHKVwgL1ZWCK0ctJnbDmwlWxvewq6+Z28iXp2XdYFt78yEfPN8Ny1q54NGs 8WuDwDBNI99ShDPmYWhDN+LUGtDimZq/+MpDfROer3h1IyLIuNarAzZwx5vPyW0Le3U5 EHHg== X-Gm-Message-State: AOJu0YwZynG+O9zsXf5oevP9w7qe+WlhaxfAE+fPa15vZzZGo/oO5pCu 5IphObq+5ohYizaK7vbcPe3SXG93k9fIMxAVtKZ914roj3im/QwWq0K73qDDNxX5U5jpGFD4lql /mHaSAP8qEvhl4zS1Wk1KVUlZO981rlnM0ZJwrqP6EOn87qW5pOkBqr3Oc5Z10VES2+OzBMgEuN BLlmLeUmNRs3PAP30rXYO7hZqw2Tll/KmdpXCB3FeJcMU= X-Received: by 2002:a05:620a:408d:b0:7a1:e93c:cd04 with SMTP id af79cd13be357-7a6896e0e1cmr1465307385a.9.1724705054756; Mon, 26 Aug 2024 13:44:14 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH4kwi4qTohFBFu7kWFKQGbZI396sZSJGmlCvwBl5XfpN0sU8TboRNIhnEyWXLKkBzi4MYP4w== X-Received: by 2002:a05:620a:408d:b0:7a1:e93c:cd04 with SMTP id af79cd13be357-7a6896e0e1cmr1465302885a.9.1724705054299; Mon, 26 Aug 2024 13:44:14 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:13 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 09/19] mm: New follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:43 -0400 Message-ID: <20240826204353.2228736-10-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce a pair of APIs to follow pfn mappings to get entry information. It's very similar to what follow_pte() does before, but different in that it recognizes huge pfn mappings. Signed-off-by: Peter Xu --- include/linux/mm.h | 31 ++++++++++ mm/memory.c | 150 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 181 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index d900f15b7650..161d496bfd18 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2373,6 +2373,37 @@ int follow_pte(struct vm_area_struct *vma, unsigned = long address, int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write); =20 +struct follow_pfnmap_args { + /** + * Inputs: + * @vma: Pointer to @vm_area_struct struct + * @address: the virtual address to walk + */ + struct vm_area_struct *vma; + unsigned long address; + /** + * Internals: + * + * The caller shouldn't touch any of these. + */ + spinlock_t *lock; + pte_t *ptep; + /** + * Outputs: + * + * @pfn: the PFN of the address + * @pgprot: the pgprot_t of the mapping + * @writable: whether the mapping is writable + * @special: whether the mapping is a special mapping (real PFN maps) + */ + unsigned long pfn; + pgprot_t pgprot; + bool writable; + bool special; +}; +int follow_pfnmap_start(struct follow_pfnmap_args *args); +void follow_pfnmap_end(struct follow_pfnmap_args *args); + extern void truncate_pagecache(struct inode *inode, loff_t new); extern void truncate_setsize(struct inode *inode, loff_t newsize); void pagecache_isize_extended(struct inode *inode, loff_t from, loff_t to); diff --git a/mm/memory.c b/mm/memory.c index 93c0c25433d0..0b136c398257 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6173,6 +6173,156 @@ int follow_pte(struct vm_area_struct *vma, unsigned= long address, } EXPORT_SYMBOL_GPL(follow_pte); =20 +static inline void pfnmap_args_setup(struct follow_pfnmap_args *args, + spinlock_t *lock, pte_t *ptep, + pgprot_t pgprot, unsigned long pfn_base, + unsigned long addr_mask, bool writable, + bool special) +{ + args->lock =3D lock; + args->ptep =3D ptep; + args->pfn =3D pfn_base + ((args->address & ~addr_mask) >> PAGE_SHIFT); + args->pgprot =3D pgprot; + args->writable =3D writable; + args->special =3D special; +} + +static inline void pfnmap_lockdep_assert(struct vm_area_struct *vma) +{ +#ifdef CONFIG_LOCKDEP + struct address_space *mapping =3D vma->vm_file->f_mapping; + + if (mapping) + lockdep_assert(lockdep_is_held(&vma->vm_file->f_mapping->i_mmap_rwsem) || + lockdep_is_held(&vma->vm_mm->mmap_lock)); + else + lockdep_assert(lockdep_is_held(&vma->vm_mm->mmap_lock)); +#endif +} + +/** + * follow_pfnmap_start() - Look up a pfn mapping at a user virtual address + * @args: Pointer to struct @follow_pfnmap_args + * + * The caller needs to setup args->vma and args->address to point to the + * virtual address as the target of such lookup. On a successful return, + * the results will be put into other output fields. + * + * After the caller finished using the fields, the caller must invoke + * another follow_pfnmap_end() to proper releases the locks and resources + * of such look up request. + * + * During the start() and end() calls, the results in @args will be valid + * as proper locks will be held. After the end() is called, all the fields + * in @follow_pfnmap_args will be invalid to be further accessed. Further + * use of such information after end() may require proper synchronizations + * by the caller with page table updates, otherwise it can create a + * security bug. + * + * If the PTE maps a refcounted page, callers are responsible to protect + * against invalidation with MMU notifiers; otherwise access to the PFN at + * a later point in time can trigger use-after-free. + * + * Only IO mappings and raw PFN mappings are allowed. The mmap semaphore + * should be taken for read, and the mmap semaphore cannot be released + * before the end() is invoked. + * + * This function must not be used to modify PTE content. + * + * Return: zero on success, negative otherwise. + */ +int follow_pfnmap_start(struct follow_pfnmap_args *args) +{ + struct vm_area_struct *vma =3D args->vma; + unsigned long address =3D args->address; + struct mm_struct *mm =3D vma->vm_mm; + spinlock_t *lock; + pgd_t *pgdp; + p4d_t *p4dp, p4d; + pud_t *pudp, pud; + pmd_t *pmdp, pmd; + pte_t *ptep, pte; + + pfnmap_lockdep_assert(vma); + + if (unlikely(address < vma->vm_start || address >=3D vma->vm_end)) + goto out; + + if (!(vma->vm_flags & (VM_IO | VM_PFNMAP))) + goto out; +retry: + pgdp =3D pgd_offset(mm, address); + if (pgd_none(*pgdp) || unlikely(pgd_bad(*pgdp))) + goto out; + + p4dp =3D p4d_offset(pgdp, address); + p4d =3D READ_ONCE(*p4dp); + if (p4d_none(p4d) || unlikely(p4d_bad(p4d))) + goto out; + + pudp =3D pud_offset(p4dp, address); + pud =3D READ_ONCE(*pudp); + if (pud_none(pud)) + goto out; + if (pud_leaf(pud)) { + lock =3D pud_lock(mm, pudp); + if (!unlikely(pud_leaf(pud))) { + spin_unlock(lock); + goto retry; + } + pfnmap_args_setup(args, lock, NULL, pud_pgprot(pud), + pud_pfn(pud), PUD_MASK, pud_write(pud), + pud_special(pud)); + return 0; + } + + pmdp =3D pmd_offset(pudp, address); + pmd =3D pmdp_get_lockless(pmdp); + if (pmd_leaf(pmd)) { + lock =3D pmd_lock(mm, pmdp); + if (!unlikely(pmd_leaf(pmd))) { + spin_unlock(lock); + goto retry; + } + pfnmap_args_setup(args, lock, NULL, pmd_pgprot(pmd), + pmd_pfn(pmd), PMD_MASK, pmd_write(pmd), + pmd_special(pmd)); + return 0; + } + + ptep =3D pte_offset_map_lock(mm, pmdp, address, &lock); + if (!ptep) + goto out; + pte =3D ptep_get(ptep); + if (!pte_present(pte)) + goto unlock; + pfnmap_args_setup(args, lock, ptep, pte_pgprot(pte), + pte_pfn(pte), PAGE_MASK, pte_write(pte), + pte_special(pte)); + return 0; +unlock: + pte_unmap_unlock(ptep, lock); +out: + return -EINVAL; +} +EXPORT_SYMBOL_GPL(follow_pfnmap_start); + +/** + * follow_pfnmap_end(): End a follow_pfnmap_start() process + * @args: Pointer to struct @follow_pfnmap_args + * + * Must be used in pair of follow_pfnmap_start(). See the start() function + * above for more information. + */ +void follow_pfnmap_end(struct follow_pfnmap_args *args) +{ + if (args->lock) + spin_unlock(args->lock); + if (args->ptep) + pte_unmap(args->ptep); +} +EXPORT_SYMBOL_GPL(follow_pfnmap_end); + #ifdef CONFIG_HAVE_IOREMAP_PROT /** * generic_access_phys - generic implementation for iomem mmap access --=20 2.45.0 From nobody Fri Dec 19 13:45:04 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8ACA919DFB6 for ; Mon, 26 Aug 2024 20:44:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705061; cv=none; b=OD1MmhGoHHXkSTCThFYYwY2nfTa7sdOlaGaAitXvrJYvz0id+fguI5K2emKOTX8HxL+EwGFaclQ5ZMQ3O/Ew4fv6hL+tFg79Mm1ul09pM0Pzku3VSqyh8L27gf+RcmspYLFKRCx+OkAjCLizAcBv05dK016OAQuAkogXsLoY0VE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705061; c=relaxed/simple; bh=Aa8NqcDK98lGFy3crPXQ+TfEjjdi6Ds7uzzU5PCZnys=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jmmmWoyyim8NO4UPc7O+UB7dWZb4vFsTM8LdPD1MGzqUrrCFahnFAFr4qR+LfwcPgl6Sf/7tQirVG4yNwfWx8UhIMMgOVScQx0QwyenwVXzRVlZJlZZwmaVfUVPZdSEPIR+PJ6PAlsuwclzqjh4jw40nx5Ffl1yaprugGnN9lMQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=MTLyfwYW; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="MTLyfwYW" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705058; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RcTv8PG6WUMces5zPq/dQ8qXZuImd+6bX+nNmvbIpbY=; b=MTLyfwYWhsMNf4pb8kDAIarW8+NJrX1jRYCOcCCbMnphHbPayWDQNhkXv+WbsSFVFDxAaU faWdGVgSt1FLMgroQvR8PefnTKq3LP7LuAT7w6shXP2qaNNOoqLv+l2IrZ2lefKUKDjAYt k5MJntAJ+WsiM6ZzFDmViVA3hIP7cWY= Received: from mail-yw1-f197.google.com (mail-yw1-f197.google.com [209.85.128.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-74-uh6gfzY6PyWrsiFCO8sivw-1; Mon, 26 Aug 2024 16:44:17 -0400 X-MC-Unique: uh6gfzY6PyWrsiFCO8sivw-1 Received: by mail-yw1-f197.google.com with SMTP id 00721157ae682-6ad26d5b061so84750427b3.2 for ; Mon, 26 Aug 2024 13:44:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705056; x=1725309856; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RcTv8PG6WUMces5zPq/dQ8qXZuImd+6bX+nNmvbIpbY=; b=bjw3IL2Obj08KdrNi1gxVFksi98QSEZ+O2+X4tz0B5qzjKDngIQiwGXgAbOCYm/Rv4 6k0OCxAO3fCBCtMYtlO9X4HKxO/plCgM+UeaD3Vq/gudodnk1Yl2yjmY8s00/Y3mWZDg iCCKQErZCzeqxvr92Ib3tBfuf1QYoXkRXNGzpLDgKVU4RxyU9refuIqRaOND98oRSqLE wi92S9chAzeBTWjdd4Q31XzLRlbfGhyOT18b3wA66Lj3NRwjNU5rz1aZYaeZpSRlOwwb 9PJsLqxajmolwNv10WLgcLSQw3GuYBO696tPTGnNo3X+QjCRY/5Yajb7ou14vJmkv6AS +9XQ== X-Gm-Message-State: AOJu0Yx/509XpVeUJ/1GL/aUdbWcznWXm12uvjdqfjAsPeDfoDpEfx6S iqmEOExCURqRXiTW7ejpDyJc1vpa8NiAKh7RB3jPfVFn4HR9tU6zo5oL++RU3qKJyoXY1233fqt 0F8j4ByayHFRHjUDOHBsDhadAgRGrhTIqRXY3W9bIXHfl3g0u0d1fBxkBrThsvOTRwesKAYGkgS 3Tv17Z9oZcYoZ7qBqT3Hbab4D9D2Y8O7ejbQEBa2spk+E= X-Received: by 2002:a05:690c:4246:b0:6b2:4154:e598 with SMTP id 00721157ae682-6c6262f43e4mr113946527b3.26.1724705056591; Mon, 26 Aug 2024 13:44:16 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGCZvjuJnmUbtCZKJF68AHOtk4V9xJHAv9nFBXLLIOg3bv7tZNlMwBWnRJYIMuW8OX1o2s1rQ== X-Received: by 2002:a05:690c:4246:b0:6b2:4154:e598 with SMTP id 00721157ae682-6c6262f43e4mr113946177b3.26.1724705056224; Mon, 26 Aug 2024 13:44:16 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:15 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 10/19] KVM: Use follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:44 -0400 Message-ID: <20240826204353.2228736-11-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the new pfnmap API to allow huge MMIO mappings for VMs. The rest work is done perfectly on the other side (host_pfn_mapping_level()). Cc: Paolo Bonzini Cc: Sean Christopherson Signed-off-by: Peter Xu --- virt/kvm/kvm_main.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index cb2b78e92910..f416d5e3f9c0 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -2860,13 +2860,11 @@ static int hva_to_pfn_remapped(struct vm_area_struc= t *vma, unsigned long addr, bool write_fault, bool *writable, kvm_pfn_t *p_pfn) { + struct follow_pfnmap_args args =3D { .vma =3D vma, .address =3D addr }; kvm_pfn_t pfn; - pte_t *ptep; - pte_t pte; - spinlock_t *ptl; int r; =20 - r =3D follow_pte(vma, addr, &ptep, &ptl); + r =3D follow_pfnmap_start(&args); if (r) { /* * get_user_pages fails for VM_IO and VM_PFNMAP vmas and does @@ -2881,21 +2879,19 @@ static int hva_to_pfn_remapped(struct vm_area_struc= t *vma, if (r) return r; =20 - r =3D follow_pte(vma, addr, &ptep, &ptl); + r =3D follow_pfnmap_start(&args); if (r) return r; } =20 - pte =3D ptep_get(ptep); - - if (write_fault && !pte_write(pte)) { + if (write_fault && !args.writable) { pfn =3D KVM_PFN_ERR_RO_FAULT; goto out; } =20 if (writable) - *writable =3D pte_write(pte); - pfn =3D pte_pfn(pte); + *writable =3D args.writable; + pfn =3D args.pfn; =20 /* * Get a reference here because callers of *hva_to_pfn* and @@ -2916,9 +2912,8 @@ static int hva_to_pfn_remapped(struct vm_area_struct = *vma, */ if (!kvm_try_get_pfn(pfn)) r =3D -EFAULT; - out: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); *p_pfn =3D pfn; =20 return r; --=20 2.45.0 From nobody Fri Dec 19 13:45:04 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 14B9B19E804 for ; Mon, 26 Aug 2024 20:44:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705063; cv=none; b=R1dcVDTKh8oBiB5KALP2ctKf8PRk9Zw7Ao9ECuC3AwCPyykZlPNSsdg6KhGVBlMX/5H+yvC6V9T5wSjHSfLf5KIV/lR7mma3or9mrZNhLoig8Uv4t7+l+2cWjJOHCG/aLgBHwGEH7HFOAQGwAuAjY80Ke4PD33Th2zOBiGeQb2w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705063; c=relaxed/simple; bh=F3zrw7GW4L3DDhmifn0KqTitiZnPFDtSbk8TCndoYpY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TKKWG2XhbTqF1y1NozHH3RSa5K2VksDuDSoruIUG+qCCaR4tRubwOC1R4wFfnFtlXJxz4vuEnlr+XtjzppcwJGcuvkVwFJYlWyayaG+5kScQtKZSh/DKend8fRdCykGWRWDMZwOWL2xtaIHTgP5gv+QQZIgAShNHvMfrI/n1Y/s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=CEepY3px; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="CEepY3px" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705061; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kaY3EfeA+k09YWgS+MfGM8qkjlv58srLyvVau/9jXnU=; b=CEepY3pxevilkMXy6m+MxhWH0JyC8nmTA/XhdMqGH/2HlbICCUCsx3wz7L8oYUtDoN7ZmX WSmiJbH3wx9P2OwWeBA9Jehz4kKiyaTWg19O2JaiUhERxxq06l8szczNhJip6oCwmXdrYp 2SyxlT9JDzvecjG7BJa/cX6ZkUChiA4= Received: from mail-oo1-f72.google.com (mail-oo1-f72.google.com [209.85.161.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-198-1nl-TaC_Oyysk9Ozd2LvyQ-1; Mon, 26 Aug 2024 16:44:20 -0400 X-MC-Unique: 1nl-TaC_Oyysk9Ozd2LvyQ-1 Received: by mail-oo1-f72.google.com with SMTP id 006d021491bc7-5d5cec5ab62so6030835eaf.3 for ; Mon, 26 Aug 2024 13:44:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705059; x=1725309859; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kaY3EfeA+k09YWgS+MfGM8qkjlv58srLyvVau/9jXnU=; b=ZQ/Bvi1D4sC1H7tDPkHqOcnth8Qgi9SgPZtRbnmf0bOdQqu7y0Au+noYy6TUKa7qWM 5zWUlFGAZXzQwEbhKx2Zak3WMHjZTucLaj/VoQ7Tc1IcKI9hEjqoZqAah7qqExFZ71nR o4hhX3ewEiQFevJo6x+kr/tvFrZKs33EMZfdPJ+xZVKb0Ty/ZTP0kNJaoM07/Um/0y6v yLesTncmNLZ+8QqdNDmprs2tlYsItqk3dzyX1eg5ZBrbAQ72l1Ffj5vsCdRK+o1be07r PBfStvrspc+XHGKyRk5KEBahg/Ir3nPRXPeaTmyhQIP75hMS5D74MPMkaoGJPmhu4FE3 epdg== X-Gm-Message-State: AOJu0Yysq/6Hm3nFIDS3F4GzAo49QMMQv1E2lSs6xmVcFLspuxvzRkFp rHTvhSKFytLlOon9xCMJhdnNZcwulqil5GE9wjAN0Woxok+BBOwt2/2+A0yKE3UeR/Qjj6+as82 Y5fBUPGjbJsi5uRIFtMvguCk5F2I9K/ePfNEWDJa08ICiqd5zUCVA0zO+FWGMudXIsFaJZvyA7s 1Oicrkdmw3c+Yoi4w11Z3iinSJYrlyTfEBTqR/MdApReg= X-Received: by 2002:a05:6358:7201:b0:1ac:f3df:3bde with SMTP id e5c5f4694b2df-1b5ebf3abdcmr117740455d.5.1724705059086; Mon, 26 Aug 2024 13:44:19 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHztBftH8jfc2wrFWJQjc0BC5uJ0IwQWlV9Rpv0YGeitSfcFoHRh3KGrU0k5GaslF87BwNh3Q== X-Received: by 2002:a05:6358:7201:b0:1ac:f3df:3bde with SMTP id e5c5f4694b2df-1b5ebf3abdcmr117734755d.5.1724705058531; Mon, 26 Aug 2024 13:44:18 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:18 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson , Niklas Schnelle , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , linux-s390@vger.kernel.org Subject: [PATCH v2 11/19] s390/pci_mmio: Use follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:45 -0400 Message-ID: <20240826204353.2228736-12-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the new API that can understand huge pfn mappings. Cc: Niklas Schnelle Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Alexander Gordeev Cc: Christian Borntraeger Cc: Sven Schnelle Cc: linux-s390@vger.kernel.org Signed-off-by: Peter Xu --- arch/s390/pci/pci_mmio.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/arch/s390/pci/pci_mmio.c b/arch/s390/pci/pci_mmio.c index 5398729bfe1b..de5c0b389a3e 100644 --- a/arch/s390/pci/pci_mmio.c +++ b/arch/s390/pci/pci_mmio.c @@ -118,12 +118,11 @@ static inline int __memcpy_toio_inuser(void __iomem *= dst, SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, mmio_addr, const void __user *, user_buffer, size_t, length) { + struct follow_pfnmap_args args =3D { }; u8 local_buf[64]; void __iomem *io_addr; void *buf; struct vm_area_struct *vma; - pte_t *ptep; - spinlock_t *ptl; long ret; =20 if (!zpci_is_enabled()) @@ -169,11 +168,13 @@ SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, m= mio_addr, if (!(vma->vm_flags & VM_WRITE)) goto out_unlock_mmap; =20 - ret =3D follow_pte(vma, mmio_addr, &ptep, &ptl); + args.address =3D mmio_addr; + args.vma =3D vma; + ret =3D follow_pfnmap_start(&args); if (ret) goto out_unlock_mmap; =20 - io_addr =3D (void __iomem *)((pte_pfn(*ptep) << PAGE_SHIFT) | + io_addr =3D (void __iomem *)((args.pfn << PAGE_SHIFT) | (mmio_addr & ~PAGE_MASK)); =20 if ((unsigned long) io_addr < ZPCI_IOMAP_ADDR_BASE) @@ -181,7 +182,7 @@ SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, mmi= o_addr, =20 ret =3D zpci_memcpy_toio(io_addr, buf, length); out_unlock_pt: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unlock_mmap: mmap_read_unlock(current->mm); out_free: @@ -260,12 +261,11 @@ static inline int __memcpy_fromio_inuser(void __user = *dst, SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mmio_addr, void __user *, user_buffer, size_t, length) { + struct follow_pfnmap_args args =3D { }; u8 local_buf[64]; void __iomem *io_addr; void *buf; struct vm_area_struct *vma; - pte_t *ptep; - spinlock_t *ptl; long ret; =20 if (!zpci_is_enabled()) @@ -308,11 +308,13 @@ SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mm= io_addr, if (!(vma->vm_flags & VM_WRITE)) goto out_unlock_mmap; =20 - ret =3D follow_pte(vma, mmio_addr, &ptep, &ptl); + args.vma =3D vma; + args.address =3D mmio_addr; + ret =3D follow_pfnmap_start(&args); if (ret) goto out_unlock_mmap; =20 - io_addr =3D (void __iomem *)((pte_pfn(*ptep) << PAGE_SHIFT) | + io_addr =3D (void __iomem *)((args.pfn << PAGE_SHIFT) | (mmio_addr & ~PAGE_MASK)); =20 if ((unsigned long) io_addr < ZPCI_IOMAP_ADDR_BASE) { @@ -322,7 +324,7 @@ SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mmio= _addr, ret =3D zpci_memcpy_fromio(buf, io_addr, length); =20 out_unlock_pt: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unlock_mmap: mmap_read_unlock(current->mm); =20 --=20 2.45.0 From nobody Fri Dec 19 13:45:04 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D4CAC19F48D for ; Mon, 26 Aug 2024 20:44:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705065; cv=none; b=oIjDtQ7aKyS2N51TJPpYc7nzjaebL9E2jgrI2cEL+pzNfxLG614h6D0XKdPFg5KMuqwjZqtE3vZlG0wvYQZ30K/pRIm+6ZwqdHmArXMUqNneBxTZutEOkUJ4Mgnio9Q1FPONOaagRsDyhLp5ChmcjIe9oBYcWpk+eEdNYeWMXUE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705065; c=relaxed/simple; bh=tylq1muqjeOnFYCjzZOC/g07tmHzw2/ZVGGdJsJupkM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CTMYJkHQKispQu96h+TlYs2yuTFhs0RnOSnQ9BHHkJ7m/FydfRd237Wrt5B4Ormf2YpWgNv6+wLLhzTBfPlQeyyrkfyD6nbaoIU6AUybcpHccd754qNSq4MANEqagRV49fvu7amI/ThyyjN1WKCgdA/xOaYV1yIrPpxR9RAHmoM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=HxcAdmuP; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="HxcAdmuP" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705063; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=moprxbUn5M7+L+63SZwwyqeq4iL0OHeeIJZbsfnkOho=; b=HxcAdmuPLoLk68HBgsFHMBqHzb3/HG0/m9q/YAKsQa2mSWy7hp18G1FsjSvE5iTW+OqvlS VMCq2Hv4UindeGb2QaUSEYKR4MrkG7UGBoHZBp3BSQXMi5rfSn2cms1lsBhXDtijYD5deA pvNZk1YZMrPOqcXq3j3aIVjCF6tY9kw= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-408-Wob4E6cPPrKtuoz43f_vBA-1; Mon, 26 Aug 2024 16:44:21 -0400 X-MC-Unique: Wob4E6cPPrKtuoz43f_vBA-1 Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-7a1d4335cceso884882085a.1 for ; Mon, 26 Aug 2024 13:44:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705061; x=1725309861; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=moprxbUn5M7+L+63SZwwyqeq4iL0OHeeIJZbsfnkOho=; b=ffX0qyfr+EYXxbDy37iwFOhMIiv5H1AephRWUHT1mY1WpDrvuhH2vCyJpMsh9GYb1y z98cDLYPh0EyQ4AHLfkVMwXydrhPbC+dIVJFLOSaxn8RWp7NoIJDZ1j/pm5wjowm+/kz cIxGUs2sFOzt+LfVJP/WbTNKoX1Mct9H3YmjiLUBii6pID8WbQDl5vurhYde2foGbbnI Fqc8+1Y13yYiYTxsXR+1aP6MBV9joInYLpdQP9ES1dzhE6eW7hExIm4pZLtPDanptjJ1 W6cQRBpGE22z+qLWhS56wxQzt1ZKSp8RKRPCxBp0gIUgmdOAtsChxSgRMa1LrHDgOGIS oAYQ== X-Gm-Message-State: AOJu0YyM9kGFZpiuEagdqFsFwhYbWTClsusftMagGoBiOi3eZL6hLprz yN4vcdWhTmuYbvVM9vMCT4VUA/OxJZ714CG5WNrRaKxRtYcLSHn+Nnm4GY4fIWeuIj3s/LD0i2J TEu+ppUJ7nRWVKzoJ5XxTfYUSUBxthoWFUuNWNsfG7WGjRZ+HmkSIh4wfYNeOACGLqbfaw2BinT OeRp+39NmDJVQdGt8RSo4lMQRJ58QTSlmtiClwpPd98wE= X-Received: by 2002:a05:620a:1a92:b0:7a1:e341:d543 with SMTP id af79cd13be357-7a7e4cf3a3fmr158799485a.28.1724705060771; Mon, 26 Aug 2024 13:44:20 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF8L+Naz3FkxDynSQ3yMse2qHglqFkf4LqBPi/5cD3sQEuFfYs6yWPMbT2Qk9XtpK98A+7trg== X-Received: by 2002:a05:620a:1a92:b0:7a1:e341:d543 with SMTP id af79cd13be357-7a7e4cf3a3fmr158793685a.28.1724705060390; Mon, 26 Aug 2024 13:44:20 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:19 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 12/19] mm/x86/pat: Use the new follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:46 -0400 Message-ID: <20240826204353.2228736-13-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the new API that can understand huge pfn mappings. Cc: x86@kernel.org Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Signed-off-by: Peter Xu --- arch/x86/mm/pat/memtype.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/arch/x86/mm/pat/memtype.c b/arch/x86/mm/pat/memtype.c index 1fa0bf6ed295..f73b5ce270b3 100644 --- a/arch/x86/mm/pat/memtype.c +++ b/arch/x86/mm/pat/memtype.c @@ -951,23 +951,20 @@ static void free_pfn_range(u64 paddr, unsigned long s= ize) static int follow_phys(struct vm_area_struct *vma, unsigned long *prot, resource_size_t *phys) { - pte_t *ptep, pte; - spinlock_t *ptl; + struct follow_pfnmap_args args =3D { .vma =3D vma, .address =3D vma->vm_s= tart }; =20 - if (follow_pte(vma, vma->vm_start, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) return -EINVAL; =20 - pte =3D ptep_get(ptep); - /* Never return PFNs of anon folios in COW mappings. */ - if (vm_normal_folio(vma, vma->vm_start, pte)) { - pte_unmap_unlock(ptep, ptl); + if (!args.special) { + follow_pfnmap_end(&args); return -EINVAL; } =20 - *prot =3D pgprot_val(pte_pgprot(pte)); - *phys =3D (resource_size_t)pte_pfn(pte) << PAGE_SHIFT; - pte_unmap_unlock(ptep, ptl); + *prot =3D pgprot_val(args.pgprot); + *phys =3D (resource_size_t)args.pfn << PAGE_SHIFT; + follow_pfnmap_end(&args); return 0; } =20 --=20 2.45.0 From nobody Fri Dec 19 13:45:04 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 138551A00D1 for ; Mon, 26 Aug 2024 20:44:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705067; cv=none; b=aHkMNnybE71Nv1iqleMykHsscnlw0SD4qN57+lvkzTIAc+ApPXhJYBrQeVDwGk0hiAXkN9gZXXDkMVnw4+PV36lHrwBoqhEqPyzSelp/YXF62a2eajqZ7N57PkA2lBxCYSbjXxHDsK6ey4R9lL6zMLelGNsxHHXT53GD0c+VH8Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705067; c=relaxed/simple; bh=BQuf5gTrb+LFvvLLmklTnMaR8Ti7tRzNRxtQXaWJ5ko=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JlE7I70UTiR8msX2u0CahdHProLEtj28gGggDRw8L6yID4yDUYaDh07bxBzN2+j51tRW7L6bc6bgjiB/qu8sfrbZ42qHI7LbjwuEKIey5Wl6eTnOIeCppqOJtHl0bYsTXQoBysU3P5agasYV8zLl1eAlqdkHOf3QIGFO6mQYosc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=ROlX2I7U; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ROlX2I7U" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705065; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HLBqit68L/U2mdwYaFkydFK3txHa+LkmL8n2YTQXnnw=; b=ROlX2I7Uk0h0z/EbbmrjDDC5L29amroUx7hw0sQuUWaJcsejLWRGRU00SYIbG5ySa8JXjC uqyVHAHgbUauLWAMse/7JAyxacnpa+tFTp2wyIVzB9I3jgPPt6i+dEz79NV8nepxwu/3Cm oJ3OMld4i7C3pSc++uPKF+BlKbQ9V3o= Received: from mail-yw1-f197.google.com (mail-yw1-f197.google.com [209.85.128.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-658-2Rh4yhghMe2FmPIezQmz3A-1; Mon, 26 Aug 2024 16:44:23 -0400 X-MC-Unique: 2Rh4yhghMe2FmPIezQmz3A-1 Received: by mail-yw1-f197.google.com with SMTP id 00721157ae682-6643016423fso96108047b3.3 for ; Mon, 26 Aug 2024 13:44:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705063; x=1725309863; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HLBqit68L/U2mdwYaFkydFK3txHa+LkmL8n2YTQXnnw=; b=HMlr6j/52tp66i3aMRgsC36P4O7LHk23fxzf7OT52NznTKJGhUVmAkkZjkzHvmX7IN xvXjpbwNtZKMxtMnWFhsPklSyrzrOiKxifq0OZQbSu1afrTE8csYMPqgfEFHrnlWYwAe OE2+PHoJRrPKREJrriperHslBqFrrLstvFO18gqjJ7nmKM3vAuvX4dqNNujAcrccIsyR iilHMYknzC+0ioEyAcCWqFZvt7bilXQKwKDcVg7SS9T3lESNsWP/sZe2u07o4QTlDxWW o5A6SZ/i7oP62DfSAio/X4V8t7bODPxh8NZFUPp369uBPZscdmy0Y7Xv9+oXTn79kQMB E/uQ== X-Gm-Message-State: AOJu0Yx2fTiRmIfDDFXhOH44XXb382Au7LuMgATU7pFY4EbA2LwIXmiJ /p0I8c2t3v6gUlD0LfW69JS4QNce4ia5W/NqYXcVcgkZ6X4gRVK6fmsqt0RFtVkGnBIFj+5dTZB EA8McMv2sl5pQb8Th62O95yrX9O5ZlxtSU9AdCwgH3RvizTFm0Q8bHfTNaEcD9bfoKzUxtxSDlV aHHzOBEEFEw6X13OgwjBpWpPSzT6gN3n7vslspomd1ZGI= X-Received: by 2002:a05:690c:39b:b0:62f:aaaa:187a with SMTP id 00721157ae682-6c625a4cc55mr151107117b3.14.1724705062820; Mon, 26 Aug 2024 13:44:22 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGeaoJv/EbB8r7mSzacjacGidgTJuNMUQKi5I30qkj93MEO5BseX564RMMiUfXmELj5ALvNSw== X-Received: by 2002:a05:690c:39b:b0:62f:aaaa:187a with SMTP id 00721157ae682-6c625a4cc55mr151106717b3.14.1724705062426; Mon, 26 Aug 2024 13:44:22 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:21 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 13/19] vfio: Use the new follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:47 -0400 Message-ID: <20240826204353.2228736-14-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the new API that can understand huge pfn mappings. Cc: Alex Williamson Cc: Jason Gunthorpe Signed-off-by: Peter Xu --- drivers/vfio/vfio_iommu_type1.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type= 1.c index 0960699e7554..bf391b40e576 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -513,12 +513,10 @@ static int follow_fault_pfn(struct vm_area_struct *vm= a, struct mm_struct *mm, unsigned long vaddr, unsigned long *pfn, bool write_fault) { - pte_t *ptep; - pte_t pte; - spinlock_t *ptl; + struct follow_pfnmap_args args =3D { .vma =3D vma, .address =3D vaddr }; int ret; =20 - ret =3D follow_pte(vma, vaddr, &ptep, &ptl); + ret =3D follow_pfnmap_start(&args); if (ret) { bool unlocked =3D false; =20 @@ -532,19 +530,17 @@ static int follow_fault_pfn(struct vm_area_struct *vm= a, struct mm_struct *mm, if (ret) return ret; =20 - ret =3D follow_pte(vma, vaddr, &ptep, &ptl); + ret =3D follow_pfnmap_start(&args); if (ret) return ret; } =20 - pte =3D ptep_get(ptep); - - if (write_fault && !pte_write(pte)) + if (write_fault && !args.writable) ret =3D -EFAULT; else - *pfn =3D pte_pfn(pte); + *pfn =3D args.pfn; =20 - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); return ret; } =20 --=20 2.45.0 From nobody Fri Dec 19 13:45:04 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EE0131A01B4 for ; Mon, 26 Aug 2024 20:44:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705069; cv=none; b=Df1NOVj3CCWdz554pG96WRzx/3UtMcf3dnqbLVJfuTL5c4NAkb92waxIKOWyVpZ710vzhxr2bN+pk8JAQFHDcH0abOlVIOXOsIVTFs14SwpVbC69d34pqTjjz8yiFMAGIc9bK54lHST2vynlWp6WS3p4nE35KxJf//XwiDKvG/E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705069; c=relaxed/simple; bh=RxaKGf99KcIXWtv8SyMqO8KfjLx2VTZrfelSTDMLhuM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Lx6h8AZnxKnXTBT7qm7g7rlIDXBgenkcYH5z4HlUFL+Jqc0XIy2fd17lzDX5IODxLRLegVuS1blHI28FJdmTZuh8qYZHcgvRrmnB1yh3UI95mindnCbIR/1GselFYTMKUTqAnXvYKej5H5k/wZF1VImXyz6/47dQVUzAUvt0BV4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=XHPLT0oc; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="XHPLT0oc" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705066; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d0kI5he4gLOJUpsF87CpEeiS1FOCa69CaYI81su9df0=; b=XHPLT0ocyeIQ6bYspsizMfyy1L6CijBqwhoWZYQQKXNk5pXtlAAmZMWKWiCexzSL0kNvjI +TbjBiSfURXsDzVHKba+Jtk3fRwp2PyRZmfY4iq+qek+n77XJpc/GLCkS2p1OUneD2wf70 i3Zj+AJdEOOwMpGLsPMKrRUv81iBb1s= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-80-dEu0rxf5Nmqf_AHydX0Tyw-1; Mon, 26 Aug 2024 16:44:25 -0400 X-MC-Unique: dEu0rxf5Nmqf_AHydX0Tyw-1 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-7a1dbbe6d6fso633463085a.3 for ; Mon, 26 Aug 2024 13:44:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705064; x=1725309864; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d0kI5he4gLOJUpsF87CpEeiS1FOCa69CaYI81su9df0=; b=Py0L0XL9lnzd3qgzwntIlrW6bFYBFcPadQVyYEhGVRI3Nc74bNDbEfgkDuKUKY7OEp XAizfzHfmApGS/RgeSTr40jybYegSfyYPwoS6P5hgJ1IDviWUNWb7umFGBFkM6sJGmQx dquDXlsEURe89kMg3vPkxCGJA51x6wrIip1QgnCsJSY8GoGOCWX/ugPf1H41IZuuiiDa IaaWqJpCtuxwZx6cPAEUWeCQnrk0KT+V/oreJkIvFaXQNCnaCWaykOVfXKvcP8PaONRU hVSc4HcEGB7ZwPOKWstV7/1Ap9j3UX/YfplCfy9zWgtzN8twR7mm2q9j68prvKLyWQgs 5neQ== X-Gm-Message-State: AOJu0YzXXgkhEO5/60yk2QiWEb6BPS5epjD0OC3xog50qNJgHIgLYDL0 kgjQBteaCbMq96i/Rha2m/AWXCPKcnmcQcqsUjjN8t5b3J/+zXNTQnoHZj+gVasAE2+oE0ji7Bb Pj9tBpdDuw78ZbI/BZRUWQOtBoDjRPmLWC+QFJPEsM38qTZhOt6ccl1xjciyOmp/FXik5RVZ+Ed PlMxbut+dvharf9PYkMAg20X8/fD70Jrf4v4NSxUZAH6Q= X-Received: by 2002:a05:620a:2484:b0:79e:ff41:fd47 with SMTP id af79cd13be357-7a7e4dc9528mr93290485a.28.1724705064691; Mon, 26 Aug 2024 13:44:24 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEloa5qz2QVMofE8PCWJJ6gTfdMgC7cjn+h2anZ9qsldjcOaIzzaCXqVOAeeqkMcyU6B13FuA== X-Received: by 2002:a05:620a:2484:b0:79e:ff41:fd47 with SMTP id af79cd13be357-7a7e4dc9528mr93286485a.28.1724705064315; Mon, 26 Aug 2024 13:44:24 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:23 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 14/19] acrn: Use the new follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:48 -0400 Message-ID: <20240826204353.2228736-15-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the new API that can understand huge pfn mappings. Signed-off-by: Peter Xu --- drivers/virt/acrn/mm.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/drivers/virt/acrn/mm.c b/drivers/virt/acrn/mm.c index db8ff1d0ac23..4c2f28715b70 100644 --- a/drivers/virt/acrn/mm.c +++ b/drivers/virt/acrn/mm.c @@ -177,9 +177,7 @@ int acrn_vm_ram_map(struct acrn_vm *vm, struct acrn_vm_= memmap *memmap) vma =3D vma_lookup(current->mm, memmap->vma_base); if (vma && ((vma->vm_flags & VM_PFNMAP) !=3D 0)) { unsigned long start_pfn, cur_pfn; - spinlock_t *ptl; bool writable; - pte_t *ptep; =20 if ((memmap->vma_base + memmap->len) > vma->vm_end) { mmap_read_unlock(current->mm); @@ -187,16 +185,20 @@ int acrn_vm_ram_map(struct acrn_vm *vm, struct acrn_v= m_memmap *memmap) } =20 for (i =3D 0; i < nr_pages; i++) { - ret =3D follow_pte(vma, memmap->vma_base + i * PAGE_SIZE, - &ptep, &ptl); + struct follow_pfnmap_args args =3D { + .vma =3D vma, + .address =3D memmap->vma_base + i * PAGE_SIZE, + }; + + ret =3D follow_pfnmap_start(&args); if (ret) break; =20 - cur_pfn =3D pte_pfn(ptep_get(ptep)); + cur_pfn =3D args.pfn; if (i =3D=3D 0) start_pfn =3D cur_pfn; - writable =3D !!pte_write(ptep_get(ptep)); - pte_unmap_unlock(ptep, ptl); + writable =3D args.writable; + follow_pfnmap_end(&args); =20 /* Disallow write access if the PTE is not writable. */ if (!writable && --=20 2.45.0 From nobody Fri Dec 19 13:45:04 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A6D2D1A01DB for ; Mon, 26 Aug 2024 20:44:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705071; cv=none; b=HxgmESalJihDU2kghNU1ZDBMpJvGpR7ohcz6yXE9nflOZ/eRyrJ1IDEixkawLHQonzRKiz65sESRKdGdv4rTvntwm2VtRuV6pnAKyBYiUVuxbUhewjoNdY1C3g7WUk2yMIqc5PfWU4NaIO2zmDxpS403LhsDf6mDeOdlDTk88Ro= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705071; c=relaxed/simple; bh=hpLmEYhsByAEmqEdBsP006I9UauZryxW/FlKEPWnJc4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=foliOG1ioSZ/GpC5VLARFpBkC3YFkbvdOgHZU5LarTYOgoAr3wvqiAZ0JC7qcxjgHFUrKOtDjB1pJux78JezjakFAFWyytFULk1zCG/7zbYLybu8eX4b8Ai1T/v40wclXGgKBqljyvl8wNwQSPTXJa+JjkWInEGjDNWuA4YT7c0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=RE2//aLs; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="RE2//aLs" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705068; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t75nynj+5ccZ6qy3UooqAYOOLyCQeJyCwmKKtTf1Ss4=; b=RE2//aLsidYqxI/Vq6eXXVa8odw+UOLLRE4cBut+e9JGmNhrdEGZav1WTRmgPolejdkoS2 GXNceF38y1X7f85RMpx7fRHAn8o442O4PKfKb9LIQkO53QWXQISHWxIDypra9ZkyCgoLm1 ySXdhrh0LQKdxI7FLpsx0tu4+ft/VvY= Received: from mail-oa1-f72.google.com (mail-oa1-f72.google.com [209.85.160.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-694-D7N-VvAvOwC6mlQfdZ9-7Q-1; Mon, 26 Aug 2024 16:44:27 -0400 X-MC-Unique: D7N-VvAvOwC6mlQfdZ9-7Q-1 Received: by mail-oa1-f72.google.com with SMTP id 586e51a60fabf-27061a48e70so6268040fac.1 for ; Mon, 26 Aug 2024 13:44:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705066; x=1725309866; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t75nynj+5ccZ6qy3UooqAYOOLyCQeJyCwmKKtTf1Ss4=; b=MNnhRXH2hWKqWD7GuqQiTNR0Oqk8pbmXq1inENNDPRmW/tD2XP9QtNI9bSGgqlSosb nngxzAAV9IUrJj/rUzEtCOfdt7LVXbQVD24mnteAeI+TbPnkODecTHdk7bSx3w7INT1a yWBw3bYN2kpqgVPv8JrIbl+PRno7gl9ntxpf9buboZZusbbm7OCzrOvOW0pKI5W3dVqh Ff4OdRzpKzJPzUoD9jgVm+I9e0j6t/L9PSwXM/kbun7C2wBzMDEn1moOODRLHAwkFpFe rHO/ZjdL1ie7b2CGcHYlaKAYTN26D0ILLzoNuyLiAu/n82697yC17+4DbK5SK71X5ntq 4Xlw== X-Gm-Message-State: AOJu0YxgU2mMFFCG6cAejK1SM2KsztbSMnnU+6yWauissbZO7DfDyiE1 V0CZ05kWYalsGeHyXlwNdKpIthMO+s6jz3jMGzY8c7wPI3e5qdTzSF+OSNvnMM8uXlAdl0Id1qR egRNEzTDiR0AhmbzoZ7kiVc7lit3BbuMp4OodkIdbkiZm4KxayAFNKOW/Ya0e9XBz/jK5QgDEvR L7TaXnVCp3rmaBSJIbjzYs9ZZOEzFPt7A+DR28JAtm7U8= X-Received: by 2002:a05:6358:3a07:b0:1ad:10ff:341e with SMTP id e5c5f4694b2df-1b5c3a3b69dmr1457845255d.4.1724705066593; Mon, 26 Aug 2024 13:44:26 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGoPajws+HFHWojCbb3D8qBEyBE/zIeN3nkxAxD5pVBn23+qUQ20h6jqSHyvqmBY4DTTGYw+g== X-Received: by 2002:a05:6358:3a07:b0:1ad:10ff:341e with SMTP id e5c5f4694b2df-1b5c3a3b69dmr1457841255d.4.1724705066240; Mon, 26 Aug 2024 13:44:26 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:25 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 15/19] mm/access_process_vm: Use the new follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:49 -0400 Message-ID: <20240826204353.2228736-16-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the new API that can understand huge pfn mappings. Signed-off-by: Peter Xu --- mm/memory.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 0b136c398257..b5d07f493d5d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6342,34 +6342,34 @@ int generic_access_phys(struct vm_area_struct *vma,= unsigned long addr, resource_size_t phys_addr; unsigned long prot =3D 0; void __iomem *maddr; - pte_t *ptep, pte; - spinlock_t *ptl; int offset =3D offset_in_page(addr); int ret =3D -EINVAL; + bool writable; + struct follow_pfnmap_args args =3D { .vma =3D vma, .address =3D addr }; =20 retry: - if (follow_pte(vma, addr, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) return -EINVAL; - pte =3D ptep_get(ptep); - pte_unmap_unlock(ptep, ptl); + prot =3D pgprot_val(args.pgprot); + phys_addr =3D (resource_size_t)args.pfn << PAGE_SHIFT; + writable =3D args.writable; + follow_pfnmap_end(&args); =20 - prot =3D pgprot_val(pte_pgprot(pte)); - phys_addr =3D (resource_size_t)pte_pfn(pte) << PAGE_SHIFT; - - if ((write & FOLL_WRITE) && !pte_write(pte)) + if ((write & FOLL_WRITE) && !writable) return -EINVAL; =20 maddr =3D ioremap_prot(phys_addr, PAGE_ALIGN(len + offset), prot); if (!maddr) return -ENOMEM; =20 - if (follow_pte(vma, addr, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) goto out_unmap; =20 - if (!pte_same(pte, ptep_get(ptep))) { - pte_unmap_unlock(ptep, ptl); + if ((prot !=3D pgprot_val(args.pgprot)) || + (phys_addr !=3D (args.pfn << PAGE_SHIFT)) || + (writable !=3D args.writable)) { + follow_pfnmap_end(&args); iounmap(maddr); - goto retry; } =20 @@ -6378,7 +6378,7 @@ int generic_access_phys(struct vm_area_struct *vma, u= nsigned long addr, else memcpy_fromio(buf, maddr + offset, len); ret =3D len; - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unmap: iounmap(maddr); =20 --=20 2.45.0 From nobody Fri Dec 19 13:45:04 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C59B01A0724 for ; Mon, 26 Aug 2024 20:44:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705073; cv=none; b=d5Bv5ZefMMEMOqplBZOLk+khfcpbcCKfDaiSUsTOoXuWRMFFcBZPZKzRo5Ltpx8cR35greogeuGtAK8luAc59ERH3KhTB+V47SW4HOQ0CyQiPNcrVbhh2AzLXFE6CrEJ5Ewp3garZ98OrsBP1LThjP1bVkm33hGaFeSDa605Ejg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705073; c=relaxed/simple; bh=Eh2leUymwhy+QIbtX8/+/NdJiGHoDFC3uu2OHYBcoWg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=foYLCKY+LPR7bntjhVyzQGe3EcXwDy3hjLV/xg8/ESLnnIcQPhyWHdWnSggDyt9jtntB0cAh7BbJ+yInyEOaZQgUUUDGSvy48KBIVzfoT/2L9wcOGrXcFWfFD4G1AL0Dy4swq6vmZdN0cx9xZ3bW3x4iPJoBOU9fhJbz3gVZ80U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Ewhu568i; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Ewhu568i" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705071; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X8WHCJ7kNzF2XCQH70BbHdio2suKNHxAf6rhsHMkPbs=; b=Ewhu568iJ9CatpSr3LyDGcOxqb3mTz4UUMA95Hw9gFMfyFgOkzBPc9QxfulJma9qSWNtQh N6emNFuCeuYPLBlfbqtm8IK2yzds4P9Jup1iYKPLmgh/ufkIMwosoZTcuV/7cTTQW7PBeF dZFCZCNY8GgfBizZUEGLUhLa5Cb3gxY= Received: from mail-ua1-f71.google.com (mail-ua1-f71.google.com [209.85.222.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-192-22pvaT7yNBmpHjIuC1GImQ-1; Mon, 26 Aug 2024 16:44:29 -0400 X-MC-Unique: 22pvaT7yNBmpHjIuC1GImQ-1 Received: by mail-ua1-f71.google.com with SMTP id a1e0cc1a2514c-842f9f7509eso1359989241.2 for ; Mon, 26 Aug 2024 13:44:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705068; x=1725309868; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=X8WHCJ7kNzF2XCQH70BbHdio2suKNHxAf6rhsHMkPbs=; b=uIlXDORgEb0FBbOgVAgiGBcytpsU+JWl8hdQYynbpQVkXhTjAj/m4RueIBcO+d3e87 2ciEIm5a/8oBCCY61w8Xi8siBl0RpqLhpxeo2Z9M7s2Zaxwpmte3A1Y8CYVTm9OGBXjb X7Ky6d5Dq86vhmfBRf8vHIy8FNFSXNIB1MQR1Fl4+ZDYEUwkI2uCkZNqPNHnF6lkKqMG 6wK81OoYXiLrFJrOosyO/gYtbfcb6Bpf3SHVm462ijjVO8k1Glvxh+UjR1h6ZM4znz1M JFQwmdwvYz6YDJppE/mf3y61t34aRZ9LOHoqAImVWTzD1k/uY2eUaKappfYwPSFBbUtq Zdcw== X-Gm-Message-State: AOJu0YyJm0eqaj/504h5Z3haJAlkWjBXqaA1ToOAEKYkdj0R/itKAZJp jP+mLc5NaxXTHk/WX12U0pH7jYXyGnpqI8LDRkPtNiT3BQTsF/zsLUrG/KWckX6Y5OqNECcB1M5 fudWhOTbN/Nj5MWHNzm44oHxGk/MG3kmJLnXxmGdToKiRLj/9+H5ze48CEZeZ9B8uTrSVTG6J3j +l/JC4xyfXtnsQz60m3fnWcXOdwwc6SCAMLDsbo+d9QVw= X-Received: by 2002:a05:6102:3e94:b0:493:e585:6ce3 with SMTP id ada2fe7eead31-49a3bd1cd83mr1005209137.31.1724705068555; Mon, 26 Aug 2024 13:44:28 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGKOe4NKlkU/HSIr+h3CCvXyG5C70MeRvYsCyQPr65oWdFZ+m1iMwzpLlsyBV/cJVa8qCplQQ== X-Received: by 2002:a05:6102:3e94:b0:493:e585:6ce3 with SMTP id ada2fe7eead31-49a3bd1cd83mr1005176137.31.1724705068153; Mon, 26 Aug 2024 13:44:28 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:27 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 16/19] mm: Remove follow_pte() Date: Mon, 26 Aug 2024 16:43:50 -0400 Message-ID: <20240826204353.2228736-17-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" follow_pte() users have been converted to follow_pfnmap*(). Remove the API. Signed-off-by: Peter Xu --- include/linux/mm.h | 2 -- mm/memory.c | 73 ---------------------------------------------- 2 files changed, 75 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 161d496bfd18..b31d4bdd65ad 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2368,8 +2368,6 @@ void free_pgd_range(struct mmu_gather *tlb, unsigned = long addr, unsigned long end, unsigned long floor, unsigned long ceiling); int copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src= _vma); -int follow_pte(struct vm_area_struct *vma, unsigned long address, - pte_t **ptepp, spinlock_t **ptlp); int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write); =20 diff --git a/mm/memory.c b/mm/memory.c index b5d07f493d5d..288f81a8698e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6100,79 +6100,6 @@ int __pmd_alloc(struct mm_struct *mm, pud_t *pud, un= signed long address) } #endif /* __PAGETABLE_PMD_FOLDED */ =20 -/** - * follow_pte - look up PTE at a user virtual address - * @vma: the memory mapping - * @address: user virtual address - * @ptepp: location to store found PTE - * @ptlp: location to store the lock for the PTE - * - * On a successful return, the pointer to the PTE is stored in @ptepp; - * the corresponding lock is taken and its location is stored in @ptlp. - * - * The contents of the PTE are only stable until @ptlp is released using - * pte_unmap_unlock(). This function will fail if the PTE is non-present. - * Present PTEs may include PTEs that map refcounted pages, such as - * anonymous folios in COW mappings. - * - * Callers must be careful when relying on PTE content after - * pte_unmap_unlock(). Especially if the PTE maps a refcounted page, - * callers must protect against invalidation with MMU notifiers; otherwise - * access to the PFN at a later point in time can trigger use-after-free. - * - * Only IO mappings and raw PFN mappings are allowed. The mmap semaphore - * should be taken for read. - * - * This function must not be used to modify PTE content. - * - * Return: zero on success, -ve otherwise. - */ -int follow_pte(struct vm_area_struct *vma, unsigned long address, - pte_t **ptepp, spinlock_t **ptlp) -{ - struct mm_struct *mm =3D vma->vm_mm; - pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *ptep; - - mmap_assert_locked(mm); - if (unlikely(address < vma->vm_start || address >=3D vma->vm_end)) - goto out; - - if (!(vma->vm_flags & (VM_IO | VM_PFNMAP))) - goto out; - - pgd =3D pgd_offset(mm, address); - if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) - goto out; - - p4d =3D p4d_offset(pgd, address); - if (p4d_none(*p4d) || unlikely(p4d_bad(*p4d))) - goto out; - - pud =3D pud_offset(p4d, address); - if (pud_none(*pud) || unlikely(pud_bad(*pud))) - goto out; - - pmd =3D pmd_offset(pud, address); - VM_BUG_ON(pmd_trans_huge(*pmd)); - - ptep =3D pte_offset_map_lock(mm, pmd, address, ptlp); - if (!ptep) - goto out; - if (!pte_present(ptep_get(ptep))) - goto unlock; - *ptepp =3D ptep; - return 0; -unlock: - pte_unmap_unlock(ptep, *ptlp); -out: - return -EINVAL; -} -EXPORT_SYMBOL_GPL(follow_pte); - static inline void pfnmap_args_setup(struct follow_pfnmap_args *args, spinlock_t *lock, pte_t *ptep, pgprot_t pgprot, unsigned long pfn_base, --=20 2.45.0 From nobody Fri Dec 19 13:45:04 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DE52A1A08A8 for ; Mon, 26 Aug 2024 20:44:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705075; cv=none; b=W0hyyX/Lrvo78J1tafM5hk56GPeU1bWMbWK0m1Y/5SJmZ/LO0AiXKg/6Azy/MuB/fDVOl3P3xo29wljf/QCHTsMPTid2R2/sN5ICRIhF64C177fJ8pD5jP7Apon/Hxj0iI5kAzLs6rJZLTpYJVJVj9SjGHXJM0iJuvXjBwj6Wfo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705075; c=relaxed/simple; bh=wO6hxJPqfj9KTLTi6qw+v+hbYJLMemjraAGsqutmV4c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HSGXv9IRhuLOlpZgErFkLTAS1h52geht77BIuCPITRsuteEvdHGNYqVYj+vKjOtnmLjs9EI+/GVdGMG0MWZgPQe3Kg7vLVZ8FEa8rkChVmvo4Pv1VMWQfJachNNKct+FDDCPzlCetB/PC/m69OUD710I9bhTmIyJSfUPDVncTLI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Xnryqcbg; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Xnryqcbg" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705073; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IC5gYR+7Z4BwAPnZp8TN8XZEGypCYGRNkok3NV40z5s=; b=XnryqcbgleFQu7wsb3ArLf6EE3uUSWK//GXhMsmI9jW89K3rTSNw8kygWmA0SGqz0athRd P0i+HBcJyIOLsfUVtZBF9Mdevx+gcNdBbkaevSQ3/zDJYfmURFjtntrnfHi53v0EQHdB73 oIVlDZAi35dyCKAnpTAe3PGb1GScXck= Received: from mail-oo1-f71.google.com (mail-oo1-f71.google.com [209.85.161.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-133-xck-W9kjNUC6aprW_elcBQ-1; Mon, 26 Aug 2024 16:44:31 -0400 X-MC-Unique: xck-W9kjNUC6aprW_elcBQ-1 Received: by mail-oo1-f71.google.com with SMTP id 006d021491bc7-5d5c7bfd8aaso6416856eaf.2 for ; Mon, 26 Aug 2024 13:44:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705070; x=1725309870; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IC5gYR+7Z4BwAPnZp8TN8XZEGypCYGRNkok3NV40z5s=; b=hTVtUKBC8VHgiJNEYCH5MPuvYyRvKMdPGhCJy0MepyWpBNXYTsqWV56rU7vGqmGPg5 Xm2pEP5blUW9fjDvAe0sbL922Wi/do3pniY8YgwxJbaBMi2t1ecHpaTZgVIX398BdBmp eL9NLFz30TA4EX4NuoAzJ+2468QBs0Vo7CO3WGF2ZVx5+Uc5Crs1svcWHqBmVHyNDO++ gA/0QlLsqKaylujD7R0x3DaP5dPwulkUPbPVTHY9MlKt8BhKEXbZBhgHwew/y0NYOM+n OWLfzYY2G3vsMXD9VyR+LkBKryRfMqYJef4YD48I8itbkirqUc+kMDR8CjmB8RfL1ntq or8A== X-Gm-Message-State: AOJu0YxwzqzsyGbJQ8yzOlKVv5SyZzSJN8UbOhKlOD/hvfxPmR5U/d+L Oc54shwdZ+jsoJc8j9Wc8ydDmSq0san2av2B5ij6qKd6IFrjx3C59OJB9cKLLaGR0mcqz/hlRuO swLAHLg4M2YLeDUIMXJ435WoWr90Qrn3T8eHMPfq9V3O0OBxZEBl37pplmwdT9i4EAip2Xw6lYY qVdXsKSoDLgRP3DqnxpR8cUhnMcsZMolO6H1F0/Gf1igk= X-Received: by 2002:a05:6358:70c3:b0:1ac:f00d:c8c6 with SMTP id e5c5f4694b2df-1b5c22ebd07mr1493543455d.27.1724705070710; Mon, 26 Aug 2024 13:44:30 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHWRaHJzCfl4jAxAoYVG6HBrfG1+v3SSmaZHfZSlba6wG4sR68STMcFscyvu7mDZcazVEGWKw== X-Received: by 2002:a05:6358:70c3:b0:1ac:f00d:c8c6 with SMTP id e5c5f4694b2df-1b5c22ebd07mr1493538355d.27.1724705070146; Mon, 26 Aug 2024 13:44:30 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:29 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 17/19] mm/x86: Support large pfn mappings Date: Mon, 26 Aug 2024 16:43:51 -0400 Message-ID: <20240826204353.2228736-18-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Helpers to install and detect special pmd/pud entries. In short, bit 9 on x86 is not used for pmd/pud, so we can directly define them the same as the pte level. One note is that it's also used in _PAGE_BIT_CPA_TEST but that is only used in the debug test, and shouldn't conflict in this case. One note is that pxx_set|clear_flags() for pmd/pud will need to be moved upper so that they can be referenced by the new special bit helpers. There's no change in the code that was moved. Cc: x86@kernel.org Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Signed-off-by: Peter Xu --- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 80 ++++++++++++++++++++++------------ 2 files changed, 53 insertions(+), 28 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index b74b9ee484da..d4dbe9717e96 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -28,6 +28,7 @@ config X86_64 select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_PER_VMA_LOCK + select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 8d12bfad6a1d..4c2d080d26b4 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -120,6 +120,34 @@ extern pmdval_t early_pmd_flags; #define arch_end_context_switch(prev) do {} while(0) #endif /* CONFIG_PARAVIRT_XXL */ =20 +static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set) +{ + pmdval_t v =3D native_pmd_val(pmd); + + return native_make_pmd(v | set); +} + +static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) +{ + pmdval_t v =3D native_pmd_val(pmd); + + return native_make_pmd(v & ~clear); +} + +static inline pud_t pud_set_flags(pud_t pud, pudval_t set) +{ + pudval_t v =3D native_pud_val(pud); + + return native_make_pud(v | set); +} + +static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) +{ + pudval_t v =3D native_pud_val(pud); + + return native_make_pud(v & ~clear); +} + /* * The following only work if pte_present() is true. * Undefined behaviour if not.. @@ -317,6 +345,30 @@ static inline int pud_devmap(pud_t pud) } #endif =20 +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +static inline bool pmd_special(pmd_t pmd) +{ + return pmd_flags(pmd) & _PAGE_SPECIAL; +} + +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return pmd_set_flags(pmd, _PAGE_SPECIAL); +} +#endif /* CONFIG_ARCH_SUPPORTS_PMD_PFNMAP */ + +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +static inline bool pud_special(pud_t pud) +{ + return pud_flags(pud) & _PAGE_SPECIAL; +} + +static inline pud_t pud_mkspecial(pud_t pud) +{ + return pud_set_flags(pud, _PAGE_SPECIAL); +} +#endif /* CONFIG_ARCH_SUPPORTS_PUD_PFNMAP */ + static inline int pgd_devmap(pgd_t pgd) { return 0; @@ -487,20 +539,6 @@ static inline pte_t pte_mkdevmap(pte_t pte) return pte_set_flags(pte, _PAGE_SPECIAL|_PAGE_DEVMAP); } =20 -static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set) -{ - pmdval_t v =3D native_pmd_val(pmd); - - return native_make_pmd(v | set); -} - -static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) -{ - pmdval_t v =3D native_pmd_val(pmd); - - return native_make_pmd(v & ~clear); -} - /* See comments above mksaveddirty_shift() */ static inline pmd_t pmd_mksaveddirty(pmd_t pmd) { @@ -595,20 +633,6 @@ static inline pmd_t pmd_mkwrite_novma(pmd_t pmd) pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma); #define pmd_mkwrite pmd_mkwrite =20 -static inline pud_t pud_set_flags(pud_t pud, pudval_t set) -{ - pudval_t v =3D native_pud_val(pud); - - return native_make_pud(v | set); -} - -static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) -{ - pudval_t v =3D native_pud_val(pud); - - return native_make_pud(v & ~clear); -} - /* See comments above mksaveddirty_shift() */ static inline pud_t pud_mksaveddirty(pud_t pud) { --=20 2.45.0 From nobody Fri Dec 19 13:45:04 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C399E1A08CC for ; Mon, 26 Aug 2024 20:44:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705077; cv=none; b=NK9mAUZvfSK2ss+igNdUs/tKtVDv1KjdSt6dSzS6AVm/sgalDPHwmTqWacfviz7aC9K0cMT6ejK3jBw8GmqBCSUw6H65dOVYX2B+K8++TkYWc0ptvV+5Ljkb/nfTpVJ3QdmZETOCVyvT4aRhUSjbpTp5DJIFaK5oB5DnUR4me58= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705077; c=relaxed/simple; bh=5/RkLDpH7n+0lRZdC6d7Fr9a2r7GbmT3sDiSd48lk+g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ospTrmDuypd99uHxSfa85KHHIcQq5xVV3MuaYImP95lhv12txVt3ulJ5cDTvXF8KHn3dP5IwEWTOzkDt0gDY9tIckVgfivYOqCXy5C6ItI15XUUFCfZTMCX+XE2V8wN6h6cNaOh+d8hoWfNMm20WnuQ0Whi0VZNX3LFTW1JDp6w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=GzpqRune; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="GzpqRune" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705075; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FEAM9DK74WZYySVYhL7YnC2YOY0xQ42haB8uzH2x2BY=; b=GzpqRune84QmLOWjJZXXnUAv2Lg2O4s+kwWDFqH1+eql3Zz4ndDdB9E4xDOTPDPX01kCsl nv8Nz7XUZj5jRIIiTqE4wUFQy8Wkxt5os9pdhW+C6xxXIhZiEGsPgRgKa/p8s9QP4vyyJg fdvsFZtNjO8sWqxga7h8Zk8go92l8Ts= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-351-8wwUoSz6PIiD_cv2n3NBXw-1; Mon, 26 Aug 2024 16:44:33 -0400 X-MC-Unique: 8wwUoSz6PIiD_cv2n3NBXw-1 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-4518f9f8a9cso62257721cf.2 for ; Mon, 26 Aug 2024 13:44:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705072; x=1725309872; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FEAM9DK74WZYySVYhL7YnC2YOY0xQ42haB8uzH2x2BY=; b=f8q0ZuMVhqSqYSF9ayTNG+wVwt1AKUXtpZuOBpfER/7nb8GQ3gzbWtIqz30f78r1pW gYjDNqfqe9zkdOKpNXnZWL2k++p98oSe8ErpJqhNwxFyDZaPl/8xcIJgZ/Yo78XDLknj qGhiDOi/FxgIojOLzJGs3gIgqqyvz6ylmJj5ZpIyXUzDaILK8Mtpe796+cFQpVHmYxWx k3Op2NSh7KhR1PmyMrnLal5rWXyfuqhca3qj7IGhh+RDvauKpP0H98MFT3vu1xyircF/ ezZSd/QWcD6YFc8a2s4b/t/iJrSg5AGCalgKADcdQgve8uNOm9yN+GyPkZHo+HICczhq AmwA== X-Gm-Message-State: AOJu0YyoozWohujrCG3egIyxp48oPnx821Oy634JiEhJLmL14oXR6F+B nwrRxihqXF/I90emxs2+z6WkbA2CuUnwudbB0/+/Ee7u3p3i9hzXpMtBiVbDg9eFM0UhmsW4O+3 4vkmyr1rvvlqnS77i11s3LO3k5Mu9H/Qkbg7TEGk7bhHE4B7pe1AtXzc5Isn7MN+jdOQoSMFbm7 uBP63SVppW1tggrDRbOsMtipKsjMJgbme4zCATwV9KEb0= X-Received: by 2002:a05:620a:430a:b0:7a4:faab:fc79 with SMTP id af79cd13be357-7a6896d1835mr1392502085a.8.1724705072645; Mon, 26 Aug 2024 13:44:32 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHDiJ56urcvTKu8a4/ieO25ZyreTkMfmYYALFQqTS8OY3wwzzWFQolk9khXQC+xv0hRJc4KwA== X-Received: by 2002:a05:620a:430a:b0:7a4:faab:fc79 with SMTP id af79cd13be357-7a6896d1835mr1392497485a.8.1724705072141; Mon, 26 Aug 2024 13:44:32 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:31 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 18/19] mm/arm64: Support large pfn mappings Date: Mon, 26 Aug 2024 16:43:52 -0400 Message-ID: <20240826204353.2228736-19-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Support huge pfnmaps by using bit 56 (PTE_SPECIAL) for "special" on pmds/puds. Provide the pmd/pud helpers to set/get special bit. There's one more thing missing for arm64 which is the pxx_pgprot() for pmd/pud. Add them too, which is mostly the same as the pte version by dropping the pfn field. These helpers are essential to be used in the new follow_pfnmap*() API to report valid pgprot_t results. Note that arm64 doesn't yet support huge PUD yet, but it's still straightforward to provide the pud helpers that we need altogether. Only PMD helpers will make an immediate benefit until arm64 will support huge PUDs first in general (e.g. in THPs). Cc: linux-arm-kernel@lists.infradead.org Cc: Catalin Marinas Cc: Will Deacon Signed-off-by: Peter Xu --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/pgtable.h | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 6494848019a0..6607ed8fdbb4 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -99,6 +99,7 @@ config ARM64 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK select ARCH_SUPPORTS_PER_VMA_LOCK + select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index b78cc4a6758b..2faecc033a19 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -578,6 +578,14 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) return pte_pmd(set_pte_bit(pmd_pte(pmd), __pgprot(PTE_DEVMAP))); } =20 +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +#define pmd_special(pte) (!!((pmd_val(pte) & PTE_SPECIAL))) +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return set_pmd_bit(pmd, __pgprot(PTE_SPECIAL)); +} +#endif + #define __pmd_to_phys(pmd) __pte_to_phys(pmd_pte(pmd)) #define __phys_to_pmd_val(phys) __phys_to_pte_val(phys) #define pmd_pfn(pmd) ((__pmd_to_phys(pmd) & PMD_MASK) >> PAGE_SHIFT) @@ -595,6 +603,27 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) #define pud_pfn(pud) ((__pud_to_phys(pud) & PUD_MASK) >> PAGE_SHIFT) #define pfn_pud(pfn,prot) __pud(__phys_to_pud_val((phys_addr_t)(pfn) << PA= GE_SHIFT) | pgprot_val(prot)) =20 +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +#define pud_special(pte) pte_special(pud_pte(pud)) +#define pud_mkspecial(pte) pte_pud(pte_mkspecial(pud_pte(pud))) +#endif + +#define pmd_pgprot pmd_pgprot +static inline pgprot_t pmd_pgprot(pmd_t pmd) +{ + unsigned long pfn =3D pmd_pfn(pmd); + + return __pgprot(pmd_val(pfn_pmd(pfn, __pgprot(0))) ^ pmd_val(pmd)); +} + +#define pud_pgprot pud_pgprot +static inline pgprot_t pud_pgprot(pud_t pud) +{ + unsigned long pfn =3D pud_pfn(pud); + + return __pgprot(pud_val(pfn_pud(pfn, __pgprot(0))) ^ pud_val(pud)); +} + static inline void __set_pte_at(struct mm_struct *mm, unsigned long __always_unused addr, pte_t *ptep, pte_t pte, unsigned int nr) --=20 2.45.0 From nobody Fri Dec 19 13:45:04 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F04101A08CA for ; Mon, 26 Aug 2024 20:44:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705079; cv=none; b=urcx2g31GdDFpWrfYlxAAHVkSdoc5rm/XKGagUFA+SZk/jK6t+TFoEdalF6Awi4o18pE3q8a4JHxDZivZpZrBY8hKbo+iS5D/pQZve7+8Dz7qWbiTdK3Aj9DwicTBrm6uaxm/7fNYQ8W2fYVQgEUdXToYQUAT/1CGeFk3DqqDs8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705079; c=relaxed/simple; bh=Dmoew7i8XEW+gHBA0zJyGyEzte9fLwrtktcen/8enVY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=siQWQxqAAidqgmIoQVZj4PlgMjqPqD5ILXB+SRDBw/8DebIzBPGNuftx6MbjnaB/vlvjtSrQdN1c4v9tUFnq7m8srwuz7W1idHRP1mKKnj7hs5G4UFg3pGpW+iCRVGknG8RYpjWxyHFDAkxyfwk64Z5LgJv1GeGGjD5TTL62X7o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=B/1cNYoV; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="B/1cNYoV" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705076; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9MPOki3YF/wyEgBnejzItxwjIF1moGifGifGb5OwzY4=; b=B/1cNYoVCU8JtQNCasWuX76tNt5qkXae3pQL3TMSvyd0LY/5KFzR34xaRtc6hh+TblYmUS LLnqdMlp1MqiCI/Ryioga/U4we8Jy1AkYjCLPuQVsuNlcIBf3z5j+uRbU6lZRMBij4u5pL p7umzSllBIqs5z41iVYSJZOt5lO1MTE= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-218--tNwjN3SMaGFcjz2lZb6fg-1; Mon, 26 Aug 2024 16:44:35 -0400 X-MC-Unique: -tNwjN3SMaGFcjz2lZb6fg-1 Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-7a1df6b3a26so630300485a.1 for ; Mon, 26 Aug 2024 13:44:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705075; x=1725309875; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9MPOki3YF/wyEgBnejzItxwjIF1moGifGifGb5OwzY4=; b=q9SWndwTKVqe0Nw9EndoQMZl6EPACIwr3cBl4FYGdDgKwMomlDp0v5jH5PUd4TOwIP EU4Ne4O3qbuDrxhCzO8Lo457pub2ErwzFyEx2wGnLlPuVOJ2LcIONOG2EwYULrx+Mj22 HO1m+/v3SHD+U5LCnLx6Vq2lkAAjfxBKtDHF8c+d/rG22sXgpjbADzV1L5QgoQNT0LYL JpqWznXKJUn6+Kps9Poh9fo75+ZrP4hLMbgrmS3NeVq1pBd/1AmqCf8IHeiQc5oUYGen dbKDmthiwxVYEtQbGtPCx9zWPoYXRCNUHZkjCrICCVL4gCfMQqd0O9njgWbfL2kkhLxc pq6w== X-Gm-Message-State: AOJu0YyU6HTMawZNCLBdLdtmRU/5JSF1P7PFcyTsRbRZe9yFAoTA8ZbV C80iTNFDXvj1tdfKRuKiwfUG6o+RjhTxY3W2vQRtZNteRfDypQVodphdyPpke7sUqAmWDAwcJhs tbrlIp2IMCD24N2xqXk9BQkY5ArhTtppg1GPyLrmWWJb2A0Bvr+nf21SwlIq75Gv5hqnzzfIHSW xXS5S+42ckV7YBoL8NhKLzonO0E4T6hLJSblqm1Yd78/M= X-Received: by 2002:a05:620a:4007:b0:79f:595:f64a with SMTP id af79cd13be357-7a6897b7b14mr1477184685a.58.1724705074750; Mon, 26 Aug 2024 13:44:34 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHpx2qE8XXNwFA/VbcshgEEhM7utf/nDzzwdfyOyUpm1p8Kz8GxiojqL+PSarI5vXlPZn99hQ== X-Received: by 2002:a05:620a:4007:b0:79f:595:f64a with SMTP id af79cd13be357-7a6897b7b14mr1477180385a.58.1724705074328; Mon, 26 Aug 2024 13:44:34 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:33 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 19/19] vfio/pci: Implement huge_fault support Date: Mon, 26 Aug 2024 16:43:53 -0400 Message-ID: <20240826204353.2228736-20-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Alex Williamson With the addition of pfnmap support in vmf_insert_pfn_{pmd,pud}() we can take advantage of PMD and PUD faults to PCI BAR mmaps and create more efficient mappings. PCI BARs are always a power of two and will typically get at least PMD alignment without userspace even trying. Userspace alignment for PUD mappings is also not too difficult. Consolidate faults through a single handler with a new wrapper for standard single page faults. The pre-faulting behavior of commit d71a989cf5d9 ("vfio/pci: Insert full vma on mmap'd MMIO fault") is removed in this refactoring since huge_fault will cover the bulk of the faults and results in more efficient page table usage. We also want to avoid that pre-faulted single page mappings preempt huge page mappings. Cc: kvm@vger.kernel.org Signed-off-by: Alex Williamson Signed-off-by: Peter Xu --- drivers/vfio/pci/vfio_pci_core.c | 60 +++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 17 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_c= ore.c index ba0ce0075b2f..2d7478e9a62d 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -1657,14 +1658,20 @@ static unsigned long vma_to_pfn(struct vm_area_stru= ct *vma) return (pci_resource_start(vdev->pdev, index) >> PAGE_SHIFT) + pgoff; } =20 -static vm_fault_t vfio_pci_mmap_fault(struct vm_fault *vmf) +static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf, + unsigned int order) { struct vm_area_struct *vma =3D vmf->vma; struct vfio_pci_core_device *vdev =3D vma->vm_private_data; unsigned long pfn, pgoff =3D vmf->pgoff - vma->vm_pgoff; - unsigned long addr =3D vma->vm_start; vm_fault_t ret =3D VM_FAULT_SIGBUS; =20 + if (order && (vmf->address & ((PAGE_SIZE << order) - 1) || + vmf->address + (PAGE_SIZE << order) > vma->vm_end)) { + ret =3D VM_FAULT_FALLBACK; + goto out; + } + pfn =3D vma_to_pfn(vma); =20 down_read(&vdev->memory_lock); @@ -1672,30 +1679,49 @@ static vm_fault_t vfio_pci_mmap_fault(struct vm_fau= lt *vmf) if (vdev->pm_runtime_engaged || !__vfio_pci_memory_enabled(vdev)) goto out_unlock; =20 - ret =3D vmf_insert_pfn(vma, vmf->address, pfn + pgoff); - if (ret & VM_FAULT_ERROR) - goto out_unlock; - - /* - * Pre-fault the remainder of the vma, abort further insertions and - * supress error if fault is encountered during pre-fault. - */ - for (; addr < vma->vm_end; addr +=3D PAGE_SIZE, pfn++) { - if (addr =3D=3D vmf->address) - continue; - - if (vmf_insert_pfn(vma, addr, pfn) & VM_FAULT_ERROR) - break; + switch (order) { + case 0: + ret =3D vmf_insert_pfn(vma, vmf->address, pfn + pgoff); + break; +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP + case PMD_ORDER: + ret =3D vmf_insert_pfn_pmd(vmf, __pfn_to_pfn_t(pfn + pgoff, + PFN_DEV), false); + break; +#endif +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP + case PUD_ORDER: + ret =3D vmf_insert_pfn_pud(vmf, __pfn_to_pfn_t(pfn + pgoff, + PFN_DEV), false); + break; +#endif + default: + ret =3D VM_FAULT_FALLBACK; } =20 out_unlock: up_read(&vdev->memory_lock); +out: + dev_dbg_ratelimited(&vdev->pdev->dev, + "%s(,order =3D %d) BAR %ld page offset 0x%lx: 0x%x\n", + __func__, order, + vma->vm_pgoff >> + (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT), + pgoff, (unsigned int)ret); =20 return ret; } =20 +static vm_fault_t vfio_pci_mmap_page_fault(struct vm_fault *vmf) +{ + return vfio_pci_mmap_huge_fault(vmf, 0); +} + static const struct vm_operations_struct vfio_pci_mmap_ops =3D { - .fault =3D vfio_pci_mmap_fault, + .fault =3D vfio_pci_mmap_page_fault, +#ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP + .huge_fault =3D vfio_pci_mmap_huge_fault, +#endif }; =20 int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struc= t *vma) --=20 2.45.0