From nobody Sun Sep 14 18:06:41 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF968C46467 for ; Thu, 19 Jan 2023 21:34:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230468AbjASVeB (ORCPT ); Thu, 19 Jan 2023 16:34:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229685AbjASVb1 (ORCPT ); Thu, 19 Jan 2023 16:31:27 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 616ACA8393; Thu, 19 Jan 2023 13:25:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163552; x=1705699552; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=0xRZR4lTYh6NOOM79Ggd51EdLXNXPAUjGASZ+kAeGKM=; b=duB0BEmlqw+ygPNkvMxSD1o+dnTZME0CnvbBTESXHyjxDrbbKpuGv4am lBKW4UPNwndUCTNGsrlGsEQNmKmAOwOJIWBP8RrPD+brqhcs1zc+tDurJ BuQkU67vPKPOeV1jIHSkWjJy+cLdLcEmc55g2vaVuk0hqnZpdxVwMAl6k 3xMdThUepL4voIWiXJjeSOb/4v/hSnV3lmRuk53yS4i5kK+bCd9uMIJ3f gZofpixT+VuWu3ryIhCgEJ0lDk6lS+E5zT5XTqPNX7V73vP6t9ANk9Tp/ Ctk17W3o+3KqGbzmxo0FF6aQS4NWDTbTCz8QoYIXSGz/chn+yoWYCfNLa A==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119409" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119409" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:42 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139038" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139038" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:41 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 11/39] x86/mm: Update pte_modify for _PAGE_COW Date: Thu, 19 Jan 2023 13:22:49 -0800 Message-Id: <20230119212317.8324-12-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Yu-cheng Yu The Write=3D0,Dirty=3D1 PTE has been used to indicate copy-on-write pages. However, newer x86 processors also regard a Write=3D0,Dirty=3D1 PTE as a shadow stack page. In order to separate the two, the software-defined _PAGE_DIRTY is changed to _PAGE_COW for the copy-on-write case, and pte_*() are updated to do this. pte_modify() takes a "raw" pgprot_t which was not necessarily created with any of the existing PTE bit helpers. That means that it can return a pte_t with Write=3D0,Dirty=3D1, a shadow stack PTE, when it did not intend = to create one. However pte_modify() changes a PTE to 'newprot', but it doesn't use the pte_*(). Modify it to also move _PAGE_DIRTY to _PAGE_COW. Do this by using the pte_mkdirty() helper. Since pte_mkdirty() also sets the soft dirty bit, extract a helper that optionally doesn't set _PAGE_SOFT_DIRTY. This helper will allow future logic for deciding when to move _PAGE_DIRTY to _PAGE_COW can live in one place. Apply the same changes to pmd_modify(). Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook --- v5: - Fix pte_modify() again, to not lose _PAGE_DIRTY, but still not set _PAGE_SOFT_DIRTY as was fixed in v4. v4: - Fix an issue in soft-dirty test, where pte_modify() would detect _PAGE_COW in pte_dirty() and set the soft dirty bit in pte_mkdirty(). v2: - Update commit log with text and suggestions from (Dave Hansen) - Drop fixup_dirty_pte() in favor of clearing the HW dirty bit along with the _PAGE_CHG_MASK masking, then calling pte_mkdirty() (Dave Hansen) arch/x86/include/asm/pgtable.h | 64 +++++++++++++++++++++++++++++----- 1 file changed, 56 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 6d2f612c04b5..7942eff2af50 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -392,9 +392,19 @@ static inline pte_t pte_mkexec(pte_t pte) return pte_clear_flags(pte, _PAGE_NX); } =20 +static inline pte_t __pte_mkdirty(pte_t pte, bool soft) +{ + pteval_t dirty =3D _PAGE_DIRTY; + + if (soft) + dirty |=3D _PAGE_SOFT_DIRTY; + + return pte_set_flags(pte, dirty); +} + static inline pte_t pte_mkdirty(pte_t pte) { - return pte_set_flags(pte, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return __pte_mkdirty(pte, true); } =20 static inline pte_t pte_mkyoung(pte_t pte) @@ -503,9 +513,19 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd) return pmd_clear_flags(pmd, _PAGE_RW); } =20 +static inline pmd_t __pmd_mkdirty(pmd_t pmd, bool soft) +{ + pmdval_t dirty =3D _PAGE_DIRTY; + + if (soft) + dirty |=3D _PAGE_SOFT_DIRTY; + + return pmd_set_flags(pmd, dirty); +} + static inline pmd_t pmd_mkdirty(pmd_t pmd) { - return pmd_set_flags(pmd, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return __pmd_mkdirty(pmd, true); } =20 static inline pmd_t pmd_mkdevmap(pmd_t pmd) @@ -715,26 +735,54 @@ static inline u64 flip_protnone_guard(u64 oldval, u64= val, u64 mask); =20 static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) { + pteval_t _page_chg_mask_no_dirty =3D _PAGE_CHG_MASK & ~_PAGE_DIRTY; pteval_t val =3D pte_val(pte), oldval =3D val; + pte_t pte_result; =20 /* * Chop off the NX bit (if present), and add the NX portion of * the newprot (if present): */ - val &=3D _PAGE_CHG_MASK; - val |=3D check_pgprot(newprot) & ~_PAGE_CHG_MASK; + val &=3D _page_chg_mask_no_dirty; + val |=3D check_pgprot(newprot) & ~_page_chg_mask_no_dirty; val =3D flip_protnone_guard(oldval, val, PTE_PFN_MASK); - return __pte(val); + + pte_result =3D __pte(val); + + /* + * Dirty bit is not preserved above so it can be done + * in a special way for the shadow stack case, where it + * may need to set _PAGE_COW. __pte_mkdirty() will do this in + * the case of shadow stack. + */ + if (pte_dirty(pte)) + pte_result =3D __pte_mkdirty(pte_result, false); + + return pte_result; } =20 static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) { + pteval_t _hpage_chg_mask_no_dirty =3D _HPAGE_CHG_MASK & ~_PAGE_DIRTY; pmdval_t val =3D pmd_val(pmd), oldval =3D val; + pmd_t pmd_result; =20 - val &=3D _HPAGE_CHG_MASK; - val |=3D check_pgprot(newprot) & ~_HPAGE_CHG_MASK; + val &=3D _hpage_chg_mask_no_dirty; + val |=3D check_pgprot(newprot) & ~_hpage_chg_mask_no_dirty; val =3D flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK); - return __pmd(val); + + pmd_result =3D __pmd(val); + + /* + * Dirty bit is not preserved above so it can be done + * in a special way for the shadow stack case, where it + * may need to set _PAGE_COW. __pmd_mkdirty() will do this in + * the case of shadow stack. + */ + if (pmd_dirty(pmd)) + pmd_result =3D __pmd_mkdirty(pmd_result, false); + + return pmd_result; } =20 /* --=20 2.17.1