From nobody Mon Apr 6 09:20:37 2026 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 76E2F1D6193 for ; Thu, 19 Mar 2026 18:31:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773945086; cv=none; b=X0i1fULnd9u/KS5O79+fZDpgeezhOsNNI+Ph8cUHU7X9t5WOIBNY9q8gU7lZJG1Vi6hFQflm9yJYQKKwPHUR1XiRXFFmgGKpkBfGFwethz54a5AuN7fHQQoEFfLINw7BFPWdP5ohz7IzIge1QFtNQHhKqnVd6X0ohSXELNulizU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773945086; c=relaxed/simple; bh=Oci7+eOyv2DvIYHwHcWPk6ScUUGwXF56RP6YDE1pfQ0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hfAcVtutWCH4j9b1c/15FP/M1UUDuE1ohd/72pJMaWN655/tDCJnOC+aDR0LMIG284w7AhWZSQlNBYQbO4LXT6bfrXUASwOrve0lyVS+I05S8gxZHV+x59ltYlwL/durRRGSw1wUre58zE52nfigI6DwlUUCEih4TAnCOpgPl0Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=DVxcO0GG; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=0v07ZlIi; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=DVxcO0GG; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=0v07ZlIi; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="DVxcO0GG"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="0v07ZlIi"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="DVxcO0GG"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="0v07ZlIi" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 95B3D5BD97; Thu, 19 Mar 2026 18:31:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1773945080; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=anyfiV6rgUNLqvdloJydtyxRF1ee9QavRldGPaFXcPE=; b=DVxcO0GGwaWmy/iEVsxUuH0tYC8PxAyd7+ebKCMjH1hkrxKNaZZ1c/OpwzZCNtJfWVRum3 PNgjxzs+J4qYe7G+rpGzOyIC3BjujeBW9wOkwS4PvcObGlYmFp3OlRBpUaF9XbWix90SEa kIqV+/xduLezQ3Lql0Pa4O69pfwBB8I= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1773945080; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=anyfiV6rgUNLqvdloJydtyxRF1ee9QavRldGPaFXcPE=; b=0v07ZlIi0C8yO71HGvvpk3vQzMbgZvMWWi8S1VhNwwmjASPjgMtxJGzQafZPd4l2akquoB bi3ovs60pdo4tbDA== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1773945080; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=anyfiV6rgUNLqvdloJydtyxRF1ee9QavRldGPaFXcPE=; b=DVxcO0GGwaWmy/iEVsxUuH0tYC8PxAyd7+ebKCMjH1hkrxKNaZZ1c/OpwzZCNtJfWVRum3 PNgjxzs+J4qYe7G+rpGzOyIC3BjujeBW9wOkwS4PvcObGlYmFp3OlRBpUaF9XbWix90SEa kIqV+/xduLezQ3Lql0Pa4O69pfwBB8I= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1773945080; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=anyfiV6rgUNLqvdloJydtyxRF1ee9QavRldGPaFXcPE=; b=0v07ZlIi0C8yO71HGvvpk3vQzMbgZvMWWi8S1VhNwwmjASPjgMtxJGzQafZPd4l2akquoB bi3ovs60pdo4tbDA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id BAB8F4273B; Thu, 19 Mar 2026 18:31:19 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 4PcbKvdAvGmbdgAAD6G6ig (envelope-from ); Thu, 19 Mar 2026 18:31:19 +0000 From: Pedro Falcato To: Andrew Morton , "Liam R. Howlett" , Lorenzo Stoakes Cc: Pedro Falcato , Vlastimil Babka , Jann Horn , David Hildenbrand , Dev Jain , Luke Yang , jhladky@redhat.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/4] mm/mprotect: encourage inlining with __always_inline Date: Thu, 19 Mar 2026 18:31:05 +0000 Message-ID: <20260319183108.1105090-2-pfalcato@suse.de> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260319183108.1105090-1-pfalcato@suse.de> References: <20260319183108.1105090-1-pfalcato@suse.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -6.80 X-Spam-Level: X-Spamd-Result: default: False [-6.80 / 50.00]; REPLY(-4.00)[]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.997]; MIME_GOOD(-0.10)[text/plain]; FUZZY_RATELIMITED(0.00)[rspamd.com]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWELVE(0.00)[12]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_TLS_ALL(0.00)[]; TO_DN_SOME(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo,suse.de:mid,suse.de:email] X-Spam-Flag: NO Content-Type: text/plain; charset="utf-8" Encourage the compiler to inline batch PTE logic and resolve constant branches by adding __always_inline strategically. Signed-off-by: Pedro Falcato --- mm/mprotect.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/mprotect.c b/mm/mprotect.c index 9681f055b9fc..1bd0d4aa07c2 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -103,7 +103,7 @@ bool can_change_pte_writable(struct vm_area_struct *vma= , unsigned long addr, return can_change_shared_pte_writable(vma, pte); } =20 -static int mprotect_folio_pte_batch(struct folio *folio, pte_t *ptep, +static __always_inline int mprotect_folio_pte_batch(struct folio *folio, p= te_t *ptep, pte_t pte, int max_nr_ptes, fpb_t flags) { /* No underlying folio, so cannot batch */ @@ -117,9 +117,9 @@ static int mprotect_folio_pte_batch(struct folio *folio= , pte_t *ptep, } =20 /* Set nr_ptes number of ptes, starting from idx */ -static void prot_commit_flush_ptes(struct vm_area_struct *vma, unsigned lo= ng addr, - pte_t *ptep, pte_t oldpte, pte_t ptent, int nr_ptes, - int idx, bool set_write, struct mmu_gather *tlb) +static __always_inline void prot_commit_flush_ptes(struct vm_area_struct *= vma, + unsigned long addr, pte_t *ptep, pte_t oldpte, pte_t ptent, + int nr_ptes, int idx, bool set_write, struct mmu_gather *tlb) { /* * Advance the position in the batch by idx; note that if idx > 0, @@ -169,7 +169,7 @@ static int page_anon_exclusive_sub_batch(int start_idx,= int max_len, * pte of the batch. Therefore, we must individually check all pages and * retrieve sub-batches. */ -static void commit_anon_folio_batch(struct vm_area_struct *vma, +static __always_inline void commit_anon_folio_batch(struct vm_area_struct = *vma, struct folio *folio, struct page *first_page, unsigned long addr, pte_t = *ptep, pte_t oldpte, pte_t ptent, int nr_ptes, struct mmu_gather *tlb) { --=20 2.53.0 From nobody Mon Apr 6 09:20:38 2026 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 199A8254B18 for ; Thu, 19 Mar 2026 18:31:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773945085; cv=none; b=NAKOWaTwLb3kVNPVZ3dZRvs7ArdtH22dtOeUrpLdrcp5RlCEw0gdTIilo70xCsiTCNk0AR5AXT3yPhv6qjihwnFDoUMJFgQy5FqLWoygb1g5a28dK2vSu2wPh5TXAPa3RquzpegJ4TTzThiPdSzlCALC/2z60JT1jqLzmvgVBK8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773945085; c=relaxed/simple; bh=mGGbC/AaH2Nw4nAbmk3HBXTk7XDsPqvns4V+yqIbHBo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=T5TE8nLpDY+i3HXcjXd0ibAyJPew3lmqHc6i0iPtUzu/Hn3Xd9kN74QDaNv5DQBDKJKUK2JaEXS3BmxHoJC9zsEyW5t/Ljnk/swjBeA2qa64hVq61ahuyzOAEcE4wbjaE/uuffwOtIF2KyTujYH2bVhGdJldsIhPnEV25pmgAzw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=1yLuBo1u; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=FNsKTwXF; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=1yLuBo1u; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=FNsKTwXF; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="1yLuBo1u"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="FNsKTwXF"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="1yLuBo1u"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="FNsKTwXF" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 5D4954D214; Thu, 19 Mar 2026 18:31:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1773945082; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=I4EDCTugoDpsygL/2msKaEtTb3dy6BTNe9WE3o1d0jk=; b=1yLuBo1uPsF7z6th72+dt19GsYUozcSNQDwQVvNoEUIO8xlkCQjkp4RfxmrQrL6tkawFkZ AbNhIEKyfDz+iyh/bBt1sqYI6gIeH05abz4Fj/osp0e56aU6fU7rX7nOZA7IaaUy75NGLv WVuGhHCKjYx1IRqIB/h+KfVVMkRPQuo= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1773945082; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=I4EDCTugoDpsygL/2msKaEtTb3dy6BTNe9WE3o1d0jk=; b=FNsKTwXFm1HxfV7fsG9TwBtg5wzBro/64pwQoXbQJMxkCYqM2xBbjAlw8C1Sl35kwD1QpK rhazKeGrVL87ZmAQ== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1773945082; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=I4EDCTugoDpsygL/2msKaEtTb3dy6BTNe9WE3o1d0jk=; b=1yLuBo1uPsF7z6th72+dt19GsYUozcSNQDwQVvNoEUIO8xlkCQjkp4RfxmrQrL6tkawFkZ AbNhIEKyfDz+iyh/bBt1sqYI6gIeH05abz4Fj/osp0e56aU6fU7rX7nOZA7IaaUy75NGLv WVuGhHCKjYx1IRqIB/h+KfVVMkRPQuo= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1773945082; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=I4EDCTugoDpsygL/2msKaEtTb3dy6BTNe9WE3o1d0jk=; b=FNsKTwXFm1HxfV7fsG9TwBtg5wzBro/64pwQoXbQJMxkCYqM2xBbjAlw8C1Sl35kwD1QpK rhazKeGrVL87ZmAQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 7B5294273B; Thu, 19 Mar 2026 18:31:21 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 6EKSGvlAvGmbdgAAD6G6ig (envelope-from ); Thu, 19 Mar 2026 18:31:21 +0000 From: Pedro Falcato To: Andrew Morton , "Liam R. Howlett" , Lorenzo Stoakes Cc: Pedro Falcato , Vlastimil Babka , Jann Horn , David Hildenbrand , Dev Jain , Luke Yang , jhladky@redhat.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/4] mm/mprotect: move softleaf code out of the main function Date: Thu, 19 Mar 2026 18:31:06 +0000 Message-ID: <20260319183108.1105090-3-pfalcato@suse.de> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260319183108.1105090-1-pfalcato@suse.de> References: <20260319183108.1105090-1-pfalcato@suse.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-6.80 / 50.00]; REPLY(-4.00)[]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.997]; MIME_GOOD(-0.10)[text/plain]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo,suse.de:mid,suse.de:email]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; TO_DN_SOME(0.00)[]; RCPT_COUNT_TWELVE(0.00)[12]; RCVD_VIA_SMTP_AUTH(0.00)[]; FUZZY_RATELIMITED(0.00)[rspamd.com]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; R_RATELIMIT(0.00)[to_ip_from(RL8d8xedm6iu8o66torxsk6bwd)]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; RCVD_TLS_ALL(0.00)[] X-Spam-Flag: NO X-Spam-Score: -6.80 X-Spam-Level: Content-Type: text/plain; charset="utf-8" Move softleaf change_pte_range code into a separate function. This makes the change_pte_range() function (or where it inlines) a good bit smaller. Plus it lessens cognitive load when reading through the function. Signed-off-by: Pedro Falcato Reviewed-by: Lorenzo Stoakes (Oracle) --- mm/mprotect.c | 128 +++++++++++++++++++++++++++----------------------- 1 file changed, 68 insertions(+), 60 deletions(-) diff --git a/mm/mprotect.c b/mm/mprotect.c index 1bd0d4aa07c2..8d4fa38a8a26 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -211,6 +211,73 @@ static void set_write_prot_commit_flush_ptes(struct vm= _area_struct *vma, commit_anon_folio_batch(vma, folio, page, addr, ptep, oldpte, ptent, nr_p= tes, tlb); } =20 +static noinline long change_pte_softleaf(struct vm_area_struct *vma, + unsigned long addr, pte_t *pte, pte_t oldpte, unsigned long cp_flags) +{ + bool uffd_wp =3D cp_flags & MM_CP_UFFD_WP; + bool uffd_wp_resolve =3D cp_flags & MM_CP_UFFD_WP_RESOLVE; + softleaf_t entry =3D softleaf_from_pte(oldpte); + pte_t newpte; + + if (softleaf_is_migration_write(entry)) { + const struct folio *folio =3D softleaf_to_folio(entry); + + /* + * A protection check is difficult so + * just be safe and disable write + */ + if (folio_test_anon(folio)) + entry =3D make_readable_exclusive_migration_entry( + swp_offset(entry)); + else + entry =3D make_readable_migration_entry(swp_offset(entry)); + newpte =3D swp_entry_to_pte(entry); + if (pte_swp_soft_dirty(oldpte)) + newpte =3D pte_swp_mksoft_dirty(newpte); + } else if (softleaf_is_device_private_write(entry)) { + /* + * We do not preserve soft-dirtiness. See + * copy_nonpresent_pte() for explanation. + */ + entry =3D make_readable_device_private_entry( + swp_offset(entry)); + newpte =3D swp_entry_to_pte(entry); + if (pte_swp_uffd_wp(oldpte)) + newpte =3D pte_swp_mkuffd_wp(newpte); + } else if (softleaf_is_marker(entry)) { + /* + * Ignore error swap entries unconditionally, + * because any access should sigbus/sigsegv + * anyway. + */ + if (softleaf_is_poison_marker(entry) || + softleaf_is_guard_marker(entry)) + return 0; + /* + * If this is uffd-wp pte marker and we'd like + * to unprotect it, drop it; the next page + * fault will trigger without uffd trapping. + */ + if (uffd_wp_resolve) { + pte_clear(vma->vm_mm, addr, pte); + return 1; + } + } else { + newpte =3D oldpte; + } + + if (uffd_wp) + newpte =3D pte_swp_mkuffd_wp(newpte); + else if (uffd_wp_resolve) + newpte =3D pte_swp_clear_uffd_wp(newpte); + + if (!pte_same(oldpte, newpte)) { + set_pte_at(vma->vm_mm, addr, pte, newpte); + return 1; + } + return 0; +} + static long change_pte_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, pgprot_t newprot, unsigned long cp_flags) @@ -317,66 +384,7 @@ static long change_pte_range(struct mmu_gather *tlb, pages++; } } else { - softleaf_t entry =3D softleaf_from_pte(oldpte); - pte_t newpte; - - if (softleaf_is_migration_write(entry)) { - const struct folio *folio =3D softleaf_to_folio(entry); - - /* - * A protection check is difficult so - * just be safe and disable write - */ - if (folio_test_anon(folio)) - entry =3D make_readable_exclusive_migration_entry( - swp_offset(entry)); - else - entry =3D make_readable_migration_entry(swp_offset(entry)); - newpte =3D swp_entry_to_pte(entry); - if (pte_swp_soft_dirty(oldpte)) - newpte =3D pte_swp_mksoft_dirty(newpte); - } else if (softleaf_is_device_private_write(entry)) { - /* - * We do not preserve soft-dirtiness. See - * copy_nonpresent_pte() for explanation. - */ - entry =3D make_readable_device_private_entry( - swp_offset(entry)); - newpte =3D swp_entry_to_pte(entry); - if (pte_swp_uffd_wp(oldpte)) - newpte =3D pte_swp_mkuffd_wp(newpte); - } else if (softleaf_is_marker(entry)) { - /* - * Ignore error swap entries unconditionally, - * because any access should sigbus/sigsegv - * anyway. - */ - if (softleaf_is_poison_marker(entry) || - softleaf_is_guard_marker(entry)) - continue; - /* - * If this is uffd-wp pte marker and we'd like - * to unprotect it, drop it; the next page - * fault will trigger without uffd trapping. - */ - if (uffd_wp_resolve) { - pte_clear(vma->vm_mm, addr, pte); - pages++; - } - continue; - } else { - newpte =3D oldpte; - } - - if (uffd_wp) - newpte =3D pte_swp_mkuffd_wp(newpte); - else if (uffd_wp_resolve) - newpte =3D pte_swp_clear_uffd_wp(newpte); - - if (!pte_same(oldpte, newpte)) { - set_pte_at(vma->vm_mm, addr, pte, newpte); - pages++; - } + pages +=3D change_pte_softleaf(vma, addr, pte, oldpte, cp_flags); } } while (pte +=3D nr_ptes, addr +=3D nr_ptes * PAGE_SIZE, addr !=3D end); lazy_mmu_mode_disable(); --=20 2.53.0 From nobody Mon Apr 6 09:20:38 2026 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 613E22DCF45 for ; Thu, 19 Mar 2026 18:31:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773945092; cv=none; b=GLAjbJcAJ3k2efCcsLjrhIAw4wGrtdNh+7czp8+YDtbuMl9pUa03/bpmUDJQX8XcXx2G3TPVmPrSu9PSCzNfmeEHvc4d7HeT2jzkJPuK74NgBvpLHnT4muLHfgJ7vGb1nMpiC6eTrd2D7Awo5n94e6t9fwjBa/9loymQsGjgiYk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773945092; c=relaxed/simple; bh=nOEE3JK1QoYh5GpiYGfLUtii2Wjnk1v54Kbm6zzMFVU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lMxx1o9WJ8FQShL7xO//D2jK2dNg3wG+2i6LQdEPVSvhXnIvMpYc5gdMhBFGJ+LFRO5zrcrZ1TTBmtNQb9qRjUp7Q5o4cL/XxVBFlm43wpYwXTqkXhM69Q/F5QRpHaOBw8tb5lYzz4QT9Y8DliJRcxWvptxp90oA5CS6+VVw/WA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id A0B535BD82; Thu, 19 Mar 2026 18:31:24 +0000 (UTC) Authentication-Results: smtp-out2.suse.de; none Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id C63284273B; Thu, 19 Mar 2026 18:31:23 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 2PTnLPtAvGmbdgAAD6G6ig (envelope-from ); Thu, 19 Mar 2026 18:31:23 +0000 From: Pedro Falcato To: Andrew Morton , "Liam R. Howlett" , Lorenzo Stoakes Cc: Pedro Falcato , Vlastimil Babka , Jann Horn , David Hildenbrand , Dev Jain , Luke Yang , jhladky@redhat.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/4] mm/mprotect: un-inline folio_pte_batch_flags() Date: Thu, 19 Mar 2026 18:31:07 +0000 Message-ID: <20260319183108.1105090-4-pfalcato@suse.de> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260319183108.1105090-1-pfalcato@suse.de> References: <20260319183108.1105090-1-pfalcato@suse.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Server: rspamd2.dmz-prg2.suse.org X-Spamd-Result: default: False [-4.00 / 50.00]; REPLY(-4.00)[] X-Rspamd-Queue-Id: A0B535BD82 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Spam-Flag: NO X-Spam-Score: -4.00 X-Spam-Level: Content-Type: text/plain; charset="utf-8" Hoist some previous, important to be inlined checks to the main loop and noinline the entirety of folio_pte_batch_flags(). The loop itself is quite large and whether it is inlined or not should not matter, as we are ideally dealing with larger orders. Signed-off-by: Pedro Falcato --- mm/mprotect.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/mm/mprotect.c b/mm/mprotect.c index 8d4fa38a8a26..aa845f5bf14d 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -103,16 +103,9 @@ bool can_change_pte_writable(struct vm_area_struct *vm= a, unsigned long addr, return can_change_shared_pte_writable(vma, pte); } =20 -static __always_inline int mprotect_folio_pte_batch(struct folio *folio, p= te_t *ptep, +static noinline int mprotect_folio_pte_batch(struct folio *folio, pte_t *p= tep, pte_t pte, int max_nr_ptes, fpb_t flags) { - /* No underlying folio, so cannot batch */ - if (!folio) - return 1; - - if (!folio_test_large(folio)) - return 1; - return folio_pte_batch_flags(folio, NULL, ptep, &pte, max_nr_ptes, flags); } =20 @@ -333,7 +326,12 @@ static long change_pte_range(struct mmu_gather *tlb, continue; } =20 - nr_ptes =3D mprotect_folio_pte_batch(folio, pte, oldpte, max_nr_ptes, f= lags); + /* No underlying folio (or not large), so cannot batch */ + if (likely(!folio || !folio_test_large(folio))) + nr_ptes =3D 1; + else + nr_ptes =3D mprotect_folio_pte_batch(folio, pte, oldpte, + max_nr_ptes, flags); =20 oldpte =3D modify_prot_start_ptes(vma, addr, pte, nr_ptes); ptent =3D pte_modify(oldpte, newprot); --=20 2.53.0 From nobody Mon Apr 6 09:20:38 2026 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9AAF11D6193 for ; Thu, 19 Mar 2026 18:31:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773945091; cv=none; b=r+0UKCiNLn0w1Ca8zjJn7rnll/NfwG25q5zlGJPdkD3xSlZAMrWuALVPaQi244D78+yY/3HNWqzgf+XU9hBrhat0ka515vtUZMBHF1RmH79N4wm7dvOebp6vTJ8m8OGdODM+HfxUSKI+bYuLtmvwwoJ7K6d+uG9WYCH04JB1cHY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773945091; c=relaxed/simple; bh=jyabvAMAbPNb8Vr0FexkWTdFAsfmhwFkB0qBjP6Kb6I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MZqX3Ys+fsQT9MXcOFdVoeygc8MQMydDr7ITTXTypCt4SNITWl8Bdu1tGpXXAUOQEyRgW3CPc0h776c4cfyBOIeCWQ9FiwGo4xnLpemXQNGwlVifWuOHFJY/igWw1Z68YvCLxOhM53ZWgV6tzVmlK3KvoI6QTi7pFxOg57ABh+A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=MkMaqk2K; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=OoND9t8j; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=MkMaqk2K; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=OoND9t8j; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="MkMaqk2K"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="OoND9t8j"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="MkMaqk2K"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="OoND9t8j" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id D3A534D214; Thu, 19 Mar 2026 18:31:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1773945086; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f7+HaQn1tFTlja+mVBj2e/YdRVEWULIlPF/DzJbJkoU=; b=MkMaqk2KTqR8SvmqBvG9RUGNCDkYTKiMQOfjr0/hHXbKyyzmarFzba+oTmNtzfU3vbH7Yb EAvlTi2UF8UqYrwf7cUZmvRmJDvUZJ8JA3DLy9yUfr8X7+OiVkTR/8iR0KamG9BHSlEqV/ jGY5C2crNdpW4i3qKkbYoXAXMSipEbk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1773945086; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f7+HaQn1tFTlja+mVBj2e/YdRVEWULIlPF/DzJbJkoU=; b=OoND9t8jg6pUIN9evDAux+9Q/DfaX6M3v2neguvPFSmQiGQHLbxW/U9fDlH+jPxBNmOnlo LDpJ+YtaRhfDp7AA== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1773945086; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f7+HaQn1tFTlja+mVBj2e/YdRVEWULIlPF/DzJbJkoU=; b=MkMaqk2KTqR8SvmqBvG9RUGNCDkYTKiMQOfjr0/hHXbKyyzmarFzba+oTmNtzfU3vbH7Yb EAvlTi2UF8UqYrwf7cUZmvRmJDvUZJ8JA3DLy9yUfr8X7+OiVkTR/8iR0KamG9BHSlEqV/ jGY5C2crNdpW4i3qKkbYoXAXMSipEbk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1773945086; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f7+HaQn1tFTlja+mVBj2e/YdRVEWULIlPF/DzJbJkoU=; b=OoND9t8jg6pUIN9evDAux+9Q/DfaX6M3v2neguvPFSmQiGQHLbxW/U9fDlH+jPxBNmOnlo LDpJ+YtaRhfDp7AA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 041D54273B; Thu, 19 Mar 2026 18:31:25 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 2PsGOf1AvGmbdgAAD6G6ig (envelope-from ); Thu, 19 Mar 2026 18:31:25 +0000 From: Pedro Falcato To: Andrew Morton , "Liam R. Howlett" , Lorenzo Stoakes Cc: Pedro Falcato , Vlastimil Babka , Jann Horn , David Hildenbrand , Dev Jain , Luke Yang , jhladky@redhat.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 4/4] mm/mprotect: special-case small folios when applying write permissions Date: Thu, 19 Mar 2026 18:31:08 +0000 Message-ID: <20260319183108.1105090-5-pfalcato@suse.de> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260319183108.1105090-1-pfalcato@suse.de> References: <20260319183108.1105090-1-pfalcato@suse.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -6.80 X-Spam-Level: X-Spamd-Result: default: False [-6.80 / 50.00]; REPLY(-4.00)[]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.996]; MIME_GOOD(-0.10)[text/plain]; FUZZY_RATELIMITED(0.00)[rspamd.com]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWELVE(0.00)[12]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; R_RATELIMIT(0.00)[to_ip_from(RL8d8xedm6iu8o66torxsk6bwd)]; TO_DN_SOME(0.00)[]; RCVD_TLS_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:mid,suse.de:email,imap1.dmz-prg2.suse.org:helo] X-Spam-Flag: NO Content-Type: text/plain; charset="utf-8" The common order-0 case is important enough to want its own branch, and avoids the hairy, large loop logic that the CPU does not seem to handle particularly well. Signed-off-by: Pedro Falcato Reviewed-by: Lorenzo Stoakes (Oracle) --- mm/mprotect.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/mm/mprotect.c b/mm/mprotect.c index aa845f5bf14d..23a741f9a4c8 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -170,6 +170,13 @@ static __always_inline void commit_anon_folio_batch(st= ruct vm_area_struct *vma, int sub_batch_idx =3D 0; int len; =20 + /* Optimize for the common order-0 case. */ + if (likely(nr_ptes =3D=3D 1)) { + prot_commit_flush_ptes(vma, addr, ptep, oldpte, ptent, 1, + 0, PageAnonExclusive(first_page), tlb); + return; + } + while (nr_ptes) { expected_anon_exclusive =3D PageAnonExclusive(first_page + sub_batch_idx= ); len =3D page_anon_exclusive_sub_batch(sub_batch_idx, nr_ptes, --=20 2.53.0