From nobody Wed Jan 22 10:03:55 2025 Received: from out-182.mta1.migadu.com (out-182.mta1.migadu.com [95.215.58.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6F7B51F470D for ; Tue, 21 Jan 2025 20:09:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737490186; cv=none; b=At3kgMzCFby5Sa6Gno4vyTpOjGnEnhmPlggzbHMNx2O9cgb8nzy/Ry7nPDUSsEjbF/duhDaBojWwBPIi24mKXMOAGXYKjp9xSnBNe0cZ0ic1wjGRmpAKNfR3sC6rsqAsSaBYcKvRwWOxKvcBJVSdK/F+x20BWrh17vrvkXW0Znc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737490186; c=relaxed/simple; bh=fI17I6HuQGCEAlHpVmI6JQkePZVm+cAT2IugwsmNoM0=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=ALnyEWGVTy5+KiZlz0pCmCiNTx4yiQVs6MZW2VtAzp6EN5fMIrMUO+eSWFAUoMV6bPEkiaZqqHwPx9CeYScZY1rMNR0wnqGOWhOf25iDuZMGlVT2p24yR/bLFCAJSWZMkCupiUlY8i8E4QN42cp8CSPRj8Otrut1/t/0ShBVtHM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=sCq8l6Vl; arc=none smtp.client-ip=95.215.58.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="sCq8l6Vl" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1737490180; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=7MsGtIYaWKOulbwuqadA1hPhFKScuzlkf6Zojvdyofw=; b=sCq8l6VlXiQ+Y9MvDQ05R6I1sl7MwDLsW1PD9fe5Ce5IQITIdW/tjnum91Sc0CC90YVzgx T5wrFIEFiywHonrhA82nF6Uro3G9+eEkylPuLA8yEWZp/oKvhevw1/aQ1xwor4viyzSV/W ofzOHQXA63oNzUfhj1vEBR/PzqOKl0Y= From: Roman Gushchin To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, Andrew Morton , Roman Gushchin , Jann Horn , Peter Zijlstra , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Hugh Dickins , linux-arch@vger.kernel.org Subject: [PATCH] mmu_gather: move tlb flush for VM_PFNMAP/VM_MIXEDMAP vmas into free_pgtables() Date: Tue, 21 Jan 2025 20:09:29 +0000 Message-ID: <20250121200929.188542-1-roman.gushchin@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" Commit b67fbebd4cf9 ("mmu_gather: Force tlb-flush VM_PFNMAP vmas") added a forced tlbflush to tlb_vma_end(), which is required to avoid a race between munmap() and unmap_mapping_range(). However it added some overhead to other paths where tlb_vma_end() is used, but vmas are not removed, e.g. madvise(MADV_DONTNEED). Fix this by moving the tlb flush out of tlb_end_vma() into free_pgtables(), somewhat similar to the stable version of the original commit: e.g. stable commit 895428ee124a ("mm: Force TLB flush for PFNMAP mappings before unlink_file_vma()"). Note, that if tlb->fullmm is set, no flush is required, as the whole mm is about to be destroyed. Suggested-by: Jann Horn Signed-off-by: Roman Gushchin Cc: Peter Zijlstra Cc: Will Deacon Cc: "Aneesh Kumar K.V" Cc: Andrew Morton Cc: Nick Piggin Cc: Hugh Dickins Cc: linux-arch@vger.kernel.org Cc: linux-mm@kvack.org --- include/asm-generic/tlb.h | 16 ++++------------ mm/memory.c | 7 +++++++ 2 files changed, 11 insertions(+), 12 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 709830274b75..411daa96f57a 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -549,22 +549,14 @@ static inline void tlb_start_vma(struct mmu_gather *t= lb, struct vm_area_struct * =20 static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_stru= ct *vma) { - if (tlb->fullmm) + if (tlb->fullmm || IS_ENABLED(CONFIG_MMU_GATHER_MERGE_VMAS)) return; =20 /* - * VM_PFNMAP is more fragile because the core mm will not track the - * page mapcount -- there might not be page-frames for these PFNs after - * all. Force flush TLBs for such ranges to avoid munmap() vs - * unmap_mapping_range() races. + * Do a TLB flush and reset the range at VMA boundaries; this avoids + * the ranges growing with the unused space between consecutive VMAs. */ - if (tlb->vma_pfn || !IS_ENABLED(CONFIG_MMU_GATHER_MERGE_VMAS)) { - /* - * Do a TLB flush and reset the range at VMA boundaries; this avoids - * the ranges growing with the unused space between consecutive VMAs. - */ - tlb_flush_mmu_tlbonly(tlb); - } + tlb_flush_mmu_tlbonly(tlb); } =20 /* diff --git a/mm/memory.c b/mm/memory.c index 398c031be9ba..2071415f68dd 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -365,6 +365,13 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_s= tate *mas, { struct unlink_vma_file_batch vb; =20 + /* + * Ensure we have no stale TLB entries by the time this mapping is + * removed from the rmap. + */ + if (tlb->vma_pfn && !tlb->fullmm) + tlb_flush_mmu(tlb); + do { unsigned long addr =3D vma->vm_start; struct vm_area_struct *next; --=20 2.48.0.rc2.279.g1de40edade-goog