From nobody Fri Dec 19 06:06:30 2025 Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4714022AE7B for ; Mon, 13 Jan 2025 14:24:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736778275; cv=none; b=HLfyF9MWd2wT03yT5rz8/QvdgzEnnyV3ftOS2kQHbYVpwle9kZKEtHpW6jCdk0i352PE6qMpvhS6rbMM0s4POgn1NROpG7UawXTM7i840ZLrW7lBZ/CoM9zB46m+1efqwRpvEjbF5RyyHhDC3rhMdaV5icHr2zxO8A3kF82+HQg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736778275; c=relaxed/simple; bh=p91sJ8ogsUGViJGmuJwacKfIF+XDLJqev8Cr+/VTREI=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version:Content-Type; b=IuTLkpXemUwcBtEEha+SghY4T/YFv/soTgb9c2lKqBsMuq5LxlVVVCqWz2bdsLBfy6vhRtYcX3qsrLb+y7Kd0VbKDq75A1rczWvKEkCuGVwgG+Osk2qwN+C55D8FYHdbIR9mkga9aHN7RN9dtylvIfD6+7rb0Ai351Y3Q/99ZLM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=HQKd0SfE; arc=none smtp.client-ip=209.85.128.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="HQKd0SfE" Received: by mail-wm1-f44.google.com with SMTP id 5b1f17b1804b1-4364a37a1d7so45998655e9.3 for ; Mon, 13 Jan 2025 06:24:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1736778270; x=1737383070; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=U6Zgp09z8lzuttgM51PWASr/v2Wid6qJyqxiEWy6j0I=; b=HQKd0SfEH6uH3uz+8Z4uPfwNoKbz/LZfQFAVo2nfvhyYfJC//lGMQnKFcOEj+3SkAE WM6mE8238VVtbLWZ92PYf9ezu3+YAL/nPmTJsg2xThMc7tAYqFy55Kp9REOr3MjIDiGn MHi+dvgmvVKm5OeInLugnR4nx4Wrcta5i+Kcu56rONhB+uCdN2epvQfJyNKw2MA4uIrb Ag43hcdjVVauGPl8fgw0RuiL+rDpjI8wPrZxZSekpoPLnsBginJDjOQ2gVMfYQVccdaI incYdq52kavlblrWYn5zTd/hE+7ofiSE38bTrOvmUV3dnFnkh13lmFEXHUN5EUZkhVcu vKKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736778270; x=1737383070; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=U6Zgp09z8lzuttgM51PWASr/v2Wid6qJyqxiEWy6j0I=; b=i7n9tYWhO176qWkUQZRfQhEPRp5Iu0sv3h6pR666gO5apVPfZRuOlJqthurbw+FF1M sp/fA0If/Wqp+NmzqZIIDQE5S5N1MmzJqQrE7+2CmZ2fj/hy0lkxNsIxVaISKh/2+l3N 2Ptj8g+jYkAqk3XqRGQZ2NZ1KGEJe0TwF4jAs0s1/GNOiT+6MRv3AsOvrCZXBjIsEkVA TQDyesi9iucOaWZWLNsDACl3DCOSAlsJt08d9brNz5K8+aN/A3Qt/ON15Neqklg1J8QF OiZbAZExvezaTQQ1XTnpNBdle3qJ3iQZeBrqAdpOcMbMtvUQxloZ4TI7IeEPJAEbhcin Cvww== X-Forwarded-Encrypted: i=1; AJvYcCUy3PA9Mp7a8mAccax5ZyV35kNzDYgn+GWD8DZXxV9n/XLwfkLrWTgOEr19Utm+DV8Kt/CX2mHU8U1A+UY=@vger.kernel.org X-Gm-Message-State: AOJu0YzTVTHRQTp9NOpqo+it265iBFSWogMDiL5Cjl6BV98OTDSsAPyP fXUiy6ecjZN8euD5fqpwZoYQ5QZjvoXaliWvyjp2n9NxmKjcSo7siV1IfvrB544= X-Gm-Gg: ASbGncs8cuA2cs+nWUWQP2Oc4k+TuX+Uunli6p4hysqWKG+sLDTHRpcwolAqoo4yHrE KGe89fqXcZ7KZ3XAEmjKD/cdOD6i0ADLydA17T35terQA5O7z8zi26/2qOytVRiKMaVR4Xbn5w3 UzTZmqn7XnsQMQzB3Nm6GJ5AoKN3mOr0L5pftqFDcEhpfz4NdfuHrYvdagJQvXsQvayW9yKSN92 zLPiHSjef0Xun9PQzQsql9WKGtLk7r2HVnZ/3zCVpwrMC/b0PndORXMzIvTK855GhUeAKCVRedE X-Google-Smtp-Source: AGHT+IGMa2ldUEkMiuq4eErSnbpALm4uDp90FJrdLywqcG2hGCMQ9lmvAIoLp9JhfgfEIFI4LBCwjA== X-Received: by 2002:a05:600c:4586:b0:434:edcf:7461 with SMTP id 5b1f17b1804b1-436e26f135bmr207312645e9.30.1736778270319; Mon, 13 Jan 2025 06:24:30 -0800 (PST) Received: from localhost.localdomain ([2001:861:3382:ef90:1d6c:a0ef:bd2f:4ccd]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-436dd11ddfdsm133376335e9.1.2025.01.13.06.24.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Jan 2025 06:24:29 -0800 (PST) From: Alexandre Ghiti To: Paul Walmsley , Palmer Dabbelt , Samuel Holland , Andrew Jones , Albert Ou , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Alexandre Ghiti , =?UTF-8?q?Cl=C3=A9ment=20L=C3=A9ger?= Subject: [PATCH v4] riscv: Call secondary mmu notifier when flushing the tlb Date: Mon, 13 Jan 2025 15:24:24 +0100 Message-Id: <20250113142424.30487-1-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable This is required to allow the IOMMU driver to correctly flush its own TLB. Signed-off-by: Alexandre Ghiti Reviewed-by: Cl=C3=A9ment L=C3=A9ger Reviewed-by: Samuel Holland --- Changes in v4: - Remove unused variables as reported by kernel test robot Changes in v3: - Rebase on top of 6.13-rc3 Changes in v2: - Rebase on top of 6.9-rc1 arch/riscv/mm/tlbflush.c | 37 ++++++++++++++++++++++--------------- 1 file changed, 22 insertions(+), 15 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 9b6e86ce3867..bb77607c87aa 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -4,6 +4,7 @@ #include #include #include +#include #include #include =20 @@ -78,10 +79,17 @@ static void __ipi_flush_tlb_range_asid(void *info) local_flush_tlb_range_asid(d->start, d->size, d->stride, d->asid); } =20 -static void __flush_tlb_range(const struct cpumask *cmask, unsigned long a= sid, +static inline unsigned long get_mm_asid(struct mm_struct *mm) +{ + return mm ? cntx2asid(atomic_long_read(&mm->context.id)) : FLUSH_TLB_NO_A= SID; +} + +static void __flush_tlb_range(struct mm_struct *mm, + const struct cpumask *cmask, unsigned long start, unsigned long size, unsigned long stride) { + unsigned long asid =3D get_mm_asid(mm); unsigned int cpu; =20 if (cpumask_empty(cmask)) @@ -105,30 +113,26 @@ static void __flush_tlb_range(const struct cpumask *c= mask, unsigned long asid, } =20 put_cpu(); -} =20 -static inline unsigned long get_mm_asid(struct mm_struct *mm) -{ - return cntx2asid(atomic_long_read(&mm->context.id)); + if (mm) + mmu_notifier_arch_invalidate_secondary_tlbs(mm, start, start + size); } =20 void flush_tlb_mm(struct mm_struct *mm) { - __flush_tlb_range(mm_cpumask(mm), get_mm_asid(mm), - 0, FLUSH_TLB_MAX_SIZE, PAGE_SIZE); + __flush_tlb_range(mm, mm_cpumask(mm), 0, FLUSH_TLB_MAX_SIZE, PAGE_SIZE); } =20 void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int page_size) { - __flush_tlb_range(mm_cpumask(mm), get_mm_asid(mm), - start, end - start, page_size); + __flush_tlb_range(mm, mm_cpumask(mm), start, end - start, page_size); } =20 void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) { - __flush_tlb_range(mm_cpumask(vma->vm_mm), get_mm_asid(vma->vm_mm), + __flush_tlb_range(vma->vm_mm, mm_cpumask(vma->vm_mm), addr, PAGE_SIZE, PAGE_SIZE); } =20 @@ -161,13 +165,13 @@ void flush_tlb_range(struct vm_area_struct *vma, unsi= gned long start, } } =20 - __flush_tlb_range(mm_cpumask(vma->vm_mm), get_mm_asid(vma->vm_mm), + __flush_tlb_range(vma->vm_mm, mm_cpumask(vma->vm_mm), start, end - start, stride_size); } =20 void flush_tlb_kernel_range(unsigned long start, unsigned long end) { - __flush_tlb_range(cpu_online_mask, FLUSH_TLB_NO_ASID, + __flush_tlb_range(NULL, cpu_online_mask, start, end - start, PAGE_SIZE); } =20 @@ -175,7 +179,7 @@ void flush_tlb_kernel_range(unsigned long start, unsign= ed long end) void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - __flush_tlb_range(mm_cpumask(vma->vm_mm), get_mm_asid(vma->vm_mm), + __flush_tlb_range(vma->vm_mm, mm_cpumask(vma->vm_mm), start, end - start, PMD_SIZE); } #endif @@ -189,7 +193,10 @@ void arch_tlbbatch_add_pending(struct arch_tlbflush_un= map_batch *batch, struct mm_struct *mm, unsigned long uaddr) { + unsigned long start =3D uaddr & PAGE_MASK; + cpumask_or(&batch->cpumask, &batch->cpumask, mm_cpumask(mm)); + mmu_notifier_arch_invalidate_secondary_tlbs(mm, start, start + PAGE_SIZE); } =20 void arch_flush_tlb_batched_pending(struct mm_struct *mm) @@ -199,7 +206,7 @@ void arch_flush_tlb_batched_pending(struct mm_struct *m= m) =20 void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) { - __flush_tlb_range(&batch->cpumask, FLUSH_TLB_NO_ASID, 0, - FLUSH_TLB_MAX_SIZE, PAGE_SIZE); + __flush_tlb_range(NULL, &batch->cpumask, + 0, FLUSH_TLB_MAX_SIZE, PAGE_SIZE); cpumask_clear(&batch->cpumask); } --=20 2.39.2