From nobody Tue Apr 7 00:50:11 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BC53B33BBAF for ; Thu, 5 Mar 2026 09:39:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772703590; cv=none; b=NRlxvhf2j7C5mRAJu/ULPXybk/uyvQza/QUvs2G2C1/B9hImwQOmRmXEz2HIsTCYyQ8Nm0Uj2gVcRvfNbnFWpf185Amm92b3bd0a1HwpxrDyiTsgun7BlibkwfYVy4Jag05wdd0jiMCy8unwHFX0jL+6UBmrfI1s01+UV7981X0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772703590; c=relaxed/simple; bh=P1HvCqMdm4NZrhz2qfIQ6WDOE/LToL9MrwGTxNicvOI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=q4iRZCJht0E1uXplvdEfIrzqvsVmpVHfX3Vl2w7JJxqcVPpfXnn27Qea2GNrh5HkkyaBKYq+XI3e+K/tu5kk4/Anhyp4sHeeEeBy+DRGTZuwUqRBqgJWpB2996vXWssdvuIhW7NsZGq+e41tIiuFz1nLYBZZ3Wqt9ArFn3w8QSw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=pass smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=N30axjuV; arc=none smtp.client-ip=198.175.65.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="N30axjuV" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1772703589; x=1804239589; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=P1HvCqMdm4NZrhz2qfIQ6WDOE/LToL9MrwGTxNicvOI=; b=N30axjuVP7FLAzf/vq4uJcCbXs6uTWBrpeQOA9Ln6+bVAbZKWuwe0WQG /Kj23a26e2bSn1VoSWpXJuW8VGY079MnM/uIdF9yHJuWSKsBi6QNNx/Pr sa2nFhqEz/v5tsIIULp3pymsoPOKkFXoHgTUYG24ffFKm6qDZdFh1QeFO vmITD9NJFqKhXnq7LRlN7BQnCAOmUD9v4FC1tnY33SJIdluOg++E/+JSO OsbbZosO/wGAg8taDBo3qL+lJI/FPO3OzNRZFRRko6tHCAnL1ks8cGuQt ZxNNJZ8Rf3V+aYgsTghQ1RP0DwZYA19QIA+NYrcSsUZWHhVJa3pOc7/lb Q==; X-CSE-ConnectionGUID: Tuo2B4l7TAab2RlgwC9lWg== X-CSE-MsgGUID: 55Q8GhRISGiMgOGPfwmstg== X-IronPort-AV: E=McAfee;i="6800,10657,11719"; a="84870953" X-IronPort-AV: E=Sophos;i="6.23,102,1770624000"; d="scan'208";a="84870953" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by orvoesa105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Mar 2026 01:39:48 -0800 X-CSE-ConnectionGUID: +uBdXiXbRQOX7B8MyuSWYQ== X-CSE-MsgGUID: JU/2s9FxTmqGmrqqaWLtQA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.23,102,1770624000"; d="scan'208";a="214684978" Received: from vpanait-mobl.ger.corp.intel.com (HELO fedora) ([10.245.244.71]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Mar 2026 01:39:44 -0800 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-xe@lists.freedesktop.org Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , Matthew Brost , =?UTF-8?q?Christian=20K=C3=B6nig?= , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Jason Gunthorpe , Andrew Morton , Simona Vetter , Dave Airlie , Alistair Popple , dri-devel@lists.freedesktop.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 1/4] mm/mmu_notifier: Allow two-pass struct mmu_interval_notifiers Date: Thu, 5 Mar 2026 10:39:06 +0100 Message-ID: <20260305093909.43623-2-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260305093909.43623-1-thomas.hellstrom@linux.intel.com> References: <20260305093909.43623-1-thomas.hellstrom@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable GPU use-cases for mmu_interval_notifiers with hmm often involve starting a gpu operation and then waiting for it to complete. These operations are typically context preemption or TLB flushing. With single-pass notifiers per GPU this doesn't scale in multi-gpu scenarios. In those scenarios we'd want to first start preemption- or TLB flushing on all GPUs and as a second pass wait for them to complete. One can do this on per-driver basis multiplexing per-driver notifiers but that would mean sharing the notifier "user" lock across all GPUs and that doesn't scale well either, so adding support for multi-pass in the core appears to be the right choice. Implement two-pass capability in the mmu_interval_notifier. Use a linked list for the final passes to minimize the impact for use-cases that don't need the multi-pass functionality by avoiding a second interval tree walk, and to be able to easily pass data between the two passes. v1: - Restrict to two passes (Jason Gunthorpe) - Improve on documentation (Jason Gunthorpe) - Improve on function naming (Alistair Popple) v2: - Include the invalidate_finish() callback in the struct mmu_interval_notifier_ops. - Update documentation (GitHub Copilot:claude-sonnet-4.6) - Use lockless list for list management. v3: - Update kerneldoc for the struct mmu_interval_notifier_finish::list member (Matthew Brost) - Add a WARN_ON_ONCE() checking for NULL invalidate_finish() op if if invalidate_start() is non-NULL. (Matthew Brost) v4: - Addressed documentation review comments by David Hildenbrand. Cc: Matthew Brost Cc: Christian K=C3=B6nig Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Cc: Jason Gunthorpe Cc: Andrew Morton Cc: Simona Vetter Cc: Dave Airlie Cc: Alistair Popple Cc: Cc: Cc: Assisted-by: GitHub Copilot:claude-sonnet-4.6 # Documentation only. Signed-off-by: Thomas Hellstr=C3=B6m Acked-by: David Hildenbrand (Arm) Reviewed-by: Maarten Lankhorst --- include/linux/mmu_notifier.h | 42 +++++++++++++++++++++++ mm/mmu_notifier.c | 65 +++++++++++++++++++++++++++++++----- 2 files changed, 98 insertions(+), 9 deletions(-) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index 07a2bbaf86e9..dcdfdf1e0b39 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -233,16 +233,58 @@ struct mmu_notifier { unsigned int users; }; =20 +/** + * struct mmu_interval_notifier_finish - mmu_interval_notifier two-pass ab= straction + * @link: Lockless list link for the notifiers pending pass list + * @notifier: The mmu_interval_notifier for which the finish pass is calle= d. + * + * Allocate, typically using GFP_NOWAIT in the interval notifier's start p= ass. + * Note that with a large number of notifiers implementing two passes, + * allocation with GFP_NOWAIT will become increasingly likely to fail, so = consider + * implementing a small pool instead of using kmalloc() allocations. + * + * If the implementation needs to pass data between the start and the fini= sh passes, + * the recommended way is to embed struct mmu_interval_notifier_finish int= o a larger + * structure that also contains the data needed to be shared. Keep in mind= that + * a notifier callback can be invoked in parallel, and each invocation nee= ds its + * own struct mmu_interval_notifier_finish. + * + * If allocation fails, then the &mmu_interval_notifier_ops->invalidate_st= art op + * needs to implements the full notifier functionality. Please refer to its + * documentation. + */ +struct mmu_interval_notifier_finish { + struct llist_node link; + struct mmu_interval_notifier *notifier; +}; + /** * struct mmu_interval_notifier_ops * @invalidate: Upon return the caller must stop using any SPTEs within th= is * range. This function can sleep. Return false only if sleep= ing * was required but mmu_notifier_range_blockable(range) is fa= lse. + * @invalidate_start: Similar to @invalidate, but intended for two-pass no= tifier + * callbacks where the call to @invalidate_start is the= first + * pass and any struct mmu_interval_notifier_finish poi= nter + * returned in the @finish parameter describes the fini= sh pass. + * If *@finish is %NULL on return, then no final pass w= ill be + * called, and @invalidate_start needs to implement the= full + * notifier, behaving like @invalidate. The value of *@= finish + * is guaranteed to be %NULL at function entry. + * @invalidate_finish: Called as the second pass for any notifier that ret= urned + * a non-NULL *@finish from @invalidate_start. The @fi= nish + * pointer passed here is the same one returned by + * @invalidate_start. */ struct mmu_interval_notifier_ops { bool (*invalidate)(struct mmu_interval_notifier *interval_sub, const struct mmu_notifier_range *range, unsigned long cur_seq); + bool (*invalidate_start)(struct mmu_interval_notifier *interval_sub, + const struct mmu_notifier_range *range, + unsigned long cur_seq, + struct mmu_interval_notifier_finish **finish); + void (*invalidate_finish)(struct mmu_interval_notifier_finish *finish); }; =20 struct mmu_interval_notifier { diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index a6cdf3674bdc..4d8a64ce8eda 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -260,6 +260,15 @@ mmu_interval_read_begin(struct mmu_interval_notifier *= interval_sub) } EXPORT_SYMBOL_GPL(mmu_interval_read_begin); =20 +static void mn_itree_finish_pass(struct llist_head *finish_passes) +{ + struct llist_node *first =3D llist_reverse_order(__llist_del_all(finish_p= asses)); + struct mmu_interval_notifier_finish *f, *next; + + llist_for_each_entry_safe(f, next, first, link) + f->notifier->ops->invalidate_finish(f); +} + static void mn_itree_release(struct mmu_notifier_subscriptions *subscripti= ons, struct mm_struct *mm) { @@ -271,6 +280,7 @@ static void mn_itree_release(struct mmu_notifier_subscr= iptions *subscriptions, .end =3D ULONG_MAX, }; struct mmu_interval_notifier *interval_sub; + LLIST_HEAD(finish_passes); unsigned long cur_seq; bool ret; =20 @@ -278,11 +288,27 @@ static void mn_itree_release(struct mmu_notifier_subs= criptions *subscriptions, mn_itree_inv_start_range(subscriptions, &range, &cur_seq); interval_sub; interval_sub =3D mn_itree_inv_next(interval_sub, &range)) { - ret =3D interval_sub->ops->invalidate(interval_sub, &range, - cur_seq); + if (interval_sub->ops->invalidate_start) { + struct mmu_interval_notifier_finish *finish =3D NULL; + + ret =3D interval_sub->ops->invalidate_start(interval_sub, + &range, + cur_seq, + &finish); + if (ret && finish) { + finish->notifier =3D interval_sub; + __llist_add(&finish->link, &finish_passes); + } + + } else { + ret =3D interval_sub->ops->invalidate(interval_sub, + &range, + cur_seq); + } WARN_ON(!ret); } =20 + mn_itree_finish_pass(&finish_passes); mn_itree_inv_end(subscriptions); } =20 @@ -430,7 +456,9 @@ static int mn_itree_invalidate(struct mmu_notifier_subs= criptions *subscriptions, const struct mmu_notifier_range *range) { struct mmu_interval_notifier *interval_sub; + LLIST_HEAD(finish_passes); unsigned long cur_seq; + int err =3D 0; =20 for (interval_sub =3D mn_itree_inv_start_range(subscriptions, range, &cur_seq); @@ -438,23 +466,41 @@ static int mn_itree_invalidate(struct mmu_notifier_su= bscriptions *subscriptions, interval_sub =3D mn_itree_inv_next(interval_sub, range)) { bool ret; =20 - ret =3D interval_sub->ops->invalidate(interval_sub, range, - cur_seq); + if (interval_sub->ops->invalidate_start) { + struct mmu_interval_notifier_finish *finish =3D NULL; + + ret =3D interval_sub->ops->invalidate_start(interval_sub, + range, + cur_seq, + &finish); + if (ret && finish) { + finish->notifier =3D interval_sub; + __llist_add(&finish->link, &finish_passes); + } + + } else { + ret =3D interval_sub->ops->invalidate(interval_sub, + range, + cur_seq); + } if (!ret) { if (WARN_ON(mmu_notifier_range_blockable(range))) continue; - goto out_would_block; + err =3D -EAGAIN; + break; } } - return 0; =20 -out_would_block: + mn_itree_finish_pass(&finish_passes); + /* * On -EAGAIN the non-blocking caller is not allowed to call * invalidate_range_end() */ - mn_itree_inv_end(subscriptions); - return -EAGAIN; + if (err) + mn_itree_inv_end(subscriptions); + + return err; } =20 static int mn_hlist_invalidate_range_start( @@ -976,6 +1022,7 @@ int mmu_interval_notifier_insert(struct mmu_interval_n= otifier *interval_sub, struct mmu_notifier_subscriptions *subscriptions; int ret; =20 + WARN_ON_ONCE(ops->invalidate_start && !ops->invalidate_finish); might_lock(&mm->mmap_lock); =20 subscriptions =3D smp_load_acquire(&mm->notifier_subscriptions); --=20 2.53.0 From nobody Tue Apr 7 00:50:11 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C17DA37AA6C for ; Thu, 5 Mar 2026 09:39:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772703594; cv=none; b=ZGTRm5gQ8eNz9iRQBz2fOiYfzyszrDL+uWPRAxHMiJ9laHRUnwwOcIacGbHlvsr7fPWjPhlVSV4evFD4xnsMusRLP92nZm7mBdSM80Dr8PstszS2mlYNvOcw5mV2kRUmyAff5ymsuE6FvwNECzqw4N6sBRS3Lm31ikqhYrknHek= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772703594; c=relaxed/simple; bh=RxDfvkY1zjSBVKQEcXFdDHmfFd96afZzeTBuuwtsgrc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=QRVif1BMsGcujJ4kXjHq0tUlHMgI9bPi0KObcFm72kafmEgMBShI+3OLqGZt9DdiiL/6bfEgUTwxVpQtBCjAyoByzoOa+qS3wHzPAN3UpwtgW2LU5pvtN0jp0RyHvnhqO2GGUbSuFBeQmkMIDoC4PXQYDBcL7jtFs0LEx88daXc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=pass smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=kKwB+4Dy; arc=none smtp.client-ip=198.175.65.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="kKwB+4Dy" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1772703593; x=1804239593; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=RxDfvkY1zjSBVKQEcXFdDHmfFd96afZzeTBuuwtsgrc=; b=kKwB+4Dyw4UXOMynvOEoxGNRDH7HCgMZ3aM5PAqm9mP2U1AetUXgY9Rk q9x4BBTCkpdybwjnLHThpMP5v7u/2N+loU92+LszmPLhvgN3rSb9b8ozL 0Y4z18kDNVS5/GO1r8Fe0elrfhT9o/8GSam9cZIcUWjwHKaeYQ2QjXADN 7t62IniLx9nu+8Fvj+g+cpFXOTfmx144s8U9eZ4apydZXJHl4oA8C/RCY LqHUWUkb0V0ckLwVNWb0s7Qh9YmOQNr5ZtGG6ep1a1lkGKNFNSnmGqELQ mPJ5b0CQ2EZQh3jsh/tN0n1Q5lFaRCx22eoUd1vidqru0jzeTGFU5U99d w==; X-CSE-ConnectionGUID: g2vsAKljQRKlNvXAZf/J9g== X-CSE-MsgGUID: usSwRVf5QJOUyYhwWFlc9A== X-IronPort-AV: E=McAfee;i="6800,10657,11719"; a="84870968" X-IronPort-AV: E=Sophos;i="6.23,102,1770624000"; d="scan'208";a="84870968" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by orvoesa105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Mar 2026 01:39:52 -0800 X-CSE-ConnectionGUID: jIoRPFlKQDGYcH4Yx7DLnw== X-CSE-MsgGUID: jic1ah2sT26hBeUow8COVQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.23,102,1770624000"; d="scan'208";a="214685012" Received: from vpanait-mobl.ger.corp.intel.com (HELO fedora) ([10.245.244.71]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Mar 2026 01:39:48 -0800 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-xe@lists.freedesktop.org Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , Matthew Brost , =?UTF-8?q?Christian=20K=C3=B6nig?= , dri-devel@lists.freedesktop.org, Jason Gunthorpe , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Simona Vetter , Dave Airlie , Alistair Popple , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 2/4] drm/xe/userptr: Convert invalidation to two-pass MMU notifier Date: Thu, 5 Mar 2026 10:39:07 +0100 Message-ID: <20260305093909.43623-3-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260305093909.43623-1-thomas.hellstrom@linux.intel.com> References: <20260305093909.43623-1-thomas.hellstrom@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable In multi-GPU scenarios, asynchronous GPU job latency is a bottleneck if each notifier waits for its own GPU before returning. The two-pass mmu_interval_notifier infrastructure allows deferring the wait to a second pass, so all GPUs can be signalled in the first pass before any of them are waited on. Convert the userptr invalidation to use the two-pass model: Use invalidate_start as the first pass to mark the VMA for repin and enable software signalling on the VM reservation fences to start any gpu work needed for signaling. Fall back to completing the work synchronously if all fences are already signalled, or if a concurrent invalidation is already using the embedded finish structure. Use invalidate_finish as the second pass to wait for the reservation fences to complete, invalidate the GPU TLB in fault mode, and unmap the gpusvm pages. Embed a struct mmu_interval_notifier_finish in struct xe_userptr to avoid dynamic allocation in the notifier callback. Use a finish_inuse flag to prevent two concurrent invalidations from using it simultaneously; fall back to the synchronous path for the second caller. v3: - Add locking asserts in notifier components (Matt Brost) - Clean up newlines (Matt Brost) - Update the userptr notifier state member locking documentation (Matt Brost) Assisted-by: GitHub Copilot:claude-sonnet-4.6 Signed-off-by: Thomas Hellstr=C3=B6m Reviewed-by: Matthew Brost --- drivers/gpu/drm/xe/xe_userptr.c | 108 +++++++++++++++++++++++++------- drivers/gpu/drm/xe/xe_userptr.h | 14 ++++- 2 files changed, 99 insertions(+), 23 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_userptr.c b/drivers/gpu/drm/xe/xe_userpt= r.c index e120323c43bc..37032b8125a6 100644 --- a/drivers/gpu/drm/xe/xe_userptr.c +++ b/drivers/gpu/drm/xe/xe_userptr.c @@ -10,6 +10,14 @@ =20 #include "xe_trace_bo.h" =20 +static void xe_userptr_assert_in_notifier(struct xe_vm *vm) +{ + lockdep_assert(lockdep_is_held_type(&vm->svm.gpusvm.notifier_lock, 0) || + (lockdep_is_held(&vm->lock) && + lockdep_is_held_type(&vm->svm.gpusvm.notifier_lock, 1) && + dma_resv_held(xe_vm_resv(vm)))); +} + /** * xe_vma_userptr_check_repin() - Advisory check for repin needed * @uvma: The userptr vma @@ -73,18 +81,46 @@ int xe_vma_userptr_pin_pages(struct xe_userptr_vma *uvm= a) &ctx); } =20 -static void __vma_userptr_invalidate(struct xe_vm *vm, struct xe_userptr_v= ma *uvma) +static void xe_vma_userptr_do_inval(struct xe_vm *vm, struct xe_userptr_vm= a *uvma, + bool is_deferred) { struct xe_userptr *userptr =3D &uvma->userptr; struct xe_vma *vma =3D &uvma->vma; - struct dma_resv_iter cursor; - struct dma_fence *fence; struct drm_gpusvm_ctx ctx =3D { .in_notifier =3D true, .read_only =3D xe_vma_read_only(vma), }; long err; =20 + xe_userptr_assert_in_notifier(vm); + + err =3D dma_resv_wait_timeout(xe_vm_resv(vm), + DMA_RESV_USAGE_BOOKKEEP, + false, MAX_SCHEDULE_TIMEOUT); + XE_WARN_ON(err <=3D 0); + + if (xe_vm_in_fault_mode(vm) && userptr->initial_bind) { + err =3D xe_vm_invalidate_vma(vma); + XE_WARN_ON(err); + } + + if (is_deferred) + userptr->finish_inuse =3D false; + drm_gpusvm_unmap_pages(&vm->svm.gpusvm, &uvma->userptr.pages, + xe_vma_size(vma) >> PAGE_SHIFT, &ctx); +} + +static struct mmu_interval_notifier_finish * +xe_vma_userptr_invalidate_pass1(struct xe_vm *vm, struct xe_userptr_vma *u= vma) +{ + struct xe_userptr *userptr =3D &uvma->userptr; + struct xe_vma *vma =3D &uvma->vma; + struct dma_resv_iter cursor; + struct dma_fence *fence; + bool signaled =3D true; + + xe_userptr_assert_in_notifier(vm); + /* * Tell exec and rebind worker they need to repin and rebind this * userptr. @@ -105,27 +141,32 @@ static void __vma_userptr_invalidate(struct xe_vm *vm= , struct xe_userptr_vma *uv */ dma_resv_iter_begin(&cursor, xe_vm_resv(vm), DMA_RESV_USAGE_BOOKKEEP); - dma_resv_for_each_fence_unlocked(&cursor, fence) + dma_resv_for_each_fence_unlocked(&cursor, fence) { dma_fence_enable_sw_signaling(fence); + if (signaled && !dma_fence_is_signaled(fence)) + signaled =3D false; + } dma_resv_iter_end(&cursor); =20 - err =3D dma_resv_wait_timeout(xe_vm_resv(vm), - DMA_RESV_USAGE_BOOKKEEP, - false, MAX_SCHEDULE_TIMEOUT); - XE_WARN_ON(err <=3D 0); - - if (xe_vm_in_fault_mode(vm) && userptr->initial_bind) { - err =3D xe_vm_invalidate_vma(vma); - XE_WARN_ON(err); + /* + * Only one caller at a time can use the multi-pass state. + * If it's already in use, or all fences are already signaled, + * proceed directly to invalidation without deferring. + */ + if (signaled || userptr->finish_inuse) { + xe_vma_userptr_do_inval(vm, uvma, false); + return NULL; } =20 - drm_gpusvm_unmap_pages(&vm->svm.gpusvm, &uvma->userptr.pages, - xe_vma_size(vma) >> PAGE_SHIFT, &ctx); + userptr->finish_inuse =3D true; + + return &userptr->finish; } =20 -static bool vma_userptr_invalidate(struct mmu_interval_notifier *mni, - const struct mmu_notifier_range *range, - unsigned long cur_seq) +static bool xe_vma_userptr_invalidate_start(struct mmu_interval_notifier *= mni, + const struct mmu_notifier_range *range, + unsigned long cur_seq, + struct mmu_interval_notifier_finish **p_finish) { struct xe_userptr_vma *uvma =3D container_of(mni, typeof(*uvma), userptr.= notifier); struct xe_vma *vma =3D &uvma->vma; @@ -138,21 +179,40 @@ static bool vma_userptr_invalidate(struct mmu_interva= l_notifier *mni, return false; =20 vm_dbg(&xe_vma_vm(vma)->xe->drm, - "NOTIFIER: addr=3D0x%016llx, range=3D0x%016llx", + "NOTIFIER PASS1: addr=3D0x%016llx, range=3D0x%016llx", xe_vma_start(vma), xe_vma_size(vma)); =20 down_write(&vm->svm.gpusvm.notifier_lock); mmu_interval_set_seq(mni, cur_seq); =20 - __vma_userptr_invalidate(vm, uvma); + *p_finish =3D xe_vma_userptr_invalidate_pass1(vm, uvma); + up_write(&vm->svm.gpusvm.notifier_lock); - trace_xe_vma_userptr_invalidate_complete(vma); + if (!*p_finish) + trace_xe_vma_userptr_invalidate_complete(vma); =20 return true; } =20 +static void xe_vma_userptr_invalidate_finish(struct mmu_interval_notifier_= finish *finish) +{ + struct xe_userptr_vma *uvma =3D container_of(finish, typeof(*uvma), userp= tr.finish); + struct xe_vma *vma =3D &uvma->vma; + struct xe_vm *vm =3D xe_vma_vm(vma); + + vm_dbg(&xe_vma_vm(vma)->xe->drm, + "NOTIFIER PASS2: addr=3D0x%016llx, range=3D0x%016llx", + xe_vma_start(vma), xe_vma_size(vma)); + + down_write(&vm->svm.gpusvm.notifier_lock); + xe_vma_userptr_do_inval(vm, uvma, true); + up_write(&vm->svm.gpusvm.notifier_lock); + trace_xe_vma_userptr_invalidate_complete(vma); +} + static const struct mmu_interval_notifier_ops vma_userptr_notifier_ops =3D= { - .invalidate =3D vma_userptr_invalidate, + .invalidate_start =3D xe_vma_userptr_invalidate_start, + .invalidate_finish =3D xe_vma_userptr_invalidate_finish, }; =20 #if IS_ENABLED(CONFIG_DRM_XE_USERPTR_INVAL_INJECT) @@ -164,6 +224,7 @@ static const struct mmu_interval_notifier_ops vma_userp= tr_notifier_ops =3D { */ void xe_vma_userptr_force_invalidate(struct xe_userptr_vma *uvma) { + static struct mmu_interval_notifier_finish *finish; struct xe_vm *vm =3D xe_vma_vm(&uvma->vma); =20 /* Protect against concurrent userptr pinning */ @@ -179,7 +240,10 @@ void xe_vma_userptr_force_invalidate(struct xe_userptr= _vma *uvma) if (!mmu_interval_read_retry(&uvma->userptr.notifier, uvma->userptr.pages.notifier_seq)) uvma->userptr.pages.notifier_seq -=3D 2; - __vma_userptr_invalidate(vm, uvma); + + finish =3D xe_vma_userptr_invalidate_pass1(vm, uvma); + if (finish) + xe_vma_userptr_do_inval(vm, uvma, true); } #endif =20 diff --git a/drivers/gpu/drm/xe/xe_userptr.h b/drivers/gpu/drm/xe/xe_userpt= r.h index ef801234991e..e1830c2f5fd2 100644 --- a/drivers/gpu/drm/xe/xe_userptr.h +++ b/drivers/gpu/drm/xe/xe_userptr.h @@ -56,7 +56,19 @@ struct xe_userptr { * @notifier: MMU notifier for user pointer (invalidation call back) */ struct mmu_interval_notifier notifier; - + /** + * @finish: MMU notifier finish structure for two-pass invalidation. + * Embedded here to avoid allocation in the notifier callback. + * Protected by struct xe_vm::svm.gpusvm.notifier_lock in write mode + * alternatively by the same lock in read mode *and* the vm resv held. + */ + struct mmu_interval_notifier_finish finish; + /** + * @finish_inuse: Whether @finish is currently in use by an in-progress + * two-pass invalidation. + * Protected using the same locking as @finish. + */ + bool finish_inuse; /** * @initial_bind: user pointer has been bound at least once. * write: vm->svm.gpusvm.notifier_lock in read mode and vm->resv held. --=20 2.53.0 From nobody Tue Apr 7 00:50:11 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F3EA337C0E1 for ; Thu, 5 Mar 2026 09:39:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772703598; cv=none; b=Fo0bj5TDn6zC2URRVo+bo0KFRM0iLd7bmjhw6SU+rbOlORApz7UWjJqSYUSM9nKgAU0vvkswAidX/PR6b+nur+IxTDBjRKi7jCLTF26gUfp5ZokJmN9Ocj3uEMy45+0m41rug7M7FttXNwR3ttmfkVz4Ybrcy0zyazQ1kFqOg4U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772703598; c=relaxed/simple; bh=vqu2mBttwBYNe3AtussgI0qd6tz0JHkp+5Nwz38MtP0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=egL1Rr4tWeLaOPxXjHDcl71JFTBmQokFDvbq5ZDPk/y+gQOg+f5+2vxW0L6ITV3nqlIm8W5s9G+yGqiG8MVkuKDjNqnIjVDX8LXoGeZIHOqAypCsXstHav8E5h/PWaM8hCHRyrc2aYyEtYWE6R2KdKzJ3lA8L+bKgmAXszfGJfE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=pass smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=cCP7LtJr; arc=none smtp.client-ip=198.175.65.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="cCP7LtJr" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1772703597; x=1804239597; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=vqu2mBttwBYNe3AtussgI0qd6tz0JHkp+5Nwz38MtP0=; b=cCP7LtJrwR/U+DwFyIpXOP/2d1FKHZRQnVKfyzTkonoM+j+VEc7agX1E RIqSRU7wQzGEjIQwCYFPShnMMSMMMq6BYXsKBcrGqUJNkN5A2eWBKOLVF /ThtZX88ni6sjTp8ng856DBxM8+IeHBEac/3afbYUJ23IxIdrFGX4ii/t 6hFBN3aI4yaVuSPxD/9f4WXUeiGoBYqK+8427/gRRgXN6O+hNCz3m0QBB KrMIaS8kLhT2sH5l4xwAT71pG9OsfH5GP2lk/YyAsfcwir8DNipoWdMBl PlH+LWqsURsQQ+19u9YBm5IVC+/2RzZVHLlQWmaeF9PkSy4/7I3MBAGBI A==; X-CSE-ConnectionGUID: KhMeNwtqQt+kLqm6GwJRUw== X-CSE-MsgGUID: aPxNXO5QSU6pxniLhyIgUQ== X-IronPort-AV: E=McAfee;i="6800,10657,11719"; a="84870986" X-IronPort-AV: E=Sophos;i="6.23,102,1770624000"; d="scan'208";a="84870986" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by orvoesa105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Mar 2026 01:39:56 -0800 X-CSE-ConnectionGUID: jBY6NJDiTZKeM7cOBB8TfQ== X-CSE-MsgGUID: DkUumHz0TWmi+je2Kt1Tww== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.23,102,1770624000"; d="scan'208";a="214685023" Received: from vpanait-mobl.ger.corp.intel.com (HELO fedora) ([10.245.244.71]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Mar 2026 01:39:53 -0800 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-xe@lists.freedesktop.org Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , Matthew Brost , =?UTF-8?q?Christian=20K=C3=B6nig?= , dri-devel@lists.freedesktop.org, Jason Gunthorpe , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Simona Vetter , Dave Airlie , Alistair Popple , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 3/4] drm/xe: Split TLB invalidation into submit and wait steps Date: Thu, 5 Mar 2026 10:39:08 +0100 Message-ID: <20260305093909.43623-4-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260305093909.43623-1-thomas.hellstrom@linux.intel.com> References: <20260305093909.43623-1-thomas.hellstrom@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable xe_vm_range_tilemask_tlb_inval() submits TLB invalidation requests to all GTs in a tile mask and then immediately waits for them to complete before returning. This is fine for the existing callers, but a subsequent patch will need to defer the wait in order to overlap TLB invalidations across multiple VMAs. Introduce xe_tlb_inval_range_tilemask_submit() and xe_tlb_inval_batch_wait() in xe_tlb_inval.c as the submit and wait halves respectively. The batch of fences is carried in the new xe_tlb_inval_batch structure. Remove xe_vm_range_tilemask_tlb_inval() and convert all three call sites to the new API. v3: - Don't wait on TLB invalidation batches if the corresponding batch submit returns an error. (Matt Brost) - s/_batch/batch/ (Matt Brost) Assisted-by: GitHub Copilot:claude-sonnet-4.6 Signed-off-by: Thomas Hellstr=C3=B6m Reviewed-by: Matthew Brost --- drivers/gpu/drm/xe/xe_svm.c | 8 ++- drivers/gpu/drm/xe/xe_tlb_inval.c | 84 +++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_tlb_inval.h | 6 ++ drivers/gpu/drm/xe/xe_tlb_inval_types.h | 14 +++++ drivers/gpu/drm/xe/xe_vm.c | 69 +++----------------- drivers/gpu/drm/xe/xe_vm.h | 3 - drivers/gpu/drm/xe/xe_vm_madvise.c | 10 ++- drivers/gpu/drm/xe/xe_vm_types.h | 1 + 8 files changed, 127 insertions(+), 68 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c index 002b6c22ad3f..a91c84487a67 100644 --- a/drivers/gpu/drm/xe/xe_svm.c +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -19,6 +19,7 @@ #include "xe_pt.h" #include "xe_svm.h" #include "xe_tile.h" +#include "xe_tlb_inval.h" #include "xe_ttm_vram_mgr.h" #include "xe_vm.h" #include "xe_vm_types.h" @@ -225,6 +226,7 @@ static void xe_svm_invalidate(struct drm_gpusvm *gpusvm, const struct mmu_notifier_range *mmu_range) { struct xe_vm *vm =3D gpusvm_to_vm(gpusvm); + struct xe_tlb_inval_batch batch; struct xe_device *xe =3D vm->xe; struct drm_gpusvm_range *r, *first; struct xe_tile *tile; @@ -276,8 +278,10 @@ static void xe_svm_invalidate(struct drm_gpusvm *gpusv= m, =20 xe_device_wmb(xe); =20 - err =3D xe_vm_range_tilemask_tlb_inval(vm, adj_start, adj_end, tile_mask); - WARN_ON_ONCE(err); + err =3D xe_tlb_inval_range_tilemask_submit(xe, vm->usm.asid, adj_start, a= dj_end, + tile_mask, &batch); + if (!WARN_ON_ONCE(err)) + xe_tlb_inval_batch_wait(&batch); =20 range_notifier_event_end: r =3D first; diff --git a/drivers/gpu/drm/xe/xe_tlb_inval.c b/drivers/gpu/drm/xe/xe_tlb_= inval.c index 933f30fb617d..10dcd4abb00f 100644 --- a/drivers/gpu/drm/xe/xe_tlb_inval.c +++ b/drivers/gpu/drm/xe/xe_tlb_inval.c @@ -486,3 +486,87 @@ bool xe_tlb_inval_idle(struct xe_tlb_inval *tlb_inval) guard(spinlock_irq)(&tlb_inval->pending_lock); return list_is_singular(&tlb_inval->pending_fences); } + +/** + * xe_tlb_inval_batch_wait() - Wait for all fences in a TLB invalidation b= atch + * @batch: Batch of TLB invalidation fences to wait on + * + * Waits for every fence in @batch to signal, then resets @batch so it can= be + * reused for a subsequent invalidation. + */ +void xe_tlb_inval_batch_wait(struct xe_tlb_inval_batch *batch) +{ + struct xe_tlb_inval_fence *fence =3D &batch->fence[0]; + unsigned int i; + + for (i =3D 0; i < batch->num_fences; ++i) + xe_tlb_inval_fence_wait(fence++); + + batch->num_fences =3D 0; +} + +/** + * xe_tlb_inval_range_tilemask_submit() - Submit TLB invalidations for an + * address range on a tile mask + * @xe: The xe device + * @asid: Address space ID + * @start: start address + * @end: end address + * @tile_mask: mask for which gt's issue tlb invalidation + * @batch: Batch of tlb invalidate fences + * + * Issue a range based TLB invalidation for gt's in tilemask + * If the function returns an error, there is no need to call + * xe_tlb_inval_batch_wait() on @batch. + * + * Returns 0 for success, negative error code otherwise. + */ +int xe_tlb_inval_range_tilemask_submit(struct xe_device *xe, u32 asid, + u64 start, u64 end, u8 tile_mask, + struct xe_tlb_inval_batch *batch) +{ + struct xe_tlb_inval_fence *fence =3D &batch->fence[0]; + struct xe_tile *tile; + u32 fence_id =3D 0; + u8 id; + int err; + + batch->num_fences =3D 0; + if (!tile_mask) + return 0; + + for_each_tile(tile, xe, id) { + if (!(tile_mask & BIT(id))) + continue; + + xe_tlb_inval_fence_init(&tile->primary_gt->tlb_inval, + &fence[fence_id], true); + + err =3D xe_tlb_inval_range(&tile->primary_gt->tlb_inval, + &fence[fence_id], start, end, + asid, NULL); + if (err) + goto wait; + ++fence_id; + + if (!tile->media_gt) + continue; + + xe_tlb_inval_fence_init(&tile->media_gt->tlb_inval, + &fence[fence_id], true); + + err =3D xe_tlb_inval_range(&tile->media_gt->tlb_inval, + &fence[fence_id], start, end, + asid, NULL); + if (err) + goto wait; + ++fence_id; + } + +wait: + batch->num_fences =3D fence_id; + if (err) + xe_tlb_inval_batch_wait(batch); + + return err; +} diff --git a/drivers/gpu/drm/xe/xe_tlb_inval.h b/drivers/gpu/drm/xe/xe_tlb_= inval.h index 62089254fa23..a76b7823a5f2 100644 --- a/drivers/gpu/drm/xe/xe_tlb_inval.h +++ b/drivers/gpu/drm/xe/xe_tlb_inval.h @@ -45,4 +45,10 @@ void xe_tlb_inval_done_handler(struct xe_tlb_inval *tlb_= inval, int seqno); =20 bool xe_tlb_inval_idle(struct xe_tlb_inval *tlb_inval); =20 +int xe_tlb_inval_range_tilemask_submit(struct xe_device *xe, u32 asid, + u64 start, u64 end, u8 tile_mask, + struct xe_tlb_inval_batch *batch); + +void xe_tlb_inval_batch_wait(struct xe_tlb_inval_batch *batch); + #endif /* _XE_TLB_INVAL_ */ diff --git a/drivers/gpu/drm/xe/xe_tlb_inval_types.h b/drivers/gpu/drm/xe/x= e_tlb_inval_types.h index 3b089f90f002..3d1797d186fd 100644 --- a/drivers/gpu/drm/xe/xe_tlb_inval_types.h +++ b/drivers/gpu/drm/xe/xe_tlb_inval_types.h @@ -9,6 +9,8 @@ #include #include =20 +#include "xe_device_types.h" + struct drm_suballoc; struct xe_tlb_inval; =20 @@ -132,4 +134,16 @@ struct xe_tlb_inval_fence { ktime_t inval_time; }; =20 +/** + * struct xe_tlb_inval_batch - Batch of TLB invalidation fences + * + * Holds one fence per GT covered by a TLB invalidation request. + */ +struct xe_tlb_inval_batch { + /** @fence: per-GT TLB invalidation fences */ + struct xe_tlb_inval_fence fence[XE_MAX_TILES_PER_DEVICE * XE_MAX_GT_PER_T= ILE]; + /** @num_fences: number of valid entries in @fence */ + unsigned int num_fences; +}; + #endif diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c index 548b0769b3ef..a3c2e8cefec7 100644 --- a/drivers/gpu/drm/xe/xe_vm.c +++ b/drivers/gpu/drm/xe/xe_vm.c @@ -3966,66 +3966,6 @@ void xe_vm_unlock(struct xe_vm *vm) dma_resv_unlock(xe_vm_resv(vm)); } =20 -/** - * xe_vm_range_tilemask_tlb_inval - Issue a TLB invalidation on this tilem= ask for an - * address range - * @vm: The VM - * @start: start address - * @end: end address - * @tile_mask: mask for which gt's issue tlb invalidation - * - * Issue a range based TLB invalidation for gt's in tilemask - * - * Returns 0 for success, negative error code otherwise. - */ -int xe_vm_range_tilemask_tlb_inval(struct xe_vm *vm, u64 start, - u64 end, u8 tile_mask) -{ - struct xe_tlb_inval_fence - fence[XE_MAX_TILES_PER_DEVICE * XE_MAX_GT_PER_TILE]; - struct xe_tile *tile; - u32 fence_id =3D 0; - u8 id; - int err; - - if (!tile_mask) - return 0; - - for_each_tile(tile, vm->xe, id) { - if (!(tile_mask & BIT(id))) - continue; - - xe_tlb_inval_fence_init(&tile->primary_gt->tlb_inval, - &fence[fence_id], true); - - err =3D xe_tlb_inval_range(&tile->primary_gt->tlb_inval, - &fence[fence_id], start, end, - vm->usm.asid, NULL); - if (err) - goto wait; - ++fence_id; - - if (!tile->media_gt) - continue; - - xe_tlb_inval_fence_init(&tile->media_gt->tlb_inval, - &fence[fence_id], true); - - err =3D xe_tlb_inval_range(&tile->media_gt->tlb_inval, - &fence[fence_id], start, end, - vm->usm.asid, NULL); - if (err) - goto wait; - ++fence_id; - } - -wait: - for (id =3D 0; id < fence_id; ++id) - xe_tlb_inval_fence_wait(&fence[id]); - - return err; -} - /** * xe_vm_invalidate_vma - invalidate GPU mappings for VMA without a lock * @vma: VMA to invalidate @@ -4040,6 +3980,7 @@ int xe_vm_invalidate_vma(struct xe_vma *vma) { struct xe_device *xe =3D xe_vma_vm(vma)->xe; struct xe_vm *vm =3D xe_vma_vm(vma); + struct xe_tlb_inval_batch batch; struct xe_tile *tile; u8 tile_mask =3D 0; int ret =3D 0; @@ -4080,12 +4021,16 @@ int xe_vm_invalidate_vma(struct xe_vma *vma) =20 xe_device_wmb(xe); =20 - ret =3D xe_vm_range_tilemask_tlb_inval(xe_vma_vm(vma), xe_vma_start(vma), - xe_vma_end(vma), tile_mask); + ret =3D xe_tlb_inval_range_tilemask_submit(xe, xe_vma_vm(vma)->usm.asid, + xe_vma_start(vma), xe_vma_end(vma), + tile_mask, &batch); =20 /* WRITE_ONCE pairs with READ_ONCE in xe_vm_has_valid_gpu_mapping() */ WRITE_ONCE(vma->tile_invalidated, vma->tile_mask); =20 + if (!ret) + xe_tlb_inval_batch_wait(&batch); + return ret; } =20 diff --git a/drivers/gpu/drm/xe/xe_vm.h b/drivers/gpu/drm/xe/xe_vm.h index f849e369432b..62f4b6fec0bc 100644 --- a/drivers/gpu/drm/xe/xe_vm.h +++ b/drivers/gpu/drm/xe/xe_vm.h @@ -240,9 +240,6 @@ struct dma_fence *xe_vm_range_rebind(struct xe_vm *vm, struct dma_fence *xe_vm_range_unbind(struct xe_vm *vm, struct xe_svm_range *range); =20 -int xe_vm_range_tilemask_tlb_inval(struct xe_vm *vm, u64 start, - u64 end, u8 tile_mask); - int xe_vm_invalidate_vma(struct xe_vma *vma); =20 int xe_vm_validate_protected(struct xe_vm *vm); diff --git a/drivers/gpu/drm/xe/xe_vm_madvise.c b/drivers/gpu/drm/xe/xe_vm_= madvise.c index 95bf53cc29e3..02daf8a93044 100644 --- a/drivers/gpu/drm/xe/xe_vm_madvise.c +++ b/drivers/gpu/drm/xe/xe_vm_madvise.c @@ -12,6 +12,7 @@ #include "xe_pat.h" #include "xe_pt.h" #include "xe_svm.h" +#include "xe_tlb_inval.h" =20 struct xe_vmas_in_madvise_range { u64 addr; @@ -235,13 +236,20 @@ static u8 xe_zap_ptes_in_madvise_range(struct xe_vm *= vm, u64 start, u64 end) static int xe_vm_invalidate_madvise_range(struct xe_vm *vm, u64 start, u64= end) { u8 tile_mask =3D xe_zap_ptes_in_madvise_range(vm, start, end); + struct xe_tlb_inval_batch batch; + int err; =20 if (!tile_mask) return 0; =20 xe_device_wmb(vm->xe); =20 - return xe_vm_range_tilemask_tlb_inval(vm, start, end, tile_mask); + err =3D xe_tlb_inval_range_tilemask_submit(vm->xe, vm->usm.asid, start, e= nd, + tile_mask, &batch); + if (!err) + xe_tlb_inval_batch_wait(&batch); + + return err; } =20 static bool madvise_args_are_sane(struct xe_device *xe, const struct drm_x= e_madvise *args) diff --git a/drivers/gpu/drm/xe/xe_vm_types.h b/drivers/gpu/drm/xe/xe_vm_ty= pes.h index 1f6f7e30e751..de6544165cfa 100644 --- a/drivers/gpu/drm/xe/xe_vm_types.h +++ b/drivers/gpu/drm/xe/xe_vm_types.h @@ -18,6 +18,7 @@ #include "xe_device_types.h" #include "xe_pt_types.h" #include "xe_range_fence.h" +#include "xe_tlb_inval_types.h" #include "xe_userptr.h" =20 struct drm_pagemap; --=20 2.53.0 From nobody Tue Apr 7 00:50:11 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 11EBB37AA6D for ; Thu, 5 Mar 2026 09:40:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772703603; cv=none; b=AyfGR4U0B0AH7E2gQF3ZIR/yecVXLHTcvI6GC8fdcDfvPZbXptGIDBzzsS3oj3q9Dh6GIE4U0dEPOf4qgvkhPSA1eiNdkIY/g+xfHotgyv3L5I8haG7bl115dLKPh9IRRENusvC1Py8DvVXmYO+LFx/MMwfJ4Uq2OjlhSnGZU0I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772703603; c=relaxed/simple; bh=EkW2TPya80geCG27da7LAu868d+oGPxc6iUTc9tfTpc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=X1tqZmiusKYTn8+g52f02EG0+q44mB1PM72kHXYz7gzEkqrQ1NpMi8z+6gocvj0TcD0QpKC0XYKqviqFd7yMaOsFqAlOMgCnZixkCVTzm9d+oYpfNBu1ifFehqBWaYfFAR5s9xMkoSL1uKIJe0Q5dg05NudzdrH5o1XZRFeNTG4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=pass smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=gkA5t/NU; arc=none smtp.client-ip=198.175.65.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="gkA5t/NU" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1772703601; x=1804239601; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=EkW2TPya80geCG27da7LAu868d+oGPxc6iUTc9tfTpc=; b=gkA5t/NUlABWk7kGkY6ME5H15/3LtDEyK4PoSSN4IUn5woQ4lpY+3mGX +RA5HH5OzrIi2FPAf4PFPmG2n+DShze1DMOuhX8a6fBbOV1MErHlGBJa7 Kt1pUQOthXC5uWcXEC3ffH4M9p4xaQLy1NJWR4wDXhtE9ryi9ZR7qzshA 6ZwBFjnq6O2sIRgyIps/8jyfCyvJyDlqiUwgJ3wzJAaqxJpNiEb7Fags3 PK2ijjDK8x4y1iby8m14E5hymRD0yrCzuA+XXX7egYSistfctAUpijT/P Y/t3+iqX0L8kK6UaRCc7n7oX/tCOJQ7Qg4ytA08yMvAwMgcb7065O3VYE Q==; X-CSE-ConnectionGUID: EryDykwwTjeR8gTB/FoWrA== X-CSE-MsgGUID: ppTI5TLIQQ+F30H5NLsYcw== X-IronPort-AV: E=McAfee;i="6800,10657,11719"; a="84871005" X-IronPort-AV: E=Sophos;i="6.23,102,1770624000"; d="scan'208";a="84871005" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by orvoesa105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Mar 2026 01:40:01 -0800 X-CSE-ConnectionGUID: O2xpvH8IT1GZu80v/B3Qmg== X-CSE-MsgGUID: aW/HBbk8RYOpVy0TQlk+eg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.23,102,1770624000"; d="scan'208";a="214685027" Received: from vpanait-mobl.ger.corp.intel.com (HELO fedora) ([10.245.244.71]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Mar 2026 01:39:57 -0800 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-xe@lists.freedesktop.org Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , Matthew Brost , =?UTF-8?q?Christian=20K=C3=B6nig?= , dri-devel@lists.freedesktop.org, Jason Gunthorpe , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Simona Vetter , Dave Airlie , Alistair Popple , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 4/4] drm/xe/userptr: Defer Waiting for TLB invalidation to the second pass if possible Date: Thu, 5 Mar 2026 10:39:09 +0100 Message-ID: <20260305093909.43623-5-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260305093909.43623-1-thomas.hellstrom@linux.intel.com> References: <20260305093909.43623-1-thomas.hellstrom@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Now that the two-pass notifier flow uses xe_vma_userptr_do_inval() for the fence-wait + TLB-invalidate work, extend it to support a further deferred TLB wait: - xe_vma_userptr_do_inval(): when the embedded finish handle is free, submit the TLB invalidation asynchronously (xe_vm_invalidate_vma_submit) and return &userptr->finish so the mmu_notifier core schedules a third pass. When the handle is occupied by a concurrent invalidation, fall back to the synchronous xe_vm_invalidate_vma() path. - xe_vma_userptr_complete_tlb_inval(): new helper called from invalidate_finish when tlb_inval_submitted is set. Waits for the previously submitted batch and unmaps the gpusvm pages. xe_vma_userptr_invalidate_finish() dispatches between the two helpers via tlb_inval_submitted, making the three possible flows explicit: pass1 (fences pending) -> invalidate_finish -> do_inval (sync TLB) pass1 (fences done) -> do_inval -> invalidate_finish -> complete_tlb_inval (deferred TLB) pass1 (finish occupied) -> do_inval (sync TLB, inline) In multi-GPU scenarios this allows TLB flushes to be submitted on all GPUs in one pass before any of them are waited on. Also adds xe_vm_invalidate_vma_submit() which submits the TLB range invalidation without blocking, populating a xe_tlb_inval_batch that the caller waits on separately. v3: - Add locking asserts and notifier state asserts (Matt Brost) - Update the locking documentation of the notifier state members (Matt Brost) - Remove unrelated code formatting changes (Matt Brost) Assisted-by: GitHub Copilot:claude-sonnet-4.6 Signed-off-by: Thomas Hellstr=C3=B6m Reviewed-by: Matthew Brost --- drivers/gpu/drm/xe/xe_userptr.c | 63 ++++++++++++++++++++++++++++----- drivers/gpu/drm/xe/xe_userptr.h | 17 +++++++++ drivers/gpu/drm/xe/xe_vm.c | 38 +++++++++++++++----- drivers/gpu/drm/xe/xe_vm.h | 2 ++ 4 files changed, 104 insertions(+), 16 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_userptr.c b/drivers/gpu/drm/xe/xe_userpt= r.c index 37032b8125a6..6761005c0b90 100644 --- a/drivers/gpu/drm/xe/xe_userptr.c +++ b/drivers/gpu/drm/xe/xe_userptr.c @@ -8,6 +8,7 @@ =20 #include =20 +#include "xe_tlb_inval.h" #include "xe_trace_bo.h" =20 static void xe_userptr_assert_in_notifier(struct xe_vm *vm) @@ -81,8 +82,8 @@ int xe_vma_userptr_pin_pages(struct xe_userptr_vma *uvma) &ctx); } =20 -static void xe_vma_userptr_do_inval(struct xe_vm *vm, struct xe_userptr_vm= a *uvma, - bool is_deferred) +static struct mmu_interval_notifier_finish * +xe_vma_userptr_do_inval(struct xe_vm *vm, struct xe_userptr_vma *uvma, boo= l is_deferred) { struct xe_userptr *userptr =3D &uvma->userptr; struct xe_vma *vma =3D &uvma->vma; @@ -93,6 +94,8 @@ static void xe_vma_userptr_do_inval(struct xe_vm *vm, str= uct xe_userptr_vma *uvm long err; =20 xe_userptr_assert_in_notifier(vm); + if (is_deferred) + xe_assert(vm->xe, userptr->finish_inuse && !userptr->tlb_inval_submitted= ); =20 err =3D dma_resv_wait_timeout(xe_vm_resv(vm), DMA_RESV_USAGE_BOOKKEEP, @@ -100,6 +103,19 @@ static void xe_vma_userptr_do_inval(struct xe_vm *vm, = struct xe_userptr_vma *uvm XE_WARN_ON(err <=3D 0); =20 if (xe_vm_in_fault_mode(vm) && userptr->initial_bind) { + if (!userptr->finish_inuse) { + /* + * Defer the TLB wait to an extra pass so the caller + * can pipeline TLB flushes across GPUs before waiting + * on any of them. + */ + xe_assert(vm->xe, !userptr->tlb_inval_submitted); + userptr->finish_inuse =3D true; + userptr->tlb_inval_submitted =3D true; + err =3D xe_vm_invalidate_vma_submit(vma, &userptr->inval_batch); + XE_WARN_ON(err); + return &userptr->finish; + } err =3D xe_vm_invalidate_vma(vma); XE_WARN_ON(err); } @@ -108,6 +124,28 @@ static void xe_vma_userptr_do_inval(struct xe_vm *vm, = struct xe_userptr_vma *uvm userptr->finish_inuse =3D false; drm_gpusvm_unmap_pages(&vm->svm.gpusvm, &uvma->userptr.pages, xe_vma_size(vma) >> PAGE_SHIFT, &ctx); + return NULL; +} + +static void +xe_vma_userptr_complete_tlb_inval(struct xe_vm *vm, struct xe_userptr_vma = *uvma) +{ + struct xe_userptr *userptr =3D &uvma->userptr; + struct xe_vma *vma =3D &uvma->vma; + struct drm_gpusvm_ctx ctx =3D { + .in_notifier =3D true, + .read_only =3D xe_vma_read_only(vma), + }; + + xe_userptr_assert_in_notifier(vm); + xe_assert(vm->xe, userptr->finish_inuse); + xe_assert(vm->xe, userptr->tlb_inval_submitted); + + xe_tlb_inval_batch_wait(&userptr->inval_batch); + userptr->tlb_inval_submitted =3D false; + userptr->finish_inuse =3D false; + drm_gpusvm_unmap_pages(&vm->svm.gpusvm, &uvma->userptr.pages, + xe_vma_size(vma) >> PAGE_SHIFT, &ctx); } =20 static struct mmu_interval_notifier_finish * @@ -153,11 +191,10 @@ xe_vma_userptr_invalidate_pass1(struct xe_vm *vm, str= uct xe_userptr_vma *uvma) * If it's already in use, or all fences are already signaled, * proceed directly to invalidation without deferring. */ - if (signaled || userptr->finish_inuse) { - xe_vma_userptr_do_inval(vm, uvma, false); - return NULL; - } + if (signaled || userptr->finish_inuse) + return xe_vma_userptr_do_inval(vm, uvma, false); =20 + /* Defer: the notifier core will call invalidate_finish once done. */ userptr->finish_inuse =3D true; =20 return &userptr->finish; @@ -205,7 +242,15 @@ static void xe_vma_userptr_invalidate_finish(struct mm= u_interval_notifier_finish xe_vma_start(vma), xe_vma_size(vma)); =20 down_write(&vm->svm.gpusvm.notifier_lock); - xe_vma_userptr_do_inval(vm, uvma, true); + /* + * If a TLB invalidation was previously submitted (deferred from the + * synchronous pass1 fallback), wait for it and unmap pages. + * Otherwise, fences have now completed: invalidate the TLB and unmap. + */ + if (uvma->userptr.tlb_inval_submitted) + xe_vma_userptr_complete_tlb_inval(vm, uvma); + else + xe_vma_userptr_do_inval(vm, uvma, true); up_write(&vm->svm.gpusvm.notifier_lock); trace_xe_vma_userptr_invalidate_complete(vma); } @@ -243,7 +288,9 @@ void xe_vma_userptr_force_invalidate(struct xe_userptr_= vma *uvma) =20 finish =3D xe_vma_userptr_invalidate_pass1(vm, uvma); if (finish) - xe_vma_userptr_do_inval(vm, uvma, true); + finish =3D xe_vma_userptr_do_inval(vm, uvma, true); + if (finish) + xe_vma_userptr_complete_tlb_inval(vm, uvma); } #endif =20 diff --git a/drivers/gpu/drm/xe/xe_userptr.h b/drivers/gpu/drm/xe/xe_userpt= r.h index e1830c2f5fd2..2a3cd1b5efbb 100644 --- a/drivers/gpu/drm/xe/xe_userptr.h +++ b/drivers/gpu/drm/xe/xe_userptr.h @@ -14,6 +14,8 @@ =20 #include =20 +#include "xe_tlb_inval_types.h" + struct xe_vm; struct xe_vma; struct xe_userptr_vma; @@ -63,12 +65,27 @@ struct xe_userptr { * alternatively by the same lock in read mode *and* the vm resv held. */ struct mmu_interval_notifier_finish finish; + /** + * @inval_batch: TLB invalidation batch for deferred completion. + * Stores an in-flight TLB invalidation submitted during a two-pass + * notifier so the wait can be deferred to a subsequent pass, allowing + * multiple GPUs to be signalled before any of them are waited on. + * Protected using the same locking as @finish. + */ + struct xe_tlb_inval_batch inval_batch; /** * @finish_inuse: Whether @finish is currently in use by an in-progress * two-pass invalidation. * Protected using the same locking as @finish. */ bool finish_inuse; + /** + * @tlb_inval_submitted: Whether a TLB invalidation has been submitted + * via @inval_batch and is pending completion. When set, the next pass + * must call xe_tlb_inval_batch_wait() before reusing @inval_batch. + * Protected using the same locking as @finish. + */ + bool tlb_inval_submitted; /** * @initial_bind: user pointer has been bound at least once. * write: vm->svm.gpusvm.notifier_lock in read mode and vm->resv held. diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c index a3c2e8cefec7..fdad9329dfb4 100644 --- a/drivers/gpu/drm/xe/xe_vm.c +++ b/drivers/gpu/drm/xe/xe_vm.c @@ -3967,20 +3967,23 @@ void xe_vm_unlock(struct xe_vm *vm) } =20 /** - * xe_vm_invalidate_vma - invalidate GPU mappings for VMA without a lock + * xe_vm_invalidate_vma_submit - Submit a job to invalidate GPU mappings f= or + * VMA. * @vma: VMA to invalidate + * @batch: TLB invalidation batch to populate; caller must later call + * xe_tlb_inval_batch_wait() on it to wait for completion * * Walks a list of page tables leaves which it memset the entries owned by= this - * VMA to zero, invalidates the TLBs, and block until TLBs invalidation is - * complete. + * VMA to zero, invalidates the TLBs, but doesn't block waiting for TLB fl= ush + * to complete, but instead populates @batch which can be waited on using + * xe_tlb_inval_batch_wait(). * * Returns 0 for success, negative error code otherwise. */ -int xe_vm_invalidate_vma(struct xe_vma *vma) +int xe_vm_invalidate_vma_submit(struct xe_vma *vma, struct xe_tlb_inval_ba= tch *batch) { struct xe_device *xe =3D xe_vma_vm(vma)->xe; struct xe_vm *vm =3D xe_vma_vm(vma); - struct xe_tlb_inval_batch batch; struct xe_tile *tile; u8 tile_mask =3D 0; int ret =3D 0; @@ -4023,14 +4026,33 @@ int xe_vm_invalidate_vma(struct xe_vma *vma) =20 ret =3D xe_tlb_inval_range_tilemask_submit(xe, xe_vma_vm(vma)->usm.asid, xe_vma_start(vma), xe_vma_end(vma), - tile_mask, &batch); + tile_mask, batch); =20 /* WRITE_ONCE pairs with READ_ONCE in xe_vm_has_valid_gpu_mapping() */ WRITE_ONCE(vma->tile_invalidated, vma->tile_mask); + return ret; +} + +/** + * xe_vm_invalidate_vma - invalidate GPU mappings for VMA without a lock + * @vma: VMA to invalidate + * + * Walks a list of page tables leaves which it memset the entries owned by= this + * VMA to zero, invalidates the TLBs, and block until TLBs invalidation is + * complete. + * + * Returns 0 for success, negative error code otherwise. + */ +int xe_vm_invalidate_vma(struct xe_vma *vma) +{ + struct xe_tlb_inval_batch batch; + int ret; =20 - if (!ret) - xe_tlb_inval_batch_wait(&batch); + ret =3D xe_vm_invalidate_vma_submit(vma, &batch); + if (ret) + return ret; =20 + xe_tlb_inval_batch_wait(&batch); return ret; } =20 diff --git a/drivers/gpu/drm/xe/xe_vm.h b/drivers/gpu/drm/xe/xe_vm.h index 62f4b6fec0bc..0bc7ed23eeae 100644 --- a/drivers/gpu/drm/xe/xe_vm.h +++ b/drivers/gpu/drm/xe/xe_vm.h @@ -242,6 +242,8 @@ struct dma_fence *xe_vm_range_unbind(struct xe_vm *vm, =20 int xe_vm_invalidate_vma(struct xe_vma *vma); =20 +int xe_vm_invalidate_vma_submit(struct xe_vma *vma, struct xe_tlb_inval_ba= tch *batch); + int xe_vm_validate_protected(struct xe_vm *vm); =20 static inline void xe_vm_queue_rebind_worker(struct xe_vm *vm) --=20 2.53.0