From nobody Sun Oct 5 03:35:59 2025 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 293F7277C96 for ; Sat, 9 Aug 2025 13:52:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754747535; cv=none; b=JDDOj5j5RfD9T9EM0iJyK3/WF76aKesVHeFXn4AZTD65sx07GQL5hPWEvOWPbVhsc0S8vIfnAmaiIqN76HPtMcdyeqH9WWvr6CYZcoZaZc+WqwwmbJ9pu8SSNpJwj+59IJCejH8Wv6p4fQwUjGFVW1HvE1onvVrNC4KAQRy8xoY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754747535; c=relaxed/simple; bh=I0S46TL3ZGNQAM5NlfugwXaH73kKnXUT2H7X0Wkzjqo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=T4zAg+H2wuY+JCsweP60Z0DlUZGVFrBR1OHmlrFkE6MyN1xxy0lNvj1Fm+IAVtq/C1ETuze+H80pPNnbOK/4+dKFqfGqc+429NeoTzvydTLWsUra7Lopc1pDwaHoknvowZTZbhhv82C+On9X9KE+OfR4vxSqQhWfTeali9/LAHM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=gQKo7GDR; arc=none smtp.client-ip=198.175.65.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="gQKo7GDR" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1754747534; x=1786283534; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=I0S46TL3ZGNQAM5NlfugwXaH73kKnXUT2H7X0Wkzjqo=; b=gQKo7GDRXdva3SdexVUMK4W43rfXtimtLTlFaq2TEEwImWAA0bS1n2vq eJSj3T8A/MVxC953uDhIQeQXpaO3hUhEBg4JC2LAI1yzOUAx/z8xZHVEG EnCGBD5dNCHiK+XKgUtGdNLPWFCO38lwTB3hRqQc71QIxch86D2qcFk+5 w1Bfj98IiPTJfnVEkaLg8vSThGja9n7VrJCLY7zgEGz50IHdO6fOPsLqs Ew6kqUQ469uLb4lsvEmP1meFoguwmoSj6ipw460ipTIxGrmVpE7rumjtd KIUB7y7AbtMD66rJI0tz9eav/yiN73OWXTJNNmV9DPCaCjNvLGDpMRTRm Q==; X-CSE-ConnectionGUID: yxxtr2GyQfSwCALL151SPw== X-CSE-MsgGUID: DCsQwJEAQd+HK+qaE5CQog== X-IronPort-AV: E=McAfee;i="6800,10657,11515"; a="68153546" X-IronPort-AV: E=Sophos;i="6.17,278,1747724400"; d="scan'208";a="68153546" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by orvoesa105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2025 06:52:14 -0700 X-CSE-ConnectionGUID: q0EFfHMxTtm9eDHUSNIpJA== X-CSE-MsgGUID: Qrzeux62Rea+AK85jOp5yg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.17,278,1747724400"; d="scan'208";a="165903744" Received: from smoticic-mobl1.ger.corp.intel.com (HELO fedora) ([10.245.244.28]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2025 06:52:11 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-xe@lists.freedesktop.org Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , Jason Gunthorpe , Andrew Morton , Simona Vetter , Dave Airlie , dri-devel@lists.freedesktop.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Matthew Brost , =?UTF-8?q?Christian=20K=C3=B6nig?= Subject: [RFC PATCH 1/6] mm/mmu_notifier: Allow multiple struct mmu_interval_notifier passes Date: Sat, 9 Aug 2025 15:51:32 +0200 Message-ID: <20250809135137.259427-2-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250809135137.259427-1-thomas.hellstrom@linux.intel.com> References: <20250809135137.259427-1-thomas.hellstrom@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable GPU use-cases for mmu_interval_notifiers with hmm often involve starting a gpu operation and then waiting for it to complete. These operations are typically context preemption or TLB flushing. With single-pass notifiers per GPU this doesn't scale in multi-gpu scenarios. In those scenarios we'd want to first start preemption- or TLB flushing on all GPUs and as a second pass wait for them to complete on all gpus. One can do this on per-driver basis multiplexing per-driver notifiers but that would mean sharing the notifier "user" lock across all GPUs and that doesn't scale well either, so adding support for multi-pass in the core appears like the right choice. Implement multi-pass capability in the mmu_interval_notifier. Use a linked list for the additional passes to minimize the impact for use-cases that don't need the multi-pass functionality. Cc: Jason Gunthorpe Cc: Andrew Morton Cc: Simona Vetter Cc: Dave Airlie Cc: Cc: Cc: Signed-off-by: Thomas Hellstr=C3=B6m --- include/linux/mmu_notifier.h | 30 ++++++++++++++++ mm/mmu_notifier.c | 67 +++++++++++++++++++++++++++++++----- 2 files changed, 88 insertions(+), 9 deletions(-) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index d1094c2d5fb6..1107a8eafd8a 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -233,6 +233,32 @@ struct mmu_notifier { unsigned int users; }; =20 +/** + * struct mmu_interval_notifier_pass - mmu_interval_notifier multi-pass ab= straction + * @link: List link for the notifiers pending pass list + * + * Allocate, typically using GFP_NOWAIT in the interval notifier's first p= ass. + * If allocation fails (which is not unlikely under memory pressure), fall= back + * to single-pass operation. + */ +struct mmu_interval_notifier_pass { + struct list_head link; + /** + * @pass: Driver callback for additionall pass. + * @additional_pass: Pointer to the mmu_interval_notifier_pass structure. + * @range: The mmu_notifier_range. + * @cur_seq: The current sequence set by the first pass. + * + * Return: Either a pointer to a valid mmu_interval_notifier_pass for + * another pass to be called, or %NULL if processing is complete for this + * notifier. There is no error reporting mechanism for additional passes. + */ + struct mmu_interval_notifier_pass * + (*pass) (struct mmu_interval_notifier_pass *additional_pass, + const struct mmu_notifier_range *range, + unsigned long cur_seq); +}; + /** * struct mmu_interval_notifier_ops * @invalidate: Upon return the caller must stop using any SPTEs within th= is @@ -243,6 +269,10 @@ struct mmu_interval_notifier_ops { bool (*invalidate)(struct mmu_interval_notifier *interval_sub, const struct mmu_notifier_range *range, unsigned long cur_seq); + bool (*invalidate_multipass)(struct mmu_interval_notifier *interval_sub, + const struct mmu_notifier_range *range, + unsigned long cur_seq, + struct mmu_interval_notifier_pass **pass); }; =20 struct mmu_interval_notifier { diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index 8e0125dc0522..dd6af87db103 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -260,6 +260,22 @@ mmu_interval_read_begin(struct mmu_interval_notifier *= interval_sub) } EXPORT_SYMBOL_GPL(mmu_interval_read_begin); =20 +static void mn_itree_additional_passes(struct list_head *additional_passes, + const struct mmu_notifier_range *range, + unsigned long cur_seq) +{ + struct mmu_interval_notifier_pass *p, *next; + + while (!list_empty(additional_passes)) { + list_for_each_entry_safe(p, next, additional_passes, link) { + list_del_init(&p->link); + p =3D p->pass(p, range, cur_seq); + if (p) + list_add_tail(&p->link, additional_passes); + } + } +} + static void mn_itree_release(struct mmu_notifier_subscriptions *subscripti= ons, struct mm_struct *mm) { @@ -272,17 +288,32 @@ static void mn_itree_release(struct mmu_notifier_subs= criptions *subscriptions, }; struct mmu_interval_notifier *interval_sub; unsigned long cur_seq; + LIST_HEAD(additional_passes); bool ret; =20 for (interval_sub =3D mn_itree_inv_start_range(subscriptions, &range, &cur_seq); interval_sub; interval_sub =3D mn_itree_inv_next(interval_sub, &range)) { - ret =3D interval_sub->ops->invalidate(interval_sub, &range, - cur_seq); + if (interval_sub->ops->invalidate_multipass) { + struct mmu_interval_notifier_pass *second =3D NULL; + + ret =3D interval_sub->ops->invalidate_multipass(interval_sub, + &range, + cur_seq, + &second); + if (ret && second) + list_add_tail(&second->link, &additional_passes); + + } else { + ret =3D interval_sub->ops->invalidate(interval_sub, + &range, + cur_seq); + } WARN_ON(!ret); } =20 + mn_itree_additional_passes(&additional_passes, &range, cur_seq); mn_itree_inv_end(subscriptions); } =20 @@ -431,6 +462,8 @@ static int mn_itree_invalidate(struct mmu_notifier_subs= criptions *subscriptions, { struct mmu_interval_notifier *interval_sub; unsigned long cur_seq; + LIST_HEAD(additional_passes); + int err =3D 0; =20 for (interval_sub =3D mn_itree_inv_start_range(subscriptions, range, &cur_seq); @@ -438,23 +471,39 @@ static int mn_itree_invalidate(struct mmu_notifier_su= bscriptions *subscriptions, interval_sub =3D mn_itree_inv_next(interval_sub, range)) { bool ret; =20 - ret =3D interval_sub->ops->invalidate(interval_sub, range, - cur_seq); + if (interval_sub->ops->invalidate_multipass) { + struct mmu_interval_notifier_pass *second =3D NULL; + + ret =3D interval_sub->ops->invalidate_multipass(interval_sub, + range, + cur_seq, + &second); + if (ret && second) + list_add_tail(&second->link, &additional_passes); + + } else { + ret =3D interval_sub->ops->invalidate(interval_sub, + range, + cur_seq); + } if (!ret) { if (WARN_ON(mmu_notifier_range_blockable(range))) continue; - goto out_would_block; + err =3D -EAGAIN; + break; } } - return 0; =20 -out_would_block: + mn_itree_additional_passes(&additional_passes, range, cur_seq); + /* * On -EAGAIN the non-blocking caller is not allowed to call * invalidate_range_end() */ - mn_itree_inv_end(subscriptions); - return -EAGAIN; + if (err) + mn_itree_inv_end(subscriptions); + + return err; } =20 static int mn_hlist_invalidate_range_start( --=20 2.50.1 From nobody Sun Oct 5 03:35:59 2025 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C510029DB81 for ; Sat, 9 Aug 2025 13:52:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754747538; cv=none; b=bkcIYtqfKDjp7AaXQ5Cm3RPs16pPkDxyU1rxQ+b2SiWAKOzjQ23LgeY5dgJw5ukYWMoXbWu4fv2OYNzCR+c33Jqdv2kaUDwM8fBcjDpTuuNDCjUTFh19V4kNFGfvGYe6vU6G+sZUlv9MjnYClJiu5p5EBdIERE7yGwPO/hQqp74= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754747538; c=relaxed/simple; bh=M1dPRbREHCZNm03J/0kaw2ckl1FuinNQkvsu+9otm4E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dyoZe5W2s2jBBxwQw4ckZOqg4djCXuFzbzT2TLyZZruL4EBsWYmZAbDBT+fWh3eGmvpBtD9PR6DvxF+JEL9n5XSl2t9kEX3ROJgOkf3iMtAM2TGN5nVGEUyVLy4UkW0QYTkmOTJkQxXl97qFIVLocg+3dJqcp6rkeH7CrLkK1fE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=KHyazKfH; arc=none smtp.client-ip=198.175.65.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="KHyazKfH" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1754747537; x=1786283537; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=M1dPRbREHCZNm03J/0kaw2ckl1FuinNQkvsu+9otm4E=; b=KHyazKfHOVdFBagKMI6boSzRKAElOs5EtZ57CpK90ETS2B+9pRx9t7UK DwBmzeMozNc1tzUYhUhj7nBSLcNqYL5khbLKxe5OsVgEhoJcYB8riOxtp sOqkGRgPhr9d7ReHshZQJwXYSI4ZVqgZGhqmfbycYlx6Lzq2IxUFQk7mn w1Y81pxHW3WgAY9+Ro/FGLC9ughpaHw5FKkW+2XCV3i5Ic/P/DSPC/RPQ VFPpKKa2+IhZjlGMpNy6cy3+aleNrPww2OWTPwG+nYTz5tOgsz4XcEN9z al1JvWQUXWPK9PhsE8qiAc1JDtfKZFvpE92S41/qWfXKww6I90Qr3T3su w==; X-CSE-ConnectionGUID: zxytRrHmRtm9r45FIoEfbA== X-CSE-MsgGUID: X3lsUGLwSGyRRwe/os6CbQ== X-IronPort-AV: E=McAfee;i="6800,10657,11515"; a="68153555" X-IronPort-AV: E=Sophos;i="6.17,278,1747724400"; d="scan'208";a="68153555" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by orvoesa105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2025 06:52:16 -0700 X-CSE-ConnectionGUID: wW5KpTcnRz+LewKSRZMlhQ== X-CSE-MsgGUID: axquRza3SkGznZ/c7UT2LA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.17,278,1747724400"; d="scan'208";a="165903751" Received: from smoticic-mobl1.ger.corp.intel.com (HELO fedora) ([10.245.244.28]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2025 06:52:14 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-xe@lists.freedesktop.org Cc: Matthew Brost , =?UTF-8?q?Christian=20K=C3=B6nig?= , dri-devel@lists.freedesktop.org, Jason Gunthorpe , Andrew Morton , Simona Vetter , Dave Airlie , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 2/6] drm/gpusvm: Update GPU SVM / Xe to twopass MMU notifier Date: Sat, 9 Aug 2025 15:51:33 +0200 Message-ID: <20250809135137.259427-3-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250809135137.259427-1-thomas.hellstrom@linux.intel.com> References: <20250809135137.259427-1-thomas.hellstrom@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Matthew Brost Update GPU SVM and Xe to use two-pass MMU notifiers, enabling pipelined TLB invalidations across VMs or multiple devices. The driver-side (Xe) implementation is not yet implemented. Signed-off-by: Matthew Brost --- drivers/gpu/drm/drm_gpusvm.c | 18 +++++++++++------- drivers/gpu/drm/xe/xe_svm.c | 9 +++++---- include/drm/drm_gpusvm.h | 11 +++++++---- 3 files changed, 23 insertions(+), 15 deletions(-) diff --git a/drivers/gpu/drm/drm_gpusvm.c b/drivers/gpu/drm/drm_gpusvm.c index 661306da6b2d..92dc7d2bd6cf 100644 --- a/drivers/gpu/drm/drm_gpusvm.c +++ b/drivers/gpu/drm/drm_gpusvm.c @@ -374,10 +374,13 @@ notifier_iter_first(struct rb_root_cached *root, unsi= gned long start, (notifier__) =3D (next__), (next__) =3D __drm_gpusvm_notifier_next(n= otifier__)) =20 /** - * drm_gpusvm_notifier_invalidate() - Invalidate a GPU SVM notifier. + * drm_gpusvm_notifier_invalidate_twopass() - Invalidate a GPU SVM notifie, + * fist pass. + * * @mni: Pointer to the mmu_interval_notifier structure. * @mmu_range: Pointer to the mmu_notifier_range structure. * @cur_seq: Current sequence number. + * @pass: First pass of MMU notifier * * This function serves as a generic MMU notifier for GPU SVM. It sets the= MMU * notifier sequence number and calls the driver invalidate vfunc under @@ -386,9 +389,10 @@ notifier_iter_first(struct rb_root_cached *root, unsig= ned long start, * Return: true if the operation succeeds, false otherwise. */ static bool -drm_gpusvm_notifier_invalidate(struct mmu_interval_notifier *mni, - const struct mmu_notifier_range *mmu_range, - unsigned long cur_seq) +drm_gpusvm_notifier_invalidate_twopass(struct mmu_interval_notifier *mni, + const struct mmu_notifier_range *mmu_range, + unsigned long cur_seq, + struct mmu_interval_notifier_pass **pass) { struct drm_gpusvm_notifier *notifier =3D container_of(mni, typeof(*notifier), notifier); @@ -399,7 +403,7 @@ drm_gpusvm_notifier_invalidate(struct mmu_interval_noti= fier *mni, =20 down_write(&gpusvm->notifier_lock); mmu_interval_set_seq(mni, cur_seq); - gpusvm->ops->invalidate(gpusvm, notifier, mmu_range); + gpusvm->ops->invalidate_twopass(gpusvm, notifier, mmu_range, pass); up_write(&gpusvm->notifier_lock); =20 return true; @@ -409,7 +413,7 @@ drm_gpusvm_notifier_invalidate(struct mmu_interval_noti= fier *mni, * drm_gpusvm_notifier_ops - MMU interval notifier operations for GPU SVM */ static const struct mmu_interval_notifier_ops drm_gpusvm_notifier_ops =3D { - .invalidate =3D drm_gpusvm_notifier_invalidate, + .invalidate_twopass =3D drm_gpusvm_notifier_invalidate_twopass, }; =20 /** @@ -440,7 +444,7 @@ int drm_gpusvm_init(struct drm_gpusvm *gpusvm, const struct drm_gpusvm_ops *ops, const unsigned long *chunk_sizes, int num_chunks) { - if (!ops->invalidate || !num_chunks) + if (!ops->invalidate_twopass || !num_chunks) return -EINVAL; =20 gpusvm->name =3D name; diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c index e35c6d4def20..23c5b363261c 100644 --- a/drivers/gpu/drm/xe/xe_svm.c +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -171,9 +171,10 @@ xe_svm_range_notifier_event_end(struct xe_vm *vm, stru= ct drm_gpusvm_range *r, mmu_range); } =20 -static void xe_svm_invalidate(struct drm_gpusvm *gpusvm, - struct drm_gpusvm_notifier *notifier, - const struct mmu_notifier_range *mmu_range) +static void xe_svm_invalidate_twopass(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier *notifier, + const struct mmu_notifier_range *mmu_range, + struct mmu_interval_notifier_pass **p) { struct xe_vm *vm =3D gpusvm_to_vm(gpusvm); struct xe_device *xe =3D vm->xe; @@ -553,7 +554,7 @@ static const struct drm_pagemap_devmem_ops dpagemap_dev= mem_ops =3D { static const struct drm_gpusvm_ops gpusvm_ops =3D { .range_alloc =3D xe_svm_range_alloc, .range_free =3D xe_svm_range_free, - .invalidate =3D xe_svm_invalidate, + .invalidate_twopass =3D xe_svm_invalidate_twopass, }; =20 static const unsigned long fault_chunk_sizes[] =3D { diff --git a/include/drm/drm_gpusvm.h b/include/drm/drm_gpusvm.h index 8d613e9b2690..8b5e159857fc 100644 --- a/include/drm/drm_gpusvm.h +++ b/include/drm/drm_gpusvm.h @@ -63,17 +63,20 @@ struct drm_gpusvm_ops { void (*range_free)(struct drm_gpusvm_range *range); =20 /** - * @invalidate: Invalidate GPU SVM notifier (required) + * @invalidate_twopass: Invalidate first pass GPU SVM notifier (required) * @gpusvm: Pointer to the GPU SVM * @notifier: Pointer to the GPU SVM notifier * @mmu_range: Pointer to the mmu_notifier_range structure + * @pass: Pass of MMU notifier, optionally populated driver side + * if a second pass of MMU notifier is desired * * Invalidate the GPU page tables. It can safely walk the notifier range * RB tree/list in this function. Called while holding the notifier lock. */ - void (*invalidate)(struct drm_gpusvm *gpusvm, - struct drm_gpusvm_notifier *notifier, - const struct mmu_notifier_range *mmu_range); + void (*invalidate_twopass)(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier *notifier, + const struct mmu_notifier_range *mmu_range, + struct mmu_interval_notifier_pass **pass); }; =20 /** --=20 2.50.1 From nobody Sun Oct 5 03:35:59 2025 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0BE2329E0F7 for ; Sat, 9 Aug 2025 13:52:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754747540; cv=none; b=EjjHWbl1Em+qtP3RwOmuI9shql5mFLojysVypz6UZXAP2nheYyV2rwPlKIxw6lmVPN6Q7iAau/IA0//dUd3m/xZQmZSWteXgCbHgJMTled8GNtWmla+g+lunK/YLdWfGpq7Ck5rW7hzC8LIjyHm4M41K5xB7rcZ7OymFeZHsMCI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754747540; c=relaxed/simple; bh=F1Va4a7oL3sStdI3GcwWNaFyPom4HkNAVRRrhM+iPZs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tNH8DXUc25PfdXYJik26rNJjYp8KIgMGKPp8FQ3FO4u0i/MSj4pUhWVQIeL7/E55hLanxBTdF9dQBw2FsE/Vriy1noY/L7Q1ufdKILDEMrb2ViRYAS0ZMXqK5f3KBUQd8EKKMYjIJZEQCtrf5jUeP/faPgN7Vn83Mk7EoXFTNpM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=ZKrliwjS; arc=none smtp.client-ip=198.175.65.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="ZKrliwjS" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1754747539; x=1786283539; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=F1Va4a7oL3sStdI3GcwWNaFyPom4HkNAVRRrhM+iPZs=; b=ZKrliwjSMqhQXrECsMFuVhRW05yTv0n1j62GSsC31hnzIPF83h6fiHzK weDaZ1/MdIsFxHVVEFlztTPmgJG3oWvAxLQ1zCEubugYbV9Ifi2eMqlWs 0TKsTw/9OmJmRD9nlM/aQo5B/Q7+VnjqXP8C3CPzgDMtYn+R9b8+dtvOW NsnP9EX2Sxa4L4lLBdCPSYilxVkGJxk6eyg/mfaXknlpQ7e/G5sEqECRa V3FWCWcJGxMfEC3/YSAul2qkqxQwQip3tHa9v/Z606nizFxlpzF08CBdZ vSGGnoYZx57Ls8Cz7wLdL65YVGUBXTbZo2QmojcwFJNyTqpUdzppS2X9E A==; X-CSE-ConnectionGUID: fXqnPmCcTvysuxu0YXJBVQ== X-CSE-MsgGUID: LMmHp4WmSquGGDtubSr9Kg== X-IronPort-AV: E=McAfee;i="6800,10657,11515"; a="68153563" X-IronPort-AV: E=Sophos;i="6.17,278,1747724400"; d="scan'208";a="68153563" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by orvoesa105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2025 06:52:19 -0700 X-CSE-ConnectionGUID: S6yn9jfERgeI3j+fOzlZuw== X-CSE-MsgGUID: jliW7cJOStiKkBK0r/YuOQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.17,278,1747724400"; d="scan'208";a="165903758" Received: from smoticic-mobl1.ger.corp.intel.com (HELO fedora) ([10.245.244.28]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2025 06:52:16 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-xe@lists.freedesktop.org Cc: Matthew Brost , =?UTF-8?q?Christian=20K=C3=B6nig?= , dri-devel@lists.freedesktop.org, Jason Gunthorpe , Andrew Morton , Simona Vetter , Dave Airlie , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 3/6] drm/gpusvm: Add drm_gpusvm_in_notifier_* helpers Date: Sat, 9 Aug 2025 15:51:34 +0200 Message-ID: <20250809135137.259427-4-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250809135137.259427-1-thomas.hellstrom@linux.intel.com> References: <20250809135137.259427-1-thomas.hellstrom@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Matthew Brost Abstract drm_gpusvm_in_notifier_lock/unlock with helpers. Intended usage is a client side 2nd pass of a MMU notifier. Signed-off-by: Matthew Brost --- include/drm/drm_gpusvm.h | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/include/drm/drm_gpusvm.h b/include/drm/drm_gpusvm.h index 8b5e159857fc..4bdbe10685cf 100644 --- a/include/drm/drm_gpusvm.h +++ b/include/drm/drm_gpusvm.h @@ -313,7 +313,7 @@ void drm_gpusvm_range_set_unmapped(struct drm_gpusvm_ra= nge *range, #endif =20 /** - * drm_gpusvm_notifier_lock() - Lock GPU SVM notifier + * drm_gpusvm_notifier_lock() - Lock GPU SVM notifier, client side * @gpusvm__: Pointer to the GPU SVM structure. * * Abstract client usage GPU SVM notifier lock, take lock @@ -322,7 +322,7 @@ void drm_gpusvm_range_set_unmapped(struct drm_gpusvm_ra= nge *range, down_read(&(gpusvm__)->notifier_lock) =20 /** - * drm_gpusvm_notifier_unlock() - Unlock GPU SVM notifier + * drm_gpusvm_notifier_unlock() - Unlock GPU SVM notifier, client side * @gpusvm__: Pointer to the GPU SVM structure. * * Abstract client usage GPU SVM notifier lock, drop lock @@ -330,6 +330,24 @@ void drm_gpusvm_range_set_unmapped(struct drm_gpusvm_r= ange *range, #define drm_gpusvm_notifier_unlock(gpusvm__) \ up_read(&(gpusvm__)->notifier_lock) =20 +/** + * drm_gpusvm_in_notifier_lock() - Lock GPU SVM notifier, in notifier + * @gpusvm__: Pointer to the GPU SVM structure. + * + * Abstract in notifier (2nd pass) usage GPU SVM notifier lock, take lock + */ +#define drm_gpusvm_in_notifier_lock(gpusvm__) \ + down_write(&(gpusvm__)->notifier_lock) + +/** + * drm_gpusvm_in_notifier_unlock() - Unlock GPU SVM notifier, in notifier + * @gpusvm__: Pointer to the GPU SVM structure. + * + * Abstract in notifier (2nd pass) GPU SVM notifier lock, drop lock + */ +#define drm_gpusvm_in_notifier_unlock(gpusvm__) \ + up_write(&(gpusvm__)->notifier_lock) + /** * drm_gpusvm_range_start() - GPU SVM range start address * @range: Pointer to the GPU SVM range --=20 2.50.1 From nobody Sun Oct 5 03:35:59 2025 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6F4D02BCF41 for ; Sat, 9 Aug 2025 13:52:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754747542; cv=none; b=WEm44OXro2R/3TPQCR/uRWGnxRfT+MIK+ijINznFTmT48XXECT5R6kvcaFGZE3fmc7Vv+wM4EpWNjECrSbpPhX+y1ybqh8yBtz5olaL+mbPBHg9QlQTn/Klv8IFh6oYSGqqjQW4HbnAzhp6BcdMp4gSOwOhbT1RqttL7BxrpGTI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754747542; c=relaxed/simple; bh=iX+aI4EV6aIpB48ENfwpl4TWhZJPqVgs0W+CzzXlU1w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=S5y3ssdwVfGm6FR1FsjD9x+o6263FR9T5SzwasQls2aym+Cx/g4CeOl0nNPiu8opEZh0wGjUsWC9/izeH5hj1zZrj5n5wbwp5puriq7s/GElfTUr0PoHbLb2iLp/PzhQ5/7uxqoAm0ZAVjJV3puKyWu5kQbsed2/lCWETtVwoyU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=kax2AiaM; arc=none smtp.client-ip=198.175.65.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="kax2AiaM" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1754747541; x=1786283541; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=iX+aI4EV6aIpB48ENfwpl4TWhZJPqVgs0W+CzzXlU1w=; b=kax2AiaMp09Abvbp8lxeJVf4phD/yBZ/+qQ8Il2ipjfAVYAymqBqujM4 c4VGkOSi9G+sCM35h80sZ2w7Si5MCMhlO2oIn9d79EwH3NMFVpw0Girq8 EPIGyzLljvtf5cHQuXSU85X+BPidX1EDbK5d9iOHZkZeCEVkWXfR52TVj oHb8xBU0rCR8I6UA5Qo929N8H8O4effvsaMseOy/cNK9SEkV1VXqvzmOn gG4KyEkKTwHAMYGuHKea9zcH4A8CBtMSBIMxg/mjx1ziABvXM2pFDeNVf wmQ7tAhPmtmsafvlspoPlaAs5+yjza/2ZIrZY9kP4xqFbU0FhuX2NHg4f Q==; X-CSE-ConnectionGUID: C79Lb2IKStafP23fDtkbWA== X-CSE-MsgGUID: uvZUah4VRFu6CpPBo5sW6A== X-IronPort-AV: E=McAfee;i="6800,10657,11515"; a="68153572" X-IronPort-AV: E=Sophos;i="6.17,278,1747724400"; d="scan'208";a="68153572" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by orvoesa105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2025 06:52:21 -0700 X-CSE-ConnectionGUID: gVHvHxq3SzW2byWIN5LLcw== X-CSE-MsgGUID: 7NQBfY0hQGygQt3TnI1X/g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.17,278,1747724400"; d="scan'208";a="165903763" Received: from smoticic-mobl1.ger.corp.intel.com (HELO fedora) ([10.245.244.28]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2025 06:52:19 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-xe@lists.freedesktop.org Cc: Matthew Brost , =?UTF-8?q?Christian=20K=C3=B6nig?= , dri-devel@lists.freedesktop.org, Jason Gunthorpe , Andrew Morton , Simona Vetter , Dave Airlie , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 4/6] drm/xe: Skip waiting on unarmed fences in xe_gt_tlb_invalidation_fence_wait Date: Sat, 9 Aug 2025 15:51:35 +0200 Message-ID: <20250809135137.259427-5-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250809135137.259427-1-thomas.hellstrom@linux.intel.com> References: <20250809135137.259427-1-thomas.hellstrom@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Matthew Brost Avoids unnecessary waits when the TLB invalidation fence has not been armed, simplifying caller logic in cases where the fence status is uncertain. Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_gt_tlb_invalidation.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/xe/xe_gt_tlb_invalidation.h b/drivers/gpu/drm/= xe/xe_gt_tlb_invalidation.h index f7f0f2eaf4b5..c6d4398d3429 100644 --- a/drivers/gpu/drm/xe/xe_gt_tlb_invalidation.h +++ b/drivers/gpu/drm/xe/xe_gt_tlb_invalidation.h @@ -34,7 +34,8 @@ void xe_gt_tlb_invalidation_fence_signal(struct xe_gt_tlb= _invalidation_fence *fe static inline void xe_gt_tlb_invalidation_fence_wait(struct xe_gt_tlb_invalidation_fence *fen= ce) { - dma_fence_wait(&fence->base, false); + if (fence->seqno) + dma_fence_wait(&fence->base, false); } =20 #endif /* _XE_GT_TLB_INVALIDATION_ */ --=20 2.50.1 From nobody Sun Oct 5 03:35:59 2025 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E46BF2BD001 for ; Sat, 9 Aug 2025 13:52:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754747545; cv=none; b=FXc0AkL2DGSxOHDJJbSOcty0JsMpccjTJlT5fBX5WSSKqfUXKV8JIgmov7xHkCsqvf9C2FxzerDD3v72sxv2k3GdLousd1OKEjQqITKJIO04arESPo7/z/i142zhv5Ypg1whNIbIOsRr6xQbKJ3t8nGo63q1I7pHtbic1k+4228= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754747545; c=relaxed/simple; bh=bt+QjLeCHx2Rrs4pAwueu1ZrAv3yQzvpcrOLeaDiagU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SnLPIjDuUtVURuu2qSQW0aWu2jODx/oLY5xybu2B1GueuE0zktMC1PJdW1QThtdhIY64rUZVVuu7LRjASWO3g351H9LxnZGoi11A9eag/8uzkpbTNUVNhQLwkPT52uF0PO4q1ICSK2uK7kAIgYCGxcvprztA4lrI+1ve3mDiySw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=BrZcuPEB; arc=none smtp.client-ip=198.175.65.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="BrZcuPEB" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1754747544; x=1786283544; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bt+QjLeCHx2Rrs4pAwueu1ZrAv3yQzvpcrOLeaDiagU=; b=BrZcuPEBSO+MoCrXGLfHSg+CXjNp1MLPZuMFQX+vclW/aZ99WPIO5/X2 NlY9WdOF7pQnRr+eZyM2PTjnQu5fcdbguqkKS1UQr/hvadIsMKTBGwdPV nD+Tbs8ZoHORASvdyvrE0AY7YVDmj1MTV3YChhBvSe0q5VHgtfOx0jTHY D7PvV5S3268PxlLZCOfOWprbdnLH9cKEzE1tt4BE6UpO2Q8S/DRXBEfgo HbDMWLyoYtCe+455amDpWdTSapBWIBWPqBIZCXgcxvXXhrhFmnlOiXIj2 hTti9HCyDwknY/bQb+mK00EYGqfsnql7a+OgFauGo20QnKU1AjZxUOo2z g==; X-CSE-ConnectionGUID: W4t88IFNTsGHKG/ZxKktYA== X-CSE-MsgGUID: XQu1XBJvRVa1KiqyGqfE0w== X-IronPort-AV: E=McAfee;i="6800,10657,11515"; a="68153585" X-IronPort-AV: E=Sophos;i="6.17,278,1747724400"; d="scan'208";a="68153585" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by orvoesa105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2025 06:52:24 -0700 X-CSE-ConnectionGUID: 6+wTSiSvTz+Kr2Xn8p5B7A== X-CSE-MsgGUID: UD2z8XlYQ1+G0MlHcfjemQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.17,278,1747724400"; d="scan'208";a="165903771" Received: from smoticic-mobl1.ger.corp.intel.com (HELO fedora) ([10.245.244.28]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2025 06:52:21 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-xe@lists.freedesktop.org Cc: Matthew Brost , =?UTF-8?q?Christian=20K=C3=B6nig?= , dri-devel@lists.freedesktop.org, Jason Gunthorpe , Andrew Morton , Simona Vetter , Dave Airlie , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 5/6] drm/xe: Add fences argument to xe_vm_range_tilemask_tlb_invalidation Date: Sat, 9 Aug 2025 15:51:36 +0200 Message-ID: <20250809135137.259427-6-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250809135137.259427-1-thomas.hellstrom@linux.intel.com> References: <20250809135137.259427-1-thomas.hellstrom@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Matthew Brost Introduce a fences argument to xe_vm_range_tilemask_tlb_invalidation, allowing callers to provide fences and defer waiting to a later point. Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_svm.c | 3 ++- drivers/gpu/drm/xe/xe_vm.c | 26 +++++++++++++++++--------- drivers/gpu/drm/xe/xe_vm.h | 6 ++++-- 3 files changed, 23 insertions(+), 12 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c index 23c5b363261c..82a598c8d56e 100644 --- a/drivers/gpu/drm/xe/xe_svm.c +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -226,7 +226,8 @@ static void xe_svm_invalidate_twopass(struct drm_gpusvm= *gpusvm, =20 xe_device_wmb(xe); =20 - err =3D xe_vm_range_tilemask_tlb_invalidation(vm, adj_start, adj_end, til= e_mask); + err =3D xe_vm_range_tilemask_tlb_invalidation(vm, NULL, adj_start, + adj_end, tile_mask); WARN_ON_ONCE(err); =20 range_notifier_event_end: diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c index 148a2425006f..52242fac6969 100644 --- a/drivers/gpu/drm/xe/xe_vm.c +++ b/drivers/gpu/drm/xe/xe_vm.c @@ -3846,6 +3846,7 @@ void xe_vm_unlock(struct xe_vm *vm) * xe_vm_range_tilemask_tlb_invalidation - Issue a TLB invalidation on thi= s tilemask for an * address range * @vm: The VM + * @fences: Caller provided fences, caller owns waiting if non-NULL * @start: start address * @end: end address * @tile_mask: mask for which gt's issue tlb invalidation @@ -3854,10 +3855,12 @@ void xe_vm_unlock(struct xe_vm *vm) * * Returns 0 for success, negative error code otherwise. */ -int xe_vm_range_tilemask_tlb_invalidation(struct xe_vm *vm, u64 start, - u64 end, u8 tile_mask) +int xe_vm_range_tilemask_tlb_invalidation(struct xe_vm *vm, + struct xe_gt_tlb_invalidation_fence *fences, + u64 start, u64 end, u8 tile_mask) { struct xe_gt_tlb_invalidation_fence fence[XE_MAX_TILES_PER_DEVICE * XE_MA= X_GT_PER_TILE]; + struct xe_gt_tlb_invalidation_fence *__fence =3D fences ?: fence; struct xe_tile *tile; u32 fence_id =3D 0; u8 id; @@ -3869,37 +3872,41 @@ int xe_vm_range_tilemask_tlb_invalidation(struct xe= _vm *vm, u64 start, for_each_tile(tile, vm->xe, id) { if (tile_mask & BIT(id)) { xe_gt_tlb_invalidation_fence_init(tile->primary_gt, - &fence[fence_id], true); + __fence, true); =20 err =3D xe_gt_tlb_invalidation_range(tile->primary_gt, - &fence[fence_id], + __fence, start, end, vm->usm.asid); if (err) goto wait; ++fence_id; + ++__fence; =20 if (!tile->media_gt) continue; =20 xe_gt_tlb_invalidation_fence_init(tile->media_gt, - &fence[fence_id], true); + __fence, true); =20 err =3D xe_gt_tlb_invalidation_range(tile->media_gt, - &fence[fence_id], + __fence, start, end, vm->usm.asid); if (err) goto wait; ++fence_id; + ++__fence; } } =20 wait: - for (id =3D 0; id < fence_id; ++id) - xe_gt_tlb_invalidation_fence_wait(&fence[id]); + if (!fences) { + for (id =3D 0; id < fence_id; ++id) + xe_gt_tlb_invalidation_fence_wait(&fence[id]); + } =20 return err; } @@ -3958,7 +3965,8 @@ int xe_vm_invalidate_vma(struct xe_vma *vma) =20 xe_device_wmb(xe); =20 - ret =3D xe_vm_range_tilemask_tlb_invalidation(xe_vma_vm(vma), xe_vma_star= t(vma), + ret =3D xe_vm_range_tilemask_tlb_invalidation(xe_vma_vm(vma), NULL, + xe_vma_start(vma), xe_vma_end(vma), tile_mask); =20 /* WRITE_ONCE pairs with READ_ONCE in xe_vm_has_valid_gpu_mapping() */ diff --git a/drivers/gpu/drm/xe/xe_vm.h b/drivers/gpu/drm/xe/xe_vm.h index 3475a118f666..d1c3c9aa8d03 100644 --- a/drivers/gpu/drm/xe/xe_vm.h +++ b/drivers/gpu/drm/xe/xe_vm.h @@ -22,6 +22,7 @@ struct dma_fence; =20 struct xe_exec_queue; struct xe_file; +struct xe_gt_tlb_invalidation_fence; struct xe_sync_entry; struct xe_svm_range; struct drm_exec; @@ -228,8 +229,9 @@ struct dma_fence *xe_vm_range_rebind(struct xe_vm *vm, struct dma_fence *xe_vm_range_unbind(struct xe_vm *vm, struct xe_svm_range *range); =20 -int xe_vm_range_tilemask_tlb_invalidation(struct xe_vm *vm, u64 start, - u64 end, u8 tile_mask); +int xe_vm_range_tilemask_tlb_invalidation(struct xe_vm *vm, + struct xe_gt_tlb_invalidation_fence *fences, + u64 start, u64 end, u8 tile_mask); =20 int xe_vm_invalidate_vma(struct xe_vma *vma); =20 --=20 2.50.1 From nobody Sun Oct 5 03:35:59 2025 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5602D2BD037 for ; Sat, 9 Aug 2025 13:52:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754747547; cv=none; b=uUx/dKE8f2aT1kEEEgcGcX7tiiGa5TlyltpSGaQJYgyBRYYcZnFiN5lexLTfuvVXoOmUNQR5CJCqPDuQLLRSWzcFApbAbnelI3CWROURY2ITmlAmZawbkkwm4WcqKAc/nS4PIhfsjTMEVzEGvCUpnKE9omdw+sS8qgEJ/wleEZU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754747547; c=relaxed/simple; bh=eVwlttIO5cPoFbzrqx0Zh2eNe66yN5Gkt7LN4QnB250=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=n4so4CqV8GtpoYEaYj9m5ArBw8Y2C/LZloZb6cZRdwDqt+M8S8LeHZ6zPgTg9rLy+uUYWyLLmaOyi/iCo38ocsYIuIZJ9Wi2UxKCZ4XU4HvFtYPK9emhh0abbPfEWDtwEkszgx0eaAZx+Ni2XduE5OYfztVGXjQpPbZqudU3qhk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=WUfugQiQ; arc=none smtp.client-ip=198.175.65.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="WUfugQiQ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1754747546; x=1786283546; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=eVwlttIO5cPoFbzrqx0Zh2eNe66yN5Gkt7LN4QnB250=; b=WUfugQiQZEq5MG97m8M4HprNo5316mcdIjF7oZS+EpdpK/zTMFtrAVuT Xi2D5s71oo127b+EsGlW1urSWstyDyIs9fUycKYdZWpkQRZNU5xSD3f1c Sb2PUGit/jyl3K4bqYjU9Xb/kC3/ShFsshg+nW6CGwJfOsFMa9MjE4WVR ZLI1czzt5sAOZA6A4DolDx1owLhg7uyGpm1KHU6QmC7FqfAjgSe98yjwM rsgvara3Ct2bIXFV2T9nIyWiKWDQrRbpvpQ1RIbr/jbV3vzjHJo78oBfv FcC0kuBK2HThEy59UrgGXbhEhfiKLG2wicWqILy4Zl2+7CVAYA4tm8KhR A==; X-CSE-ConnectionGUID: Cj4DhNNfToGikl5mcVbbwA== X-CSE-MsgGUID: DCkEvYEvQwOCh6aEHdqkWg== X-IronPort-AV: E=McAfee;i="6800,10657,11515"; a="68153594" X-IronPort-AV: E=Sophos;i="6.17,278,1747724400"; d="scan'208";a="68153594" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by orvoesa105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2025 06:52:26 -0700 X-CSE-ConnectionGUID: R3aCHRgpQ4+mVFpxSPBY2Q== X-CSE-MsgGUID: zuNnvxMYTpSWm7CSeb31rg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.17,278,1747724400"; d="scan'208";a="165903777" Received: from smoticic-mobl1.ger.corp.intel.com (HELO fedora) ([10.245.244.28]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2025 06:52:24 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-xe@lists.freedesktop.org Cc: Matthew Brost , =?UTF-8?q?Christian=20K=C3=B6nig?= , dri-devel@lists.freedesktop.org, Jason Gunthorpe , Andrew Morton , Simona Vetter , Dave Airlie , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 6/6] drm/xe: Implement two pass MMU notifiers for SVM Date: Sat, 9 Aug 2025 15:51:37 +0200 Message-ID: <20250809135137.259427-7-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250809135137.259427-1-thomas.hellstrom@linux.intel.com> References: <20250809135137.259427-1-thomas.hellstrom@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Matthew Brost Implement two-pass MMU notifiers for SVM, enabling multiple VMs or devices with GPU mappings to pipeline costly TLB invalidations by issuing them in the first pass and waiting for completion in the second. Signed-off-by: Matthew Brost --- drivers/gpu/drm/drm_gpusvm.c | 2 +- drivers/gpu/drm/xe/xe_svm.c | 74 ++++++++++++++++++++++++++++++------ 2 files changed, 63 insertions(+), 13 deletions(-) diff --git a/drivers/gpu/drm/drm_gpusvm.c b/drivers/gpu/drm/drm_gpusvm.c index 92dc7d2bd6cf..f153df1bc862 100644 --- a/drivers/gpu/drm/drm_gpusvm.c +++ b/drivers/gpu/drm/drm_gpusvm.c @@ -413,7 +413,7 @@ drm_gpusvm_notifier_invalidate_twopass(struct mmu_inter= val_notifier *mni, * drm_gpusvm_notifier_ops - MMU interval notifier operations for GPU SVM */ static const struct mmu_interval_notifier_ops drm_gpusvm_notifier_ops =3D { - .invalidate_twopass =3D drm_gpusvm_notifier_invalidate_twopass, + .invalidate_multipass =3D drm_gpusvm_notifier_invalidate_twopass, }; =20 /** diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c index 82a598c8d56e..5728394806ca 100644 --- a/drivers/gpu/drm/xe/xe_svm.c +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -144,15 +144,8 @@ xe_svm_range_notifier_event_begin(struct xe_vm *vm, st= ruct drm_gpusvm_range *r, * invalidations spanning multiple ranges. */ for_each_tile(tile, xe, id) - if (xe_pt_zap_ptes_range(tile, vm, range)) { + if (xe_pt_zap_ptes_range(tile, vm, range)) tile_mask |=3D BIT(id); - /* - * WRITE_ONCE pairs with READ_ONCE in - * xe_vm_has_valid_gpu_mapping() - */ - WRITE_ONCE(range->tile_invalidated, - range->tile_invalidated | BIT(id)); - } =20 return tile_mask; } @@ -161,16 +154,60 @@ static void xe_svm_range_notifier_event_end(struct xe_vm *vm, struct drm_gpusvm_range = *r, const struct mmu_notifier_range *mmu_range) { + struct xe_svm_range *range =3D to_xe_range(r); struct drm_gpusvm_ctx ctx =3D { .in_notifier =3D true, }; =20 xe_svm_assert_in_notifier(vm); =20 + /* + * WRITE_ONCE pairs with READ_ONCE in xe_vm_has_valid_gpu_mapping() + */ + WRITE_ONCE(range->tile_invalidated, range->tile_present); + drm_gpusvm_range_unmap_pages(&vm->svm.gpusvm, r, &ctx); if (!xe_vm_is_closed(vm) && mmu_range->event =3D=3D MMU_NOTIFY_UNMAP) xe_svm_garbage_collector_add_range(vm, to_xe_range(r), mmu_range); } =20 +struct xe_svm_invalidate_pass { + struct drm_gpusvm *gpusvm; + struct drm_gpusvm_notifier *notifier; +#define XE_SVM_INVALIDATE_FENCE_COUNT \ + (XE_MAX_TILES_PER_DEVICE * XE_MAX_GT_PER_TILE) + struct xe_gt_tlb_invalidation_fence fences[XE_SVM_INVALIDATE_FENCE_COUNT]; + struct mmu_interval_notifier_pass p; +}; + +static struct mmu_interval_notifier_pass * +xe_svm_invalidate_second(struct mmu_interval_notifier_pass *p, + const struct mmu_notifier_range *mmu_range, + unsigned long cur_seq) +{ + struct xe_svm_invalidate_pass *pass =3D container_of(p, typeof(*pass), p); + struct drm_gpusvm *gpusvm =3D pass->gpusvm; + struct drm_gpusvm_notifier *notifier =3D pass->notifier; + struct drm_gpusvm_range *r =3D NULL; + struct xe_vm *vm =3D gpusvm_to_vm(gpusvm); + u64 adj_start =3D mmu_range->start, adj_end =3D mmu_range->end; + int id; + + /* Adjust invalidation to notifier boundaries */ + adj_start =3D max(drm_gpusvm_notifier_start(notifier), adj_start); + adj_end =3D min(drm_gpusvm_notifier_end(notifier), adj_end); + + for (id =3D 0; id < XE_SVM_INVALIDATE_FENCE_COUNT; ++id) + xe_gt_tlb_invalidation_fence_wait(&pass->fences[id]); + + drm_gpusvm_in_notifier_lock(gpusvm); + drm_gpusvm_for_each_range(r, notifier, adj_start, adj_end) + xe_svm_range_notifier_event_end(vm, r, mmu_range); + drm_gpusvm_in_notifier_unlock(gpusvm); + + kfree(pass); + return NULL; +} + static void xe_svm_invalidate_twopass(struct drm_gpusvm *gpusvm, struct drm_gpusvm_notifier *notifier, const struct mmu_notifier_range *mmu_range, @@ -179,6 +216,8 @@ static void xe_svm_invalidate_twopass(struct drm_gpusvm= *gpusvm, struct xe_vm *vm =3D gpusvm_to_vm(gpusvm); struct xe_device *xe =3D vm->xe; struct drm_gpusvm_range *r, *first; + struct xe_svm_invalidate_pass *pass =3D NULL; + struct xe_gt_tlb_invalidation_fence *fences =3D NULL; u64 adj_start =3D mmu_range->start, adj_end =3D mmu_range->end; u8 tile_mask =3D 0; long err; @@ -226,14 +265,25 @@ static void xe_svm_invalidate_twopass(struct drm_gpus= vm *gpusvm, =20 xe_device_wmb(xe); =20 - err =3D xe_vm_range_tilemask_tlb_invalidation(vm, NULL, adj_start, + pass =3D kzalloc(sizeof(*pass), GFP_NOWAIT); + if (pass) { + pass->gpusvm =3D gpusvm; + pass->notifier =3D notifier; + pass->p.pass =3D xe_svm_invalidate_second; + fences =3D pass->fences; + *p =3D &pass->p; + } + + err =3D xe_vm_range_tilemask_tlb_invalidation(vm, fences, adj_start, adj_end, tile_mask); WARN_ON_ONCE(err); =20 range_notifier_event_end: - r =3D first; - drm_gpusvm_for_each_range(r, notifier, adj_start, adj_end) - xe_svm_range_notifier_event_end(vm, r, mmu_range); + if (!pass) { + r =3D first; + drm_gpusvm_for_each_range(r, notifier, adj_start, adj_end) + xe_svm_range_notifier_event_end(vm, r, mmu_range); + } } =20 static int __xe_svm_garbage_collector(struct xe_vm *vm, --=20 2.50.1