From nobody Fri Oct 2 07:45:30 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 730AF331EB3 for ; Tue, 4 Aug 2026 04:26:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817616; cv=none; b=R97HxWRxBPcsJU8GStzkUXXOXuhNVHiuo33gqpiPbbOKwVazlTkGPAEFsP0HY23I4u8ZpFWPLNXkY8Y3OKNWvj1BtWEZEA4RhZoimRPmDoG8d1xaADFDU6+MPzdHYSihJh+1XWU3G/QkDHhC0wPdlKAe0MKYBPhApxA5z3NjrTg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817616; c=relaxed/simple; bh=m/YGHZfxZaxAhm7Vn+an85b7whoDrdoSP6zeSZnUEy0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=e1XWKC7CgX5Y3+PI/zIYNVx8fCqBXc6DS93Y8G3rf/IWkKkO+c11xrZDhTVTtzVhRlOdqfP24xNrAMaWNRcjpTBf5rjXJgfiRA5WcYAHMN6ogIPp9qpy46BthcVPi3OjSk+LiNxz06GZlv95i4rF0+QQgj8scwbRsyHAGqhGhDI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=NSh7peBC; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=GUVI4Xo+; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="NSh7peBC"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="GUVI4Xo+" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1785817614; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hwL36eymgoLnccdesFRfeMb71MKt5ljy6aFM5YklG3k=; b=NSh7peBCTp9IoqIxQOFe7tS+KtLq9BgslC69+cqGeUsmSmflR6GLssZo96S+1D1sm/TGQM v0lfCWjw9kUJJFy5S5AeBQTEYU7QpdHJkh6U4gRenEC5O1dj6bWBprI/gegyeSdmcJPZgx /6DNZY9nEKV2po+2+1PTT9k5cS63rbI= Received: from mail-lf1-f71.google.com (mail-lf1-f71.google.com [209.85.167.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-561-XgnD_Z6mPwm1Ifm9AtRKXg-1; Tue, 04 Aug 2026 00:26:52 -0400 X-MC-Unique: XgnD_Z6mPwm1Ifm9AtRKXg-1 X-Mimecast-MFC-AGG-ID: XgnD_Z6mPwm1Ifm9AtRKXg_1785817611 Received: by mail-lf1-f71.google.com with SMTP id 2adb3069b0e04-5aebb33d6b8so2747822e87.0 for ; Mon, 03 Aug 2026 21:26:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1785817611; x=1786422411; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=hwL36eymgoLnccdesFRfeMb71MKt5ljy6aFM5YklG3k=; b=GUVI4Xo+too8rLuhNspdFV6+ITRCkwFCn/8YLpJ/YMUJ32onCwxFJsBncq3NgJ3bCr Y7gKwHI9i0YV2kJInOyjk/trhBbZNedIuaMhYriwni2NH80T42ruz+mE0jfQQQTAwHA3 YJNdikZjSW5doMX0fe5aGIFb3a0Y+9JZnGxNZT8+Z8uiLHZJqLTwGMF4S1ozKanqIJWE S44tLuoPA/0f4nETXXW8tQpEwbzArDxZ6Khpd3WqX3w3/potF6HdbKBqPNedVq8Oiuid KFcW6SaiFtd+iDRfOSgYbObcM23QoiY0ILhu45o3Za0sVoxXMdxYMAzi8BBMtp1LXIRU JHBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785817611; x=1786422411; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=hwL36eymgoLnccdesFRfeMb71MKt5ljy6aFM5YklG3k=; b=k6QFwZzFw3DoTP8XtXhXVdomeRewgAHHO0xH4r+QS3qVQ33KBzv1AeFO1SbFGe+uxz 2LAksvcTkW1pXDpQDwq6lcpOsUgyRLf17mQfW8m/TyAS3/x3sP3E+pGj6F9BdJDwVHhz 5KYdnSVCjZgNykZ+vErx1FFXfTqqDA24GaoZRwFPdCgErFXBmwybvbC9mC7j0xTMN05s uUqshSUCZpgSfSKBugngmGm3OXPT0f9wZwMdRPXmyHvb2xKJCqzSw/t6YR3gFOwb3h7f vWcwqvLlJA0SMTO+e2m7+BjMXQgSlPLcXKv7AFDGD7uxmHHXga0BcLe2fC6VpzWz1yg6 EMAw== X-Forwarded-Encrypted: i=1; AHgh+RqztUd5dZ9F4O43VSosCK8Dwk9PAO3chf3W3xD3odu6NndHWXrFJkM5CKI1Tu/ToU+0D8e0VHHBZDQ4Lto=@vger.kernel.org X-Gm-Message-State: AOJu0Yy87VPnx/cpXTdWLsYhjctKeGWLnmRiVddwYJ+gpY+FZzb3qXs/ NQb1KxPHevpjEi0JS5qVq2a18Mfmox8zbcM7g79vZC50p25KKV7+dVGSeUP8OWxE5EIJFPl31kb CTKR/fESO8MBotE9Tw5+ah8R3OGfevrkB6vkBmmU9c+05iPgmYoocryCOW3ucoWlA X-Gm-Gg: AR+sD10WqAIK9KY5zLBm5XjJRGONcrDF2QbA17hMeZlnPPGV5nP2PO3jhyaxo3uNMW1 aL51xnJysQD1RbH9tdBIPfwnzKld7gRGBEVA0nbwLLyEFVhvo30GcDZIDCPm5a35cQ7cruHdW/C qZ+SaLz6bNTJyBPke50sohMwO8nrfU9Vt0Bs1SRSHvrX5buZQgl4k2E5qI5M41sL9OwvP5tYDVX ejiNnNZpZZU/EH6Z0hg4eKYin6x6/wAR0ANtQdNRcfJInD0gbddMcbdqr9ScZF7Sk3Kurp7EHcb dHu4cXz4wFbmjrKE3MSehGKMSFl5oJbRRBNfc9Pp5yNPDz7nzsOTbSENte5gmwIftt+S9MaRrpH U3v8aCFlNxAZ+zMlb X-Received: by 2002:a05:6512:3c96:b0:5b2:991f:90d1 with SMTP id 2adb3069b0e04-5b2e4f3cb5bmr2664340e87.26.1785817610754; Mon, 03 Aug 2026 21:26:50 -0700 (PDT) X-Received: by 2002:a05:6512:3c96:b0:5b2:991f:90d1 with SMTP id 2adb3069b0e04-5b2e4f3cb5bmr2664315e87.26.1785817610290; Mon, 03 Aug 2026 21:26:50 -0700 (PDT) Received: from fedora (85-23-51-1.bb.dnainternet.fi. [85.23.51.1]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b2e23c46c4sm2311351e87.17.2026.08.03.21.26.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Aug 2026 21:26:48 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v13 01/11] mm/Kconfig: changes for migrate on fault for device pages Date: Tue, 4 Aug 2026 07:26:21 +0300 Message-ID: <20260804042631.2175585-2-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260804042631.2175585-1-mpenttil@redhat.com> References: <20260804042631.2175585-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 HMM depends on MMU notifiers. With the unified HMM/migrate_device page table walk migrate_device needs HMM enabled. Enable them explicitly to avoid breaking random configs. Cc: Andrew Morton Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: "Liam R. Howlett" Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Signed-off-by: Mika Penttil=C3=A4 Acked-by: David Hildenbrand (Arm) --- mm/Kconfig | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index 9e0ca4824905..7647671f37f0 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -699,6 +699,7 @@ config MIGRATION =20 config DEVICE_MIGRATION def_bool MIGRATION && ZONE_DEVICE + select HMM_MIRROR =20 config ARCH_ENABLE_HUGEPAGE_MIGRATION bool @@ -1260,6 +1261,7 @@ config ZONE_DEVICE config HMM_MIRROR bool depends on MMU + select MMU_NOTIFIER =20 config GET_FREE_REGION bool --=20 2.55.0 From nobody Fri Oct 2 07:45:30 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EE33233A02B for ; Tue, 4 Aug 2026 04:27:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817624; cv=none; b=SeIrGm7T8wy5HQY8SzBCBq/SUPgPKfkTz9o3YehVO161J2VCGF5DsLfZx5c3QGTXWyMLMLPXN+dpfMO30NiGictdOA5wYOz1Pgz8qTuo/Kuspmnxbo8LFJzqI1XPaFXim64O//ZVl1/MA9eiM1MTmhvkfzOrKFK8vdAWYs0SEWA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817624; c=relaxed/simple; bh=mu6vL51u5TRVdfK0l7AImxgf3JCs8ho3Pl4NoaJ4PNs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=l80HP+sPzg60ibWPWirHOZqviibFQALh4HhmtV67h7hEJbYgx2HKSlqHHDhaEHH79/hMZkzIz4AL5tcX3PtdilC5O78xtkHBkY2JdCDQNzrNz7zfY6licGMe1WOtdlkEboWZ8oYzX5d06gVmWU9WMkXjfp+HJny0nTQdhcIziLE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=cgANNcrJ; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=Qq/jACnk; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="cgANNcrJ"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="Qq/jACnk" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1785817621; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ihGGkS1db6JGdRXvy1Z/cqvAlkfUAiXTBTA07KJEz14=; b=cgANNcrJwSKW0d5pENmqC5EXj22CGP04NXzbYcTp9T45OVAV+GBdArZfdTTAKcYp9gKA7K TGq7BNHf1ewBsE3pxpX7KDVa9H0KIV3S3nOn0QKl0jh81Cdn9xFNxbw4339WaMlPx64Mv4 2pnJaxJHzT7lJc7uepFlghewb+0VzY8= Received: from mail-lf1-f70.google.com (mail-lf1-f70.google.com [209.85.167.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-344-RB4VGqCQPBWFL8K8GzoQvg-1; Tue, 04 Aug 2026 00:26:59 -0400 X-MC-Unique: RB4VGqCQPBWFL8K8GzoQvg-1 X-Mimecast-MFC-AGG-ID: RB4VGqCQPBWFL8K8GzoQvg_1785817618 Received: by mail-lf1-f70.google.com with SMTP id 2adb3069b0e04-5b109e0a1d8so1963695e87.3 for ; Mon, 03 Aug 2026 21:26:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1785817618; x=1786422418; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=ihGGkS1db6JGdRXvy1Z/cqvAlkfUAiXTBTA07KJEz14=; b=Qq/jACnk1y9lnPXlIHR0w80dycfphVcYqw6NC9oT2kZmH1j3zvZAZxIyDs/H2HxbyJ OphsrK9kF4MZ0oADUq0RLxDr/TnX56eeu+VovZTc3iJZdJMkICuXSXW1ZGQmo01i6FCE PVx9YolZ2C8HW/QPUQKH+yBNn2XwZVknVZscLc2GZ8TD6EPaQzzUXLOXTOwRMJ3OGm9i Pof270stmukLOxH5KUTK6JrtdQDoXrMOI1rtE2MRVAWaERuV0ZK9Cejlc0M/GjXCQMOU l+PKuSpwkTMrbpPOj5vioj4aRHACvOK1Meg4Mct1jhnzlAqxjILghkbmI+2Q+VP+KusC nM7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785817618; x=1786422418; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=ihGGkS1db6JGdRXvy1Z/cqvAlkfUAiXTBTA07KJEz14=; b=nN1hGvtzx5IdLgb9ZJsC9MM00Mu8iOTiSeEOssmcXOVqUn6Ra0SsFcKsE0K4EK9HJx 6N8qr/zFc6txkbygUbequiEvgvFsNR85rxB62/6DBSkXQzxogPAXLjQ5fvnb25dly/MG lSBP8O7Nw3Vw2aeDyKHTjkdmXgSUL2cdc8pMvaQDvyGjA1VLOsjbxtd7RiWppT/jBHG+ 1Ej143jmZkvm9iHHVxTexQD3l0rf35bDmiLTaW0RL3MyFuIFSshPatejx6AxTmy0l6sz lMq6EWLQZL8U1Pah+tAI/cmiw5qUMnjRDAa4SAe/+GVpAveeE1FeSzNdeX19mlXW6wLQ 8dEw== X-Forwarded-Encrypted: i=1; AHgh+RqpApLu9X77IOt3OAZYjY3qTZQWds85oFHUv9WQIWXgOtE1VAHfmgtbvK6R0r57Mie+tTWOh0brfVOuqrs=@vger.kernel.org X-Gm-Message-State: AOJu0YwZgIT2Hn91vpd4qytJyZ0Tnkx6I0BPBReoe+w6dCUQ8NcpOjTi 0DFAzzRj7xBXCo3HfIhUh+93Xfq5f4rkGCQHZUvzpNiVBmSPmcbb34pu7ofQmePcGWh1yDGZZX0 ND6+5KQ4SQ0wZoz2nz8jYWIVYI+6eSrZH3B5h3w/ZcUXN+hzSRSIzUUn940i9jyqv X-Gm-Gg: AR+sD101096+8kY9IUjA4DL4Agxk8k0HI0R6py5hWb7KKk+k10f2krbWzWtB3da4feh jg9dHP1zZsPb2TZOdTydNwcC10LjPRlG8E789EVLmfW6FqoEgnPoVEZu0KB4iPZvUApilklaRVC yfTLTMpwMURpojtcGKvZdYd+BRP7SlENSSUsO5sFVcIJNlRUXAGGSrBo9ff1/G5OqTc7UdqA4Ur d9jbd/9S/PD0VQ4DJA2rKkZ8WzbUZ4Aw9yql37ezp47gsmQvKpQCAC0QKVvtxxcZK92D/e4FIDp /l21jx7K8XiAI7Ukjv/XIKnr3kzSYQ94CZvwm9mHHK8rUatXC0ML7LoBfTdRd4+aJaGyYTPthf2 GnNtGA0xqQvCYF0DT X-Received: by 2002:a05:6512:39c3:b0:5ae:bd5e:95f7 with SMTP id 2adb3069b0e04-5b2e4f1aed0mr2402078e87.4.1785817618212; Mon, 03 Aug 2026 21:26:58 -0700 (PDT) X-Received: by 2002:a05:6512:39c3:b0:5ae:bd5e:95f7 with SMTP id 2adb3069b0e04-5b2e4f1aed0mr2402060e87.4.1785817617697; Mon, 03 Aug 2026 21:26:57 -0700 (PDT) Received: from fedora (85-23-51-1.bb.dnainternet.fi. [85.23.51.1]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b2e23c46c4sm2311351e87.17.2026.08.03.21.26.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Aug 2026 21:26:51 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v13 02/11] mm: add helper to convert HMM pfn to migrate pfn Date: Tue, 4 Aug 2026 07:26:22 +0300 Message-ID: <20260804042631.2175585-3-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260804042631.2175585-1-mpenttil@redhat.com> References: <20260804042631.2175585-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 The unified HMM/migrate_device pagewalk does the "collecting" on the HMM side, so we need a helper to transfer pfns to the migrate_vma world. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- include/linux/hmm.h | 19 +++++++++++++- include/linux/migrate.h | 3 ++- mm/migrate_device.c | 55 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 75 insertions(+), 2 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index db75ffc949a7..bfedafc1c143 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -13,6 +13,8 @@ =20 struct mmu_interval_notifier; =20 +struct migrate_vma; + /* * On output: * 0 - The page is faultable and a future call with=20 @@ -27,6 +29,13 @@ struct mmu_interval_notifier; * HMM_PFN_P2PDMA_BUS - Bus mapped P2P transfer * HMM_PFN_DMA_MAPPED - Flag preserved on input-to-output transformation * to mark that page is already DMA mapped + * HMM_PFN_MIGRATE - The entry is to be migrated. Note, HMM_PFN_MIGRATE + * alone without HMM_PFN_VALID denotes the + * empty page. + * This flag together with HMM_PFN_COMPOUND are + * indicators for migrate_hmm_range_setup() to + * setup the migrate pfns. + * HMM_PFN_COMPOUND - The entry is represents a > 0 order page * * On input: * 0 - Return the current state of the page, do not fault = it. @@ -34,6 +43,8 @@ struct mmu_interval_notifier; * will fail * HMM_PFN_REQ_WRITE - The output must have HMM_PFN_WRITE or hmm_range_fau= lt() * will fail. Must be combined with HMM_PFN_REQ_FAULT. + * HMM_PFN_REQ_MIGRATE - For default_flags, request to migrate, according = to + * hmm_range.migrate.flags */ enum hmm_pfn_flags { /* Output fields and flags */ @@ -48,11 +59,15 @@ enum hmm_pfn_flags { HMM_PFN_P2PDMA =3D 1UL << (BITS_PER_LONG - 5), HMM_PFN_P2PDMA_BUS =3D 1UL << (BITS_PER_LONG - 6), =20 - HMM_PFN_ORDER_SHIFT =3D (BITS_PER_LONG - 11), + /* Migrate request */ + HMM_PFN_MIGRATE =3D 1UL << (BITS_PER_LONG - 7), + HMM_PFN_COMPOUND =3D 1UL << (BITS_PER_LONG - 8), + HMM_PFN_ORDER_SHIFT =3D (BITS_PER_LONG - 13), =20 /* Input flags */ HMM_PFN_REQ_FAULT =3D HMM_PFN_VALID, HMM_PFN_REQ_WRITE =3D HMM_PFN_WRITE, + HMM_PFN_REQ_MIGRATE =3D HMM_PFN_MIGRATE, =20 HMM_PFN_FLAGS =3D ~((1UL << HMM_PFN_ORDER_SHIFT) - 1), }; @@ -107,6 +122,7 @@ static inline unsigned int hmm_pfn_to_map_order(unsigne= d long hmm_pfn) * @default_flags: default flags for the range (write, read, ... see hmm d= oc) * @pfn_flags_mask: allows to mask pfn flags so that only default_flags ma= tter * @dev_private_owner: owner of device private pages + * @migrate: structure for migrating a range of a VMA */ struct hmm_range { struct mmu_interval_notifier *notifier; @@ -117,6 +133,7 @@ struct hmm_range { unsigned long default_flags; unsigned long pfn_flags_mask; void *dev_private_owner; + struct migrate_vma *migrate; }; =20 /* diff --git a/include/linux/migrate.h b/include/linux/migrate.h index d5af2b7f577b..425ab5242da0 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -3,6 +3,7 @@ #define _LINUX_MIGRATE_H =20 #include +#include #include #include #include @@ -200,7 +201,7 @@ void migrate_device_pages(unsigned long *src_pfns, unsi= gned long *dst_pfns, unsigned long npages); void migrate_device_finalize(unsigned long *src_pfns, unsigned long *dst_pfns, unsigned long npages); - +void migrate_hmm_range_setup(struct hmm_range *range); #endif /* CONFIG_MIGRATION */ =20 #endif /* _LINUX_MIGRATE_H */ diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 908d2d4ec43a..a6df6566f89e 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -1487,3 +1487,58 @@ int migrate_device_coherent_folio(struct folio *foli= o) return 0; return -EBUSY; } + +/** + * migrate_hmm_range_setup() - prepare to migrate a range of memory + * @range: contains pointer to struct migrate_vma to be set up. + * + * When collecting has been done with hmm_range_fault(), this + * should be called next, and completes range->migrate by + * populating migrate->src[] and migrate->dst[] + * using range->hmm_pfns[]. + * Also, migrate->cpages and migrate->npages get initialized. + * After migrate_hmm_range_setup(), range->migrate is good + * for the rest of the migrate_vma_* flow. + */ +void migrate_hmm_range_setup(struct hmm_range *range) +{ + struct migrate_vma *migrate =3D range->migrate; + + if (!migrate) + return; + + migrate->npages =3D (migrate->end - migrate->start) >> PAGE_SHIFT; + migrate->cpages =3D 0; + + for (unsigned long i =3D 0; i < migrate->npages; i++) { + unsigned long pfn =3D range->hmm_pfns[i]; + + /* + * We are only interested in entries to be + * migrated. + */ + if (!(pfn & HMM_PFN_MIGRATE)) { + migrate->src[i] =3D 0; + migrate->dst[i] =3D 0; + continue; + } + + migrate->cpages++; + + /* HMM_PFN_MIGRATE without HMM_PFN_VALID denotes the special zero page */ + if (pfn & HMM_PFN_VALID) + migrate->src[i] =3D migrate_pfn(page_to_pfn(hmm_pfn_to_page(pfn))); + else + migrate->src[i] =3D 0; + + migrate->src[i] |=3D MIGRATE_PFN_MIGRATE; + migrate->src[i] |=3D (pfn & HMM_PFN_WRITE) ? MIGRATE_PFN_WRITE : 0; + migrate->src[i] |=3D (pfn & HMM_PFN_COMPOUND) ? MIGRATE_PFN_COMPOUND : 0; + migrate->dst[i] =3D 0; + } + + if (migrate->cpages) + migrate_vma_unmap(migrate); + +} +EXPORT_SYMBOL(migrate_hmm_range_setup); --=20 2.55.0 From nobody Fri Oct 2 07:45:30 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8BDBC3382CD for ; Tue, 4 Aug 2026 04:27:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817630; cv=none; b=lsL6wdi1QmbgWDgtalV8FOLJADoE4K0CaN2DgZc33d0WxgaVnVMS9CR0SzpGBMbC91NGLmEO+qvzwGCd+c4W2rCqvJrtxDMduRIU+OFPj2Gek3l4sVJSXrCIRGn9r301i87d1VT+GOOZOGLx3YYBv/aMhsxVC5QU5GCwfNlpNv0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817630; c=relaxed/simple; bh=N+QiSPZQloKU/Gz/pXSk4NHFZ99qjsUrFcKHHiI8jR8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=c9xdiwC8g9VVbr54mgOYr42ACO3VRrcL2V5kh7Ytx2pgB0wgCv7amtTUTF+CI7KM7pMl9LhC8XEytIuYQhrorlkNxLxu2lkw2+CCZXIFFw4Vd0M6eCAkqKIzxjQ4lisFPcGHCxXYYQQsn8yEZoVy7kvacrd64cray7MFJnMuzi8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=WIa9IyqH; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=tl0HjgCF; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="WIa9IyqH"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="tl0HjgCF" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1785817627; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Bp/HOa9o+BFxZP9Tl5+LymilN/hYIUdOgGYp2AmAGQw=; b=WIa9IyqHnitOSikrNKlgrQ06Ws4jvoYl7IAXA+L4vLVf6YjU67CIANS69VoXo2XnAudcIW 1oY2c2U2NZKPea6HbgZ2JiF6c0pKNin5Ect+MX5tbjGfvfDuNGM6hwBGO0MqasaKQyYEyk xgI5BpiadK0zQjHE/9/xDNWHDW33NaQ= Received: from mail-lf1-f71.google.com (mail-lf1-f71.google.com [209.85.167.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-465-2KfBWpqlPP6h-i5Hj0E4EA-1; Tue, 04 Aug 2026 00:27:06 -0400 X-MC-Unique: 2KfBWpqlPP6h-i5Hj0E4EA-1 X-Mimecast-MFC-AGG-ID: 2KfBWpqlPP6h-i5Hj0E4EA_1785817624 Received: by mail-lf1-f71.google.com with SMTP id 2adb3069b0e04-5b2a9f6b0b2so2285331e87.2 for ; Mon, 03 Aug 2026 21:27:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1785817624; x=1786422424; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=Bp/HOa9o+BFxZP9Tl5+LymilN/hYIUdOgGYp2AmAGQw=; b=tl0HjgCFY2Q6L1izT/QYxECO0whkG+7g/SEEoDIhUDKC+N+GzBqv2NUJ/0+zo27EC8 rS+cwT+Npjf2U3JS83FgLaSsXQHlFvWLNT0Rr5Gj6BKlRkF33NmJOvctwnmiROU8Crs8 /XJexu/Qcth+ME+plofHrXNBQWQakq5TP0DIPdlT93T/tg9LQBibbKFJvaZBtHUXvB+u BvyJXJjC0hX0EkaTk/1Ky0lJBnFTHknEFsLgwZKo6i/Nq9vh3SHeJIYNPyV7S62SQAor e/RLdrV+zAS7gUC+QN5rAXjvKmrJwvUc5WyHtRAktDACqsA5H7EOIcgk3JT63E2FYxa6 386Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785817624; x=1786422424; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=Bp/HOa9o+BFxZP9Tl5+LymilN/hYIUdOgGYp2AmAGQw=; b=OM/KnPcCXtrA9Yvt0VqsUCf6HSVvQPFHUPAvQSryBKNjqd8NKwjAuF1xLWkniNjivc epPzGpS6J9h8ksU+McB4wxl+XZzCIeLAq52d5yjO32DBOvxdP1gauD/MsTksiHffF2yB JslO1Ll7om5fUQ08lFEuVaqsifQ5NNTjVa9fvQzZ5y0sZTvSrhj+NfVn6z+gJYjMVVot wwPE3raTcGQz5HwW1r6XOxYahKdYuEXt4XsRnP4fpZUWz1b0T5VV1iv9xOT/k/reYImo 5iO5jETJfnwJpfB108tPglMMgx11l7aTVFCf5QpM4SZ1UogL/XEwl/P7BHc1hUyxZv02 mDkw== X-Forwarded-Encrypted: i=1; AHgh+RpdEbkUwN9ez+xqHwENJHX+JgSucRHgosqchzhNX9+XzQxNk78sEs1RznhdfokTUxIDgftqa/pcwyO2Z24=@vger.kernel.org X-Gm-Message-State: AOJu0Yz4iAba0Uun3b+UBBO9t9mPj3ELluYoGD19d0wEDf7N9A8TWJvO d28m2YeE7xmN5WleEbHuy+vOuS7Y/A7MhJFn7zqOGbgnWHSsQxfvfXVjZbe6922MD2DCwQCyVI4 XEhO2Xrw7+z383SpopWunQuHkKIhVwgxayqs8gMn86LVSU4MCg2kcd3bsloLHXH64 X-Gm-Gg: AR+sD10AUsVLi2kQjf/8Tk7fOVvHHIt6CUjPOkktGvVyZW1umQ6Mi8pbTnhpsI9VCMA H8iEUHSxvsjQ9MY3UBIBxMR9j0DiQIQPVjD+FSRk82po+FWDS85VmJZ8FpSXfEUfeq3J4mOtyE4 dwQQEDHKKl7r6Kelq49axzcYCa8wMm4O2bi3RQdiLNUZ/o+h7L339H6d2Z6QsyJ5Rotrr884I03 tAY9CxkrRzpyCnfwgmP24zV3e/mj21qsImdiX+gOXbr+iq709qAdihOdT1Sstf1QmDfu+QlPb7k /sx0GMF49JG7apUK7OUuCuWO58fTOGfit2w0+gE4hKh3oeyo2mn0y/0Iuzs1CWlryDY055MTqTQ GmsJ/r9h1Ny4Z1x4F X-Received: by 2002:a05:6512:63cb:b0:5b2:b85d:4802 with SMTP id 2adb3069b0e04-5b2e4f19bb8mr1748254e87.4.1785817624369; Mon, 03 Aug 2026 21:27:04 -0700 (PDT) X-Received: by 2002:a05:6512:63cb:b0:5b2:b85d:4802 with SMTP id 2adb3069b0e04-5b2e4f19bb8mr1748235e87.4.1785817623922; Mon, 03 Aug 2026 21:27:03 -0700 (PDT) Received: from fedora (85-23-51-1.bb.dnainternet.fi. [85.23.51.1]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b2e23c46c4sm2311351e87.17.2026.08.03.21.26.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Aug 2026 21:27:00 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v13 03/11] mm/hmm: preparations for HMM to participate in migration Date: Tue, 4 Aug 2026 07:26:23 +0300 Message-ID: <20260804042631.2175585-4-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260804042631.2175585-1-mpenttil@redhat.com> References: <20260804042631.2175585-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 For migration to happen after hmm_range_fault(), the vma has to retrieved. Luckily, pagewalk already resolves that for us, so just have to save it for further use. For migration, mmap_lock has to be hold for the whole operation, so vma stays stable. Take care also of firing the mmu_notifier when migrating. Also, prepare for differencies of how mm_struct * is retrieved in fault and migrate_vma paths. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- include/linux/migrate.h | 45 ++++++++++++++++++++- lib/test_hmm.c | 2 +- mm/hmm.c | 86 ++++++++++++++++++++++++++++++++++++++--- 3 files changed, 125 insertions(+), 8 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 425ab5242da0..7380a89756ee 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -106,6 +106,28 @@ static inline void softleaf_entry_wait_on_locked(softl= eaf_t entry, spinlock_t *p spin_unlock(ptl); } =20 +enum migrate_vma_info { + MIGRATE_VMA_SELECT_NONE =3D 0, + MIGRATE_VMA_SELECT_COMPOUND =3D MIGRATE_VMA_SELECT_NONE, +}; + +static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *r= ange) +{ + return MIGRATE_VMA_SELECT_NONE; +} + +static inline void hmm_fill_migrate_vma(struct hmm_range *range, + struct vm_area_struct *vma, + unsigned long start, + unsigned long end) +{ +} + +static inline struct mm_struct *hmm_range_fault_mm(struct hmm_range *range) +{ + return range->notifier->mm; +} + #endif /* CONFIG_MIGRATION */ =20 #ifdef CONFIG_NUMA_BALANCING @@ -149,7 +171,7 @@ static inline unsigned long migrate_pfn(unsigned long p= fn) return (pfn << MIGRATE_PFN_SHIFT) | MIGRATE_PFN_VALID; } =20 -enum migrate_vma_direction { +enum migrate_vma_info { MIGRATE_VMA_SELECT_SYSTEM =3D 1 << 0, MIGRATE_VMA_SELECT_DEVICE_PRIVATE =3D 1 << 1, MIGRATE_VMA_SELECT_DEVICE_COHERENT =3D 1 << 2, @@ -191,6 +213,27 @@ struct migrate_vma { struct page *fault_page; }; =20 +// TODO: enable migration +static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *r= ange) +{ + return 0; +} + +static inline struct mm_struct *hmm_range_fault_mm(struct hmm_range *range) +{ + return range->notifier ? range->notifier->mm : range->migrate->vma->vm_mm; +} + +static inline void hmm_fill_migrate_vma(struct hmm_range *range, + struct vm_area_struct *vma, + unsigned long start, + unsigned long end) +{ + range->migrate->vma =3D vma; + range->migrate->start =3D start; + range->migrate->end =3D end; +} + int migrate_vma_setup(struct migrate_vma *args); void migrate_vma_pages(struct migrate_vma *migrate); void migrate_vma_finalize(struct migrate_vma *migrate); diff --git a/lib/test_hmm.c b/lib/test_hmm.c index c4adbf98fac7..79cb7487d211 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -145,7 +145,7 @@ static bool dmirror_is_private_zone(struct dmirror_devi= ce *mdevice) HMM_DMIRROR_MEMORY_DEVICE_PRIVATE); } =20 -static enum migrate_vma_direction +static enum migrate_vma_info dmirror_select_device(struct dmirror *dmirror) { return (dmirror->mdevice->zone_device_type =3D=3D diff --git a/mm/hmm.c b/mm/hmm.c index c72c9ddfdb2f..637c00188b3e 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -27,12 +28,17 @@ #include #include #include +#include =20 #include "internal.h" =20 struct hmm_vma_walk { - struct hmm_range *range; - unsigned long last; + struct mmu_notifier_range mmu_range; + struct vm_area_struct *vma; + struct hmm_range *range; + unsigned long start; + unsigned long end; + unsigned long last; }; =20 enum { @@ -393,6 +399,57 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *w= alk, unsigned long start, } #endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ =20 +static int hmm_vma_capture_migrate_range(unsigned long start, + unsigned long end, + struct mm_walk *walk) +{ + struct hmm_vma_walk *hmm_vma_walk =3D walk->private; + struct hmm_range *range =3D hmm_vma_walk->range; + + if (!hmm_select_migrate(range)) + return 0; + + if (hmm_vma_walk->vma && (hmm_vma_walk->vma !=3D walk->vma)) + return -ERANGE; + + hmm_vma_walk->vma =3D walk->vma; + hmm_vma_walk->start =3D start; + hmm_vma_walk->end =3D end; + + if (end - start > range->end - range->start) + return -ERANGE; + + if (!hmm_vma_walk->mmu_range.owner) { + mmu_notifier_range_init_owner(&hmm_vma_walk->mmu_range, MMU_NOTIFY_MIGRA= TE, 0, + walk->vma->vm_mm, start, end, + range->dev_private_owner); + mmu_notifier_invalidate_range_start(&hmm_vma_walk->mmu_range); + } + + return 0; +} + +static void hmm_vma_post_range_fault(struct hmm_vma_walk *hmm_vma_walk) +{ + + struct hmm_range *range =3D hmm_vma_walk->range; + + if (hmm_select_migrate(range) && + hmm_vma_walk->mmu_range.owner) { + /* + * The migrate_vma path has the following initialized, + * so take care of fault path below. + */ + if (range->notifier) { + hmm_fill_migrate_vma(range, + hmm_vma_walk->vma, + hmm_vma_walk->start, + hmm_vma_walk->end); + } + mmu_notifier_invalidate_range_end(&hmm_vma_walk->mmu_range); + } +} + static int hmm_vma_walk_pmd(pmd_t *pmdp, unsigned long start, unsigned long end, @@ -600,6 +657,11 @@ static int hmm_vma_walk_test(unsigned long start, unsi= gned long end, struct hmm_vma_walk *hmm_vma_walk =3D walk->private; struct hmm_range *range =3D hmm_vma_walk->range; struct vm_area_struct *vma =3D walk->vma; + int r; + + r =3D hmm_vma_capture_migrate_range(start, end, walk); + if (r) + return r; =20 if (!(vma->vm_flags & (VM_IO | VM_PFNMAP)) && vma->vm_flags & VM_READ) @@ -662,16 +724,25 @@ int hmm_range_fault(struct hmm_range *range) .range =3D range, .last =3D range->start, }; - struct mm_struct *mm =3D range->notifier->mm; + /* + * Could be serving a device fault or come from migrate + * entry point. For the former we have not resolved the vma + * yet, and the latter we don't have a notifier (but have a vma). + * + */ + struct mm_struct *mm =3D hmm_range_fault_mm(range); int ret; =20 mmap_assert_locked(mm); =20 do { /* If range is no longer valid force retry. */ - if (mmu_interval_check_retry(range->notifier, - range->notifier_seq)) - return -EBUSY; + if (range->notifier && mmu_interval_check_retry(range->notifier, + range->notifier_seq)) { + ret =3D -EBUSY; + break; + } + ret =3D walk_page_range(mm, hmm_vma_walk.last, range->end, &hmm_walk_ops, &hmm_vma_walk); /* @@ -681,6 +752,9 @@ int hmm_range_fault(struct hmm_range *range) * output, and all >=3D are still at their input values. */ } while (ret =3D=3D -EBUSY); + + hmm_vma_post_range_fault(&hmm_vma_walk); + return ret; } EXPORT_SYMBOL(hmm_range_fault); --=20 2.55.0 From nobody Fri Oct 2 07:45:30 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D05B39FCD8 for ; Tue, 4 Aug 2026 04:27:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817636; cv=none; b=gcPPDxrYxQAxPN5tlIPiXBl+SM4Hp5RrT0+NsvSZNpDWCgoHkr3j3XhAu5p4nYf3mQP9gh1DMugJw8YDMmfX2naVH/TXmjpDrtl16fsJajYA9p06hocxitYDUP+S3lSwQGWOkjaZ3Cn/sYzsXw6PWIWDyZoR2L3yrRRZ9pHNCmM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817636; c=relaxed/simple; bh=j/JLaEjLi+f6LpDWhpx27RFftj4iZYjqrWGSd2DasN0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=PQ8WcSxvjykHRbcMGDvPTdp35BjP1ipMK8j/txMVGPtuRMiGz8oQk63/oMZ4xAqITrtQKHq4FLTxdO13A8bsaMXTpuClKAN+V52SETGxqNUBMJDNBAaBCwkzUBOe6jH0V4PkWIeNNGASFlqrG9dX/nDUk2+K838Dp+f+jHsRt7k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=bNeuss6m; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=m0WyVpoQ; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="bNeuss6m"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="m0WyVpoQ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1785817633; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XfGZlzkw7Q/Ct5X5pPytInsf700IjSbqTVJGP4dmkUY=; b=bNeuss6mCGur5yBWo6R9k+Bce8lQaaOOAIWQI/y2yahhYCpFpDQRpD3Urgx7sTAkL88JSa Y+Qk/gWhwl6RDLlsarvJK/6qEnTbGZY99pXsATyUl05ZeOfQe25hSp5TPaN8F/xnnLsj7H yZ0mvWknpkyzZr9b8gzZrY+Ho30MX+I= Received: from mail-lf1-f71.google.com (mail-lf1-f71.google.com [209.85.167.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-391-Al9T5CiCPAac6QTR7HzLgQ-1; Tue, 04 Aug 2026 00:27:11 -0400 X-MC-Unique: Al9T5CiCPAac6QTR7HzLgQ-1 X-Mimecast-MFC-AGG-ID: Al9T5CiCPAac6QTR7HzLgQ_1785817630 Received: by mail-lf1-f71.google.com with SMTP id 2adb3069b0e04-5b01965458eso2789243e87.0 for ; Mon, 03 Aug 2026 21:27:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1785817630; x=1786422430; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=XfGZlzkw7Q/Ct5X5pPytInsf700IjSbqTVJGP4dmkUY=; b=m0WyVpoQ3izPErhszOAfkXeaJaYDcVAWxXKeZDhA1R1zUhbzWBudX80FSNy1P2QVII rxwzhIJxK+yaPj6MQ0dTjGcD09cg4uxWsxlTr5s5lEs6Exew3pahvdPDkikXveTs5rUg d1rHAJinitfTMSOCzc2d+p55BzyZxhMzwtZyazfBF6s7BwZWMA+F6k4Yc3PSMw7lO+1n tSlxLRQgFgcsb67qCgm1U3c/fUsfdOwebZtQEkKRvVhMIqmussBwpE2rlp/0mScS7D6N +/LeuVYhakuBw3M1Pp3aU6gHpo/OPsoeFOucqq3alm8wdyh1uKYRiT6/363iLEvtwPK0 dtGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785817630; x=1786422430; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=XfGZlzkw7Q/Ct5X5pPytInsf700IjSbqTVJGP4dmkUY=; b=jOuzdN23akEzH9Awzb17paYvbx+IbN9wpN7bkm0zQbPRwX9r6Xgxr36vYaeDO7wj2l v6xIjPRM8sWrKlc7PovRsxri2PBaxZoIQCeEy6nCH1ZpOt5giLBWpiA2vIvt6ZG1p8QS 1CzR2x4rSyATRxnBN7jlTGW6qC0hYd+9dq9SkzpjNSTi1VEHIpDi7qs4NyuoFx286Rp9 gPm0R/TiVbfR6sIb0e93HyLareEaeSXEiuazhwsIRL4tREW1dUnG2jUgq9xtARaLE3zD GI08HPYxMJEmuagYJ/QyuYkEJZbeWzonrDvTk92xfG2Lr5RLwvsZpABAj+0nzL2+Vkuv XMkw== X-Forwarded-Encrypted: i=1; AHgh+RpqYzGoIwo4Lu3wLeHsqQxfgmxfcmShjxuoAiqdpeQ9e9p9GsVCHkyLzWzLGMwsZAVNEtF8BVuQ5GzM5zs=@vger.kernel.org X-Gm-Message-State: AOJu0YzsLxoZphqR5EyDvghUbv8RAA0BXAJfY9esXOQmk2R0fdj75RsC rqW2VvgkvODhjLGGApzWhPBPoMusopeOaiooUWI+didzLXHc/AVGWTwpgOOBZvDGWY+uu/S9Adi ZHtRvRrUrIs2/6iBw5eAkxAgkMezOK5G5W9lXnCp7wliNl1JB6OY0LizGLcgfFawR X-Gm-Gg: AR+sD10Gm2t2RGV9eLFJeEjVvT7Qv7O1VI4v75tQG42pJd8nPIfTNXt08BgUPuBh+/T Wo9YZ5+HSRViA+wbLKYwS/Orb0nlCqTRoTWeggTaUBJT3Kp4KxDS+5g2ai4dmSlqFgdf2Vstjmj v8AXTurmS+W3pV0HBawGbcvUlqY+NhFX/jDHyPzD2K8tnMIJda58ULAthTnxh3ZhUsVAT9uW4cv yzoUvNxqABKFZiD6Dk14htoY7YMtt1nQ1/e9sT1rcald8/h5c8/j9jiinBSq9XtQ01Eh9g/oeIl TTCYTgNO0N74Hwp7+yMBoaeCuNkm86MAl3aRuKdy3aQnSl2D/gQeXhfu2uXI6bvlhIXpad50g7U JrcD3fS3G5Iqm+Pwp X-Received: by 2002:ac2:55a1:0:b0:5b1:5ab5:e9f7 with SMTP id 2adb3069b0e04-5b2e4f6d5b9mr1644428e87.39.1785817629706; Mon, 03 Aug 2026 21:27:09 -0700 (PDT) X-Received: by 2002:ac2:55a1:0:b0:5b1:5ab5:e9f7 with SMTP id 2adb3069b0e04-5b2e4f6d5b9mr1644415e87.39.1785817629185; Mon, 03 Aug 2026 21:27:09 -0700 (PDT) Received: from fedora (85-23-51-1.bb.dnainternet.fi. [85.23.51.1]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b2e23c46c4sm2311351e87.17.2026.08.03.21.27.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Aug 2026 21:27:06 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v13 04/11] mm/hmm: do the plumbing for HMM to participate in migration Date: Tue, 4 Aug 2026 07:26:24 +0300 Message-ID: <20260804042631.2175585-5-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260804042631.2175585-1-mpenttil@redhat.com> References: <20260804042631.2175585-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 Do the preparations in hmm_range_fault() and pagewalk callbacks to do the "collecting" part of migration, needed for migration. These steps include locking for pmd/pte if migrating, and calling the still dummy hmm_vma_handle_migrate_prepare_pmd() and hmm_vma_handle_migrate_prepare() functions in the pagewalk. When doing migration, have to have pmd/pte table locked for the duration of both hmm_vma_handle*() and hmm_vma_handle_migrate_prepare*() walks, and unlock when handling the faults. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- mm/hmm.c | 360 +++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 309 insertions(+), 51 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index 637c00188b3e..75e19be6ef76 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -39,8 +39,33 @@ struct hmm_vma_walk { unsigned long start; unsigned long end; unsigned long last; + /* + * For migration we need pte/pmd + * locked for the handle_* and + * prepare_* regions. While faulting + * we have to drop the locks and + * start again. + * ptelocked and pmdlocked + * hold the state and tells if need + * to drop locks before faulting. + * ptl is the lock held for pte or pmd. + * + */ + bool ptelocked; + bool pmdlocked; + spinlock_t *ptl; }; =20 +#define HMM_ASSERT_PTE_LOCKED(hmm_vma_walk, locked) \ + WARN_ON_ONCE((hmm_vma_walk)->ptelocked !=3D locked) + +#define HMM_ASSERT_PMD_LOCKED(hmm_vma_walk, locked) \ + WARN_ON_ONCE((hmm_vma_walk)->pmdlocked !=3D locked) + +#define HMM_ASSERT_UNLOCKED(hmm_vma_walk) \ + WARN_ON_ONCE((hmm_vma_walk)->ptelocked || \ + (hmm_vma_walk)->pmdlocked) + enum { HMM_NEED_FAULT =3D 1 << 0, HMM_NEED_WRITE_FAULT =3D 1 << 1, @@ -54,14 +79,38 @@ enum { }; =20 static int hmm_pfns_fill(unsigned long addr, unsigned long end, - struct hmm_range *range, unsigned long cpu_flags) + struct hmm_vma_walk *hmm_vma_walk, unsigned long cpu_flags) { + struct hmm_range *range =3D hmm_vma_walk->range; unsigned long i =3D (addr - range->start) >> PAGE_SHIFT; + enum migrate_vma_info minfo; + bool migrate =3D false; + + minfo =3D hmm_select_migrate(range); + if (cpu_flags !=3D HMM_PFN_ERROR) { + if (minfo && (vma_is_anonymous(hmm_vma_walk->vma))) { + cpu_flags |=3D HMM_PFN_MIGRATE; + migrate =3D true; + } + } + + if (migrate && thp_migration_supported() && + (minfo & MIGRATE_VMA_SELECT_COMPOUND) && + IS_ALIGNED(addr, HPAGE_PMD_SIZE) && + IS_ALIGNED(end, HPAGE_PMD_SIZE) && + end-addr =3D=3D HPAGE_PMD_SIZE) { + range->hmm_pfns[i] &=3D HMM_PFN_INOUT_FLAGS; + range->hmm_pfns[i] |=3D cpu_flags | HMM_PFN_COMPOUND; + addr +=3D PAGE_SIZE; + i++; + cpu_flags =3D 0; + } =20 for (; addr < end; addr +=3D PAGE_SIZE, i++) { range->hmm_pfns[i] &=3D HMM_PFN_INOUT_FLAGS; range->hmm_pfns[i] |=3D cpu_flags; } + return 0; } =20 @@ -84,6 +133,7 @@ static int hmm_vma_fault(unsigned long addr, unsigned lo= ng end, unsigned int fault_flags =3D FAULT_FLAG_REMOTE; =20 WARN_ON_ONCE(!required_fault); + HMM_ASSERT_UNLOCKED(hmm_vma_walk); hmm_vma_walk->last =3D addr; =20 if (required_fault & HMM_NEED_WRITE_FAULT) { @@ -177,11 +227,16 @@ static int hmm_vma_walk_hole(unsigned long addr, unsi= gned long end, if (!walk->vma) { if (required_fault) return -EFAULT; - return hmm_pfns_fill(addr, end, range, HMM_PFN_ERROR); + return hmm_pfns_fill(addr, end, hmm_vma_walk, HMM_PFN_ERROR); } - if (required_fault) + if (required_fault) { + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } return hmm_vma_fault(addr, end, required_fault, walk); - return hmm_pfns_fill(addr, end, range, 0); + } + return hmm_pfns_fill(addr, end, hmm_vma_walk, 0); } =20 static inline unsigned long hmm_pfn_flags_order(unsigned long order) @@ -214,8 +269,13 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, un= signed long addr, cpu_flags =3D pmd_to_hmm_pfn_flags(range, pmd); required_fault =3D hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, cpu_flags); - if (required_fault) + if (required_fault) { + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } return hmm_vma_fault(addr, end, required_fault, walk); + } =20 pfn =3D pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); for (i =3D 0; addr < end; addr +=3D PAGE_SIZE, i++, pfn++) { @@ -295,14 +355,25 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, u= nsigned long addr, goto fault; =20 if (softleaf_is_migration(entry)) { - pte_unmap(ptep); - hmm_vma_walk->last =3D addr; - migration_entry_wait(walk->mm, pmdp, addr); - return -EBUSY; + if (!hmm_select_migrate(range)) { + HMM_ASSERT_UNLOCKED(hmm_vma_walk); + pte_unmap(ptep); + hmm_vma_walk->last =3D addr; + migration_entry_wait(walk->mm, pmdp, addr); + return -EBUSY; + } + return 0; } =20 /* Report error for everything else */ - pte_unmap(ptep); + + if (hmm_vma_walk->ptelocked) { + pte_unmap_unlock(ptep, hmm_vma_walk->ptl); + hmm_vma_walk->ptelocked =3D false; + } else { + pte_unmap(ptep); + } + return -EFAULT; } =20 @@ -319,7 +390,13 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, un= signed long addr, if (!vm_normal_page(walk->vma, addr, pte) && !is_zero_pfn(pte_pfn(pte))) { if (hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0)) { - pte_unmap(ptep); + if (hmm_vma_walk->ptelocked) { + pte_unmap_unlock(ptep, hmm_vma_walk->ptl); + hmm_vma_walk->ptelocked =3D false; + } else { + pte_unmap(ptep); + } + return -EFAULT; } new_pfn_flags =3D HMM_PFN_ERROR; @@ -332,7 +409,12 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, un= signed long addr, return 0; =20 fault: - pte_unmap(ptep); + if (hmm_vma_walk->ptelocked) { + pte_unmap_unlock(ptep, hmm_vma_walk->ptl); + hmm_vma_walk->ptelocked =3D false; + } else { + pte_unmap(ptep); + } /* Fault any virtual address we were asked to fault */ return hmm_vma_fault(addr, end, required_fault, walk); } @@ -376,13 +458,18 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *= walk, unsigned long start, required_fault =3D hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0); if (required_fault) { - if (softleaf_is_device_private(entry)) + if (softleaf_is_device_private(entry)) { + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } return hmm_vma_fault(addr, end, required_fault, walk); + } else return -EFAULT; } =20 - return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR); + return hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR); } #else static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long s= tart, @@ -390,15 +477,72 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *= walk, unsigned long start, pmd_t pmd) { struct hmm_vma_walk *hmm_vma_walk =3D walk->private; - struct hmm_range *range =3D hmm_vma_walk->range; unsigned long npages =3D (end - start) >> PAGE_SHIFT; =20 if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0)) return -EFAULT; - return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR); + return hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR); } #endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ =20 +#ifdef CONFIG_DEVICE_MIGRATION +static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk, + pmd_t *pmdp, + unsigned long start, + unsigned long end, + unsigned long *hmm_pfn) +{ + // TODO: implement migration entry insertion + return 0; +} + +static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk, + pmd_t *pmdp, + pte_t *ptep, + unsigned long addr, + unsigned long *hmm_pfn, + bool *unmapped) +{ + // TODO: implement migration entry insertion + return 0; +} + +static int hmm_vma_walk_split(pmd_t *pmdp, + unsigned long addr, + struct mm_walk *walk) +{ + // TODO : implement split + return 0; +} + +#else +static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk, + pmd_t *pmdp, + unsigned long start, + unsigned long end, + unsigned long *hmm_pfn) +{ + return 0; +} + +static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk, + pmd_t *pmdp, + pte_t *ptep, + unsigned long addr, + unsigned long *hmm_pfn, + bool *unmapped) +{ + return 0; +} + +static int hmm_vma_walk_split(pmd_t *pmdp, + unsigned long addr, + struct mm_walk *walk) +{ + return 0; +} +#endif + static int hmm_vma_capture_migrate_range(unsigned long start, unsigned long end, struct mm_walk *walk) @@ -457,46 +601,128 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, { struct hmm_vma_walk *hmm_vma_walk =3D walk->private; struct hmm_range *range =3D hmm_vma_walk->range; - unsigned long *hmm_pfns =3D - &range->hmm_pfns[(start - range->start) >> PAGE_SHIFT]; unsigned long npages =3D (end - start) >> PAGE_SHIFT; + struct mm_struct *mm =3D walk->vma->vm_mm; + unsigned long *hmm_pfns, *hmm_pfns_start; + enum migrate_vma_info minfo; unsigned long addr =3D start; + bool unmapped =3D false; + unsigned long i; pte_t *ptep; pmd_t pmd; + int r =3D 0; =20 + minfo =3D hmm_select_migrate(range); + hmm_pfns_start =3D &range->hmm_pfns[(start - range->start) >> PAGE_SHIFT]; again: - pmd =3D pmdp_get_lockless(pmdp); - if (pmd_none(pmd)) - return hmm_vma_walk_hole(start, end, -1, walk); + hmm_pfns =3D &range->hmm_pfns[(addr - range->start) >> PAGE_SHIFT]; + hmm_vma_walk->ptelocked =3D false; + hmm_vma_walk->pmdlocked =3D false; + + if (minfo) { + hmm_vma_walk->ptl =3D pmd_lock(mm, pmdp); + hmm_vma_walk->pmdlocked =3D true; + pmd =3D pmdp_get(pmdp); + } else + pmd =3D pmdp_get_lockless(pmdp); + + if (pmd_none(pmd)) { + r =3D hmm_vma_walk_hole(start, end, -1, walk); + + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } + return r; + } =20 if (thp_migration_supported() && pmd_is_migration_entry(pmd)) { - if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0)) { - hmm_vma_walk->last =3D addr; - pmd_migration_entry_wait(walk->mm, pmdp); - return -EBUSY; + if (!minfo) { + if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns_start, npages, 0)) { + hmm_vma_walk->last =3D addr; + pmd_migration_entry_wait(walk->mm, pmdp); + return -EBUSY; + } + for (i =3D 0; start < end; start +=3D PAGE_SIZE, i++) + hmm_pfns_start[i] &=3D HMM_PFN_INOUT_FLAGS; } - return hmm_pfns_fill(start, end, range, 0); + + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } + + return 0; } =20 - if (!pmd_present(pmd)) - return hmm_vma_handle_absent_pmd(walk, start, end, hmm_pfns, - pmd); + if (pmd_trans_huge(pmd) || !pmd_present(pmd)) { + if (!pmd_present(pmd)) { + r =3D hmm_vma_handle_absent_pmd(walk, start, end, hmm_pfns_start, + pmd); + // If not migrating we are done + if (r || !minfo) { + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } + return r; + } + } =20 - if (pmd_trans_huge(pmd)) { - /* - * No need to take pmd_lock here, even if some other thread - * is splitting the huge pmd we will get that event through - * mmu_notifier callback. - * - * So just read pmd value and check again it's a transparent - * huge or device mapping one and compute corresponding pfn - * values. - */ - pmd =3D pmdp_get_lockless(pmdp); - if (!pmd_trans_huge(pmd)) - goto again; + if (pmd_trans_huge(pmd)) { + /* + * No need to take pmd_lock here if not migrating, + * even if some other thread is splitting the huge + * pmd we will get that event through mmu_notifier callback. + * + * So just read pmd value and check again it's a transparent + * huge or device mapping one and compute corresponding pfn + * values. + */ + + if (!minfo) { + pmd =3D pmdp_get_lockless(pmdp); + if (!pmd_trans_huge(pmd)) + goto again; + } + + r =3D hmm_vma_handle_pmd(walk, start, end, hmm_pfns_start, pmd); + + // If not migrating we are done + if (r || !minfo) { + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } + return r; + } + } + + r =3D hmm_vma_handle_migrate_prepare_pmd(walk, pmdp, start, end, hmm_pfn= s_start); + + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } + + if (r =3D=3D -ENOENT) { + r =3D hmm_vma_walk_split(pmdp, addr, walk); + if (r) { + /* Split not successful, skip */ + return hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR); + } + + /* Split successful, reloop */ + hmm_vma_walk->last =3D addr; + return -EBUSY; + } + return r; =20 - return hmm_vma_handle_pmd(walk, addr, end, hmm_pfns, pmd); + } + + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; } =20 /* @@ -506,24 +732,56 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, * recover. */ if (pmd_bad(pmd)) { - if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0)) + if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns_start, npages, 0)) return -EFAULT; - return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR); + return hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR); } =20 - ptep =3D pte_offset_map(pmdp, addr); - if (!ptep) + if (minfo) { + ptep =3D pte_offset_map_lock(mm, pmdp, addr, &hmm_vma_walk->ptl); + if (ptep) + hmm_vma_walk->ptelocked =3D true; + } else { + ptep =3D pte_offset_map(pmdp, addr); + } + if (!ptep) { + addr =3D start; goto again; + } + for (; addr < end; addr +=3D PAGE_SIZE, ptep++, hmm_pfns++) { - int r; =20 r =3D hmm_vma_handle_pte(walk, addr, end, pmdp, ptep, hmm_pfns); if (r) { - /* hmm_vma_handle_pte() did pte_unmap() */ + /* hmm_vma_handle_pte() did pte_unmap() / pte_unmap_unlock */ return r; } + + r =3D hmm_vma_handle_migrate_prepare(walk, pmdp, ptep, addr, hmm_pfns, &= unmapped); + if (r =3D=3D -EAGAIN) { + HMM_ASSERT_UNLOCKED(hmm_vma_walk); + if (unmapped) { + flush_tlb_range(walk->vma, start, addr); + unmapped =3D false; + } + goto again; + } + if (r) { + hmm_pfns_fill(addr, end, hmm_vma_walk, HMM_PFN_ERROR); + break; + } + } + + if (unmapped) + flush_tlb_range(walk->vma, start, addr); + + if (hmm_vma_walk->ptelocked) { + pte_unmap_unlock(ptep - 1, hmm_vma_walk->ptl); + hmm_vma_walk->ptelocked =3D false; + } else { + pte_unmap(ptep - 1); } - pte_unmap(ptep - 1); + return 0; } =20 @@ -684,7 +942,7 @@ static int hmm_vma_walk_test(unsigned long start, unsig= ned long end, (end - start) >> PAGE_SHIFT, 0)) return -EFAULT; =20 - hmm_pfns_fill(start, end, range, HMM_PFN_ERROR); + hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR); =20 /* Skip this vma and continue processing the next vma. */ return 1; --=20 2.55.0 From nobody Fri Oct 2 07:45:30 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6BE6442046B for ; Tue, 4 Aug 2026 04:27:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817638; cv=none; b=lUToT8hEoGV0FqXLhnhRf9Vz8QAvoDrvnUcDuJuw+MKl2Qn/zh6vHwNwUb3qbHeiqpCFJ7Pj+0bCbDLznbKgtRlt/+I76o4GdJR//83W2hN5ofWPIr70CNxwEj9TfwRBEuEHx9fNG4zOUcOtKS+m9YNLjnICIGelz0l4I4g/yM4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817638; c=relaxed/simple; bh=0KEr+fo7jJPOTUONZ2E2IrL9hPA+w5fyK2+TRLhJAFA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=BgC3A3cvIs1JqOrItquvzFmOfNkF5uI3+Lw0VVHUSLCeW5QWYNONn87eqssxhf/aOp/CmEPleRYMpL4X3n6L2+VarSnOyMMmaV9CF2inAHgSL83D4rz0Yn5aGHcBt+Wf608NME+saXITFVodgUqPEFZuHseVaDsb47CjxyoCIIE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=TghJtegt; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=Qm46aNL/; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="TghJtegt"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="Qm46aNL/" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1785817636; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NWLoSmzvlWrra6TV4DFt42DXFKEy+9KqLNEl12wX4Kg=; b=TghJtegtvFHidVTTpSne5c6AvZMKCN40p55IVxlnZvnjJVpA+RdZvZ6iYvkpu6mK6clHMH 0vYgYhT1sxsVWF5QZjruAQLBSiaF7w+XsrufdQTEtvQNwffqGO2GPsWfSg2gULU/UaUHHt 0nLNMmJepJ4J9enkIxkooiAsCtfU+NQ= Received: from mail-lf1-f69.google.com (mail-lf1-f69.google.com [209.85.167.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-244-BjAylndcOS2ysWCl9WD4Ug-1; Tue, 04 Aug 2026 00:27:14 -0400 X-MC-Unique: BjAylndcOS2ysWCl9WD4Ug-1 X-Mimecast-MFC-AGG-ID: BjAylndcOS2ysWCl9WD4Ug_1785817633 Received: by mail-lf1-f69.google.com with SMTP id 2adb3069b0e04-5b29e7e8ab6so267859e87.0 for ; Mon, 03 Aug 2026 21:27:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1785817633; x=1786422433; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=NWLoSmzvlWrra6TV4DFt42DXFKEy+9KqLNEl12wX4Kg=; b=Qm46aNL/K84FeDpWTzA/dYtWaXSWnkcyi4XlnM0iap43sKkThtCNI1Q0z+bqnuT16Y t4fnDaS7Kt1bBirTdd4XrkyM0X6+yciCLUkqZzMzbdjphmXqSFDKWZDUWy0gJnTPPSCN 70bxgItLD06AA4cfVHmQJ//M2dET6SVLt/sIUlDoCPF0Qvm1P2+ToXUcVGSk772PcjNT 9qUfzRBB6CHKk2pmjnYoKshGV8jZzVIJXc9TX3Mh3flLA6LdflfIDiCUOvbzovkZ8pU/ ewcpUgReIpNh7lkRC5RjFdeVh0/HshBmL/KkH0Vt0pG4ZMJZdpQ44CwCWiE+mjg+x/Oc YwWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785817633; x=1786422433; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=NWLoSmzvlWrra6TV4DFt42DXFKEy+9KqLNEl12wX4Kg=; b=SMfPPhiEtXFGW0VjCIIT7GICm4Pr/OO0Y7KjntNre3wUVLNpjOtrMrIUMLrPzmSogs MxMvhcopFcs/20UYexQKIEexNGV54ET5GC9HulboNPPJF1vTmhiEVeceLfEf4RngPNRT xh8VBVaTZPbzOR9pDZGzOuKGfQSlyz9H9GRXTXFOJh8zWE34oYvCCqOzbWzBOLGCqRAv kp/MgwKlAZ4iQNBtTC3qYEnkkhQVejQ9QKpya9F9wEdcpCupMncwm9HWCMpv/eZvcDOd x58L8u8OHPG1e+od4ImovN3BehaVaNnHcpDRIRemD5R0Y8cYr9Tg16Be4ew3uCiclrDZ +W2A== X-Forwarded-Encrypted: i=1; AHgh+Roi0up1k10t17rZsfPN5NLF+1CRGxDBfuM2b7wrj1gHN23D1a/FDZsn+HhGc10ZYvhNa4uw75ykd/jc8BE=@vger.kernel.org X-Gm-Message-State: AOJu0Yw3mGD1LxxdQ77jXeHl6U2L9cDvJbToBtP+xt87wtEz1vBYw2Lm JzH+08fjVYc+333rZWx6tP1W7kW0upVfXybLquC7rHt2Xg2amdBw+z8TVs/luUFCRuiLtoPEiLB EypX2GKL2lPgGIVrGWMkG8+TGAJsxLfiJNu47O8crp07sSMi49PpcaIU9PiV9Unc/ X-Gm-Gg: AR+sD10GNCICk2oUNH5CELkkCIM31u2HANnESWA8IPaotwNsMSTVax4msAbIvNHzuml 9hSPFbnPLqab5WemPFFsabaIn4Wgn+XQ2wnljo4SbCGQVLvfFyqA5cQ1tlZ7XB7GjpiBOGIXSGd 868rGAZrK8oXNYsK0VzeJecTD5iFjSKqQMaQMPHTmEhwvnmlsyogF7eNI9TEY7K72teldTFktTj vsHuNCnJlOifd5e0boq4yghUKJV8BFitMpvOXVaF11UoSzcXLtPWLqtJP0lHcYqYrEOCsxJ6wAb Ny8LbBcEnrbHH5X9JCvB9P3tYpMtckEE06dXmOu4V9EBpBHlxAHcHp9vcHWW2k4mMVRQOMhMxbm e3IJbBh2f/qPSqq/q X-Received: by 2002:a05:6512:6ca:b0:5ae:b7c7:5334 with SMTP id 2adb3069b0e04-5b2ee622c99mr384781e87.17.1785817633349; Mon, 03 Aug 2026 21:27:13 -0700 (PDT) X-Received: by 2002:a05:6512:6ca:b0:5ae:b7c7:5334 with SMTP id 2adb3069b0e04-5b2ee622c99mr384770e87.17.1785817632836; Mon, 03 Aug 2026 21:27:12 -0700 (PDT) Received: from fedora (85-23-51-1.bb.dnainternet.fi. [85.23.51.1]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b2e23c46c4sm2311351e87.17.2026.08.03.21.27.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Aug 2026 21:27:10 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v13 05/11] mm/hmm: implement folio split for migrate needs in HMM pagewalk Date: Tue, 4 Aug 2026 07:26:25 +0300 Message-ID: <20260804042631.2175585-6-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260804042631.2175585-1-mpenttil@redhat.com> References: <20260804042631.2175585-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 Implement the helper to split PMD size folios during pagewalk. Splitting is needed if the start and end addresses are not PMD aligned, or setting up pmd migration entry fails. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- mm/hmm.c | 47 ++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 44 insertions(+), 3 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index 75e19be6ef76..3621d1df51d3 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -511,10 +511,51 @@ static int hmm_vma_walk_split(pmd_t *pmdp, unsigned long addr, struct mm_walk *walk) { - // TODO : implement split - return 0; -} + struct hmm_vma_walk *hmm_vma_walk =3D walk->private; + struct hmm_range *range =3D hmm_vma_walk->range; + struct migrate_vma *migrate =3D range->migrate; + struct folio *folio, *fault_folio; + spinlock_t *ptl; + int ret =3D 0; + + HMM_ASSERT_UNLOCKED(hmm_vma_walk); + + fault_folio =3D (migrate && migrate->fault_page) ? + page_folio(migrate->fault_page) : NULL; + + ptl =3D pmd_lock(walk->mm, pmdp); + if (unlikely(!pmd_trans_huge(*pmdp))) { + spin_unlock(ptl); + goto out; + } =20 + folio =3D pmd_folio(*pmdp); + if (is_huge_zero_folio(folio)) { + spin_unlock(ptl); + split_huge_pmd(walk->vma, pmdp, addr); + } else { + folio_get(folio); + spin_unlock(ptl); + + if (folio !=3D fault_folio) { + if (unlikely(!folio_trylock(folio))) { + folio_put(folio); + ret =3D -EBUSY; + goto out; + } + } else { + folio_put(folio); + } + + ret =3D split_folio(folio); + if (fault_folio !=3D folio) { + folio_unlock(folio); + folio_put(folio); + } + } +out: + return ret; +} #else static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk, pmd_t *pmdp, --=20 2.55.0 From nobody Fri Oct 2 07:45:30 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6904A33ADB3 for ; Tue, 4 Aug 2026 04:27:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817646; cv=none; b=g204i8Qh/TDETn8UZfCoL6eWeHwoQ+v/Y8rA4PpB0KGud36yv+uW26skQHORqDNERWomAALXdOK787P0nqFU0MJo4wtI/6n5m6TpAE00sWaTL0S9Vhmx6Uvcn8ugwwtaez0ykES8/nc5Q75jmC2pN3t7RR2BKIqj9tlWDnGs+Bo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817646; c=relaxed/simple; bh=d14mUJPIWhmWzAEhhRR5qBtIt+cpauPTZgkwIHlJNKw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=dVt3uhJf5jUIMsBStbDzKWdkIESRpQez8mq3sIYjWC7LiucR9w6WlBQpey7++4rIPPsdAYR3XWdkwJz8tZFLHroC+F7/fcoqvwygI0XolJw/Uhh5ItiLVOGSTcZ6yxOLko9wPlmv/B+JEq6tcE/5I25XeVw1KegeACb3MqDE91I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=XA4oCeQu; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=agJRxFMr; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="XA4oCeQu"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="agJRxFMr" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1785817643; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kzj2oXiMIPJc8DDmgyLO85pt6xHacr5zfN3Pc28UGUc=; b=XA4oCeQuBAJUV1zdBJCq6Tclzu0hk3CkSuMilqiTY2sMZ5NYEh5ujTMtHyZYhmGvVKtwM3 zoUutJEMKjrVuggkPkoHPNwQVN1KIFOdfBx2AJTUWlHEpQvkshftJ0ME7m8R8OJHiZylTR wepq6Yb1rUihbcR+cPYbjzCWx1IQfmo= Received: from mail-lf1-f69.google.com (mail-lf1-f69.google.com [209.85.167.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-103-G2qEPA72NvWBvPwjYBAejw-1; Tue, 04 Aug 2026 00:27:21 -0400 X-MC-Unique: G2qEPA72NvWBvPwjYBAejw-1 X-Mimecast-MFC-AGG-ID: G2qEPA72NvWBvPwjYBAejw_1785817640 Received: by mail-lf1-f69.google.com with SMTP id 2adb3069b0e04-5b14d51dae1so1537101e87.3 for ; Mon, 03 Aug 2026 21:27:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1785817640; x=1786422440; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=kzj2oXiMIPJc8DDmgyLO85pt6xHacr5zfN3Pc28UGUc=; b=agJRxFMrwlyGfgOBXPrTehQHvzBroM/XcR9H//qOAS0Kb7SJU/sZ1sCZZDlaV2YzuI F0FsIyNiMZ6UC3uF4fI/hg0Y5UDvxQ0dOmIEB78SEu6NqkMzGo3TmAb78qyjE5NqB7nN txvtVZpzxvHwm93zo00jmZvcWkKEkRobaiv1hb8XzcbXKJXPrZiIL8xYJlIP2kE2DGGK InIohrpyT8kfEBRcrPJRHxqN0ydz48/DaQ4prJYdgv00lGky2nHQ86dweGNH2mgYZpEp +pU7rmRc0Kf1O1uCuWOKE3tDLdmv2nsr91D1n2770a+gixui1s+Z14uZPIrbBFofDB60 DYIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785817640; x=1786422440; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=kzj2oXiMIPJc8DDmgyLO85pt6xHacr5zfN3Pc28UGUc=; b=keCpQC3wNW8MwsPMxq66kd+rNtzGAd9/7dmU0cRq4qu+oCRtP/01TcRUIyN0FIo+Dk NhCYxJnm5egcOWxd4dsuZKWGBEZAFCMoqd85uhSXgFfxzaKiUBdMIXjrTmvoTPfiY/Tc sa0uwXkX4Fmw+m6hdGVzkzMg8qIe/wV+1sK/TYoXhHB43ujzDvsfdBXh8VJk1hE+G4uY uvE6QG7cyXmKbzel0moAz+cwyB/kEtNqHWS7xchgihcUa8AJCr6KXuIiHsOZfdLLaVbK aml/7IzOknZdGFr/We2dKj9P72zQfJNxArWcOWfnKGLsYw8yqpHb3T+4PBhthsTbDKoZ Rd6w== X-Forwarded-Encrypted: i=1; AHgh+Rouoc6Ehish8EJZMovgytnu6+3GiB9lKxOt4ipAIpDseocc3Wgd8Zf+WIw0u/Cvr4iN1wcYwgiUgapgUYc=@vger.kernel.org X-Gm-Message-State: AOJu0YyL0QWWy6uqcxGRKtRNldvWxv7/9+GrwxmUvsqe01DPbur1z1IY Y2gakOBeUnH3p7uiJtib5gVZoEyryYp+ctaEYEfydYpxscZ2rQl9o/tVv/0AVfG1scNTUQA/lIk N8pbJjabNdtYGauDelUkLCXZtbJ+cUesOYqJeNGUKGLhkda7Nch6fgQo4CWrnvh29 X-Gm-Gg: AR+sD13aZMR2LUDwnyC3wRp07xeydq575uMdsQ4kwj2v/FS8bhSh/Y2sHifwnTmc0vf SEHffK6yg2H1b/sKJc6F9eh9kNB3VGWN7Tbap3dAAkmgqGZkvsCh0R7x5gAWpHnjj8jo3juCHVj xTrVAW2ITLjKIzDfJnc49V7+UOFhidjYQrmY61MWcZV8YkTzvO2UthQRzP4B90Ru5DSMe5WcAWO Xrs5c3qmm3BWsGiG2sTh6FX5dHPhaSmtlaDTO5TZfOpB17N/dQmKg5v1ga2rFkNaHoHeKdt+3aj cVDfW0B4dDvcjwwuIzAxRzud9V9n741onbgOYNw3sPEwJ9/+PbAJda22I2UYw+uZSPDqNTH3t2Q wvqB17XYoApvb/tKN X-Received: by 2002:ac2:532a:0:b0:5b1:56c7:5753 with SMTP id 2adb3069b0e04-5b2e4f8064emr1809690e87.47.1785817640027; Mon, 03 Aug 2026 21:27:20 -0700 (PDT) X-Received: by 2002:ac2:532a:0:b0:5b1:56c7:5753 with SMTP id 2adb3069b0e04-5b2e4f8064emr1809678e87.47.1785817639563; Mon, 03 Aug 2026 21:27:19 -0700 (PDT) Received: from fedora (85-23-51-1.bb.dnainternet.fi. [85.23.51.1]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b2e23c46c4sm2311351e87.17.2026.08.03.21.27.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Aug 2026 21:27:15 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v13 06/11] mm/hmm: migrate collection in HMM pagewalk - pte level Date: Tue, 4 Aug 2026 07:26:26 +0300 Message-ID: <20260804042631.2175585-7-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260804042631.2175585-1-mpenttil@redhat.com> References: <20260804042631.2175585-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 Implement the needed hmm_vma_handle_migrate_prepare() function which is mostly carried over from migrate_device.c's migrate_vma_collect_pmd() function. Also implement the migrate_vma_split_folio(), for splitting pte mapped large folios. It is also mostly from migrate_device.c, with care taken to reference folio before relasing page table lock. With HMM pagewalk based migration, the idea is that hmm_vma_handle_*() are responsible for faulting, and the pfn collecting part. hmm_vma_handle_migrate_prepare*() do the migration decisions (with HMM_PFN_MIGRATE), possibly split folios, and insert migration ptes/pmds. HMM pagewalk based migration is enabled in later commit, for now now hmm_select_migrate() returns 0. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- mm/hmm.c | 262 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 261 insertions(+), 1 deletion(-) diff --git a/mm/hmm.c b/mm/hmm.c index 3621d1df51d3..2d902b8bcd58 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -486,6 +486,63 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *w= alk, unsigned long start, #endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ =20 #ifdef CONFIG_DEVICE_MIGRATION +/** + * migrate_vma_split_folio() - Helper function to split a THP folio + * @folio: the folio to split + * @fault_page: struct page associated with the fault if any + * @hmm_vma_walk: walk in progress + * @ptep: pte_t * for unmap and unlock ptl + * + * Returns 0 on success + */ +static int migrate_vma_split_folio(struct folio *folio, + struct page *fault_page, + struct hmm_vma_walk *hmm_vma_walk, + pte_t *ptep) +{ + int ret; + struct folio *fault_folio =3D fault_page ? page_folio(fault_page) : NULL; + struct folio *new_fault_folio =3D NULL; + + if (folio !=3D fault_folio) + folio_get(folio); + + pte_unmap_unlock(ptep, hmm_vma_walk->ptl); + hmm_vma_walk->ptelocked =3D false; + + if (folio !=3D fault_folio) + folio_lock(folio); + + ret =3D split_folio(folio); + if (ret) { + if (folio !=3D fault_folio) { + folio_unlock(folio); + folio_put(folio); + } + return ret; + } + + new_fault_folio =3D fault_page ? page_folio(fault_page) : NULL; + + /* + * Ensure the lock is held on the correct + * folio after the split + */ + if (!new_fault_folio) { + folio_unlock(folio); + folio_put(folio); + } else if (folio !=3D new_fault_folio) { + if (new_fault_folio !=3D fault_folio) { + folio_get(new_fault_folio); + folio_lock(new_fault_folio); + } + folio_unlock(folio); + folio_put(folio); + } + + return 0; +} + static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk, pmd_t *pmdp, unsigned long start, @@ -496,6 +553,11 @@ static int hmm_vma_handle_migrate_prepare_pmd(const st= ruct mm_walk *walk, return 0; } =20 +/* + * Install migration entries if migration requested, either from fault + * or migrate paths. + * + */ static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk, pmd_t *pmdp, pte_t *ptep, @@ -503,8 +565,206 @@ static int hmm_vma_handle_migrate_prepare(const struc= t mm_walk *walk, unsigned long *hmm_pfn, bool *unmapped) { - // TODO: implement migration entry insertion + struct hmm_vma_walk *hmm_vma_walk =3D walk->private; + struct hmm_range *range =3D hmm_vma_walk->range; + struct migrate_vma *migrate =3D range->migrate; + struct mm_struct *mm =3D walk->vma->vm_mm; + struct folio *fault_folio =3D NULL; + enum migrate_vma_info minfo; + struct dev_pagemap *pgmap; + bool anon_exclusive; + struct folio *folio; + unsigned long pfn; + struct page *page; + softleaf_t entry; + pte_t pte, swp_pte; + bool writable =3D false; + + // Do we want to migrate at all? + minfo =3D hmm_select_migrate(range); + if (!minfo) + return 0; + + WARN_ON_ONCE(!migrate); + HMM_ASSERT_PTE_LOCKED(hmm_vma_walk, true); + + fault_folio =3D migrate->fault_page ? + page_folio(migrate->fault_page) : NULL; + + pte =3D ptep_get(ptep); + + if (pte_none(pte)) { + if (vma_is_anonymous(walk->vma)) { + *hmm_pfn &=3D HMM_PFN_INOUT_FLAGS; + *hmm_pfn |=3D HMM_PFN_MIGRATE; + goto out; + } + } + + if (!(hmm_pfn[0] & HMM_PFN_VALID)) + goto out; + + if (!pte_present(pte)) { + /* + * Only care about unaddressable device page special + * page table entry. Other special swap entries are not + * migratable, and we ignore regular swapped page. + */ + entry =3D softleaf_from_pte(pte); + if (!softleaf_is_device_private(entry)) + goto out; + + if (!(minfo & MIGRATE_VMA_SELECT_DEVICE_PRIVATE)) + goto out; + + page =3D softleaf_to_page(entry); + folio =3D page_folio(page); + if (folio->pgmap->owner !=3D migrate->pgmap_owner) + goto out; + + if (folio_test_large(folio)) { + int ret; + + ret =3D migrate_vma_split_folio(folio, + migrate->fault_page, + hmm_vma_walk, + ptep); + if (ret) + goto out_error; + return -EAGAIN; + } + + pfn =3D page_to_pfn(page); + if (softleaf_is_device_private_write(entry)) + writable =3D true; + } else { + pfn =3D pte_pfn(pte); + if (is_zero_pfn(pfn) && + (minfo & MIGRATE_VMA_SELECT_SYSTEM)) { + *hmm_pfn =3D HMM_PFN_MIGRATE; + goto out; + } + page =3D vm_normal_page(walk->vma, addr, pte); + if (page && !is_zone_device_page(page) && + !(minfo & MIGRATE_VMA_SELECT_SYSTEM)) { + goto out; + } else if (page && is_device_coherent_page(page)) { + pgmap =3D page_pgmap(page); + + if (!(minfo & + MIGRATE_VMA_SELECT_DEVICE_COHERENT) || + pgmap->owner !=3D migrate->pgmap_owner) + goto out; + } + + folio =3D page ? page_folio(page) : NULL; + if (folio && folio_test_large(folio)) { + int ret; + + ret =3D migrate_vma_split_folio(folio, + migrate->fault_page, + hmm_vma_walk, + ptep); + if (ret) + goto out_error; + return -EAGAIN; + } + + writable =3D pte_write(pte); + } + + if (!page || !page->mapping) + goto out; + + /* + * By getting a reference on the folio we pin it and that blocks + * any kind of migration. Side effect is that it "freezes" the + * pte. + * + * We drop this reference after isolating the folio from the lru + * for non device folio (device folio are not on the lru and thus + * can't be dropped from it). + */ + folio =3D page_folio(page); + folio_get(folio); + + /* + * We rely on folio_trylock() to avoid deadlock between + * concurrent migrations where each is waiting on the others + * folio lock. If we can't immediately lock the folio we fail this + * migration as it is only best effort anyway. + * + * If we can lock the folio it's safe to set up a migration entry + * now. In the common case where the folio is mapped once in a + * single process setting up the migration entry now is an + * optimisation to avoid walking the rmap later with + * try_to_migrate(). + */ + + if (fault_folio =3D=3D folio || folio_trylock(folio)) { + anon_exclusive =3D folio_test_anon(folio) && + PageAnonExclusive(page); + + if (pte_present(pte)) + flush_cache_page(walk->vma, addr, pfn); + + if (anon_exclusive) { + pte =3D ptep_clear_flush(walk->vma, addr, ptep); + + if (folio_try_share_anon_rmap_pte(folio, page)) { + set_pte_at(mm, addr, ptep, pte); + folio_unlock(folio); + folio_put(folio); + goto out; + } + } else { + pte =3D ptep_get_and_clear(mm, addr, ptep); + } + + if (pte_present(pte) && pte_dirty(pte)) + folio_mark_dirty(folio); + + /* Setup special migration page table entry */ + if (writable) + entry =3D make_writable_migration_entry(pfn); + else if (anon_exclusive) + entry =3D make_readable_exclusive_migration_entry(pfn); + else + entry =3D make_readable_migration_entry(pfn); + + if (pte_present(pte)) { + if (pte_young(pte)) + entry =3D make_migration_entry_young(entry); + if (pte_dirty(pte)) + entry =3D make_migration_entry_dirty(entry); + } + + swp_pte =3D swp_entry_to_pte(entry); + if (pte_present(pte)) { + if (pte_soft_dirty(pte)) + swp_pte =3D pte_swp_mksoft_dirty(swp_pte); + if (pte_uffd_wp(pte)) + swp_pte =3D pte_swp_mkuffd_wp(swp_pte); + } else { + if (pte_swp_soft_dirty(pte)) + swp_pte =3D pte_swp_mksoft_dirty(swp_pte); + if (pte_swp_uffd_wp(pte)) + swp_pte =3D pte_swp_mkuffd_wp(swp_pte); + } + + set_pte_at(mm, addr, ptep, swp_pte); + folio_remove_rmap_pte(folio, page, walk->vma); + folio_put(folio); + *hmm_pfn |=3D HMM_PFN_MIGRATE; + if (pte_present(pte)) + *unmapped =3D true; + } else { + folio_put(folio); + } +out: return 0; +out_error: + return -EFAULT; } =20 static int hmm_vma_walk_split(pmd_t *pmdp, --=20 2.55.0 From nobody Fri Oct 2 07:45:30 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CD70D346A11 for ; Tue, 4 Aug 2026 04:27:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817655; cv=none; b=QiW7RF8Zb3OVSchwrpOqr0BBY+e0P5j71jGGxlVCV1gU/5o4rHah4NqfYRsxOvel/PEAG7BvHC/1xJfMReMpDyuo0MTh8vtjV5MFuhUYgB2HWpkS2+3DBeW6oBNA2DDDSglsXZFdQPiTmTXOSdpckpHdngjaE1H8FL37Ry0qS34= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817655; c=relaxed/simple; bh=opHX033rfmO7a//8tZ3kVaD1IEXGm33r9foejSOn4Tc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=UWCOiArpPs9hjm2RZyoCldtjXkY2gxzbJEKXukcXrmP89B6rtlkLaeJ7P0g4NGoqlFH2LESeJczibq2dKwxfv6/TPlt/4JdcnP6Zp3NssS31mUrjqB0qxLlQvW07s51BWbhibKL2TtR6waSeMYNv0+OpIqr/ie5YSK1Yw5eetgo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=gphsZ2Hk; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=AHU+/wL7; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="gphsZ2Hk"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="AHU+/wL7" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1785817652; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=p+Wrj+fEG23grALcs/D4e+T/chcmptdybXX8nN5bDvk=; b=gphsZ2HkT6CZ41KXOT0vTZy2FhZuMTtEMFnOqw8rR7XKBLDc/4U/g05pyH0vaabqOpSteQ /Gf5SzZL6+z3Im9N+5JFh/r0//zYu97ruHfCJdtbk0KeEu8TaUQ0Stp/CMQG2w/k9d92n0 or7ovJ92OxMRd1sdLVVLfudtiIe5sNE= Received: from mail-lj1-f199.google.com (mail-lj1-f199.google.com [209.85.208.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-642-H57AL0FaNISLLHJHyeJmJA-1; Tue, 04 Aug 2026 00:27:26 -0400 X-MC-Unique: H57AL0FaNISLLHJHyeJmJA-1 X-Mimecast-MFC-AGG-ID: H57AL0FaNISLLHJHyeJmJA_1785817645 Received: by mail-lj1-f199.google.com with SMTP id 38308e7fff4ca-39c82feaaa3so2176971fa.0 for ; Mon, 03 Aug 2026 21:27:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1785817645; x=1786422445; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=p+Wrj+fEG23grALcs/D4e+T/chcmptdybXX8nN5bDvk=; b=AHU+/wL7SFZZdRMmu+q6CQTXZVveOIiEMUqirp2NziIQTKoJczIJQmFt6ENDkak5zV CDziBnXatcBlMaRVmPD+3uTR1yCTrkOy0eOnvbCEe9Zb18XRN4ErC/uj0ZFrYgjxwwmF qFUw+JU9y4Bf9SrrDu9a5iDPHeak527y56bn1SEBHBZOrksvLbL26DthFeSAyxbCSTJW u9UxZE5x8y0+WRu92eWTRbDZy5EShVXcpn8lVpM9SgUAxE7wYTvZ44IrATtLaYgtTwoi hpEwXq6x06P4nq2+nZ08twJg67KW9HnYFxKHfH1rDx9pKdL4jPRlhz6bkXJ33qFOUJlq fEqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785817645; x=1786422445; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=p+Wrj+fEG23grALcs/D4e+T/chcmptdybXX8nN5bDvk=; b=FbnolZ1aY2WeY1aRwqMHZ7Yf0Wyz0HujOsCHJSiUCWO8vjmWz76HmDOD5nEoA2Nsfa jMSVjQFiPn6u4qFaT9ZzAN/VazYyJWuqTbr4Ub2LWPOUBC/UlFhA6JjuzbGv33KERE+w t+Vu3We0eNKNoBQtY1bgAIw0swPCC6jgD1JFfRwMd1fMkDLYj2wAifh/hZz3wYGAZuhN PgZ/jp1K+N9xhrvdvkREVqUtVp8AcxJhXMbn1WOgjVp2GcTHf/yCn6i2xAZWtU9aiITV 7DGV+MFju2n3USAWSVRjChbotwtCIrUO3GDTdkZ+HnWHnZeRBPOh4PtMzqeHlzl8Awl8 fzOg== X-Forwarded-Encrypted: i=1; AHgh+RpRjO/VuzLlx3jjZ/lAi9CMYClw7S6smk/aW+hCQYVQVJkyWdFIf7wSGnZP+ZddQBttDlXQTsBFkMV10Rk=@vger.kernel.org X-Gm-Message-State: AOJu0YziwKq7ns5n45o0csWUZ+fof46XxUyKtrpPwtTgONvLd87DgN2H zTYmxRhmspK+Vdm+Ne0jUXQzIhZnWZq+amzlCZuT4DS52j1QJpp5ZVtZM6y35sxYszYaa6uj0Z4 KSQa0iyWnvAGugso9CbA8WUAPS5okw7KrJZNMLOPfw4V1Hwd26J6GCjrOU0pGs+On6kj7hwrKH8 A= X-Gm-Gg: AR+sD12/A6sOAI//teF7Yd0VMTO7bP/LQqhmBbqSBPFFQnDpKJbgO1hxVn/yTuUJXvG 3NVW39Gt93Q1BNfbRAGNg8H0QO3etDu1QXjGeSN+uftMT6pwplFO17ALM+qpgTmANIA6rLIAa+a ioTVAaXDGKPwMPWvHx0tbsqp3INcvYvd0seSCuHoypcJqFhZB/BHRkVGuod5uA2ybOy30WPS7uf 9z5td2UKSoLD5T0okenamHcf+5SjNe6Q/iVJHlCGmOMtGvLdCQPdmjiUGuVV+UtQ1lSqoC1ZV6c sGaH0Q9UxrTujqEo4YuA3igsWnCtyZBDNRdmBT1La+mWRhsaHthKfzoU6sSiRUZUtna20hdGo2g 6+XtiHlox/TFAOdMo X-Received: by 2002:a05:6512:400a:b0:5b2:aa73:ea5c with SMTP id 2adb3069b0e04-5b2f282e27cmr151466e87.20.1785817644602; Mon, 03 Aug 2026 21:27:24 -0700 (PDT) X-Received: by 2002:a05:6512:400a:b0:5b2:aa73:ea5c with SMTP id 2adb3069b0e04-5b2f282e27cmr151455e87.20.1785817644120; Mon, 03 Aug 2026 21:27:24 -0700 (PDT) Received: from fedora (85-23-51-1.bb.dnainternet.fi. [85.23.51.1]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b2e23c46c4sm2311351e87.17.2026.08.03.21.27.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Aug 2026 21:27:22 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v13 07/11] mm/hmm: migrate collection in HMM pagewalk - pmd level Date: Tue, 4 Aug 2026 07:26:27 +0300 Message-ID: <20260804042631.2175585-8-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260804042631.2175585-1-mpenttil@redhat.com> References: <20260804042631.2175585-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 Implement the needed hmm_vma_handle_migrate_prepare_pmd() function which is mostly carried over from migrate_device.c's migrate_vma_collect_huge_pmd() function. With HMM pagewalk based migration, the idea is that hmm_vma_handle_*() are responsible for faulting, and the pfn collecting part. hmm_vma_handle_migrate_prepare*() do the migration decisions (with HMM_PFN_MIGRATE), possibly split folios, and insert migration ptes/pmds. HMM pagewalk based migration is enabled in later commit, for now now hmm_select_migrate() returns 0. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- mm/hmm.c | 96 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 94 insertions(+), 2 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index 2d902b8bcd58..61b9147fd8ed 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -549,8 +549,100 @@ static int hmm_vma_handle_migrate_prepare_pmd(const s= truct mm_walk *walk, unsigned long end, unsigned long *hmm_pfn) { - // TODO: implement migration entry insertion - return 0; + struct hmm_vma_walk *hmm_vma_walk =3D walk->private; + struct hmm_range *range =3D hmm_vma_walk->range; + struct migrate_vma *migrate =3D range->migrate; + struct folio *fault_folio =3D NULL; + enum migrate_vma_info minfo; + struct folio *folio; + unsigned long i; + int r =3D 0; + + // Do we want to migrate at all? + minfo =3D hmm_select_migrate(range); + if (!minfo) + return r; + + WARN_ON_ONCE(!migrate); + HMM_ASSERT_PMD_LOCKED(hmm_vma_walk, true); + + fault_folio =3D migrate->fault_page ? + page_folio(migrate->fault_page) : NULL; + + if (pmd_none(*pmdp)) + return hmm_pfns_fill(start, end, hmm_vma_walk, 0); + + if (!(hmm_pfn[0] & HMM_PFN_VALID)) + goto out; + + if (pmd_trans_huge(*pmdp)) { + if (!(minfo & MIGRATE_VMA_SELECT_SYSTEM)) + goto out; + + folio =3D pmd_folio(*pmdp); + if (is_huge_zero_folio(folio)) + return hmm_pfns_fill(start, end, hmm_vma_walk, 0); + + } else if (!pmd_present(*pmdp)) { + const softleaf_t entry =3D softleaf_from_pmd(*pmdp); + + if (!softleaf_is_device_private(entry)) + goto out; + + if (!(minfo & MIGRATE_VMA_SELECT_DEVICE_PRIVATE)) + goto out; + + folio =3D softleaf_to_folio(entry); + if (folio->pgmap->owner !=3D migrate->pgmap_owner) + goto out; + } else { + hmm_vma_walk->last =3D start; + return -EBUSY; + } + + folio_get(folio); + + if (folio !=3D fault_folio && unlikely(!folio_trylock(folio))) { + folio_put(folio); + hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR); + return 0; + } + + if (thp_migration_supported() && + (migrate->flags & MIGRATE_VMA_SELECT_COMPOUND) && + (IS_ALIGNED(start, HPAGE_PMD_SIZE) && + IS_ALIGNED(end, HPAGE_PMD_SIZE))) { + struct page_vma_mapped_walk pvmw =3D { + .ptl =3D hmm_vma_walk->ptl, + .address =3D start, + .pmd =3D pmdp, + .vma =3D walk->vma, + }; + + hmm_pfn[0] |=3D HMM_PFN_MIGRATE | HMM_PFN_COMPOUND; + + r =3D set_pmd_migration_entry(&pvmw, folio_page(folio, 0)); + if (r) { + hmm_pfn[0] &=3D ~(HMM_PFN_MIGRATE | HMM_PFN_COMPOUND); + r =3D -ENOENT; // fallback + goto unlock_out; + } + for (i =3D 1, start +=3D PAGE_SIZE; start < end; start +=3D PAGE_SIZE, i= ++) + hmm_pfn[i] &=3D HMM_PFN_INOUT_FLAGS; + + } else { + r =3D -ENOENT; // fallback + goto unlock_out; + } + +out: + return r; + +unlock_out: + if (folio !=3D fault_folio) + folio_unlock(folio); + folio_put(folio); + goto out; } =20 /* --=20 2.55.0 From nobody Fri Oct 2 07:45:30 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 30A7B343886 for ; Tue, 4 Aug 2026 04:27:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817654; cv=none; b=UqJQ9dhO16SjRecWSu+QRvlHwZCyDYCgix0UqKsblGUQHmUndcVw4fe2swodsnE33zataJaRvF4/TwI5bGNY1iTPO9pQZswh3Y69gjglXRMFFZvFHVCC0puaAWerzrDbmR4OaxhaXmBgDUuLSlafEl0vQuR859vCFCAalNrDhfs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817654; c=relaxed/simple; bh=t/Qv0M4+uVMUbM4q8mIjTJxHj9K9KeCjzK5Bua9wiQE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=hZyht7bRy8Jy7AmedeuSpQIoomahuS7n6I3MBC0Y2PzF6ENCpXyZBmRuCJfVyLp/fmiCWwiLxtiv3bFqDitF9jWk99yHnVXi4b/VEhxQPTvbKQzWIvswR48PbLqF3cmA8WKv8EgFz3qNeZfbqn1v7RRcFeQt02O8YR4XggQNji8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Lkj5FDX7; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=LrjVYUx/; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Lkj5FDX7"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="LrjVYUx/" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1785817652; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lxWvLz6f1rN1o4scxqJnXGGSQRdc4CCCHPneXFEhe/4=; b=Lkj5FDX7aTCGZr8HUVe4QfoBGDDtkJRQ8iQHkg2yTBD9nTNInzyceJ2sMMZgUem9aMsFNg ZgRyWuixMEHAcWAB1z8858ZjPREa8Xi7lM2sAXVLf0sSjN4vJG1dR599qKx2Kdj0+wwhLh 9GDTa+NEaUBiZ0Zecjh8irkjZQzzwjM= Received: from mail-lf1-f72.google.com (mail-lf1-f72.google.com [209.85.167.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-270-dB-l48N3Pb-jJ3OY0fth6w-1; Tue, 04 Aug 2026 00:27:30 -0400 X-MC-Unique: dB-l48N3Pb-jJ3OY0fth6w-1 X-Mimecast-MFC-AGG-ID: dB-l48N3Pb-jJ3OY0fth6w_1785817649 Received: by mail-lf1-f72.google.com with SMTP id 2adb3069b0e04-5b2d7c7ec76so1826114e87.1 for ; Mon, 03 Aug 2026 21:27:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1785817648; x=1786422448; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=lxWvLz6f1rN1o4scxqJnXGGSQRdc4CCCHPneXFEhe/4=; b=LrjVYUx/ngAnE64IvAe65ZsDujmCB18JmdWa26xOjp98aTk3YWx/SFtaZKVkCx0P/i FAbEClij4rdvoNNiAVyS1CZ4wXb+3SV2ptFcoNkDQPAIL0g4ptzCjb3YkHscx1vipSLs JtVao23B5QgARR/PLbz66VmSW2EmHXiTSbV14dmgFRfC27fGhkakIMT4R9P9MU/bdaxp wenfPhoDJ/HdZ4UUe2k0RfqkBooBZPTbKGZoBct+STdh//8eV5IK0oLgjlskgEcP7lei PaCfxkRaUftfn/2qIA0GAA3fMm0fqNY7+iwbBPVRRMitg/ZHww8y7eLimpmJQSFHmZDq quhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785817648; x=1786422448; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=lxWvLz6f1rN1o4scxqJnXGGSQRdc4CCCHPneXFEhe/4=; b=U56k5ByD0W+E7Ec9muS2KIhsaCxTHXT1wQ7hI4kx9MX32LIpfnlXE3rdYld59QB2Gj 3QGfzXfJVD61F7phA6Tur2whzIpya5uL1dd9TDizHus9g66FRzveczuyUsg5ELc2ddMc OyB8/h62hu73RgBTAfiAhHIazSEf9+TwDj7qBSBpVhqUo5srP2n7J/vDn4kpw1GQGqA4 n46nwSRg6YJ3mgDxnMGcaMVVaC3tF0oQQREF6lL9jpUhMgJbr/8OKkmhgmpXpvFBaMRu DaVLId0HwjzHhOt5IqvQsmswqglDqXnUSu+HSr5aVFe04qn0MEZrE6D5kwuGeTgfDB8Q UuOg== X-Forwarded-Encrypted: i=1; AHgh+RoJuU7Z5az4A8DdxmSw5ctAaoPYVRCiMSejwk4MrhoYL0w+aQUsXDGscfw19s0Dze1HhVBFgDpwKKzNufM=@vger.kernel.org X-Gm-Message-State: AOJu0YyYpIRvVZ/BK/Y+befGWnvX1nsjO3Jglppmvwrtik3//tcWqq/z 63qnpGovljS9iNWmeheA94gqDz4JuMAHUuZ82teiTXtsWhNomnlmxikexAUeO1+7qvfrO+Min4O GWN6LlCM0G+RtMnnWpgL8/Iqzhlkdqiq3jCZPxlFGvmqKp56GTV7GP94saseWcOP/qIT2ipUBdB 0= X-Gm-Gg: AR+sD119qlGrS1Iej9ux4bsKRiWIZHc16VIXukFKKHXGniqjeToVrAxgHHJrujRCO4U /3ABpRVddXImgmDRgvld7PsNL82hrqPWAwC2IL7fk/FVgHkEcoxcL42tk3qd63Qr/R6FF6uUvls 7e6jhwatiuZyb7i0gTgUN0aI8ApvcV4bzv2KaLmItExywJwjpz/hDZgchqvM6T/TU1VOJX1XdZL jvUUdNTTEJYtU+qru5OXzWC9WoDpR2H+VMmZY6DMShOvRutUK/mHvHXiPT/y1IQY5nrn3cfNNAO X9QtldocNK/t+Ruemh1q9M4fkyN3l5dT1r2+AGEktF6tF8h7fi/4ywyKwPDgSBxvQDWz+la7MIt uzVByqeWBuE7VfTxL X-Received: by 2002:a05:6512:8051:b0:5ae:b86b:71ee with SMTP id 2adb3069b0e04-5b2e4f6d5c4mr1858466e87.38.1785817648605; Mon, 03 Aug 2026 21:27:28 -0700 (PDT) X-Received: by 2002:a05:6512:8051:b0:5ae:b86b:71ee with SMTP id 2adb3069b0e04-5b2e4f6d5c4mr1858457e87.38.1785817648175; Mon, 03 Aug 2026 21:27:28 -0700 (PDT) Received: from fedora (85-23-51-1.bb.dnainternet.fi. [85.23.51.1]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b2e23c46c4sm2311351e87.17.2026.08.03.21.27.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Aug 2026 21:27:26 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v13 08/11] mm/hmm: add lazy MMU mode support for migration in HMM pagewalk Date: Tue, 4 Aug 2026 07:26:28 +0300 Message-ID: <20260804042631.2175585-9-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260804042631.2175585-1-mpenttil@redhat.com> References: <20260804042631.2175585-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 Add calls to lazy_mmu_mode_enable() and lazy_mmu_mode_disable() while doing migration. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- mm/hmm.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/hmm.c b/mm/hmm.c index 61b9147fd8ed..631f6b4d52fc 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -368,6 +368,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, uns= igned long addr, /* Report error for everything else */ =20 if (hmm_vma_walk->ptelocked) { + lazy_mmu_mode_disable(); pte_unmap_unlock(ptep, hmm_vma_walk->ptl); hmm_vma_walk->ptelocked =3D false; } else { @@ -391,6 +392,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, uns= igned long addr, !is_zero_pfn(pte_pfn(pte))) { if (hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0)) { if (hmm_vma_walk->ptelocked) { + lazy_mmu_mode_disable(); pte_unmap_unlock(ptep, hmm_vma_walk->ptl); hmm_vma_walk->ptelocked =3D false; } else { @@ -410,6 +412,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, uns= igned long addr, =20 fault: if (hmm_vma_walk->ptelocked) { + lazy_mmu_mode_disable(); pte_unmap_unlock(ptep, hmm_vma_walk->ptl); hmm_vma_walk->ptelocked =3D false; } else { @@ -507,6 +510,7 @@ static int migrate_vma_split_folio(struct folio *folio, if (folio !=3D fault_folio) folio_get(folio); =20 + lazy_mmu_mode_disable(); pte_unmap_unlock(ptep, hmm_vma_walk->ptl); hmm_vma_walk->ptelocked =3D false; =20 @@ -1132,8 +1136,10 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, =20 if (minfo) { ptep =3D pte_offset_map_lock(mm, pmdp, addr, &hmm_vma_walk->ptl); - if (ptep) + if (ptep) { + lazy_mmu_mode_enable(); hmm_vma_walk->ptelocked =3D true; + } } else { ptep =3D pte_offset_map(pmdp, addr); } @@ -1169,6 +1175,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, flush_tlb_range(walk->vma, start, addr); =20 if (hmm_vma_walk->ptelocked) { + lazy_mmu_mode_disable(); pte_unmap_unlock(ptep - 1, hmm_vma_walk->ptl); hmm_vma_walk->ptelocked =3D false; } else { --=20 2.55.0 From nobody Fri Oct 2 07:45:30 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C3BF935966 for ; Tue, 4 Aug 2026 04:27:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817665; cv=none; b=uNSgFHr/8x75USqCNhyFvrCmnNdhZ55jjdPkjfwYrY3tRJ9iFHcRB4xy0vblqTU9E5lEs/RQ0wds94U/XOP3iSl+oJaTb4zqU7E1JswVTbGufitJhl062NBtrjqgWC/A41UOBdBdFdmb0P0w7Y/a1sdom/GziiYgZCp7MwqlTp8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817665; c=relaxed/simple; bh=cXoQkyyXfXDi6EeAvKIQ+4F4h1tKWVLYSbGn0nVkvXo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=QK+/racbGgYNxgaURG4lx/svqNn8oGuFvvMI2F8R4tqmI8mcIStzPfVS2XAHCUjsGC3gmBQxJleNtdK/+YN3y0jNPlWT4JP0Zx63H3P7kOPIpmzMgAtPIxhTPd3LsFL0anx943IilSgXDNfC17u9wtu9TU6vg/yH5MgK6S3Ijac= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=VTqUA1aD; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=aIjCGpop; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="VTqUA1aD"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="aIjCGpop" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1785817662; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=csSbLn2HNgFXzmHTSRr7vcVb4Wk3QmqGPDut2K4sshw=; b=VTqUA1aDn+yI3lr5PlkXEBO9vUb6QudUNU1MLlwz8lQe1oQnPq3i8s8NMDS1Iqh3QgGocO 2KcSITw7sOqlWNuOzVrcSO3jXVmzZ/O4vzTzwewIcDXJ5m4O1mKaKkeMxOti/P8an0oyGj eG8sFoEl6v7+3qgKbn942eMeQixalvQ= Received: from mail-lf1-f70.google.com (mail-lf1-f70.google.com [209.85.167.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-459-HNmyUF5VMiyEtkkJQ8EZsg-1; Tue, 04 Aug 2026 00:27:34 -0400 X-MC-Unique: HNmyUF5VMiyEtkkJQ8EZsg-1 X-Mimecast-MFC-AGG-ID: HNmyUF5VMiyEtkkJQ8EZsg_1785817653 Received: by mail-lf1-f70.google.com with SMTP id 2adb3069b0e04-5b15484404cso2283758e87.0 for ; Mon, 03 Aug 2026 21:27:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1785817653; x=1786422453; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=csSbLn2HNgFXzmHTSRr7vcVb4Wk3QmqGPDut2K4sshw=; b=aIjCGpop5XjXRBsijO0mwA2P0BI0gv8s5R14MnBN4DrANgKCCrJWhePjAmTXI8jZyu jbLRpUMnY5R6jZw/BtHNK+mPR40UBsaUIX4nYzni0klPu6Avv7lyT8v2kC4zgiU/rcwo zWETNX/RYgEvx3RrdGupNEpPeVafcJ5r5iuyg+2LEOtAPBn4uEK+AtvXF6DmvDopi7VA dz/90tmS0SNlUAtzP7sc6wn49JvydnxK2iJrbV9Ogm0YQuFaKma6vlU5Z8kcYVjrRWvQ eFB4IvrLQYknMEBViOEnUJnif3PeSwQMhrOANoYLm7DVOiN48zuho1r7+Kb2RB+112+M t3Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785817653; x=1786422453; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=csSbLn2HNgFXzmHTSRr7vcVb4Wk3QmqGPDut2K4sshw=; b=ZfLJ257OaKmutrTdsj5uIkUWQ6s+cDyKtPUlWT7V23cntDb1UXXtsqC3SvjAcLRAuC pYeLsqkNNRUNOnctA5gT4e4RCMugbA3v1yrQAk9kZ8RGiKoq5wblak1bRKF+U0UHh1CB sC8a2lVi0jtHG8XfzCF2wyJeV3BovUfQgZZxT/a7FF/p+HKWkoATFnD1ss4ALfRW8G98 x98kW9bdwTTNmBtW8+hBDhLPejO/FHxlg5STUsMmiOMAvp4dlAVFJ3WQ7rCVDdOcCRE4 wWsDiB5H4CDNPQQSI1K3KL4rj19kBR72jzKlxJg0BhJwmLwHT5Z4XOT8ub4HUlQTeGiS /ctw== X-Forwarded-Encrypted: i=1; AHgh+Ropxk2E6vmNHCAfd9yDmamH9YaWV0P+jki5fLLag5iunRjmmng9jaEXx1FfLBZbwVU+gJdtpRVbfCd0XDY=@vger.kernel.org X-Gm-Message-State: AOJu0YwaIZo1ArtEAE4omG60+L9tQBhG8M1JB6CVb5YPOADtm0vUtySc 8j63WPB8aYIIN93pyLn9QSLTELcNFF3BBEh3q+7yiOwYjrMxQBu+3GsazokyJYv3tNFdf4Bs70n ZrH3l0rNBHj75noys7BQwlENpHmdiLRE945JShbE64+5h0Mr/R1Z1vatyl9ZmWUuvgtdB1wZZNl M= X-Gm-Gg: AR+sD10+cJmmnL0wTVTuTQVXMnwvp8tj2EN8za3GHCVQ/woKmxP+0fJTuDfffthyrb6 GeC7y/tvA63KFJ1icyqWfDTIQksVdU0oTbs87Rj5h9Zv5pj1Nw16X9f693Hvbwz+kuF6w0hPbGG uHE9bAzkqhop6DLp+Hk8dwz28+hOrZDtU0NG/Bloy2DLKWm2d99MYWFY5vywLKGjTRf4AdO6/8N Jgs265wpdydIMmPxYvHga/SkuQZGtk+ncwvkN3p2M8KhQD28aWPQQHATXSMLnt3BOvlhPRE2yX7 ncJku747/diy1ndVFmAg7IS366VKcnbjIzukc5aJbpuxp7Qn1yEocow00Hqn19Kxd5duc7ILGWw ThCAlMqcKtbw3u4uB X-Received: by 2002:a05:6512:1386:b0:5b1:4e5e:559c with SMTP id 2adb3069b0e04-5b2f2391564mr119394e87.30.1785817652763; Mon, 03 Aug 2026 21:27:32 -0700 (PDT) X-Received: by 2002:a05:6512:1386:b0:5b1:4e5e:559c with SMTP id 2adb3069b0e04-5b2f2391564mr119387e87.30.1785817652298; Mon, 03 Aug 2026 21:27:32 -0700 (PDT) Received: from fedora (85-23-51-1.bb.dnainternet.fi. [85.23.51.1]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b2e23c46c4sm2311351e87.17.2026.08.03.21.27.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Aug 2026 21:27:30 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v13 09/11] mm/hmm: implement rollback for device page migration in HMM pagewalk Date: Tue, 4 Aug 2026 07:26:29 +0300 Message-ID: <20260804042631.2175585-10-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260804042631.2175585-1-mpenttil@redhat.com> References: <20260804042631.2175585-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 During the migration pagewalk, the PTE table could be cleared and/or changed into PMD leaf or even another PTE table while dropped locks. In these cases the possibly inserted migration ptes are gone. We have to however undo the collecting done so far, so unlock the folios and drop reference taken. During the pagewalk we notice such scenarios if going to recollect a pfn but have already committed to migrate the entry with HMM_PFN_MIGRATE, in which case rollback. If we encounter migration ptes they are just skipped to allow for restart own walks. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- include/linux/hmm.h | 22 +++++++++++++ mm/hmm.c | 76 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 98 insertions(+) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index bfedafc1c143..fbc9bb74d160 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -111,6 +111,28 @@ static inline unsigned int hmm_pfn_to_map_order(unsign= ed long hmm_pfn) return (hmm_pfn >> HMM_PFN_ORDER_SHIFT) & 0x1F; } =20 +/* + * hmm_pfn_collected() - is this pfn entry prepared for migration ? + * If collected the folio's refcount is increased and the folio + * is locked. + */ +static inline bool hmm_pfn_collected(unsigned long hmm_pfn) +{ + return (hmm_pfn & (HMM_PFN_VALID | HMM_PFN_MIGRATE)) =3D=3D + (HMM_PFN_VALID | HMM_PFN_MIGRATE); +} + +/* + * hmm_pfn_rollback() - undoes the collecction of hmm_pfn + * + * Note for total rollback the folio's refcount has to be put + * and folio has to be unlocked. + */ +static inline unsigned long hmm_pfn_rollback_collected(unsigned long hmm_p= fn) +{ + return hmm_pfn & ~(HMM_PFN_VALID | HMM_PFN_MIGRATE | HMM_PFN_COMPOUND); +} + /* * struct hmm_range - track invalidation lock on virtual address range * diff --git a/mm/hmm.c b/mm/hmm.c index 631f6b4d52fc..03622455302a 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -78,6 +78,11 @@ enum { HMM_PFN_P2PDMA_BUS, }; =20 +static void hmm_vma_handle_migrate_prepare_rollback(const struct hmm_vma_w= alk *hmm_vma_walk, + unsigned long start, + unsigned long end, + unsigned long *hmm_pfn); + static int hmm_pfns_fill(unsigned long addr, unsigned long end, struct hmm_vma_walk *hmm_vma_walk, unsigned long cpu_flags) { @@ -94,6 +99,8 @@ static int hmm_pfns_fill(unsigned long addr, unsigned lon= g end, } } =20 + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr, end, &range->= hmm_pfns[i]); + if (migrate && thp_migration_supported() && (minfo & MIGRATE_VMA_SELECT_COMPOUND) && IS_ALIGNED(addr, HPAGE_PMD_SIZE) && @@ -277,6 +284,8 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, uns= igned long addr, return hmm_vma_fault(addr, end, required_fault, walk); } =20 + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr, + end, hmm_pfns); pfn =3D pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); for (i =3D 0; addr < end; addr +=3D PAGE_SIZE, i++, pfn++) { hmm_pfns[i] &=3D HMM_PFN_INOUT_FLAGS; @@ -407,6 +416,9 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, uns= igned long addr, =20 new_pfn_flags =3D pte_pfn(pte) | cpu_flags; out: + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr, + addr + PAGE_SIZE, + hmm_pfn); *hmm_pfn =3D (*hmm_pfn & HMM_PFN_INOUT_FLAGS) | new_pfn_flags; return 0; =20 @@ -445,6 +457,9 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *wa= lk, unsigned long start, if (softleaf_is_device_private_write(entry)) cpu_flags |=3D HMM_PFN_WRITE; =20 + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, + start, end, + hmm_pfns); /* * Fully populate the PFN list though subsequent PFNs could be * inferred, because drivers which are not yet aware of large @@ -547,6 +562,48 @@ static int migrate_vma_split_folio(struct folio *folio, return 0; } =20 +/* + * Due to dropping ptl locks for splitting for instance, would we + * overwrite already collected pfns? This could happen when pmd + * pointing to a page table has vanished and been replaced + * with a leaf pmd, or another page table. + * In that case unref and unlock the folios, + * the pfns of which were collected from the disappeared + * page tables. + */ +static void hmm_vma_handle_migrate_prepare_rollback(const struct hmm_vma_w= alk *hmm_vma_walk, + unsigned long start, + unsigned long end, + unsigned long *hmm_pfn) +{ + struct hmm_range *range =3D hmm_vma_walk->range; + struct migrate_vma *migrate =3D range->migrate; + struct folio *fault_folio =3D NULL; + enum migrate_vma_info minfo; + struct folio *folio; + unsigned long i; + + minfo =3D hmm_select_migrate(range); + if (!minfo) + return; + + WARN_ON_ONCE(!migrate); + + fault_folio =3D migrate->fault_page ? + page_folio(migrate->fault_page) : NULL; + + for (i =3D 0; start < end; start +=3D PAGE_SIZE, i++) { + if (hmm_pfn_collected(hmm_pfn[i])) { + folio =3D page_folio(hmm_pfn_to_page(hmm_pfn[i])); + if (folio !=3D fault_folio) + folio_unlock(folio); + folio_put(folio); + hmm_pfn[i] =3D hmm_pfn_rollback_collected(hmm_pfn[i]); + + } + } +} + static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk, pmd_t *pmdp, unsigned long start, @@ -690,6 +747,11 @@ static int hmm_vma_handle_migrate_prepare(const struct= mm_walk *walk, pte =3D ptep_get(ptep); =20 if (pte_none(pte)) { + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, + addr, + addr + PAGE_SIZE, + hmm_pfn); + if (vma_is_anonymous(walk->vma)) { *hmm_pfn &=3D HMM_PFN_INOUT_FLAGS; *hmm_pfn |=3D HMM_PFN_MIGRATE; @@ -737,6 +799,10 @@ static int hmm_vma_handle_migrate_prepare(const struct= mm_walk *walk, pfn =3D pte_pfn(pte); if (is_zero_pfn(pfn) && (minfo & MIGRATE_VMA_SELECT_SYSTEM)) { + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, + addr, + addr + PAGE_SIZE, + hmm_pfn); *hmm_pfn =3D HMM_PFN_MIGRATE; goto out; } @@ -913,6 +979,13 @@ static int hmm_vma_walk_split(pmd_t *pmdp, return ret; } #else +static void hmm_vma_handle_migrate_prepare_rollback(const struct hmm_vma_w= alk *hmm_vma_walk, + unsigned long start, + unsigned long end, + unsigned long *hmm_pfn) +{ +} + static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk, pmd_t *pmdp, unsigned long start, @@ -1139,6 +1212,9 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, if (ptep) { lazy_mmu_mode_enable(); hmm_vma_walk->ptelocked =3D true; + } else { + /* The pte table is gone */ + hmm_vma_handle_migrate_prepare_rollback(walk->private, addr, end, hmm_p= fns); } } else { ptep =3D pte_offset_map(pmdp, addr); --=20 2.55.0 From nobody Fri Oct 2 07:45:30 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E62CE33688B for ; Tue, 4 Aug 2026 04:27:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817667; cv=none; b=CACLxyM3fFDy+uRK9/4sbtg6Kz34AOt+4mw0CYCBUNR9L6XA9qULjUrSI8zsmkPIC7eTVxiCJI5rz6446ZSwz68Yn/gk/9kKtsGW+jqHtekB+5DjQ3mq0KU/uQnPnWFhsKniC5c/w+LD5CIT9PvGjXKF/GuzRVlJ7JMZpUEWQzc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817667; c=relaxed/simple; bh=rwPeoqy+QPwiRCyxWTk//1L4E++k4S4AczoCx6+zoCo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=iQZyOmwbyErFfm3PXIDhTM5A4Sj+W7nEgRnR4qT43xH5IxKRURWR4Ss6VTLFgPRElNpQe+mn9rR78NH+AJG4Sv6ViknhJySffakoeqG2tMJZj6tuKbjbwutaPlIt29LS/Y7Wb5Lo5v42LLagEID5B5GSzD5sClxzW8BCDLE6STY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=QTZyObRO; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=NoIdsyJO; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="QTZyObRO"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="NoIdsyJO" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1785817663; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tpEIqxoBbLX0SvQdg20oXD/NXhm9ONjr2trVRywE48w=; b=QTZyObRO2msfnFju+QYuSY/ONORFn3D40iB1Ybiin5++mv3BEqsK0K6bwuAtdAjYwrwSnV n/gryAxQvtKzzRTr9VDzpw2OxjKn7gHvMCo8JOO3CbdrT28QSJtP06r7z6vdZcXZVd8R7F nlsyiDdSyRhi35cr8dWUCXUhnJB4zSw= Received: from mail-lf1-f69.google.com (mail-lf1-f69.google.com [209.85.167.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-133-2dNAaRiTMweV3evsn2R3EQ-1; Tue, 04 Aug 2026 00:27:40 -0400 X-MC-Unique: 2dNAaRiTMweV3evsn2R3EQ-1 X-Mimecast-MFC-AGG-ID: 2dNAaRiTMweV3evsn2R3EQ_1785817659 Received: by mail-lf1-f69.google.com with SMTP id 2adb3069b0e04-5b2a35b9306so2526812e87.1 for ; Mon, 03 Aug 2026 21:27:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1785817659; x=1786422459; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=tpEIqxoBbLX0SvQdg20oXD/NXhm9ONjr2trVRywE48w=; b=NoIdsyJO7qLeKMFuVenkAO70Tg1tLtH8zGTD5SU8FUoMuSCNEBnSV+jD05+SQe5jJP sTSQWuf07/E8P689n5PUiMMo2LUMXVCLaEkEuizJIZlLyp8ZHo7lxzMm1XQcd3aO2bUq /nRt+5M03R+P3e8mVLgJhrjJsG6AMfPhzacOJDk61ofMqOyYiJMAqPpEJ7b1WT0MkvWG w9Caj7W4Sq1V92QftVH5FY9UEfAnmD+aWfvNb0zjK/5e80LZl341hBqqC52INWgRq00v R+HDfx0LkzxqsFYxXEbKiAcH4xdkukXp6MaM5IPN2pLrueDfcIDLThOXUMmy1UFpVU+k KYeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785817659; x=1786422459; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=tpEIqxoBbLX0SvQdg20oXD/NXhm9ONjr2trVRywE48w=; b=WkYktMM0DCgMYOJQ6q0HrqXbl85oCY8+u5ROyxxaX1uJJJ+BLNdTpvAxH+UH2z0bvT RiCnZQxA5a3G7v305Zm3Z0EsDAfLDMK+vQpyX6LmZFEGQBMmnRRWeDqVAfDxiqmwBlic aTGZzdaGVoTy4CICajTRabXJ8XHk74QmYi9vk+NtX2+cxb0Wtej1JL8etTpHita9J2K0 q5KaMZbnyG6sNNaJpLbNtpOGNQAjAVWYcxte1rLuUog1wXAcKztu6zuNjgf723/ulFZj IxKy6oNUhaen2Q4mevXbuNM7Qti5XnVhSchJuoVqlHIcrzl8ujijilSnICIsz2k/7j4m wc2w== X-Forwarded-Encrypted: i=1; AHgh+Rp8cRZ7242ohdaXErtfz/s6gOEyyHoe30V7wnO32pcJBfSH8KEtz3m5bYG/caC7mtXryYc8xhYs8fb/cro=@vger.kernel.org X-Gm-Message-State: AOJu0Yx7lWDmNlvdv3LHnuMbawyZClIbLAHIvtQNgzjzse0jXMHKV9gF qc8HHzkJWc8RGFGOy2TuGmwr8dSbU2hkBVHwaSs2sS0fQ2XWJJ0B1WHbNp9PKuwjy8fo5HvWko1 ce4fQESElwWI20Db/mpxpiDr4ghuA4kn/g082haB9Io8hBdza29Wr1+eI1kKiBy7d X-Gm-Gg: AR+sD13ZgvsbJiAhlYPB6zVQU2t+vM/qK+LS7atI+HQ9rKCWOORqQtRnaJh5uEUTrWZ SwoUEgRG4cPPtDrUQNk/RtUXzTRMIR4tAuPWH4yL6419Em6bJqOH9kyQE4WioN5sKmYQ9NqWk1X Ui5apq+Wm/TbtgiX6oRoLfOnVZWc88uFGHGki9oj4awy9+QgDjUc4qYtqSsypWgR9esqrcgmpZq ZL8m2y1Hj/0GsqgzLAtm8YdBMyTkjAGfFcMyVWnJr76Vdk185dr5oQEwQ5xPLpdZj6H1KnAguIH xIvfFCGuFc4XqshbNuKeSoU/j4bs2XC1zpJ6LpzDvKHPnwkeQkfGg196sujTN9aBCcWIvwWq8FX SfZLmhxf86wqj3X9A X-Received: by 2002:a05:6512:6694:20b0:5b2:a6eb:ff2f with SMTP id 2adb3069b0e04-5b2e4f7bdf2mr1488647e87.43.1785817658599; Mon, 03 Aug 2026 21:27:38 -0700 (PDT) X-Received: by 2002:a05:6512:6694:20b0:5b2:a6eb:ff2f with SMTP id 2adb3069b0e04-5b2e4f7bdf2mr1488627e87.43.1785817657987; Mon, 03 Aug 2026 21:27:37 -0700 (PDT) Received: from fedora (85-23-51-1.bb.dnainternet.fi. [85.23.51.1]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b2e23c46c4sm2311351e87.17.2026.08.03.21.27.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Aug 2026 21:27:33 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v13 10/11] mm: enable device page migration from HMM pagewalk Date: Tue, 4 Aug 2026 07:26:30 +0300 Message-ID: <20260804042631.2175585-11-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260804042631.2175585-1-mpenttil@redhat.com> References: <20260804042631.2175585-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 HMM pagewalk has now the machinery to do the first phase of device page migration, collecting the pfns and installing migration ptes. Enable migration in hmm_range_fault(), and change migrate_vma_setup() to use HMM pagewalk path. Two new flags, MIGRATE_VMA_FAULT and MIGRATE_VMA_WRITE are introduced for migrate_vma_setup(), to request for faulting missing pages, and requesting write access. Also, the migrate_vma_collect*() based paths are now unused, so delete them. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- include/linux/migrate.h | 10 +- mm/hmm.c | 28 +++ mm/migrate_device.c | 533 ++-------------------------------------- 3 files changed, 63 insertions(+), 508 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 7380a89756ee..2b137e6a56d0 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -176,6 +176,8 @@ enum migrate_vma_info { MIGRATE_VMA_SELECT_DEVICE_PRIVATE =3D 1 << 1, MIGRATE_VMA_SELECT_DEVICE_COHERENT =3D 1 << 2, MIGRATE_VMA_SELECT_COMPOUND =3D 1 << 3, + MIGRATE_VMA_FAULT =3D 1 << 4, + MIGRATE_VMA_WRITE =3D 1 << 5, }; =20 struct migrate_vma { @@ -213,10 +215,14 @@ struct migrate_vma { struct page *fault_page; }; =20 -// TODO: enable migration static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *r= ange) { - return 0; + enum migrate_vma_info minfo; + + minfo =3D (range->default_flags & HMM_PFN_REQ_MIGRATE) ? + range->migrate->flags : 0; + + return minfo; } =20 static inline struct mm_struct *hmm_range_fault_mm(struct hmm_range *range) diff --git a/mm/hmm.c b/mm/hmm.c index 03622455302a..53df67bbd87e 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -1448,9 +1448,37 @@ static const struct mm_walk_ops hmm_walk_ops =3D { * the invalidation to finish. * -EFAULT: A page was requested to be valid and could not be made val= id * ie it has no backing VMA or it is illegal to access + * -ERANGE: The range crosses multiple VMAs, or space for hmm_pfns arr= ay + * is too low. * * This is similar to get_user_pages(), except that it can read the page t= ables * without mutating them (ie causing faults). + * + * If want to do migration after faulting, call hmm_range_fault() with + * range.default_flags of HMM_PFN_REQ_MIGRATE, and optionally + * HMM_PFN_REQ_FAULT|HMM_PFN_REQ_WRITE, and initialize range->migrate fiel= d. + * range->migrate->vma will be populated during the call, + * and must be stable across the whole migrate process, which is + * why mmap_lock must be held around this call. + * + * When HMM_PFN_REQ_MIGRATE is set, migration collection may be partial on + * return and the caller takes responsibility for completing or aborting i= t. + * + * On success, the caller must call migrate_hmm_range_setup() and may then + * proceed with the normal migrate_vma sequence: prepare destination pages, + * call migrate_vma_pages(), update device mappings as needed, and finally + * call migrate_vma_finalize(). + * + * On -EBUSY, the caller may retry hmm_range_fault() using the same range = and + * PFN array without undoing entries collected by the previous attempt. If + * the caller stops retrying, it must abort the partial migration. + * + * On any other error, or when abandoning a retry, the caller must call + * migrate_hmm_range_setup(), migrate_vma_pages() with no valid destination + * entries, and migrate_vma_finalize() to abort the partial migration. + * + * The mmap read lock must remain held until the migration has either been + * completed or aborted. */ int hmm_range_fault(struct hmm_range *range) { diff --git a/mm/migrate_device.c b/mm/migrate_device.c index a6df6566f89e..3d021b2e81f0 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -18,509 +18,6 @@ #include #include "internal.h" =20 -static int migrate_vma_collect_skip(unsigned long start, - unsigned long end, - struct mm_walk *walk) -{ - struct migrate_vma *migrate =3D walk->private; - unsigned long addr; - - for (addr =3D start; addr < end; addr +=3D PAGE_SIZE) { - migrate->dst[migrate->npages] =3D 0; - migrate->src[migrate->npages++] =3D 0; - } - - return 0; -} - -static int migrate_vma_collect_hole(unsigned long start, - unsigned long end, - __always_unused int depth, - struct mm_walk *walk) -{ - struct migrate_vma *migrate =3D walk->private; - unsigned long addr; - - /* Only allow populating anonymous memory. */ - if (!vma_is_anonymous(walk->vma)) - return migrate_vma_collect_skip(start, end, walk); - - if (thp_migration_supported() && - (migrate->flags & MIGRATE_VMA_SELECT_COMPOUND) && - (IS_ALIGNED(start, HPAGE_PMD_SIZE) && - IS_ALIGNED(end, HPAGE_PMD_SIZE))) { - migrate->src[migrate->npages] =3D MIGRATE_PFN_MIGRATE | - MIGRATE_PFN_COMPOUND; - migrate->dst[migrate->npages] =3D 0; - migrate->npages++; - migrate->cpages++; - - /* - * Collect the remaining entries as holes, in case we - * need to split later - */ - return migrate_vma_collect_skip(start + PAGE_SIZE, end, walk); - } - - for (addr =3D start; addr < end; addr +=3D PAGE_SIZE) { - migrate->src[migrate->npages] =3D MIGRATE_PFN_MIGRATE; - migrate->dst[migrate->npages] =3D 0; - migrate->npages++; - migrate->cpages++; - } - - return 0; -} - -/** - * migrate_vma_split_folio() - Helper function to split a THP folio - * @folio: the folio to split - * @fault_page: struct page associated with the fault if any - * - * Returns 0 on success - */ -static int migrate_vma_split_folio(struct folio *folio, - struct page *fault_page) -{ - int ret; - struct folio *fault_folio =3D fault_page ? page_folio(fault_page) : NULL; - struct folio *new_fault_folio =3D NULL; - - if (folio !=3D fault_folio) { - folio_get(folio); - folio_lock(folio); - } - - ret =3D split_folio(folio); - if (ret) { - if (folio !=3D fault_folio) { - folio_unlock(folio); - folio_put(folio); - } - return ret; - } - - new_fault_folio =3D fault_page ? page_folio(fault_page) : NULL; - - /* - * Ensure the lock is held on the correct - * folio after the split - */ - if (!new_fault_folio) { - folio_unlock(folio); - folio_put(folio); - } else if (folio !=3D new_fault_folio) { - if (new_fault_folio !=3D fault_folio) { - folio_get(new_fault_folio); - folio_lock(new_fault_folio); - } - folio_unlock(folio); - folio_put(folio); - } - - return 0; -} - -/** migrate_vma_collect_huge_pmd - collect THP pages without splitting the - * folio for device private pages. - * @pmdp: pointer to pmd entry - * @start: start address of the range for migration - * @end: end address of the range for migration - * @walk: mm_walk callback structure - * @fault_folio: folio associated with the fault if any - * - * Collect the huge pmd entry at @pmdp for migration and set the - * MIGRATE_PFN_COMPOUND flag in the migrate src entry to indicate that - * migration will occur at HPAGE_PMD granularity - */ -static int migrate_vma_collect_huge_pmd(pmd_t *pmdp, unsigned long start, - unsigned long end, struct mm_walk *walk, - struct folio *fault_folio) -{ - struct mm_struct *mm =3D walk->mm; - struct folio *folio; - struct migrate_vma *migrate =3D walk->private; - spinlock_t *ptl; - int ret; - unsigned long write =3D 0; - - ptl =3D pmd_lock(mm, pmdp); - if (pmd_none(*pmdp)) { - spin_unlock(ptl); - return migrate_vma_collect_hole(start, end, -1, walk); - } - - if (pmd_trans_huge(*pmdp)) { - if (!(migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) { - spin_unlock(ptl); - return migrate_vma_collect_skip(start, end, walk); - } - - folio =3D pmd_folio(*pmdp); - if (is_huge_zero_folio(folio)) { - spin_unlock(ptl); - return migrate_vma_collect_hole(start, end, -1, walk); - } - if (pmd_write(*pmdp)) - write =3D MIGRATE_PFN_WRITE; - } else if (!pmd_present(*pmdp)) { - const softleaf_t entry =3D softleaf_from_pmd(*pmdp); - - folio =3D softleaf_to_folio(entry); - - if (!softleaf_is_device_private(entry) || - !(migrate->flags & MIGRATE_VMA_SELECT_DEVICE_PRIVATE) || - (folio->pgmap->owner !=3D migrate->pgmap_owner)) { - spin_unlock(ptl); - return migrate_vma_collect_skip(start, end, walk); - } - - if (softleaf_is_device_private_write(entry)) - write =3D MIGRATE_PFN_WRITE; - } else { - spin_unlock(ptl); - return -EAGAIN; - } - - folio_get(folio); - if (folio !=3D fault_folio && unlikely(!folio_trylock(folio))) { - spin_unlock(ptl); - folio_put(folio); - return migrate_vma_collect_skip(start, end, walk); - } - - if (thp_migration_supported() && - (migrate->flags & MIGRATE_VMA_SELECT_COMPOUND) && - (IS_ALIGNED(start, HPAGE_PMD_SIZE) && - IS_ALIGNED(end, HPAGE_PMD_SIZE))) { - - struct page_vma_mapped_walk pvmw =3D { - .ptl =3D ptl, - .address =3D start, - .pmd =3D pmdp, - .vma =3D walk->vma, - }; - - unsigned long pfn =3D page_to_pfn(folio_page(folio, 0)); - - migrate->src[migrate->npages] =3D migrate_pfn(pfn) | write - | MIGRATE_PFN_MIGRATE - | MIGRATE_PFN_COMPOUND; - migrate->dst[migrate->npages++] =3D 0; - migrate->cpages++; - ret =3D set_pmd_migration_entry(&pvmw, folio_page(folio, 0)); - if (ret) { - migrate->npages--; - migrate->cpages--; - migrate->src[migrate->npages] =3D 0; - migrate->dst[migrate->npages] =3D 0; - goto fallback; - } - migrate_vma_collect_skip(start + PAGE_SIZE, end, walk); - spin_unlock(ptl); - return 0; - } - -fallback: - spin_unlock(ptl); - if (!folio_test_large(folio)) - goto done; - ret =3D split_folio(folio); - if (fault_folio !=3D folio) - folio_unlock(folio); - folio_put(folio); - if (ret) - return migrate_vma_collect_skip(start, end, walk); - if (pmd_none(pmdp_get_lockless(pmdp))) - return migrate_vma_collect_hole(start, end, -1, walk); - -done: - return -ENOENT; -} - -static int migrate_vma_collect_pmd(pmd_t *pmdp, - unsigned long start, - unsigned long end, - struct mm_walk *walk) -{ - struct migrate_vma *migrate =3D walk->private; - struct vm_area_struct *vma =3D walk->vma; - struct mm_struct *mm =3D vma->vm_mm; - unsigned long addr =3D start, unmapped =3D 0; - spinlock_t *ptl; - struct folio *fault_folio =3D migrate->fault_page ? - page_folio(migrate->fault_page) : NULL; - pte_t *ptep; - -again: - if (pmd_trans_huge(*pmdp) || !pmd_present(*pmdp)) { - int ret =3D migrate_vma_collect_huge_pmd(pmdp, start, end, walk, fault_f= olio); - - if (ret =3D=3D -EAGAIN) - goto again; - if (ret =3D=3D 0) - return 0; - } - - ptep =3D pte_offset_map_lock(mm, pmdp, start, &ptl); - if (!ptep) - goto again; - lazy_mmu_mode_enable(); - ptep +=3D (addr - start) / PAGE_SIZE; - - for (; addr < end; addr +=3D PAGE_SIZE, ptep++) { - struct dev_pagemap *pgmap; - unsigned long mpfn =3D 0, pfn; - struct folio *folio; - struct page *page; - softleaf_t entry; - pte_t pte; - - pte =3D ptep_get(ptep); - - if (pte_none(pte)) { - if (vma_is_anonymous(vma)) { - mpfn =3D MIGRATE_PFN_MIGRATE; - migrate->cpages++; - } - goto next; - } - - if (!pte_present(pte)) { - /* - * Only care about unaddressable device page special - * page table entry. Other special swap entries are not - * migratable, and we ignore regular swapped page. - */ - entry =3D softleaf_from_pte(pte); - if (!softleaf_is_device_private(entry)) - goto next; - - page =3D softleaf_to_page(entry); - pgmap =3D page_pgmap(page); - if (!(migrate->flags & - MIGRATE_VMA_SELECT_DEVICE_PRIVATE) || - pgmap->owner !=3D migrate->pgmap_owner) - goto next; - - folio =3D page_folio(page); - if (folio_test_large(folio)) { - int ret; - - lazy_mmu_mode_disable(); - pte_unmap_unlock(ptep, ptl); - ret =3D migrate_vma_split_folio(folio, - migrate->fault_page); - - if (ret) { - if (unmapped) - flush_tlb_range(walk->vma, start, end); - - return migrate_vma_collect_skip(addr, end, walk); - } - - goto again; - } - - mpfn =3D migrate_pfn(page_to_pfn(page)) | - MIGRATE_PFN_MIGRATE; - if (softleaf_is_device_private_write(entry)) - mpfn |=3D MIGRATE_PFN_WRITE; - } else { - pfn =3D pte_pfn(pte); - if (is_zero_pfn(pfn) && - (migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) { - mpfn =3D MIGRATE_PFN_MIGRATE; - migrate->cpages++; - goto next; - } - page =3D vm_normal_page(migrate->vma, addr, pte); - if (page && !is_zone_device_page(page) && - !(migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) { - goto next; - } else if (page && is_device_coherent_page(page)) { - pgmap =3D page_pgmap(page); - - if (!(migrate->flags & - MIGRATE_VMA_SELECT_DEVICE_COHERENT) || - pgmap->owner !=3D migrate->pgmap_owner) - goto next; - } - folio =3D page ? page_folio(page) : NULL; - if (folio && folio_test_large(folio)) { - int ret; - - lazy_mmu_mode_disable(); - pte_unmap_unlock(ptep, ptl); - ret =3D migrate_vma_split_folio(folio, - migrate->fault_page); - - if (ret) { - if (unmapped) - flush_tlb_range(walk->vma, start, end); - - return migrate_vma_collect_skip(addr, end, walk); - } - - goto again; - } - mpfn =3D migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE; - mpfn |=3D pte_write(pte) ? MIGRATE_PFN_WRITE : 0; - } - - if (!page || !page->mapping) { - mpfn =3D 0; - goto next; - } - - /* - * By getting a reference on the folio we pin it and that blocks - * any kind of migration. Side effect is that it "freezes" the - * pte. - * - * We drop this reference after isolating the folio from the lru - * for non device folio (device folio are not on the lru and thus - * can't be dropped from it). - */ - folio =3D page_folio(page); - folio_get(folio); - - /* - * We rely on folio_trylock() to avoid deadlock between - * concurrent migrations where each is waiting on the others - * folio lock. If we can't immediately lock the folio we fail this - * migration as it is only best effort anyway. - * - * If we can lock the folio it's safe to set up a migration entry - * now. In the common case where the folio is mapped once in a - * single process setting up the migration entry now is an - * optimisation to avoid walking the rmap later with - * try_to_migrate(). - */ - if (fault_folio =3D=3D folio || folio_trylock(folio)) { - bool anon_exclusive; - pte_t swp_pte; - - if (pte_present(pte)) - flush_cache_page(vma, addr, pte_pfn(pte)); - anon_exclusive =3D folio_test_anon(folio) && - PageAnonExclusive(page); - if (anon_exclusive) { - pte =3D ptep_clear_flush(vma, addr, ptep); - - if (folio_try_share_anon_rmap_pte(folio, page)) { - set_pte_at(mm, addr, ptep, pte); - if (fault_folio !=3D folio) - folio_unlock(folio); - folio_put(folio); - mpfn =3D 0; - goto next; - } - } else { - pte =3D ptep_get_and_clear(mm, addr, ptep); - } - - migrate->cpages++; - - /* Set the dirty flag on the folio now the pte is gone. */ - if (pte_present(pte) && pte_dirty(pte)) - folio_mark_dirty(folio); - - /* Setup special migration page table entry */ - if (mpfn & MIGRATE_PFN_WRITE) - entry =3D make_writable_migration_entry( - page_to_pfn(page)); - else if (anon_exclusive) - entry =3D make_readable_exclusive_migration_entry( - page_to_pfn(page)); - else - entry =3D make_readable_migration_entry( - page_to_pfn(page)); - if (pte_present(pte)) { - if (pte_young(pte)) - entry =3D make_migration_entry_young(entry); - if (pte_dirty(pte)) - entry =3D make_migration_entry_dirty(entry); - } - swp_pte =3D swp_entry_to_pte(entry); - if (pte_present(pte)) { - if (pte_soft_dirty(pte)) - swp_pte =3D pte_swp_mksoft_dirty(swp_pte); - if (pte_uffd_wp(pte)) - swp_pte =3D pte_swp_mkuffd_wp(swp_pte); - } else { - if (pte_swp_soft_dirty(pte)) - swp_pte =3D pte_swp_mksoft_dirty(swp_pte); - if (pte_swp_uffd_wp(pte)) - swp_pte =3D pte_swp_mkuffd_wp(swp_pte); - } - set_pte_at(mm, addr, ptep, swp_pte); - - /* - * This is like regular unmap: we remove the rmap and - * drop the folio refcount. The folio won't be freed, as - * we took a reference just above. - */ - folio_remove_rmap_pte(folio, page, vma); - folio_put(folio); - - if (pte_present(pte)) - unmapped++; - } else { - folio_put(folio); - mpfn =3D 0; - } - -next: - migrate->dst[migrate->npages] =3D 0; - migrate->src[migrate->npages++] =3D mpfn; - } - - /* Only flush the TLB if we actually modified any entries */ - if (unmapped) - flush_tlb_range(walk->vma, start, end); - - lazy_mmu_mode_disable(); - pte_unmap_unlock(ptep - 1, ptl); - - return 0; -} - -static const struct mm_walk_ops migrate_vma_walk_ops =3D { - .pmd_entry =3D migrate_vma_collect_pmd, - .pte_hole =3D migrate_vma_collect_hole, - .walk_lock =3D PGWALK_RDLOCK, -}; - -/* - * migrate_vma_collect() - collect pages over a range of virtual addresses - * @migrate: migrate struct containing all migration information - * - * This will walk the CPU page table. For each virtual address backed by a - * valid page, it updates the src array and takes a reference on the page,= in - * order to pin the page until we lock it and unmap it. - */ -static void migrate_vma_collect(struct migrate_vma *migrate) -{ - struct mmu_notifier_range range; - - /* - * Note that the pgmap_owner is passed to the mmu notifier callback so - * that the registered device driver can skip invalidating device - * private page mappings that won't be migrated. - */ - mmu_notifier_range_init_owner(&range, MMU_NOTIFY_MIGRATE, 0, - migrate->vma->vm_mm, migrate->start, migrate->end, - migrate->pgmap_owner); - mmu_notifier_invalidate_range_start(&range); - - walk_page_range(migrate->vma->vm_mm, migrate->start, migrate->end, - &migrate_vma_walk_ops, migrate); - - mmu_notifier_invalidate_range_end(&range); - migrate->end =3D migrate->start + (migrate->npages << PAGE_SHIFT); -} - /* * migrate_vma_check_page() - check if page is pinned or not * @page: struct page to check @@ -729,10 +226,20 @@ static void migrate_vma_unmap(struct migrate_vma *mig= rate) */ int migrate_vma_setup(struct migrate_vma *args) { + int ret; long nr_pages =3D (args->end - args->start) >> PAGE_SHIFT; + struct hmm_range range =3D { + .notifier =3D NULL, + .hmm_pfns =3D args->src, + .dev_private_owner =3D args->pgmap_owner, + .migrate =3D args, + .default_flags =3D HMM_PFN_REQ_MIGRATE + }; =20 args->start &=3D PAGE_MASK; args->end &=3D PAGE_MASK; + range.start =3D args->start; + range.end =3D args->end; if (!args->vma || is_vm_hugetlb_page(args->vma) || (args->vma->vm_flags & VM_SPECIAL) || vma_is_dax(args->vma)) return -EINVAL; @@ -754,10 +261,24 @@ int migrate_vma_setup(struct migrate_vma *args) args->cpages =3D 0; args->npages =3D 0; =20 - migrate_vma_collect(args); + if (args->flags & MIGRATE_VMA_FAULT) + range.default_flags |=3D HMM_PFN_REQ_FAULT; =20 - if (args->cpages) - migrate_vma_unmap(args); + if (args->flags & MIGRATE_VMA_WRITE) + range.default_flags |=3D HMM_PFN_REQ_FAULT | HMM_PFN_REQ_WRITE; + + ret =3D hmm_range_fault(&range); + + migrate_hmm_range_setup(&range); + + /* Remove migration PTEs */ + if (ret) { + migrate_vma_pages(args); + migrate_vma_finalize(args); + memset(args->src, 0, sizeof(*args->src) * nr_pages); + args->cpages =3D 0; + args->npages =3D 0; + } =20 /* * At this point pages are locked and unmapped, and thus they have --=20 2.55.0 From nobody Fri Oct 2 07:45:30 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D514742BE87 for ; Tue, 4 Aug 2026 04:27:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817670; cv=none; b=gH5/ujNNqK6tnKum+vEfGZ9axn8QiHcudPBLv0b1AqUZq1F6yJzP6Wq9aUUxGiHVBe85oMoVYuFNotga9Q9l6o/FGZAfBAgLVAzVDI9mEC/Hhb3dcUWG9oxUTFOnjzwf7wyEYpIxDbNwW6tvYk0iXOCn+/1i2XlnkuBzdTwa72g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817670; c=relaxed/simple; bh=BKyFkjBXdTVfeHF7s8Wdtcu2uTxQWYq7ooOrKQW93vc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=kJKy+txzozxgVCNcr7eV3wh2NmpQWs4PacdzAxS5D4Gb3yEAZfvdoteONBRk7Q7LowTZ8HJJkjjuKGeA87Bzwxt8hDJ71PP2LsMa/1HhZE8vBLsB41nTbuYWsl6qsuyhChfG+splRaROwP1TM9OJAfLwYJtRqUd5X0FHbEMoakc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=GvFZCm5w; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=khHYLoTx; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="GvFZCm5w"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="khHYLoTx" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1785817667; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GvY3DG2kuXvuZDjGJcBo0Sva6rHYNngOo11VFCjg+Yg=; b=GvFZCm5wiMrYwdO+E0vHCHl7ND1EoGNRrZit4oHqGTd3zk7BtP4Ybf8k1RTfwTARjLY7pl BxR5i3bQQxjLn5J6Ph0HvcEyWYUA6RQCRu4bAPg5XERM83tO9MZRCAG2OsluCgm9DDL67R qUomgjdWJpTw82X+mq2/hEBBRHSxTP0= Received: from mail-lf1-f69.google.com (mail-lf1-f69.google.com [209.85.167.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-248-_l3iDQYSOyOxCrE9981qDg-1; Tue, 04 Aug 2026 00:27:44 -0400 X-MC-Unique: _l3iDQYSOyOxCrE9981qDg-1 X-Mimecast-MFC-AGG-ID: _l3iDQYSOyOxCrE9981qDg_1785817663 Received: by mail-lf1-f69.google.com with SMTP id 2adb3069b0e04-5aebadc6167so2338207e87.3 for ; Mon, 03 Aug 2026 21:27:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1785817663; x=1786422463; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=GvY3DG2kuXvuZDjGJcBo0Sva6rHYNngOo11VFCjg+Yg=; b=khHYLoTxQnXxGwAa4ShGj/ca+XoJtAh/HPgQHWg4aBknGCKb6mfrRPHVfFWZxgcVnj cfIfHpfNtNrepQZL/53DQDSpm6p5l1wLWINHQBDFnewKlArBitA89tmY28+s4qoD7j9L lIz9m48Wl0SPFWQp/DMLGSXtf2DAiVqox+0hkDYAnKSFkTk8pQ1gFsvlvPeaK/48UVKS QOb/95kFtx+WEKfV8C6Vs+kGRD2S27cANEmSLzU76hJ4xmTdXmj5J49gYABt8Oza6rXg 78noYn+O3QiCY0Zt7IIga79PsJtTE2g2mW5iq2Cl/TY78FOvNTCZUDaQ/xCcaWTrq3DV dQCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785817663; x=1786422463; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=GvY3DG2kuXvuZDjGJcBo0Sva6rHYNngOo11VFCjg+Yg=; b=hkoTXNu4jz/e55VePVIb75Yop7Au6l4Uc7/HRov/LsiwP6TqDJxvDg38jJzj9/TxWZ Za99TqmHLdTq/N3ltHY9kcs8xcGkBVMnSo5TemWiut5/5w9G+tdZ80ihlTNa+Hs++mh2 9yhS32XnMnBNcTb7BBzyb0RMtQmhAPUPQ+VSF0XWUofjCfm6baE5065n6YwvXeRnouX3 v1Jqy30Apj3RD+bvlasDJMA7jzs9bSEELzPsoO0+axAxPSCFLHM2h9RT+eP8FvZ+QNJY aHe2MD/bcXZni2kBGKBL5ejbNl+ATo+wBwdBbHQ9oyJjw/EQryXgRrbBkuCBtDeXKKD7 ShXQ== X-Forwarded-Encrypted: i=1; AHgh+Ro95ESR+7HuuNevMcBj3lD/pPC/JG25h1dUAhM93nVOQjRin5LpXBCPYBzHm4gO8AOz80Xf1di++SBomQk=@vger.kernel.org X-Gm-Message-State: AOJu0YyUJiR/rBXO9OoqiYQR3oA9wAc1pckR4XYfyjXtVaiijsaufiRz syLAiEITo8EgfXI3/ojngMDOSnyfztkOeGc3ZqP3VsO5Im8WD/DiBsX0TROZ686AxXdFnfNIyIL G2nC0BMolao+mfLP1Gma/G9tmprl3viCxu9Cw9wRSJICE2rwPAvYGGMF7fMjbJ1Bd X-Gm-Gg: AR+sD13LJYtthr+oItMbR/aOUY7NdHW2Z0/eICoZjsK6skoqrGJ8zYeYOSRCG0xfaU0 i8c+UZxb9tU3MuBp4VMPQqJP3ekVDlc9/H65iQuGuh6zVNnrcBv0l4WyjsTXDGt7AT3s+HOPEQf F2N49yg3fqC8plHECFGhK7IK/bKbUQTW0NxzfBflG3WIYg4GV7dYHjRk1/MqxQNx3UqGW+h8iG9 IDPp5ZW6JdM7pf37vmjNSulORIX2qM9BkNhTKTOXBfCtCFKYL1WI0hqrSJsc9cbRrx1Kc2bZm6u y3b1s38IRR6Njz+FnLG1/Wy+avLMag8vDI0LXLwdeqMRFnVJEloxVVEpYDV0M5qsWyQz9EktTIr iCK6jNIcff1311vag X-Received: by 2002:a05:6512:3153:b0:5b2:aaa2:1d80 with SMTP id 2adb3069b0e04-5b2e4f8415dmr1510636e87.47.1785817663130; Mon, 03 Aug 2026 21:27:43 -0700 (PDT) X-Received: by 2002:a05:6512:3153:b0:5b2:aaa2:1d80 with SMTP id 2adb3069b0e04-5b2e4f8415dmr1510633e87.47.1785817662567; Mon, 03 Aug 2026 21:27:42 -0700 (PDT) Received: from fedora (85-23-51-1.bb.dnainternet.fi. [85.23.51.1]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b2e23c46c4sm2311351e87.17.2026.08.03.21.27.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Aug 2026 21:27:40 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Marco Pagani Subject: [PATCH v13 11/11] lib/test_hmm: add a new testcase for the migrate on fault Date: Tue, 4 Aug 2026 07:26:31 +0300 Message-ID: <20260804042631.2175585-12-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260804042631.2175585-1-mpenttil@redhat.com> References: <20260804042631.2175585-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 Enhance the hmm test driver (lib/test_hmm) with migrate on fault case. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Signed-off-by: Marco Pagani Signed-off-by: Mika Penttil=C3=A4 --- lib/test_hmm.c | 130 ++++++++++++++++++++++++- lib/test_hmm_uapi.h | 19 ++-- tools/testing/selftests/mm/hmm-tests.c | 54 ++++++++++ 3 files changed, 190 insertions(+), 13 deletions(-) diff --git a/lib/test_hmm.c b/lib/test_hmm.c index 79cb7487d211..46a83e0cb709 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -36,6 +36,7 @@ #define DMIRROR_RANGE_FAULT_TIMEOUT 1000 #define DEVMEM_CHUNK_SIZE (256 * 1024 * 1024U) #define DEVMEM_CHUNKS_RESERVE 16 +#define PFNS_ARRAY_SIZE 64 =20 /* * For device_private pages, dpage is just a dummy struct page @@ -355,6 +356,7 @@ static int dmirror_range_fault(struct dmirror *dmirror, struct mm_struct *mm =3D dmirror->notifier.mm; unsigned long timeout =3D jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT); + bool migrate =3D range->default_flags & HMM_PFN_REQ_MIGRATE; int ret; =20 while (true) { @@ -364,9 +366,15 @@ static int dmirror_range_fault(struct dmirror *dmirror, } =20 range->notifier_seq =3D mmu_interval_read_begin(range->notifier); - mmap_read_lock(mm); - ret =3D hmm_range_fault(range); - mmap_read_unlock(mm); + + /* mmap lock held for whole migrate on fault operation */ + if (!migrate) { + mmap_read_lock(mm); + ret =3D hmm_range_fault(range); + mmap_read_unlock(mm); + } else { + ret =3D hmm_range_fault(range); + } if (ret) { if (ret =3D=3D -EBUSY) continue; @@ -382,7 +390,9 @@ static int dmirror_range_fault(struct dmirror *dmirror, break; } =20 - ret =3D dmirror_do_fault(dmirror, range); + /* update device page table after migration */ + if (!migrate) + ret =3D dmirror_do_fault(dmirror, range); =20 mutex_unlock(&dmirror->mutex); out: @@ -1281,6 +1291,114 @@ static int dmirror_migrate_to_device(struct dmirror= *dmirror, return ret; } =20 +static int do_fault_and_migrate(struct dmirror *dmirror, struct hmm_range = *range) +{ + struct migrate_vma *migrate =3D range->migrate; + int ret; + + mmap_read_lock(dmirror->notifier.mm); + + /* Fault-in pages for migration */ + ret =3D dmirror_range_fault(dmirror, range); + /* + * Set this up even on error: hmm_range_fault() may have collected + * part of the range before failing. + */ + migrate_hmm_range_setup(range); + if (ret) { + /* + * dst[] entries are empty, so this marks every collected migration + * as failed. finalize then restores the source mappings and drops + * the associated locks/references. + */ + migrate_vma_pages(migrate); + migrate_vma_finalize(migrate); + goto out; + } + + pr_debug("Migrating from sys mem to device mem\n"); + + dmirror_migrate_alloc_and_copy(migrate, dmirror); + migrate_vma_pages(migrate); + dmirror_migrate_finalize_and_map(migrate, dmirror); + migrate_vma_finalize(migrate); +out: + mmap_read_unlock(dmirror->notifier.mm); + return ret; +} + +static int dmirror_fault_and_migrate_to_device(struct dmirror *dmirror, + struct hmm_dmirror_cmd *cmd) +{ + unsigned long start, size, end, next; + unsigned long src_pfns[PFNS_ARRAY_SIZE] =3D { 0 }; + unsigned long dst_pfns[PFNS_ARRAY_SIZE] =3D { 0 }; + struct migrate_vma migrate =3D { 0 }; + struct hmm_range range =3D { 0 }; + struct dmirror_bounce bounce; + int ret =3D 0; + + /* Whole range */ + start =3D cmd->addr; + size =3D cmd->npages << PAGE_SHIFT; + end =3D start + size; + + if (!mmget_not_zero(dmirror->notifier.mm)) { + ret =3D -EFAULT; + goto out; + } + + migrate.pgmap_owner =3D dmirror->mdevice; + migrate.src =3D src_pfns; + migrate.dst =3D dst_pfns; + migrate.flags =3D MIGRATE_VMA_SELECT_SYSTEM; + + range.migrate =3D &migrate; + range.hmm_pfns =3D src_pfns; + range.pfn_flags_mask =3D 0; + range.default_flags =3D HMM_PFN_REQ_FAULT | HMM_PFN_REQ_MIGRATE; + range.dev_private_owner =3D dmirror->mdevice; + range.notifier =3D &dmirror->notifier; + + for (next =3D start; next < end; next =3D range.end) { + range.start =3D next; + range.end =3D min(end, next + (PFNS_ARRAY_SIZE << PAGE_SHIFT)); + + pr_debug("Fault and migrate range start:%#lx end:%#lx\n", + range.start, range.end); + + ret =3D do_fault_and_migrate(dmirror, &range); + if (ret) + goto out_mmput; + } + + /* + * Return the migrated data for verification. + * Only for pages in device zone + */ + ret =3D dmirror_bounce_init(&bounce, start, size); + if (ret) + goto out_mmput; + + mutex_lock(&dmirror->mutex); + ret =3D dmirror_do_read(dmirror, start, end, &bounce); + mutex_unlock(&dmirror->mutex); + if (ret =3D=3D 0) { + ret =3D copy_to_user(u64_to_user_ptr(cmd->ptr), bounce.ptr, bounce.size); + if (ret) + ret =3D -EFAULT; + } + + cmd->cpages =3D bounce.cpages; + dmirror_bounce_fini(&bounce); + + +out_mmput: + mmput(dmirror->notifier.mm); +out: + return ret; +} + static void dmirror_mkentry(struct dmirror *dmirror, struct hmm_range *ran= ge, unsigned char *perm, unsigned long entry) { @@ -1547,6 +1665,10 @@ static long dmirror_fops_unlocked_ioctl(struct file = *filp, ret =3D dmirror_migrate_to_device(dmirror, &cmd); break; =20 + case HMM_DMIRROR_MIGRATE_ON_FAULT_TO_DEV: + ret =3D dmirror_fault_and_migrate_to_device(dmirror, &cmd); + break; + case HMM_DMIRROR_MIGRATE_TO_SYS: ret =3D dmirror_migrate_to_system(dmirror, &cmd); break; diff --git a/lib/test_hmm_uapi.h b/lib/test_hmm_uapi.h index f94c6d457338..0b6e7a419e36 100644 --- a/lib/test_hmm_uapi.h +++ b/lib/test_hmm_uapi.h @@ -29,15 +29,16 @@ struct hmm_dmirror_cmd { }; =20 /* Expose the address space of the calling process through hmm device file= */ -#define HMM_DMIRROR_READ _IOWR('H', 0x00, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_WRITE _IOWR('H', 0x01, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_MIGRATE_TO_DEV _IOWR('H', 0x02, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_MIGRATE_TO_SYS _IOWR('H', 0x03, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_SNAPSHOT _IOWR('H', 0x04, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_EXCLUSIVE _IOWR('H', 0x05, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_CHECK_EXCLUSIVE _IOWR('H', 0x06, struct hmm_dmirror_cm= d) -#define HMM_DMIRROR_RELEASE _IOWR('H', 0x07, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_FLAGS _IOWR('H', 0x08, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_READ _IOWR('H', 0x00, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_WRITE _IOWR('H', 0x01, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_MIGRATE_TO_DEV _IOWR('H', 0x02, struct hmm_dmirror_cm= d) +#define HMM_DMIRROR_MIGRATE_ON_FAULT_TO_DEV _IOWR('H', 0x03, struct hmm_dm= irror_cmd) +#define HMM_DMIRROR_MIGRATE_TO_SYS _IOWR('H', 0x04, struct hmm_dmirror_cm= d) +#define HMM_DMIRROR_SNAPSHOT _IOWR('H', 0x05, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_EXCLUSIVE _IOWR('H', 0x06, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_CHECK_EXCLUSIVE _IOWR('H', 0x07, struct hmm_dmirror_c= md) +#define HMM_DMIRROR_RELEASE _IOWR('H', 0x08, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_FLAGS _IOWR('H', 0x09, struct hmm_dmirror_cmd) =20 #define HMM_DMIRROR_FLAG_FAIL_ALLOC (1ULL << 0) =20 diff --git a/tools/testing/selftests/mm/hmm-tests.c b/tools/testing/selftes= ts/mm/hmm-tests.c index 2f2b9879d100..4ac17cf6f866 100644 --- a/tools/testing/selftests/mm/hmm-tests.c +++ b/tools/testing/selftests/mm/hmm-tests.c @@ -283,6 +283,13 @@ static int hmm_migrate_sys_to_dev(int fd, return hmm_dmirror_cmd(fd, HMM_DMIRROR_MIGRATE_TO_DEV, buffer, npages); } =20 +static int hmm_migrate_on_fault_sys_to_dev(int fd, + struct hmm_buffer *buffer, + unsigned long npages) +{ + return hmm_dmirror_cmd(fd, HMM_DMIRROR_MIGRATE_ON_FAULT_TO_DEV, buffer, n= pages); +} + static int hmm_migrate_dev_to_sys(int fd, struct hmm_buffer *buffer, unsigned long npages) @@ -990,6 +997,53 @@ TEST_F(hmm, migrate) hmm_buffer_free(buffer); } =20 + +/* + * Fault and migrate anonymous memory to device private memory. + */ +TEST_F(hmm, migrate_on_fault) +{ + struct hmm_buffer *buffer; + unsigned long npages; + unsigned long size; + unsigned long i; + int *ptr; + int ret; + + npages =3D ALIGN(HMM_BUFFER_SIZE, self->page_size) >> self->page_shift; + ASSERT_NE(npages, 0); + size =3D npages << self->page_shift; + + buffer =3D malloc(sizeof(*buffer)); + ASSERT_NE(buffer, NULL); + + buffer->fd =3D -1; + buffer->size =3D size; + buffer->mirror =3D malloc(size); + ASSERT_NE(buffer->mirror, NULL); + + buffer->ptr =3D mmap(NULL, size, + PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, + buffer->fd, 0); + ASSERT_NE(buffer->ptr, MAP_FAILED); + + /* Initialize buffer in system memory. */ + for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) + ptr[i] =3D i; + + /* Fault and migrate memory to device. */ + ret =3D hmm_migrate_on_fault_sys_to_dev(self->fd, buffer, npages); + ASSERT_EQ(ret, 0); + ASSERT_EQ(buffer->cpages, npages); + + /* Check what the device read. */ + for (i =3D 0, ptr =3D buffer->mirror; i < size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i], i); + + hmm_buffer_free(buffer); +} + /* * Migrate private file memory to device private memory. */ --=20 2.55.0