From nobody Thu Sep 24 12:53:00 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 335C541D638 for ; Thu, 24 Sep 2026 06:53:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232830; cv=none; b=ldruZoZNeRBGq0N9xV9g3n12ufT7/NQpy9sz/NRodRC9nCNKCCcToAI5z/On8jae45vDn5Af2IikMdM7j9aKQ8yhYwxtx5c+ysPxfCDhiZywCIb1A6oOcVyushDSEG6jaEqseDDWh1AdLxEJfvYeUI6Igpy3S2FryNgZS9nZKJc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232830; c=relaxed/simple; bh=YMM3m1N8fC5PJuvtWQuu+nad9ANW0tPUhYNaGEkuucE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=EGyWsZbdq3CfK4BzfOAV9X7LzS8WOQW9j7X2Ub8NyHhcuiqKxwS98tHdRX+uPxvdJEaGh7kIlxFJ4zeUz9QZaOMrxMaQgtci2i5SbrwyELTmddcgLpDcBefHzMDsIM7gKYmvkB4SJVndtcK+VRussh73wclOYOs3wNrlaDCf/qc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=cg0AAlg+; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=HFGHzGip; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="cg0AAlg+"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="HFGHzGip" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790232815; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h/j5EYytItjUwk1/CqHyOHSfaQ3+6rA1AecV+p7hbPA=; b=cg0AAlg+T4e8SoiFKy7kbgHE4WwhHAIOPh7rQw79WqiqJnp+HXkWJ1xhCtBw02dOPuFOfI gt1H/3J6aybmE+59zcaSuGMQ6dE/Ylt3oyc32/svFZWWGrcbUvzAS52ZmPUx7K2wFdt1T5 f5D+21IHnYeqnUJG4AiC+6S3uNl0aLo= Received: from mail-lj1-f197.google.com (mail-lj1-f197.google.com [209.85.208.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-156-gt2HRUQANeOrdMQvxUFuYQ-1; Thu, 24 Sep 2026 02:53:33 -0400 X-MC-Unique: gt2HRUQANeOrdMQvxUFuYQ-1 X-Mimecast-MFC-AGG-ID: gt2HRUQANeOrdMQvxUFuYQ_1790232812 Received: by mail-lj1-f197.google.com with SMTP id 38308e7fff4ca-3a35015f2afso3927231fa.1 for ; Wed, 23 Sep 2026 23:53:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790232812; x=1790837612; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=h/j5EYytItjUwk1/CqHyOHSfaQ3+6rA1AecV+p7hbPA=; b=HFGHzGipc/o76on2+jTKs3/X7j/6s4R92AE70wlApxRh4Xwtj4oz9Gl1oYSt+uPIjT cgKal1KYQdaMDcNvC0ASrtvQFC47RiKJWJ+m/XefxP5oXqjlDkhzqzpn7VJQGgah0djX 5Dj91fHabL8yvBJ8hZd8aIlr6A61BTjtpY7Vyko5Ex+TlO9wyMcEUfpbwXHRYZF6XbbA L8V/WS9r4ZQEGSHcSi5y4jGC9UjcIL81xaQhvaIFVPaZtI2Mzibg3mOLnOpTYrRq6w3A Ab21oNojv6nK6FtiYldhFZALUbi8wHRW2jG1RwDhIlaxj+w6GwL7NQzQJw+RDVMn1f9G y8iQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790232812; x=1790837612; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=h/j5EYytItjUwk1/CqHyOHSfaQ3+6rA1AecV+p7hbPA=; b=oACzQi3FmSJhk3Be6OaK5InmA7+oTqs0CK0JegzKp/etL5sB3DVN+b9b1sytyHLmaK QBVuA5JOEowRyloM+ZS5XLCp4DhUaut3w7Yw5IdQAMEYXLkUX/mDHJPbRtUJyOqkF7dZ N675EVQs9gsnE62zjRqNgG6iTrBZAoPdlB8KOf4BLxa12XB18+xzS3czgGiggTWcaaGz qCNTTT07tuJMXLgJKHJFOzjyUMVBt8+S5fTSep0j2aCaThWqiq6sd18gX7ChljYFbffz ryTvTeJeujxXkiDSIdHz+Mb17vJ15nycQFjsxc3idKkKwkPERXIoTl7c0INQruNWkkJs yNQA== X-Forwarded-Encrypted: i=1; AKwUvByoKW9Qmb+vnWCUZE+9qoKwcj7kVDnk16k3oYnVpSl7HzDQpFhFVd8QGI4fCbSvN+5hqJ1PlVpg3oCKhqU=@vger.kernel.org X-Gm-Message-State: AFuF++nqBOKzBXKGGr5d5DoGrPTmRgKP+1fAusRlEcldGQh6UF1PuVW8 NWAxP7vmkrfFt2mnKMNniknmcKBcgdDqK/KDUk+pX5qY/a9uhwMPTJhgsXgTL0TLUoe2Kh8dfht iro6HXvsA+qyy2OJPH5vWmDLn2naPtjmsGrzHv7ICqYK/06miPsXN4zepzHqcL2Ce X-Gm-Gg: AYBFou32b5zqQWXRdbHM8eb+JFfWLsWcyaSXef+qt0DKVtSR0fOkZdsZw1qiL+y8aPb Ngwysrva8SmVQmhGtwUXqeQJfDYgehKogtdqu4BDd/t2gegxul42J1jXcuxOPewWEHDHYSaQJ1U SECwIPrKoGJKXetopQzdhy4zNEJurF51FLssTWtQO3ZSYRSY1iMMRiztYFP6dCrVWL56tkvvDjb B1e0g4hmnL41gVZxsQSLkNhzIgXbPl2nHrgv0SLmkeKGZnc6jH/I7I6qUEFpRGa/k1XwBAikVWl U+YyyRG7ASImGrANSkDAs4SGt/+vQ8oGYoiizmUOs76AA2WbnZNmNywKz1P6ZvamlYrchnxUvsS EJqRsjgOLMjhQ/YVKSzam X-Received: by 2002:a2e:a593:0:b0:3a6:d50:e26e with SMTP id 38308e7fff4ca-3a63bf1c44fmr3484761fa.2.1790232812391; Wed, 23 Sep 2026 23:53:32 -0700 (PDT) X-Received: by 2002:a2e:a593:0:b0:3a6:d50:e26e with SMTP id 38308e7fff4ca-3a63bf1c44fmr3484571fa.2.1790232811952; Wed, 23 Sep 2026 23:53:31 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-3a63bf57909sm4803141fa.29.2026.09.23.23.53.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Sep 2026 23:53:31 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v15 01/11] mm/Kconfig: changes for migrate on fault for device pages Date: Thu, 24 Sep 2026 09:53:03 +0300 Message-ID: <20260924065313.899730-2-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260924065313.899730-1-mpenttil@redhat.com> References: <20260924065313.899730-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 HMM depends on MMU notifiers. With the unified HMM/migrate_device page table walk migrate_device needs HMM enabled. Enable them explicitly to avoid breaking random configs. Cc: Andrew Morton Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: "Liam R. Howlett" Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Signed-off-by: Mika Penttil=C3=A4 Acked-by: David Hildenbrand (Arm) Acked-by: Balbir Singh --- mm/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/Kconfig b/mm/Kconfig index 604c58199acb..b7a485beb9a7 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -687,6 +687,7 @@ config MIGRATION =20 config DEVICE_MIGRATION def_bool MIGRATION && ZONE_DEVICE + select HMM_MIRROR =20 config ARCH_ENABLE_HUGEPAGE_MIGRATION bool --=20 2.55.0 From nobody Thu Sep 24 12:53:00 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3D95E4219E9 for ; Thu, 24 Sep 2026 06:53:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232827; cv=none; b=U22A5qDxIuJ3NzUJJOxPvpZIhS7SebsI58RxcmtH8tYVrzWfz66IvpwkFtkTnyjTn5L82LYqvoNdt95HuFSxJQoweUhIHczRa8cSrxpavTuzzpV6PgaWgRV02plljv5PN8LoLbUAyDczRPYi472ZQiPaMZnYMaXgYFnueXcqaLs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232827; c=relaxed/simple; bh=Vw8DEcu9a1Jd1CJ9KWanGL9Yzn9S0TsNI8wC/ggW76Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Oo1U9LxlYQMVAVCdSOlRlUkgn8Mza4r7kxqR2YUeiF9twnfGMBsMminQOeLdsFu7Ow/D277NVxcs9dv+mUkU2rZR5Bs9V885mcp9uNxuhuUtpbhJ/XxBPaVvqkzKfxxstFBazpA5FXvG6cT0k/g2gdjNe6mZrX97whEaXKyu0Yo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=QlOlxhim; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=sh1lDjT2; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="QlOlxhim"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="sh1lDjT2" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790232818; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XHN2LiOQU7r0d0fe1KkDFs3khSrbeSD24b1q7hC8HmQ=; b=QlOlxhimajsjWWcWqWyw17Wtiaq1cUJmGLrnaWDDpXfZGlumGLQ+ejEtaKH0IHOIQVf37f X7unWH329T/dFwsv7gjc5hR6V7jfNTeXWNvXimHAwSpkSe0s4ONb0Ab1pcDYBY24Fo6JGl 4FAKmfM1zCt36k0Uj9xEJa2mc6fWqBc= Received: from mail-lj1-f199.google.com (mail-lj1-f199.google.com [209.85.208.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-651-Z1tPTtFkN8O2fPuWQtHWiA-1; Thu, 24 Sep 2026 02:53:37 -0400 X-MC-Unique: Z1tPTtFkN8O2fPuWQtHWiA-1 X-Mimecast-MFC-AGG-ID: Z1tPTtFkN8O2fPuWQtHWiA_1790232816 Received: by mail-lj1-f199.google.com with SMTP id 38308e7fff4ca-3a592f20c7cso8787061fa.0 for ; Wed, 23 Sep 2026 23:53:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790232815; x=1790837615; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=XHN2LiOQU7r0d0fe1KkDFs3khSrbeSD24b1q7hC8HmQ=; b=sh1lDjT2DjiCcdZLEUS5MgMZgWMWTEMbcl5ynWdu2bPme6JVJKfm7gr4jjRki1MGAt mbXf3qqfBcmlseRyNdBrPydagG3mAG8SAZnRdQVagFw8OUfVrKjdlND+s34MehTlAEt4 WtSW6+oEUSv6N/aVphcDeMOWVCFlqf45Y3HaRR+zdcS2zG3f0u1TQ6gUqta5wDxfrIwq 70o1+TsqyFWZyrDhs0HzDzFxzy93gmPWRvETbLlc3AYoccITyWXCvodlH+dmutClBr8v HQHz33JHKn0unXhSpl36Fg4eIysxSS/4Cz5FIUpBqYypXD/asAoVSF3W7eLhYg2Pkjfn n+Yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790232815; x=1790837615; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=XHN2LiOQU7r0d0fe1KkDFs3khSrbeSD24b1q7hC8HmQ=; b=MMj5TtNtELtJJCIbNcWRUjkMI/p7uClnwBrld/iN4Nz7qh65hw401R4HS+4LDY/JGZ R7QbgDKuxMQVUziZtAtsbPUdoRVLPSP2waaNsoYcqHcjVrQsc+qhRO73zglIt5n2XuTd Cm5raY9+nPvu0qgBuM6/28MUMJq0CvhEEAl5Kmx+O/DRxSIRQ1VcJyrxEcmX4vFOW+UV hE715NFdXDzq+tijbWourlxB2UsPCbwHo7elHogPTggXNKXBAWDCGGYvcDg4bGFwpSJu koewyAues0kT/cMm4pBr99dA2n1jUUv0MT+TO1v/e44Y0AXCVgAJApmgAEmUKjh23vsq fjqg== X-Forwarded-Encrypted: i=1; AKwUvBzAGyM+npn6KssxhCI4wqNTvyOERyGVJ+/AActRYdbr2rxlWfVdzDXLQOI2Wnn1FCLSYQXB+rpmWIo0VZ0=@vger.kernel.org X-Gm-Message-State: AFuF++lNwK5xVKBjr2bkG782xztRPHJd8LvwXCsGH0QOJl7jXkT1YcHg 5zA/++6s5zVulErsZA2fSYckKWNbWVWeHK1WQtaj01l8RBPtPOOQS+JIeymASsXIQ3YO48xOpHB 0Zprd+rXuusZAIsbCTfQLhmII9WyCg76jmT9U+pWTQchb2pQfh8dPok/ZSmwbweQTAHkRhqSY4F 0= X-Gm-Gg: AYBFou1f+CvwoRgdW64P9a1fTz6zpnhCX15eMMQhoOB6NmryVWDpk88MXfd4PS5cwMB juOBUfflbBHU2QjrjyyulsaN1eayYSkPTg9zd6xQOIWPGnoU1+203gaQFYzIgU2aObx+2eIuX+L 4Z41c+Mb+xfUAisDlTSOYlcC7/VdGtdaoWmZG9K5yKLYANdAiaDk/vlaAwuSkY61JdJztOUNdhT CHFl6fCJvqu/W+ohWN/hRrzdj8oLKaXacoovEgUfljnKyq9jDH8TyiD3tXuIR6AW6WJRdlBFDY0 FGOBFul8aIDewk8U9pxqSpmh+GegmF0XCIM445NVzgEm6oDQvsX8vMugUT+CPtlHF1KyQkFESBy 0JcL3gwYKTTg30xhesC15 X-Received: by 2002:a05:651c:211b:b0:3a1:980f:5907 with SMTP id 38308e7fff4ca-3a63c086e9bmr3379281fa.8.1790232815507; Wed, 23 Sep 2026 23:53:35 -0700 (PDT) X-Received: by 2002:a05:651c:211b:b0:3a1:980f:5907 with SMTP id 38308e7fff4ca-3a63c086e9bmr3379231fa.8.1790232815017; Wed, 23 Sep 2026 23:53:35 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-3a63bf57909sm4803141fa.29.2026.09.23.23.53.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Sep 2026 23:53:34 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v15 02/11] mm: add helper to convert HMM pfn to migrate pfn Date: Thu, 24 Sep 2026 09:53:04 +0300 Message-ID: <20260924065313.899730-3-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260924065313.899730-1-mpenttil@redhat.com> References: <20260924065313.899730-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 The unified HMM/migrate_device pagewalk does the "collecting" on the HMM side, so we need a helper to transfer pfns to the migrate_vma world. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- include/linux/hmm.h | 29 +++++++++++++++++----- include/linux/migrate.h | 3 ++- mm/migrate_device.c | 55 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 80 insertions(+), 7 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 6f04e3932f5b..4f56f3419cb4 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -13,6 +13,8 @@ =20 struct mmu_interval_notifier; =20 +struct migrate_vma; + /* * On output: * 0 - The page is faultable and a future call with=20 @@ -27,13 +29,22 @@ struct mmu_interval_notifier; * HMM_PFN_P2PDMA_BUS - Bus mapped P2P transfer * HMM_PFN_DMA_MAPPED - Flag preserved on input-to-output transformation * to mark that page is already DMA mapped + * HMM_PFN_MIGRATE - The entry is to be migrated. Note, HMM_PFN_MIGRATE + * alone without HMM_PFN_VALID denotes the + * empty page. + * This flag optionally together with HMM_PFN_COMPOUND + * are indicators for migrate_hmm_range_setup() to + * setup the migrate pfns. + * HMM_PFN_COMPOUND - The entry represents a > 0 order page * * On input: - * 0 - Return the current state of the page, do not fault = it. - * HMM_PFN_REQ_FAULT - The output must have HMM_PFN_VALID or hmm_range_fau= lt() - * will fail - * HMM_PFN_REQ_WRITE - The output must have HMM_PFN_WRITE or hmm_range_fau= lt() - * will fail. Must be combined with HMM_PFN_REQ_FAULT. + * 0 - Return the current state of the page, do not faul= t it. + * HMM_PFN_REQ_FAULT - The output must have HMM_PFN_VALID or hmm_range_f= ault() + * will fail + * HMM_PFN_REQ_WRITE - The output must have HMM_PFN_WRITE or hmm_range_f= ault() + * will fail. Must be combined with HMM_PFN_REQ_FAUL= T. + * HMM_PFN_REQ_MIGRATE - For default_flags only, request to migrate the ra= nge, + * according to hmm_range.migrate.flags */ enum hmm_pfn_flags { /* Output fields and flags */ @@ -48,11 +59,15 @@ enum hmm_pfn_flags { HMM_PFN_P2PDMA =3D 1UL << (BITS_PER_LONG - 5), HMM_PFN_P2PDMA_BUS =3D 1UL << (BITS_PER_LONG - 6), =20 - HMM_PFN_ORDER_SHIFT =3D (BITS_PER_LONG - 11), + /* Migrate request */ + HMM_PFN_MIGRATE =3D 1UL << (BITS_PER_LONG - 7), + HMM_PFN_COMPOUND =3D 1UL << (BITS_PER_LONG - 8), + HMM_PFN_ORDER_SHIFT =3D (BITS_PER_LONG - 13), =20 /* Input flags */ HMM_PFN_REQ_FAULT =3D HMM_PFN_VALID, HMM_PFN_REQ_WRITE =3D HMM_PFN_WRITE, + HMM_PFN_REQ_MIGRATE =3D HMM_PFN_MIGRATE, =20 HMM_PFN_FLAGS =3D ~((1UL << HMM_PFN_ORDER_SHIFT) - 1), }; @@ -107,6 +122,7 @@ static inline unsigned int hmm_pfn_to_map_order(unsigne= d long hmm_pfn) * @default_flags: default flags for the range (write, read, ... see hmm d= oc) * @pfn_flags_mask: allows to mask pfn flags so that only default_flags ma= tter * @dev_private_owner: owner of device private pages + * @migrate: structure for migrating a range of a VMA */ struct hmm_range { struct mmu_interval_notifier *notifier; @@ -117,6 +133,7 @@ struct hmm_range { unsigned long default_flags; unsigned long pfn_flags_mask; void *dev_private_owner; + struct migrate_vma *migrate; }; =20 /* diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 78424b3824c2..280ec9a866c7 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -3,6 +3,7 @@ #define _LINUX_MIGRATE_H =20 #include +#include #include #include #include @@ -202,7 +203,7 @@ void migrate_device_pages(unsigned long *src_pfns, unsi= gned long *dst_pfns, unsigned long npages); void migrate_device_finalize(unsigned long *src_pfns, unsigned long *dst_pfns, unsigned long npages); - +void migrate_hmm_range_setup(struct hmm_range *range); #endif /* CONFIG_MIGRATION */ =20 #endif /* _LINUX_MIGRATE_H */ diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 009bfa8b212d..43e571e82cca 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -1528,3 +1528,58 @@ int migrate_device_coherent_folio(struct folio *foli= o) return 0; return -EBUSY; } + +/** + * migrate_hmm_range_setup() - prepare to migrate a range of memory + * @range: contains pointer to struct migrate_vma to be set up. + * + * When collecting has been done with hmm_range_fault(), this + * should be called next, and completes range->migrate by + * populating migrate->src[] and migrate->dst[] + * using range->hmm_pfns[]. + * Also, migrate->cpages and migrate->npages get initialized. + * After migrate_hmm_range_setup(), range->migrate is good + * for the rest of the migrate_vma_* flow. + */ +void migrate_hmm_range_setup(struct hmm_range *range) +{ + struct migrate_vma *migrate =3D range->migrate; + + if (!migrate) + return; + + migrate->npages =3D (migrate->end - migrate->start) >> PAGE_SHIFT; + migrate->cpages =3D 0; + + for (unsigned long i =3D 0; i < migrate->npages; i++) { + unsigned long pfn =3D range->hmm_pfns[i]; + + /* + * We are only interested in entries to be + * migrated. + */ + if (!(pfn & HMM_PFN_MIGRATE)) { + migrate->src[i] =3D 0; + migrate->dst[i] =3D 0; + continue; + } + + migrate->cpages++; + + /* HMM_PFN_MIGRATE without HMM_PFN_VALID denotes the special zero page */ + if (pfn & HMM_PFN_VALID) + migrate->src[i] =3D migrate_pfn(page_to_pfn(hmm_pfn_to_page(pfn))); + else + migrate->src[i] =3D 0; + + migrate->src[i] |=3D MIGRATE_PFN_MIGRATE; + migrate->src[i] |=3D (pfn & HMM_PFN_WRITE) ? MIGRATE_PFN_WRITE : 0; + migrate->src[i] |=3D (pfn & HMM_PFN_COMPOUND) ? MIGRATE_PFN_COMPOUND : 0; + migrate->dst[i] =3D 0; + } + + if (migrate->cpages) + migrate_vma_unmap(migrate); + +} +EXPORT_SYMBOL(migrate_hmm_range_setup); --=20 2.55.0 From nobody Thu Sep 24 12:53:00 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 31D2D418342 for ; Thu, 24 Sep 2026 06:53:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232835; cv=none; b=iEVLm8iAGFJK454YrcFCdeiGqqZYvporr4tFIcbt7lR7TI5wN6xteOl6G79q6rBSaRB38L9YHMeyiFGp2sPLMLq+92rrj6xBYahqANnk9LwLOVRvmIrEcJpdKFQ7qx2MaVo2zu8+Y7AlhP2M3nTQaKWIx9L3My2EqDJs0Vit2vk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232835; c=relaxed/simple; bh=VrmX+POa5H5GflwtqXqdfjjgLIkBtAzF+h//OZga+AY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=iXEn9fldaqEW/RE4cDieMFOh9sEaIaM/FrDAUVz8DbYfL5XmzDZWiAP4c+nhPY6/7zHHReR6jBXBVMYAEspTVQnzEuSkR5veLydrUuhwINs9ST1kLdpFwAJ805TlfqtP4kP+47Z91TkzVAipfX+w9SqsBISkrpfcV7AoAn9lr+A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=EMqr4TXK; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=fkCbbLc5; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="EMqr4TXK"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="fkCbbLc5" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790232822; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Kki3z56ATUJrZFB3MBKMIKwGUeQgKMm6Uy7jrjVIGAU=; b=EMqr4TXKXe0tgZaaXEXNblm2wTy4LEK6EJsBY+tZ5jBePu1JFSYm67sodHjnKkjA3BjjkL J4B65SaYytivESNTnM3mhg0gOunQ4HWqfDS/ed+Vm6ucPUSTRFogCinnsEPeAaAv+9TIww BUNMVQdwl9SV+EdQLnYyp98sJeBUxlQ= Received: from mail-lj1-f199.google.com (mail-lj1-f199.google.com [209.85.208.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-623-z4E4nGeiPdKyEYzt4XNNTA-1; Thu, 24 Sep 2026 02:53:40 -0400 X-MC-Unique: z4E4nGeiPdKyEYzt4XNNTA-1 X-Mimecast-MFC-AGG-ID: z4E4nGeiPdKyEYzt4XNNTA_1790232819 Received: by mail-lj1-f199.google.com with SMTP id 38308e7fff4ca-3a27721662eso7920391fa.1 for ; Wed, 23 Sep 2026 23:53:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790232819; x=1790837619; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=Kki3z56ATUJrZFB3MBKMIKwGUeQgKMm6Uy7jrjVIGAU=; b=fkCbbLc58K5rCEyBgVACOVqL/2ZXUdHf64ppWFjd9maFuX0gk7Hw2xQpotJ4Oavf/s JpLelxEDoC/x+xpOOccOltjXAQBuz7yFI6eqcJwTIe1im8HXjzenfDF6w84MieN5iy8j KDQ4soYP69Bdyt5YCZhqU1/57uf02gceQQlqxYe5J/fXkmPrnj/fP8yhkrEEP/5ry+Yt HtzomuuGGBNSqwZFCJZidvOxH2gJ2cD2Vc/Mm41gm7eUIyl7MorhMaEzNNhhlcwCU8rH bunJqtzKLAoIwJNT60lHMfiOguiD2jMfaUx9E1YZdfoCqqjYwEiOCY0gdwi0jROrmB/T qtHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790232819; x=1790837619; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=Kki3z56ATUJrZFB3MBKMIKwGUeQgKMm6Uy7jrjVIGAU=; b=Pqiygt/BmfNycEkleHuqxvDM+yC1i3klcynWK7VjU04TPV/1+adOSqD7WTYt9910As m1qEj9mMZhTxIcuOd79mbJ/eZv2naecZ32Lftkk3C2rXf/2dvB3idv5rtn1TOPHloBaD Zs5a8nc0VKe/n5I9whCqkQdta7UMya3YMnkkqM2NlJi9J6IW/USiyo+6j1Go7tjkqla7 +7hc4FJu+8rfPzEjNR7+v89zgqEPWCXOJU9jbSfh6qhDlOHy94+6l/fDWCxF3XEnwqRT zTcu9UrfwXkfWm+QzpKFHDM0XhpXvGDhOVTiLkC+e0CqCgX9pmWAJeYOkRug2j4qW27G II7A== X-Forwarded-Encrypted: i=1; AKwUvBz4ndll8MOULD3rEGrno24i+JKdPokLHSaunogRRznBuBdtGbxcDOwwx8fuHMYYs1uBNECjXdmNQJssCxs=@vger.kernel.org X-Gm-Message-State: AFuF++m1XdnW1gxI3BX/wMjmgw7hwhZAnUbkHBWJjkec8AnHCZENo2jK PPD3S7T2IrT+xvb4H1877Hw1vtobXlHCjZ6GL+X0WL3ax55NkFRbDg6IhJUxedTzvVjTu0wNSJw yeq++/GA933ZpiWa4akJWLsgB44JZJudLzTRyn7SQLXIJccJRX+Dan8li+2ZKDcrz X-Gm-Gg: AYBFou1/HV/95XKbHhUmORACJFCUWI3ZCdmrxvFdWRNmwfsSzColriFKczy3NsjJEYY HZk5Ihcqfi+AFthuDE5Gi3nFQbLm61kf2lnHi966irP4f5RXR1KUBXGeJjp+1Ep6Dl0NYMOmESS DEufOB9ZUBUW/QpsFulJ+9QaUj3W12Zlj/cNIE2oBVrfb0tFpenoJEz7iefJ2ArRznPIhW99r7K fw7eBwaXierNeYpk7QBhhqenE2jOS88HOuqLphzXlQ1HwpwT+88d0OU63luUA9GjuxkRH+bsywD oBvmop3FpKcMwCtg9l7qU3ZadV94dhXr/CmdrNWAqL85fx7MRN3FkYqlCJf/OEPOLULM9AAWJ4U fkpbsgbgz3Po6O2gN8w3JN8zdZZU5B/Y= X-Received: by 2002:a05:651c:b0a:b0:3a3:749e:319b with SMTP id 38308e7fff4ca-3a63de182e4mr2715091fa.15.1790232819299; Wed, 23 Sep 2026 23:53:39 -0700 (PDT) X-Received: by 2002:a05:651c:b0a:b0:3a3:749e:319b with SMTP id 38308e7fff4ca-3a63de182e4mr2714961fa.15.1790232818795; Wed, 23 Sep 2026 23:53:38 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-3a63bf57909sm4803141fa.29.2026.09.23.23.53.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Sep 2026 23:53:36 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v15 03/11] mm/hmm: preparations for HMM to participate in migration Date: Thu, 24 Sep 2026 09:53:05 +0300 Message-ID: <20260924065313.899730-4-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260924065313.899730-1-mpenttil@redhat.com> References: <20260924065313.899730-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 For migration to happen after hmm_range_fault(), the vma has to retrieved. Luckily, pagewalk already resolves that for us, so just have to save it for further use. For migration, mmap_lock has to be hold for the whole operation, so vma stays stable. In case lock-drop behavior is requested, return an error while migrating. Also, rename some struct hmm_vma_walk fields to solve namespace collisions, and we are later introducing more xxlocked fields so be unambiguous. Take care also of firing the mmu_notifier when migrating. Also, prepare for differencies of how mm_struct * is retrieved in fault and migrate_vma paths. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- include/linux/migrate.h | 49 +++++++++++++++++- lib/test_hmm.c | 2 +- mm/hmm.c | 110 ++++++++++++++++++++++++++++++++++------ 3 files changed, 144 insertions(+), 17 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 280ec9a866c7..01c5f62a56e9 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include =20 @@ -108,6 +109,29 @@ static inline void softleaf_entry_wait_on_locked(softl= eaf_t entry, spinlock_t *p spin_unlock(ptl); } =20 +enum migrate_vma_info { + MIGRATE_VMA_SELECT_NONE =3D 0, + MIGRATE_VMA_SELECT_COMPOUND =3D MIGRATE_VMA_SELECT_NONE, +}; + +static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *r= ange) +{ + return MIGRATE_VMA_SELECT_NONE; +} + +static inline void hmm_fill_migrate_vma(struct hmm_range *range, + struct vm_area_struct *vma, + unsigned long start, + unsigned long end) +{ +} + +#ifdef CONFIG_MMU_NOTIFIER +static inline struct mm_struct *hmm_range_fault_mm(struct hmm_range *range) +{ + return range->notifier->mm; +} +#endif #endif /* CONFIG_MIGRATION */ =20 #ifdef CONFIG_NUMA_BALANCING @@ -151,7 +175,7 @@ static inline unsigned long migrate_pfn(unsigned long p= fn) return (pfn << MIGRATE_PFN_SHIFT) | MIGRATE_PFN_VALID; } =20 -enum migrate_vma_direction { +enum migrate_vma_info { MIGRATE_VMA_SELECT_SYSTEM =3D 1 << 0, MIGRATE_VMA_SELECT_DEVICE_PRIVATE =3D 1 << 1, MIGRATE_VMA_SELECT_DEVICE_COHERENT =3D 1 << 2, @@ -193,6 +217,29 @@ struct migrate_vma { struct page *fault_page; }; =20 +// TODO: enable migration +static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *r= ange) +{ + return 0; +} + +#ifdef CONFIG_MMU_NOTIFIER +static inline struct mm_struct *hmm_range_fault_mm(struct hmm_range *range) +{ + return range->notifier ? range->notifier->mm : range->migrate->vma->vm_mm; +} +#endif + +static inline void hmm_fill_migrate_vma(struct hmm_range *range, + struct vm_area_struct *vma, + unsigned long start, + unsigned long end) +{ + range->migrate->vma =3D vma; + range->migrate->start =3D start; + range->migrate->end =3D end; +} + int migrate_vma_setup(struct migrate_vma *args); void migrate_vma_pages(struct migrate_vma *migrate); void migrate_vma_finalize(struct migrate_vma *migrate); diff --git a/lib/test_hmm.c b/lib/test_hmm.c index 6911daa9f854..cd88e8177d0a 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -145,7 +145,7 @@ static bool dmirror_is_private_zone(struct dmirror_devi= ce *mdevice) HMM_DMIRROR_MEMORY_DEVICE_PRIVATE); } =20 -static enum migrate_vma_direction +static enum migrate_vma_info dmirror_select_device(struct dmirror *dmirror) { return (dmirror->mdevice->zone_device_type =3D=3D diff --git a/mm/hmm.c b/mm/hmm.c index 2f1e98c6b644..4e9542aedc25 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -28,15 +29,23 @@ #include #include #include +#include =20 #include "internal.h" =20 struct hmm_vma_walk { - struct hmm_range *range; - bool *locked; - unsigned long last; - unsigned long end; - unsigned int required_fault; + struct mmu_notifier_range mmu_range; + /* Set once mmu_range is armed with mmu_notifier_invalidate_range_start()= */ + bool mmu_armed; + struct vm_area_struct *vma; + struct hmm_range *range; + unsigned long start; + unsigned long end; + unsigned long last; + /* fault and lock drop related fields */ + bool *mmlocked; + unsigned long fault_end; + unsigned int required_fault; }; =20 /* @@ -95,7 +104,7 @@ static int hmm_record_fault(unsigned long addr, unsigned= long end, =20 WARN_ON_ONCE(!required_fault); hmm_vma_walk->last =3D addr; - hmm_vma_walk->end =3D end; + hmm_vma_walk->fault_end =3D end; hmm_vma_walk->required_fault =3D required_fault; return HMM_FAULT_PENDING; } @@ -400,6 +409,58 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *w= alk, unsigned long start, } #endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */ =20 +static int hmm_vma_capture_migrate_range(unsigned long start, + unsigned long end, + struct mm_walk *walk) +{ + struct hmm_vma_walk *hmm_vma_walk =3D walk->private; + struct hmm_range *range =3D hmm_vma_walk->range; + + if (!hmm_select_migrate(range)) + return 0; + + if (hmm_vma_walk->vma && (hmm_vma_walk->vma !=3D walk->vma)) + return -ERANGE; + + hmm_vma_walk->vma =3D walk->vma; + + if (end - start > range->end - range->start) + return -ERANGE; + + if (!hmm_vma_walk->mmu_armed) { + hmm_vma_walk->start =3D start; + hmm_vma_walk->end =3D end; + mmu_notifier_range_init_owner(&hmm_vma_walk->mmu_range, MMU_NOTIFY_MIGRA= TE, 0, + walk->vma->vm_mm, start, end, + range->dev_private_owner); + mmu_notifier_invalidate_range_start(&hmm_vma_walk->mmu_range); + hmm_vma_walk->mmu_armed =3D true; + } + + return 0; +} + +static void hmm_vma_post_range_fault(struct hmm_vma_walk *hmm_vma_walk) +{ + + struct hmm_range *range =3D hmm_vma_walk->range; + + if (hmm_select_migrate(range) && + hmm_vma_walk->mmu_armed) { + /* + * The migrate_vma path has the following initialized, + * so take care of fault path below. + */ + if (range->notifier) { + hmm_fill_migrate_vma(range, + hmm_vma_walk->vma, + hmm_vma_walk->start, + hmm_vma_walk->end); + } + mmu_notifier_invalidate_range_end(&hmm_vma_walk->mmu_range); + } +} + static int hmm_vma_walk_pmd(pmd_t *pmdp, unsigned long start, unsigned long end, @@ -594,6 +655,11 @@ static int hmm_vma_walk_test(unsigned long start, unsi= gned long end, struct hmm_vma_walk *hmm_vma_walk =3D walk->private; struct hmm_range *range =3D hmm_vma_walk->range; struct vm_area_struct *vma =3D walk->vma; + int r; + + r =3D hmm_vma_capture_migrate_range(start, end, walk); + if (r) + return r; =20 if (!(vma->vm_flags & (VM_IO | VM_PFNMAP)) && vma->vm_flags & VM_READ) @@ -646,12 +712,12 @@ static int hmm_do_fault(struct mm_struct *mm, struct hmm_vma_walk *hmm_vma_walk) { unsigned long addr =3D hmm_vma_walk->last; - unsigned long end =3D hmm_vma_walk->end; + unsigned long end =3D hmm_vma_walk->fault_end; unsigned int required_fault =3D hmm_vma_walk->required_fault; unsigned int fault_flags =3D FAULT_FLAG_REMOTE; struct vm_area_struct *vma; =20 - if (hmm_vma_walk->locked) + if (hmm_vma_walk->mmlocked) fault_flags |=3D FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; =20 vma =3D vma_lookup(mm, addr); @@ -670,8 +736,8 @@ static int hmm_do_fault(struct mm_struct *mm, ret =3D handle_mm_fault(vma, addr, fault_flags, NULL); =20 if (ret & (VM_FAULT_COMPLETED | VM_FAULT_RETRY)) { - if (hmm_vma_walk->locked) /* needed by sparse */ - *hmm_vma_walk->locked =3D false; + if (hmm_vma_walk->mmlocked) /* needed by sparse */ + *hmm_vma_walk->mmlocked =3D false; else WARN_ON_ONCE(1); /* broken fault handler */ return HMM_FAULT_UNLOCKED; @@ -694,19 +760,31 @@ static int hmm_range_fault_locked(struct hmm_range *r= ange, bool *locked) { struct hmm_vma_walk hmm_vma_walk =3D { .range =3D range, - .locked =3D locked, + .mmlocked =3D locked, .last =3D range->start, }; - struct mm_struct *mm =3D range->notifier->mm; + /* + * Could be serving a device fault or come from migrate + * entry point. For the former we have not resolved the vma + * yet, and the latter we don't have a notifier (but have a vma). + * + */ + struct mm_struct *mm =3D hmm_range_fault_mm(range); int ret; =20 + /* Migration is incompatible with mmap lock drop */ + if (locked && hmm_select_migrate(range)) + return -EINVAL; + mmap_assert_locked(mm); =20 do { /* If range is no longer valid force retry. */ - if (mmu_interval_check_retry(range->notifier, - range->notifier_seq)) - return -EBUSY; + if (range->notifier && mmu_interval_check_retry(range->notifier, + range->notifier_seq)) { + ret =3D -EBUSY; + break; + } ret =3D walk_page_range(mm, hmm_vma_walk.last, range->end, &hmm_walk_ops, &hmm_vma_walk); /* @@ -732,6 +810,8 @@ static int hmm_range_fault_locked(struct hmm_range *ran= ge, bool *locked) * output, and all >=3D are still at their input values. */ } while (ret =3D=3D -EBUSY); + + hmm_vma_post_range_fault(&hmm_vma_walk); return ret; } =20 --=20 2.55.0 From nobody Thu Sep 24 12:53:00 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3711441F35A for ; Thu, 24 Sep 2026 06:53:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232846; cv=none; b=pi7/18gIcjmT4huT6db47QSV9f5QxMLJLZ30IWmKdi64pwiKGO7OlUGsurTBgfoYd20ex+l7Iw1CnM3cpPvzvQSRQp6ZM2CBcvFK7sxFMjNboq/sfTFJVVRclBPl+EHT6FREj21gHLMM0s/nEUGIN05kkFG2nzmVXRA1vIi4RUA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232846; c=relaxed/simple; bh=CPruOQCSNAIM+qAHYchuaWnwGVf5fK8LoGSdGiN4bvM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=T4ZSXPMewWu9nUw9Dw+zDoFeOSFgjSCdHyyqIzmAvluH2z5AL8D0rpj1p3y6YsEM+OF14yOCvu2edm8np2Efr30e7k3AjWIIYXkinPgP3lGizO3/iSowupRgy7pIAwOJiziFtdjZ7wmPp5rcpBIRQubqJbVWqaW320hdPwMHxjk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=XhqYTIGt; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=hwQYC6UH; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="XhqYTIGt"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="hwQYC6UH" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790232825; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t+NNYQBmmJ6tcL5bo4gUEuagh51ZsPRBTyRL58Mk91k=; b=XhqYTIGtRa+CSJrPi9IZntkJroTvO3S9SsKNUlVSUy/naaqnyDSS7aFaLwISWa5pRXL/Mr XYJ1y7oPOfsbmS6NOJKXAqZUWIY4+n0ieas2Ysnke+wufNvOQ8WLMXAVHm4s4x/LOMGrIF vt5cZqx/DsQOqzte5FKjEPjiTaBvENk= Received: from mail-lj1-f197.google.com (mail-lj1-f197.google.com [209.85.208.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-223-TcPp75s_OUmTZ1mRlPc9yQ-1; Thu, 24 Sep 2026 02:53:44 -0400 X-MC-Unique: TcPp75s_OUmTZ1mRlPc9yQ-1 X-Mimecast-MFC-AGG-ID: TcPp75s_OUmTZ1mRlPc9yQ_1790232823 Received: by mail-lj1-f197.google.com with SMTP id 38308e7fff4ca-3a33567d27dso8532321fa.2 for ; Wed, 23 Sep 2026 23:53:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790232823; x=1790837623; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=t+NNYQBmmJ6tcL5bo4gUEuagh51ZsPRBTyRL58Mk91k=; b=hwQYC6UHU2Pvi8OfT4juBDXGxzeOSwnPRtlOJqB2VCCiDWU+pdvcRPxL0OBos333Cw C6NfK27DRkFrIXBWGS7XkYesr2AOGt1jSe1gtw4rQGMZD+KpjyXL9JdFmR+hIrxw/g62 aC9WKR4V+kZYMdAjHPxEWsNcaCrznIrvWqOzIgWzj/RRFNMEJNx2gH6NBQ/Q/+mNHrqJ F4NNp0Np8dIX4MsTggAfiAoKUfjAj2smtzwrNr/9xJYZfum3QraqsFNSV029y47xt+Xq 4gmq00/YvPjevlUcSa8wUsLFK3WCMCUXR/EwFLcpQ46eeTqCyMK+92uTu+ABNB+Yqr3f XYTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790232823; x=1790837623; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=t+NNYQBmmJ6tcL5bo4gUEuagh51ZsPRBTyRL58Mk91k=; b=oFRJYA7jSdx7DvgMGrWA5aTprixX8B75kmdrHokGYou/kEf10U8k4ATCTpwNBTobiF t2MqxtT/Zr8UBss4JT8q73VaFqMFKlRGfc3Q6/eU8JKkvN4pAEF7ir30u6G0znbmNC9o Rou8YQ46z1KmtdK3STGAj1U5/wf3thqY2n36yPHJA9JTB0B0mGwuDabpjzYBW6rnMJtx OanfeGQhec2zZ9DVkdyvxxoibu0KNzdtgsIZjaUigFIHuKDO48BGSGakiDJICr/5Dmml Q/b3ay5c+YnEkT2c5BnoA5UdP+VeeHgRD7GyCp4L1vkatEHqyjkXK54pQPJICXM2o0nF uhRg== X-Forwarded-Encrypted: i=1; AKwUvByWHwlsWAkxPSxezrpnKAvJriZhD//7Bnxb62LIJmBArkGnXV8qax3MRgHXxMazzOUMSHBTlM/Rr69uvG0=@vger.kernel.org X-Gm-Message-State: AFuF++kfVlPt4dy1BvmvHp+3NuXehEN1ZLKzI0CtTit3zAgjdbAM5uNh 4IadFh196Z2rp0nnVaC3Om7EqoJSldeD0e9P9faWytsuRyHbCiAw69msnSctKTpuPhKgdrFVI3h 8ZV6u9csVYmi1XlzMkfEWyNEM5HmitaHFgP7CWojosZfkMnC96nvXR/+aJ7Mr+/Tt X-Gm-Gg: AYBFou1myNTKGebIBCk4bD8tAe/EDXxl9vL9PDvUnXyTESmVj8ZCPPZ6T7HIyTSAuXQ DlACZg9pK9Gypno1k0lbTGZoL9Xz2pbUUvypFNYQ/a+FlAUHoFqJw9cJV+GRD/L0r+0NCEYXINZ 1+BRJ0DdhG6rtvXE3snoEpmHcWbfZbx44M+puYNHcp80CTxLLWS6KNWtrQIMn5XtW6OGN5Pw0wE 1QN18Gcx+kgy+uG//DFcgJwesxe2BaGA5ZqJgBkpW4eyXM3CxKA7wMCjH7IO8r8rFGXbdA4++H+ PpkmbLwxQamLt9eN3eNWJLO/yY2Ib+0x6ujKiYYwu2nQ23HCd38lAPsgDEgGOhLGQLZQwa9jyPX pCtkNas8WnEtZkRtuFWKR X-Received: by 2002:a05:651c:1990:b0:3a6:16aa:23c4 with SMTP id 38308e7fff4ca-3a63c435595mr3305281fa.26.1790232822709; Wed, 23 Sep 2026 23:53:42 -0700 (PDT) X-Received: by 2002:a05:651c:1990:b0:3a6:16aa:23c4 with SMTP id 38308e7fff4ca-3a63c435595mr3305011fa.26.1790232821294; Wed, 23 Sep 2026 23:53:41 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-3a63bf57909sm4803141fa.29.2026.09.23.23.53.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Sep 2026 23:53:40 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v15 04/11] mm/hmm: do the plumbing for HMM to participate in migration Date: Thu, 24 Sep 2026 09:53:06 +0300 Message-ID: <20260924065313.899730-5-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260924065313.899730-1-mpenttil@redhat.com> References: <20260924065313.899730-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 Do the preparations in hmm_range_fault() and pagewalk callbacks to do the "collecting" part of migration, needed for migration. These steps include locking for pmd/pte if migrating, and calling the still dummy hmm_vma_handle_migrate_prepare_pmd() and hmm_vma_handle_migrate_prepare() functions in the pagewalk. When doing migration, have to have pmd/pte table locked for the duration of both hmm_vma_handle*() and hmm_vma_handle_migrate_prepare*() walks, and unlock when handling the faults. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- mm/hmm.c | 344 ++++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 290 insertions(+), 54 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index 4e9542aedc25..25f537c87b5f 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -42,6 +42,16 @@ struct hmm_vma_walk { unsigned long start; unsigned long end; unsigned long last; + /* + * For migration we need pte/pmd locked for the handle_* and + * prepare_* regions. While faulting we have to drop the locks + * and start again. ptelocked and pmdlocked hold the state + * and tell if need to drop locks before faulting. + * ptl is the lock held for pte or pmd. + */ + bool ptelocked; + bool pmdlocked; + spinlock_t *ptl; /* fault and lock drop related fields */ bool *mmlocked; unsigned long fault_end; @@ -63,6 +73,16 @@ struct hmm_vma_walk { */ #define HMM_FAULT_UNLOCKED -ENOLCK =20 +#define HMM_ASSERT_PTE_LOCKED(hmm_vma_walk, locked) \ + WARN_ON_ONCE((hmm_vma_walk)->ptelocked !=3D locked) + +#define HMM_ASSERT_PMD_LOCKED(hmm_vma_walk, locked) \ + WARN_ON_ONCE((hmm_vma_walk)->pmdlocked !=3D locked) + +#define HMM_ASSERT_UNLOCKED(hmm_vma_walk) \ + WARN_ON_ONCE((hmm_vma_walk)->ptelocked || \ + (hmm_vma_walk)->pmdlocked) + enum { HMM_NEED_FAULT =3D 1 << 0, HMM_NEED_WRITE_FAULT =3D 1 << 1, @@ -76,14 +96,38 @@ enum { }; =20 static int hmm_pfns_fill(unsigned long addr, unsigned long end, - struct hmm_range *range, unsigned long cpu_flags) + struct hmm_vma_walk *hmm_vma_walk, unsigned long cpu_flags) { + struct hmm_range *range =3D hmm_vma_walk->range; unsigned long i =3D (addr - range->start) >> PAGE_SHIFT; + enum migrate_vma_info minfo; + bool migrate =3D false; + + minfo =3D hmm_select_migrate(range); + if (cpu_flags !=3D HMM_PFN_ERROR) { + if (minfo && (vma_is_anonymous(hmm_vma_walk->vma))) { + cpu_flags |=3D HMM_PFN_MIGRATE; + migrate =3D true; + } + } + + if (migrate && thp_migration_supported() && + (minfo & MIGRATE_VMA_SELECT_COMPOUND) && + IS_ALIGNED(addr, HPAGE_PMD_SIZE) && + IS_ALIGNED(end, HPAGE_PMD_SIZE) && + end-addr =3D=3D HPAGE_PMD_SIZE) { + range->hmm_pfns[i] &=3D HMM_PFN_INOUT_FLAGS; + range->hmm_pfns[i] |=3D cpu_flags | HMM_PFN_COMPOUND; + addr +=3D PAGE_SIZE; + i++; + cpu_flags =3D 0; + } =20 for (; addr < end; addr +=3D PAGE_SIZE, i++) { range->hmm_pfns[i] &=3D HMM_PFN_INOUT_FLAGS; range->hmm_pfns[i] |=3D cpu_flags; } + return 0; } =20 @@ -103,6 +147,7 @@ static int hmm_record_fault(unsigned long addr, unsigne= d long end, struct hmm_vma_walk *hmm_vma_walk =3D walk->private; =20 WARN_ON_ONCE(!required_fault); + HMM_ASSERT_UNLOCKED(hmm_vma_walk); hmm_vma_walk->last =3D addr; hmm_vma_walk->fault_end =3D end; hmm_vma_walk->required_fault =3D required_fault; @@ -187,11 +232,16 @@ static int hmm_vma_walk_hole(unsigned long addr, unsi= gned long end, if (!walk->vma) { if (required_fault) return -EFAULT; - return hmm_pfns_fill(addr, end, range, HMM_PFN_ERROR); + return hmm_pfns_fill(addr, end, hmm_vma_walk, HMM_PFN_ERROR); } - if (required_fault) + if (required_fault) { + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } return hmm_record_fault(addr, end, required_fault, walk); - return hmm_pfns_fill(addr, end, range, 0); + } + return hmm_pfns_fill(addr, end, hmm_vma_walk, 0); } =20 static inline unsigned long hmm_pfn_flags_order(unsigned long order) @@ -224,8 +274,13 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, un= signed long addr, cpu_flags =3D pmd_to_hmm_pfn_flags(range, pmd); required_fault =3D hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, cpu_flags); - if (required_fault) + if (required_fault) { + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } return hmm_record_fault(addr, end, required_fault, walk); + } =20 pfn =3D pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); for (i =3D 0; addr < end; addr +=3D PAGE_SIZE, i++, pfn++) { @@ -305,14 +360,25 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, u= nsigned long addr, goto fault; =20 if (softleaf_is_migration(entry)) { - pte_unmap(ptep); - hmm_vma_walk->last =3D addr; - migration_entry_wait(walk->mm, pmdp, addr); - return -EBUSY; + if (!hmm_select_migrate(range)) { + HMM_ASSERT_UNLOCKED(hmm_vma_walk); + pte_unmap(ptep); + hmm_vma_walk->last =3D addr; + migration_entry_wait(walk->mm, pmdp, addr); + return -EBUSY; + } + return 0; } =20 /* Report error for everything else */ - pte_unmap(ptep); + + if (hmm_vma_walk->ptelocked) { + pte_unmap_unlock(ptep, hmm_vma_walk->ptl); + hmm_vma_walk->ptelocked =3D false; + } else { + pte_unmap(ptep); + } + return -EFAULT; } =20 @@ -329,7 +395,13 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, un= signed long addr, if (!vm_normal_page(walk->vma, addr, pte) && !is_zero_pfn(pte_pfn(pte))) { if (hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0)) { - pte_unmap(ptep); + if (hmm_vma_walk->ptelocked) { + pte_unmap_unlock(ptep, hmm_vma_walk->ptl); + hmm_vma_walk->ptelocked =3D false; + } else { + pte_unmap(ptep); + } + return -EFAULT; } new_pfn_flags =3D HMM_PFN_ERROR; @@ -342,7 +414,12 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, un= signed long addr, return 0; =20 fault: - pte_unmap(ptep); + if (hmm_vma_walk->ptelocked) { + pte_unmap_unlock(ptep, hmm_vma_walk->ptl); + hmm_vma_walk->ptelocked =3D false; + } else { + pte_unmap(ptep); + } /* Fault any virtual address we were asked to fault */ return hmm_record_fault(addr, end, required_fault, walk); } @@ -386,13 +463,18 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *= walk, unsigned long start, required_fault =3D hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0); if (required_fault) { - if (softleaf_is_device_private(entry)) + if (softleaf_is_device_private(entry)) { + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } return hmm_record_fault(addr, end, required_fault, walk); + } else return -EFAULT; } =20 - return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR); + return hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR); } #else static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long s= tart, @@ -400,15 +482,58 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *= walk, unsigned long start, pmd_t pmd) { struct hmm_vma_walk *hmm_vma_walk =3D walk->private; - struct hmm_range *range =3D hmm_vma_walk->range; unsigned long npages =3D (end - start) >> PAGE_SHIFT; =20 if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0)) return -EFAULT; - return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR); + return hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR); } #endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */ =20 +#ifdef CONFIG_DEVICE_MIGRATION +static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk, + pmd_t *pmdp, + unsigned long start, + unsigned long end, + unsigned long *hmm_pfn) +{ + // TODO: implement migration entry insertion + return 0; +} + +static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk, + pmd_t *pmdp, + pte_t *ptep, + unsigned long addr, + unsigned long *hmm_pfn, + bool *unmapped) +{ + // TODO: implement migration entry insertion + return 0; +} + +#else +static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk, + pmd_t *pmdp, + unsigned long start, + unsigned long end, + unsigned long *hmm_pfn) +{ + return 0; +} + +static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk, + pmd_t *pmdp, + pte_t *ptep, + unsigned long addr, + unsigned long *hmm_pfn, + bool *unmapped) +{ + return 0; +} + +#endif + static int hmm_vma_capture_migrate_range(unsigned long start, unsigned long end, struct mm_walk *walk) @@ -468,46 +593,117 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, { struct hmm_vma_walk *hmm_vma_walk =3D walk->private; struct hmm_range *range =3D hmm_vma_walk->range; - unsigned long *hmm_pfns =3D - &range->hmm_pfns[(start - range->start) >> PAGE_SHIFT]; unsigned long npages =3D (end - start) >> PAGE_SHIFT; + struct mm_struct *mm =3D walk->vma->vm_mm; + unsigned long *hmm_pfns, *hmm_pfns_start; + enum migrate_vma_info minfo; unsigned long addr =3D start; + bool unmapped =3D false; + unsigned long i; pte_t *ptep; pmd_t pmd; + int r =3D 0; =20 + minfo =3D hmm_select_migrate(range); + hmm_pfns_start =3D &range->hmm_pfns[(start - range->start) >> PAGE_SHIFT]; again: - pmd =3D pmdp_get_lockless(pmdp); - if (pmd_none(pmd)) - return hmm_vma_walk_hole(start, end, -1, walk); + hmm_pfns =3D &range->hmm_pfns[(addr - range->start) >> PAGE_SHIFT]; + hmm_vma_walk->ptelocked =3D false; + hmm_vma_walk->pmdlocked =3D false; + + if (minfo) { + hmm_vma_walk->ptl =3D pmd_lock(mm, pmdp); + hmm_vma_walk->pmdlocked =3D true; + pmd =3D pmdp_get(pmdp); + } else + pmd =3D pmdp_get_lockless(pmdp); + + if (pmd_none(pmd)) { + r =3D hmm_vma_walk_hole(start, end, -1, walk); + + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } + return r; + } =20 if (thp_migration_supported() && pmd_is_migration_entry(pmd)) { - if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0)) { - hmm_vma_walk->last =3D addr; - pmd_migration_entry_wait(walk->mm, pmdp); - return -EBUSY; + if (!minfo) { + if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns_start, npages, 0)) { + hmm_vma_walk->last =3D addr; + pmd_migration_entry_wait(walk->mm, pmdp); + return -EBUSY; + } + for (i =3D 0; start < end; start +=3D PAGE_SIZE, i++) + hmm_pfns_start[i] &=3D HMM_PFN_INOUT_FLAGS; + } + + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; } - return hmm_pfns_fill(start, end, range, 0); + + return 0; } =20 - if (!pmd_present(pmd)) - return hmm_vma_handle_absent_pmd(walk, start, end, hmm_pfns, - pmd); + if (pmd_trans_huge(pmd) || !pmd_present(pmd)) { + if (!pmd_present(pmd)) { + r =3D hmm_vma_handle_absent_pmd(walk, start, end, hmm_pfns_start, + pmd); + // If not migrating we are done + if (r || !minfo) { + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } + return r; + } + } =20 - if (pmd_trans_huge(pmd)) { - /* - * No need to take pmd_lock here, even if some other thread - * is splitting the huge pmd we will get that event through - * mmu_notifier callback. - * - * So just read pmd value and check again it's a transparent - * huge or device mapping one and compute corresponding pfn - * values. - */ - pmd =3D pmdp_get_lockless(pmdp); - if (!pmd_trans_huge(pmd)) - goto again; + if (pmd_trans_huge(pmd)) { + /* + * No need to take pmd_lock here if not migrating, + * even if some other thread is splitting the huge + * pmd we will get that event through mmu_notifier callback. + * + * So just read pmd value and check again it's a transparent + * huge or device mapping one and compute corresponding pfn + * values. + */ + + if (!minfo) { + pmd =3D pmdp_get_lockless(pmdp); + if (!pmd_trans_huge(pmd)) + goto again; + } =20 - return hmm_vma_handle_pmd(walk, addr, end, hmm_pfns, pmd); + r =3D hmm_vma_handle_pmd(walk, start, end, hmm_pfns_start, pmd); + + // If not migrating we are done + if (r || !minfo) { + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } + return r; + } + } + + r =3D hmm_vma_handle_migrate_prepare_pmd(walk, pmdp, start, end, hmm_pfn= s_start); + + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } + + return r; + + } + + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; } =20 /* @@ -517,24 +713,64 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, * recover. */ if (pmd_bad(pmd)) { - if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0)) + if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns_start, npages, 0)) return -EFAULT; - return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR); + return hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR); } =20 - ptep =3D pte_offset_map(pmdp, addr); - if (!ptep) + if (minfo) { + ptep =3D pte_offset_map_lock(mm, pmdp, addr, &hmm_vma_walk->ptl); + if (ptep) + hmm_vma_walk->ptelocked =3D true; + } else { + ptep =3D pte_offset_map(pmdp, addr); + } + if (!ptep) { + addr =3D start; goto again; + } + for (; addr < end; addr +=3D PAGE_SIZE, ptep++, hmm_pfns++) { - int r; =20 r =3D hmm_vma_handle_pte(walk, addr, end, pmdp, ptep, hmm_pfns); if (r) { - /* hmm_vma_handle_pte() did pte_unmap() */ + /* hmm_vma_handle_pte() did pte_unmap() / pte_unmap_unlock */ + if (unmapped) + flush_tlb_range(walk->vma, start, addr); return r; } + + r =3D hmm_vma_handle_migrate_prepare(walk, pmdp, ptep, addr, hmm_pfns, &= unmapped); + if (r =3D=3D -EAGAIN) { + HMM_ASSERT_UNLOCKED(hmm_vma_walk); + if (unmapped) { + flush_tlb_range(walk->vma, start, addr); + unmapped =3D false; + } + goto again; + } + if (r) { + /* A non -EAGAIN error here means migrate_vma_split_folio() + * already dropped the PTE lock and cleared ptelocked. + */ + HMM_ASSERT_UNLOCKED(hmm_vma_walk); + if (unmapped) + flush_tlb_range(walk->vma, start, addr); + hmm_pfns_fill(addr, end, hmm_vma_walk, HMM_PFN_ERROR); + return 0; + } } - pte_unmap(ptep - 1); + + if (unmapped) + flush_tlb_range(walk->vma, start, addr); + + if (hmm_vma_walk->ptelocked) { + pte_unmap_unlock(ptep - 1, hmm_vma_walk->ptl); + hmm_vma_walk->ptelocked =3D false; + } else { + pte_unmap(ptep - 1); + } + return 0; } =20 @@ -682,7 +918,7 @@ static int hmm_vma_walk_test(unsigned long start, unsig= ned long end, (end - start) >> PAGE_SHIFT, 0)) return -EFAULT; =20 - hmm_pfns_fill(start, end, range, HMM_PFN_ERROR); + hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR); =20 /* Skip this vma and continue processing the next vma. */ return 1; @@ -798,9 +1034,9 @@ static int hmm_range_fault_locked(struct hmm_range *ra= nge, bool *locked) if (ret =3D=3D HMM_FAULT_PENDING) { ret =3D hmm_do_fault(mm, &hmm_vma_walk); if (ret =3D=3D HMM_FAULT_UNLOCKED) { - if (fatal_signal_pending(current)) - return -EINTR; - return -EBUSY; + ret =3D fatal_signal_pending(current) ? -EINTR : + -EBUSY; + break; } } /* --=20 2.55.0 From nobody Thu Sep 24 12:53:00 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C293C410D36 for ; Thu, 24 Sep 2026 06:53:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232849; cv=none; b=A9HP28awQOkkuSpQ+4WEg3SOTF97q2cUh9DT20hwwFCinNoNwGs/5VLupDFnVwI5hboHD5Rmr/5QhuqCs8j3W8IUjaK13GmpuIQCNKIWKboas7qzWoqwA/XwRmEODtYlwXNDUBRT0e8Hd/+MkRmte+mpo+jEIY/ZEgZoox61tx8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232849; c=relaxed/simple; bh=1LPu3V9Ei+sHzoWF2SKd5yIugDRbYsEKXozoa/7EZmM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=LquGSS9j5UoUf7Ag3cXcDuU265eTAiWjvEHhR+eav8U603Mh6UzdxYbWem91L9hduXASYTl4NRUE09usffg/t2XLIx4EVU/4cfzQCSCIFmCHz0TOUzPY8DRGDHF2XxyG4Q4lOQZbeTtH3bcZHjEUbNU1Ijsq8vBPDA85GvpkR3I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=LtOYRA/c; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=OHzKLovn; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="LtOYRA/c"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="OHzKLovn" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790232831; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sKufge1uGAYO+MtJk2WzsKER/XWhb707zmmEu/uQZ4Q=; b=LtOYRA/cC6ac6vuo5MVaw4avQXg154tpb35+411NPIa6i0AhUB/OwJNOMpVLLTW9OjynAl da0bKBUky5Nu0FRhfpZffTQUkL9IwIWrrhjbSIDMNAWdeeOulv91RHdRcfvjvdccaP3cth TsWW7n+Djh8VmuZrEzDbn2ezLzDSp+Y= Received: from mail-lj1-f200.google.com (mail-lj1-f200.google.com [209.85.208.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-19-MbqZQIKHOVC4UwFBZmNFsw-1; Thu, 24 Sep 2026 02:53:50 -0400 X-MC-Unique: MbqZQIKHOVC4UwFBZmNFsw-1 X-Mimecast-MFC-AGG-ID: MbqZQIKHOVC4UwFBZmNFsw_1790232829 Received: by mail-lj1-f200.google.com with SMTP id 38308e7fff4ca-3a1fa5bb969so7167291fa.0 for ; Wed, 23 Sep 2026 23:53:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790232829; x=1790837629; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=sKufge1uGAYO+MtJk2WzsKER/XWhb707zmmEu/uQZ4Q=; b=OHzKLovn3u/e7vudiLAugY2JB93YoeiWSXxPWf3yCVu8UhEM98IyfLiB6pp6QwSngW OSYIsW4nQB+vm5TFaRbmlL5P8Zu0poeip4DZOCZ5flqGddbHdPb+CK8aqOSIl1mvz1/W dhs8Qop1gzO8n2b0WRMSPVieXJy0VktuZPt2kgF7GF+2uJM0GoHP5jmojoMaiObFh3lJ 7YbPpI8SPlwnAaMFy2PAMjI2l91JgntJJLFpUfDIVnbEFi7fSE+NFKinP23bdO1WSGzG GHLBbt5/8sby+GOE22xfYgBxGRmcfuUxuxvtv4Qj5jObwwuQOUwY0UmuvxuhVM95rn1+ xEvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790232829; x=1790837629; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=sKufge1uGAYO+MtJk2WzsKER/XWhb707zmmEu/uQZ4Q=; b=Xp/X8iX1BSZXuTvE7z0aPFEqUqjigyA6jAMbA3nrjo7Rdtwttk6pOT/4D4voYQegAz ETFDIde8WCTjwEv3r1qcE/Z0Paj7fG59x1XZyT7URwaDZovG/OV4LkI1nUbS+3Wa+I4Q MFfI38SACmAMofkThZyf825Zx8RyN9qtjpqTv0CqyIfkDeLU5NaHC07rpsExw/Okuppb 8YrH86peladctVubW/hkTesrt26tOERQtibRw+Z21WFKvNqm1GMB5a4ydZTgXJCNz5Z8 9q8nRNjkzHH2UlPy1VlTn1tyLY4r2YvBF5oXrPcGofRreXyIZubHmVFmX6VgNFLhdlNY M4yA== X-Forwarded-Encrypted: i=1; AKwUvBy5JU59nt8x+FEx+S++5zZwGiAmkFyZHyD2BA4aN+4C8eaBdOSSP9bc7CaODKYJjr8IIZm12kjxaNKc6T0=@vger.kernel.org X-Gm-Message-State: AFuF++lcANiM628LfiK8dSwr3ISKUM1FfMefvTOwZy9Ul/yMQ4HYtFgh vfsb0Tz+R/G7ikRoiGNVaHhwjeOMNCW6HMb6x4h9LUE/CWU4EAf60shsMIEeYj+/MHmqnptZJdf rRtDH0JnkQJOBZCC3lSley0sdPV99E+G0lzG3WBC/pCXXObwjCNIc8i44HzRUYxan X-Gm-Gg: AYBFou0XzVVCa3b5ilZILbje6HIbRUprc3C6vrjMjI1ChzG5cRDr75W05JHe49MV6mf QQJqW+8fOMkyitMcrxPIhBxx0aw4q3xzKDU6AxMZCuItBAxm8o6BsXi+FA9lVCFNCy3tR5NEQ6W 4VKVlHIAmd5DMQmw9Bqz6PSSwkz+m9BHCNELzUMJRouxptQe3r0DUzag/kAvn+PE3hNgnTsQSSM 2Z+dBlETJYvhEaR0UBCin6CdMlRapGbXh+Mg+96c7TzrbQEeihqsYXeppybK1ccoam4YOpzSmvE 2c7DdLYaM3bgEKTDfwZiuPg6J5p2h+REFnmvqb0yaZYy4dNatvD+ogIop7i8Scj9Ztwqxvsy4yK NLn8xy8P8/S5P44IvoJFC X-Received: by 2002:a2e:a58e:0:b0:3a5:d070:9ffd with SMTP id 38308e7fff4ca-3a63bf294a6mr3893291fa.2.1790232828685; Wed, 23 Sep 2026 23:53:48 -0700 (PDT) X-Received: by 2002:a2e:a58e:0:b0:3a5:d070:9ffd with SMTP id 38308e7fff4ca-3a63bf294a6mr3893181fa.2.1790232828134; Wed, 23 Sep 2026 23:53:48 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-3a63bf57909sm4803141fa.29.2026.09.23.23.53.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Sep 2026 23:53:47 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v15 05/11] mm/hmm: migrate collection in HMM pagewalk - pte level Date: Thu, 24 Sep 2026 09:53:07 +0300 Message-ID: <20260924065313.899730-6-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260924065313.899730-1-mpenttil@redhat.com> References: <20260924065313.899730-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 Implement the needed hmm_vma_handle_migrate_prepare() function which is mostly carried over from migrate_device.c's migrate_vma_collect_pmd() function. Also implement the migrate_vma_split_folio(), for splitting pte mapped large folios. It is also mostly from migrate_device.c, with care taken to reference folio before relasing page table lock. With HMM pagewalk based migration, the idea is that hmm_vma_handle_*() are responsible for faulting, and the pfn collecting part. hmm_vma_handle_migrate_prepare*() do the migration decisions (with HMM_PFN_MIGRATE), possibly split folios, and insert migration ptes/pmds. HMM pagewalk based migration is enabled in later commit, for now now hmm_select_migrate() returns 0. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- mm/hmm.c | 263 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 262 insertions(+), 1 deletion(-) diff --git a/mm/hmm.c b/mm/hmm.c index 25f537c87b5f..a60b66963049 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -491,6 +491,63 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *w= alk, unsigned long start, #endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */ =20 #ifdef CONFIG_DEVICE_MIGRATION +/** + * migrate_vma_split_folio() - Helper function to split a THP folio + * @folio: the folio to split + * @fault_page: struct page associated with the fault if any + * @hmm_vma_walk: walk in progress + * @ptep: pte_t * for unmap and unlock ptl + * + * Returns 0 on success + */ +static int migrate_vma_split_folio(struct folio *folio, + struct page *fault_page, + struct hmm_vma_walk *hmm_vma_walk, + pte_t *ptep) +{ + int ret; + struct folio *fault_folio =3D fault_page ? page_folio(fault_page) : NULL; + struct folio *new_fault_folio =3D NULL; + + if (folio !=3D fault_folio) + folio_get(folio); + + pte_unmap_unlock(ptep, hmm_vma_walk->ptl); + hmm_vma_walk->ptelocked =3D false; + + if (folio !=3D fault_folio) + folio_lock(folio); + + ret =3D split_folio(folio); + if (ret) { + if (folio !=3D fault_folio) { + folio_unlock(folio); + folio_put(folio); + } + return ret; + } + + new_fault_folio =3D fault_page ? page_folio(fault_page) : NULL; + + /* + * Ensure the lock is held on the correct + * folio after the split + */ + if (!new_fault_folio) { + folio_unlock(folio); + folio_put(folio); + } else if (folio !=3D new_fault_folio) { + if (new_fault_folio !=3D fault_folio) { + folio_get(new_fault_folio); + folio_lock(new_fault_folio); + } + folio_unlock(folio); + folio_put(folio); + } + + return 0; +} + static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk, pmd_t *pmdp, unsigned long start, @@ -501,6 +558,11 @@ static int hmm_vma_handle_migrate_prepare_pmd(const st= ruct mm_walk *walk, return 0; } =20 +/* + * Install migration entries if migration requested, either from fault + * or migrate paths. + * + */ static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk, pmd_t *pmdp, pte_t *ptep, @@ -508,8 +570,207 @@ static int hmm_vma_handle_migrate_prepare(const struc= t mm_walk *walk, unsigned long *hmm_pfn, bool *unmapped) { - // TODO: implement migration entry insertion + struct hmm_vma_walk *hmm_vma_walk =3D walk->private; + struct hmm_range *range =3D hmm_vma_walk->range; + struct migrate_vma *migrate =3D range->migrate; + struct mm_struct *mm =3D walk->vma->vm_mm; + struct folio *fault_folio =3D NULL; + enum migrate_vma_info minfo; + struct dev_pagemap *pgmap; + bool anon_exclusive; + struct folio *folio; + unsigned long pfn; + struct page *page; + softleaf_t entry; + pte_t pte, swp_pte; + bool writable =3D false; + + // Do we want to migrate at all? + minfo =3D hmm_select_migrate(range); + if (!minfo) + return 0; + + WARN_ON_ONCE(!migrate); + HMM_ASSERT_PTE_LOCKED(hmm_vma_walk, true); + + fault_folio =3D migrate->fault_page ? + page_folio(migrate->fault_page) : NULL; + + pte =3D ptep_get(ptep); + + if (pte_none(pte)) { + if (vma_is_anonymous(walk->vma)) { + *hmm_pfn &=3D HMM_PFN_INOUT_FLAGS; + *hmm_pfn |=3D HMM_PFN_MIGRATE; + goto out; + } + } + + if (!(hmm_pfn[0] & HMM_PFN_VALID)) + goto out; + + if (!pte_present(pte)) { + /* + * Only care about unaddressable device page special + * page table entry. Other special swap entries are not + * migratable, and we ignore regular swapped page. + */ + entry =3D softleaf_from_pte(pte); + if (!softleaf_is_device_private(entry)) + goto out; + + if (!(minfo & MIGRATE_VMA_SELECT_DEVICE_PRIVATE)) + goto out; + + page =3D softleaf_to_page(entry); + folio =3D page_folio(page); + if (folio->pgmap->owner !=3D migrate->pgmap_owner) + goto out; + + if (folio_test_large(folio)) { + int ret; + + ret =3D migrate_vma_split_folio(folio, + migrate->fault_page, + hmm_vma_walk, + ptep); + if (ret) + goto out_error; + return -EAGAIN; + } + + pfn =3D page_to_pfn(page); + if (softleaf_is_device_private_write(entry)) + writable =3D true; + } else { + pfn =3D pte_pfn(pte); + if (is_zero_pfn(pfn) && + (minfo & MIGRATE_VMA_SELECT_SYSTEM)) { + *hmm_pfn =3D HMM_PFN_MIGRATE; + goto out; + } + page =3D vm_normal_page(walk->vma, addr, pte); + if (page && !is_zone_device_page(page) && + !(minfo & MIGRATE_VMA_SELECT_SYSTEM)) { + goto out; + } else if (page && is_device_coherent_page(page)) { + pgmap =3D page_pgmap(page); + + if (!(minfo & + MIGRATE_VMA_SELECT_DEVICE_COHERENT) || + pgmap->owner !=3D migrate->pgmap_owner) + goto out; + } + + folio =3D page ? page_folio(page) : NULL; + if (folio && folio_test_large(folio)) { + int ret; + + ret =3D migrate_vma_split_folio(folio, + migrate->fault_page, + hmm_vma_walk, + ptep); + if (ret) + goto out_error; + return -EAGAIN; + } + + writable =3D pte_write(pte); + } + + if (!page || !page->mapping) + goto out; + + /* + * By getting a reference on the folio we pin it and that blocks + * any kind of migration. Side effect is that it "freezes" the + * pte. + * + * We drop this reference after isolating the folio from the lru + * for non device folio (device folio are not on the lru and thus + * can't be dropped from it). + */ + folio =3D page_folio(page); + folio_get(folio); + + /* + * We rely on folio_trylock() to avoid deadlock between + * concurrent migrations where each is waiting on the others + * folio lock. If we can't immediately lock the folio we fail this + * migration as it is only best effort anyway. + * + * If we can lock the folio it's safe to set up a migration entry + * now. In the common case where the folio is mapped once in a + * single process setting up the migration entry now is an + * optimisation to avoid walking the rmap later with + * try_to_migrate(). + */ + + if (fault_folio =3D=3D folio || folio_trylock(folio)) { + anon_exclusive =3D folio_test_anon(folio) && + PageAnonExclusive(page); + + if (pte_present(pte)) + flush_cache_page(walk->vma, addr, pfn); + + if (anon_exclusive) { + pte =3D ptep_clear_flush(walk->vma, addr, ptep); + + if (folio_try_share_anon_rmap_pte(folio, page)) { + set_pte_at(mm, addr, ptep, pte); + if (fault_folio !=3D folio) + folio_unlock(folio); + folio_put(folio); + goto out; + } + } else { + pte =3D ptep_get_and_clear(mm, addr, ptep); + } + + if (pte_present(pte) && pte_dirty(pte)) + folio_mark_dirty(folio); + + /* Setup special migration page table entry */ + if (writable) + entry =3D make_writable_migration_entry(pfn); + else if (anon_exclusive) + entry =3D make_readable_exclusive_migration_entry(pfn); + else + entry =3D make_readable_migration_entry(pfn); + + if (pte_present(pte)) { + if (pte_young(pte)) + entry =3D make_migration_entry_young(entry); + if (pte_dirty(pte)) + entry =3D make_migration_entry_dirty(entry); + } + + swp_pte =3D swp_entry_to_pte(entry); + if (pte_present(pte)) { + if (pte_soft_dirty(pte)) + swp_pte =3D pte_swp_mksoft_dirty(swp_pte); + if (pte_uffd(pte)) + swp_pte =3D pte_swp_mkuffd(swp_pte); + } else { + if (pte_swp_soft_dirty(pte)) + swp_pte =3D pte_swp_mksoft_dirty(swp_pte); + if (pte_swp_uffd(pte)) + swp_pte =3D pte_swp_mkuffd(swp_pte); + } + + set_pte_at(mm, addr, ptep, swp_pte); + folio_remove_rmap_pte(folio, page, walk->vma); + folio_put(folio); + *hmm_pfn |=3D HMM_PFN_MIGRATE; + if (pte_present(pte)) + *unmapped =3D true; + } else { + folio_put(folio); + } +out: return 0; +out_error: + return -EFAULT; } =20 #else --=20 2.55.0 From nobody Thu Sep 24 12:53:00 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6FD141D207 for ; Thu, 24 Sep 2026 06:54:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232855; cv=none; b=LyRaA3VP0cu7Sw09/1WMfNVjdEjyob9Dg4/7kr5pvbhcx9flGuIzI5nRmDlqTylIWBJ0GZxBHr3jjZFTjAuNQbiL7fYJNTj+hjUwUPa4uaWNZG0v/ucDiQeNZAoZ1jTOgnnCXM/T5yh8rNLvhZfLEQPuVpDp9K+g6T3Ss/Cy7ZM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232855; c=relaxed/simple; bh=8pe60ShfWybuaCh7m4Hq1ZUgjlmzJ1u0Ok6lD6w1+c8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=PjlavwvFhbneE69I8EcAHOsw1rOK2d0rusqijP5EuvrQdq7or4+LtSPJoKy9f7jUNnVDCgPwRCNhzrylEV2zIpeYk2dku1RvG70VflZL8idgzvgoGSe2jCve8R2ejAin7X9/QUIv0EsMR+MAj35TmKHYV3A/KMZMuw84+o+EK2Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Ic87cHHr; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=EmDFQ+2g; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Ic87cHHr"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="EmDFQ+2g" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790232839; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JpYqOYoRlVZdrcLlVTpkBL4vVH/KqUerRKv2x17E1pA=; b=Ic87cHHrTOGB6Eq4zxXvbshwClNbb9CmxHM8ynVkU/h1U2JrbyTkkL9GafXsm4irFYAMUN iKjSQwzNhLnZJylIFCzrlZ2dW+UaUFNecMv/hdPJw5X+YoPFre5jHuChTuathfS1CFj164 1FbGytcyI3l2OQXW71vNPvI3jN9+zMk= Received: from mail-lj1-f199.google.com (mail-lj1-f199.google.com [209.85.208.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-36-OnmJEGtQP4u5v0JD8soHEw-1; Thu, 24 Sep 2026 02:53:55 -0400 X-MC-Unique: OnmJEGtQP4u5v0JD8soHEw-1 X-Mimecast-MFC-AGG-ID: OnmJEGtQP4u5v0JD8soHEw_1790232834 Received: by mail-lj1-f199.google.com with SMTP id 38308e7fff4ca-3a5b062d0dbso3518731fa.0 for ; Wed, 23 Sep 2026 23:53:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790232834; x=1790837634; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=JpYqOYoRlVZdrcLlVTpkBL4vVH/KqUerRKv2x17E1pA=; b=EmDFQ+2g821EWhcS6eJUpxpeWtL4I501ijMTA6RbX8wdx1S0IPf3CJwJGRmwTS1ndn ka9OXC7R/tZLW5PnyXh657e+UKY4mEf8n5Aa8Q1B6klk5ZJJe8qjGr0vy4pcKEottyqs LGsZEhhmMWwZ0ZhCbLaL8N/nFFYxM2Ezzl6cmIYbwd9YFZTxWW5ZBXyHmYoIoMFgbMuS s1RE/dX0jD5jJgqQKewCoB5iCTILTrz/ljIGr8C0K/+ZrqlEhFEkoXmH07fbLjd+MPRe SGZVW0qvR2mCCNga6djaeV7zfLXbvQ19e8kEkxgk1KrqZ9qVoXkkPMMl/VbhJy3u6VuU MpQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790232834; x=1790837634; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=JpYqOYoRlVZdrcLlVTpkBL4vVH/KqUerRKv2x17E1pA=; b=XUxNH9c2+SbsRDkk+NBGrfst2d9MG0+yccLnXFVnkAPevjImh2+KSfmPI47BMngVNW CK39T0qpeg4I3YkgLgFhI5CYSoLJAL42E58Fx4vAwlZl0C07dPtIYrVpIsqEvLutagxV O9IuQE+lKUHozVaFNXJdj9X+kzCmJJYqenBlb/IKaKKtOP+gLorzzwh/P+STC0TwU8uC oy8G3KfyeG55/bG6tOruJHXXn+YaCerlVT4/+ztKw48x1PogPK/pQuzri1JPfOZOzTYH v9Z9AJsXBaFEXdO0+X+77YhSJQ8m8P551B5npAa01voptn0LSfuErKHE4PhoiNt2CnjY eZcA== X-Forwarded-Encrypted: i=1; AKwUvBw2znXyDEbGcELINWZlspNekAcK+nie6wXpQLXLHdoJVFUW3WDFmc235qSCDsC5Geplam1hnv4vyNBPtEc=@vger.kernel.org X-Gm-Message-State: AFuF++nDoP6fjQGFQej2tcva5jMe8EEYQ4EhFNzRcY4HBVVsAVTgYDg7 xxLkUbxK64V5Vy7NsbPDKoQdLb+dNxXf8JKBxo0ZbND+Hno729LFco7nHvKpx/6fQZx3jhwGaLe yQfvApHAxCj14QSMcLX7y9q/TBIJA3wFaxBKYGAnrNMeEQmIkQ9QF/A5cZ2JGg8U6 X-Gm-Gg: AYBFou02vX5/o9xYAqQR09dZTsecYPgCQQi1m28zEv+2V8zhpumUdgf9DxmgRPr6GrQ iAlwqTe0OV9DKb2Ojouf9Ula0wKi/bVFz2LkYRjRR3CEd97HjKD8s5ZuIVVNbbDnZnV3TRQ8tTs Rt58Jt6k//rowSWwb+h9M/mt5bdsJ2UyDBAL5IAqWhUEvnXruzBHWwBI90i8d4WTcl407EK0HaP tyV0TlMR73kuDOLkopEWeXOvTpZPxyMH5ZbqsF2TVSzhRUiSZGwPpStD+ri6EaYhzCPeJtsgl2u Jt3+3bWziJQQXaPKA2Q1LcbcAJVIU+lC0vRQSEwP+c3mvPbGxhaLvk0Yn4mH/GGefacJIRXnvFu JS4GikA0oIkB2FkALkfF+ X-Received: by 2002:a05:651c:54f:b0:3a4:9881:8de0 with SMTP id 38308e7fff4ca-3a63bf5cf4cmr3206521fa.7.1790232834180; Wed, 23 Sep 2026 23:53:54 -0700 (PDT) X-Received: by 2002:a05:651c:54f:b0:3a4:9881:8de0 with SMTP id 38308e7fff4ca-3a63bf5cf4cmr3206461fa.7.1790232833678; Wed, 23 Sep 2026 23:53:53 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-3a63bf57909sm4803141fa.29.2026.09.23.23.53.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Sep 2026 23:53:53 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v15 06/11] mm/hmm: migrate collection in HMM pagewalk - pmd level Date: Thu, 24 Sep 2026 09:53:08 +0300 Message-ID: <20260924065313.899730-7-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260924065313.899730-1-mpenttil@redhat.com> References: <20260924065313.899730-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 Implement the needed hmm_vma_handle_migrate_prepare_pmd() function which is mostly carried over from migrate_device.c's migrate_vma_collect_huge_pmd() function. With HMM pagewalk based migration, the idea is that hmm_vma_handle_*() are responsible for faulting, and the pfn collecting part. hmm_vma_handle_migrate_prepare*() do the migration decisions (with HMM_PFN_MIGRATE), possibly split folios, and insert migration ptes/pmds. HMM pagewalk based migration is enabled in later commit, for now now hmm_select_migrate() returns 0. Also avoid the problems present in the migrate_vma_collect_huge_pmd() split fallback path this HMM path supersedes: - extra refcount for fault_folio makes split_folio() fail - wrong folio unlocked after split_folio() Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- mm/hmm.c | 173 ++++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 151 insertions(+), 22 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index a60b66963049..ea77429c5178 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -492,31 +492,19 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *= walk, unsigned long start, =20 #ifdef CONFIG_DEVICE_MIGRATION /** - * migrate_vma_split_folio() - Helper function to split a THP folio + * __migrate_vma_split_folio() - split a folio and move the lock/ref to the + * order-0 folio backing @fault_page after the split * @folio: the folio to split - * @fault_page: struct page associated with the fault if any - * @hmm_vma_walk: walk in progress - * @ptep: pte_t * for unmap and unlock ptl + * @fault_page: fault page if any * - * Returns 0 on success + * Returns 0 on success. */ -static int migrate_vma_split_folio(struct folio *folio, - struct page *fault_page, - struct hmm_vma_walk *hmm_vma_walk, - pte_t *ptep) +static int __migrate_vma_split_folio(struct folio *folio, + struct page *fault_page) { - int ret; struct folio *fault_folio =3D fault_page ? page_folio(fault_page) : NULL; struct folio *new_fault_folio =3D NULL; - - if (folio !=3D fault_folio) - folio_get(folio); - - pte_unmap_unlock(ptep, hmm_vma_walk->ptl); - hmm_vma_walk->ptelocked =3D false; - - if (folio !=3D fault_folio) - folio_lock(folio); + int ret; =20 ret =3D split_folio(folio); if (ret) { @@ -548,14 +536,145 @@ static int migrate_vma_split_folio(struct folio *fol= io, return 0; } =20 +/** + * migrate_vma_split_folio() - drop the pte lock and split a THP folio + * @folio: the folio to split + * @fault_page: struct page associated with the fault if any + * @hmm_vma_walk: walk in progress + * @ptep: pte_t * for unmap and unlock ptl + * + * Returns 0 on success + */ +static int migrate_vma_split_folio(struct folio *folio, + struct page *fault_page, + struct hmm_vma_walk *hmm_vma_walk, + pte_t *ptep) +{ + struct folio *fault_folio =3D fault_page ? page_folio(fault_page) : NULL; + + if (folio !=3D fault_folio) + folio_get(folio); + + pte_unmap_unlock(ptep, hmm_vma_walk->ptl); + hmm_vma_walk->ptelocked =3D false; + + if (folio !=3D fault_folio) + folio_lock(folio); + + return __migrate_vma_split_folio(folio, fault_page); +} + static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk, pmd_t *pmdp, unsigned long start, unsigned long end, unsigned long *hmm_pfn) { - // TODO: implement migration entry insertion - return 0; + struct hmm_vma_walk *hmm_vma_walk =3D walk->private; + struct hmm_range *range =3D hmm_vma_walk->range; + struct migrate_vma *migrate =3D range->migrate; + struct folio *fault_folio =3D NULL; + enum migrate_vma_info minfo; + struct folio *folio; + unsigned long i; + int r =3D 0; + + // Do we want to migrate at all? + minfo =3D hmm_select_migrate(range); + if (!minfo) + return r; + + WARN_ON_ONCE(!migrate); + HMM_ASSERT_PMD_LOCKED(hmm_vma_walk, true); + + fault_folio =3D migrate->fault_page ? + page_folio(migrate->fault_page) : NULL; + + if (pmd_none(*pmdp)) + return hmm_pfns_fill(start, end, hmm_vma_walk, 0); + + if (!(hmm_pfn[0] & HMM_PFN_VALID)) + goto out; + + if (pmd_trans_huge(*pmdp)) { + if (!(minfo & MIGRATE_VMA_SELECT_SYSTEM)) + goto out; + + folio =3D pmd_folio(*pmdp); + if (is_huge_zero_folio(folio)) + return hmm_pfns_fill(start, end, hmm_vma_walk, 0); + + } else if (!pmd_present(*pmdp)) { + const softleaf_t entry =3D softleaf_from_pmd(*pmdp); + + if (!softleaf_is_device_private(entry)) + goto out; + + if (!(minfo & MIGRATE_VMA_SELECT_DEVICE_PRIVATE)) + goto out; + + folio =3D softleaf_to_folio(entry); + if (folio->pgmap->owner !=3D migrate->pgmap_owner) + goto out; + } else { + hmm_vma_walk->last =3D start; + return -EBUSY; + } + + folio_get(folio); + + if (folio !=3D fault_folio && unlikely(!folio_trylock(folio))) { + folio_put(folio); + hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR); + return 0; + } + + if (thp_migration_supported() && + (migrate->flags & MIGRATE_VMA_SELECT_COMPOUND) && + (IS_ALIGNED(start, HPAGE_PMD_SIZE) && + IS_ALIGNED(end, HPAGE_PMD_SIZE))) { + struct page_vma_mapped_walk pvmw =3D { + .ptl =3D hmm_vma_walk->ptl, + .address =3D start, + .pmd =3D pmdp, + .vma =3D walk->vma, + }; + + hmm_pfn[0] |=3D HMM_PFN_MIGRATE | HMM_PFN_COMPOUND; + + r =3D set_pmd_migration_entry(&pvmw, folio_page(folio, 0)); + if (r) { + hmm_pfn[0] &=3D ~(HMM_PFN_MIGRATE | HMM_PFN_COMPOUND); + goto split; /* fall back to splitting the pmd */ + } + for (i =3D 1, start +=3D PAGE_SIZE; start < end; start +=3D PAGE_SIZE, i= ++) + hmm_pfn[i] &=3D HMM_PFN_INOUT_FLAGS; + + } else { + goto split; /* fall back to splitting the pmd */ + } + +out: + return r; + +split: + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + + /* + * folio_get() above took an extra reference. For the fault folio the + * caller still holds a reference and the lock, which is the + * precondition of __migrate_vma_split_folio(), so drop the extra one. + */ + if (folio =3D=3D fault_folio) + folio_put(folio); + + r =3D __migrate_vma_split_folio(folio, migrate->fault_page); + if (r) + return r; + + hmm_vma_walk->last =3D start; + return -EBUSY; } =20 /* @@ -958,8 +1077,18 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, hmm_vma_walk->pmdlocked =3D false; } =20 + /* + * hmm_vma_handle_migrate_prepare_pmd() splits the huge pmd in + * place when needed and returns -EBUSY to re-walk the range as + * PTEs; any other error means the split failed. + */ + if (r =3D=3D -EBUSY) + return -EBUSY; + if (r) { + /* Split not successful, skip */ + return hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR); + } return r; - } =20 if (hmm_vma_walk->pmdlocked) { --=20 2.55.0 From nobody Thu Sep 24 12:53:00 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 73FCA41E6A7 for ; Thu, 24 Sep 2026 06:54:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232856; cv=none; b=D5ml/mM4t6s6NUk+yFZGeeV6eYe3RYhLcEKEb0blVYfWeyD7udyemP1mgA1XiLHFNdcn4wJ2ZkSqltvefeMR9varMspv11ltjeYVRl4KmEjHUmmb0d8PYifR0biDKMzR/7hoMlmr7sZd2SUGQ3B9QdB0ajveoraOXBy1iBYv6Fw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232856; c=relaxed/simple; bh=8VWcZuhGT+A2DQa1VOQzXzrGT7n50TAj5hHqQcOXivY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Q+Xh4Zzq2uNniJpZSCK8a9xQF989XaiAFzFGLw575dcz/DUvt952PgZPFhN9mnI65hHCA6KDsRWD5bjiKaRzOS5G4A6AcLQAGe5iq514KZq7dQoyZvnssybmSUZuGX3O3Rp4eIdsWtDDnFFMmUpfahG15UYibQeSxYzBlaAp+Yo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Sl23vWiK; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=U/rtdRdg; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Sl23vWiK"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="U/rtdRdg" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790232842; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vz+W5yHAoWI5ZxzvIYZX65gg07ZvreXTcI/Y7c3cDnc=; b=Sl23vWiK1SlrSeVkszoxOCTdUzwfOSoLlgVwHCcRdZtGbic/c8a9RcLBoeLAqQIHuHOqjA +Q84cYwobS+/ZYsFIYK2Si7sgsRidcEpL9p0JtC2HE8Zatx/5ldJndVoG3fR6chhPm6bbs vTjlTUJCUJFS8RkXTfI+hdXZAuGEvf0= Received: from mail-lj1-f200.google.com (mail-lj1-f200.google.com [209.85.208.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-157-a3DtspxcNO6jZx5Dbmb7ig-1; Thu, 24 Sep 2026 02:54:00 -0400 X-MC-Unique: a3DtspxcNO6jZx5Dbmb7ig-1 X-Mimecast-MFC-AGG-ID: a3DtspxcNO6jZx5Dbmb7ig_1790232839 Received: by mail-lj1-f200.google.com with SMTP id 38308e7fff4ca-3a27d89a894so7817551fa.3 for ; Wed, 23 Sep 2026 23:54:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790232839; x=1790837639; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=vz+W5yHAoWI5ZxzvIYZX65gg07ZvreXTcI/Y7c3cDnc=; b=U/rtdRdgeed7RvMjBYz/no3/LItu28U7JCS4g0WzjN5Gnz5kRDSDau0vKfJadfzQL1 8fNF8bLwqBK3rySP/5blVHGMEMG0xMev2KGP1ahFAUGzPE5p6Cah1Z9zHOQbZ70tyiNT Sj4w4YOhuPGl+e0lv6UqIRu+rGkxcRuU4/VzWXQQN9SOroEGEOfN052+buysbBec87gh XkX38RyX/AFGbXiQ9IsPFXC4EN8r2R37o7w47E6W/1xgXWNcJWNPMUj4cbVp6pFr3+5c bRrRnc1GGae2ZyDJtWdv/WVBu7LRre3Rd//H/L9gwyLxPGZs0lgMPGW0TuCPs/dyGstp GVmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790232839; x=1790837639; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=vz+W5yHAoWI5ZxzvIYZX65gg07ZvreXTcI/Y7c3cDnc=; b=rbV7QY12tYVA7x06FxhrZbHPQ3LtiXotgm0eaXULjQpwhRKB2yM7gjvrcOwc8Ul0Pk gd09/wo00/x/xK9VNYwiL+PFiEocN/NxCa2A6RKcvSOrz/fi6OLokq/4cyvpz5V9PZNr kRj9jS7Gx6SXWUACQEfbD6ORBMrqB327V68n4S/6fQlosRigMztTiMrEuu//OiyRSEja kB/y/I/OtyQxzxl2nNjsKy0ew+ksFtrfKRDgYWRDeCA8f+BVyX9KhoTdRF2YJ1E87Nom GhRNgNYAudfZ5MAEOs8m1Il0FvknP5uI60fdFfWkwf32EnSptqh958mmg9U+oP0VgCvU PmMQ== X-Forwarded-Encrypted: i=1; AKwUvByRRnAmYnq5AZakrUrtYzYQWKICs1HbOxJRPU7cxzLF0XCHJrx0aaDNLjC2XB/FF0/8/KZRTAtHCtgadVc=@vger.kernel.org X-Gm-Message-State: AFuF++lajKtmmLFLICkhzsP/kLcUL+u2IwiPNd/heiBaijBP0Sc6+UV7 eUBb8dMsNKBzR5daYXGXopwmZlglgY05mbZlTkdLF8JRs4wgzJ1YidGDNkscMMnaTwpB/iiY3Rc zyE1lD+ZY7/sxNrSyOlFpzYQdif+F4KaOZkkhv2wkYe4V2zZsnuMoFo1/N4RL418G X-Gm-Gg: AYBFou3zVksHqPJ9sZTUHmqEIaTjJhYEP+Wl1fPFPntvvYn5Oi/pQ3kii1wlgu/OGFc 4imKdtJyVYlD//ytHaL2yF16Lho7ZM/VnBgZFEp1/GeaTVCScLCaHCGy9MLqpJbuWTfS99eE3EZ WZZMQLhReJU266Li6TZpuYIs9djHk/8XvgUZePasjQA+Ktdk48tb6ZoWKI3mQ3cAMHOAqtkIYRQ tic0cLKqZD9Awzxi6tmrPAEmu6xU+MfJ69fSko2jgHi3WcqP12dpUEgPMSYDE2+7Gw49E/XYP8W eX1vZ5P5ffKmWNH4gHCxzI2Mg/EGGkICKm1H8M+nu4+uTyi3w8hCDkb6cP0rCPNVnYvjn82Nd3c 6zlOPYLQf4O2lHtIx5bPw X-Received: by 2002:a05:651c:a209:20b0:3a3:7680:67fc with SMTP id 38308e7fff4ca-3a63c2ffcbemr2308961fa.27.1790232839207; Wed, 23 Sep 2026 23:53:59 -0700 (PDT) X-Received: by 2002:a05:651c:a209:20b0:3a3:7680:67fc with SMTP id 38308e7fff4ca-3a63c2ffcbemr2308741fa.27.1790232838723; Wed, 23 Sep 2026 23:53:58 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-3a63bf57909sm4803141fa.29.2026.09.23.23.53.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Sep 2026 23:53:57 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v15 07/11] mm/hmm: add lazy MMU mode support for migration in HMM pagewalk Date: Thu, 24 Sep 2026 09:53:09 +0300 Message-ID: <20260924065313.899730-8-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260924065313.899730-1-mpenttil@redhat.com> References: <20260924065313.899730-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 Add calls to lazy_mmu_mode_enable() and lazy_mmu_mode_disable() while doing migration. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- mm/hmm.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/hmm.c b/mm/hmm.c index ea77429c5178..9fdd945cc026 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -373,6 +373,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, uns= igned long addr, /* Report error for everything else */ =20 if (hmm_vma_walk->ptelocked) { + lazy_mmu_mode_disable(); pte_unmap_unlock(ptep, hmm_vma_walk->ptl); hmm_vma_walk->ptelocked =3D false; } else { @@ -396,6 +397,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, uns= igned long addr, !is_zero_pfn(pte_pfn(pte))) { if (hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0)) { if (hmm_vma_walk->ptelocked) { + lazy_mmu_mode_disable(); pte_unmap_unlock(ptep, hmm_vma_walk->ptl); hmm_vma_walk->ptelocked =3D false; } else { @@ -415,6 +417,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, uns= igned long addr, =20 fault: if (hmm_vma_walk->ptelocked) { + lazy_mmu_mode_disable(); pte_unmap_unlock(ptep, hmm_vma_walk->ptl); hmm_vma_walk->ptelocked =3D false; } else { @@ -555,6 +558,7 @@ static int migrate_vma_split_folio(struct folio *folio, if (folio !=3D fault_folio) folio_get(folio); =20 + lazy_mmu_mode_disable(); pte_unmap_unlock(ptep, hmm_vma_walk->ptl); hmm_vma_walk->ptelocked =3D false; =20 @@ -1110,8 +1114,10 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, =20 if (minfo) { ptep =3D pte_offset_map_lock(mm, pmdp, addr, &hmm_vma_walk->ptl); - if (ptep) + if (ptep) { + lazy_mmu_mode_enable(); hmm_vma_walk->ptelocked =3D true; + } } else { ptep =3D pte_offset_map(pmdp, addr); } @@ -1155,6 +1161,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, flush_tlb_range(walk->vma, start, addr); =20 if (hmm_vma_walk->ptelocked) { + lazy_mmu_mode_disable(); pte_unmap_unlock(ptep - 1, hmm_vma_walk->ptl); hmm_vma_walk->ptelocked =3D false; } else { --=20 2.55.0 From nobody Thu Sep 24 12:53:00 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4300C418360 for ; Thu, 24 Sep 2026 06:54:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232853; cv=none; b=Mh45QgC6xR6Go8oHFwWYuN5CYF72y6EatIYF4A4kdM7NMoELAZKwhoBOkoea8C37/KNONyjRIXbx05kN98UaUQXg0F7jAgC/c49nzuC8Jkii/3Et9pW0IWaFf/hV9KGZDynHfAYM+xSbnIiBMY507+iLZMLnd/cuCTCWOg2Shzg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232853; c=relaxed/simple; bh=pJJ4XpXMtlG54hweUc7WruCWO5RNNm8NBpCqI7fWZZQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=rnKFbqnZFJEIKOT/tuo2tdTh3P5TxuaNI8VGoQjfC+plsgl5IWBZiO6Ur+tgTEEvX4SwCvd9l8pAMswP7lQn1fwtc7Ygc7E6RU3ZaOpwSze1pefXurwVzu+vlkFDur39PJTz+SZjv/0QpLd+9om+RGBgMxsFDLuVhxrvyHoP0zc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=LdSpMSxR; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=Ao+Mj4Oy; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="LdSpMSxR"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="Ao+Mj4Oy" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790232845; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FpTd5JgQBNZxOTQ5E1xh4waIRSPVZay1TNJaxL2j0i8=; b=LdSpMSxRHC6sr2XcIZmndgi3ly8lnG0gd/TJ9QqeGArbFJ75qrZAo17xNDSFbfGPV758At D2rlaFhlqn9ifVsknn8qT8nsSXlYtPjUwfQrWcyhggXcGJNbcBAVT9ql6TAwgNLBLWbw+k 7Va84Jb1nlh4+oj/hRv4cGLyk7C2eQA= Received: from mail-lj1-f199.google.com (mail-lj1-f199.google.com [209.85.208.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-677-hvfJOuA9Ovey8X_fUb-Jqg-1; Thu, 24 Sep 2026 02:54:03 -0400 X-MC-Unique: hvfJOuA9Ovey8X_fUb-Jqg-1 X-Mimecast-MFC-AGG-ID: hvfJOuA9Ovey8X_fUb-Jqg_1790232842 Received: by mail-lj1-f199.google.com with SMTP id 38308e7fff4ca-3a3540f42beso8614771fa.2 for ; Wed, 23 Sep 2026 23:54:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790232842; x=1790837642; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=FpTd5JgQBNZxOTQ5E1xh4waIRSPVZay1TNJaxL2j0i8=; b=Ao+Mj4OybVOSZbBPtLONNzytcEmm+SuB4CoyD/ERvvqeByFtlvUZL+12rPsMdcRCMV GYejJvBKIDCU4+myOB8zIsksuMjLvbbzf3Pt0QSJt3dfgdBDTc51HU3f3U3486JuPx6+ Mlrv5sOA0TZlYki+dQ+IG5Z/wdWOWRdEgQa80hUQQi6X5C2UCFyF+t0q1qiED6xdn8A5 wMHVPXfAk8CQo0C3w89kNLXvTgz6FjAr6duhxb7kDkL7SxKXoVm7Ykn2FzEvmZ1IiBW2 YQ9vplO7m+Ypq6WqL6+1VjYxYwVDso2F7bBp7x7UdJO6sLP/B+EIVpYMVkWAMVgB30tC qP4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790232842; x=1790837642; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=FpTd5JgQBNZxOTQ5E1xh4waIRSPVZay1TNJaxL2j0i8=; b=IDsRVqt1H3wh71L1hDiqcy413CjttTDzTP+a6wrkqWx6FVe8u/oTUDlwF0EAJQ9XV+ 6QWrN6cHn7U/AyYmjm6DGLSz+MHxQoeY6j12s28MNNF06zXogVVYXw0TD8xTlUV5tMlL qHSWYMbuAK9NBnbNb5xl3L3Jpn08DsFkKsBce4XeNwUs6lX+VU8xFlM21YzEV0Wfn9/5 jEIBMXhovMxhdHA3V3detKav734alaCq81X0rH/VARGvJCxhE81V4g2fcjGkh4xg1XD7 ruFfOeJ0X0WW8Rholo9v2Wkor10rOgjNXEtJHKWI/YFCIbseotULD5mtrF4aeUXQLOH6 MHQw== X-Forwarded-Encrypted: i=1; AKwUvBzxKu1ChOgCGduQcCM1g0UxyN6nmesli6zbciYZIM6+mytUXur8oIjb2ZDR+iOEW6lq+YuQwo5B0VJh8IY=@vger.kernel.org X-Gm-Message-State: AFuF++num60cfW6o6D/bmTERBfZWrK5QUua/ILXCGTBW6ZIWR4hw6nyx UiPjvHqF6OpDYGkNs7SnNN52BPCB2xmG53YE7bqUKJ0tH7ZE6BkFWWTUcpJZTKu5/O9lscSJ0Ba cR/TXXDMFXIwDR/sC6750j9lqG7nuPxA8HgZiUUKehgTV9AP7IZzypyW9OHrQYRhT X-Gm-Gg: AYBFou2ZnmNL4gJ7ISjeL5eKEuXWjRGFst2Z1Yu5/KuwW9hXJTLflNQjQrFPMdKl1ZN 4lbcqlNiG7SULW0jRNLmISfQz/7kZCso1JT1jWHrSLORrkjQYBKYEDL2jbhYWLRPszsmUsAdygD R8DAHGOui1zVoOCExxcNZU5cJ66sb080u+4qsYj7TNQva6l4n9nVFuvcf5Jci4DIkraR7cCbluA qU1NHrcLhvL0C8EhDMGjAbz5rKzoRUV8vIPSzGe7pHfAnGvLXGIW+dgE08l+y5dsvzq2qiMtjQB giEFJ9LZ+DK/wNwVQcCOf2ZNeBWS10jGx2MDnUtMlHicKKBmUJswXvznZDiOWlTIr39s2Z2Ac4N q23NEcYgmFVs4fMmkciNS X-Received: by 2002:a05:651c:211b:b0:3a3:74b7:fc0c with SMTP id 38308e7fff4ca-3a63c4090b9mr3873541fa.24.1790232841667; Wed, 23 Sep 2026 23:54:01 -0700 (PDT) X-Received: by 2002:a05:651c:211b:b0:3a3:74b7:fc0c with SMTP id 38308e7fff4ca-3a63c4090b9mr3873461fa.24.1790232841170; Wed, 23 Sep 2026 23:54:01 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-3a63bf57909sm4803141fa.29.2026.09.23.23.54.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Sep 2026 23:54:00 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v15 08/11] mm/hmm: implement rollback for device page migration in HMM pagewalk Date: Thu, 24 Sep 2026 09:53:10 +0300 Message-ID: <20260924065313.899730-9-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260924065313.899730-1-mpenttil@redhat.com> References: <20260924065313.899730-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 During the migration pagewalk, the PTE table could be cleared and/or changed into PMD leaf or even another PTE table while dropped locks. In these cases the possibly inserted migration ptes are gone. We have to however undo the collecting done so far, so unlock the folios and drop reference taken. During the pagewalk we notice such scenarios if going to recollect a pfn but have already committed to migrate the entry with HMM_PFN_MIGRATE, in which case rollback. If we encounter migration ptes they are just skipped to allow for restart own walks. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- include/linux/hmm.h | 22 +++++++++++++ mm/hmm.c | 76 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 98 insertions(+) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 4f56f3419cb4..b08ebc1343dd 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -111,6 +111,28 @@ static inline unsigned int hmm_pfn_to_map_order(unsign= ed long hmm_pfn) return (hmm_pfn >> HMM_PFN_ORDER_SHIFT) & 0x1F; } =20 +/* + * hmm_pfn_collected() - is this pfn entry prepared for migration ? + * If collected the folio's refcount is increased and the folio + * is locked. + */ +static inline bool hmm_pfn_collected(unsigned long hmm_pfn) +{ + return (hmm_pfn & (HMM_PFN_VALID | HMM_PFN_MIGRATE)) =3D=3D + (HMM_PFN_VALID | HMM_PFN_MIGRATE); +} + +/* + * hmm_pfn_rollback_collected() - undoes the collection of hmm_pfn + * + * Note for total rollback the folio's refcount has to be put + * and folio has to be unlocked. + */ +static inline unsigned long hmm_pfn_rollback_collected(unsigned long hmm_p= fn) +{ + return hmm_pfn & ~(HMM_PFN_VALID | HMM_PFN_MIGRATE | HMM_PFN_COMPOUND); +} + /* * struct hmm_range - track invalidation lock on virtual address range * diff --git a/mm/hmm.c b/mm/hmm.c index 9fdd945cc026..daf83f809151 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -95,6 +95,11 @@ enum { HMM_PFN_P2PDMA_BUS, }; =20 +static void hmm_vma_handle_migrate_prepare_rollback(const struct hmm_vma_w= alk *hmm_vma_walk, + unsigned long start, + unsigned long end, + unsigned long *hmm_pfn); + static int hmm_pfns_fill(unsigned long addr, unsigned long end, struct hmm_vma_walk *hmm_vma_walk, unsigned long cpu_flags) { @@ -111,6 +116,8 @@ static int hmm_pfns_fill(unsigned long addr, unsigned l= ong end, } } =20 + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr, end, &range->= hmm_pfns[i]); + if (migrate && thp_migration_supported() && (minfo & MIGRATE_VMA_SELECT_COMPOUND) && IS_ALIGNED(addr, HPAGE_PMD_SIZE) && @@ -282,6 +289,8 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, uns= igned long addr, return hmm_record_fault(addr, end, required_fault, walk); } =20 + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr, + end, hmm_pfns); pfn =3D pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); for (i =3D 0; addr < end; addr +=3D PAGE_SIZE, i++, pfn++) { hmm_pfns[i] &=3D HMM_PFN_INOUT_FLAGS; @@ -412,6 +421,9 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, uns= igned long addr, =20 new_pfn_flags =3D pte_pfn(pte) | cpu_flags; out: + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr, + addr + PAGE_SIZE, + hmm_pfn); *hmm_pfn =3D (*hmm_pfn & HMM_PFN_INOUT_FLAGS) | new_pfn_flags; return 0; =20 @@ -450,6 +462,9 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *wa= lk, unsigned long start, if (softleaf_is_device_private_write(entry)) cpu_flags |=3D HMM_PFN_WRITE; =20 + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, + start, end, + hmm_pfns); /* * Fully populate the PFN list though subsequent PFNs could be * inferred, because drivers which are not yet aware of large @@ -568,6 +583,48 @@ static int migrate_vma_split_folio(struct folio *folio, return __migrate_vma_split_folio(folio, fault_page); } =20 +/* + * Due to dropping ptl locks for splitting for instance, would we + * overwrite already collected pfns? This could happen when pmd + * pointing to a page table has vanished and been replaced + * with a leaf pmd, or another page table. + * In that case unref and unlock the folios, + * the pfns of which were collected from the disappeared + * page tables. + */ +static void hmm_vma_handle_migrate_prepare_rollback(const struct hmm_vma_w= alk *hmm_vma_walk, + unsigned long start, + unsigned long end, + unsigned long *hmm_pfn) +{ + struct hmm_range *range =3D hmm_vma_walk->range; + struct migrate_vma *migrate =3D range->migrate; + struct folio *fault_folio =3D NULL; + enum migrate_vma_info minfo; + struct folio *folio; + unsigned long i; + + minfo =3D hmm_select_migrate(range); + if (!minfo) + return; + + WARN_ON_ONCE(!migrate); + + fault_folio =3D migrate->fault_page ? + page_folio(migrate->fault_page) : NULL; + + for (i =3D 0; start < end; start +=3D PAGE_SIZE, i++) { + if (hmm_pfn_collected(hmm_pfn[i])) { + folio =3D page_folio(hmm_pfn_to_page(hmm_pfn[i])); + if (folio !=3D fault_folio) + folio_unlock(folio); + folio_put(folio); + hmm_pfn[i] =3D hmm_pfn_rollback_collected(hmm_pfn[i]); + + } + } +} + static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk, pmd_t *pmdp, unsigned long start, @@ -722,6 +779,11 @@ static int hmm_vma_handle_migrate_prepare(const struct= mm_walk *walk, pte =3D ptep_get(ptep); =20 if (pte_none(pte)) { + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, + addr, + addr + PAGE_SIZE, + hmm_pfn); + if (vma_is_anonymous(walk->vma)) { *hmm_pfn &=3D HMM_PFN_INOUT_FLAGS; *hmm_pfn |=3D HMM_PFN_MIGRATE; @@ -769,6 +831,10 @@ static int hmm_vma_handle_migrate_prepare(const struct= mm_walk *walk, pfn =3D pte_pfn(pte); if (is_zero_pfn(pfn) && (minfo & MIGRATE_VMA_SELECT_SYSTEM)) { + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, + addr, + addr + PAGE_SIZE, + hmm_pfn); *hmm_pfn =3D HMM_PFN_MIGRATE; goto out; } @@ -897,6 +963,13 @@ static int hmm_vma_handle_migrate_prepare(const struct= mm_walk *walk, } =20 #else +static void hmm_vma_handle_migrate_prepare_rollback(const struct hmm_vma_w= alk *hmm_vma_walk, + unsigned long start, + unsigned long end, + unsigned long *hmm_pfn) +{ +} + static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk, pmd_t *pmdp, unsigned long start, @@ -1117,6 +1190,9 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, if (ptep) { lazy_mmu_mode_enable(); hmm_vma_walk->ptelocked =3D true; + } else { + /* The pte table is gone */ + hmm_vma_handle_migrate_prepare_rollback(walk->private, addr, end, hmm_p= fns); } } else { ptep =3D pte_offset_map(pmdp, addr); --=20 2.55.0 From nobody Thu Sep 24 12:53:00 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 81734423EBA for ; Thu, 24 Sep 2026 06:54:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232888; cv=none; b=r6Q2vEwqj//ql2hL7G4VDySLNi0vVCB4WqBktD6JTnc62hsXNDMduS8DRQo6LuSQzJI/bYzp8iMYE5LUiEv94RUcjiO/i9tTXLKnnOndjpj+M6vMjLs1QN4UJIuGxyfAGdPpUh8+YMUJozq/Xu6MATsErUDZCy2kIwVrt55VV+0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232888; c=relaxed/simple; bh=IS7n5gWoXpslMWI/90FexsmzfWyzNAALfZJLjo3VqA8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Zc1xQ/KGUbDYSoZK5of9G1lHSSV1NRx8XL1NpBvvc1OZnCrmL5oP1rGYLyLOGBNGkxZePD1mtQesfuBemmXqZ194qNN4gKQOd2NReIZIxSIxXOUq21pVjixjm6HYU9mM8syNdRIMlMnWG/WjzNXhSsIfLhxHxv9eUZBwpxhri8k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=a5nJmonK; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=L0CDq3wd; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="a5nJmonK"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="L0CDq3wd" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790232850; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Sv/cs7ZJuFY+/3q9oQyGV7uOngHjsWLpUE1ceSZJPbM=; b=a5nJmonKYR8HyPqLrN7K+8UqE8RBNJz/2k2Q4zYVszvjXLChCj3GeqoSzIHxrpxwcuK3bI lGPi0zTQtww0RMhgXw68pBI5rsxD5oKLmYN8XQCabkUzrtMrzLkDVPG15rZSqkF/eBZ1s9 m2nXql8GKwQ67UvL60Eqm1iiVjucidg= Received: from mail-lj1-f198.google.com (mail-lj1-f198.google.com [209.85.208.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-262-AKV9PnmoNxOx1BfHbSjGVQ-1; Thu, 24 Sep 2026 02:54:06 -0400 X-MC-Unique: AKV9PnmoNxOx1BfHbSjGVQ-1 X-Mimecast-MFC-AGG-ID: AKV9PnmoNxOx1BfHbSjGVQ_1790232845 Received: by mail-lj1-f198.google.com with SMTP id 38308e7fff4ca-3a5d09935b0so8260261fa.3 for ; Wed, 23 Sep 2026 23:54:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790232845; x=1790837645; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=Sv/cs7ZJuFY+/3q9oQyGV7uOngHjsWLpUE1ceSZJPbM=; b=L0CDq3wdXNeUnWP8g2HMaRVMk78zDc5VInt7XUGQdNgCXbG/ju5YghIINgFEc4co0W a1sxJh34vZLukdiocn8jWu4WxmTY+USpSJDk8jLXMWuNLeCxBgdTWMmEN+o+0yfT7fOJ 5tAOYjBQVmTM9R43Mz6o9dhG3oYpVbm2page0ihJvKyb84/Lrmp8OsK0vMD320zjI693 sDzkOHmyAeGUOEhxslb0XOy/IdCkZ0Uiy1eV4H7TTuevp+vHcbZkVxLO7DAB4x17nczn M3bp/R7eYsOyeU1Zl99ODhMpQyjY2a3K2+LRF379U+NnzuCTxeJ5JExksRpteAD0jDiw mq0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790232845; x=1790837645; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=Sv/cs7ZJuFY+/3q9oQyGV7uOngHjsWLpUE1ceSZJPbM=; b=Pdfgbkxfichl2rcm1+gXkBV4XbtQjSmyzB0Z0r45PYMWF/7Y5+OAj18PSj2R25boJB ShI5zbLIqWVyYQNRrDLPJT5OUH0s8jEV8Oah777qu9e+z9Sb7lRO2dcyoQ7Hmvfls8L/ CdzHyMDMxn/So0JQkn5Boa8F30pLHBxwMclT5TU05CAuxyLzcwTBPwlxzG2d/Kk/9S+N 6PqXt16YFCV3sYbL8VHvjmvqJc52L5VS4wRy+w+K47iBOsCcQpvJYvpOjlB3yBgBAV7w 7l/cSK2fjdfVC1cD8l+kRGc/tp74FEEdVX+5D/kZLBbJU7It+u1z8LLDkj+I7ahEwAI3 8GPg== X-Forwarded-Encrypted: i=1; AKwUvBysRz6ZKS6o0SiGhNCAHOIH7hQrvv5fH9A5DEzwSpXca55JNMZCPFOzyaqToACikDWkUf1g2OtN9/YjCNw=@vger.kernel.org X-Gm-Message-State: AFuF++mDWqazLmW2g4eMQKN5CPWElWV5ra/r3Yc1p11yFhMV8sGdsn3D qJ8+KxRZqrQEoglmBOJ0dMP4gLsdXhYf1RD06D5tIPPtcwxVwIPNHx9q4tSc6W1sflyi4H3tA2M MbONdTkjftgqc/EuZ5dOfzERaIuGjo0QBcI4M96Ij4EwdFwkbFDsziht5kUndgb+F X-Gm-Gg: AYBFou1fRwAXghp8kqTN2yqlKOT31SeNZ4yXx40lyG49VCHS/8JG9qPRA88mTsTVISq jWxwoMofFS1WiUKUH/gqHpR9Ip1pR2vqV98uV9eO+j56y95VA51JCT8H/pJAAzxuxRHZzSZFUyO OzekoUXOWrcAM49f6yw4XQ4WHLDIrD6BNEvQedKJ8wxJgfFiRbB3+heMhKci3LsKG0a+kPwRNFr 88sr7asSExxpPx0ykwOi6pIm069DD7RotZzRI2qGQOX53US3sT4eO7r2cXcMc9CDVWs0Oqm5onf JbR2+67nP42L/BD4U5gn+b0EjJG2TZI5XFa48OK3sYHbD/J3xQelOZVRBFpZ5JSq/p3WPNSPqWF HWBNcdlxJ8X2THP0tVjM6 X-Received: by 2002:a05:651c:31dc:b0:3a3:7682:49a8 with SMTP id 38308e7fff4ca-3a63c3030camr3548941fa.29.1790232844779; Wed, 23 Sep 2026 23:54:04 -0700 (PDT) X-Received: by 2002:a05:651c:31dc:b0:3a3:7682:49a8 with SMTP id 38308e7fff4ca-3a63c3030camr3548831fa.29.1790232844285; Wed, 23 Sep 2026 23:54:04 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-3a63bf57909sm4803141fa.29.2026.09.23.23.54.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Sep 2026 23:54:02 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v15 09/11] mm: enable device page migration from HMM pagewalk Date: Thu, 24 Sep 2026 09:53:11 +0300 Message-ID: <20260924065313.899730-10-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260924065313.899730-1-mpenttil@redhat.com> References: <20260924065313.899730-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 HMM pagewalk has now the machinery to do the first phase of device page migration, collecting the pfns and installing migration ptes. Enable migration in hmm_range_fault(), and change migrate_vma_setup() to use HMM pagewalk path. Two new flags, MIGRATE_VMA_FAULT and MIGRATE_VMA_WRITE are introduced for migrate_vma_setup(), to request for faulting missing pages, and requesting write access. Also, the migrate_vma_collect*() based paths are now unused, so delete them. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- include/linux/migrate.h | 10 +- mm/hmm.c | 33 ++- mm/migrate_device.c | 562 ++++------------------------------------ 3 files changed, 83 insertions(+), 522 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 01c5f62a56e9..c73e41625ac2 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -180,6 +180,8 @@ enum migrate_vma_info { MIGRATE_VMA_SELECT_DEVICE_PRIVATE =3D 1 << 1, MIGRATE_VMA_SELECT_DEVICE_COHERENT =3D 1 << 2, MIGRATE_VMA_SELECT_COMPOUND =3D 1 << 3, + MIGRATE_VMA_FAULT =3D 1 << 4, + MIGRATE_VMA_WRITE =3D 1 << 5, }; =20 struct migrate_vma { @@ -217,10 +219,14 @@ struct migrate_vma { struct page *fault_page; }; =20 -// TODO: enable migration static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *r= ange) { - return 0; + enum migrate_vma_info minfo; + + minfo =3D (range->default_flags & HMM_PFN_REQ_MIGRATE) ? + range->migrate->flags : 0; + + return minfo; } =20 #ifdef CONFIG_MMU_NOTIFIER diff --git a/mm/hmm.c b/mm/hmm.c index daf83f809151..556ee63dd6d8 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -1539,14 +1539,43 @@ static int hmm_range_fault_locked(struct hmm_range = *range, bool *locked) * the invalidation to finish. * -EFAULT: A page was requested to be valid and could not be made val= id * ie it has no backing VMA or it is illegal to access + * -ERANGE: The range crosses multiple VMAs, or space for hmm_pfns arr= ay + * is too low. * * This is similar to get_user_pages(), except that it can read the page t= ables * without mutating them (ie causing faults). * * The mmap lock must be held by the caller and will remain held on return. * New users should prefer hmm_range_fault_unlocked_timeout() unless they - * specifically need to keep the mmap lock held across the call. This help= er - * cannot support VMAs whose fault handlers need to drop the mmap lock. + * specifically need to keep the mmap lock held across the call like while + * migrating. This helper cannot support VMAs whose fault handlers need to + * drop the mmap lock. + * + * If want to do migration after faulting, call hmm_range_fault() with + * range.default_flags of HMM_PFN_REQ_MIGRATE, and optionally + * HMM_PFN_REQ_FAULT|HMM_PFN_REQ_WRITE, and initialize range->migrate fiel= d. + * range->migrate->vma will be populated during the call, + * and must be stable across the whole migrate process, which is + * why mmap_lock must be held around this call. + * + * When HMM_PFN_REQ_MIGRATE is set, migration collection may be partial on + * return and the caller takes responsibility for completing or aborting i= t. + * + * On success, the caller must call migrate_hmm_range_setup() and may then + * proceed with the normal migrate_vma sequence: prepare destination pages, + * call migrate_vma_pages(), update device mappings as needed, and finally + * call migrate_vma_finalize(). + * + * On -EBUSY, the caller may retry hmm_range_fault() using the same range = and + * PFN array without undoing entries collected by the previous attempt. If + * the caller stops retrying, it must abort the partial migration. + * + * On any other error, or when abandoning a retry, the caller must call + * migrate_hmm_range_setup(), migrate_vma_pages() with no valid destination + * entries, and migrate_vma_finalize() to abort the partial migration. + * + * The mmap read lock must remain held until the migration has either been + * completed or aborted. */ int hmm_range_fault(struct hmm_range *range) { diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 43e571e82cca..1e1a592e1267 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -18,519 +18,6 @@ #include #include "internal.h" =20 -static int migrate_vma_collect_skip(unsigned long start, - unsigned long end, - struct mm_walk *walk) -{ - struct migrate_vma *migrate =3D walk->private; - unsigned long addr; - - for (addr =3D start; addr < end; addr +=3D PAGE_SIZE) { - migrate->dst[migrate->npages] =3D 0; - migrate->src[migrate->npages++] =3D 0; - } - - return 0; -} - -static int migrate_vma_collect_hole(unsigned long start, - unsigned long end, - __always_unused int depth, - struct mm_walk *walk) -{ - struct migrate_vma *migrate =3D walk->private; - unsigned long addr; - - /* Only allow populating anonymous memory. */ - if (!vma_is_anonymous(walk->vma)) - return migrate_vma_collect_skip(start, end, walk); - - if (thp_migration_supported() && - (migrate->flags & MIGRATE_VMA_SELECT_COMPOUND) && - (IS_ALIGNED(start, HPAGE_PMD_SIZE) && - IS_ALIGNED(end, HPAGE_PMD_SIZE))) { - migrate->src[migrate->npages] =3D MIGRATE_PFN_MIGRATE | - MIGRATE_PFN_COMPOUND; - migrate->dst[migrate->npages] =3D 0; - migrate->npages++; - migrate->cpages++; - - /* - * Collect the remaining entries as holes, in case we - * need to split later - */ - return migrate_vma_collect_skip(start + PAGE_SIZE, end, walk); - } - - for (addr =3D start; addr < end; addr +=3D PAGE_SIZE) { - migrate->src[migrate->npages] =3D MIGRATE_PFN_MIGRATE; - migrate->dst[migrate->npages] =3D 0; - migrate->npages++; - migrate->cpages++; - } - - return 0; -} - -/** - * migrate_vma_split_folio() - Helper function to split a THP folio - * @folio: the folio to split - * @fault_page: struct page associated with the fault if any - * - * If @folio is not the folio containing @fault_page, the caller must hold= a - * reference on @folio. The helper consumes that reference. - * - * Returns 0 on success - */ -static int migrate_vma_split_folio(struct folio *folio, - struct page *fault_page) -{ - int ret; - struct folio *fault_folio =3D fault_page ? page_folio(fault_page) : NULL; - struct folio *new_fault_folio =3D NULL; - - if (folio !=3D fault_folio) - folio_lock(folio); - - ret =3D split_folio(folio); - if (ret) { - if (folio !=3D fault_folio) { - folio_unlock(folio); - folio_put(folio); - } - return ret; - } - - new_fault_folio =3D fault_page ? page_folio(fault_page) : NULL; - - /* - * Ensure the lock is held on the correct - * folio after the split - */ - if (!new_fault_folio) { - folio_unlock(folio); - folio_put(folio); - } else if (folio !=3D new_fault_folio) { - if (new_fault_folio !=3D fault_folio) { - folio_get(new_fault_folio); - folio_lock(new_fault_folio); - } - folio_unlock(folio); - folio_put(folio); - } - - return 0; -} - -/** migrate_vma_collect_huge_pmd - collect THP pages without splitting the - * folio for device private pages. - * @pmdp: pointer to pmd entry - * @start: start address of the range for migration - * @end: end address of the range for migration - * @walk: mm_walk callback structure - * @fault_folio: folio associated with the fault if any - * - * Collect the huge pmd entry at @pmdp for migration and set the - * MIGRATE_PFN_COMPOUND flag in the migrate src entry to indicate that - * migration will occur at HPAGE_PMD granularity - */ -static int migrate_vma_collect_huge_pmd(pmd_t *pmdp, unsigned long start, - unsigned long end, struct mm_walk *walk, - struct folio *fault_folio) -{ - struct mm_struct *mm =3D walk->mm; - struct folio *folio; - struct migrate_vma *migrate =3D walk->private; - spinlock_t *ptl; - int ret; - unsigned long write =3D 0; - - ptl =3D pmd_lock(mm, pmdp); - if (pmd_none(*pmdp)) { - spin_unlock(ptl); - return migrate_vma_collect_hole(start, end, -1, walk); - } - - if (pmd_trans_huge(*pmdp)) { - if (!(migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) { - spin_unlock(ptl); - return migrate_vma_collect_skip(start, end, walk); - } - - folio =3D pmd_folio(*pmdp); - if (is_huge_zero_folio(folio)) { - spin_unlock(ptl); - return migrate_vma_collect_hole(start, end, -1, walk); - } - if (pmd_write(*pmdp)) - write =3D MIGRATE_PFN_WRITE; - } else if (!pmd_present(*pmdp)) { - const softleaf_t entry =3D softleaf_from_pmd(*pmdp); - - if (!softleaf_is_device_private(entry) || - !(migrate->flags & MIGRATE_VMA_SELECT_DEVICE_PRIVATE)) { - spin_unlock(ptl); - return migrate_vma_collect_skip(start, end, walk); - } - - folio =3D softleaf_to_folio(entry); - if (folio->pgmap->owner !=3D migrate->pgmap_owner) { - spin_unlock(ptl); - return migrate_vma_collect_skip(start, end, walk); - } - - if (softleaf_is_device_private_write(entry)) - write =3D MIGRATE_PFN_WRITE; - } else { - spin_unlock(ptl); - return -EAGAIN; - } - - folio_get(folio); - if (folio !=3D fault_folio && unlikely(!folio_trylock(folio))) { - spin_unlock(ptl); - folio_put(folio); - return migrate_vma_collect_skip(start, end, walk); - } - - if (thp_migration_supported() && - (migrate->flags & MIGRATE_VMA_SELECT_COMPOUND) && - (IS_ALIGNED(start, HPAGE_PMD_SIZE) && - IS_ALIGNED(end, HPAGE_PMD_SIZE))) { - - struct page_vma_mapped_walk pvmw =3D { - .ptl =3D ptl, - .address =3D start, - .pmd =3D pmdp, - .vma =3D walk->vma, - }; - - unsigned long pfn =3D page_to_pfn(folio_page(folio, 0)); - - migrate->src[migrate->npages] =3D migrate_pfn(pfn) | write - | MIGRATE_PFN_MIGRATE - | MIGRATE_PFN_COMPOUND; - migrate->dst[migrate->npages++] =3D 0; - migrate->cpages++; - ret =3D set_pmd_migration_entry(&pvmw, folio_page(folio, 0)); - if (ret) { - migrate->npages--; - migrate->cpages--; - migrate->src[migrate->npages] =3D 0; - migrate->dst[migrate->npages] =3D 0; - goto fallback; - } - migrate_vma_collect_skip(start + PAGE_SIZE, end, walk); - spin_unlock(ptl); - return 0; - } - -fallback: - spin_unlock(ptl); - if (!folio_test_large(folio)) - goto done; - ret =3D split_folio(folio); - if (fault_folio !=3D folio) - folio_unlock(folio); - folio_put(folio); - if (ret) - return migrate_vma_collect_skip(start, end, walk); - if (pmd_none(pmdp_get_lockless(pmdp))) - return migrate_vma_collect_hole(start, end, -1, walk); - -done: - return -ENOENT; -} - -static int migrate_vma_collect_pmd(pmd_t *pmdp, - unsigned long start, - unsigned long end, - struct mm_walk *walk) -{ - struct migrate_vma *migrate =3D walk->private; - struct vm_area_struct *vma =3D walk->vma; - struct mm_struct *mm =3D vma->vm_mm; - unsigned long addr =3D start, unmapped =3D 0; - spinlock_t *ptl; - struct folio *fault_folio =3D migrate->fault_page ? - page_folio(migrate->fault_page) : NULL; - pte_t *ptep; - -again: - if (pmd_trans_huge(*pmdp) || !pmd_present(*pmdp)) { - int ret =3D migrate_vma_collect_huge_pmd(pmdp, start, end, walk, fault_f= olio); - - if (ret =3D=3D -EAGAIN) - goto again; - if (ret =3D=3D 0) - return 0; - } - - ptep =3D pte_offset_map_lock(mm, pmdp, start, &ptl); - if (!ptep) - goto again; - lazy_mmu_mode_enable(); - ptep +=3D (addr - start) / PAGE_SIZE; - - for (; addr < end; addr +=3D PAGE_SIZE, ptep++) { - struct dev_pagemap *pgmap; - unsigned long mpfn =3D 0, pfn; - struct folio *folio; - struct page *page; - softleaf_t entry; - pte_t pte; - - pte =3D ptep_get(ptep); - - if (pte_none(pte)) { - if (vma_is_anonymous(vma)) { - mpfn =3D MIGRATE_PFN_MIGRATE; - migrate->cpages++; - } - goto next; - } - - if (!pte_present(pte)) { - /* - * Only care about unaddressable device page special - * page table entry. Other special swap entries are not - * migratable, and we ignore regular swapped page. - */ - entry =3D softleaf_from_pte(pte); - if (!softleaf_is_device_private(entry)) - goto next; - - page =3D softleaf_to_page(entry); - pgmap =3D page_pgmap(page); - if (!(migrate->flags & - MIGRATE_VMA_SELECT_DEVICE_PRIVATE) || - pgmap->owner !=3D migrate->pgmap_owner) - goto next; - - folio =3D page_folio(page); - if (folio_test_large(folio)) { - int ret; - - /* migrate_vma_split_folio() consumes this reference */ - if (folio !=3D fault_folio) - folio_get(folio); - lazy_mmu_mode_disable(); - pte_unmap_unlock(ptep, ptl); - ret =3D migrate_vma_split_folio(folio, - migrate->fault_page); - - if (ret) { - if (unmapped) - flush_tlb_range(walk->vma, start, end); - - return migrate_vma_collect_skip(addr, end, walk); - } - - goto again; - } - - mpfn =3D migrate_pfn(page_to_pfn(page)) | - MIGRATE_PFN_MIGRATE; - if (softleaf_is_device_private_write(entry)) - mpfn |=3D MIGRATE_PFN_WRITE; - } else { - pfn =3D pte_pfn(pte); - if (is_zero_pfn(pfn) && - (migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) { - mpfn =3D MIGRATE_PFN_MIGRATE; - migrate->cpages++; - goto next; - } - page =3D vm_normal_page(migrate->vma, addr, pte); - if (page && !is_zone_device_page(page) && - !(migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) { - goto next; - } else if (page && is_device_coherent_page(page)) { - pgmap =3D page_pgmap(page); - - if (!(migrate->flags & - MIGRATE_VMA_SELECT_DEVICE_COHERENT) || - pgmap->owner !=3D migrate->pgmap_owner) - goto next; - } - folio =3D page ? page_folio(page) : NULL; - if (folio && folio_test_large(folio)) { - int ret; - - /* migrate_vma_split_folio() consumes this reference */ - if (folio !=3D fault_folio) - folio_get(folio); - lazy_mmu_mode_disable(); - pte_unmap_unlock(ptep, ptl); - ret =3D migrate_vma_split_folio(folio, - migrate->fault_page); - - if (ret) { - if (unmapped) - flush_tlb_range(walk->vma, start, end); - - return migrate_vma_collect_skip(addr, end, walk); - } - - goto again; - } - mpfn =3D migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE; - mpfn |=3D pte_write(pte) ? MIGRATE_PFN_WRITE : 0; - } - - if (!page || !page->mapping) { - mpfn =3D 0; - goto next; - } - - /* - * By getting a reference on the folio we pin it and that blocks - * any kind of migration. Side effect is that it "freezes" the - * pte. - * - * We drop this reference after isolating the folio from the lru - * for non device folio (device folio are not on the lru and thus - * can't be dropped from it). - */ - folio =3D page_folio(page); - folio_get(folio); - - /* - * We rely on folio_trylock() to avoid deadlock between - * concurrent migrations where each is waiting on the others - * folio lock. If we can't immediately lock the folio we fail this - * migration as it is only best effort anyway. - * - * If we can lock the folio it's safe to set up a migration entry - * now. In the common case where the folio is mapped once in a - * single process setting up the migration entry now is an - * optimisation to avoid walking the rmap later with - * try_to_migrate(). - */ - if (fault_folio =3D=3D folio || folio_trylock(folio)) { - bool anon_exclusive; - pte_t swp_pte; - - if (pte_present(pte)) - flush_cache_page(vma, addr, pte_pfn(pte)); - anon_exclusive =3D folio_test_anon(folio) && - PageAnonExclusive(page); - if (anon_exclusive) { - pte =3D ptep_clear_flush(vma, addr, ptep); - - if (folio_try_share_anon_rmap_pte(folio, page)) { - set_pte_at(mm, addr, ptep, pte); - if (fault_folio !=3D folio) - folio_unlock(folio); - folio_put(folio); - mpfn =3D 0; - goto next; - } - } else { - pte =3D ptep_get_and_clear(mm, addr, ptep); - } - - migrate->cpages++; - - /* Set the dirty flag on the folio now the pte is gone. */ - if (pte_present(pte) && pte_dirty(pte)) - folio_mark_dirty(folio); - - /* Setup special migration page table entry */ - if (mpfn & MIGRATE_PFN_WRITE) - entry =3D make_writable_migration_entry( - page_to_pfn(page)); - else if (anon_exclusive) - entry =3D make_readable_exclusive_migration_entry( - page_to_pfn(page)); - else - entry =3D make_readable_migration_entry( - page_to_pfn(page)); - if (pte_present(pte)) { - if (pte_young(pte)) - entry =3D make_migration_entry_young(entry); - if (pte_dirty(pte)) - entry =3D make_migration_entry_dirty(entry); - } - swp_pte =3D swp_entry_to_pte(entry); - if (pte_present(pte)) { - if (pte_soft_dirty(pte)) - swp_pte =3D pte_swp_mksoft_dirty(swp_pte); - if (pte_uffd(pte)) - swp_pte =3D pte_swp_mkuffd(swp_pte); - } else { - if (pte_swp_soft_dirty(pte)) - swp_pte =3D pte_swp_mksoft_dirty(swp_pte); - if (pte_swp_uffd(pte)) - swp_pte =3D pte_swp_mkuffd(swp_pte); - } - set_pte_at(mm, addr, ptep, swp_pte); - - /* - * This is like regular unmap: we remove the rmap and - * drop the folio refcount. The folio won't be freed, as - * we took a reference just above. - */ - folio_remove_rmap_pte(folio, page, vma); - folio_put(folio); - - if (pte_present(pte)) - unmapped++; - } else { - folio_put(folio); - mpfn =3D 0; - } - -next: - migrate->dst[migrate->npages] =3D 0; - migrate->src[migrate->npages++] =3D mpfn; - } - - /* Only flush the TLB if we actually modified any entries */ - if (unmapped) - flush_tlb_range(walk->vma, start, end); - - lazy_mmu_mode_disable(); - pte_unmap_unlock(ptep - 1, ptl); - - return 0; -} - -static const struct mm_walk_ops migrate_vma_walk_ops =3D { - .pmd_entry =3D migrate_vma_collect_pmd, - .pte_hole =3D migrate_vma_collect_hole, - .walk_lock =3D PGWALK_RDLOCK, -}; - -/* - * migrate_vma_collect() - collect pages over a range of virtual addresses - * @migrate: migrate struct containing all migration information - * - * This will walk the CPU page table. For each virtual address backed by a - * valid page, it updates the src array and takes a reference on the page,= in - * order to pin the page until we lock it and unmap it. - */ -static void migrate_vma_collect(struct migrate_vma *migrate) -{ - struct mmu_notifier_range range; - - /* - * Note that the pgmap_owner is passed to the mmu notifier callback so - * that the registered device driver can skip invalidating device - * private page mappings that won't be migrated. - */ - mmu_notifier_range_init_owner(&range, MMU_NOTIFY_MIGRATE, 0, - migrate->vma->vm_mm, migrate->start, migrate->end, - migrate->pgmap_owner); - mmu_notifier_invalidate_range_start(&range); - - walk_page_range_vma(migrate->vma, migrate->start, migrate->end, - &migrate_vma_walk_ops, migrate); - - mmu_notifier_invalidate_range_end(&range); - migrate->end =3D migrate->start + (migrate->npages << PAGE_SHIFT); -} - /* * migrate_vma_check_page() - check if page is pinned or not * @page: struct page to check @@ -677,8 +164,9 @@ static void migrate_vma_unmap(struct migrate_vma *migra= te) * migrate_vma_setup() - prepare to migrate a range of memory * @args: contains the vma, start, and pfns arrays for the migration * - * Returns: negative errno on failures, 0 when 0 or more pages were migrat= ed - * without an error. + * Returns: -EINVAL on invalid arguments, 0 otherwise. A 0 return does not + * imply anything was collected; on return @args->cpages holds the number = of + * pages prepared for migration (possibly 0). See the note on faulting bel= ow. * * Prepare to migrate a range of memory virtual address range by collectin= g all * the pages backing each virtual address in the range, saving them inside= the @@ -688,6 +176,20 @@ static void migrate_vma_unmap(struct migrate_vma *migr= ate) * corresponding src array entry. Then restores any pages that are pinned= , by * remapping and unlocking those pages. * + * By default migrate_vma_setup() is best-effort: virtual addresses that c= annot + * be collected are left with an empty src[] entry and the rest of the ran= ge + * still proceeds. + * + * The optional flags MIGRATE_VMA_FAULT and MIGRATE_VMA_WRITE in @args->fl= ags + * additionally fault in missing pages before collecting them, for read an= d for + * write respectively. Unlike the best-effort default, if such a fault can= not be + * satisfied the whole range is aborted: the partial collection is rolled = back, + * @args->src is cleared, @args->cpages and @args->npages are reset to 0, + * and migrate_vma_setup() still returns 0. The fault error is therefore n= ot + * observable through this interface. Callers that need to act on fault fa= ilures + * should instead drive the migration directly via hmm_range_fault() follo= wed by + * migrate_hmm_range_setup(), which preserves the error code. + * * The caller should then allocate destination memory and copy source memo= ry to * it for all those entries (ie with MIGRATE_PFN_VALID and MIGRATE_PFN_MIG= RATE * flag set). Once these are allocated and copied, the caller must update= each @@ -739,10 +241,20 @@ static void migrate_vma_unmap(struct migrate_vma *mig= rate) */ int migrate_vma_setup(struct migrate_vma *args) { + int ret; long nr_pages =3D (args->end - args->start) >> PAGE_SHIFT; + struct hmm_range range =3D { + .notifier =3D NULL, + .hmm_pfns =3D args->src, + .dev_private_owner =3D args->pgmap_owner, + .migrate =3D args, + .default_flags =3D HMM_PFN_REQ_MIGRATE + }; =20 args->start &=3D PAGE_MASK; args->end &=3D PAGE_MASK; + range.start =3D args->start; + range.end =3D args->end; if (!args->vma || is_vm_hugetlb_page(args->vma) || (args->vma->vm_flags & VM_SPECIAL) || vma_is_dax(args->vma)) return -EINVAL; @@ -764,10 +276,24 @@ int migrate_vma_setup(struct migrate_vma *args) args->cpages =3D 0; args->npages =3D 0; =20 - migrate_vma_collect(args); + if (args->flags & MIGRATE_VMA_FAULT) + range.default_flags |=3D HMM_PFN_REQ_FAULT; =20 - if (args->cpages) - migrate_vma_unmap(args); + if (args->flags & MIGRATE_VMA_WRITE) + range.default_flags |=3D HMM_PFN_REQ_FAULT | HMM_PFN_REQ_WRITE; + + ret =3D hmm_range_fault(&range); + + migrate_hmm_range_setup(&range); + + /* Remove migration PTEs */ + if (ret) { + migrate_vma_pages(args); + migrate_vma_finalize(args); + memset(args->src, 0, sizeof(*args->src) * nr_pages); + args->cpages =3D 0; + args->npages =3D 0; + } =20 /* * At this point pages are locked and unmapped, and thus they have --=20 2.55.0 From nobody Thu Sep 24 12:53:00 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9803341BA72 for ; Thu, 24 Sep 2026 06:54:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232867; cv=none; b=SJnplrVSmMLx2bgeqyZ5jAmWqpu03QHymQ5nHIAT6i0XwowhQeIN856b6CsnLAdfGd9+1ei0BQkKJUhZaLXltPhA/PcZzgKthrhgy3SOVRa8/Fin4jOUMhZJse/JUfldkDghN9RL0yMMP9KLJRLOKlrlTah1jDLtYQ56o5LN7GQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232867; c=relaxed/simple; bh=O54bS2E2a+qUgKDMDZcRSIyg39+SO3sB3ayy8FsFdKc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=lVLYkP1yIwOxH5i7ERw5aE0tA133ro7iVPiIdEnDH5vfiIyxsaKVu9JHZG0R+BLlh7PIjw90FFUaYFt9VqWPjHWbN2M34lfj+O11odyyI4Lwg55dyPf+3z+9PbLaQQrpgb0U62xNj5dDAl2CBiRbRgPmGYZthZ+LHBc0cXrBXm4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=UHbcLD07; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=YmKxyxVn; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="UHbcLD07"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="YmKxyxVn" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790232850; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2s7sy+SVvRc4UpGAfkya5y1O6jc//tVZw1Co8TmbAlo=; b=UHbcLD07+fhmSglViDYTB2Qx5McEW83uB8CLJnDeZ9XFzjYyCFhkKTC44vdCM8nEWeLx7B FUE2aruRV3pphlZ9Sop3RLW6giBd9/DrFfUR7RNqX+GnM5KloojALruMqc0Bl/1WE9PuUr 8t7x4v4tCnM1WdwTBKiwfknVN+UVPnc= Received: from mail-lj1-f198.google.com (mail-lj1-f198.google.com [209.85.208.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-374-Md_KUXjBNu2DXP_XZbkdSw-1; Thu, 24 Sep 2026 02:54:08 -0400 X-MC-Unique: Md_KUXjBNu2DXP_XZbkdSw-1 X-Mimecast-MFC-AGG-ID: Md_KUXjBNu2DXP_XZbkdSw_1790232847 Received: by mail-lj1-f198.google.com with SMTP id 38308e7fff4ca-3a27d89a894so7817881fa.3 for ; Wed, 23 Sep 2026 23:54:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790232847; x=1790837647; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=2s7sy+SVvRc4UpGAfkya5y1O6jc//tVZw1Co8TmbAlo=; b=YmKxyxVnpdUsQ17ldKVt1CngSHFWjJH5WV2CUxhOhbxwGHD+z6OcoCwvAAWQJAcRXt 98v1f2AhHqCYPw6uREOKyh8lXApAiTvpZP3CDvTmDUg6HQzRgeybp/Fl0+4WLEemD49n pLwMoi1qIHv3W3nL0MyC85f7dG4hfOqEklVNBhI0t/NCAwWSGAbVSiIhyaJq6qBZ8J5g VIixzAHHj0+K+J0J20ORdgxYyYPqAoN0YY2PUue+f3XbhwSp9uSeFSk8y0kEHp3EjpuM roAC5XK82KrgHGYTiflCzpOGmx1TViMXQYGj+WQM46GWR3jtFKyiKxERLvLNlKeAt42y DHTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790232847; x=1790837647; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=2s7sy+SVvRc4UpGAfkya5y1O6jc//tVZw1Co8TmbAlo=; b=eJu8BdFips5EpAr47OWiD7QdJuF4LwDPG8TwpJ/xWFyTbOioLQrLzoBtpWkzWoub9/ jn4WHj2NGHQ1rNGFdjKQIYlkZtHe8osAmkW9I+ysqrFW9I44ZIlt2jyvnBF9MpOxhxp9 LwiIVBFbpu9nygDMEXax/LQOU9hSRItWGoLVdCHEvvlw/way25x+vSDgrED1+oXXe543 pXZoxj127FSIhktunzBv8N2JvZpoNnPcVxzzRKzUVieLDMi4k0SR/Pvj6lCg3SiwcUP8 omaFrE1vZyoetBSHhMsLFEdSFPQHWHzsyRUu3HQuH9YM42Mw6F2SaRFufiWkkDxtffR+ XpGQ== X-Forwarded-Encrypted: i=1; AKwUvBxpCSAdHFObgbbqG9N9c17bn+FAaXNnEp6OSQYYdnx0ED0ZOAddxTvpyADJkitP2RPtTqaKBphoA/W7SM0=@vger.kernel.org X-Gm-Message-State: AFuF++mlKCKdZhA8FHjrZjYb0oA4QFEdujkLl//NcK2xmAbiu7L0krLT b+o1IpLmwqXt5uMpxBZDk+fba2B6P0+m1A3I4gF22ELyBntb0AFUPo8jkZ2kgzGHH3hYDxeegXN wGdnI6JQXsrpHXhgEVZ6H+yMv8RUS2qLUg+Pfj9S0u8eRv4528Q5RfRI4pK1yIbXZ X-Gm-Gg: AYBFou2gFel7qKMK087w4ZJdwSO7sZiT7omuf9DN8if9Y7FqwyVbmHpda/LWRhX82mF jKZqya8bZ3cz6owmNdheLirTUyqSxo3YlW2m1K/MHg0weAROFGcYAwdPTJTu70rcA4ZkysnOyMy zaZFWmRgOmtDS/dKPKGdwUzXuqHARIr70Lqycr8OLxHewOq9QZDQ/k1eRGev9j5l7nnwRy09+Kx 4BTPk8OhAvOELLE6Z5kV7VUOF953Il6sFt3xZuIIqXPJGMjjoLGGQXFHuyUQsaCkbVE0rd3KS00 CEfRslArPAyJp7ZlFnFAAvqR1SBr3ASlw4hIk0dDJYEuTIza/6x//gydB0SASHZf6LX0SbEC7CO iEXfk8LZXaig3puvHczgv X-Received: by 2002:a2e:be1d:0:b0:3a5:dfe2:7f5c with SMTP id 38308e7fff4ca-3a63c0df7bemr3352971fa.16.1790232847010; Wed, 23 Sep 2026 23:54:07 -0700 (PDT) X-Received: by 2002:a2e:be1d:0:b0:3a5:dfe2:7f5c with SMTP id 38308e7fff4ca-3a63c0df7bemr3352851fa.16.1790232846449; Wed, 23 Sep 2026 23:54:06 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-3a63bf57909sm4803141fa.29.2026.09.23.23.54.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Sep 2026 23:54:05 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Marco Pagani Subject: [PATCH v15 10/11] lib/test_hmm: add a new testcase for the migrate on fault Date: Thu, 24 Sep 2026 09:53:12 +0300 Message-ID: <20260924065313.899730-11-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260924065313.899730-1-mpenttil@redhat.com> References: <20260924065313.899730-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 Enhance the hmm test driver (lib/test_hmm) with migrate on fault case. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Signed-off-by: Marco Pagani Signed-off-by: Mika Penttil=C3=A4 --- lib/test_hmm.c | 172 +++++++++++++++++++++++-- lib/test_hmm_uapi.h | 21 +-- tools/testing/selftests/mm/hmm-tests.c | 54 ++++++++ 3 files changed, 224 insertions(+), 23 deletions(-) diff --git a/lib/test_hmm.c b/lib/test_hmm.c index cd88e8177d0a..31608b6998bb 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -36,6 +36,7 @@ #define DMIRROR_RANGE_FAULT_TIMEOUT 1000 #define DEVMEM_CHUNK_SIZE (256 * 1024 * 1024U) #define DEVMEM_CHUNKS_RESERVE 16 +#define PFNS_ARRAY_SIZE 64 =20 /* * For device_private pages, dpage is just a dummy struct page @@ -355,6 +356,7 @@ static int dmirror_range_fault(struct dmirror *dmirror, struct mm_struct *mm =3D dmirror->notifier.mm; unsigned long timeout =3D jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT); + bool migrate =3D range->default_flags & HMM_PFN_REQ_MIGRATE; int ret; =20 while (true) { @@ -364,9 +366,15 @@ static int dmirror_range_fault(struct dmirror *dmirror, } =20 range->notifier_seq =3D mmu_interval_read_begin(range->notifier); - mmap_read_lock(mm); - ret =3D hmm_range_fault(range); - mmap_read_unlock(mm); + + /* mmap lock held for whole migrate on fault operation */ + if (!migrate) { + mmap_read_lock(mm); + ret =3D hmm_range_fault(range); + mmap_read_unlock(mm); + } else { + ret =3D hmm_range_fault(range); + } if (ret) { if (ret =3D=3D -EBUSY) continue; @@ -382,7 +390,9 @@ static int dmirror_range_fault(struct dmirror *dmirror, break; } =20 - ret =3D dmirror_do_fault(dmirror, range); + /* update device page table after migration */ + if (!migrate) + ret =3D dmirror_do_fault(dmirror, range); =20 mutex_unlock(&dmirror->mutex); out: @@ -994,8 +1004,12 @@ static int dmirror_atomic_map(unsigned long addr, str= uct page *page, return 0; } =20 -static int dmirror_migrate_finalize_and_map(struct migrate_vma *args, - struct dmirror *dmirror) +/* + * Map the migrated pages into the device's page table. Caller must hold + * dmirror->mutex. + */ +static int __dmirror_migrate_map_locked(struct migrate_vma *args, + struct dmirror *dmirror) { unsigned long start =3D args->start; unsigned long end =3D args->end; @@ -1005,9 +1019,6 @@ static int dmirror_migrate_finalize_and_map(struct mi= grate_vma *args, const unsigned long start_pfn =3D start >> PAGE_SHIFT; const unsigned long end_pfn =3D end >> PAGE_SHIFT; =20 - /* Map the migrated pages into the device's page tables. */ - mutex_lock(&dmirror->mutex); - for (pfn =3D start_pfn; pfn < end_pfn; pfn++, src++, dst++) { struct page *dpage; void *entry; @@ -1036,17 +1047,27 @@ static int dmirror_migrate_finalize_and_map(struct = migrate_vma *args, if (*dst & MIGRATE_PFN_WRITE) entry =3D xa_tag_pointer(entry, DPT_XA_TAG_WRITE); entry =3D xa_store(&dmirror->pt, pfn + i, entry, GFP_ATOMIC); - if (xa_is_err(entry)) { - mutex_unlock(&dmirror->mutex); + if (xa_is_err(entry)) return xa_err(entry); - } } } =20 - mutex_unlock(&dmirror->mutex); return 0; } =20 +static int dmirror_migrate_finalize_and_map(struct migrate_vma *args, + struct dmirror *dmirror) +{ + int ret; + + /* Map the migrated pages into the device's page tables. */ + mutex_lock(&dmirror->mutex); + ret =3D __dmirror_migrate_map_locked(args, dmirror); + mutex_unlock(&dmirror->mutex); + + return ret; +} + static int dmirror_exclusive(struct dmirror *dmirror, struct hmm_dmirror_cmd *cmd) { @@ -1377,6 +1398,127 @@ static int dmirror_migrate_to_device(struct dmirror= *dmirror, return ret; } =20 +static int do_fault_and_migrate(struct dmirror *dmirror, struct hmm_range = *range) +{ + struct migrate_vma *migrate =3D range->migrate; + unsigned long timeout =3D + jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT); + int ret; + + do { + mmap_read_lock(dmirror->notifier.mm); + + /* Fault-in pages for migration */ + ret =3D dmirror_range_fault(dmirror, range); + /* + * Set this up even on error: hmm_range_fault() may have collected + * part of the range before failing. + */ + migrate_hmm_range_setup(range); + if (ret) { + /* + * dst[] entries are empty, so this marks every collected + * migration as failed. finalize then restores the source + * mappings and drops the associated locks/references. + */ + migrate_vma_pages(migrate); + migrate_vma_finalize(migrate); + goto unlock; + } + + pr_debug("Migrating from sys mem to device mem\n"); + + dmirror_migrate_alloc_and_copy(migrate, dmirror); + migrate_vma_pages(migrate); + + mutex_lock(&dmirror->mutex); + if (mmu_interval_read_retry(&dmirror->notifier, + range->notifier_seq)) + ret =3D -EBUSY; + else + ret =3D __dmirror_migrate_map_locked(migrate, dmirror); + mutex_unlock(&dmirror->mutex); + + migrate_vma_finalize(migrate); +unlock: + mmap_read_unlock(dmirror->notifier.mm); + } while (ret =3D=3D -EBUSY && !time_after(jiffies, timeout)); + + return ret; +} + +static int dmirror_fault_and_migrate_to_device(struct dmirror *dmirror, + struct hmm_dmirror_cmd *cmd) +{ + unsigned long start, size, end, next; + unsigned long src_pfns[PFNS_ARRAY_SIZE] =3D { 0 }; + unsigned long dst_pfns[PFNS_ARRAY_SIZE] =3D { 0 }; + struct migrate_vma migrate =3D { 0 }; + struct hmm_range range =3D { 0 }; + struct dmirror_bounce bounce; + int ret =3D 0; + + /* Whole range */ + start =3D cmd->addr; + size =3D cmd->npages << PAGE_SHIFT; + end =3D start + size; + + if (!mmget_not_zero(dmirror->notifier.mm)) { + ret =3D -EFAULT; + goto out; + } + + migrate.pgmap_owner =3D dmirror->mdevice; + migrate.src =3D src_pfns; + migrate.dst =3D dst_pfns; + migrate.flags =3D MIGRATE_VMA_SELECT_SYSTEM; + + range.migrate =3D &migrate; + range.hmm_pfns =3D src_pfns; + range.pfn_flags_mask =3D 0; + range.default_flags =3D HMM_PFN_REQ_FAULT | HMM_PFN_REQ_MIGRATE; + range.dev_private_owner =3D dmirror->mdevice; + range.notifier =3D &dmirror->notifier; + + for (next =3D start; next < end; next =3D range.end) { + range.start =3D next; + range.end =3D min(end, next + (PFNS_ARRAY_SIZE << PAGE_SHIFT)); + + pr_debug("Fault and migrate range start:%#lx end:%#lx\n", + range.start, range.end); + + ret =3D do_fault_and_migrate(dmirror, &range); + if (ret) + goto out_mmput; + } + + /* + * Return the migrated data for verification. + * Only for pages in device zone + */ + ret =3D dmirror_bounce_init(&bounce, start, size); + if (ret) + goto out_mmput; + + mutex_lock(&dmirror->mutex); + ret =3D dmirror_do_read(dmirror, start, end, &bounce); + mutex_unlock(&dmirror->mutex); + if (ret =3D=3D 0) { + ret =3D copy_to_user(u64_to_user_ptr(cmd->ptr), bounce.ptr, bounce.size); + if (ret) + ret =3D -EFAULT; + } + + cmd->cpages =3D bounce.cpages; + dmirror_bounce_fini(&bounce); + + +out_mmput: + mmput(dmirror->notifier.mm); +out: + return ret; +} + static void dmirror_mkentry(struct dmirror *dmirror, struct hmm_range *ran= ge, unsigned char *perm, unsigned long entry) { @@ -1643,6 +1785,10 @@ static long dmirror_fops_unlocked_ioctl(struct file = *filp, ret =3D dmirror_migrate_to_device(dmirror, &cmd); break; =20 + case HMM_DMIRROR_MIGRATE_ON_FAULT_TO_DEV: + ret =3D dmirror_fault_and_migrate_to_device(dmirror, &cmd); + break; + case HMM_DMIRROR_MIGRATE_TO_SYS: ret =3D dmirror_migrate_to_system(dmirror, &cmd); break; diff --git a/lib/test_hmm_uapi.h b/lib/test_hmm_uapi.h index ea9b0ec404fb..67d267a3b999 100644 --- a/lib/test_hmm_uapi.h +++ b/lib/test_hmm_uapi.h @@ -29,16 +29,17 @@ struct hmm_dmirror_cmd { }; =20 /* Expose the address space of the calling process through hmm device file= */ -#define HMM_DMIRROR_READ _IOWR('H', 0x00, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_WRITE _IOWR('H', 0x01, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_MIGRATE_TO_DEV _IOWR('H', 0x02, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_MIGRATE_TO_SYS _IOWR('H', 0x03, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_SNAPSHOT _IOWR('H', 0x04, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_EXCLUSIVE _IOWR('H', 0x05, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_CHECK_EXCLUSIVE _IOWR('H', 0x06, struct hmm_dmirror_cm= d) -#define HMM_DMIRROR_RELEASE _IOWR('H', 0x07, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_FLAGS _IOWR('H', 0x08, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_READ_UNLOCKED _IOWR('H', 0x09, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_READ _IOWR('H', 0x00, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_WRITE _IOWR('H', 0x01, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_MIGRATE_TO_DEV _IOWR('H', 0x02, struct hmm_dmirror_cm= d) +#define HMM_DMIRROR_MIGRATE_TO_SYS _IOWR('H', 0x03, struct hmm_dmirror_cm= d) +#define HMM_DMIRROR_SNAPSHOT _IOWR('H', 0x04, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_EXCLUSIVE _IOWR('H', 0x05, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_CHECK_EXCLUSIVE _IOWR('H', 0x06, struct hmm_dmirror_c= md) +#define HMM_DMIRROR_RELEASE _IOWR('H', 0x07, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_FLAGS _IOWR('H', 0x08, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_READ_UNLOCKED _IOWR('H', 0x09, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_MIGRATE_ON_FAULT_TO_DEV _IOWR('H', 0x0a, struct hmm_dm= irror_cmd) =20 #define HMM_DMIRROR_FLAG_FAIL_ALLOC (1ULL << 0) =20 diff --git a/tools/testing/selftests/mm/hmm-tests.c b/tools/testing/selftes= ts/mm/hmm-tests.c index e2642eca0d02..a2d1b8f494b6 100644 --- a/tools/testing/selftests/mm/hmm-tests.c +++ b/tools/testing/selftests/mm/hmm-tests.c @@ -334,6 +334,13 @@ static int hmm_migrate_sys_to_dev(int fd, return hmm_dmirror_cmd(fd, HMM_DMIRROR_MIGRATE_TO_DEV, buffer, npages); } =20 +static int hmm_migrate_on_fault_sys_to_dev(int fd, + struct hmm_buffer *buffer, + unsigned long npages) +{ + return hmm_dmirror_cmd(fd, HMM_DMIRROR_MIGRATE_ON_FAULT_TO_DEV, buffer, n= pages); +} + static int hmm_migrate_dev_to_sys(int fd, struct hmm_buffer *buffer, unsigned long npages) @@ -938,6 +945,53 @@ TEST_F(hmm, migrate) hmm_buffer_free(buffer); } =20 + +/* + * Fault and migrate anonymous memory to device private memory. + */ +TEST_F(hmm, migrate_on_fault) +{ + struct hmm_buffer *buffer; + unsigned long npages; + unsigned long size; + unsigned long i; + int *ptr; + int ret; + + npages =3D ALIGN(HMM_BUFFER_SIZE, self->page_size) >> self->page_shift; + ASSERT_NE(npages, 0); + size =3D npages << self->page_shift; + + buffer =3D malloc(sizeof(*buffer)); + ASSERT_NE(buffer, NULL); + + buffer->fd =3D -1; + buffer->size =3D size; + buffer->mirror =3D malloc(size); + ASSERT_NE(buffer->mirror, NULL); + + buffer->ptr =3D mmap(NULL, size, + PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, + buffer->fd, 0); + ASSERT_NE(buffer->ptr, MAP_FAILED); + + /* Initialize buffer in system memory. */ + for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) + ptr[i] =3D i; + + /* Fault and migrate memory to device. */ + ret =3D hmm_migrate_on_fault_sys_to_dev(self->fd, buffer, npages); + ASSERT_EQ(ret, 0); + ASSERT_EQ(buffer->cpages, npages); + + /* Check what the device read. */ + for (i =3D 0, ptr =3D buffer->mirror; i < size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i], i); + + hmm_buffer_free(buffer); +} + /* * Migrate private file memory to device private memory. */ --=20 2.55.0 From nobody Thu Sep 24 12:53:00 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9489C42641B for ; Thu, 24 Sep 2026 06:54:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232864; cv=none; b=G0Fnibc9fp9i4s+9C9MXvRwrgnp8XscTH9+8k3zArM6tWf7eUsH5wGqbEbqNK7bhOLcbxuzUMQhHeszXA6Z8wyyXwdJ3Rsy1vQ3MCYJVdBVGWJSTL+swdFsNs4uog1Qki7LzFgkLP5jmIE6yzwgxXSccVUuH+qRCK97KuYSUYzs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790232864; c=relaxed/simple; bh=G51B54owKuBfMBL6BmQ3fz1GoUN0Axehqo+N3yadsZc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=KGfCBlg3kuj7nZZBL9a2D4pfRcc1Infa2lu08vRGRwBcIzZyT5I9t9F4LYRQCWVJ17Qj6OjzGWSfeX3/cWB8CNRCrHmue5FZjvgz8CYI/VTiU1/vDto1gq7YKc9RavpNvWM+31gaRaVPx9WBNbm3DaIRqxPzpdlMWkPd/ubqFJo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=csuWkobV; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=e5iCKh4w; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="csuWkobV"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="e5iCKh4w" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790232855; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xEEzUKmwPcuLDb8KHUBfxekosp3Yw5qO1BP0yTFVDsQ=; b=csuWkobV7AR0/0ikrVcxv1tIMzYCRSkqqESrCJEmsRoxJ1vunVQys8AsTV58pjQO3TcisE 7PWhTT4tJmsYsueovM36/Riy7vijN8F+NH+BTtF3QOYLOdvNN4kdR8uk1kZCm4mut/7u3v juCzlBSW55JL61VAvjFDnrgeSufmd1s= Received: from mail-lj1-f200.google.com (mail-lj1-f200.google.com [209.85.208.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-635-xFT3pKH2NSuhBCouvYicQw-1; Thu, 24 Sep 2026 02:54:13 -0400 X-MC-Unique: xFT3pKH2NSuhBCouvYicQw-1 X-Mimecast-MFC-AGG-ID: xFT3pKH2NSuhBCouvYicQw_1790232852 Received: by mail-lj1-f200.google.com with SMTP id 38308e7fff4ca-3a27d89a894so7818171fa.3 for ; Wed, 23 Sep 2026 23:54:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790232852; x=1790837652; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=xEEzUKmwPcuLDb8KHUBfxekosp3Yw5qO1BP0yTFVDsQ=; b=e5iCKh4wlZKJ1iGEkx5EMiOuFQpDHTje7tQgznZXHhGUxAstjMu0d9RuzprjwQ2zmv EN/+t29cWHVjHG2KjUwrTurbPubqzgV62xM94XALyxdm3yfcFWRbZom3pEKZB6Kp5bSv bHPw294TFHnWqelSaI9B+d7lVee5VBrkzSA8cPfURws3h45byftaqVvqQzyEj/ZbdtML FmRnyFto8Im7poQ1pYBFxcI9TjaI8WZ2bYdARDI2NBc0R6j5i5HAYv2MapRpdAeRUmks orpqiJhsXrRsGMjW4+xNmjpZ6KUzrxcJHubCMyWRwZI4zDW4mawt6wFUrMMmixBcElvP gVIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790232852; x=1790837652; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=xEEzUKmwPcuLDb8KHUBfxekosp3Yw5qO1BP0yTFVDsQ=; b=KGsDGR4JjXUZQcG0DwNv1ubqXBldAlesX/w8WhwUafluLsvtqL+KAlQQ9AjiweOXHi PTv1I+Tekf4u8EclZqsNnf7bjG+RV/TBnLHNwbQulm+03WmAia4loxh9qL05RzGwjbTF Icx4iBzToHbLM6tsQ8CUXxx7TeyZfbmgzJXEdw0KB8/vuce9UQbNFiZIsM/hGMG9sesH ZZpEwuy2NYP5RBCubb9vMB6QeJ/TdPOaHoJhvK3EFjH3bVQ7RsHxeOUfWG7/BmgpPK2V sz4kcWwrv8fqUafxO72r36xo989GNniM9H1Apr07qjmiSCgoaf3rbYjrF/BDc2IKtenr ze1w== X-Forwarded-Encrypted: i=1; AKwUvBwF0uLPbBEwtILxU6gdLpIg4aHvME44CT8o7erl+l075aCrZmkCo1mWtCuHomuvSrWCtYfml2767yP9fbg=@vger.kernel.org X-Gm-Message-State: AFuF++kwex1UBY1eQd78mK1Rj9mEk4/DNjC85GqjL9rEu72a1E0UHzij tFu55/AfNRvR2tTc7sAPDhhwU35U0nj1gm9wTYiCfMuaa/Df9+mCy1gm4jojHs/rm/IXth0EC1e 7qOPLy2BwmEWiPo8tD+ZikkUsQ58xB4hRqePPRRoekXeNzNMg/Rb9NSCfY4LQ4Wii X-Gm-Gg: AYBFou1/EyvDdOuUZc8taIgb6vPXgT6JmczSvX4NGyh4ZOncEMu0KBI7J6zDoSHIpTo 9+6ODvc5EaC/8bSxjUOaBnxYyGnnAVGB2fquRCDpws9kZc61zxkcdrM7ibAoFR0c5eJ/jUGB4LJ xumN1WEn7YgJzenqXJXmwOVVU1dSJkPfIgxY+D1G/Q9o1S0WD9tXpvPb6ZVsKGGNvYYSD5LKuow A5pRbR0Dfb/uWzbViv1kCd6qVr468H3cSYIFsT1ObCEofKncBz4czePFWBtjhebB6xqBwg0S9UO S4qKl8fz4X6XPmAjWEzLQdHNHtqXe2It5FG7WYcVNzDQdoRW5TlGZ3eVvX3plUnlXx+NXQVr8be 8OayeWTh0hq/6bNtkLkuA X-Received: by 2002:a2e:b893:0:b0:3a2:f233:5c6e with SMTP id 38308e7fff4ca-3a63bf652b3mr3928291fa.6.1790232852068; Wed, 23 Sep 2026 23:54:12 -0700 (PDT) X-Received: by 2002:a2e:b893:0:b0:3a2:f233:5c6e with SMTP id 38308e7fff4ca-3a63bf652b3mr3928141fa.6.1790232851558; Wed, 23 Sep 2026 23:54:11 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-3a63bf57909sm4803141fa.29.2026.09.23.23.54.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Sep 2026 23:54:08 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v15 11/11] Documentation/mm/hmm: document migration through hmm_range_fault() Date: Thu, 24 Sep 2026 09:53:13 +0300 Message-ID: <20260924065313.899730-12-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260924065313.899730-1-mpenttil@redhat.com> References: <20260924065313.899730-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 Describe the two new MIGRATE_VMA_FAULT / MIGRATE_VMA_WRITE flags of migrate_vma_setup(), and add a section on driving the migration collection phase directly from hmm_range_fault() via HMM_PFN_REQ_MIGRATE and migrate_hmm_range_setup(). Signed-off-by: Mika Penttil=C3=A4 --- Documentation/mm/hmm.rst | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/Documentation/mm/hmm.rst b/Documentation/mm/hmm.rst index fc1b8dc19825..58424e5f5873 100644 --- a/Documentation/mm/hmm.rst +++ b/Documentation/mm/hmm.rst @@ -348,6 +348,13 @@ between device driver specific code and shared common = code: Currently only anonymous private VMA ranges can be migrated to or from system memory and device private memory. =20 + By default only pages already present are collected. Two additional fla= gs + ask migrate_vma_setup() to fault in missing pages first: + + * ``MIGRATE_VMA_FAULT`` faults in missing pages with read access. + * ``MIGRATE_VMA_WRITE`` faults in missing pages with write access + (implies faulting). + One of the first steps migrate_vma_setup() does is to invalidate other device's MMUs with the ``mmu_notifier_invalidate_range_start()`` and ``mmu_notifier_invalidate_range_end()`` calls around the page table @@ -427,6 +434,38 @@ between device driver specific code and shared common = code: =20 The lock can now be released. =20 +Migration collection through hmm_range_fault() +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The collection phase of migration (steps 1 and 2 above) can also be driven= by +hmm_range_fault() directly, sharing its page table walk. This lets a driver +fault in and collect a range for migration in one walk, which is useful for +migrate on fault. + +To do so, the driver sets ``HMM_PFN_REQ_MIGRATE`` in ``range->default_flag= s`` +and points ``range->migrate`` at a ``struct migrate_vma`` it has filled in +(``flags``, ``src``, ``dst``, ``pgmap_owner``). Usually ``HMM_PFN_REQ_FAUL= T`` +(and ``HMM_PFN_REQ_WRITE``) is set as well, so missing pages are faulted in +before being collected. The mmap_read_lock() has to be held for the whole +migration, since the vma must stay stable. + +hmm_range_fault() collects the entries the same way migrate_vma_setup() do= es, +taking a folio reference, locking it and installing a migration PTE. Colle= cted +entries are marked with ``HMM_PFN_VALID | HMM_PFN_MIGRATE`` in +``range->hmm_pfns``. If the page tables change while locks are dropped the +partially collected entries are rolled back automatically. + +After hmm_range_fault() returns, the driver calls:: + + void migrate_hmm_range_setup(struct hmm_range *range); + +to translate ``range->hmm_pfns`` into ``migrate->src[]`` (``migrate->dst[]= `` is +zeroed) and initialize ``migrate->cpages`` and ``migrate->npages``. From h= ere on +the ``range->migrate`` struct is ready for the rest of the flow (steps 3 +onwards), i.e. migrate_vma_pages() and migrate_vma_finalize(). This should= be +called even on error, since hmm_range_fault() may have collected part of t= he +range before failing. + Exclusive access memory =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 --=20 2.55.0