From nobody Fri Sep 25 18:51:54 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BDC1D48A8D9 for ; Tue, 22 Sep 2026 05:34:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055289; cv=none; b=U2h18I3F1jGBlHHoBCAzNf3R6axxhxXHlT5+G8uItZbFEfETJK1+FYcq5FkTHqajSrd9qdu3fCsH8cPsmMSPHaOL7nZFYuhNGYSvYjQ/xUfwjclyyGJSKxP6oXsu4+6y1WCgWLLbmX3VZfMFiSe96wAM3FQyeuFD8Sor8TRwCLM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055289; c=relaxed/simple; bh=xGg8IJyyA8p0Uz2J69fZOcNcHmNV7VrbGKBqc0XBB8c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=LZ3HFCkqKixhr/uPhyKDOCfRPAiC8gawkDLFniMCKuoCla+NQ7a1CFSB2GBBtZ2kJocxANOrpnGfAseS+mal+GYXA58Ru8ksLjJd40gynddeVeBwUOOOXBl8YFEcV875bvM0mBXYFes5dfnbICit93w2YS/tXYdCPFGNkJZxAHM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=AqOLuhL6; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=oUGJLp5j; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="AqOLuhL6"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="oUGJLp5j" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790055286; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RZqncuA8V9KN45lL/iPHOVqmTI1CzmtLR7XQjhWRrog=; b=AqOLuhL6ICB7NAHKvpndRYIH00FR9Tm2ndaUlHHpC6eftbteHGuYQFPNaTiEolX/XEnimW kCQnjtfAyV+yOF1cS81b6jGMoL3E59jITAoHPNp7dabHosjnuJrdgSxehJHx97PzNoagJv Az1cZTkzN+ahS7/Soj8FhIP0Sd6zJxs= Received: from mail-lf1-f72.google.com (mail-lf1-f72.google.com [209.85.167.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-678-pspDJMPmNcCEFNBlmoxgcg-1; Tue, 22 Sep 2026 01:34:45 -0400 X-MC-Unique: pspDJMPmNcCEFNBlmoxgcg-1 X-Mimecast-MFC-AGG-ID: pspDJMPmNcCEFNBlmoxgcg_1790055284 Received: by mail-lf1-f72.google.com with SMTP id 2adb3069b0e04-5b4b020612aso3337416e87.0 for ; Mon, 21 Sep 2026 22:34:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790055283; x=1790660083; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=RZqncuA8V9KN45lL/iPHOVqmTI1CzmtLR7XQjhWRrog=; b=oUGJLp5jpIJ8I7kjDTv7DXWvEvFcVDnkArrrM2OZjdzXcsrtcc/CtoDm2jo9wUca66 mqCkphVYQCUIGDMXf6FnK6mtjGPJaqCsklnRTxP9UHtCwpYy8GsW4yXwb5udSUTGFK2s WX6VsfPCAGu0VvhtBs1DKBebc5yS8dXhyUWXLofRvb1CAObPI6n9YvXpSYYwVHOuPAzh TetJevxbMi//7+N3p4rXtHrXfuwMQDNTqAZ8Xz6DSOGTB3bQ8z0HQXvH45r5q8I+XCgn DVluYxkq42h5aIoHyQfCeGRx8MVd9yNTpR+AJ9wRY2wsCeh+FUzF8npCCpK4rbZ0/8sr Vg9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790055283; x=1790660083; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=RZqncuA8V9KN45lL/iPHOVqmTI1CzmtLR7XQjhWRrog=; b=grSuJR/i6DWH9r/LI/JpcSNsEBC/w9DxrQ4/x6FFRBM4/I9PCd/Zwvza0GpU95UEYU HoinpDmKfFtKYSNZCREbyZwc6d+Z3HcLgw6BdvujQjOTEMu4bLCCETrCpV8gU4mFyfN1 s1+q4XvPLp4DQ/IUpfato6jpxCGEWbiUOIRFiR4s8zYaYmTTp9PFaqUxi6RaPAjJ4jXG eihU/9hyRSlG692rzTaAeJ2kLtW6BeBF8PoUaM1LUEmx9+EN/XfY73AupSU6qIHfv4Cy BzTyceg64Acy8XBKUY3sJlq+Ahjdv062DRoaH5VATg+4dlgySds506z0gXAP9cn7Afmr BjIQ== X-Forwarded-Encrypted: i=1; AKwUvBznncCxakY12eNbu8dMqzx8dDgPtTgvAMNsoS7ycZ3+PHtydOEpYupn1xvU9sh4dKCY+SKvivG4NWmUFGM=@vger.kernel.org X-Gm-Message-State: AFuF++lxI1Nxot1NkNFKerJ28XoLbdXiKk1MQGqxvetxJ4gcPdLG3Frs d0mPH5dZJORGu4pg2FsurPpbxw0axWN5vBxRLVaGCXH4OgtosiysP5Gm93Zj36RqjoQrpLQBeOA NH+GuFlRPETmwl2odaFLezE6N+ypjz4V90fcOMXOjUZ1vQP2l7yTxaKTJNHw2xlWx X-Gm-Gg: AYBFou2bd0NBLNT0A2kA+OCfek/AdkVPo6zYaL1TnRkBGthUwJHq5csOSCgAdpwX2AM TtWqtowQjwQ++0Lbme7C+3gVgH6nsik9sqQpxGOLrF9aml7HULvvwjcHaO8sCcYGvSm/uEzqx64 EgX6VNYTYCw38gh1V5q+Zr2BQeuViN6Stjp40I6mySvnZNsXaVOU1LqFmCd3sB498xbEA21E0+W 5p6YK7xYPBlfiKJCBXcV/8B8txGCGTAVOah3Ms7hwlDkudhojsodiqBOftKKY40yqjmD0wEkSyM PDglYjAANkA0XCKgV8YjEutt21capkf+HGBF8nUdt3+8m96L1Goe8QxEebINam+BzDMjCWhuhSP b4VxL7JUpVo/JzhF31KJ0 X-Received: by 2002:a05:6512:3402:b0:5b8:b3e0:74af with SMTP id 2adb3069b0e04-5b8c195ea85mr3626235e87.59.1790055283561; Mon, 21 Sep 2026 22:34:43 -0700 (PDT) X-Received: by 2002:a05:6512:3402:b0:5b8:b3e0:74af with SMTP id 2adb3069b0e04-5b8c195ea85mr3626231e87.59.1790055283051; Mon, 21 Sep 2026 22:34:43 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b8d46e0966sm191742e87.55.2026.09.21.22.34.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 22:34:42 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v14 01/12] mm/Kconfig: changes for migrate on fault for device pages Date: Tue, 22 Sep 2026 08:34:10 +0300 Message-ID: <20260922053421.4092027-2-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922053421.4092027-1-mpenttil@redhat.com> References: <20260922053421.4092027-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 HMM depends on MMU notifiers. With the unified HMM/migrate_device page table walk migrate_device needs HMM enabled. Enable them explicitly to avoid breaking random configs. Cc: Andrew Morton Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: "Liam R. Howlett" Cc: Vlastimil Babka Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Signed-off-by: Mika Penttil=C3=A4 Acked-by: David Hildenbrand (Arm) Acked-by: Balbir Singh --- mm/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/Kconfig b/mm/Kconfig index 604c58199acb..b7a485beb9a7 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -687,6 +687,7 @@ config MIGRATION =20 config DEVICE_MIGRATION def_bool MIGRATION && ZONE_DEVICE + select HMM_MIRROR =20 config ARCH_ENABLE_HUGEPAGE_MIGRATION bool --=20 2.55.0 From nobody Fri Sep 25 18:51:54 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 36D8048E0E3 for ; Tue, 22 Sep 2026 05:34:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055292; cv=none; b=P1siPvI498nNdDUIhNrNJqm+1pfFq9eathm/Cs78rjPgmKu3ERuH5+TphSR1058ub+M6rhZiJuwj7dVIpRQmbXmNDhdngkZyLS5vWhS25lEE9hOsukk2qXft7VUKW53y05tDzxm0k/I8xOZ9Zy2/ZoBdSZrxux4ndMSz/mC+Qgc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055292; c=relaxed/simple; bh=Vw8DEcu9a1Jd1CJ9KWanGL9Yzn9S0TsNI8wC/ggW76Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=BoD/WwQONVQTaYBWhptQKYwvw25/VWzSFCVxcUcwCf759nBoIEIL5wrYYl0BiqoGoxkxpqvwTFCZBmx2Ip677Pxs8Gt2F32oQuBHYKyPwt2y9lyUW/okns+kxYIC5xXE5SaK8FSoHzZZGosEkzli4GUlTJt/TrBLToKq9n0tFsI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=buc4URW/; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=WyPoC1UZ; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="buc4URW/"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="WyPoC1UZ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790055289; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XHN2LiOQU7r0d0fe1KkDFs3khSrbeSD24b1q7hC8HmQ=; b=buc4URW/cSs9Mjmv1WYP1x9blxpske7TcKhzUHcoq5D64DH37EzGePjmXaJ73cNOwLvRnN xnCAOIKhMe7glelyxQxff8H4OXM12qSBuQX+WgL+k6TuRAqIf1YlhxHZQDJKFDDarrrpQt yW4WFTZVOgEJlI0ViGip0kv/BS4AnFY= Received: from mail-lj1-f198.google.com (mail-lj1-f198.google.com [209.85.208.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-18-0P50kFA2MKiKebBGKT1prw-1; Tue, 22 Sep 2026 01:34:47 -0400 X-MC-Unique: 0P50kFA2MKiKebBGKT1prw-1 X-Mimecast-MFC-AGG-ID: 0P50kFA2MKiKebBGKT1prw_1790055286 Received: by mail-lj1-f198.google.com with SMTP id 38308e7fff4ca-3a58d16b9abso17422701fa.3 for ; Mon, 21 Sep 2026 22:34:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790055286; x=1790660086; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=XHN2LiOQU7r0d0fe1KkDFs3khSrbeSD24b1q7hC8HmQ=; b=WyPoC1UZ1H9kFJtsbz8ZyODvxujjaQ7mROUKAkqHsXhLTo+inzDeIq8kK8fqIP8hO1 LBkK1tbinq8utm4I+b7QgrEA11UFjxDSyBx13diE2AbCZBgUPzEbuFfOf4vB9Sh4gPat 67ekAZCVgjYcd33LzFK5S4nyruz6H2aEpEg8RvH69IGnqHE33x1UHkbeF/q1Zjct8A1N bOyZEp6WW/RFxvyXhCYjdH1hZOqVIUq10jGIUKsl+9pl5yP+Ey0Uyp1SGlMeeH7E+umh Tf4ZoFU6JpZ2N4gHC6Yw0FE0o5KbBsoN88Jf5xgslyL5DkjdhAXvKJvF507djS81gy9o eI5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790055286; x=1790660086; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=XHN2LiOQU7r0d0fe1KkDFs3khSrbeSD24b1q7hC8HmQ=; b=14XjWN0Xj9XP8QJTg5HxK1AzU4EKvDanaVZYqdfhKjZalzBJH6fK5D5hM7p9+NROvP x6o5jgZGCs0GcYIFNBieGEJ+2CTn8VOrn+1YqvZ8rbn09ZPHfDUJHknWSCr6CqVvQO73 7LR5ovy1GGvDZVx/Mgom2Sg23NoPVqmyyW5FfhIOu27o2CP5pSr0Rtc/LUkRzkb7ab45 YhDP0IgDBwhYmuXwT7d8FjiCQ95rnoQZRR8l8A0nwtol1EzE4eaNwwgMEONPhktk0KqE FCAP/L70QG3Exjg+pf0c8reFijzfrPrAMGAFTMhsPDhsVjnblI0rTWMrrl7Dr/zYOyt8 jeNg== X-Forwarded-Encrypted: i=1; AKwUvByblrmEbh+MoAvSuVJqc4wRLWQW9wWJmOYRin6S6+PWu26Vd6mTetBbvDnXfSH5gknwjWAK0MaKzyfgQtA=@vger.kernel.org X-Gm-Message-State: AFuF++k0TXrfMZxCZKD9oFkrqimcXHU1/VxlOh9ttDg9WMPr2tBDLJbS YxwRrJP9gHc1K+LoeIfSN3SybYtsr7rPeI4u1FEebCq5mXZimndolRE0YZXMrOqho0kxa1R+hE+ g84EdXnVzNd0OkzliZXQFwQFsSqH8Rrklvm8HZu/4em1uWUa+pH/jhnzpMnzLgLlB X-Gm-Gg: AYBFou2XoeYtmR2AocCh+eiRMYM2f/w66E1yBRV+svLWeOfmNROo4S0MmVoIe18JGTY OdDArnw5/o7n1ihpVrss50yM7RKS2O19luTFR8+vXNec4M7mE55CI538N20RfXs1bmMQwGjC+CZ SxXq2uUKRILAwadTjBIYu3Rd2Wgz37ZcMZu7G8Mzv2iMwTkXcYIOAG1a/kGVQU0PGvcZoY5De3b cLBfr+Vd6nF2dBwWM0twsqod3RZIln/SqTY7uRpKFuCoNMZY6fJZo5sLj61Z/UfgtbSWQUJHvK1 RT9CjF9hqzB3JY6kCBVwREbLsF2GAVrwjYzeCIS0o+SIahpHgtqvm8da6thHxcKUHXp6FymcfqV wGy7qa7Gk5S/qSSoRTxki X-Received: by 2002:a05:6512:3ca7:b0:5b4:fdbe:ab4e with SMTP id 2adb3069b0e04-5b8c184e399mr3985139e87.25.1790055285988; Mon, 21 Sep 2026 22:34:45 -0700 (PDT) X-Received: by 2002:a05:6512:3ca7:b0:5b4:fdbe:ab4e with SMTP id 2adb3069b0e04-5b8c184e399mr3985136e87.25.1790055285556; Mon, 21 Sep 2026 22:34:45 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b8d46e0966sm191742e87.55.2026.09.21.22.34.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 22:34:44 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v14 02/12] mm: add helper to convert HMM pfn to migrate pfn Date: Tue, 22 Sep 2026 08:34:11 +0300 Message-ID: <20260922053421.4092027-3-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922053421.4092027-1-mpenttil@redhat.com> References: <20260922053421.4092027-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 The unified HMM/migrate_device pagewalk does the "collecting" on the HMM side, so we need a helper to transfer pfns to the migrate_vma world. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- include/linux/hmm.h | 29 +++++++++++++++++----- include/linux/migrate.h | 3 ++- mm/migrate_device.c | 55 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 80 insertions(+), 7 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 6f04e3932f5b..4f56f3419cb4 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -13,6 +13,8 @@ =20 struct mmu_interval_notifier; =20 +struct migrate_vma; + /* * On output: * 0 - The page is faultable and a future call with=20 @@ -27,13 +29,22 @@ struct mmu_interval_notifier; * HMM_PFN_P2PDMA_BUS - Bus mapped P2P transfer * HMM_PFN_DMA_MAPPED - Flag preserved on input-to-output transformation * to mark that page is already DMA mapped + * HMM_PFN_MIGRATE - The entry is to be migrated. Note, HMM_PFN_MIGRATE + * alone without HMM_PFN_VALID denotes the + * empty page. + * This flag optionally together with HMM_PFN_COMPOUND + * are indicators for migrate_hmm_range_setup() to + * setup the migrate pfns. + * HMM_PFN_COMPOUND - The entry represents a > 0 order page * * On input: - * 0 - Return the current state of the page, do not fault = it. - * HMM_PFN_REQ_FAULT - The output must have HMM_PFN_VALID or hmm_range_fau= lt() - * will fail - * HMM_PFN_REQ_WRITE - The output must have HMM_PFN_WRITE or hmm_range_fau= lt() - * will fail. Must be combined with HMM_PFN_REQ_FAULT. + * 0 - Return the current state of the page, do not faul= t it. + * HMM_PFN_REQ_FAULT - The output must have HMM_PFN_VALID or hmm_range_f= ault() + * will fail + * HMM_PFN_REQ_WRITE - The output must have HMM_PFN_WRITE or hmm_range_f= ault() + * will fail. Must be combined with HMM_PFN_REQ_FAUL= T. + * HMM_PFN_REQ_MIGRATE - For default_flags only, request to migrate the ra= nge, + * according to hmm_range.migrate.flags */ enum hmm_pfn_flags { /* Output fields and flags */ @@ -48,11 +59,15 @@ enum hmm_pfn_flags { HMM_PFN_P2PDMA =3D 1UL << (BITS_PER_LONG - 5), HMM_PFN_P2PDMA_BUS =3D 1UL << (BITS_PER_LONG - 6), =20 - HMM_PFN_ORDER_SHIFT =3D (BITS_PER_LONG - 11), + /* Migrate request */ + HMM_PFN_MIGRATE =3D 1UL << (BITS_PER_LONG - 7), + HMM_PFN_COMPOUND =3D 1UL << (BITS_PER_LONG - 8), + HMM_PFN_ORDER_SHIFT =3D (BITS_PER_LONG - 13), =20 /* Input flags */ HMM_PFN_REQ_FAULT =3D HMM_PFN_VALID, HMM_PFN_REQ_WRITE =3D HMM_PFN_WRITE, + HMM_PFN_REQ_MIGRATE =3D HMM_PFN_MIGRATE, =20 HMM_PFN_FLAGS =3D ~((1UL << HMM_PFN_ORDER_SHIFT) - 1), }; @@ -107,6 +122,7 @@ static inline unsigned int hmm_pfn_to_map_order(unsigne= d long hmm_pfn) * @default_flags: default flags for the range (write, read, ... see hmm d= oc) * @pfn_flags_mask: allows to mask pfn flags so that only default_flags ma= tter * @dev_private_owner: owner of device private pages + * @migrate: structure for migrating a range of a VMA */ struct hmm_range { struct mmu_interval_notifier *notifier; @@ -117,6 +133,7 @@ struct hmm_range { unsigned long default_flags; unsigned long pfn_flags_mask; void *dev_private_owner; + struct migrate_vma *migrate; }; =20 /* diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 78424b3824c2..280ec9a866c7 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -3,6 +3,7 @@ #define _LINUX_MIGRATE_H =20 #include +#include #include #include #include @@ -202,7 +203,7 @@ void migrate_device_pages(unsigned long *src_pfns, unsi= gned long *dst_pfns, unsigned long npages); void migrate_device_finalize(unsigned long *src_pfns, unsigned long *dst_pfns, unsigned long npages); - +void migrate_hmm_range_setup(struct hmm_range *range); #endif /* CONFIG_MIGRATION */ =20 #endif /* _LINUX_MIGRATE_H */ diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 009bfa8b212d..43e571e82cca 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -1528,3 +1528,58 @@ int migrate_device_coherent_folio(struct folio *foli= o) return 0; return -EBUSY; } + +/** + * migrate_hmm_range_setup() - prepare to migrate a range of memory + * @range: contains pointer to struct migrate_vma to be set up. + * + * When collecting has been done with hmm_range_fault(), this + * should be called next, and completes range->migrate by + * populating migrate->src[] and migrate->dst[] + * using range->hmm_pfns[]. + * Also, migrate->cpages and migrate->npages get initialized. + * After migrate_hmm_range_setup(), range->migrate is good + * for the rest of the migrate_vma_* flow. + */ +void migrate_hmm_range_setup(struct hmm_range *range) +{ + struct migrate_vma *migrate =3D range->migrate; + + if (!migrate) + return; + + migrate->npages =3D (migrate->end - migrate->start) >> PAGE_SHIFT; + migrate->cpages =3D 0; + + for (unsigned long i =3D 0; i < migrate->npages; i++) { + unsigned long pfn =3D range->hmm_pfns[i]; + + /* + * We are only interested in entries to be + * migrated. + */ + if (!(pfn & HMM_PFN_MIGRATE)) { + migrate->src[i] =3D 0; + migrate->dst[i] =3D 0; + continue; + } + + migrate->cpages++; + + /* HMM_PFN_MIGRATE without HMM_PFN_VALID denotes the special zero page */ + if (pfn & HMM_PFN_VALID) + migrate->src[i] =3D migrate_pfn(page_to_pfn(hmm_pfn_to_page(pfn))); + else + migrate->src[i] =3D 0; + + migrate->src[i] |=3D MIGRATE_PFN_MIGRATE; + migrate->src[i] |=3D (pfn & HMM_PFN_WRITE) ? MIGRATE_PFN_WRITE : 0; + migrate->src[i] |=3D (pfn & HMM_PFN_COMPOUND) ? MIGRATE_PFN_COMPOUND : 0; + migrate->dst[i] =3D 0; + } + + if (migrate->cpages) + migrate_vma_unmap(migrate); + +} +EXPORT_SYMBOL(migrate_hmm_range_setup); --=20 2.55.0 From nobody Fri Sep 25 18:51:54 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E0CAE488DB6 for ; Tue, 22 Sep 2026 05:34:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055294; cv=none; b=hCQoLc4R+8zykdh7u3YX091wQj1DfHtHFSgSzSnhjjOhytM3fogOz8J5Tjfc/kfWUFM3cojBL8ueMmlKAdN6ycFvcOug1xF6LqSc2bUjXUoEYvI/Ir3oWXNPPw5fRLJrCBV4iqDfExUROQKDXUJQDctzpMHnJ3CpmmGlvlrEDc0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055294; c=relaxed/simple; bh=+YqWnUjPR4QlAzmXvCT0RkKU/SuOrgunbWSL3HKniBU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Kv9ZWNA/8xXMfgSbO7II+Tr2H0htst1r8dfu8UqHvCijvKs1h9kcqDx99oqiNx7aDjJ1sqWsqY8afoRa+YnAzIhvxVqE6b3C3/Axt7UN1XkksRZ+Ep4V/RdshWLG8VpLfYhG2e3Ml65Ba+gjm65JOvYu0rwcfAAvdXU5psmlgGo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=bU0Si8+q; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=JSLUxzxi; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="bU0Si8+q"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="JSLUxzxi" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790055291; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ggFf9vsm/p6yby6h1KHVQFgSLVs+0wCPIsczXGIceWQ=; b=bU0Si8+qdlCmbYYjsaz5hxhfgyeyQ6nH8goihzL7cvjPsa2mTy5QLbRTuE1iBNz6PWgx5l 5jEyv5DvdeZytfLfIOMod1jkX8varMMF8q1qyS6YfzsCuY1Gank0rNEc4f5TUgSb2X6dvP 0VjmgNgK448Gmhn1IHXywEkZQ1WjovQ= Received: from mail-lf1-f71.google.com (mail-lf1-f71.google.com [209.85.167.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-567-hh0UDpvUPdGenAr7uzWyMQ-1; Tue, 22 Sep 2026 01:34:50 -0400 X-MC-Unique: hh0UDpvUPdGenAr7uzWyMQ-1 X-Mimecast-MFC-AGG-ID: hh0UDpvUPdGenAr7uzWyMQ_1790055289 Received: by mail-lf1-f71.google.com with SMTP id 2adb3069b0e04-5b609be5e96so2237383e87.3 for ; Mon, 21 Sep 2026 22:34:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790055289; x=1790660089; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=ggFf9vsm/p6yby6h1KHVQFgSLVs+0wCPIsczXGIceWQ=; b=JSLUxzxi2o1CZ/efEKO3iLkwRbLpJD76XEWzEpNIT6+ukMg4hAwd1lB9pi1+tdNEov bBpwO8qGdYU8EE/4s0ihG/Ji8HU0B52lx3xuf6UsxB3P/maIcQDbH8Dy/12VFY/vR5kp mgTzvDDpV0Px6AXHBf0/RYGoJ4lQYb/3auj5r9mB/aIEfIMyPfS38ESznTfMa0WB5rHV eoLNKRgNyZjCvaJ4LIX+uCnnkVjAxwXfB1JMSODZkzpbNiGIshPDXgBBs/TMxenkPmPv bVb7A9XVAF7DAQmYt7h1OaC/QdUfu78jqXSmXs5FJw6eB5veO67QuNPzPuvwmlwDFmuY 7Sdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790055289; x=1790660089; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=ggFf9vsm/p6yby6h1KHVQFgSLVs+0wCPIsczXGIceWQ=; b=oq5pPWjbmVsNS6v8MXIUNX2rE+SWV+kPaWjClb7TIBxSG7CX7LFenOKuhV68hMnBGh 8TFxpoyuHHeKXYzQgS7V39vfW9U4ALxz8/WZM8GeSM2kapFfWKdbnKuXVrdwxSHGvBOu L/oca94J+Xd9NthmVrLofrBG3PYRW+Gi3gy+7efLw7pAbE7SvqKIV7ZgZ+DVGb4aCD5j yv04dSqD0RNqZbF8/1Vyv0Z2oRMVM9bX9eBNNcGM7/OqkSQOeZLiTqJx5w0alxYoT5ty J7MYhxIBCwncOGsYDkM09WNij9eo+Qi84wClO3/2JmtkqEUkRXGpUJwuJ2jM/nQ4icqz vGiQ== X-Forwarded-Encrypted: i=1; AKwUvBx/xHg/dCqv6aSjGc5178+xcZowOuZS2m0/cSl1BLA3TGyWEbrckIZuPa4EcQ6G23TAVoGRfGiWvwSe0T4=@vger.kernel.org X-Gm-Message-State: AFuF++lVmaJmaPnGECHfHF2fiK/tGR+r92+XhND6RHCq/MNEg7v+QuMe WmQZYfAJmQsdjx6VmEvm8NNHvC9WVVEMkxckpFW66Qd6DiRb50JpzHy00wQfUfLa/JvFuhiTUT4 FZJPi4di+84aajQRLx5cPTmSPDHIMlO2mPl36YgQLFzY9ZmLfUMHjt6V/JvGT4xa7 X-Gm-Gg: AYBFou2r9qHMSHhcR89QdsqrJmajAdVqz4i8aBj+kAiJwX7gSpRFcJb0k+nyqPOYG78 r2H7GhN8Qw64c6tTeQijU8tY+p26zH4ICM4WAy42Q4D745ip1CcwTqDKOV3kXkpnie0kv1Jpl7R a5FvBFT16RFWIQcbLlnX4uNhJ2rRFP8AnYhfh0bUdKEL7ogM4U4/DK3bsjynFwpgxLa5bRfYLm3 XRy1bcoOkzwLKGcEjUaxeKu44OyCknzJANNaJb+OSAXFD+jJH9ESj0HVwxGaKWWdw7joUvkdy9O PT4Rl2ZkO9Mn9sciS5axhhaNn6dG667NsJSQp2xPCxC+3a+tgn2IQBDX2aT/7ieestaiREgCh6V YWcnLY+HagMMay/MaVjUC X-Received: by 2002:ac2:51c3:0:b0:5b4:b03e:b17 with SMTP id 2adb3069b0e04-5b8c17f19eemr4703898e87.4.1790055288826; Mon, 21 Sep 2026 22:34:48 -0700 (PDT) X-Received: by 2002:ac2:51c3:0:b0:5b4:b03e:b17 with SMTP id 2adb3069b0e04-5b8c17f19eemr4703877e87.4.1790055288311; Mon, 21 Sep 2026 22:34:48 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b8d46e0966sm191742e87.55.2026.09.21.22.34.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 22:34:47 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v14 03/12] mm/hmm: preparations for HMM to participate in migration Date: Tue, 22 Sep 2026 08:34:12 +0300 Message-ID: <20260922053421.4092027-4-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922053421.4092027-1-mpenttil@redhat.com> References: <20260922053421.4092027-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 For migration to happen after hmm_range_fault(), the vma has to retrieved. Luckily, pagewalk already resolves that for us, so just have to save it for further use. For migration, mmap_lock has to be hold for the whole operation, so vma stays stable. In case lock-drop behavior is requested, return an error while migrating. Also, rename some struct hmm_vma_walk fields to solve namespace collisions, and we are later introducing more xxlocked fields so be unambiguous. Take care also of firing the mmu_notifier when migrating. Also, prepare for differencies of how mm_struct * is retrieved in fault and migrate_vma paths. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- include/linux/migrate.h | 49 +++++++++++++++++- lib/test_hmm.c | 2 +- mm/hmm.c | 107 ++++++++++++++++++++++++++++++++++------ 3 files changed, 141 insertions(+), 17 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 280ec9a866c7..01c5f62a56e9 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include =20 @@ -108,6 +109,29 @@ static inline void softleaf_entry_wait_on_locked(softl= eaf_t entry, spinlock_t *p spin_unlock(ptl); } =20 +enum migrate_vma_info { + MIGRATE_VMA_SELECT_NONE =3D 0, + MIGRATE_VMA_SELECT_COMPOUND =3D MIGRATE_VMA_SELECT_NONE, +}; + +static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *r= ange) +{ + return MIGRATE_VMA_SELECT_NONE; +} + +static inline void hmm_fill_migrate_vma(struct hmm_range *range, + struct vm_area_struct *vma, + unsigned long start, + unsigned long end) +{ +} + +#ifdef CONFIG_MMU_NOTIFIER +static inline struct mm_struct *hmm_range_fault_mm(struct hmm_range *range) +{ + return range->notifier->mm; +} +#endif #endif /* CONFIG_MIGRATION */ =20 #ifdef CONFIG_NUMA_BALANCING @@ -151,7 +175,7 @@ static inline unsigned long migrate_pfn(unsigned long p= fn) return (pfn << MIGRATE_PFN_SHIFT) | MIGRATE_PFN_VALID; } =20 -enum migrate_vma_direction { +enum migrate_vma_info { MIGRATE_VMA_SELECT_SYSTEM =3D 1 << 0, MIGRATE_VMA_SELECT_DEVICE_PRIVATE =3D 1 << 1, MIGRATE_VMA_SELECT_DEVICE_COHERENT =3D 1 << 2, @@ -193,6 +217,29 @@ struct migrate_vma { struct page *fault_page; }; =20 +// TODO: enable migration +static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *r= ange) +{ + return 0; +} + +#ifdef CONFIG_MMU_NOTIFIER +static inline struct mm_struct *hmm_range_fault_mm(struct hmm_range *range) +{ + return range->notifier ? range->notifier->mm : range->migrate->vma->vm_mm; +} +#endif + +static inline void hmm_fill_migrate_vma(struct hmm_range *range, + struct vm_area_struct *vma, + unsigned long start, + unsigned long end) +{ + range->migrate->vma =3D vma; + range->migrate->start =3D start; + range->migrate->end =3D end; +} + int migrate_vma_setup(struct migrate_vma *args); void migrate_vma_pages(struct migrate_vma *migrate); void migrate_vma_finalize(struct migrate_vma *migrate); diff --git a/lib/test_hmm.c b/lib/test_hmm.c index 6911daa9f854..cd88e8177d0a 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -145,7 +145,7 @@ static bool dmirror_is_private_zone(struct dmirror_devi= ce *mdevice) HMM_DMIRROR_MEMORY_DEVICE_PRIVATE); } =20 -static enum migrate_vma_direction +static enum migrate_vma_info dmirror_select_device(struct dmirror *dmirror) { return (dmirror->mdevice->zone_device_type =3D=3D diff --git a/mm/hmm.c b/mm/hmm.c index 2f1e98c6b644..be39d2e5403a 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -28,15 +29,21 @@ #include #include #include +#include =20 #include "internal.h" =20 struct hmm_vma_walk { - struct hmm_range *range; - bool *locked; - unsigned long last; - unsigned long end; - unsigned int required_fault; + struct mmu_notifier_range mmu_range; + struct vm_area_struct *vma; + struct hmm_range *range; + unsigned long start; + unsigned long end; + unsigned long last; + /* fault and lock drop related fields */ + bool *mmlocked; + unsigned long fault_end; + unsigned int required_fault; }; =20 /* @@ -95,7 +102,7 @@ static int hmm_record_fault(unsigned long addr, unsigned= long end, =20 WARN_ON_ONCE(!required_fault); hmm_vma_walk->last =3D addr; - hmm_vma_walk->end =3D end; + hmm_vma_walk->fault_end =3D end; hmm_vma_walk->required_fault =3D required_fault; return HMM_FAULT_PENDING; } @@ -400,6 +407,57 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *w= alk, unsigned long start, } #endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */ =20 +static int hmm_vma_capture_migrate_range(unsigned long start, + unsigned long end, + struct mm_walk *walk) +{ + struct hmm_vma_walk *hmm_vma_walk =3D walk->private; + struct hmm_range *range =3D hmm_vma_walk->range; + + if (!hmm_select_migrate(range)) + return 0; + + if (hmm_vma_walk->vma && (hmm_vma_walk->vma !=3D walk->vma)) + return -ERANGE; + + hmm_vma_walk->vma =3D walk->vma; + hmm_vma_walk->start =3D start; + hmm_vma_walk->end =3D end; + + if (end - start > range->end - range->start) + return -ERANGE; + + if (!hmm_vma_walk->mmu_range.owner) { + mmu_notifier_range_init_owner(&hmm_vma_walk->mmu_range, MMU_NOTIFY_MIGRA= TE, 0, + walk->vma->vm_mm, start, end, + range->dev_private_owner); + mmu_notifier_invalidate_range_start(&hmm_vma_walk->mmu_range); + } + + return 0; +} + +static void hmm_vma_post_range_fault(struct hmm_vma_walk *hmm_vma_walk) +{ + + struct hmm_range *range =3D hmm_vma_walk->range; + + if (hmm_select_migrate(range) && + hmm_vma_walk->mmu_range.owner) { + /* + * The migrate_vma path has the following initialized, + * so take care of fault path below. + */ + if (range->notifier) { + hmm_fill_migrate_vma(range, + hmm_vma_walk->vma, + hmm_vma_walk->start, + hmm_vma_walk->end); + } + mmu_notifier_invalidate_range_end(&hmm_vma_walk->mmu_range); + } +} + static int hmm_vma_walk_pmd(pmd_t *pmdp, unsigned long start, unsigned long end, @@ -594,6 +652,11 @@ static int hmm_vma_walk_test(unsigned long start, unsi= gned long end, struct hmm_vma_walk *hmm_vma_walk =3D walk->private; struct hmm_range *range =3D hmm_vma_walk->range; struct vm_area_struct *vma =3D walk->vma; + int r; + + r =3D hmm_vma_capture_migrate_range(start, end, walk); + if (r) + return r; =20 if (!(vma->vm_flags & (VM_IO | VM_PFNMAP)) && vma->vm_flags & VM_READ) @@ -646,12 +709,12 @@ static int hmm_do_fault(struct mm_struct *mm, struct hmm_vma_walk *hmm_vma_walk) { unsigned long addr =3D hmm_vma_walk->last; - unsigned long end =3D hmm_vma_walk->end; + unsigned long end =3D hmm_vma_walk->fault_end; unsigned int required_fault =3D hmm_vma_walk->required_fault; unsigned int fault_flags =3D FAULT_FLAG_REMOTE; struct vm_area_struct *vma; =20 - if (hmm_vma_walk->locked) + if (hmm_vma_walk->mmlocked) fault_flags |=3D FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; =20 vma =3D vma_lookup(mm, addr); @@ -670,8 +733,8 @@ static int hmm_do_fault(struct mm_struct *mm, ret =3D handle_mm_fault(vma, addr, fault_flags, NULL); =20 if (ret & (VM_FAULT_COMPLETED | VM_FAULT_RETRY)) { - if (hmm_vma_walk->locked) /* needed by sparse */ - *hmm_vma_walk->locked =3D false; + if (hmm_vma_walk->mmlocked) /* needed by sparse */ + *hmm_vma_walk->mmlocked =3D false; else WARN_ON_ONCE(1); /* broken fault handler */ return HMM_FAULT_UNLOCKED; @@ -694,19 +757,31 @@ static int hmm_range_fault_locked(struct hmm_range *r= ange, bool *locked) { struct hmm_vma_walk hmm_vma_walk =3D { .range =3D range, - .locked =3D locked, + .mmlocked =3D locked, .last =3D range->start, }; - struct mm_struct *mm =3D range->notifier->mm; + /* + * Could be serving a device fault or come from migrate + * entry point. For the former we have not resolved the vma + * yet, and the latter we don't have a notifier (but have a vma). + * + */ + struct mm_struct *mm =3D hmm_range_fault_mm(range); int ret; =20 + /* Migration is incompatible with mmap lock drop */ + if (locked && hmm_select_migrate(range)) + return -EINVAL; + mmap_assert_locked(mm); =20 do { /* If range is no longer valid force retry. */ - if (mmu_interval_check_retry(range->notifier, - range->notifier_seq)) - return -EBUSY; + if (range->notifier && mmu_interval_check_retry(range->notifier, + range->notifier_seq)) { + ret =3D -EBUSY; + break; + } ret =3D walk_page_range(mm, hmm_vma_walk.last, range->end, &hmm_walk_ops, &hmm_vma_walk); /* @@ -732,6 +807,8 @@ static int hmm_range_fault_locked(struct hmm_range *ran= ge, bool *locked) * output, and all >=3D are still at their input values. */ } while (ret =3D=3D -EBUSY); + + hmm_vma_post_range_fault(&hmm_vma_walk); return ret; } =20 --=20 2.55.0 From nobody Fri Sep 25 18:51:54 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A864E48EC62 for ; Tue, 22 Sep 2026 05:34:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055298; cv=none; b=WPilF1oGhYk1rf6snYuUrwWzYTyfZC8kjYHFykaYCWhdAZ+fuViRBOekD7jRIsnz9ZOz4up9iuIDTXDp+EtUEP7ig7bwtt1zo3o59r6BTutJP5WHXqvyPxCEG1iSgbsJ1E/qU6pB9DroJSMeUzGPan30YLoqbkqWNzsBMo4bMCk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055298; c=relaxed/simple; bh=8huE7awOA9jy28uWgIltai/48X1SQ/3W8P943KuRd1k=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=pKAp/QOD2Xqi7eDqf47SQvjpXQ7jYH+lYQXgu1/qgbQj8eenk/R9zMaey7yjt+MO2OFA8iBE05Z+gVxJ33FtVueJKHCSkwdoOV1cLj9HsTgWsSXWX1gmeBU5n1b6xhnm6qrX1auM31QRG1wtOnb39de3DIFfbsJy90lThel9wak= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=HOBNsqYZ; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=gwSR5KJJ; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="HOBNsqYZ"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="gwSR5KJJ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790055294; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AT0RwoJ8r2XRPNe8c9sAnpXjwtunlXN+XFUKufZZink=; b=HOBNsqYZMM+GhWXjnf1OcFXuZe3S33usCEnkthb1xHvqOXodOlljgmm5MNw1qsaBNIOp2w DUX5NlYjx+TSn+an+XSJtF5/3x4j71r/ft/dAcjEYU6Qf0komstZzMjZkoFAwQDwx8SxND YDitMYol48fTFr8ioJR4zBu672B8MC4= Received: from mail-lf1-f69.google.com (mail-lf1-f69.google.com [209.85.167.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-323-j6yLAfBKOgeErkf2iueraw-1; Tue, 22 Sep 2026 01:34:53 -0400 X-MC-Unique: j6yLAfBKOgeErkf2iueraw-1 X-Mimecast-MFC-AGG-ID: j6yLAfBKOgeErkf2iueraw_1790055292 Received: by mail-lf1-f69.google.com with SMTP id 2adb3069b0e04-5b609be5e96so2237407e87.3 for ; Mon, 21 Sep 2026 22:34:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790055292; x=1790660092; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=AT0RwoJ8r2XRPNe8c9sAnpXjwtunlXN+XFUKufZZink=; b=gwSR5KJJSCRkxw0HU85W0IUWms8/oeyQZJTRIHlFytghgweQNlbRpwOUjYPLFJDpyz 6JTbYx9Js03TqKWJSV28LyhF+X5ue72wT2GH88toZZjFfmvft7TIrtpxMYYNA/sBLrUo YwTGuFoQjOve2ZoMfE0D2DSQZj/j19WiSbd1mso+ZVoKgB1Sqp6BAuQIpON/D7CbxGXS bkRxCgRyLpGhlmf/wp46ZknHm49imVS/TgsZ1BuzJtp21CAQXCmStzqhgwKv8HCUrwly aQSVH3NoqC1s598dbo8RBp/rsxu8YaRFfrjTFV6IUA0FzSdXEhGThN3qgJLnWOb2v6+6 8V5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790055292; x=1790660092; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=AT0RwoJ8r2XRPNe8c9sAnpXjwtunlXN+XFUKufZZink=; b=t/I/KRr+JecUWzffGMxZ1L2IJyKf/302fzzpOvkqn9d7VNtbiUHOiUG+rjAXF1+fgL AJnsA5mcL9rYcc3ubhggaf6X4ARpI5XErO04oXYqEFUKDSv1mfwIguLfcjt2RUrz89r2 Qr+wXcZjTYv/g34pUHmgxJ8u7eM5NywR11pfNW3ZXJsrhZq4cj9sY2HHC5gp5WuI4XrO wpIhg9gf4A5WAEcBl2iOgrv3r0KcqbT0awmRzFohYCXwb5CLYL9bIR7ML828/gSTdT+D hAGkP3i18aBkYA+W+yn50H6WhTxndtizPU/ySRnEOBCqQthBzyRFut10MKvNOmTBztb+ R6VQ== X-Forwarded-Encrypted: i=1; AKwUvBzM53Zn/EWQKi0lMccYBR2LTAcGbVyxE2uNq2JKRz4TIl/S8Qt9X2LKrB6qJaLukELnuvKaL1XlOBQWBgU=@vger.kernel.org X-Gm-Message-State: AFuF++mQekXyip09r3P2VuomHY53Yw4eauSaws+JkOpT9nL9/CdRyMen vw4wITLbsUZdoUnXXL7MNbvH1Ork2edi7ccIumtqIU/FFoJJQ82maqwRw9YXKvVdiflpM/LDO8H 4a8wrsacC781Ciy8eIfiXwmcoYjdfjkHWjgIBkMwEPHVRvIfPRzl7c5p8dNn12q2n X-Gm-Gg: AYBFou15K4061pCMttP6D2CDx2NmVWy4VWV4abIOe818wHgHMs5SmKwE4Sa8OsblQ/F 6oplE1WQRUopytj1CG4mMQy/GvEd1KylM62X9LWyIdhkdBRPBU6MtQYX0LDxnOgRAQYwpEKjVFz UDJ8VtAj6lBMGUcvb/x0KjiJkc29BOGOQLBlnKxfuzdUQpArat0SSxMH1XHtTSeZ3m+QORsETOt wI7rkBrYniN3KA7JphGYB23BYGWdE4mPCYjvalt28ga72Jx8BS7D24jIa+bwTIbFOtyaNnUO9b6 AzcMeGhropSqPXaPSM4aOrs7GZM7uSp4tTUpuHIC5YfNFAZQnSH3hsL2/bqyt5JI6n+0nf7RlJN oRJWVRkIYsTY1vw3uj2j8 X-Received: by 2002:a05:6512:39d1:b0:5b4:24a3:ecf9 with SMTP id 2adb3069b0e04-5b8c180b6ffmr4840048e87.21.1790055291511; Mon, 21 Sep 2026 22:34:51 -0700 (PDT) X-Received: by 2002:a05:6512:39d1:b0:5b4:24a3:ecf9 with SMTP id 2adb3069b0e04-5b8c180b6ffmr4840036e87.21.1790055290968; Mon, 21 Sep 2026 22:34:50 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b8d46e0966sm191742e87.55.2026.09.21.22.34.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 22:34:49 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v14 04/12] mm/hmm: do the plumbing for HMM to participate in migration Date: Tue, 22 Sep 2026 08:34:13 +0300 Message-ID: <20260922053421.4092027-5-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922053421.4092027-1-mpenttil@redhat.com> References: <20260922053421.4092027-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 Do the preparations in hmm_range_fault() and pagewalk callbacks to do the "collecting" part of migration, needed for migration. These steps include locking for pmd/pte if migrating, and calling the still dummy hmm_vma_handle_migrate_prepare_pmd() and hmm_vma_handle_migrate_prepare() functions in the pagewalk. When doing migration, have to have pmd/pte table locked for the duration of both hmm_vma_handle*() and hmm_vma_handle_migrate_prepare*() walks, and unlock when handling the faults. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- mm/hmm.c | 361 +++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 310 insertions(+), 51 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index be39d2e5403a..4805c88347e6 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -40,6 +40,16 @@ struct hmm_vma_walk { unsigned long start; unsigned long end; unsigned long last; + /* + * For migration we need pte/pmd locked for the handle_* and + * prepare_* regions. While faulting we have to drop the locks + * and start again. ptelocked and pmdlocked hold the state + * and tell if need to drop locks before faulting. + * ptl is the lock held for pte or pmd. + */ + bool ptelocked; + bool pmdlocked; + spinlock_t *ptl; /* fault and lock drop related fields */ bool *mmlocked; unsigned long fault_end; @@ -61,6 +71,16 @@ struct hmm_vma_walk { */ #define HMM_FAULT_UNLOCKED -ENOLCK =20 +#define HMM_ASSERT_PTE_LOCKED(hmm_vma_walk, locked) \ + WARN_ON_ONCE((hmm_vma_walk)->ptelocked !=3D locked) + +#define HMM_ASSERT_PMD_LOCKED(hmm_vma_walk, locked) \ + WARN_ON_ONCE((hmm_vma_walk)->pmdlocked !=3D locked) + +#define HMM_ASSERT_UNLOCKED(hmm_vma_walk) \ + WARN_ON_ONCE((hmm_vma_walk)->ptelocked || \ + (hmm_vma_walk)->pmdlocked) + enum { HMM_NEED_FAULT =3D 1 << 0, HMM_NEED_WRITE_FAULT =3D 1 << 1, @@ -74,14 +94,38 @@ enum { }; =20 static int hmm_pfns_fill(unsigned long addr, unsigned long end, - struct hmm_range *range, unsigned long cpu_flags) + struct hmm_vma_walk *hmm_vma_walk, unsigned long cpu_flags) { + struct hmm_range *range =3D hmm_vma_walk->range; unsigned long i =3D (addr - range->start) >> PAGE_SHIFT; + enum migrate_vma_info minfo; + bool migrate =3D false; + + minfo =3D hmm_select_migrate(range); + if (cpu_flags !=3D HMM_PFN_ERROR) { + if (minfo && (vma_is_anonymous(hmm_vma_walk->vma))) { + cpu_flags |=3D HMM_PFN_MIGRATE; + migrate =3D true; + } + } + + if (migrate && thp_migration_supported() && + (minfo & MIGRATE_VMA_SELECT_COMPOUND) && + IS_ALIGNED(addr, HPAGE_PMD_SIZE) && + IS_ALIGNED(end, HPAGE_PMD_SIZE) && + end-addr =3D=3D HPAGE_PMD_SIZE) { + range->hmm_pfns[i] &=3D HMM_PFN_INOUT_FLAGS; + range->hmm_pfns[i] |=3D cpu_flags | HMM_PFN_COMPOUND; + addr +=3D PAGE_SIZE; + i++; + cpu_flags =3D 0; + } =20 for (; addr < end; addr +=3D PAGE_SIZE, i++) { range->hmm_pfns[i] &=3D HMM_PFN_INOUT_FLAGS; range->hmm_pfns[i] |=3D cpu_flags; } + return 0; } =20 @@ -101,6 +145,7 @@ static int hmm_record_fault(unsigned long addr, unsigne= d long end, struct hmm_vma_walk *hmm_vma_walk =3D walk->private; =20 WARN_ON_ONCE(!required_fault); + HMM_ASSERT_UNLOCKED(hmm_vma_walk); hmm_vma_walk->last =3D addr; hmm_vma_walk->fault_end =3D end; hmm_vma_walk->required_fault =3D required_fault; @@ -185,11 +230,16 @@ static int hmm_vma_walk_hole(unsigned long addr, unsi= gned long end, if (!walk->vma) { if (required_fault) return -EFAULT; - return hmm_pfns_fill(addr, end, range, HMM_PFN_ERROR); + return hmm_pfns_fill(addr, end, hmm_vma_walk, HMM_PFN_ERROR); } - if (required_fault) + if (required_fault) { + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } return hmm_record_fault(addr, end, required_fault, walk); - return hmm_pfns_fill(addr, end, range, 0); + } + return hmm_pfns_fill(addr, end, hmm_vma_walk, 0); } =20 static inline unsigned long hmm_pfn_flags_order(unsigned long order) @@ -222,8 +272,13 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, un= signed long addr, cpu_flags =3D pmd_to_hmm_pfn_flags(range, pmd); required_fault =3D hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, cpu_flags); - if (required_fault) + if (required_fault) { + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } return hmm_record_fault(addr, end, required_fault, walk); + } =20 pfn =3D pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); for (i =3D 0; addr < end; addr +=3D PAGE_SIZE, i++, pfn++) { @@ -303,14 +358,25 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, u= nsigned long addr, goto fault; =20 if (softleaf_is_migration(entry)) { - pte_unmap(ptep); - hmm_vma_walk->last =3D addr; - migration_entry_wait(walk->mm, pmdp, addr); - return -EBUSY; + if (!hmm_select_migrate(range)) { + HMM_ASSERT_UNLOCKED(hmm_vma_walk); + pte_unmap(ptep); + hmm_vma_walk->last =3D addr; + migration_entry_wait(walk->mm, pmdp, addr); + return -EBUSY; + } + return 0; } =20 /* Report error for everything else */ - pte_unmap(ptep); + + if (hmm_vma_walk->ptelocked) { + pte_unmap_unlock(ptep, hmm_vma_walk->ptl); + hmm_vma_walk->ptelocked =3D false; + } else { + pte_unmap(ptep); + } + return -EFAULT; } =20 @@ -327,7 +393,13 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, un= signed long addr, if (!vm_normal_page(walk->vma, addr, pte) && !is_zero_pfn(pte_pfn(pte))) { if (hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0)) { - pte_unmap(ptep); + if (hmm_vma_walk->ptelocked) { + pte_unmap_unlock(ptep, hmm_vma_walk->ptl); + hmm_vma_walk->ptelocked =3D false; + } else { + pte_unmap(ptep); + } + return -EFAULT; } new_pfn_flags =3D HMM_PFN_ERROR; @@ -340,7 +412,12 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, un= signed long addr, return 0; =20 fault: - pte_unmap(ptep); + if (hmm_vma_walk->ptelocked) { + pte_unmap_unlock(ptep, hmm_vma_walk->ptl); + hmm_vma_walk->ptelocked =3D false; + } else { + pte_unmap(ptep); + } /* Fault any virtual address we were asked to fault */ return hmm_record_fault(addr, end, required_fault, walk); } @@ -384,13 +461,18 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *= walk, unsigned long start, required_fault =3D hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0); if (required_fault) { - if (softleaf_is_device_private(entry)) + if (softleaf_is_device_private(entry)) { + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } return hmm_record_fault(addr, end, required_fault, walk); + } else return -EFAULT; } =20 - return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR); + return hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR); } #else static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long s= tart, @@ -398,15 +480,72 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *= walk, unsigned long start, pmd_t pmd) { struct hmm_vma_walk *hmm_vma_walk =3D walk->private; - struct hmm_range *range =3D hmm_vma_walk->range; unsigned long npages =3D (end - start) >> PAGE_SHIFT; =20 if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0)) return -EFAULT; - return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR); + return hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR); } #endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */ =20 +#ifdef CONFIG_DEVICE_MIGRATION +static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk, + pmd_t *pmdp, + unsigned long start, + unsigned long end, + unsigned long *hmm_pfn) +{ + // TODO: implement migration entry insertion + return 0; +} + +static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk, + pmd_t *pmdp, + pte_t *ptep, + unsigned long addr, + unsigned long *hmm_pfn, + bool *unmapped) +{ + // TODO: implement migration entry insertion + return 0; +} + +static int hmm_vma_walk_split(pmd_t *pmdp, + unsigned long addr, + struct mm_walk *walk) +{ + // TODO : implement split + return 0; +} + +#else +static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk, + pmd_t *pmdp, + unsigned long start, + unsigned long end, + unsigned long *hmm_pfn) +{ + return 0; +} + +static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk, + pmd_t *pmdp, + pte_t *ptep, + unsigned long addr, + unsigned long *hmm_pfn, + bool *unmapped) +{ + return 0; +} + +static int hmm_vma_walk_split(pmd_t *pmdp, + unsigned long addr, + struct mm_walk *walk) +{ + return 0; +} +#endif + static int hmm_vma_capture_migrate_range(unsigned long start, unsigned long end, struct mm_walk *walk) @@ -465,46 +604,128 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, { struct hmm_vma_walk *hmm_vma_walk =3D walk->private; struct hmm_range *range =3D hmm_vma_walk->range; - unsigned long *hmm_pfns =3D - &range->hmm_pfns[(start - range->start) >> PAGE_SHIFT]; unsigned long npages =3D (end - start) >> PAGE_SHIFT; + struct mm_struct *mm =3D walk->vma->vm_mm; + unsigned long *hmm_pfns, *hmm_pfns_start; + enum migrate_vma_info minfo; unsigned long addr =3D start; + bool unmapped =3D false; + unsigned long i; pte_t *ptep; pmd_t pmd; + int r =3D 0; =20 + minfo =3D hmm_select_migrate(range); + hmm_pfns_start =3D &range->hmm_pfns[(start - range->start) >> PAGE_SHIFT]; again: - pmd =3D pmdp_get_lockless(pmdp); - if (pmd_none(pmd)) - return hmm_vma_walk_hole(start, end, -1, walk); + hmm_pfns =3D &range->hmm_pfns[(addr - range->start) >> PAGE_SHIFT]; + hmm_vma_walk->ptelocked =3D false; + hmm_vma_walk->pmdlocked =3D false; + + if (minfo) { + hmm_vma_walk->ptl =3D pmd_lock(mm, pmdp); + hmm_vma_walk->pmdlocked =3D true; + pmd =3D pmdp_get(pmdp); + } else + pmd =3D pmdp_get_lockless(pmdp); + + if (pmd_none(pmd)) { + r =3D hmm_vma_walk_hole(start, end, -1, walk); + + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } + return r; + } =20 if (thp_migration_supported() && pmd_is_migration_entry(pmd)) { - if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0)) { - hmm_vma_walk->last =3D addr; - pmd_migration_entry_wait(walk->mm, pmdp); - return -EBUSY; + if (!minfo) { + if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns_start, npages, 0)) { + hmm_vma_walk->last =3D addr; + pmd_migration_entry_wait(walk->mm, pmdp); + return -EBUSY; + } + for (i =3D 0; start < end; start +=3D PAGE_SIZE, i++) + hmm_pfns_start[i] &=3D HMM_PFN_INOUT_FLAGS; + } + + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; } - return hmm_pfns_fill(start, end, range, 0); + + return 0; } =20 - if (!pmd_present(pmd)) - return hmm_vma_handle_absent_pmd(walk, start, end, hmm_pfns, - pmd); + if (pmd_trans_huge(pmd) || !pmd_present(pmd)) { + if (!pmd_present(pmd)) { + r =3D hmm_vma_handle_absent_pmd(walk, start, end, hmm_pfns_start, + pmd); + // If not migrating we are done + if (r || !minfo) { + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } + return r; + } + } =20 - if (pmd_trans_huge(pmd)) { - /* - * No need to take pmd_lock here, even if some other thread - * is splitting the huge pmd we will get that event through - * mmu_notifier callback. - * - * So just read pmd value and check again it's a transparent - * huge or device mapping one and compute corresponding pfn - * values. - */ - pmd =3D pmdp_get_lockless(pmdp); - if (!pmd_trans_huge(pmd)) - goto again; + if (pmd_trans_huge(pmd)) { + /* + * No need to take pmd_lock here if not migrating, + * even if some other thread is splitting the huge + * pmd we will get that event through mmu_notifier callback. + * + * So just read pmd value and check again it's a transparent + * huge or device mapping one and compute corresponding pfn + * values. + */ + + if (!minfo) { + pmd =3D pmdp_get_lockless(pmdp); + if (!pmd_trans_huge(pmd)) + goto again; + } + + r =3D hmm_vma_handle_pmd(walk, start, end, hmm_pfns_start, pmd); =20 - return hmm_vma_handle_pmd(walk, addr, end, hmm_pfns, pmd); + // If not migrating we are done + if (r || !minfo) { + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } + return r; + } + } + + r =3D hmm_vma_handle_migrate_prepare_pmd(walk, pmdp, start, end, hmm_pfn= s_start); + + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; + } + + if (r =3D=3D -ENOENT) { + r =3D hmm_vma_walk_split(pmdp, addr, walk); + if (r) { + /* Split not successful, skip */ + return hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR); + } + + /* Split successful, reloop */ + hmm_vma_walk->last =3D addr; + return -EBUSY; + } + return r; + + } + + if (hmm_vma_walk->pmdlocked) { + spin_unlock(hmm_vma_walk->ptl); + hmm_vma_walk->pmdlocked =3D false; } =20 /* @@ -514,24 +735,62 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, * recover. */ if (pmd_bad(pmd)) { - if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0)) + if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns_start, npages, 0)) return -EFAULT; - return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR); + return hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR); } =20 - ptep =3D pte_offset_map(pmdp, addr); - if (!ptep) + if (minfo) { + ptep =3D pte_offset_map_lock(mm, pmdp, addr, &hmm_vma_walk->ptl); + if (ptep) + hmm_vma_walk->ptelocked =3D true; + } else { + ptep =3D pte_offset_map(pmdp, addr); + } + if (!ptep) { + addr =3D start; goto again; + } + for (; addr < end; addr +=3D PAGE_SIZE, ptep++, hmm_pfns++) { - int r; =20 r =3D hmm_vma_handle_pte(walk, addr, end, pmdp, ptep, hmm_pfns); if (r) { - /* hmm_vma_handle_pte() did pte_unmap() */ + /* hmm_vma_handle_pte() did pte_unmap() / pte_unmap_unlock */ return r; } + + r =3D hmm_vma_handle_migrate_prepare(walk, pmdp, ptep, addr, hmm_pfns, &= unmapped); + if (r =3D=3D -EAGAIN) { + HMM_ASSERT_UNLOCKED(hmm_vma_walk); + if (unmapped) { + flush_tlb_range(walk->vma, start, addr); + unmapped =3D false; + } + goto again; + } + if (r) { + /* A non -EAGAIN error here means migrate_vma_split_folio() + * already dropped the PTE lock and cleared ptelocked. + */ + HMM_ASSERT_UNLOCKED(hmm_vma_walk); + if (unmapped) + flush_tlb_range(walk->vma, start, addr); + hmm_pfns_fill(addr, end, hmm_vma_walk, HMM_PFN_ERROR); + return 0; + } } - pte_unmap(ptep - 1); + + if (unmapped) + flush_tlb_range(walk->vma, start, addr); + + if (hmm_vma_walk->ptelocked) { + pte_unmap_unlock(ptep - 1, hmm_vma_walk->ptl); + hmm_vma_walk->ptelocked =3D false; + } else { + pte_unmap(ptep - 1); + } + return 0; } =20 @@ -679,7 +938,7 @@ static int hmm_vma_walk_test(unsigned long start, unsig= ned long end, (end - start) >> PAGE_SHIFT, 0)) return -EFAULT; =20 - hmm_pfns_fill(start, end, range, HMM_PFN_ERROR); + hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR); =20 /* Skip this vma and continue processing the next vma. */ return 1; --=20 2.55.0 From nobody Fri Sep 25 18:51:54 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BF81D48E0C6 for ; Tue, 22 Sep 2026 05:34:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055300; cv=none; b=fVb7J8Sm6+hAMshNFV7cCNqf1wxdyqFX3PZJ4YHboapv0SJIOrFc/MUbipu6sAoJgQ56T2JKE61N3wZQlQrektyad+4qf9C+CGdpw5MaHaU7rxs77FDdd1byrXVm3fbmaeb1Q/XW1vL/68PRcU/r0FWE5tAgtqQ9XfSROlXkEpw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055300; c=relaxed/simple; bh=/85IW+OvTho1athsusF8SuiwgPPvr04BfdSaZRV7Nfw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=VAMWhAvxkBI3jkFZwhEIIxp1O5spH8QoFEJH6BkwEsa9FSfG+7fUI8NUcDByiJio7iznqMBOpG5nBsb9MpsfgQKFuA70ti3lO9CcS2JNSSr0rh2xi4aSaqhEf7ZmIT6FSwNqoIV0JIwh4+4hHj6U5qSOPwp2kL92VLTflizQEoo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=dwiJnxNm; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=kIz9kijC; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="dwiJnxNm"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="kIz9kijC" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790055297; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Qu0BSxnttTs+uKLg/qx0GDVYgMyi2m2Up9fDq2tLMMQ=; b=dwiJnxNmaeeb5e74Yv2dL4KbncwDm25K1/OQuBU8KIMHr3AYIqPJt6oXI/5cAyohe++HKA iwXLSUuG6Y0lFsiNQcaERL3JU1oDrHNWJ6NU/v5OuanMYTOTDUfUVZJrL9nvocB/obFQEf o4YxVir+WKQcthFDRBrcT0bENZzzmdw= Received: from mail-lf1-f70.google.com (mail-lf1-f70.google.com [209.85.167.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-471-JnWxvv8oPyeMkCeXm1vxkA-1; Tue, 22 Sep 2026 01:34:55 -0400 X-MC-Unique: JnWxvv8oPyeMkCeXm1vxkA-1 X-Mimecast-MFC-AGG-ID: JnWxvv8oPyeMkCeXm1vxkA_1790055294 Received: by mail-lf1-f70.google.com with SMTP id 2adb3069b0e04-5b767214d60so283475e87.1 for ; Mon, 21 Sep 2026 22:34:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790055294; x=1790660094; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=Qu0BSxnttTs+uKLg/qx0GDVYgMyi2m2Up9fDq2tLMMQ=; b=kIz9kijCWaeGmtCpNPHg+wzGANdVwMLD0UwyjgxC54VVB/iLj1IsOW6RgUBZ7fAMhm W+tBRdvgTM+7n16TzbaNTdbm+tKYdZnkkb9OzyIvXZ5/Xckiht48so5zvJGsUNb+mtNI 8mj4ivfl6gPqRKx7npZAJnC+xAwLajC6zfpBs0hXIS8E2N0CXRB4lWqP5loExlrwQmsr 5zsDrluMJgQUm+Qks7+bmlhLEYfDRGPdJ2kpqWfs5aeVGMRjVSmO9ccyCWqYffymSbtJ qBPsYvU3xdDhz4aZJLCx1Wy9LX1oQWm7XY+qajvRCsbn6fQUV/aZbS05EPITDhvDZcjp +cRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790055294; x=1790660094; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=Qu0BSxnttTs+uKLg/qx0GDVYgMyi2m2Up9fDq2tLMMQ=; b=G0bQLu8cbqCxTPZJM3447UanZ4etcdUjACdAABh4TbA/jsuIYAtxsqEggxGPPnSfB5 SXvg/ut5xlN5NFIGU91Nv8lcmiIRuAd7xZYvY4d6UZm9WLpJ+wj1pSidSG+lBTOEJ5Nm MIWy5GUW8ZAJRg7T3g+5jWjq73phSVUrb4HRajmqfpjc8ksTPwCDw0uqm91/HbvY0YfC IEv1mMBNr+N1NpAxHbK410o3IVTPwZvSm/R5ieAsM95D0sijCiie5/jcw4uSS73b43Nk zEDRWNJ60hh1noRr03hqD/n2R2FfVD+fPV4o9oE+XTLMLU19sWwPUCWrjATTrWTgBVOg WO3w== X-Forwarded-Encrypted: i=1; AKwUvBwW0jemGeVU0rgIbUvE8DAiPQHwz+3xHHGXgqjHCDnnk/9qMif8dKFNqHr+2kqHgU6j1Ye+HdyzkDqN3Mk=@vger.kernel.org X-Gm-Message-State: AFuF++kCAqvQySFCaVvGJW19t3j6Q3XVaqavPfKlt7d6jCPcd98zYRod SgUz1HXqTSsd9tJCF2H43U82N3qm9YxWnWaHsB7sv+BaybzSSbAO9bf+/1Y6+AAFR2Mg6HAxj68 hZ8iHhmW1FtpbS/H16g4d3WEZGQJE3Wrvl6ZgzMRAXTn6LJ9o0Ft9u4R3/whgfjUa X-Gm-Gg: AYBFou07DnmyPntAm7Nnk51/Evfbi8tWowAs5u6dEeAo/I3rFCGbK6BKDSWIX4BV58M 6xGil5Yu20pT6cmbCvrZJNV7rznsQqP8cM1hlkbUWt3iAFgByqnZQc20AvDcz65gTJhntshXuQw XOcnwW2y06Vx6VNWaUggsO187QEomEexf3fvnudG6mZJAhonoE0ciaOR1HlxtnhWvemY1OImVJd DS7hu+0176EoVoMeqvjE3la6xf0yRQm9O86vzsC8G4EIXhZdTekQXPMmfo+CqcY+sFdiS9huv9X /iwj3SAnzja+9BxVFOWJyn4biO1rgYvGAVjyjIGe0e/WQ5Few0A2x2n4J3cfDLFzCfAR7Ur/HAJ 2DYqQhJd4UBJ150DeYlMI X-Received: by 2002:a05:6512:aca:b0:5ae:b7c7:5334 with SMTP id 2adb3069b0e04-5b8d28fbc51mr474012e87.17.1790055294082; Mon, 21 Sep 2026 22:34:54 -0700 (PDT) X-Received: by 2002:a05:6512:aca:b0:5ae:b7c7:5334 with SMTP id 2adb3069b0e04-5b8d28fbc51mr474009e87.17.1790055293609; Mon, 21 Sep 2026 22:34:53 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b8d46e0966sm191742e87.55.2026.09.21.22.34.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 22:34:52 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v14 05/12] mm/hmm: implement folio split for migrate needs in HMM pagewalk Date: Tue, 22 Sep 2026 08:34:14 +0300 Message-ID: <20260922053421.4092027-6-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922053421.4092027-1-mpenttil@redhat.com> References: <20260922053421.4092027-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 Implement the helper to split PMD size folios during pagewalk. Splitting is needed if the start and end addresses are not PMD aligned, or setting up pmd migration entry fails. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- mm/hmm.c | 47 ++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 44 insertions(+), 3 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index 4805c88347e6..e6469ef4ae6a 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -514,10 +514,51 @@ static int hmm_vma_walk_split(pmd_t *pmdp, unsigned long addr, struct mm_walk *walk) { - // TODO : implement split - return 0; -} + struct hmm_vma_walk *hmm_vma_walk =3D walk->private; + struct hmm_range *range =3D hmm_vma_walk->range; + struct migrate_vma *migrate =3D range->migrate; + struct folio *folio, *fault_folio; + spinlock_t *ptl; + int ret =3D 0; + + HMM_ASSERT_UNLOCKED(hmm_vma_walk); + + fault_folio =3D (migrate && migrate->fault_page) ? + page_folio(migrate->fault_page) : NULL; + + ptl =3D pmd_lock(walk->mm, pmdp); + if (unlikely(!pmd_trans_huge(*pmdp))) { + spin_unlock(ptl); + goto out; + } =20 + folio =3D pmd_folio(*pmdp); + if (is_huge_zero_folio(folio)) { + spin_unlock(ptl); + split_huge_pmd(walk->vma, pmdp, addr); + } else { + folio_get(folio); + spin_unlock(ptl); + + if (folio !=3D fault_folio) { + if (unlikely(!folio_trylock(folio))) { + folio_put(folio); + ret =3D -EBUSY; + goto out; + } + } else { + folio_put(folio); + } + + ret =3D split_folio(folio); + if (fault_folio !=3D folio) { + folio_unlock(folio); + folio_put(folio); + } + } +out: + return ret; +} #else static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk, pmd_t *pmdp, --=20 2.55.0 From nobody Fri Sep 25 18:51:54 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E55E048E0F2 for ; Tue, 22 Sep 2026 05:35:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055303; cv=none; b=HQeeIkY2fkok1Mehj4dFQzuApcH5cBUC5AfQv+qCKUYn8MoS8vNN/lx4nnVh7ZmkRbyTIqyYqxw6bePdbpGGvaoUkLgN8gi1Bo3sbFkDmSnQ54HKuykBzPQuQF1LagkKGN0P1yI4k11wolrNAWLMticrh5Xaa1CADjpr7SoZEiM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055303; c=relaxed/simple; bh=/bavorzbX6fdsAS1auZYVgDQ9isR87ThoKZFo3OyPWw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ZUpAQeQ2EN1gENrInRl1YpYctKXNbILk08gim0b2AR/IgYT0TDjzYU9ue4JWnmVrPSwW5KRwHQB7VZNaWD6BO6W44B860KPvSmjQN8MXtO11c03gwTGzpayL7eDXZobU4fw5DUvsf4ipze3EWmYW2nN2AGuKN/c5jw2xnUII824= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Zb9Az9dY; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=gODo+R0A; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Zb9Az9dY"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="gODo+R0A" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790055300; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OVe9F9UnqicBMpqnz2rAf3G6pQga7qmnVnonC4zTIr8=; b=Zb9Az9dY3swZEZQ/31UW2k5RBfNs56Dlas7yzhSDLs30jWdNOfeQDCtm32UtsEoQj+q1ms qKTlH1+baoO75yr9pCzdKNayC0qFOZgBgyDw/yHjEjPAiCBhrHJx1CYJKgKXABraunIQ9e sO+yslki8c3Qc9p+NvXJaEjLwF1RpHw= Received: from mail-lf1-f69.google.com (mail-lf1-f69.google.com [209.85.167.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-660-EQA0pumqOOaPSmIR5ALlbQ-1; Tue, 22 Sep 2026 01:34:59 -0400 X-MC-Unique: EQA0pumqOOaPSmIR5ALlbQ-1 X-Mimecast-MFC-AGG-ID: EQA0pumqOOaPSmIR5ALlbQ_1790055298 Received: by mail-lf1-f69.google.com with SMTP id 2adb3069b0e04-5b667b38b04so1993639e87.2 for ; Mon, 21 Sep 2026 22:34:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790055297; x=1790660097; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=OVe9F9UnqicBMpqnz2rAf3G6pQga7qmnVnonC4zTIr8=; b=gODo+R0AQXrBI8e0ZFq/n4JuC+i92NOwD/co52yPAu1WhfuwGok4rZjqdt+bQqAZ+A s2m3fregqRkikh7mVPP+RJN9ydqtxfro420Rc7/lI5ZfSn41oibwV3677hsEg57HP7SN un6zN2Mcljhc0PSFyenWTKdsSPBH/qt56CPiAlbIapXta4saDMJ0IBh0CMCE3mFZtcKJ zDNiCmtZ+N198dkst3RpNWUo1uwQ4S6CWpEFu10VKn3+2/RVO2er/G8yamYkmhVbQBSb PxLE+fbgTbHeLa8MwElUyR0/m/WTsnEX22jgTCH5cK4aBzZnUUmI463zeVe1moY+AUn8 XSxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790055297; x=1790660097; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=OVe9F9UnqicBMpqnz2rAf3G6pQga7qmnVnonC4zTIr8=; b=qtyLTEJaJafIvHOrBKovN3rfGvltCOUigFkwEY0sIv+FKuvt9C79CCAvE8ArGifl1R SAckYi8Js2ut3DOOVOIxUPHom5LdoD2sIMaU8r2jctYwgu1MAn/qT8pliFTy0ckPk+/q 2bvrqb0JblM/0dLDlj6i1XIZqUveP5iQAPSaPymtCo541VgoFkTYS+kDeicRRHi2yI0s CkWtoJ9e0YsNKnlhSYhQRGz4q3ejCaF+XmMXtuSU0ElngMPxXm4aym7lasyWKnAUNmNM 86hPF1x9rCZTds9D4xQcFRp9barinjQ2neUgf3Fh6RK6ZvKX7Bwy/xsqO5ZRAXYKDu6e Jnyw== X-Forwarded-Encrypted: i=1; AKwUvBxdyp4DsllJS4fo6XHHKRmbLTOYx9nIVFLFO51bFnSx5Fqk9LM7RaTe04a4bJLntJF+LLLaFKlVgzj6ZxU=@vger.kernel.org X-Gm-Message-State: AFuF++mYnvaF8cS8r5t3uteRWuyfBmvR0+SDifioiO3kpEMnW680pOAu KIzGuXOJPslaQJiRy6lfWE4WbHJwnG8E8eq/vtHW9ys8qBO1qD04r6iHdPc/OkVsCZpqlrbUoZQ yhYeqPu3p6Khz5PHQbQE+hBRaQPA3EchTWHl1Z7maV/Win5KOP1chxZrlI+SzSsBr X-Gm-Gg: AYBFou1jiSqK1mM6fu5sKK+OVuOoXhHtPzzbWpeTpi5ys6zTe7m16ML99bm61ye5/CW A96Jkuuxbek3wnLc171PyzPU+33l2MlJ/kzSwm4tM3Itva6IRs6O16ZBlhnTpO4yvEcXzfXaexb 06SX1gRxX2YgurNxHnQ+8/Sx/WIrLCcsqC0mUF6Byjl5ozondVxVbXMJ0zvE3vGFd8bOI9mdKQ/ H7sItVAJ2rbaZJplmvXrRSMXwnLYwFXVJp3i4wEJ1gA2n3WJI/INgtITN6R9AYbZT755QPPt6jC KR0CFKUJ/H0zMKVtpjWDhqqkHYtbKZQu6aZqnK3Tq9KAWCn1uqT7pOSX8J0AzwHxhcqUc+3NGiL xWcP2mzVNm7B9yIhlt494 X-Received: by 2002:a05:6512:12cc:b0:5b6:183c:5c9e with SMTP id 2adb3069b0e04-5b8c1959154mr3783083e87.60.1790055297566; Mon, 21 Sep 2026 22:34:57 -0700 (PDT) X-Received: by 2002:a05:6512:12cc:b0:5b6:183c:5c9e with SMTP id 2adb3069b0e04-5b8c1959154mr3783074e87.60.1790055297099; Mon, 21 Sep 2026 22:34:57 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b8d46e0966sm191742e87.55.2026.09.21.22.34.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 22:34:55 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v14 06/12] mm/hmm: migrate collection in HMM pagewalk - pte level Date: Tue, 22 Sep 2026 08:34:15 +0300 Message-ID: <20260922053421.4092027-7-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922053421.4092027-1-mpenttil@redhat.com> References: <20260922053421.4092027-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 Implement the needed hmm_vma_handle_migrate_prepare() function which is mostly carried over from migrate_device.c's migrate_vma_collect_pmd() function. Also implement the migrate_vma_split_folio(), for splitting pte mapped large folios. It is also mostly from migrate_device.c, with care taken to reference folio before relasing page table lock. With HMM pagewalk based migration, the idea is that hmm_vma_handle_*() are responsible for faulting, and the pfn collecting part. hmm_vma_handle_migrate_prepare*() do the migration decisions (with HMM_PFN_MIGRATE), possibly split folios, and insert migration ptes/pmds. HMM pagewalk based migration is enabled in later commit, for now now hmm_select_migrate() returns 0. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- mm/hmm.c | 262 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 261 insertions(+), 1 deletion(-) diff --git a/mm/hmm.c b/mm/hmm.c index e6469ef4ae6a..b3b79d13c797 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -489,6 +489,63 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *w= alk, unsigned long start, #endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */ =20 #ifdef CONFIG_DEVICE_MIGRATION +/** + * migrate_vma_split_folio() - Helper function to split a THP folio + * @folio: the folio to split + * @fault_page: struct page associated with the fault if any + * @hmm_vma_walk: walk in progress + * @ptep: pte_t * for unmap and unlock ptl + * + * Returns 0 on success + */ +static int migrate_vma_split_folio(struct folio *folio, + struct page *fault_page, + struct hmm_vma_walk *hmm_vma_walk, + pte_t *ptep) +{ + int ret; + struct folio *fault_folio =3D fault_page ? page_folio(fault_page) : NULL; + struct folio *new_fault_folio =3D NULL; + + if (folio !=3D fault_folio) + folio_get(folio); + + pte_unmap_unlock(ptep, hmm_vma_walk->ptl); + hmm_vma_walk->ptelocked =3D false; + + if (folio !=3D fault_folio) + folio_lock(folio); + + ret =3D split_folio(folio); + if (ret) { + if (folio !=3D fault_folio) { + folio_unlock(folio); + folio_put(folio); + } + return ret; + } + + new_fault_folio =3D fault_page ? page_folio(fault_page) : NULL; + + /* + * Ensure the lock is held on the correct + * folio after the split + */ + if (!new_fault_folio) { + folio_unlock(folio); + folio_put(folio); + } else if (folio !=3D new_fault_folio) { + if (new_fault_folio !=3D fault_folio) { + folio_get(new_fault_folio); + folio_lock(new_fault_folio); + } + folio_unlock(folio); + folio_put(folio); + } + + return 0; +} + static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk, pmd_t *pmdp, unsigned long start, @@ -499,6 +556,11 @@ static int hmm_vma_handle_migrate_prepare_pmd(const st= ruct mm_walk *walk, return 0; } =20 +/* + * Install migration entries if migration requested, either from fault + * or migrate paths. + * + */ static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk, pmd_t *pmdp, pte_t *ptep, @@ -506,8 +568,206 @@ static int hmm_vma_handle_migrate_prepare(const struc= t mm_walk *walk, unsigned long *hmm_pfn, bool *unmapped) { - // TODO: implement migration entry insertion + struct hmm_vma_walk *hmm_vma_walk =3D walk->private; + struct hmm_range *range =3D hmm_vma_walk->range; + struct migrate_vma *migrate =3D range->migrate; + struct mm_struct *mm =3D walk->vma->vm_mm; + struct folio *fault_folio =3D NULL; + enum migrate_vma_info minfo; + struct dev_pagemap *pgmap; + bool anon_exclusive; + struct folio *folio; + unsigned long pfn; + struct page *page; + softleaf_t entry; + pte_t pte, swp_pte; + bool writable =3D false; + + // Do we want to migrate at all? + minfo =3D hmm_select_migrate(range); + if (!minfo) + return 0; + + WARN_ON_ONCE(!migrate); + HMM_ASSERT_PTE_LOCKED(hmm_vma_walk, true); + + fault_folio =3D migrate->fault_page ? + page_folio(migrate->fault_page) : NULL; + + pte =3D ptep_get(ptep); + + if (pte_none(pte)) { + if (vma_is_anonymous(walk->vma)) { + *hmm_pfn &=3D HMM_PFN_INOUT_FLAGS; + *hmm_pfn |=3D HMM_PFN_MIGRATE; + goto out; + } + } + + if (!(hmm_pfn[0] & HMM_PFN_VALID)) + goto out; + + if (!pte_present(pte)) { + /* + * Only care about unaddressable device page special + * page table entry. Other special swap entries are not + * migratable, and we ignore regular swapped page. + */ + entry =3D softleaf_from_pte(pte); + if (!softleaf_is_device_private(entry)) + goto out; + + if (!(minfo & MIGRATE_VMA_SELECT_DEVICE_PRIVATE)) + goto out; + + page =3D softleaf_to_page(entry); + folio =3D page_folio(page); + if (folio->pgmap->owner !=3D migrate->pgmap_owner) + goto out; + + if (folio_test_large(folio)) { + int ret; + + ret =3D migrate_vma_split_folio(folio, + migrate->fault_page, + hmm_vma_walk, + ptep); + if (ret) + goto out_error; + return -EAGAIN; + } + + pfn =3D page_to_pfn(page); + if (softleaf_is_device_private_write(entry)) + writable =3D true; + } else { + pfn =3D pte_pfn(pte); + if (is_zero_pfn(pfn) && + (minfo & MIGRATE_VMA_SELECT_SYSTEM)) { + *hmm_pfn =3D HMM_PFN_MIGRATE; + goto out; + } + page =3D vm_normal_page(walk->vma, addr, pte); + if (page && !is_zone_device_page(page) && + !(minfo & MIGRATE_VMA_SELECT_SYSTEM)) { + goto out; + } else if (page && is_device_coherent_page(page)) { + pgmap =3D page_pgmap(page); + + if (!(minfo & + MIGRATE_VMA_SELECT_DEVICE_COHERENT) || + pgmap->owner !=3D migrate->pgmap_owner) + goto out; + } + + folio =3D page ? page_folio(page) : NULL; + if (folio && folio_test_large(folio)) { + int ret; + + ret =3D migrate_vma_split_folio(folio, + migrate->fault_page, + hmm_vma_walk, + ptep); + if (ret) + goto out_error; + return -EAGAIN; + } + + writable =3D pte_write(pte); + } + + if (!page || !page->mapping) + goto out; + + /* + * By getting a reference on the folio we pin it and that blocks + * any kind of migration. Side effect is that it "freezes" the + * pte. + * + * We drop this reference after isolating the folio from the lru + * for non device folio (device folio are not on the lru and thus + * can't be dropped from it). + */ + folio =3D page_folio(page); + folio_get(folio); + + /* + * We rely on folio_trylock() to avoid deadlock between + * concurrent migrations where each is waiting on the others + * folio lock. If we can't immediately lock the folio we fail this + * migration as it is only best effort anyway. + * + * If we can lock the folio it's safe to set up a migration entry + * now. In the common case where the folio is mapped once in a + * single process setting up the migration entry now is an + * optimisation to avoid walking the rmap later with + * try_to_migrate(). + */ + + if (fault_folio =3D=3D folio || folio_trylock(folio)) { + anon_exclusive =3D folio_test_anon(folio) && + PageAnonExclusive(page); + + if (pte_present(pte)) + flush_cache_page(walk->vma, addr, pfn); + + if (anon_exclusive) { + pte =3D ptep_clear_flush(walk->vma, addr, ptep); + + if (folio_try_share_anon_rmap_pte(folio, page)) { + set_pte_at(mm, addr, ptep, pte); + folio_unlock(folio); + folio_put(folio); + goto out; + } + } else { + pte =3D ptep_get_and_clear(mm, addr, ptep); + } + + if (pte_present(pte) && pte_dirty(pte)) + folio_mark_dirty(folio); + + /* Setup special migration page table entry */ + if (writable) + entry =3D make_writable_migration_entry(pfn); + else if (anon_exclusive) + entry =3D make_readable_exclusive_migration_entry(pfn); + else + entry =3D make_readable_migration_entry(pfn); + + if (pte_present(pte)) { + if (pte_young(pte)) + entry =3D make_migration_entry_young(entry); + if (pte_dirty(pte)) + entry =3D make_migration_entry_dirty(entry); + } + + swp_pte =3D swp_entry_to_pte(entry); + if (pte_present(pte)) { + if (pte_soft_dirty(pte)) + swp_pte =3D pte_swp_mksoft_dirty(swp_pte); + if (pte_uffd(pte)) + swp_pte =3D pte_swp_mkuffd(swp_pte); + } else { + if (pte_swp_soft_dirty(pte)) + swp_pte =3D pte_swp_mksoft_dirty(swp_pte); + if (pte_swp_uffd(pte)) + swp_pte =3D pte_swp_mkuffd(swp_pte); + } + + set_pte_at(mm, addr, ptep, swp_pte); + folio_remove_rmap_pte(folio, page, walk->vma); + folio_put(folio); + *hmm_pfn |=3D HMM_PFN_MIGRATE; + if (pte_present(pte)) + *unmapped =3D true; + } else { + folio_put(folio); + } +out: return 0; +out_error: + return -EFAULT; } =20 static int hmm_vma_walk_split(pmd_t *pmdp, --=20 2.55.0 From nobody Fri Sep 25 18:51:54 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8650E48E0C6 for ; Tue, 22 Sep 2026 05:35:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055306; cv=none; b=sPm4y+niODITP2YxGPgExY5LM8GFSm/5UctSeduPeFYt5HDzxi43+W2qCLWBExAy76XAw6D3CeQ7ZcHiHps/eep34QiwJrQOtGeOOtL8lpyRUXWdomrN1DCCgr+a7gE+9o6/PEldb2HBOewLb03oRLK6uWVyYV2FJyNkjNVzeKM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055306; c=relaxed/simple; bh=0FXsJV1z79oMxRtSOr/aKFhpIItT9GM5SpOVBZoF7BI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=SMwQqnBjUfvXLiz3ruiBZjtuTX4ZTh0tBq3p4iKPo8XWZ0lA/mpXQY7PXG9HnHJ0curLJuzTIT1h7X9GAMEwt3xH4pA3DmjJS/CG3XZ39mp/ryWjKAgFP8IbHBupZuH+cSjxbFxfzvmq5MdTRJEuYelrj6quevXuUzOOEJ8gHis= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=ccXPWX4u; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=HmuPoGEw; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ccXPWX4u"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="HmuPoGEw" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790055303; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4Dwb+61ZiIhZrjnlVPFK1XZrDSM5l5u6qowPzm9WLF0=; b=ccXPWX4uvzF6Wa1aba6tsfuJuT0R0Yz+PeoDGBs9Yd3nMnEJWn5g9gXx8wtoK5pLaMzs8C IgidwsNpXmtHeNSh3iTId1THLd4OxgJyoL6vzyx0wQoQoUhXoZ/4t7Rny4shBfBWgX8U4l 4JhrROQypMpzi4FqbG98+fN2Pq422jI= Received: from mail-lf1-f72.google.com (mail-lf1-f72.google.com [209.85.167.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-607-r9iqtV6BNiKqDPnvXmS_EQ-1; Tue, 22 Sep 2026 01:35:02 -0400 X-MC-Unique: r9iqtV6BNiKqDPnvXmS_EQ-1 X-Mimecast-MFC-AGG-ID: r9iqtV6BNiKqDPnvXmS_EQ_1790055300 Received: by mail-lf1-f72.google.com with SMTP id 2adb3069b0e04-5b8c58d7635so1656862e87.1 for ; Mon, 21 Sep 2026 22:35:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790055300; x=1790660100; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=4Dwb+61ZiIhZrjnlVPFK1XZrDSM5l5u6qowPzm9WLF0=; b=HmuPoGEwiPGJ2FRmRWT6BPNruvUycnktFWlmHsR0Dg9urdWrZF++hQvYqP02boMIDR O4+3wOTZfs/2V7c/tWede93905W3lE/o3QSQzP95s1Sl85wL+PGi78t4/RtooWHHvGP9 YpFsAG8Lt9IfnA4bUT4K9ZuKi/7ISkhuj+L4/Z6bcMgqNfPTeh0S2XI4CbigvNfjemSH BItJnEQe6LVjwabT6Bf75yHo9xMHyADMKWXuaxPxzm4PIGWtjUmKXjIH5/eJdLnLfBbW 7kUDS66NvJohbO15z6EQ1PdNobosAZJdAlJ47JXtEQMMviI4bHzxenPRgE7UC0mitIoD ErNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790055300; x=1790660100; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=4Dwb+61ZiIhZrjnlVPFK1XZrDSM5l5u6qowPzm9WLF0=; b=RNnubjS3fwSOPu+ajdYgwz3MLrlvOtvOC0yyuituztXCHvh5fZ2ksRmaMikABmIRxu bXMe5usH6v2ByaaESo2Md2yq89kex+u0he8umNWw39vWjlJJr/P5SoPl43qYlugjDHG9 ZYkr2T4UVvNsFPuqKff6C5M6ZgW+rJyfZdW+PVULaqRWoRWnDlxWEx4voQVVr1OdreZl G99/Su9yd+v5Hxk7hxXCm7SbHMyT2UikA142ESZy1ujFtNmVaseRyeV8JlxdB6d++Gjw Qdm2cD3eTS2z/GNpF31bl2CoxC2vjhLmNNM6NiyUbeqpoWMWGpkUDlL/B3INfx50Av0M RkyA== X-Forwarded-Encrypted: i=1; AKwUvBx8/+bGGrhflKz1U/7lW5u+B3zMfKPWOuxMansxWyeJXhD4TpHdvOtymE+UIegJ3RaBQ/IUZh15PjnlGQs=@vger.kernel.org X-Gm-Message-State: AFuF++nFuBQx4VlBcxFMfmxGrX6yzcERzPzGBra9vMgi4hlM3v8e5EsI XDboSFWnSzDGf6huK1q00U9m+dgUPh3OlBmneo0Rvqdi6zdFU7asgXM0gMD1oF8XXID0jbcWscZ CT4dBswHZjVvtry7IUVUSOOEt5tgCffeuXVrUFhCVW27sneB3+I+8hyUEkBn+DaGI X-Gm-Gg: AYBFou3yq3LC23AXbxVFiCY9JuIaXbJjd34M3cHNRiQuW1WY6D4hnZDWSAO7TY1+oBL 3EbVTlzr4JSI9qJYfzR0MNhb2LwHEOREI2P2zockeDN/2bGwU1O5vIIVvoIorqCBzZHcmBbfuvM PUGF7955Ob4FYqGFO4q3lO3BEx9wa8B2uNEGohtRjDA3XOyt0GlFpLiqS2qQKu3MJOapwIzn/mz ulnqkp27WisGH/GR97Sqqyi6Ynq1ag1PJzA9B7pfq+Ftlwwrae1EoEn7aEjgG/VSSeRpbkJozuh JrCVHB9hzJzwtc9zdkFqZ9Hl2sKl+G7vodjY8/R0LgChzlSQ93H6a5CkC6nZOX3Em/IoBqrLf4w yAm4sGnsV02dsVMuXxlLN X-Received: by 2002:a05:6512:3a85:b0:5b6:186e:b79d with SMTP id 2adb3069b0e04-5b8c194df51mr4033236e87.57.1790055300381; Mon, 21 Sep 2026 22:35:00 -0700 (PDT) X-Received: by 2002:a05:6512:3a85:b0:5b6:186e:b79d with SMTP id 2adb3069b0e04-5b8c194df51mr4033212e87.57.1790055299969; Mon, 21 Sep 2026 22:34:59 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b8d46e0966sm191742e87.55.2026.09.21.22.34.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 22:34:58 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v14 07/12] mm/hmm: migrate collection in HMM pagewalk - pmd level Date: Tue, 22 Sep 2026 08:34:16 +0300 Message-ID: <20260922053421.4092027-8-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922053421.4092027-1-mpenttil@redhat.com> References: <20260922053421.4092027-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 Implement the needed hmm_vma_handle_migrate_prepare_pmd() function which is mostly carried over from migrate_device.c's migrate_vma_collect_huge_pmd() function. With HMM pagewalk based migration, the idea is that hmm_vma_handle_*() are responsible for faulting, and the pfn collecting part. hmm_vma_handle_migrate_prepare*() do the migration decisions (with HMM_PFN_MIGRATE), possibly split folios, and insert migration ptes/pmds. HMM pagewalk based migration is enabled in later commit, for now now hmm_select_migrate() returns 0. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- mm/hmm.c | 96 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 94 insertions(+), 2 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index b3b79d13c797..a1459bd34c04 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -552,8 +552,100 @@ static int hmm_vma_handle_migrate_prepare_pmd(const s= truct mm_walk *walk, unsigned long end, unsigned long *hmm_pfn) { - // TODO: implement migration entry insertion - return 0; + struct hmm_vma_walk *hmm_vma_walk =3D walk->private; + struct hmm_range *range =3D hmm_vma_walk->range; + struct migrate_vma *migrate =3D range->migrate; + struct folio *fault_folio =3D NULL; + enum migrate_vma_info minfo; + struct folio *folio; + unsigned long i; + int r =3D 0; + + // Do we want to migrate at all? + minfo =3D hmm_select_migrate(range); + if (!minfo) + return r; + + WARN_ON_ONCE(!migrate); + HMM_ASSERT_PMD_LOCKED(hmm_vma_walk, true); + + fault_folio =3D migrate->fault_page ? + page_folio(migrate->fault_page) : NULL; + + if (pmd_none(*pmdp)) + return hmm_pfns_fill(start, end, hmm_vma_walk, 0); + + if (!(hmm_pfn[0] & HMM_PFN_VALID)) + goto out; + + if (pmd_trans_huge(*pmdp)) { + if (!(minfo & MIGRATE_VMA_SELECT_SYSTEM)) + goto out; + + folio =3D pmd_folio(*pmdp); + if (is_huge_zero_folio(folio)) + return hmm_pfns_fill(start, end, hmm_vma_walk, 0); + + } else if (!pmd_present(*pmdp)) { + const softleaf_t entry =3D softleaf_from_pmd(*pmdp); + + if (!softleaf_is_device_private(entry)) + goto out; + + if (!(minfo & MIGRATE_VMA_SELECT_DEVICE_PRIVATE)) + goto out; + + folio =3D softleaf_to_folio(entry); + if (folio->pgmap->owner !=3D migrate->pgmap_owner) + goto out; + } else { + hmm_vma_walk->last =3D start; + return -EBUSY; + } + + folio_get(folio); + + if (folio !=3D fault_folio && unlikely(!folio_trylock(folio))) { + folio_put(folio); + hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR); + return 0; + } + + if (thp_migration_supported() && + (migrate->flags & MIGRATE_VMA_SELECT_COMPOUND) && + (IS_ALIGNED(start, HPAGE_PMD_SIZE) && + IS_ALIGNED(end, HPAGE_PMD_SIZE))) { + struct page_vma_mapped_walk pvmw =3D { + .ptl =3D hmm_vma_walk->ptl, + .address =3D start, + .pmd =3D pmdp, + .vma =3D walk->vma, + }; + + hmm_pfn[0] |=3D HMM_PFN_MIGRATE | HMM_PFN_COMPOUND; + + r =3D set_pmd_migration_entry(&pvmw, folio_page(folio, 0)); + if (r) { + hmm_pfn[0] &=3D ~(HMM_PFN_MIGRATE | HMM_PFN_COMPOUND); + r =3D -ENOENT; // fallback + goto unlock_out; + } + for (i =3D 1, start +=3D PAGE_SIZE; start < end; start +=3D PAGE_SIZE, i= ++) + hmm_pfn[i] &=3D HMM_PFN_INOUT_FLAGS; + + } else { + r =3D -ENOENT; // fallback + goto unlock_out; + } + +out: + return r; + +unlock_out: + if (folio !=3D fault_folio) + folio_unlock(folio); + folio_put(folio); + goto out; } =20 /* --=20 2.55.0 From nobody Fri Sep 25 18:51:54 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7E46248EBE4 for ; Tue, 22 Sep 2026 05:35:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055309; cv=none; b=FY7vnnnBpUi6vzOXdAcpBnxsLuAQbZlHmVjpbJa31cRvpBTCIfq7lXjcKOwlsRMn+aFQN14A0mryrL49rdwwNiIpsZEmH/J1CAJZatp7GjwV4RXul9Wrri09OsHlRcyStWwT1npqLVG+wSm/m2nTD2u0ssDzp8jSeiaC64igghk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055309; c=relaxed/simple; bh=gaafXcZcjznd8IeK7dwBsQcyVhCGqTDT+dHWEFWiQ8E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=qeDyWz1HgQ6A0wpU9BvnJcCCe0xon8nKQuPKx0ZpNPRmczei4Jtg9m2lIhhFZ95x9iU8EWoaqAyQtTqVcrrT43+eWBdJkW6FaUSaiW5NIk+2IYcd8FVDd22Iz9XbPFU4Y0GVMCq/v+dPnTdH5rMiHIJjeTyrHIf9yeBjHrrmIgo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=VQS5CPWR; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=UBHec5BK; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="VQS5CPWR"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="UBHec5BK" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790055306; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j3/indgfaSaWc6lWBv7sZZH36jWcDt4waMnqUjoKZgQ=; b=VQS5CPWRqzGS58O82hPZzXW8492UTAchvT4Ny4pzwxjXnEzTb/XEbe5YjFk4Adl9WSdhP/ 1cNoyEPGNxHz3Ej8Lar+4Kunw8DFDdnKryzYF5opBAjC5d9baDoxJGRD01Vrg8AkjcDRmb GdkC0qUaBDpLEuzonmayUSm5UXRwtxI= Received: from mail-lf1-f70.google.com (mail-lf1-f70.google.com [209.85.167.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-693-p8E0DT3WMbixoANhotkCpQ-1; Tue, 22 Sep 2026 01:35:05 -0400 X-MC-Unique: p8E0DT3WMbixoANhotkCpQ-1 X-Mimecast-MFC-AGG-ID: p8E0DT3WMbixoANhotkCpQ_1790055304 Received: by mail-lf1-f70.google.com with SMTP id 2adb3069b0e04-5b4fe10d092so2566868e87.0 for ; Mon, 21 Sep 2026 22:35:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790055303; x=1790660103; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=j3/indgfaSaWc6lWBv7sZZH36jWcDt4waMnqUjoKZgQ=; b=UBHec5BKzeq9YLQk3Dr1xRZ3X79glHMDRGYQgnso0HMx7LdNTR3a8hKHjCJzxUld8w svVqUH97yF2vURih3f2x78wB372pOFfoG4Pgvfv1HFUw4M6BHGMPwN4lYr+j+AR/VE7h iOiRM0BYq0fG9bXyF+7pgxFkmd/zbO7jTo38dtnzeYEKSyTRqL+u+dFgpatkyt9jcXLY x1eHafRQ+Vsx15ML/HOWwxaoCH5l9ozloStTGddwwHO/wb61pcsEyMZtDwS+z1kaUnp0 SKA1yGRMWaIU2m/fw1paz8ynS/VIzMmDHK8conlZJJLXwP/0zV6bQLs7fcNqb08MHoPk jO3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790055303; x=1790660103; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=j3/indgfaSaWc6lWBv7sZZH36jWcDt4waMnqUjoKZgQ=; b=bU9YaRchBfko6IZiruUTlewduUtOPHoAbRLqn+RmBN5VEp4VFxSU2Ul9t4bMcAqtYB NU2AN5Zi/BF/Oi4H9qzQtqGdfjcVcU6W3dJNycaIQgd0k74LXbTzIQ7HwsIIvpf1F5jG g0kbWvCaEesgtttS5Jffm/e4ovkQnmkwd8KhFgMMl/MsT+Vt4emqqIJiNyVs63zjww9k KO8/+qg6O8WE+0x+/ygp5om5yLp+vasBUIy7yOIXf4TYug102NTAIEM6UQkOXKNPz/8c Avzi/B+/bPV3Hwjsq2zU82MxdzTFkWUmOjWc6LF2Y7E4CxlLnAwFY3JyudnkB5/sOJn5 QXdg== X-Forwarded-Encrypted: i=1; AKwUvBxtBA/9drj0BS7e2qwoU85IwNRHVHha7N0Ssjqvvhn+ssB3jwUazzSdKPZH36OLS7haAW0I/rAz71VXJ1Q=@vger.kernel.org X-Gm-Message-State: AFuF++lsDcOvk3O5qf8hKTkp8au0GmvhjDQFt7sd0Or0eSwVaQwoXb7S huTvYUmZQGOa6NnSJy0xgYywpmt/BU6u6CcGok9guytNDSRBqxoDfknJIzkSvmHN8B0jb7j+SD2 DVeZnXZGnSAnPSM7mEgKjgoa7TzaI3QBBN8ppOkPA2rZZG+vCKYkGsv3BnnDFUney X-Gm-Gg: AYBFou0OkOippQWi7CJ7IGHSciPfOzPZPznWBbKGM9wgSB0eHVYUGK4xePsaJkskmUB TClpTUeGBcen4CRsNKckNa7WK6XjYhMZ+YOtO0sq58FjG7ojHLbROmc6GFBPHdlxIYQCAduP70Z Mqr7f1B1lTn8fmO0UkevxJce1KvyU6tzTwZOe2HWkEmMLXfwRn4UuQK7+8tu/BP1MC9hV7H0WTH xhDcjMEikx6lfdQYOOSfO7KDsRTShdWjhRTbVQ3Kk7YidSja+J1nq7T4KbB6RaHK42aoC4FmwYy euLX8a72mVg92Q7y6AhMeKDwItBfMKbss+gYuvuToYpquqbc38NimAG0CkP61MATP2zg3tfke8K PH8B+tTujMelDwanAcIpLoSWFC3aPRVc= X-Received: by 2002:a05:6512:110b:b0:5b6:183b:d215 with SMTP id 2adb3069b0e04-5b8c195af81mr3584654e87.60.1790055303484; Mon, 21 Sep 2026 22:35:03 -0700 (PDT) X-Received: by 2002:a05:6512:110b:b0:5b6:183b:d215 with SMTP id 2adb3069b0e04-5b8c195af81mr3584632e87.60.1790055303030; Mon, 21 Sep 2026 22:35:03 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b8d46e0966sm191742e87.55.2026.09.21.22.35.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 22:35:01 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v14 08/12] mm/hmm: add lazy MMU mode support for migration in HMM pagewalk Date: Tue, 22 Sep 2026 08:34:17 +0300 Message-ID: <20260922053421.4092027-9-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922053421.4092027-1-mpenttil@redhat.com> References: <20260922053421.4092027-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 Add calls to lazy_mmu_mode_enable() and lazy_mmu_mode_disable() while doing migration. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- mm/hmm.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/hmm.c b/mm/hmm.c index a1459bd34c04..a67a23e54551 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -371,6 +371,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, uns= igned long addr, /* Report error for everything else */ =20 if (hmm_vma_walk->ptelocked) { + lazy_mmu_mode_disable(); pte_unmap_unlock(ptep, hmm_vma_walk->ptl); hmm_vma_walk->ptelocked =3D false; } else { @@ -394,6 +395,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, uns= igned long addr, !is_zero_pfn(pte_pfn(pte))) { if (hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0)) { if (hmm_vma_walk->ptelocked) { + lazy_mmu_mode_disable(); pte_unmap_unlock(ptep, hmm_vma_walk->ptl); hmm_vma_walk->ptelocked =3D false; } else { @@ -413,6 +415,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, uns= igned long addr, =20 fault: if (hmm_vma_walk->ptelocked) { + lazy_mmu_mode_disable(); pte_unmap_unlock(ptep, hmm_vma_walk->ptl); hmm_vma_walk->ptelocked =3D false; } else { @@ -510,6 +513,7 @@ static int migrate_vma_split_folio(struct folio *folio, if (folio !=3D fault_folio) folio_get(folio); =20 + lazy_mmu_mode_disable(); pte_unmap_unlock(ptep, hmm_vma_walk->ptl); hmm_vma_walk->ptelocked =3D false; =20 @@ -1135,8 +1139,10 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, =20 if (minfo) { ptep =3D pte_offset_map_lock(mm, pmdp, addr, &hmm_vma_walk->ptl); - if (ptep) + if (ptep) { + lazy_mmu_mode_enable(); hmm_vma_walk->ptelocked =3D true; + } } else { ptep =3D pte_offset_map(pmdp, addr); } @@ -1178,6 +1184,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, flush_tlb_range(walk->vma, start, addr); =20 if (hmm_vma_walk->ptelocked) { + lazy_mmu_mode_disable(); pte_unmap_unlock(ptep - 1, hmm_vma_walk->ptl); hmm_vma_walk->ptelocked =3D false; } else { --=20 2.55.0 From nobody Fri Sep 25 18:51:54 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A5A84489FA5 for ; Tue, 22 Sep 2026 05:35:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055314; cv=none; b=LAHkPnmYHkVSL5F5klfhiC3qIVkBlSVwfjEW7jUinRXGXju4hpJ79UfiQtJ336YFvxNOAbRcfQ1lBqHbcRUBmx6p4zvQarkK7tuCdzWzXTCfjYMhAGX9p0vLlczAyp8Q1wD1GI/1Dz9cmsMRt9nDypjwZ+9rlFByos+DUxPbIXo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055314; c=relaxed/simple; bh=yJ4/zDONc3VCH7yVvFo4yUz4R7da9SYeZ7rFR34pviU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=KvROQnuJECF4Y0AEdqlcZrgIi4wQyyF+SoakSx+joiNwZNMKmUMXwkbvN/NHypSUj1Kf+dwWOBIvqONjYF5oCt77GbgLTlcUx50goB4CIWPrcB5lT786j2bpFALbd3WDflI91Pu/jHVvBlOZBpwu+Kx5N+JzbQYWAtzgIRe2V1g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=PXHJ5KCA; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=cYQN3rzp; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="PXHJ5KCA"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="cYQN3rzp" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790055311; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bfS04+9MgqTJr7/AuJyf/BgMvbHeqVFk8HMCC4JI/vk=; b=PXHJ5KCABOnYV7gYljO1rX9sC27ooKUiQNj4HYHhVdyA4/j1d2M116/ZjhEIRAONX9BizT 4tIhXGD0sddbq/syhvSsCj+q1/k5kgIlvtwkA1QYn/7X6vLdR7OCYH+uEg/RhkWzASCQWo 0CukFT4hh9Bpddo/qecfRxngltrIDy8= Received: from mail-lf1-f72.google.com (mail-lf1-f72.google.com [209.85.167.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-574-SVYkO5SIOeSXamBk1sShAQ-1; Tue, 22 Sep 2026 01:35:09 -0400 X-MC-Unique: SVYkO5SIOeSXamBk1sShAQ-1 X-Mimecast-MFC-AGG-ID: SVYkO5SIOeSXamBk1sShAQ_1790055308 Received: by mail-lf1-f72.google.com with SMTP id 2adb3069b0e04-5b767214d60so283660e87.1 for ; Mon, 21 Sep 2026 22:35:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790055308; x=1790660108; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=bfS04+9MgqTJr7/AuJyf/BgMvbHeqVFk8HMCC4JI/vk=; b=cYQN3rzpBTSfyZ5ycs9TUbyPSjJKoMCoeSYV6yZCxlFsaqJaHpCMNK35gGK600G7Zb qEQK37QK4gUHvqaKfCT/GBwPIy+CNp3cWOx+RMGZmu3LbTUCmzZbaDbEmRMqJAjrvAiK UyyM5HAJGeSZ2QDQMbOxIP1kaAFsF4UVGDa6B0JwRghSD1nspalxlYt8iou+b1cjqtRQ b16KFlvvbWOGLSkxEMa+lpHeZst7BVZHrRokxW9lvQiwjWgtrlM8pdNKH0HRoI3SzA6V uIMBhe9hfGWK/dxFHpJmKdQ+z/ri4ulp/b+CUj82yfaij9FgbHRaFlEglTzKVu55gLz7 Jvfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790055308; x=1790660108; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=bfS04+9MgqTJr7/AuJyf/BgMvbHeqVFk8HMCC4JI/vk=; b=L48mFfE5pzDoMacSjx2/l7JCyxrRZ+SKo4Pt/zjrgUkSa9VT7yAIs6uqGK47Ub76lI +ycPVdM03OALj+mb4wEBNy52JZQ9hXl7dIwxAZBM6spxDlQNLSYKpsB5kQp5qZz4Me9E Uq5nm2W11uOwGC7AWzPTo1VuhV80UsVQRdHszrv7KGZ2SSVCi8gap7BXZqaq5qeTbkb5 iDibe5225wFgbeejEpk2bXwTlNkJ6C4A6NKuwn6bOU9W+v7o/m3T2pMJwHEso6F7OtrA cIJ7/izfNUZyjsZGzj9gG2+MSqG6d4d5/0VkXQ3DWQDroHGcgSZ3zcExK46I/UGX3Et/ oj8w== X-Forwarded-Encrypted: i=1; AKwUvBwlUrGhTz2xicUWRPPECD1FIgyTB7z8X0LSAjuUVOc21FCSin98y7lJio5w0bUWBAXiFrZ4OXaTtdDfasg=@vger.kernel.org X-Gm-Message-State: AFuF++nuK/pboh/QQA7KrXvRwvkmPW31CziFTZrddE/gQ9fgleukyXI4 5vq0GmeQxDNJ1pxcZZaOZkG0cVp04aDzPtf7OgvmQ3mLAJZrf8P8vzFafrQQlhpQjbkdywSFrFG qELWM8hZevWxgg0/bUiBuRrFlmW+amXI3sJIRGV3Gl9M927+VqwSCv/Mlg0J3yGYN X-Gm-Gg: AYBFou0bIqva9ha0AhkYfGvmy5lax7IeEkhsr/FdWlKHhYgJmBAn7OcyHdxt72Qu88W HU5hsgyC+1Xzyhxjn+J4xdaLTVOY367/KGCXCrjharar4eD4i6AsPjMF59c5g+CaOZqitvLy6QP M5/Yh0ziVmwiPeRVXQvwmi3oNbtN9NBaoFyRV44AVOq64swAsnNFYrVhn0OtSbCQZVGrh7aslRR E+MpmJdO84OejJRnlq3NECjlriYzgH0xNveWyzKs9eD8BhVzgWENDjvxSwbXQOziYilq0fwJYEf 2PZP341CXDuxKdrLoMS0TghAeTCx9KadjRE1hxDec2JSa95NjGBZz5TKiBYOP1xnsyWrJDebcfv qtR8FXoIqFhpms75apqwh X-Received: by 2002:a05:6512:4884:b0:5b6:10bc:dc47 with SMTP id 2adb3069b0e04-5b8d2945a50mr296474e87.30.1790055308114; Mon, 21 Sep 2026 22:35:08 -0700 (PDT) X-Received: by 2002:a05:6512:4884:b0:5b6:10bc:dc47 with SMTP id 2adb3069b0e04-5b8d2945a50mr296458e87.30.1790055307602; Mon, 21 Sep 2026 22:35:07 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b8d46e0966sm191742e87.55.2026.09.21.22.35.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 22:35:05 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v14 09/12] mm/hmm: implement rollback for device page migration in HMM pagewalk Date: Tue, 22 Sep 2026 08:34:18 +0300 Message-ID: <20260922053421.4092027-10-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922053421.4092027-1-mpenttil@redhat.com> References: <20260922053421.4092027-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 During the migration pagewalk, the PTE table could be cleared and/or changed into PMD leaf or even another PTE table while dropped locks. In these cases the possibly inserted migration ptes are gone. We have to however undo the collecting done so far, so unlock the folios and drop reference taken. During the pagewalk we notice such scenarios if going to recollect a pfn but have already committed to migrate the entry with HMM_PFN_MIGRATE, in which case rollback. If we encounter migration ptes they are just skipped to allow for restart own walks. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- include/linux/hmm.h | 22 +++++++++++++ mm/hmm.c | 76 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 98 insertions(+) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 4f56f3419cb4..b08ebc1343dd 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -111,6 +111,28 @@ static inline unsigned int hmm_pfn_to_map_order(unsign= ed long hmm_pfn) return (hmm_pfn >> HMM_PFN_ORDER_SHIFT) & 0x1F; } =20 +/* + * hmm_pfn_collected() - is this pfn entry prepared for migration ? + * If collected the folio's refcount is increased and the folio + * is locked. + */ +static inline bool hmm_pfn_collected(unsigned long hmm_pfn) +{ + return (hmm_pfn & (HMM_PFN_VALID | HMM_PFN_MIGRATE)) =3D=3D + (HMM_PFN_VALID | HMM_PFN_MIGRATE); +} + +/* + * hmm_pfn_rollback_collected() - undoes the collection of hmm_pfn + * + * Note for total rollback the folio's refcount has to be put + * and folio has to be unlocked. + */ +static inline unsigned long hmm_pfn_rollback_collected(unsigned long hmm_p= fn) +{ + return hmm_pfn & ~(HMM_PFN_VALID | HMM_PFN_MIGRATE | HMM_PFN_COMPOUND); +} + /* * struct hmm_range - track invalidation lock on virtual address range * diff --git a/mm/hmm.c b/mm/hmm.c index a67a23e54551..e64444cbc62a 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -93,6 +93,11 @@ enum { HMM_PFN_P2PDMA_BUS, }; =20 +static void hmm_vma_handle_migrate_prepare_rollback(const struct hmm_vma_w= alk *hmm_vma_walk, + unsigned long start, + unsigned long end, + unsigned long *hmm_pfn); + static int hmm_pfns_fill(unsigned long addr, unsigned long end, struct hmm_vma_walk *hmm_vma_walk, unsigned long cpu_flags) { @@ -109,6 +114,8 @@ static int hmm_pfns_fill(unsigned long addr, unsigned l= ong end, } } =20 + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr, end, &range->= hmm_pfns[i]); + if (migrate && thp_migration_supported() && (minfo & MIGRATE_VMA_SELECT_COMPOUND) && IS_ALIGNED(addr, HPAGE_PMD_SIZE) && @@ -280,6 +287,8 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, uns= igned long addr, return hmm_record_fault(addr, end, required_fault, walk); } =20 + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr, + end, hmm_pfns); pfn =3D pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); for (i =3D 0; addr < end; addr +=3D PAGE_SIZE, i++, pfn++) { hmm_pfns[i] &=3D HMM_PFN_INOUT_FLAGS; @@ -410,6 +419,9 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, uns= igned long addr, =20 new_pfn_flags =3D pte_pfn(pte) | cpu_flags; out: + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr, + addr + PAGE_SIZE, + hmm_pfn); *hmm_pfn =3D (*hmm_pfn & HMM_PFN_INOUT_FLAGS) | new_pfn_flags; return 0; =20 @@ -448,6 +460,9 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *wa= lk, unsigned long start, if (softleaf_is_device_private_write(entry)) cpu_flags |=3D HMM_PFN_WRITE; =20 + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, + start, end, + hmm_pfns); /* * Fully populate the PFN list though subsequent PFNs could be * inferred, because drivers which are not yet aware of large @@ -550,6 +565,48 @@ static int migrate_vma_split_folio(struct folio *folio, return 0; } =20 +/* + * Due to dropping ptl locks for splitting for instance, would we + * overwrite already collected pfns? This could happen when pmd + * pointing to a page table has vanished and been replaced + * with a leaf pmd, or another page table. + * In that case unref and unlock the folios, + * the pfns of which were collected from the disappeared + * page tables. + */ +static void hmm_vma_handle_migrate_prepare_rollback(const struct hmm_vma_w= alk *hmm_vma_walk, + unsigned long start, + unsigned long end, + unsigned long *hmm_pfn) +{ + struct hmm_range *range =3D hmm_vma_walk->range; + struct migrate_vma *migrate =3D range->migrate; + struct folio *fault_folio =3D NULL; + enum migrate_vma_info minfo; + struct folio *folio; + unsigned long i; + + minfo =3D hmm_select_migrate(range); + if (!minfo) + return; + + WARN_ON_ONCE(!migrate); + + fault_folio =3D migrate->fault_page ? + page_folio(migrate->fault_page) : NULL; + + for (i =3D 0; start < end; start +=3D PAGE_SIZE, i++) { + if (hmm_pfn_collected(hmm_pfn[i])) { + folio =3D page_folio(hmm_pfn_to_page(hmm_pfn[i])); + if (folio !=3D fault_folio) + folio_unlock(folio); + folio_put(folio); + hmm_pfn[i] =3D hmm_pfn_rollback_collected(hmm_pfn[i]); + + } + } +} + static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk, pmd_t *pmdp, unsigned long start, @@ -693,6 +750,11 @@ static int hmm_vma_handle_migrate_prepare(const struct= mm_walk *walk, pte =3D ptep_get(ptep); =20 if (pte_none(pte)) { + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, + addr, + addr + PAGE_SIZE, + hmm_pfn); + if (vma_is_anonymous(walk->vma)) { *hmm_pfn &=3D HMM_PFN_INOUT_FLAGS; *hmm_pfn |=3D HMM_PFN_MIGRATE; @@ -740,6 +802,10 @@ static int hmm_vma_handle_migrate_prepare(const struct= mm_walk *walk, pfn =3D pte_pfn(pte); if (is_zero_pfn(pfn) && (minfo & MIGRATE_VMA_SELECT_SYSTEM)) { + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, + addr, + addr + PAGE_SIZE, + hmm_pfn); *hmm_pfn =3D HMM_PFN_MIGRATE; goto out; } @@ -916,6 +982,13 @@ static int hmm_vma_walk_split(pmd_t *pmdp, return ret; } #else +static void hmm_vma_handle_migrate_prepare_rollback(const struct hmm_vma_w= alk *hmm_vma_walk, + unsigned long start, + unsigned long end, + unsigned long *hmm_pfn) +{ +} + static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk, pmd_t *pmdp, unsigned long start, @@ -1142,6 +1215,9 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, if (ptep) { lazy_mmu_mode_enable(); hmm_vma_walk->ptelocked =3D true; + } else { + /* The pte table is gone */ + hmm_vma_handle_migrate_prepare_rollback(walk->private, addr, end, hmm_p= fns); } } else { ptep =3D pte_offset_map(pmdp, addr); --=20 2.55.0 From nobody Fri Sep 25 18:51:54 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1717F48D881 for ; Tue, 22 Sep 2026 05:35:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055318; cv=none; b=n2Dn6U+SXcaAz/vREwKfW8bWLXiTTmVMU+4pJ4ApnCMSEnInRrDKC0pZyh/5DrPk/AfnDGrybP92tUX2JHfWVsvrS6TCVWAV9XnqBr45/5RuCOIqpuuzSd6Ly7bGc4hgjtV8jTLzSMx3R2zliInW528u2wu4aYfyFnfKrU8d5CI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055318; c=relaxed/simple; bh=fenRUEXGWoKb7BjuVIpaYwzRL6B6mDm53pogy2ThnfQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=FALyezhfJnXNNb/RlZJGIz9fnless/trXvTZM3ir7EpTH6ih+7lkiMLFjJOLybD9bP/GC8bvAwC98CUHuVO5hpmQocj1MI52RiVon4xS/H1BkqWFHFNzDX5uuEk6MJETAdtTIk3YqY62GBkAnNemFEzrQtNCaCrAkzppL9rQeTA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=SHDlQUOv; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=SSv76M6i; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="SHDlQUOv"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="SSv76M6i" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790055315; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UKL+vFX2Cnp1o1+u97ic85G8uQdKpjbv7FGhiJsOCec=; b=SHDlQUOvPioP7mspt0Prc7vOa7AxOLBtjHk9Xz3I+IYvLcQWye5KK/fZTE3pOeY5hiFhDE LX9TA0Yg6ux6YtZTzlGKGeriZ38ZhNEzDWSx3dUXl67pgr62xkEGFCLunjIJ8C41c5sLdU u6U4xT39rpvYigm+3GFJRfsBEL+m4G0= Received: from mail-lf1-f69.google.com (mail-lf1-f69.google.com [209.85.167.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-8-_btReQqVMcmROKtQllFvgQ-1; Tue, 22 Sep 2026 01:35:12 -0400 X-MC-Unique: _btReQqVMcmROKtQllFvgQ-1 X-Mimecast-MFC-AGG-ID: _btReQqVMcmROKtQllFvgQ_1790055311 Received: by mail-lf1-f69.google.com with SMTP id 2adb3069b0e04-5b5bfdb26b1so1747274e87.2 for ; Mon, 21 Sep 2026 22:35:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790055311; x=1790660111; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=UKL+vFX2Cnp1o1+u97ic85G8uQdKpjbv7FGhiJsOCec=; b=SSv76M6iMkrxV/Yv+wt7CISrRqw6UEE6szg7xiitCA502didsic5Ua0/FCbA9vMFyx FPbpvXhdBBB7cmnxFr7V1d7lw1oXhnQGr1EEBwENy9f6IBrBS95ibVG9deI9m6ZxdJv9 SLhcEJybpGPBKM+0xIYEHLY/QzISA/58ivtyWVRpqfCJSKHQ/9v3yQg4xXwRVXngaYVC BLqrvGS//Kj637xvoS2zrVBhAW1yADdP/QId89aJ4qfIqIaD0ICyzJOtC71vIWBdmJKB yx47kqHaRrtXLHQHK2fuzacW5DadGLYxqLd3KkgfakOaxugMQ+QH2UjftMNvpVQB91ay IEcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790055311; x=1790660111; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=UKL+vFX2Cnp1o1+u97ic85G8uQdKpjbv7FGhiJsOCec=; b=OQLzjeYqCj0ONtbuJ50OxtFzXAh7juqKyeYqR1pN/Xi2o2DtcV1USnq9vZmStm4rCY I5KOXihUgo4z1p6Daodbas2pi8RkLoDH2v/3AmsRF/x0Kr5r/zpK1z5cdpxZu9Mdrga5 5gwmsZ0AdNDKUDBy//ElDMqhzOpBqwmOUlxJp/JZnHpHiwZbrTfrsq5fkSaMMKutaida bcnh50EmXTJVcLl7ARb670UrimKIyLih/GJMXUUHPYYlTGirKgRREQlH1LCq9IGmlR+R 7rTKVNaw2GAw2amTEBC7G2VZBxdoNOFgnJsXe7hv7QAHyaJ7QSbWnjnasMaNPxU576pc 2D3g== X-Forwarded-Encrypted: i=1; AKwUvByt2Qyqe11M0lW1WGp8S8UvTiR+A6Rk1ualErP4aynEwjbTv2kKuNmQBfGfEmkfeoRM2SbWHkUM76WjXKs=@vger.kernel.org X-Gm-Message-State: AFuF++m2BVnUl3t4RucQLXPlDEJRX1bWX7u3OpszihpNkgyb6iAt28QK ezCmlJQAZiM7HRFboWrQ7qBrJqZ16r8lzSh54jTgz0rn9UzubnyDMCw6PCLES41H0sxbIg338ky 9LU4S35p7fzohqGyGgI88UwIX6b96JMzUxcLK/8Kf6E3rn97rPmQWLwUS5k+64Qzi X-Gm-Gg: AYBFou2zKsvz4qEh31qjnbfJYpUPFs2afSndZTsbZC0NSfGZLzBXvqkP+E1BrgIBrQV AYj6Vqkda5Uo76rP1myA/HlfiYbsSRGmiUXG4IcfEEdotAnKRKxdALJZdSxrJJvVsKxwhT0hc84 DZmsbB4G8zwiH6wHpXlaJppUAK86t3k6+ykNUV4SHQtjeyu6u+hUxcQdcAG9kpYijaqHdPRXjEk E14qC8LXKBwXb6EntCYglmrIfaDtYxcuejcr1JCE8XAaJJp0oPJ9A3ZWtblFpV+zgEj/a69efRZ 1PP1o8nsY3miOHcVUaGaKcrprSixBknCtI71pNB6lXb9IVup+1/cFJN/wLyekhxGSw/gOXm1Acu E9zHWidMEErvPEpW6DJt2 X-Received: by 2002:a05:6512:23a2:b0:5b6:6b8:1fc5 with SMTP id 2adb3069b0e04-5b8c18226eamr4272907e87.17.1790055311065; Mon, 21 Sep 2026 22:35:11 -0700 (PDT) X-Received: by 2002:a05:6512:23a2:b0:5b6:6b8:1fc5 with SMTP id 2adb3069b0e04-5b8c18226eamr4272895e87.17.1790055310478; Mon, 21 Sep 2026 22:35:10 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b8d46e0966sm191742e87.55.2026.09.21.22.35.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 22:35:09 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v14 10/12] mm: enable device page migration from HMM pagewalk Date: Tue, 22 Sep 2026 08:34:19 +0300 Message-ID: <20260922053421.4092027-11-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922053421.4092027-1-mpenttil@redhat.com> References: <20260922053421.4092027-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 HMM pagewalk has now the machinery to do the first phase of device page migration, collecting the pfns and installing migration ptes. Enable migration in hmm_range_fault(), and change migrate_vma_setup() to use HMM pagewalk path. Two new flags, MIGRATE_VMA_FAULT and MIGRATE_VMA_WRITE are introduced for migrate_vma_setup(), to request for faulting missing pages, and requesting write access. Also, the migrate_vma_collect*() based paths are now unused, so delete them. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttil=C3=A4 --- include/linux/migrate.h | 10 +- mm/hmm.c | 33 ++- mm/migrate_device.c | 562 ++++------------------------------------ 3 files changed, 83 insertions(+), 522 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 01c5f62a56e9..c73e41625ac2 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -180,6 +180,8 @@ enum migrate_vma_info { MIGRATE_VMA_SELECT_DEVICE_PRIVATE =3D 1 << 1, MIGRATE_VMA_SELECT_DEVICE_COHERENT =3D 1 << 2, MIGRATE_VMA_SELECT_COMPOUND =3D 1 << 3, + MIGRATE_VMA_FAULT =3D 1 << 4, + MIGRATE_VMA_WRITE =3D 1 << 5, }; =20 struct migrate_vma { @@ -217,10 +219,14 @@ struct migrate_vma { struct page *fault_page; }; =20 -// TODO: enable migration static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *r= ange) { - return 0; + enum migrate_vma_info minfo; + + minfo =3D (range->default_flags & HMM_PFN_REQ_MIGRATE) ? + range->migrate->flags : 0; + + return minfo; } =20 #ifdef CONFIG_MMU_NOTIFIER diff --git a/mm/hmm.c b/mm/hmm.c index e64444cbc62a..87ebd2500df6 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -1562,14 +1562,43 @@ static int hmm_range_fault_locked(struct hmm_range = *range, bool *locked) * the invalidation to finish. * -EFAULT: A page was requested to be valid and could not be made val= id * ie it has no backing VMA or it is illegal to access + * -ERANGE: The range crosses multiple VMAs, or space for hmm_pfns arr= ay + * is too low. * * This is similar to get_user_pages(), except that it can read the page t= ables * without mutating them (ie causing faults). * * The mmap lock must be held by the caller and will remain held on return. * New users should prefer hmm_range_fault_unlocked_timeout() unless they - * specifically need to keep the mmap lock held across the call. This help= er - * cannot support VMAs whose fault handlers need to drop the mmap lock. + * specifically need to keep the mmap lock held across the call like while + * migrating. This helper cannot support VMAs whose fault handlers need to + * drop the mmap lock. + * + * If want to do migration after faulting, call hmm_range_fault() with + * range.default_flags of HMM_PFN_REQ_MIGRATE, and optionally + * HMM_PFN_REQ_FAULT|HMM_PFN_REQ_WRITE, and initialize range->migrate fiel= d. + * range->migrate->vma will be populated during the call, + * and must be stable across the whole migrate process, which is + * why mmap_lock must be held around this call. + * + * When HMM_PFN_REQ_MIGRATE is set, migration collection may be partial on + * return and the caller takes responsibility for completing or aborting i= t. + * + * On success, the caller must call migrate_hmm_range_setup() and may then + * proceed with the normal migrate_vma sequence: prepare destination pages, + * call migrate_vma_pages(), update device mappings as needed, and finally + * call migrate_vma_finalize(). + * + * On -EBUSY, the caller may retry hmm_range_fault() using the same range = and + * PFN array without undoing entries collected by the previous attempt. If + * the caller stops retrying, it must abort the partial migration. + * + * On any other error, or when abandoning a retry, the caller must call + * migrate_hmm_range_setup(), migrate_vma_pages() with no valid destination + * entries, and migrate_vma_finalize() to abort the partial migration. + * + * The mmap read lock must remain held until the migration has either been + * completed or aborted. */ int hmm_range_fault(struct hmm_range *range) { diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 43e571e82cca..1e1a592e1267 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -18,519 +18,6 @@ #include #include "internal.h" =20 -static int migrate_vma_collect_skip(unsigned long start, - unsigned long end, - struct mm_walk *walk) -{ - struct migrate_vma *migrate =3D walk->private; - unsigned long addr; - - for (addr =3D start; addr < end; addr +=3D PAGE_SIZE) { - migrate->dst[migrate->npages] =3D 0; - migrate->src[migrate->npages++] =3D 0; - } - - return 0; -} - -static int migrate_vma_collect_hole(unsigned long start, - unsigned long end, - __always_unused int depth, - struct mm_walk *walk) -{ - struct migrate_vma *migrate =3D walk->private; - unsigned long addr; - - /* Only allow populating anonymous memory. */ - if (!vma_is_anonymous(walk->vma)) - return migrate_vma_collect_skip(start, end, walk); - - if (thp_migration_supported() && - (migrate->flags & MIGRATE_VMA_SELECT_COMPOUND) && - (IS_ALIGNED(start, HPAGE_PMD_SIZE) && - IS_ALIGNED(end, HPAGE_PMD_SIZE))) { - migrate->src[migrate->npages] =3D MIGRATE_PFN_MIGRATE | - MIGRATE_PFN_COMPOUND; - migrate->dst[migrate->npages] =3D 0; - migrate->npages++; - migrate->cpages++; - - /* - * Collect the remaining entries as holes, in case we - * need to split later - */ - return migrate_vma_collect_skip(start + PAGE_SIZE, end, walk); - } - - for (addr =3D start; addr < end; addr +=3D PAGE_SIZE) { - migrate->src[migrate->npages] =3D MIGRATE_PFN_MIGRATE; - migrate->dst[migrate->npages] =3D 0; - migrate->npages++; - migrate->cpages++; - } - - return 0; -} - -/** - * migrate_vma_split_folio() - Helper function to split a THP folio - * @folio: the folio to split - * @fault_page: struct page associated with the fault if any - * - * If @folio is not the folio containing @fault_page, the caller must hold= a - * reference on @folio. The helper consumes that reference. - * - * Returns 0 on success - */ -static int migrate_vma_split_folio(struct folio *folio, - struct page *fault_page) -{ - int ret; - struct folio *fault_folio =3D fault_page ? page_folio(fault_page) : NULL; - struct folio *new_fault_folio =3D NULL; - - if (folio !=3D fault_folio) - folio_lock(folio); - - ret =3D split_folio(folio); - if (ret) { - if (folio !=3D fault_folio) { - folio_unlock(folio); - folio_put(folio); - } - return ret; - } - - new_fault_folio =3D fault_page ? page_folio(fault_page) : NULL; - - /* - * Ensure the lock is held on the correct - * folio after the split - */ - if (!new_fault_folio) { - folio_unlock(folio); - folio_put(folio); - } else if (folio !=3D new_fault_folio) { - if (new_fault_folio !=3D fault_folio) { - folio_get(new_fault_folio); - folio_lock(new_fault_folio); - } - folio_unlock(folio); - folio_put(folio); - } - - return 0; -} - -/** migrate_vma_collect_huge_pmd - collect THP pages without splitting the - * folio for device private pages. - * @pmdp: pointer to pmd entry - * @start: start address of the range for migration - * @end: end address of the range for migration - * @walk: mm_walk callback structure - * @fault_folio: folio associated with the fault if any - * - * Collect the huge pmd entry at @pmdp for migration and set the - * MIGRATE_PFN_COMPOUND flag in the migrate src entry to indicate that - * migration will occur at HPAGE_PMD granularity - */ -static int migrate_vma_collect_huge_pmd(pmd_t *pmdp, unsigned long start, - unsigned long end, struct mm_walk *walk, - struct folio *fault_folio) -{ - struct mm_struct *mm =3D walk->mm; - struct folio *folio; - struct migrate_vma *migrate =3D walk->private; - spinlock_t *ptl; - int ret; - unsigned long write =3D 0; - - ptl =3D pmd_lock(mm, pmdp); - if (pmd_none(*pmdp)) { - spin_unlock(ptl); - return migrate_vma_collect_hole(start, end, -1, walk); - } - - if (pmd_trans_huge(*pmdp)) { - if (!(migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) { - spin_unlock(ptl); - return migrate_vma_collect_skip(start, end, walk); - } - - folio =3D pmd_folio(*pmdp); - if (is_huge_zero_folio(folio)) { - spin_unlock(ptl); - return migrate_vma_collect_hole(start, end, -1, walk); - } - if (pmd_write(*pmdp)) - write =3D MIGRATE_PFN_WRITE; - } else if (!pmd_present(*pmdp)) { - const softleaf_t entry =3D softleaf_from_pmd(*pmdp); - - if (!softleaf_is_device_private(entry) || - !(migrate->flags & MIGRATE_VMA_SELECT_DEVICE_PRIVATE)) { - spin_unlock(ptl); - return migrate_vma_collect_skip(start, end, walk); - } - - folio =3D softleaf_to_folio(entry); - if (folio->pgmap->owner !=3D migrate->pgmap_owner) { - spin_unlock(ptl); - return migrate_vma_collect_skip(start, end, walk); - } - - if (softleaf_is_device_private_write(entry)) - write =3D MIGRATE_PFN_WRITE; - } else { - spin_unlock(ptl); - return -EAGAIN; - } - - folio_get(folio); - if (folio !=3D fault_folio && unlikely(!folio_trylock(folio))) { - spin_unlock(ptl); - folio_put(folio); - return migrate_vma_collect_skip(start, end, walk); - } - - if (thp_migration_supported() && - (migrate->flags & MIGRATE_VMA_SELECT_COMPOUND) && - (IS_ALIGNED(start, HPAGE_PMD_SIZE) && - IS_ALIGNED(end, HPAGE_PMD_SIZE))) { - - struct page_vma_mapped_walk pvmw =3D { - .ptl =3D ptl, - .address =3D start, - .pmd =3D pmdp, - .vma =3D walk->vma, - }; - - unsigned long pfn =3D page_to_pfn(folio_page(folio, 0)); - - migrate->src[migrate->npages] =3D migrate_pfn(pfn) | write - | MIGRATE_PFN_MIGRATE - | MIGRATE_PFN_COMPOUND; - migrate->dst[migrate->npages++] =3D 0; - migrate->cpages++; - ret =3D set_pmd_migration_entry(&pvmw, folio_page(folio, 0)); - if (ret) { - migrate->npages--; - migrate->cpages--; - migrate->src[migrate->npages] =3D 0; - migrate->dst[migrate->npages] =3D 0; - goto fallback; - } - migrate_vma_collect_skip(start + PAGE_SIZE, end, walk); - spin_unlock(ptl); - return 0; - } - -fallback: - spin_unlock(ptl); - if (!folio_test_large(folio)) - goto done; - ret =3D split_folio(folio); - if (fault_folio !=3D folio) - folio_unlock(folio); - folio_put(folio); - if (ret) - return migrate_vma_collect_skip(start, end, walk); - if (pmd_none(pmdp_get_lockless(pmdp))) - return migrate_vma_collect_hole(start, end, -1, walk); - -done: - return -ENOENT; -} - -static int migrate_vma_collect_pmd(pmd_t *pmdp, - unsigned long start, - unsigned long end, - struct mm_walk *walk) -{ - struct migrate_vma *migrate =3D walk->private; - struct vm_area_struct *vma =3D walk->vma; - struct mm_struct *mm =3D vma->vm_mm; - unsigned long addr =3D start, unmapped =3D 0; - spinlock_t *ptl; - struct folio *fault_folio =3D migrate->fault_page ? - page_folio(migrate->fault_page) : NULL; - pte_t *ptep; - -again: - if (pmd_trans_huge(*pmdp) || !pmd_present(*pmdp)) { - int ret =3D migrate_vma_collect_huge_pmd(pmdp, start, end, walk, fault_f= olio); - - if (ret =3D=3D -EAGAIN) - goto again; - if (ret =3D=3D 0) - return 0; - } - - ptep =3D pte_offset_map_lock(mm, pmdp, start, &ptl); - if (!ptep) - goto again; - lazy_mmu_mode_enable(); - ptep +=3D (addr - start) / PAGE_SIZE; - - for (; addr < end; addr +=3D PAGE_SIZE, ptep++) { - struct dev_pagemap *pgmap; - unsigned long mpfn =3D 0, pfn; - struct folio *folio; - struct page *page; - softleaf_t entry; - pte_t pte; - - pte =3D ptep_get(ptep); - - if (pte_none(pte)) { - if (vma_is_anonymous(vma)) { - mpfn =3D MIGRATE_PFN_MIGRATE; - migrate->cpages++; - } - goto next; - } - - if (!pte_present(pte)) { - /* - * Only care about unaddressable device page special - * page table entry. Other special swap entries are not - * migratable, and we ignore regular swapped page. - */ - entry =3D softleaf_from_pte(pte); - if (!softleaf_is_device_private(entry)) - goto next; - - page =3D softleaf_to_page(entry); - pgmap =3D page_pgmap(page); - if (!(migrate->flags & - MIGRATE_VMA_SELECT_DEVICE_PRIVATE) || - pgmap->owner !=3D migrate->pgmap_owner) - goto next; - - folio =3D page_folio(page); - if (folio_test_large(folio)) { - int ret; - - /* migrate_vma_split_folio() consumes this reference */ - if (folio !=3D fault_folio) - folio_get(folio); - lazy_mmu_mode_disable(); - pte_unmap_unlock(ptep, ptl); - ret =3D migrate_vma_split_folio(folio, - migrate->fault_page); - - if (ret) { - if (unmapped) - flush_tlb_range(walk->vma, start, end); - - return migrate_vma_collect_skip(addr, end, walk); - } - - goto again; - } - - mpfn =3D migrate_pfn(page_to_pfn(page)) | - MIGRATE_PFN_MIGRATE; - if (softleaf_is_device_private_write(entry)) - mpfn |=3D MIGRATE_PFN_WRITE; - } else { - pfn =3D pte_pfn(pte); - if (is_zero_pfn(pfn) && - (migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) { - mpfn =3D MIGRATE_PFN_MIGRATE; - migrate->cpages++; - goto next; - } - page =3D vm_normal_page(migrate->vma, addr, pte); - if (page && !is_zone_device_page(page) && - !(migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) { - goto next; - } else if (page && is_device_coherent_page(page)) { - pgmap =3D page_pgmap(page); - - if (!(migrate->flags & - MIGRATE_VMA_SELECT_DEVICE_COHERENT) || - pgmap->owner !=3D migrate->pgmap_owner) - goto next; - } - folio =3D page ? page_folio(page) : NULL; - if (folio && folio_test_large(folio)) { - int ret; - - /* migrate_vma_split_folio() consumes this reference */ - if (folio !=3D fault_folio) - folio_get(folio); - lazy_mmu_mode_disable(); - pte_unmap_unlock(ptep, ptl); - ret =3D migrate_vma_split_folio(folio, - migrate->fault_page); - - if (ret) { - if (unmapped) - flush_tlb_range(walk->vma, start, end); - - return migrate_vma_collect_skip(addr, end, walk); - } - - goto again; - } - mpfn =3D migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE; - mpfn |=3D pte_write(pte) ? MIGRATE_PFN_WRITE : 0; - } - - if (!page || !page->mapping) { - mpfn =3D 0; - goto next; - } - - /* - * By getting a reference on the folio we pin it and that blocks - * any kind of migration. Side effect is that it "freezes" the - * pte. - * - * We drop this reference after isolating the folio from the lru - * for non device folio (device folio are not on the lru and thus - * can't be dropped from it). - */ - folio =3D page_folio(page); - folio_get(folio); - - /* - * We rely on folio_trylock() to avoid deadlock between - * concurrent migrations where each is waiting on the others - * folio lock. If we can't immediately lock the folio we fail this - * migration as it is only best effort anyway. - * - * If we can lock the folio it's safe to set up a migration entry - * now. In the common case where the folio is mapped once in a - * single process setting up the migration entry now is an - * optimisation to avoid walking the rmap later with - * try_to_migrate(). - */ - if (fault_folio =3D=3D folio || folio_trylock(folio)) { - bool anon_exclusive; - pte_t swp_pte; - - if (pte_present(pte)) - flush_cache_page(vma, addr, pte_pfn(pte)); - anon_exclusive =3D folio_test_anon(folio) && - PageAnonExclusive(page); - if (anon_exclusive) { - pte =3D ptep_clear_flush(vma, addr, ptep); - - if (folio_try_share_anon_rmap_pte(folio, page)) { - set_pte_at(mm, addr, ptep, pte); - if (fault_folio !=3D folio) - folio_unlock(folio); - folio_put(folio); - mpfn =3D 0; - goto next; - } - } else { - pte =3D ptep_get_and_clear(mm, addr, ptep); - } - - migrate->cpages++; - - /* Set the dirty flag on the folio now the pte is gone. */ - if (pte_present(pte) && pte_dirty(pte)) - folio_mark_dirty(folio); - - /* Setup special migration page table entry */ - if (mpfn & MIGRATE_PFN_WRITE) - entry =3D make_writable_migration_entry( - page_to_pfn(page)); - else if (anon_exclusive) - entry =3D make_readable_exclusive_migration_entry( - page_to_pfn(page)); - else - entry =3D make_readable_migration_entry( - page_to_pfn(page)); - if (pte_present(pte)) { - if (pte_young(pte)) - entry =3D make_migration_entry_young(entry); - if (pte_dirty(pte)) - entry =3D make_migration_entry_dirty(entry); - } - swp_pte =3D swp_entry_to_pte(entry); - if (pte_present(pte)) { - if (pte_soft_dirty(pte)) - swp_pte =3D pte_swp_mksoft_dirty(swp_pte); - if (pte_uffd(pte)) - swp_pte =3D pte_swp_mkuffd(swp_pte); - } else { - if (pte_swp_soft_dirty(pte)) - swp_pte =3D pte_swp_mksoft_dirty(swp_pte); - if (pte_swp_uffd(pte)) - swp_pte =3D pte_swp_mkuffd(swp_pte); - } - set_pte_at(mm, addr, ptep, swp_pte); - - /* - * This is like regular unmap: we remove the rmap and - * drop the folio refcount. The folio won't be freed, as - * we took a reference just above. - */ - folio_remove_rmap_pte(folio, page, vma); - folio_put(folio); - - if (pte_present(pte)) - unmapped++; - } else { - folio_put(folio); - mpfn =3D 0; - } - -next: - migrate->dst[migrate->npages] =3D 0; - migrate->src[migrate->npages++] =3D mpfn; - } - - /* Only flush the TLB if we actually modified any entries */ - if (unmapped) - flush_tlb_range(walk->vma, start, end); - - lazy_mmu_mode_disable(); - pte_unmap_unlock(ptep - 1, ptl); - - return 0; -} - -static const struct mm_walk_ops migrate_vma_walk_ops =3D { - .pmd_entry =3D migrate_vma_collect_pmd, - .pte_hole =3D migrate_vma_collect_hole, - .walk_lock =3D PGWALK_RDLOCK, -}; - -/* - * migrate_vma_collect() - collect pages over a range of virtual addresses - * @migrate: migrate struct containing all migration information - * - * This will walk the CPU page table. For each virtual address backed by a - * valid page, it updates the src array and takes a reference on the page,= in - * order to pin the page until we lock it and unmap it. - */ -static void migrate_vma_collect(struct migrate_vma *migrate) -{ - struct mmu_notifier_range range; - - /* - * Note that the pgmap_owner is passed to the mmu notifier callback so - * that the registered device driver can skip invalidating device - * private page mappings that won't be migrated. - */ - mmu_notifier_range_init_owner(&range, MMU_NOTIFY_MIGRATE, 0, - migrate->vma->vm_mm, migrate->start, migrate->end, - migrate->pgmap_owner); - mmu_notifier_invalidate_range_start(&range); - - walk_page_range_vma(migrate->vma, migrate->start, migrate->end, - &migrate_vma_walk_ops, migrate); - - mmu_notifier_invalidate_range_end(&range); - migrate->end =3D migrate->start + (migrate->npages << PAGE_SHIFT); -} - /* * migrate_vma_check_page() - check if page is pinned or not * @page: struct page to check @@ -677,8 +164,9 @@ static void migrate_vma_unmap(struct migrate_vma *migra= te) * migrate_vma_setup() - prepare to migrate a range of memory * @args: contains the vma, start, and pfns arrays for the migration * - * Returns: negative errno on failures, 0 when 0 or more pages were migrat= ed - * without an error. + * Returns: -EINVAL on invalid arguments, 0 otherwise. A 0 return does not + * imply anything was collected; on return @args->cpages holds the number = of + * pages prepared for migration (possibly 0). See the note on faulting bel= ow. * * Prepare to migrate a range of memory virtual address range by collectin= g all * the pages backing each virtual address in the range, saving them inside= the @@ -688,6 +176,20 @@ static void migrate_vma_unmap(struct migrate_vma *migr= ate) * corresponding src array entry. Then restores any pages that are pinned= , by * remapping and unlocking those pages. * + * By default migrate_vma_setup() is best-effort: virtual addresses that c= annot + * be collected are left with an empty src[] entry and the rest of the ran= ge + * still proceeds. + * + * The optional flags MIGRATE_VMA_FAULT and MIGRATE_VMA_WRITE in @args->fl= ags + * additionally fault in missing pages before collecting them, for read an= d for + * write respectively. Unlike the best-effort default, if such a fault can= not be + * satisfied the whole range is aborted: the partial collection is rolled = back, + * @args->src is cleared, @args->cpages and @args->npages are reset to 0, + * and migrate_vma_setup() still returns 0. The fault error is therefore n= ot + * observable through this interface. Callers that need to act on fault fa= ilures + * should instead drive the migration directly via hmm_range_fault() follo= wed by + * migrate_hmm_range_setup(), which preserves the error code. + * * The caller should then allocate destination memory and copy source memo= ry to * it for all those entries (ie with MIGRATE_PFN_VALID and MIGRATE_PFN_MIG= RATE * flag set). Once these are allocated and copied, the caller must update= each @@ -739,10 +241,20 @@ static void migrate_vma_unmap(struct migrate_vma *mig= rate) */ int migrate_vma_setup(struct migrate_vma *args) { + int ret; long nr_pages =3D (args->end - args->start) >> PAGE_SHIFT; + struct hmm_range range =3D { + .notifier =3D NULL, + .hmm_pfns =3D args->src, + .dev_private_owner =3D args->pgmap_owner, + .migrate =3D args, + .default_flags =3D HMM_PFN_REQ_MIGRATE + }; =20 args->start &=3D PAGE_MASK; args->end &=3D PAGE_MASK; + range.start =3D args->start; + range.end =3D args->end; if (!args->vma || is_vm_hugetlb_page(args->vma) || (args->vma->vm_flags & VM_SPECIAL) || vma_is_dax(args->vma)) return -EINVAL; @@ -764,10 +276,24 @@ int migrate_vma_setup(struct migrate_vma *args) args->cpages =3D 0; args->npages =3D 0; =20 - migrate_vma_collect(args); + if (args->flags & MIGRATE_VMA_FAULT) + range.default_flags |=3D HMM_PFN_REQ_FAULT; =20 - if (args->cpages) - migrate_vma_unmap(args); + if (args->flags & MIGRATE_VMA_WRITE) + range.default_flags |=3D HMM_PFN_REQ_FAULT | HMM_PFN_REQ_WRITE; + + ret =3D hmm_range_fault(&range); + + migrate_hmm_range_setup(&range); + + /* Remove migration PTEs */ + if (ret) { + migrate_vma_pages(args); + migrate_vma_finalize(args); + memset(args->src, 0, sizeof(*args->src) * nr_pages); + args->cpages =3D 0; + args->npages =3D 0; + } =20 /* * At this point pages are locked and unmapped, and thus they have --=20 2.55.0 From nobody Fri Sep 25 18:51:54 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8BAA548EC67 for ; Tue, 22 Sep 2026 05:35:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055322; cv=none; b=nnXgerJkPRGRxxxTplfj2BLnXvh6r4rXxH2UYgHXS08FZ8aCNPDVRd7MgVnXlwKC60cU3n0nwWuYK7dUmrQGxDBNLqMOT8F/oMkXxrd+SULTdiYqEKnwNNW0QNuW0VV/7cxNIo7UE4uAO2AOHKjImRIXV4MM+RvqG0ir2UUuijY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055322; c=relaxed/simple; bh=jTTuFQ42teG21/NODquZooAEx+rVK66SJ1HWPla0emA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ngiZSt+TEgmjNGnwc0uedI3w4ZTkVW1b/xdCAP8aG1SsrBDrJU91oYa2972KbMEh29vBmmLD/fQUQ6ikSMafIuAFFrIc1+hPEMPho4TbDYUZVwS/+iaAG2C6MNJuj6Om5ve1U77jMUleg3T85QYKDWlpH8Zs5y2qAmg9vdueGhY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=gy1olaf7; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=fqx7qgbL; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="gy1olaf7"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="fqx7qgbL" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790055319; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rU6cW44LYgRUyAQrC4VYer8F2kJCybD5egmuRztFQqU=; b=gy1olaf7U49N7haoKw1ENxwNsfHuU3tpQ+XtiQfGOTpGpOVj2c7wLWmBsHQSwnzbGdzcQh Ia1aG/KjbK8uNBxKwt4WuI53F3Wgqs+/5ocuvjMQlK7uU0WIFLcdqOykVDEpfsZKYMUtbY iXQ4BK8duVGmGnckUSH5ynlxzuk4hxs= Received: from mail-lj1-f198.google.com (mail-lj1-f198.google.com [209.85.208.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-108-elHOsZOEN9qHpJ7ewt3djA-1; Tue, 22 Sep 2026 01:35:15 -0400 X-MC-Unique: elHOsZOEN9qHpJ7ewt3djA-1 X-Mimecast-MFC-AGG-ID: elHOsZOEN9qHpJ7ewt3djA_1790055314 Received: by mail-lj1-f198.google.com with SMTP id 38308e7fff4ca-3a468782b61so14482661fa.0 for ; Mon, 21 Sep 2026 22:35:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790055314; x=1790660114; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=rU6cW44LYgRUyAQrC4VYer8F2kJCybD5egmuRztFQqU=; b=fqx7qgbLZM6S/5y9s9zcfjTm3SK9IUuFbER5ja8SbIYypjpdnS9PQHO1OVXNsX6fKL A64NrMNVLKgxizi5aW/Ytsm9WnvG6r8KSNJV0u5YDbzAe8bhq/mR/BEJziHRgbP8QlN4 owUtwOxGgME1nCsfVQ/uaHX2ui1bWO39WW+TzBvEkyRm6/5nljbq2+1Ukonjo6Ckzg19 K3LXCG61imn0lGNyk8gyQdmS72f+4UqAdMCP7ykWcR6HX7eW5so8k4ayg2mIuRLJQfuU zzLVqjYfJaRrd1bB08otqaedWgWsAQUu4patPO1De7fUzqqCi5HVGnfDaTammrOwd8mZ tbGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790055314; x=1790660114; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=rU6cW44LYgRUyAQrC4VYer8F2kJCybD5egmuRztFQqU=; b=D/qHRocEjESziQRmK/0ncFCnJHQFFSo32PB9/ShntSAqwwpqRoSh2yyaT+5UUgFHWy YFOf7VV40phjSp1RSE3xLU/Rlym56Sh+dpVGj5RC39JHKAU77vpCGrp4JESE87NkUd5F R8qtrsE8C3KAfuo0gt0DUBacdr96mByG4lpBkwByxzF/wL6Y6rZB3up3AWeUoX6ugJ73 jf8/0m47GACi9RDqxH51zh87evOM2sB06ynZgadC/CKUT9uj6nsR8TDfEFVVQ7hq6tWt ER7mzDUHZi19WrTsjjaTHRAVZG5rYPRq4nyDDcVQi5zmdHpKt5OXGzHhB/KlMgqxvm4W aJmg== X-Forwarded-Encrypted: i=1; AKwUvByoR2r29PyKqiQz8tJk3xMp68j1KhPkg1yyXRFjan1hsjLD9TriuorjOC7DKwT5zT81VwryjpJmgnu8hJc=@vger.kernel.org X-Gm-Message-State: AFuF++ko2ChaEEngrNE2yjFwO8REDs0B1q+cp9QaeJVYHtTOqUjIovUV bIeAznG/LQxIMJWTwDyx/6Tx+qW/Jie9jfa95YhL4GhgCIPmcExuAZIKKaQgDQbCTokb0QtRy98 oWTT53EYqF7tR740EdfPsc/N+zoMMLhAenVhvKjjbEnllQvWi71b1dpNlhqbOtGYC X-Gm-Gg: AYBFou3ogMPBDojwE/NILkge3NHYBBnNgMFgDl+3db45927+cKJNhsS5eHvr1nc6l3e r48tAyRVFRNsxUwTOHzmryjHy4KUtVvi9CllCMf10hvf/PMuBbN4fBf0KZzEjBxbJFia2msYXED NhJyNxuy7zT0FkZIrshBaZVu0s8sihK+81qdTBG1kIge61BfwVJ/yRZyha3WUJ/hTViH29ow3JB sPI3sxYzgAFrC42kwkgl9iapkbJp8sky6gLIHSEhVrMfQpsICYDbnhDZeOi17R0aw1g9WeutMs+ NG2F30tYRXGNVku2GT5Qpaukxz+QCWcWzSZ8lKwxDzCl+OKQ5Lr+ZqMyaEgyB8dJiHouoajT2ae 9tn0v04ak6bHhrGQQqcIA X-Received: by 2002:a05:6512:33c6:b0:5b6:183c:5c9d with SMTP id 2adb3069b0e04-5b8c1958e1fmr3832808e87.59.1790055314053; Mon, 21 Sep 2026 22:35:14 -0700 (PDT) X-Received: by 2002:a05:6512:33c6:b0:5b6:183c:5c9d with SMTP id 2adb3069b0e04-5b8c1958e1fmr3832800e87.59.1790055313530; Mon, 21 Sep 2026 22:35:13 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b8d46e0966sm191742e87.55.2026.09.21.22.35.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 22:35:12 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Marco Pagani Subject: [PATCH v14 11/12] lib/test_hmm: add a new testcase for the migrate on fault Date: Tue, 22 Sep 2026 08:34:20 +0300 Message-ID: <20260922053421.4092027-12-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922053421.4092027-1-mpenttil@redhat.com> References: <20260922053421.4092027-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 Enhance the hmm test driver (lib/test_hmm) with migrate on fault case. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Signed-off-by: Marco Pagani Signed-off-by: Mika Penttil=C3=A4 --- lib/test_hmm.c | 130 ++++++++++++++++++++++++- lib/test_hmm_uapi.h | 21 ++-- tools/testing/selftests/mm/hmm-tests.c | 54 ++++++++++ 3 files changed, 191 insertions(+), 14 deletions(-) diff --git a/lib/test_hmm.c b/lib/test_hmm.c index cd88e8177d0a..b39522c05d83 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -36,6 +36,7 @@ #define DMIRROR_RANGE_FAULT_TIMEOUT 1000 #define DEVMEM_CHUNK_SIZE (256 * 1024 * 1024U) #define DEVMEM_CHUNKS_RESERVE 16 +#define PFNS_ARRAY_SIZE 64 =20 /* * For device_private pages, dpage is just a dummy struct page @@ -355,6 +356,7 @@ static int dmirror_range_fault(struct dmirror *dmirror, struct mm_struct *mm =3D dmirror->notifier.mm; unsigned long timeout =3D jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT); + bool migrate =3D range->default_flags & HMM_PFN_REQ_MIGRATE; int ret; =20 while (true) { @@ -364,9 +366,15 @@ static int dmirror_range_fault(struct dmirror *dmirror, } =20 range->notifier_seq =3D mmu_interval_read_begin(range->notifier); - mmap_read_lock(mm); - ret =3D hmm_range_fault(range); - mmap_read_unlock(mm); + + /* mmap lock held for whole migrate on fault operation */ + if (!migrate) { + mmap_read_lock(mm); + ret =3D hmm_range_fault(range); + mmap_read_unlock(mm); + } else { + ret =3D hmm_range_fault(range); + } if (ret) { if (ret =3D=3D -EBUSY) continue; @@ -382,7 +390,9 @@ static int dmirror_range_fault(struct dmirror *dmirror, break; } =20 - ret =3D dmirror_do_fault(dmirror, range); + /* update device page table after migration */ + if (!migrate) + ret =3D dmirror_do_fault(dmirror, range); =20 mutex_unlock(&dmirror->mutex); out: @@ -1377,6 +1387,114 @@ static int dmirror_migrate_to_device(struct dmirror= *dmirror, return ret; } =20 +static int do_fault_and_migrate(struct dmirror *dmirror, struct hmm_range = *range) +{ + struct migrate_vma *migrate =3D range->migrate; + int ret; + + mmap_read_lock(dmirror->notifier.mm); + + /* Fault-in pages for migration */ + ret =3D dmirror_range_fault(dmirror, range); + /* + * Set this up even on error: hmm_range_fault() may have collected + * part of the range before failing. + */ + migrate_hmm_range_setup(range); + if (ret) { + /* + * dst[] entries are empty, so this marks every collected migration + * as failed. finalize then restores the source mappings and drops + * the associated locks/references. + */ + migrate_vma_pages(migrate); + migrate_vma_finalize(migrate); + goto out; + } + + pr_debug("Migrating from sys mem to device mem\n"); + + dmirror_migrate_alloc_and_copy(migrate, dmirror); + migrate_vma_pages(migrate); + dmirror_migrate_finalize_and_map(migrate, dmirror); + migrate_vma_finalize(migrate); +out: + mmap_read_unlock(dmirror->notifier.mm); + return ret; +} + +static int dmirror_fault_and_migrate_to_device(struct dmirror *dmirror, + struct hmm_dmirror_cmd *cmd) +{ + unsigned long start, size, end, next; + unsigned long src_pfns[PFNS_ARRAY_SIZE] =3D { 0 }; + unsigned long dst_pfns[PFNS_ARRAY_SIZE] =3D { 0 }; + struct migrate_vma migrate =3D { 0 }; + struct hmm_range range =3D { 0 }; + struct dmirror_bounce bounce; + int ret =3D 0; + + /* Whole range */ + start =3D cmd->addr; + size =3D cmd->npages << PAGE_SHIFT; + end =3D start + size; + + if (!mmget_not_zero(dmirror->notifier.mm)) { + ret =3D -EFAULT; + goto out; + } + + migrate.pgmap_owner =3D dmirror->mdevice; + migrate.src =3D src_pfns; + migrate.dst =3D dst_pfns; + migrate.flags =3D MIGRATE_VMA_SELECT_SYSTEM; + + range.migrate =3D &migrate; + range.hmm_pfns =3D src_pfns; + range.pfn_flags_mask =3D 0; + range.default_flags =3D HMM_PFN_REQ_FAULT | HMM_PFN_REQ_MIGRATE; + range.dev_private_owner =3D dmirror->mdevice; + range.notifier =3D &dmirror->notifier; + + for (next =3D start; next < end; next =3D range.end) { + range.start =3D next; + range.end =3D min(end, next + (PFNS_ARRAY_SIZE << PAGE_SHIFT)); + + pr_debug("Fault and migrate range start:%#lx end:%#lx\n", + range.start, range.end); + + ret =3D do_fault_and_migrate(dmirror, &range); + if (ret) + goto out_mmput; + } + + /* + * Return the migrated data for verification. + * Only for pages in device zone + */ + ret =3D dmirror_bounce_init(&bounce, start, size); + if (ret) + goto out_mmput; + + mutex_lock(&dmirror->mutex); + ret =3D dmirror_do_read(dmirror, start, end, &bounce); + mutex_unlock(&dmirror->mutex); + if (ret =3D=3D 0) { + ret =3D copy_to_user(u64_to_user_ptr(cmd->ptr), bounce.ptr, bounce.size); + if (ret) + ret =3D -EFAULT; + } + + cmd->cpages =3D bounce.cpages; + dmirror_bounce_fini(&bounce); + + +out_mmput: + mmput(dmirror->notifier.mm); +out: + return ret; +} + static void dmirror_mkentry(struct dmirror *dmirror, struct hmm_range *ran= ge, unsigned char *perm, unsigned long entry) { @@ -1643,6 +1761,10 @@ static long dmirror_fops_unlocked_ioctl(struct file = *filp, ret =3D dmirror_migrate_to_device(dmirror, &cmd); break; =20 + case HMM_DMIRROR_MIGRATE_ON_FAULT_TO_DEV: + ret =3D dmirror_fault_and_migrate_to_device(dmirror, &cmd); + break; + case HMM_DMIRROR_MIGRATE_TO_SYS: ret =3D dmirror_migrate_to_system(dmirror, &cmd); break; diff --git a/lib/test_hmm_uapi.h b/lib/test_hmm_uapi.h index ea9b0ec404fb..67d267a3b999 100644 --- a/lib/test_hmm_uapi.h +++ b/lib/test_hmm_uapi.h @@ -29,16 +29,17 @@ struct hmm_dmirror_cmd { }; =20 /* Expose the address space of the calling process through hmm device file= */ -#define HMM_DMIRROR_READ _IOWR('H', 0x00, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_WRITE _IOWR('H', 0x01, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_MIGRATE_TO_DEV _IOWR('H', 0x02, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_MIGRATE_TO_SYS _IOWR('H', 0x03, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_SNAPSHOT _IOWR('H', 0x04, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_EXCLUSIVE _IOWR('H', 0x05, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_CHECK_EXCLUSIVE _IOWR('H', 0x06, struct hmm_dmirror_cm= d) -#define HMM_DMIRROR_RELEASE _IOWR('H', 0x07, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_FLAGS _IOWR('H', 0x08, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_READ_UNLOCKED _IOWR('H', 0x09, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_READ _IOWR('H', 0x00, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_WRITE _IOWR('H', 0x01, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_MIGRATE_TO_DEV _IOWR('H', 0x02, struct hmm_dmirror_cm= d) +#define HMM_DMIRROR_MIGRATE_TO_SYS _IOWR('H', 0x03, struct hmm_dmirror_cm= d) +#define HMM_DMIRROR_SNAPSHOT _IOWR('H', 0x04, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_EXCLUSIVE _IOWR('H', 0x05, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_CHECK_EXCLUSIVE _IOWR('H', 0x06, struct hmm_dmirror_c= md) +#define HMM_DMIRROR_RELEASE _IOWR('H', 0x07, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_FLAGS _IOWR('H', 0x08, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_READ_UNLOCKED _IOWR('H', 0x09, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_MIGRATE_ON_FAULT_TO_DEV _IOWR('H', 0x0a, struct hmm_dm= irror_cmd) =20 #define HMM_DMIRROR_FLAG_FAIL_ALLOC (1ULL << 0) =20 diff --git a/tools/testing/selftests/mm/hmm-tests.c b/tools/testing/selftes= ts/mm/hmm-tests.c index e2642eca0d02..a2d1b8f494b6 100644 --- a/tools/testing/selftests/mm/hmm-tests.c +++ b/tools/testing/selftests/mm/hmm-tests.c @@ -334,6 +334,13 @@ static int hmm_migrate_sys_to_dev(int fd, return hmm_dmirror_cmd(fd, HMM_DMIRROR_MIGRATE_TO_DEV, buffer, npages); } =20 +static int hmm_migrate_on_fault_sys_to_dev(int fd, + struct hmm_buffer *buffer, + unsigned long npages) +{ + return hmm_dmirror_cmd(fd, HMM_DMIRROR_MIGRATE_ON_FAULT_TO_DEV, buffer, n= pages); +} + static int hmm_migrate_dev_to_sys(int fd, struct hmm_buffer *buffer, unsigned long npages) @@ -938,6 +945,53 @@ TEST_F(hmm, migrate) hmm_buffer_free(buffer); } =20 + +/* + * Fault and migrate anonymous memory to device private memory. + */ +TEST_F(hmm, migrate_on_fault) +{ + struct hmm_buffer *buffer; + unsigned long npages; + unsigned long size; + unsigned long i; + int *ptr; + int ret; + + npages =3D ALIGN(HMM_BUFFER_SIZE, self->page_size) >> self->page_shift; + ASSERT_NE(npages, 0); + size =3D npages << self->page_shift; + + buffer =3D malloc(sizeof(*buffer)); + ASSERT_NE(buffer, NULL); + + buffer->fd =3D -1; + buffer->size =3D size; + buffer->mirror =3D malloc(size); + ASSERT_NE(buffer->mirror, NULL); + + buffer->ptr =3D mmap(NULL, size, + PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, + buffer->fd, 0); + ASSERT_NE(buffer->ptr, MAP_FAILED); + + /* Initialize buffer in system memory. */ + for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) + ptr[i] =3D i; + + /* Fault and migrate memory to device. */ + ret =3D hmm_migrate_on_fault_sys_to_dev(self->fd, buffer, npages); + ASSERT_EQ(ret, 0); + ASSERT_EQ(buffer->cpages, npages); + + /* Check what the device read. */ + for (i =3D 0, ptr =3D buffer->mirror; i < size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i], i); + + hmm_buffer_free(buffer); +} + /* * Migrate private file memory to device private memory. */ --=20 2.55.0 From nobody Fri Sep 25 18:51:54 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C6B1C48EC75 for ; Tue, 22 Sep 2026 05:35:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055324; cv=none; b=KohVoFPkL3nCLhBFvb7DZ0QIitkBPUY9oI+VF7pT4FQECzIG/I/Gyldk8G2oqGxaUB2+7RbQDv0CDIGeYICfjnb7J8sdNUAOOvXZHAX5IYj7ypsdjckc26H0orwmBAUM2+sqMPcE7zm0f29zaU9ujiQqUxkBykwdTqt5y1jturs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790055324; c=relaxed/simple; bh=G51B54owKuBfMBL6BmQ3fz1GoUN0Axehqo+N3yadsZc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=oIWeIFJNKiWNjqcDANOnSi5VQnbfE+Pkm6UIVKDvNIz6dnMPxhSxo5jXrAKESQFn1aIL9cbOtFKKjlZnlB1yA+bAuV6RVZnFSk7u6NVOd0qhncGh1kYQMM04WR4LT0BixNgIRl9xAks75otbTSBkcfZUG7MHImRG9WKec58D7jw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=UvASsNjw; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=JC1NNLTP; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="UvASsNjw"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="JC1NNLTP" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1790055321; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xEEzUKmwPcuLDb8KHUBfxekosp3Yw5qO1BP0yTFVDsQ=; b=UvASsNjwNk+oVdt28xpUo6LmNEH3PULy/Ax6+G6tBe+SnHIzAslC/+xJKtHVre//VJTvjO HiEgB7jdPUqotdgZnSpFHZPwwh+LRqN7HommmQDq087NIWzsTbMKSO/5L6l28dMmG6pgtd BLqRiAlGJWBtg7be+xSoCS2eJ8tkQgI= Received: from mail-lf1-f70.google.com (mail-lf1-f70.google.com [209.85.167.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-630-ilZFXAlzP5G7TnX2Buk02g-1; Tue, 22 Sep 2026 01:35:20 -0400 X-MC-Unique: ilZFXAlzP5G7TnX2Buk02g-1 X-Mimecast-MFC-AGG-ID: ilZFXAlzP5G7TnX2Buk02g_1790055319 Received: by mail-lf1-f70.google.com with SMTP id 2adb3069b0e04-5b8c58cd8c0so2124869e87.2 for ; Mon, 21 Sep 2026 22:35:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1790055318; x=1790660118; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=xEEzUKmwPcuLDb8KHUBfxekosp3Yw5qO1BP0yTFVDsQ=; b=JC1NNLTPtStO2FogkAZ/ONMYy2vmTxuTeNtCT4SVAHeYDEzXdskDmrzIiojYbtEryk O9RUyzYmfJQjig4iu76Eagq01MvZ25gF8t/X9+g6I+3Ljj1XgtD5IibLYu8PDUSmaWgx oUe/wMhuSI7HcBul+60xWj4XmY52pj4ASAgRzGX5jU1GyRH34IoQxJaC5AIgzoEvM/1x 4+5wcdvoXMGctqSE+iVXVUK4yVWzEd4wL4/bCK0SDjOy+xknZg1B77JmZHS7lp8IWwmd cmBwp7Skm+2aE7mFtDcw1dG+MuGox7iJ9fD6lo9zxc2UhaZgJ6sYFyK74nnLMoLBeQ4P 6IPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790055318; x=1790660118; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=xEEzUKmwPcuLDb8KHUBfxekosp3Yw5qO1BP0yTFVDsQ=; b=WVnLiT5WsStLfIy5YFMxwOCZEOK/ysS3K0ZJzaLom++buP7xSFBpxpiZxxxGd7OmbT 2Vrj238wAnuCzW5JvYNNxO/IAdeBr6J4gdrsf0CeHLywBg2RtdTKkMWTVlK9fcuTZ7xo RwJyObWZJ8lvd579CLYmlZJgvQIsf9xHTq+ulpYX43qZgRatktQG2JO8E2jhAnMQ/l6E 0eK4Wpe1udk1tPd6YnoI3LZzbaDnnse7cMGkfIoIz9Cy8cWWdtUm24x//afJwmlRg4Hn 80z+veFX+Sdjlr2jW8PhMIqzdufu85uqKXhTcOtJwAxkcZVy5YwoiWTSwhv5/4WRHjzt WPgg== X-Forwarded-Encrypted: i=1; AKwUvBzYFurU+JOlRVM2WqXZRawFpEOioFv0l3My7quP9H6WPjw7iXAyfy/wCmC1DLzNfUoHKrZyxbWzwgDh9F0=@vger.kernel.org X-Gm-Message-State: AFuF++mccidNy9y2QSeqZ/Hu4f+1Qm5PrrKjR/DE/VzYh79vIpplm8XY mj5SrTSGxVXJcVqjX8fr+EeKB+tRCcTUp5he42xFkNWqi02MNhrr3/MrCdaK3RkbIuLk55MFOS7 AzcDp76kNf8K0KoMT5HlNchvlEs13q/DR92pA2YJEsn7SHIrU47f2y5WdSghmI2Of X-Gm-Gg: AYBFou1JxFrB1DvZ5aLw27sqbZFqoUATewA9wnIVoQ/S991Z371xLTOJUkUr/rHnd/c tCp1+Qob/5K2ha8J+RxvZgbh6FAVMUAtRsMbchZZgiDy92JQ4CsdguBylyET/8Q9+XPJuPEDpDH otZeiOtEUQ3OmqYv1s4JhGWutBb3ThHM35Kq98cR6n9X8vtkVQiEAw00V1QiFy1KW4xlcq3mSrJ O1IWNfr1Ze0fqTilRn5fi++I9ZS3NgJHu7etZn8KbgPacdC7cuux8REHH1wAg6X4wzYJbQBybEp dBlYTIPcn0+sMrJFvvzcxb1V6y/Rr92b9SbI6+Var6pAQ7N85+GPw0HGZ9iQVJk6DiAezkv18qi KwERaE2Acyu6RdZovKcnK X-Received: by 2002:ac2:51c3:0:b0:5b5:e86f:3550 with SMTP id 2adb3069b0e04-5b8c1833648mr3732748e87.9.1790055318591; Mon, 21 Sep 2026 22:35:18 -0700 (PDT) X-Received: by 2002:ac2:51c3:0:b0:5b5:e86f:3550 with SMTP id 2adb3069b0e04-5b8c1833648mr3732729e87.9.1790055318145; Mon, 21 Sep 2026 22:35:18 -0700 (PDT) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b8d46e0966sm191742e87.55.2026.09.21.22.35.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 22:35:15 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v14 12/12] Documentation/mm/hmm: document migration through hmm_range_fault() Date: Tue, 22 Sep 2026 08:34:21 +0300 Message-ID: <20260922053421.4092027-13-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922053421.4092027-1-mpenttil@redhat.com> References: <20260922053421.4092027-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Mika Penttil=C3=A4 Describe the two new MIGRATE_VMA_FAULT / MIGRATE_VMA_WRITE flags of migrate_vma_setup(), and add a section on driving the migration collection phase directly from hmm_range_fault() via HMM_PFN_REQ_MIGRATE and migrate_hmm_range_setup(). Signed-off-by: Mika Penttil=C3=A4 --- Documentation/mm/hmm.rst | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/Documentation/mm/hmm.rst b/Documentation/mm/hmm.rst index fc1b8dc19825..58424e5f5873 100644 --- a/Documentation/mm/hmm.rst +++ b/Documentation/mm/hmm.rst @@ -348,6 +348,13 @@ between device driver specific code and shared common = code: Currently only anonymous private VMA ranges can be migrated to or from system memory and device private memory. =20 + By default only pages already present are collected. Two additional fla= gs + ask migrate_vma_setup() to fault in missing pages first: + + * ``MIGRATE_VMA_FAULT`` faults in missing pages with read access. + * ``MIGRATE_VMA_WRITE`` faults in missing pages with write access + (implies faulting). + One of the first steps migrate_vma_setup() does is to invalidate other device's MMUs with the ``mmu_notifier_invalidate_range_start()`` and ``mmu_notifier_invalidate_range_end()`` calls around the page table @@ -427,6 +434,38 @@ between device driver specific code and shared common = code: =20 The lock can now be released. =20 +Migration collection through hmm_range_fault() +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The collection phase of migration (steps 1 and 2 above) can also be driven= by +hmm_range_fault() directly, sharing its page table walk. This lets a driver +fault in and collect a range for migration in one walk, which is useful for +migrate on fault. + +To do so, the driver sets ``HMM_PFN_REQ_MIGRATE`` in ``range->default_flag= s`` +and points ``range->migrate`` at a ``struct migrate_vma`` it has filled in +(``flags``, ``src``, ``dst``, ``pgmap_owner``). Usually ``HMM_PFN_REQ_FAUL= T`` +(and ``HMM_PFN_REQ_WRITE``) is set as well, so missing pages are faulted in +before being collected. The mmap_read_lock() has to be held for the whole +migration, since the vma must stay stable. + +hmm_range_fault() collects the entries the same way migrate_vma_setup() do= es, +taking a folio reference, locking it and installing a migration PTE. Colle= cted +entries are marked with ``HMM_PFN_VALID | HMM_PFN_MIGRATE`` in +``range->hmm_pfns``. If the page tables change while locks are dropped the +partially collected entries are rolled back automatically. + +After hmm_range_fault() returns, the driver calls:: + + void migrate_hmm_range_setup(struct hmm_range *range); + +to translate ``range->hmm_pfns`` into ``migrate->src[]`` (``migrate->dst[]= `` is +zeroed) and initialize ``migrate->cpages`` and ``migrate->npages``. From h= ere on +the ``range->migrate`` struct is ready for the rest of the flow (steps 3 +onwards), i.e. migrate_vma_pages() and migrate_vma_finalize(). This should= be +called even on error, since hmm_range_fault() may have collected part of t= he +range before failing. + Exclusive access memory =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 --=20 2.55.0