From nobody Mon Aug 24 08:07:33 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A59D481A9A for ; Wed, 5 Aug 2026 23:10:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785971454; cv=none; b=Hbyju9x3YUEOn5jImLpxEv5Fi3rA2HFZrEuVfyZLBb0mpgi9PrNxxQcInG9wCzjYRrLD6EfNr/DDCWW//UAZGDY/PYxDItzTQ3cTWHryisaZ+qHyiz/MPiWHUTd7u+t35bQPLhnnKp5QGFYVYMbQait/6Y060+aX7LpDugCv9A8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785971454; c=relaxed/simple; bh=dNJ/oi0EebM2qITDTJ9UJ0BoOZBi/aCxva+AfF14at8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=XWuj7ajBDx+pZSE+f4C0n382g1iVC62rCIpxljWSIepAn4Rpk6UrTg5SF4G7zPPPy8wf94zHL58k45ZLYJyyiAR0JgYxt09bNzW1Uj5G/ZkQOUkSbcSJhBOzKun/5KjVr0a20k31Ll0F8qwZ7mahleVID5jxC2lYklMkNhgZiy4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Mo0gf365; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Mo0gf365" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1785971451; x=1817507451; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dNJ/oi0EebM2qITDTJ9UJ0BoOZBi/aCxva+AfF14at8=; b=Mo0gf365n/KhVFQsPA0udEeuRib0ao4X4g8V/J/0uVaIHjRpLr+jP1tJ mlNCqY9oBGv3m9ILkUIcbyoFA5qci1I2LrcMWzOtLs0jA1RdTskyijXVk I7UjVNxRdezAufqyZum6AQ91NkjvyX+cSeQzobGGo4GAGvg9cqc7ddiex zZhqD1jk7BXarm2QwlAnEc80jhiG6yvPlIxLgML1Xt1Kn30mM40C32JC5 6DSN5ygDfbBrqw/50ShtIL49qyuLRKfBDOKJyC2rgapNC1xJCvRitsCpv aotU6AUEG1zSeUh6oTLHMDu+LeVOyxzg4i3FpMY8s4qeM1mVAOvGYnbgt g==; X-CSE-ConnectionGUID: m5diZvaUQSCIoJyDZgHzUg== X-CSE-MsgGUID: +cq1h87NSISbn2ogTU4EdA== X-IronPort-AV: E=McAfee;i="6800,10657,11866"; a="90376832" X-IronPort-AV: E=Sophos;i="6.25,207,1779174000"; d="scan'208";a="90376832" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Aug 2026 16:10:49 -0700 X-CSE-ConnectionGUID: gdzBu8tUSsWVVT+WYsw11w== X-CSE-MsgGUID: Douz6uonSw2oqZmSx0TiGg== X-ExtLoop1: 1 Received: from gsse-cloud1.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Aug 2026 16:10:48 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Arvind Yadav , Andrew Morton , David Hildenbrand , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Zi Yan , Balbir Singh Subject: [PATCH v3 1/6] mm/migrate_device: Clear stale mapping after freeing swapcache Date: Wed, 5 Aug 2026 16:10:36 -0700 Message-Id: <20260805231041.3791771-2-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260805231041.3791771-1-matthew.brost@intel.com> References: <20260805231041.3791771-1-matthew.brost@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Arvind Yadav __migrate_device_pages() reads the folio mapping before calling folio_free_swap(). When folio_free_swap() succeeds, the folio is removed from the swap cache, but the saved mapping still points to swap_space. Passing the stale mapping to folio_migrate_mapping() makes it use the mapped-folio path for a folio that is no longer in swapcache. It can then operate on swap_space.i_pages with invalid reference accounting, eventually triggering a folio reference count BUG. After a successful split, nr still contains the number of pages in the original large folio, although each resulting page is now a separate order-0 folio. Reset nr to 1 so each split folio is processed separately, including its own swapcache removal and mapping lookup. Refresh the saved mapping after folio_free_swap() so the current folio state is used during migration. v2: - Refresh the mapping using folio_mapping(), as suggested by Zi Yan. v3: - Reset nr to 1 after a successful split so each resulting folio is processed independently, as suggested by Zi Yan. v4: - Re-read each source folio's mapping immediately before folio_migrate_mapping(), as suggested by Balbir Singh. - Add a warning to validate the post-split order-0 invariant, as suggested by Balbir Singh. Fixes: df263d9a7dff ("mm/migrate_device: try to handle swapcache pages") Cc: Andrew Morton Cc: David Hildenbrand Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Reviewed-by: Zi Yan Reviewed-by: Balbir Singh Signed-off-by: Arvind Yadav --- mm/migrate_device.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 908d2d4ec43a..162d29b2807a 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -1183,6 +1183,13 @@ static void __migrate_device_pages(unsigned long *sr= c_pfns, MIGRATE_PFN_COMPOUND); goto next; } + + /* + * reset nr so that only first after-split folio + * is processed below + */ + VM_WARN_ON_ONCE(folio_test_large(folio)); + nr =3D 1; } else if ((src_pfns[i] & MIGRATE_PFN_MIGRATE) && (dst_pfns[i] & MIGRATE_PFN_COMPOUND) && !(src_pfns[i] & MIGRATE_PFN_COMPOUND)) { @@ -1222,6 +1229,12 @@ static void __migrate_device_pages(unsigned long *sr= c_pfns, folio =3D page_folio(migrate_pfn_to_page(src_pfns[i+j])); newfolio =3D page_folio(migrate_pfn_to_page(dst_pfns[i+j])); =20 + /* + * folio_free_swap() removed the folio from the swap + * cache. Refresh the saved mapping before migration. + */ + mapping =3D folio_mapping(folio); + r =3D folio_migrate_mapping(mapping, newfolio, folio, extra_cnt); if (r) src_pfns[i+j] &=3D ~MIGRATE_PFN_MIGRATE; --=20 2.34.1 From nobody Mon Aug 24 08:07:33 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A013E3EA97E; Wed, 5 Aug 2026 23:10:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785971456; cv=none; b=uNnBumisf8uYuZaJbJgmyEOlolBYib+ZiE14ISaymKn5Rln8sCXUbMrXVhPqIuuWWcjDNjv/AIxgf7vfIPPCY85AVDJRq4tXvSTzEjD2Fh16NHcR2i8DvjQZmorY5itfPYwg/F7cOIdwXEw1yAbnraryUhfFghRXVh9cwD8RWYg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785971456; c=relaxed/simple; bh=/Jjq0f7O0WvJkLs4JYtXmw67Rzu5MIrXEQREZTu1NqM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=uD7TWf2q/WYMwd5TJGWsI+PktQ9N8YLkT2XaXf7xoKdkFtQOzczDkdT1KurvFE+PQQb3y3iSwxoswSoslUgAVzOVOexzsfIKymnhTBf08idY5xHH6AxBMl5vHxsUFqnKw7E1oH1lSBc7B+iIFfLgfcIg7ddgdwHyFxZRp8yhtmA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=RBJPhayk; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="RBJPhayk" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1785971452; x=1817507452; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=/Jjq0f7O0WvJkLs4JYtXmw67Rzu5MIrXEQREZTu1NqM=; b=RBJPhaykjoIPurZCdOYS7QJ5Js8FYO/W98xrqE8Leh8/1e+k9uXMh9mZ zeUYwtHoTS+q2dla92O3bo4Rg1I9ANHEI+9L2IzWoTUEef9CEWBw1dYRp eaHXSYOSghB877HnsuqJV02chhIgyYbwQr1qTWcuhr/qEzjNofev7aGZl LsktUfhwmm7n0XADzwI3yfYCJTHnVdM45ZE2kJTNjRvwLYf5nGHxRHZk2 tE/ZUmIuM5uUFelI0ITDHv2At9y4QYBAGCt7PoUbvViC4VeRJ4Y+XLlOD 3jxrLABCxhWgHvphAfoDVrId4Ndorg1W8aeifn3lVdtNwhSb1C17rP/iE w==; X-CSE-ConnectionGUID: v3R2amyfSwWyXpUTqsLXIA== X-CSE-MsgGUID: CIixlMg1SBGDCBjFU+7P/A== X-IronPort-AV: E=McAfee;i="6800,10657,11866"; a="90376843" X-IronPort-AV: E=Sophos;i="6.25,207,1779174000"; d="scan'208";a="90376843" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Aug 2026 16:10:49 -0700 X-CSE-ConnectionGUID: dbZ5mZZVTkeTIfdkt0zbgA== X-CSE-MsgGUID: 9Pr+QSRTQgSOCeG8Ps6dLQ== X-ExtLoop1: 1 Received: from gsse-cloud1.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Aug 2026 16:10:49 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sashiko , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Balbir Singh , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter , =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , Francois Dugast , stable@vger.kernel.org Subject: [PATCH v3 2/6] mm/migrate_device: Do not write past the end of the src_pfns array Date: Wed, 5 Aug 2026 16:10:37 -0700 Message-Id: <20260805231041.3791771-3-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260805231041.3791771-1-matthew.brost@intel.com> References: <20260805231041.3791771-1-matthew.brost@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable migrate_device_range() and migrate_device_pfns() zero the tail entries of a large folio without checking them against @npages: for (j =3D 1; j < nr; j++) src_pfns[i+j] =3D 0; @nr comes from the folio, not from the array, so a folio that extends past the end of the range being migrated writes beyond src_pfns[]. Callers size that array for @npages entries, so this corrupts whatever follows it. Bound the loop by @npages. The subsequent "i +=3D j - 1" still terminates the outer loop correctly: on a bounded exit j is @npages - i, leaving i at @npages after the increment. Reported-by: Sashiko Fixes: a30b48bf1b24 ("mm/migrate_device: implement THP migration of zone de= vice pages") Cc: Andrew Morton Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Zi Yan Cc: Baolin Wang Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Balbir Singh Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Simona Vetter Cc: Thomas Hellstr=C3=B6m Cc: Francois Dugast Cc: dri-devel@lists.freedesktop.org Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Cc: stable@vger.kernel.org Assisted-by: GitHub_Copilot:claude-opus-5 Signed-off-by: Matthew Brost Reviewed-by: Balbir Singh --- mm/migrate_device.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 162d29b2807a..ae9027421b80 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -1415,7 +1415,7 @@ int migrate_device_range(unsigned long *src_pfns, uns= igned long start, nr =3D folio_nr_pages(folio); if (nr > 1) { src_pfns[i] |=3D MIGRATE_PFN_COMPOUND; - for (j =3D 1; j < nr; j++) + for (j =3D 1; j < nr && (i + j) < npages; j++) src_pfns[i+j] =3D 0; i +=3D j - 1; pfn +=3D j - 1; @@ -1449,7 +1449,7 @@ int migrate_device_pfns(unsigned long *src_pfns, unsi= gned long npages) nr =3D folio_nr_pages(folio); if (nr > 1) { src_pfns[i] |=3D MIGRATE_PFN_COMPOUND; - for (j =3D 1; j < nr; j++) + for (j =3D 1; j < nr && (i + j) < npages; j++) src_pfns[i+j] =3D 0; i +=3D j - 1; } --=20 2.34.1 From nobody Mon Aug 24 08:07:33 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 065A2481AB0; Wed, 5 Aug 2026 23:10:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785971460; cv=none; b=uctQhyK3U6ETBVC4bQtmKl8e/SVmxq3KowDej76lbxEEoa57zMe8CuEYDesadkvIezfMWkTOWSsMZoxleZM8HJ49NQhMq9GNs5HAZCsJDbgJJHVNiG81MZeyw8qRmJtmAomJjdbJK/ZnTkv4BeQ0lcQ1EMkG2FHu8cwDKcHFwuA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785971460; c=relaxed/simple; bh=nJTeZC7RsvVra/mCHHpW5c/cjGj3D4a/B232Qmgslhg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=c0D+QgGW9eyPy6/qbv02HntrVKJldNJr/BkytUaNZ2j8w5H2NDSZ+GOQ9tyQNstyN2aPxG0ycwzu6p6n/v6F2RCyLO8gaVT5oYl7MrIV1fbMiMS9TWP/UXNazzjU9zoFneMh+BaRwJPHmLObtR4TW+j+As+qgwzAnz2GeUps64M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=CpXdKVYZ; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="CpXdKVYZ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1785971455; x=1817507455; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=nJTeZC7RsvVra/mCHHpW5c/cjGj3D4a/B232Qmgslhg=; b=CpXdKVYZw0ZIdG7bc4MYLaPDXOOnf9TTZP4iNvZ1DrIon7Niatt245Q0 fGuSvMlIYuV3jqMN4pNeAp2Y2MdQG2qC7Mr3+UmH+UlqK896mev3Val+H 8xeGKD4pHkOyB+jvjYDqr5sJwRXG9IeI4eSb0d354NsxmZjfz6vuhwgxD EweDBk2uaK0Y+bSb/ov3mlpaSizaqGwthj1mSUBoDtENgDJMGfe8vT701 r3XPbQx7P+2fQZGprJ8JVIxF8EcojtYF/27kwuxi16ypC+l3PnhkWxNEX GZ47gtB7UAE1TsG+ZuiYDT6u3Qi0R1Gju6VjApusyrAW6t5xuIFHiHxSo A==; X-CSE-ConnectionGUID: GjEJVH5lRWGLZuHPaiYvmQ== X-CSE-MsgGUID: fZYbNVWqQem5fx01XkH0+A== X-IronPort-AV: E=McAfee;i="6800,10657,11866"; a="90376858" X-IronPort-AV: E=Sophos;i="6.25,207,1779174000"; d="scan'208";a="90376858" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Aug 2026 16:10:50 -0700 X-CSE-ConnectionGUID: LCw9d/qKTamCBaXMnbTsfg== X-CSE-MsgGUID: cCa1JkdvTTm7q16LPZCmyg== X-ExtLoop1: 1 Received: from gsse-cloud1.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Aug 2026 16:10:49 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Balbir Singh , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter , =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , Francois Dugast , stable@vger.kernel.org Subject: [PATCH v3 3/6] mm/migrate_device: Fix THP splitting of a CPU faulted device private folio Date: Wed, 5 Aug 2026 16:10:38 -0700 Message-Id: <20260805231041.3791771-4-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260805231041.3791771-1-matthew.brost@intel.com> References: <20260805231041.3791771-1-matthew.brost@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable When a CPU faults on a device private PMD and the device driver can only allocate order-0 destination folios, __migrate_device_pages() has to split the source THP via migrate_vma_split_unmapped_folio(). That path is broken in two independent ways when the fault is what triggered the migration. First, the split never succeeds. At the point folio_split_unmapped() is called the folio carries two references beyond the ones it is entitled to: 1 - taken by do_huge_pmd_device_private() for the duration of the ->migrate_to_ram() callback 2 - taken by migrate_vma_collect_huge_pmd() when the folio was collected (the mapping reference having been dropped by set_pmd_migration_entry()). folio_split_unmapped() requires folio_expected_ref_count(folio) =3D=3D folio_ref_count(folio) - 1, i.e. it tolerates exactly one caller reference. With both of the above held the check sees 2 against an expected 0 and returns -EAGAIN, so the migration is abandoned and the CPU fault makes no progress. The PTE-based split path does not have this problem: migrate_vma_split_folio() is called before any collect reference is taken and explicitly skips folio_get() for the fault folio, so the fault reference is the single caller reference the split expects. Fix it by dropping the fault reference across the split and re-taking it afterwards. do_huge_pmd_device_private() derives the fault page from the PMD entry, so it is always the head page of the folio and always ends up in the head folio of an uniform split to order 0; re-taking the reference on the folio therefore puts it back exactly where do_huge_pmd_device_private() will release it. The folio cannot be freed while the reference is dropped because the collect reference is still held. Second, the folio is split globally but the page tables were demoted only locally: split_huge_pmd_address(migrate->vma, addr, true); ret =3D folio_split_unmapped(folio, 0); migrate_device_unmap() unmaps via try_to_migrate(folio, 0), deliberately without TTU_SPLIT_HUGE_PMD, so every VMA that PMD maps the folio is left holding a PMD sized migration entry. A folio that was PMD mapped in more than one VMA -- after fork(), for example -- therefore keeps huge migration entries in all the other VMAs while only migrate->vma is demoted. folio_split_unmapped() does not notice: the folio is fully unmapped, so it only looks at the refcount and happily splits to order 0. The other VMAs are then left pointing a huge PMD at an order-0 folio, and migrate_vma_finalize() -> remove_migration_ptes() walks into it: page dumped because: VM_BUG_ON_FOLIO(folio_test_hugetlb(folio) || !folio_test_pmd_mappable(folio)) kernel BUG at mm/migrate.c:368! RIP: 0010:remove_migration_pte+0x56a/0x9b0 Call Trace: rmap_walk_anon+0xfc/0x260 remove_migration_ptes+0x79/0xb0 __migrate_device_finalize+0x113/0x290 __drm_pagemap_migrate_to_ram+0x278/0x360 [drm_gpusvm_helper] drm_pagemap_migrate_to_ram+0x5c/0x80 [drm_gpusvm_helper] do_huge_pmd_device_private+0x160/0x280 Without CONFIG_DEBUG_VM the VM_BUG_ON_FOLIO() is compiled out and remove_migration_pmd() installs a huge PMD pointing at an order-0 page instead, along with add_mm_counter(mm, MM_ANONPAGES, HPAGE_PMD_NR). The victim mm then maps 2MB of address space onto a single 4K page, which shows up later as bad rss-counter state, leaked page tables and page allocator freelist corruption in unrelated processes. Note this second problem was latent before the refcount fix above: the split always failed, and the failed attempt left migrate->vma demoted, so the retried fault took the PTE path, where __folio_split() unmaps with TTU_SPLIT_HUGE_PMD and demotes every VMA. Fix it by walking the rmap and demoting every PMD sized migration entry mapping the folio before splitting it. Demote with freeze =3D false: entry creation in __split_huge_pmd_locked() is dispatched on pmd_is_migration_entry(), not on freeze, so a migration PMD becomes PTE sized migration entries either way, and freeze only controls a trailing put_page(). With freeze =3D false there is no refcount change at all, which makes the demotion idempotent across N VMAs. rmap_walk_control.anon_lock is deliberately left unset: folio_lock_anon_vma_read() depends on folio_mapped(), and the folio is already fully unmapped here. This mirrors remove_migration_ptes(). Finally, refuse the split for a folio that is not anonymous. The rmap walk would otherwise reach a file backed VMA, where split_huge_pmd_address() zaps the PMD instead of demoting it. Fixes: 4265d67e405a ("mm/migrate_device: add THP splitting during migration= ") Cc: Andrew Morton Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Zi Yan Cc: Baolin Wang Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Balbir Singh Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Simona Vetter Cc: Thomas Hellstr=C3=B6m Cc: Francois Dugast Cc: dri-devel@lists.freedesktop.org Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Cc: stable@vger.kernel.org Assisted-by: GitHub_Copilot:claude-opus-5 Signed-off-by: Matthew Brost Reviewed-by: Balbir Singh --- mm/migrate_device.c | 98 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 89 insertions(+), 9 deletions(-) diff --git a/mm/migrate_device.c b/mm/migrate_device.c index ae9027421b80..ae17bd516d24 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -899,22 +899,104 @@ static int migrate_vma_insert_huge_pmd_page(struct m= igrate_vma *migrate, return 0; } =20 +static bool migrate_vma_split_pmd_one(struct folio *folio, + struct vm_area_struct *vma, + unsigned long addr, void *arg) +{ + DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, addr, PVMW_SYNC | PVMW_MIGRATION); + + while (page_vma_mapped_walk(&pvmw)) { + if (pvmw.pte) + continue; + + addr =3D pvmw.address; + page_vma_mapped_walk_done(&pvmw); + + /* + * Demote with freeze =3D false: the PMD already holds a + * migration entry, so __split_huge_pmd_locked() creates PTE + * sized migration entries from it and leaves the refcount + * alone. There is at most one PMD mapping @folio per VMA, so + * stop the walk here. + */ + split_huge_pmd_address(vma, addr, false); + break; + } + + return true; +} + +/* + * Demote every PMD sized migration entry that maps @folio to PTE sized on= es. + * + * migrate_device_unmap() unmaps with try_to_migrate(folio, 0), i.e. witho= ut + * TTU_SPLIT_HUGE_PMD, so a folio that was PMD mapped in several VMAs -- a= fter + * fork(), for instance -- ends up with a PMD sized migration entry in eve= ry one + * of them. folio_split_unmapped() below does not care, it only looks at t= he + * refcount, so splitting the folio without demoting all of those first wo= uld + * leave the other VMAs pointing a huge PMD at what is now an order-0 foli= o. + * remove_migration_ptes() trips over that in migrate_vma_finalize(). + */ +static void migrate_vma_split_pmd_mappings(struct folio *folio) +{ + struct rmap_walk_control rwc =3D { + .rmap_one =3D migrate_vma_split_pmd_one, + }; + + /* + * Do not pass .anon_lock: folio_lock_anon_vma_read() requires + * folio_mapped(), and @folio is already fully unmapped here. + */ + rmap_walk(folio, &rwc); +} + static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate, - unsigned long idx, unsigned long addr, + unsigned long idx, struct folio *folio) { unsigned long i; unsigned long pfn; unsigned long flags; + bool fault_folio; int ret =3D 0; =20 /* - * take a reference, since split_huge_pmd_address() with freeze =3D true - * drops a reference at the end. + * migrate_vma_split_pmd_mappings() walks the rmap, and + * split_huge_pmd_address() zaps rather than demotes a PMD in a VMA that + * is not anonymous. migrate_vma_collect_huge_pmd() does not check the + * VMA type, so a file THP can reach here; the rest of the migrate_vma() + * machinery only supports anonymous memory anyway. */ - folio_get(folio); - split_huge_pmd_address(migrate->vma, addr, true); + if (!folio_test_anon(folio)) + return -EINVAL; + + /* + * A CPU fault on a device private PMD holds an extra reference on the + * folio, taken by do_huge_pmd_device_private(). folio_split_unmapped() + * only tolerates a single caller reference, so the split would always + * fail with -EAGAIN while this fault reference is held. + * + * do_huge_pmd_device_private() derives the fault page from the PMD + * entry, so it is always the head page of @folio, and therefore always + * ends up in the head folio after an uniform split to order 0. Drop + * the reference across the split and re-take it on the head folio + * afterwards, leaving the reference exactly where it is expected to be + * released. + * + * The folio cannot go away while the reference is dropped: the + * reference taken by migrate_vma_collect_huge_pmd() is still held. + */ + fault_folio =3D migrate->fault_page && + page_folio(migrate->fault_page) =3D=3D folio; + + migrate_vma_split_pmd_mappings(folio); + + if (fault_folio) + folio_put(folio); ret =3D folio_split_unmapped(folio, 0); + if (fault_folio) + folio_get(folio); + if (ret) return ret; migrate->src[idx] &=3D ~MIGRATE_PFN_COMPOUND; @@ -935,7 +1017,7 @@ static int migrate_vma_insert_huge_pmd_page(struct mig= rate_vma *migrate, } =20 static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate, - unsigned long idx, unsigned long addr, + unsigned long idx, struct folio *folio) { return 0; @@ -1103,7 +1185,6 @@ static void __migrate_device_pages(unsigned long *src= _pfns, struct mmu_notifier_range range; unsigned long i, j; bool notified =3D false; - unsigned long addr; =20 for (i =3D 0; i < npages; ) { struct page *newpage =3D migrate_pfn_to_page(dst_pfns[i]); @@ -1177,8 +1258,7 @@ static void __migrate_device_pages(unsigned long *src= _pfns, goto next; } nr =3D 1 << folio_order(folio); - addr =3D migrate->start + i * PAGE_SIZE; - if (migrate_vma_split_unmapped_folio(migrate, i, addr, folio)) { + if (migrate_vma_split_unmapped_folio(migrate, i, folio)) { src_pfns[i] &=3D ~(MIGRATE_PFN_MIGRATE | MIGRATE_PFN_COMPOUND); goto next; --=20 2.34.1 From nobody Mon Aug 24 08:07:33 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 28A95481FBF; Wed, 5 Aug 2026 23:10:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785971459; cv=none; b=dnpQdfXApBKG2boa0UCAm+IkNN96ScQ8KhLOkbhNJRUfvSoK+vndSUhxXi9vu/Dt91pjAQLJzbALiOddOJr6Fs0wMB2So2iJb8HPJ5O6/wN1KZ6yLqrV6UaoTclJnck+Gpek5rbmHsiQwsQj3HyH0oHcEofy3TBT/IFVxXMX8+Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785971459; c=relaxed/simple; bh=0lyTOIeWDhVM9+t9ZjBtctLh8tev93wX9ypZPs331q0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=W1bsluMEBSHB3HcW3zWZo7lgfnJCWcKBiw2oiVod8liXE5mtMTqrCuhIp6HE0qQdBzoYtU2ACmL8eVNFWAsJYxd9RPYNK7W421dYXMJHC4kWJAMkj/DXT1OfmSIfKdhL5DcgbTBqdbyKU/yRVCu3w7hm+AWSOR2+ZuOTjF9SrzQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=l/araYlg; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="l/araYlg" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1785971455; x=1817507455; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=0lyTOIeWDhVM9+t9ZjBtctLh8tev93wX9ypZPs331q0=; b=l/araYlg9Xef2qVorBaLsIXEDpkfFmsbWt2ZbcoyyFSRcZQDuaNy27Mz cgV+DWvvAQTBwwI/uslZ2l7I3RR7Slg6f/ikVIz8uxY41vitpa4xfJEV2 9JhAUPCFdEnf4tukn9715Hg8TF45dIDjLHizGojGCawauQb6wE5HHbUPn sw5JMDJRIyqLEC+0JTDW5vFDJTO1fBT1z9PGMExc2OpWCCj1YC3eKaqCn vUH/tL5hCnD58mtC1ReLYhVg9V7UO42xF4Mm26y4HlowIv1C+wSXe1sqP 3Y3+P1j5bfOmkZ/Ncq6NlC6zIkaNu4VhK61rIImr1hzSOpIFo3M/tp7gY A==; X-CSE-ConnectionGUID: nVEjYr/WTjWhbfejF6+zDg== X-CSE-MsgGUID: OkAPmeDGQM+S5B5k+r1a4w== X-IronPort-AV: E=McAfee;i="6800,10657,11866"; a="90376873" X-IronPort-AV: E=Sophos;i="6.25,207,1779174000"; d="scan'208";a="90376873" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Aug 2026 16:10:51 -0700 X-CSE-ConnectionGUID: ioLIX4KjSZeTqDkxaD3SeA== X-CSE-MsgGUID: 7XkeOWqkTcmRm/99NBgSrg== X-ExtLoop1: 1 Received: from gsse-cloud1.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Aug 2026 16:10:50 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sashiko , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Balbir Singh , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter , =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , Francois Dugast , stable@vger.kernel.org Subject: [PATCH v3 4/6] drm/pagemap: dma-unmap pages before handling migration errors Date: Wed, 5 Aug 2026 16:10:39 -0700 Message-Id: <20260805231041.3791771-5-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260805231041.3791771-1-matthew.brost@intel.com> References: <20260805231041.3791771-1-matthew.brost@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable drm_pagemap_migrate_unmap_pages() relies on the pages array to determine which pages require DMA unmapping. However, drm_pagemap_migration_unlock_put_pages() clears the array as part of its cleanup, leaving drm_pagemap_migrate_unmap_pages() with no valid page information if it is called afterward. Call drm_pagemap_migrate_unmap_pages() before drm_pagemap_migration_unlock_put_pages() so the pages array remains valid during DMA unmapping. Reported-by: Sashiko Fixes: f86ad0ed620c ("drm/gpusvm, drm/pagemap: Move migration functionality= to drm_pagemap") Cc: Andrew Morton Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Zi Yan Cc: Baolin Wang Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Balbir Singh Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Simona Vetter Cc: Thomas Hellstr=C3=B6m Cc: Francois Dugast Cc: dri-devel@lists.freedesktop.org Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Cc: stable@vger.kernel.org Signed-off-by: Matthew Brost Reviewed-by: Himal Prasad Ghimiray --- drivers/gpu/drm/drm_pagemap.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/gpu/drm/drm_pagemap.c b/drivers/gpu/drm/drm_pagemap.c index 892b325fa99b..aef1fcea663d 100644 --- a/drivers/gpu/drm/drm_pagemap.c +++ b/drivers/gpu/drm/drm_pagemap.c @@ -1175,12 +1175,12 @@ int drm_pagemap_evict_to_ram(struct drm_pagemap_dev= mem *devmem_allocation) goto err_finalize; =20 err_finalize: + drm_pagemap_migrate_unmap_pages(devmem_allocation->dev, pagemap_addr, dst= , npages, + DMA_FROM_DEVICE, &state); if (err) drm_pagemap_migration_unlock_put_pages(npages, dst); migrate_device_pages(src, dst, npages); migrate_device_finalize(src, dst, npages); - drm_pagemap_migrate_unmap_pages(devmem_allocation->dev, pagemap_addr, dst= , npages, - DMA_FROM_DEVICE, &state); =20 err_free: kvfree(buf); @@ -1305,14 +1305,14 @@ static int __drm_pagemap_migrate_to_ram(struct vm_a= rea_struct *vas, goto err_finalize; =20 err_finalize: - if (err) - drm_pagemap_migration_unlock_put_pages(npages, migrate.dst); - migrate_vma_pages(&migrate); - migrate_vma_finalize(&migrate); if (dev) drm_pagemap_migrate_unmap_pages(dev, pagemap_addr, migrate.dst, npages, DMA_FROM_DEVICE, &state); + if (err) + drm_pagemap_migration_unlock_put_pages(npages, migrate.dst); + migrate_vma_pages(&migrate); + migrate_vma_finalize(&migrate); err_free: kvfree(buf); err_out: --=20 2.34.1 From nobody Mon Aug 24 08:07:33 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7A21481FCB; Wed, 5 Aug 2026 23:10:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785971461; cv=none; b=Tzm9UmtLLnd/S2aTn6tQECFbK6JbAepDgPZ5NmhM84CfEGbLV92hlr0/LXp3Uxr9zlkJO4PPI6YmbCNtqfgh7bK13ELlW2dK/uK4AqkDXVfC339pWvHkD6812+3KmxpmahygR1wiP+YroCbQC58nyOmHLCAEmsVNcKxyyH8PYRE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785971461; c=relaxed/simple; bh=7IwZ720KYefy5lGHWlIcMDNlKqXcJ+g12XviECt04CM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=aIErMJnK848WCIcrPrS0sOZLo43Vn3/Z8SypIWdgzmXEn7Qr/Z48pBYBxjAU/rkKj2gDSCO0uDyL/Zllkz2kjtXPU9P9oBh+0eGwHuSErhSCCGNCXfnfZ3d2acsKWIeIvtU4VLcj4tMaLsOypome+A2DgEmzaxtPOcayPQpr3Z4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=QDPrNWfA; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="QDPrNWfA" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1785971457; x=1817507457; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=7IwZ720KYefy5lGHWlIcMDNlKqXcJ+g12XviECt04CM=; b=QDPrNWfAHArovYB/8euoHp+tGKoz3tyDrV/kh3MdAcVpLYtZl1O586LF 51ITz/Dok7f8OvYZWGoXpMdu2Y0ijE8vPCP4lYNjaat0/CkUFqvEclgw6 HU+mF/kUfDF7TJCJpQ5Ydxw6vHsGIMJBQpihhqZjsR3MsL5YWCXsSbSwY uVTGsZwUiyjTNXNqxXhRXQus/q4LL0ZngX9ZkJ8sj7/vASwFTmgEoe+pW +2nIXVbIJ69LTWLduNb3Z/dulv1voBUG8UGmM80/258GdJ1DxeXSrk9H3 VR20eTKTYxBPKd4I/W4mCmGVHOGN8arKVT24hGLMlpFIue9Zf1op23uN0 A==; X-CSE-ConnectionGUID: 5asSuSpwQTerYGk0stSlqg== X-CSE-MsgGUID: u6Tz/ZBDR3WkyKqoBGi2ww== X-IronPort-AV: E=McAfee;i="6800,10657,11866"; a="90376889" X-IronPort-AV: E=Sophos;i="6.25,207,1779174000"; d="scan'208";a="90376889" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Aug 2026 16:10:51 -0700 X-CSE-ConnectionGUID: TRftW6ygQaOc//3wYA5uIA== X-CSE-MsgGUID: DCKgb19kSGe3y8k33VrtkA== X-ExtLoop1: 1 Received: from gsse-cloud1.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Aug 2026 16:10:51 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Balbir Singh , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter , =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , Francois Dugast , stable@vger.kernel.org Subject: [PATCH v3 5/6] drm/pagemap: Fix folio allocation fallback and use-after-put Date: Wed, 5 Aug 2026 16:10:40 -0700 Message-Id: <20260805231041.3791771-6-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260805231041.3791771-1-matthew.brost@intel.com> References: <20260805231041.3791771-1-matthew.brost@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable drm_pagemap_migrate_populate_ram_pfn() had two issues when populating RAM PFNs with higher-order folios: 1. The higher-order vma_alloc_folio()/folio_alloc() calls did not pass __GFP_NOWARN, so a THP allocation failure under memory pressure would spam the kernel log, and there was no fallback path despite a TODO comment stating one was needed. Add __GFP_NOWARN to the higher-order allocation and, on failure, fall back to order-0 allocations for the entire range originally covered by the failed higher-order allocation, leaving MIGRATE_PFN_COMPOUND unset for those PFNs. 2. In the free_pages error path, order was computed via folio_order(page_folio(page)) *after* put_page(page) had already dropped the reference, resulting in a use-after-free/put when that was the last reference on the page. Compute order before releasing the page. Introducing the fallback in 1. also requires the source page array handed to ->copy_to_ram() to be built differently. Both callers only populated the entry at the head of each source folio, relying on the copy callback to derive the rest of the folio from the order recorded in the matching drm_pagemap_addr. Once the destination has been demoted to order-0 folios the drm_pagemap_addr entries are per-page, so a source page is needed for every one of them; leaving them NULL makes the copy callback stop after the first page and the remainder of the range is never copied. The source folio is only split later, by migrate_vma_pages() / migrate_device_pages(), so its order cannot be used to detect the demotion - test the destination for MIGRATE_PFN_COMPOUND instead. Factor the array population out into drm_pagemap_migrate_populate_src_pages() and use it from both drm_pagemap_evict_to_ram() and __drm_pagemap_migrate_to_ram(). Fixes: ddeda6136038 ("drm/pagemap: Allocate folios when possible") Cc: Andrew Morton Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Zi Yan Cc: Baolin Wang Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Balbir Singh Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Simona Vetter Cc: Thomas Hellstr=C3=B6m Cc: Francois Dugast Cc: dri-devel@lists.freedesktop.org Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Cc: stable@vger.kernel.org Assisted-by: GitHub_Copilot:claude-opus-5 Signed-off-by: Matthew Brost Reviewed-by: Himal Prasad Ghimiray --- v2: Add THP-mid-PMD invariant (Sashiko) --- drivers/gpu/drm/drm_pagemap.c | 128 +++++++++++++++++++++++++++------- 1 file changed, 103 insertions(+), 25 deletions(-) diff --git a/drivers/gpu/drm/drm_pagemap.c b/drivers/gpu/drm/drm_pagemap.c index aef1fcea663d..51c6f12e4256 100644 --- a/drivers/gpu/drm/drm_pagemap.c +++ b/drivers/gpu/drm/drm_pagemap.c @@ -383,6 +383,58 @@ drm_pagemap_migrate_map_system_pages(struct device *de= v, return 0; } =20 +/** + * drm_pagemap_migrate_populate_src_pages() - Populate the source page arr= ay + * @pages: Array of source pages to populate + * @src_mpfn: Source array of migrate PFNs + * @dst_mpfn: Destination array of migrate PFNs + * @npages: Number of pages in the arrays + * + * Populate @pages with the device pages the copy callback is to read from. + * + * Entries are normally only populated at the head of each source folio, w= ith + * the copy callback deriving the rest of the folio from the order recorde= d in + * the corresponding drm_pagemap_addr. That does not work where + * drm_pagemap_migrate_populate_ram_pfn() had to demote a higher-order sou= rce + * folio to order-0 destination folios: the drm_pagemap_addr entries are t= hen + * per-page, and the copy callback needs a source page for each of them. + * Populate every entry for those ranges. + * + * Note that the source folio itself is only split later, by + * migrate_vma_pages() / migrate_device_pages(), so its order cannot be us= ed to + * detect the demotion - the destination has to be inspected instead. + */ +static void drm_pagemap_migrate_populate_src_pages(struct page **pages, + unsigned long *src_mpfn, + unsigned long *dst_mpfn, + unsigned long npages) +{ + unsigned long i; + + for (i =3D 0; i < npages;) { + struct page *page =3D migrate_pfn_to_page(src_mpfn[i]); + unsigned int order =3D 0; + unsigned long j, nr; + + if (!page) { + i++; + continue; + } + + order =3D folio_order(page_folio(page)); + nr =3D NR_PAGES(order); + + if (order && !(dst_mpfn[i] & MIGRATE_PFN_COMPOUND)) { + for (j =3D 0; j < nr && i + j < npages; j++) + pages[i + j] =3D folio_page(page_folio(page), j); + } else { + pages[i] =3D page; + } + + i +=3D nr; + } +} + /** * drm_pagemap_migrate_unmap_pages() - Unmap pages previously mapped for G= PU SVM migration * @dev: The device for which the pages were mapped @@ -875,6 +927,7 @@ static int drm_pagemap_migrate_populate_ram_pfn(struct = vm_area_struct *vas, struct page *page =3D NULL, *src_page; struct folio *folio; unsigned int order =3D 0; + gfp_t gfp =3D GFP_HIGHUSER; =20 if (!(src_mpfn[i] & MIGRATE_PFN_MIGRATE)) goto next; @@ -891,11 +944,51 @@ static int drm_pagemap_migrate_populate_ram_pfn(struc= t vm_area_struct *vas, =20 order =3D folio_order(page_folio(src_page)); =20 - /* TODO: Support fallback to single pages if THP allocation fails */ + /* + * A large source folio is always collected whole, at its head + * page, PMD aligned and flagged MIGRATE_PFN_COMPOUND: anything + * else is split before it reaches us, either by + * migrate_vma_collect_pmd() or, for the eviction path, by + * migrate_device_pfns(). Both the order-0 fallback below and + * drm_pagemap_migrate_populate_src_pages() rely on that, as + * they index the folio from @i. + */ + WARN_ON_ONCE(order && + (src_page !=3D folio_page(page_folio(src_page), 0) || + !(src_mpfn[i] & MIGRATE_PFN_COMPOUND))); + + if (order) + gfp |=3D __GFP_NOWARN; + if (vas) - folio =3D vma_alloc_folio(GFP_HIGHUSER, order, vas, addr); + folio =3D vma_alloc_folio(gfp, order, vas, addr); else - folio =3D folio_alloc(GFP_HIGHUSER, order); + folio =3D folio_alloc(gfp, order); + + if (!folio && order) { + /* + * Higher-order allocation failed, fall back to + * order-0 allocations for the entire range covered + * by the original higher-order allocation, without + * setting MIGRATE_PFN_COMPOUND, until we move past + * that range. + */ + unsigned long nr =3D NR_PAGES(order); + unsigned long j; + + gfp &=3D ~__GFP_NOWARN; + for (j =3D 0; j < nr && i < npages; j++, i++, addr +=3D PAGE_SIZE) { + folio =3D vas ? + vma_alloc_folio(gfp, 0, vas, addr) : + folio_alloc(gfp, 0); + if (!folio) + goto free_pages; + + page =3D folio_page(folio, 0); + mpfn[i] =3D migrate_pfn(page_to_pfn(page)); + } + continue; + } =20 if (!folio) goto free_pages; @@ -940,11 +1033,11 @@ static int drm_pagemap_migrate_populate_ram_pfn(stru= ct vm_area_struct *vas, if (!page) goto next_put; =20 + order =3D folio_order(page_folio(page)); + put_page(page); mpfn[i] =3D 0; =20 - order =3D folio_order(page_folio(page)); - next_put: i +=3D NR_PAGES(order); } @@ -1120,7 +1213,7 @@ int drm_pagemap_evict_to_ram(struct drm_pagemap_devme= m *devmem_allocation) unsigned long *src, *dst; struct drm_pagemap_addr *pagemap_addr; void *buf; - int i, err =3D 0; + int err =3D 0; unsigned int retry_count =3D 2; =20 npages =3D devmem_allocation->size >> PAGE_SHIFT; @@ -1160,15 +1253,7 @@ int drm_pagemap_evict_to_ram(struct drm_pagemap_devm= em *devmem_allocation) if (err) goto err_finalize; =20 - for (i =3D 0; i < npages;) { - unsigned int order =3D 0; - - pages[i] =3D migrate_pfn_to_page(src[i]); - if (pages[i]) - order =3D folio_order(page_folio(pages[i])); - - i +=3D NR_PAGES(order); - } + drm_pagemap_migrate_populate_src_pages(pages, src, dst, npages); =20 err =3D ops->copy_to_ram(pages, pagemap_addr, npages, NULL); if (err) @@ -1235,7 +1320,7 @@ static int __drm_pagemap_migrate_to_ram(struct vm_are= a_struct *vas, struct drm_pagemap_addr *pagemap_addr; unsigned long start, end; void *buf; - int i, err =3D 0; + int err =3D 0; =20 zdd =3D drm_pagemap_page_zone_device_data(page); if (time_before64(get_jiffies_64(), zdd->devmem_allocation->timeslice_exp= iration)) @@ -1290,15 +1375,8 @@ static int __drm_pagemap_migrate_to_ram(struct vm_ar= ea_struct *vas, if (err) goto err_finalize; =20 - for (i =3D 0; i < npages;) { - unsigned int order =3D 0; - - pages[i] =3D migrate_pfn_to_page(migrate.src[i]); - if (pages[i]) - order =3D folio_order(page_folio(pages[i])); - - i +=3D NR_PAGES(order); - } + drm_pagemap_migrate_populate_src_pages(pages, migrate.src, migrate.dst, + npages); =20 err =3D ops->copy_to_ram(pages, pagemap_addr, npages, NULL); if (err) --=20 2.34.1 From nobody Mon Aug 24 08:07:33 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5FD6848035A for ; Wed, 5 Aug 2026 23:10:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785971464; cv=none; b=DyOW3rCY2hyQh8i0kKW+JYsSQAajKFYY9Vffh1hRImoAWjUZGczqhYUvAu1R+vtYLoAWvcRFKPzvPDhUJ5cbB09wCrGPa9090rfmxl1i/aejFAIW6Z+FBRCNkRf6ctVU9widrxn1G4bo9D7AGl5JcdeypXeRcoEAb3U9sbuqUqs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785971464; c=relaxed/simple; bh=NZLOXaen9jDVMjpnEaM9wnAKmSoircGZzGky2M1Tvb0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=lhtduBLqKS1UejW4DkcTPPQ98qx6t4V4FhbR8bGDY7cTv2xxrMGWtZK0P09JcZJ6H5tTkgSajsEeziXzJYvm+u4297HzgDI0t6vBb/lQ+NIAs5LAa5j/d2bzHHGAhZcvD/wT60EUSYQacfgU5RLtBGrphPNJHENkI5ZKueUkHgo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=fxFAd2K3; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="fxFAd2K3" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1785971460; x=1817507460; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=NZLOXaen9jDVMjpnEaM9wnAKmSoircGZzGky2M1Tvb0=; b=fxFAd2K3n9YmGGXyYtlrwt56A3kxPiw/t8dn1zSmvX2dyOLor0s7VnyL pD6P9GvKd/zCaEJBvA2N3+3Kv2cQuW6bXG6QMp1R6smOFgnL2/8/W0dmx +OTmx2PyklEx90LZiT4/FD28SAr2jBnMHHfcMQfqtzRD2MfAOXSoNJiE5 fSjjp7nYqdYKQ4NGgwdwBCsveLxrOfXL+8y6ByoiDaUlQv3rA2bCgxfxT bVE1xNOF+CoGrIk1Btwrv/9KAQcu8aVQbGue8olBx7VoK9VWzxJK7gmWE 69rHJHxHO4Q+gEbiCXTd+/W5dk/L0KX1qTOcnjsDPaZWbrlRFC0RgBdXT g==; X-CSE-ConnectionGUID: phF/KWv1SOuH1fTajprXSg== X-CSE-MsgGUID: Ym72pCEFQfaoXbCNYPZjxw== X-IronPort-AV: E=McAfee;i="6800,10657,11866"; a="90376904" X-IronPort-AV: E=Sophos;i="6.25,207,1779174000"; d="scan'208";a="90376904" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Aug 2026 16:10:52 -0700 X-CSE-ConnectionGUID: Mu2CKANrRdOv2SSKNqnKZw== X-CSE-MsgGUID: W3bcdhw1So+UvqvPffSiGw== X-ExtLoop1: 1 Received: from gsse-cloud1.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Aug 2026 16:10:51 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Balbir Singh , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter , =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , Francois Dugast Subject: [PATCH v3 6/6] drm/pagemap: Add fault injection for higher-order RAM folio allocation Date: Wed, 5 Aug 2026 16:10:41 -0700 Message-Id: <20260805231041.3791771-7-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260805231041.3791771-1-matthew.brost@intel.com> References: <20260805231041.3791771-1-matthew.brost@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Migrating a device-private THP back to system memory has two distinct paths in __migrate_device_pages(): the fast path where both source and destination carry MIGRATE_PFN_COMPOUND, and the fallback path where the destination could only be satisfied with order-0 folios and the source THP therefore has to be split via migrate_vma_split_unmapped_folio(). The fallback path only triggers under genuine memory pressure, which makes it both rare and awkward to reproduce, yet it is the path where the interesting refcounting happens (the CPU fault holds an extra reference on the device folio taken by do_huge_pmd_device_private()). Add a fault_attr, modelled on backup_fault_inject in ttm_pool.c, that forces the higher-order allocation in drm_pagemap_migrate_populate_ram_pfn() to fail so the existing order-0 fallback is taken deterministically. The attribute is exposed at /sys/kernel/debug/drm_pagemap_fault_inject and requires CONFIG_FAULT_INJECTION_DEBUG_FS. With CONFIG_FAULT_INJECTION disabled the helper compiles out to a constant false and the injection has no cost. Cc: Andrew Morton Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Zi Yan Cc: Baolin Wang Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Balbir Singh Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Simona Vetter Cc: Thomas Hellstr=C3=B6m Cc: Francois Dugast Cc: dri-devel@lists.freedesktop.org Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Assisted-by: GitHub_Copilot:claude-opus-5 Signed-off-by: Matthew Brost --- drivers/gpu/drm/drm_pagemap.c | 36 ++++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/drm_pagemap.c b/drivers/gpu/drm/drm_pagemap.c index 51c6f12e4256..6ae8c9aa36cc 100644 --- a/drivers/gpu/drm/drm_pagemap.c +++ b/drivers/gpu/drm/drm_pagemap.c @@ -3,6 +3,7 @@ * Copyright =C2=A9 2024-2025 Intel Corporation */ =20 +#include #include #include #include @@ -12,6 +13,27 @@ #include #include =20 +#ifdef CONFIG_FAULT_INJECTION +#include +static DECLARE_FAULT_ATTR(migrate_to_ram_fault_inject); + +/* + * Force a higher-order destination folio allocation to fail in + * drm_pagemap_migrate_populate_ram_pfn(), exercising the order-0 fallback + * (and, in turn, the THP split path in __migrate_device_pages()) without + * having to drive the system into actual memory pressure. + */ +static bool drm_pagemap_fault_inject_folio(void) +{ + return should_fail(&migrate_to_ram_fault_inject, 1); +} +#else +static bool drm_pagemap_fault_inject_folio(void) +{ + return false; +} +#endif + /** * DOC: Overview * @@ -960,7 +982,9 @@ static int drm_pagemap_migrate_populate_ram_pfn(struct = vm_area_struct *vas, if (order) gfp |=3D __GFP_NOWARN; =20 - if (vas) + if (order && drm_pagemap_fault_inject_folio()) + folio =3D NULL; + else if (vas) folio =3D vma_alloc_folio(gfp, order, vas, addr); else folio =3D folio_alloc(gfp, order); @@ -1554,6 +1578,16 @@ void drm_pagemap_destroy(struct drm_pagemap *dpagema= p, bool is_atomic_or_reclaim kfree(dpagemap); } =20 +static int __init drm_pagemap_module_init(void) +{ +#if defined(CONFIG_DEBUG_FS) && defined(CONFIG_FAULT_INJECTION) + fault_create_debugfs_attr("drm_pagemap_fault_inject", NULL, + &migrate_to_ram_fault_inject); +#endif + return 0; +} +module_init(drm_pagemap_module_init); + static void drm_pagemap_exit(void) { flush_work(&drm_pagemap_work); --=20 2.34.1