From nobody Mon Feb 9 01:19:29 2026 Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1675B76C68 for ; Mon, 25 Mar 2024 22:33:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711406025; cv=none; b=iMfkBfQJhDlWi92Icvqe5UOBhVOR+W7HIE/yk0mhfbSP5SGlK0C4YpzxDGh2e3MhZ/fi2i627Wo1NbaoGgbuFd6tp/W2HWTg598kb0P8HEwDoKhAG6KVE/aHiaJy5HBOsw60kefluJzR3OZmmZIV5KX3/1GWzX0q0dMmGgsEDsE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711406025; c=relaxed/simple; bh=5doG0I4O79bhHZe6ovaO/RH1HdZ3bLY0+cmxlMVxMuM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LwAD9duvV1QcDnOQuWji98JUWdCDwqE/qTWW3LakFUS/RMPg2ZYIKbAuJo6eKspzdwYmOjtsw4LGYlvm5XPbGX/PljXZ93HiltLC+WyDtPqaXFb9/OW922Jq1T/nN+BWBAYxLD66x9uT8xUabNAwqSc6TmUr4H+Fr+VpwOvdRfY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Iy3HWiU4; arc=none smtp.client-ip=209.85.215.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Iy3HWiU4" Received: by mail-pg1-f181.google.com with SMTP id 41be03b00d2f7-5d3907ff128so3627601a12.3 for ; Mon, 25 Mar 2024 15:33:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711406023; x=1712010823; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7wuUkLEO8wS3NnK8pO29sbwl4okjZZVm7bwiudBCZKQ=; b=Iy3HWiU43r2HGm/vK/lXHBwol1h40nGt3g/lPPD46I+PdkED+Inv8m75++XI0XBl5R BsH2oBrFQyMSTNKAHgRRAuHMyrwbVdJ7sZ8ENTXb+gA4JYJXp+lChIWrfzn8iYPFKHHs 0fYHg2yyKGdVLrcGc9oRS5dMQh4uOYZJa+S1HG9E8YFgPpm4H5FnG6BVVNyQfccmtbeQ vxUJf6OHcwxYqXjtwdSXLUr/HoDEMJs/RqZDEvkXDR5BGbbSFHnWFh4P9Tbpasfj77DQ RUKaPoi/vjW7C/TYStEW+6ac38CmE2w23j/rjWoHpii4AdU6pepLQaY6FD3NV3kLErSn whYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711406023; x=1712010823; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7wuUkLEO8wS3NnK8pO29sbwl4okjZZVm7bwiudBCZKQ=; b=KVbgatBV3WQJL+1HFDqCpApsCR1AvU/xt5mZeZutcHGu2eezSymu+FKf10OMq8154B SrpqhDKV3FISF9394uLDIh90uuTTXI3/3fVDosCwQKj53EgzTWjbD1L0pxyfT69E7wMy tYbbHUfcaAqFjKw3rlWBXJOlTTMUPNkOZml/8vbxx7Tk2Tj6IZeBjcx3SrB2dYJJCaze 1/secLxYZXegJcVibW4K9QRa52L2YP4q3flFaJYx2Gy/qKshJrync3TSHel2V6gOGdia DYlPaXfD0e5J4+DGfbRWx4bm/bqWFkvqdA7h70pYVFxWL3OT5bjNCvE9ECiqLWJryYNs QS7A== X-Gm-Message-State: AOJu0YxgioqZeOLZWfpMuTYBpBPbiGyRhZy5eDy/e8V0lzW0gtwwjrdN PuW6mT5ZKUmUJTfeMQpVBbov/SMokjZUyFB5Jbf+3s1uZcqyTf47 X-Google-Smtp-Source: AGHT+IFl6KFavtRUNTSrxnwpqS8RSMDEAwpK70R/1RBLnR40J5C5dnwH9mZD8fjRJHapYDc5t8I8Zw== X-Received: by 2002:a05:6a21:3942:b0:1a3:cd06:20cf with SMTP id ac2-20020a056a21394200b001a3cd0620cfmr4148137pzc.16.1711406023376; Mon, 25 Mar 2024 15:33:43 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id sx16-20020a17090b2cd000b002a053cdd4e5sm4356173pjb.9.2024.03.25.15.33.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 15:33:43 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, muchun.song@linux.dev, "Vishal Moola (Oracle)" Subject: [PATCH 1/5] hugetlb: Convert hugetlb_fault() to use struct vm_fault Date: Mon, 25 Mar 2024 15:33:35 -0700 Message-ID: <20240325223339.169350-2-vishal.moola@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325223339.169350-1-vishal.moola@gmail.com> References: <20240325223339.169350-1-vishal.moola@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that hugetlb_fault() has a vm_fault available for fault tracking, use it throughout. This cleans up the code by removing 2 variables, and prepares hugetlb_fault() to take in a struct vm_fault argument. Signed-off-by: Vishal Moola (Oracle) --- mm/hugetlb.c | 84 +++++++++++++++++++++++++--------------------------- 1 file changed, 41 insertions(+), 43 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 688017ca0cc2..81e8ade53b64 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6330,8 +6330,6 @@ u32 hugetlb_fault_mutex_hash(struct address_space *ma= pping, pgoff_t idx) vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, unsigned int flags) { - pte_t *ptep, entry; - spinlock_t *ptl; vm_fault_t ret; u32 hash; struct folio *folio =3D NULL; @@ -6339,13 +6337,13 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, stru= ct vm_area_struct *vma, struct hstate *h =3D hstate_vma(vma); struct address_space *mapping; int need_wait_lock =3D 0; - unsigned long haddr =3D address & huge_page_mask(h); struct vm_fault vmf =3D { .vma =3D vma, - .address =3D haddr, + .address =3D address & huge_page_mask(h), .real_address =3D address, .flags =3D flags, - .pgoff =3D vma_hugecache_offset(h, vma, haddr), + .pgoff =3D vma_hugecache_offset(h, vma, + address & huge_page_mask(h)), /* TODO: Track hugetlb faults using vm_fault */ =20 /* @@ -6365,22 +6363,22 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, stru= ct vm_area_struct *vma, =20 /* * Acquire vma lock before calling huge_pte_alloc and hold - * until finished with ptep. This prevents huge_pmd_unshare from - * being called elsewhere and making the ptep no longer valid. + * until finished with vmf.pte. This prevents huge_pmd_unshare from + * being called elsewhere and making the vmf.pte no longer valid. */ hugetlb_vma_lock_read(vma); - ptep =3D huge_pte_alloc(mm, vma, haddr, huge_page_size(h)); - if (!ptep) { + vmf.pte =3D huge_pte_alloc(mm, vma, vmf.address, huge_page_size(h)); + if (!vmf.pte) { hugetlb_vma_unlock_read(vma); mutex_unlock(&hugetlb_fault_mutex_table[hash]); return VM_FAULT_OOM; } =20 - entry =3D huge_ptep_get(ptep); - if (huge_pte_none_mostly(entry)) { - if (is_pte_marker(entry)) { + vmf.orig_pte =3D huge_ptep_get(vmf.pte); + if (huge_pte_none_mostly(vmf.orig_pte)) { + if (is_pte_marker(vmf.orig_pte)) { pte_marker marker =3D - pte_marker_get(pte_to_swp_entry(entry)); + pte_marker_get(pte_to_swp_entry(vmf.orig_pte)); =20 if (marker & PTE_MARKER_POISONED) { ret =3D VM_FAULT_HWPOISON_LARGE; @@ -6395,20 +6393,20 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, stru= ct vm_area_struct *vma, * mutex internally, which make us return immediately. */ return hugetlb_no_page(mm, vma, mapping, vmf.pgoff, address, - ptep, entry, flags, &vmf); + vmf.pte, vmf.orig_pte, flags, &vmf); } =20 ret =3D 0; =20 /* - * entry could be a migration/hwpoison entry at this point, so this - * check prevents the kernel from going below assuming that we have - * an active hugepage in pagecache. This goto expects the 2nd page - * fault, and is_hugetlb_entry_(migration|hwpoisoned) check will - * properly handle it. + * vmf.orig_pte could be a migration/hwpoison vmf.orig_pte at this + * point, so this check prevents the kernel from going below assuming + * that we have an active hugepage in pagecache. This goto expects + * the 2nd page fault, and is_hugetlb_entry_(migration|hwpoisoned) + * check will properly handle it. */ - if (!pte_present(entry)) { - if (unlikely(is_hugetlb_entry_migration(entry))) { + if (!pte_present(vmf.orig_pte)) { + if (unlikely(is_hugetlb_entry_migration(vmf.orig_pte))) { /* * Release the hugetlb fault lock now, but retain * the vma lock, because it is needed to guard the @@ -6417,9 +6415,9 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct= vm_area_struct *vma, * be released there. */ mutex_unlock(&hugetlb_fault_mutex_table[hash]); - migration_entry_wait_huge(vma, ptep); + migration_entry_wait_huge(vma, vmf.pte); return 0; - } else if (unlikely(is_hugetlb_entry_hwpoisoned(entry))) + } else if (unlikely(is_hugetlb_entry_hwpoisoned(vmf.orig_pte))) ret =3D VM_FAULT_HWPOISON_LARGE | VM_FAULT_SET_HINDEX(hstate_index(h)); goto out_mutex; @@ -6433,13 +6431,13 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, stru= ct vm_area_struct *vma, * determine if a reservation has been consumed. */ if ((flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) && - !(vma->vm_flags & VM_MAYSHARE) && !huge_pte_write(entry)) { - if (vma_needs_reservation(h, vma, haddr) < 0) { + !(vma->vm_flags & VM_MAYSHARE) && !huge_pte_write(vmf.orig_pte)) { + if (vma_needs_reservation(h, vma, vmf.address) < 0) { ret =3D VM_FAULT_OOM; goto out_mutex; } /* Just decrements count, does not deallocate */ - vma_end_reservation(h, vma, haddr); + vma_end_reservation(h, vma, vmf.address); =20 pagecache_folio =3D filemap_lock_hugetlb_folio(h, mapping, vmf.pgoff); @@ -6447,17 +6445,17 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, stru= ct vm_area_struct *vma, pagecache_folio =3D NULL; } =20 - ptl =3D huge_pte_lock(h, mm, ptep); + vmf.ptl =3D huge_pte_lock(h, mm, vmf.pte); =20 /* Check for a racing update before calling hugetlb_wp() */ - if (unlikely(!pte_same(entry, huge_ptep_get(ptep)))) + if (unlikely(!pte_same(vmf.orig_pte, huge_ptep_get(vmf.pte)))) goto out_ptl; =20 /* Handle userfault-wp first, before trying to lock more pages */ - if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(ptep)) && - (flags & FAULT_FLAG_WRITE) && !huge_pte_write(entry)) { + if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(vmf.pte)) && + (flags & FAULT_FLAG_WRITE) && !huge_pte_write(vmf.orig_pte)) { if (!userfaultfd_wp_async(vma)) { - spin_unlock(ptl); + spin_unlock(vmf.ptl); if (pagecache_folio) { folio_unlock(pagecache_folio); folio_put(pagecache_folio); @@ -6467,18 +6465,18 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, stru= ct vm_area_struct *vma, return handle_userfault(&vmf, VM_UFFD_WP); } =20 - entry =3D huge_pte_clear_uffd_wp(entry); - set_huge_pte_at(mm, haddr, ptep, entry, + vmf.orig_pte =3D huge_pte_clear_uffd_wp(vmf.orig_pte); + set_huge_pte_at(mm, vmf.address, vmf.pte, vmf.orig_pte, huge_page_size(hstate_vma(vma))); /* Fallthrough to CoW */ } =20 /* - * hugetlb_wp() requires page locks of pte_page(entry) and + * hugetlb_wp() requires page locks of pte_page(vmf.orig_pte) and * pagecache_folio, so here we need take the former one * when folio !=3D pagecache_folio or !pagecache_folio. */ - folio =3D page_folio(pte_page(entry)); + folio =3D page_folio(pte_page(vmf.orig_pte)); if (folio !=3D pagecache_folio) if (!folio_trylock(folio)) { need_wait_lock =3D 1; @@ -6488,24 +6486,24 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, stru= ct vm_area_struct *vma, folio_get(folio); =20 if (flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) { - if (!huge_pte_write(entry)) { - ret =3D hugetlb_wp(mm, vma, address, ptep, flags, - pagecache_folio, ptl, &vmf); + if (!huge_pte_write(vmf.orig_pte)) { + ret =3D hugetlb_wp(mm, vma, address, vmf.pte, flags, + pagecache_folio, vmf.ptl, &vmf); goto out_put_page; } else if (likely(flags & FAULT_FLAG_WRITE)) { - entry =3D huge_pte_mkdirty(entry); + vmf.orig_pte =3D huge_pte_mkdirty(vmf.orig_pte); } } - entry =3D pte_mkyoung(entry); - if (huge_ptep_set_access_flags(vma, haddr, ptep, entry, + vmf.orig_pte =3D pte_mkyoung(vmf.orig_pte); + if (huge_ptep_set_access_flags(vma, vmf.address, vmf.pte, vmf.orig_pte, flags & FAULT_FLAG_WRITE)) - update_mmu_cache(vma, haddr, ptep); + update_mmu_cache(vma, vmf.address, vmf.pte); out_put_page: if (folio !=3D pagecache_folio) folio_unlock(folio); folio_put(folio); out_ptl: - spin_unlock(ptl); + spin_unlock(vmf.ptl); =20 if (pagecache_folio) { folio_unlock(pagecache_folio); --=20 2.43.0 From nobody Mon Feb 9 01:19:29 2026 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 052107F48F for ; Mon, 25 Mar 2024 22:33:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711406026; cv=none; b=X5doCUPFxhAr36zG94WILr5NgTOHOWFBnICtSIr6M1ZEQFlJ2uQ24t8QjULFxCAsFNDrOqNNLEK1u0LlVksnijE+obagaFVLhZ7x+e24AxBJBJX1MNhtcU03z+vucRkWLn1neBtm5sYNUNwdHgjj9+W8sgWdfAWBgOcH4yImT+Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711406026; c=relaxed/simple; bh=V+cGmqZ84ky38wT7ptHjAMrWPr+0QyKC4IpVVER8XpY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cAbU+78s6OOXtNhMScinoE75aiT0l+7e8Xt46ST6/QI0JjTMeGOLXsj/eIdCB1wq7vdhIE6REqtkKyNto/rEUng+mil+Q/fmMfVvOPdEjNOcN2srPubMsJ5UC0HkHgu4iABgc3mMP7og3Ffy+x8kD6u38hT/CcWfrdlx7tINODI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=kLdYuMmy; arc=none smtp.client-ip=209.85.216.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kLdYuMmy" Received: by mail-pj1-f44.google.com with SMTP id 98e67ed59e1d1-29f749b6667so3579290a91.0 for ; Mon, 25 Mar 2024 15:33:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711406024; x=1712010824; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sP/HrRtOaF5YRIfX9u0RpV4cBbAlDIrGYsjsP3+O3X8=; b=kLdYuMmyJxwm1RO9iE1ZEmc0jvoP7PDRFNsWSlcA5E91axCFPcEDZ0YM2twQgPW+tT SBWTTlODtG1sWcQiKb7i+5By2r+xfdrKevRRyBuAFqdwV+/ikmheL9UxwwsnbtJ4Wuv/ 6YCoC6V7snkjKTZ+8tHzT+sECYNnR4R9sSkU2FCxeLfO5LLhRiEjHEzprn0egdNvIM1S Pue+McX1Pkmao4dhLEeo18Xr9+MZraSg0aqPakc3PcMtMIjDbimZZ7ThDLWiXhebkhGb /hAoq2wf3MjtMET4cq+BM+nJm12gUjZaamojamf1FZBqc2SSQa/p7LBtgtLcLakemU// 7HzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711406024; x=1712010824; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sP/HrRtOaF5YRIfX9u0RpV4cBbAlDIrGYsjsP3+O3X8=; b=NGqUqg8vN8fc02thXEZdBKS/8v3Sb7rzuB+wbxWntGElhqNoGyXkyIqqE+GH8GwiFm dJaa88XSYx9ghqjNaVu4bIHk/s6NFeNlbFackvgnbsnPuXBD4O/VafZtRHggW9UKDYzJ o6n1tkoS360S7N5FC9rb+6OMQ+pgIQwGYJD12vCXfL8uCVGoRNoF6CkjNnEQPBETURp4 0XdI9bP+fz2hZ9a1fL1iGKuHTlrmiZBg/VYnl54y3fG5qD3nIa7kXyqG6G6pYr8CcmKz yPfUpfHE9oM0NVlIOLmrryg4TFOEyNn/Q2w6dOZ23v2rbuLCff4BWguPqBUlW4lmKd/D rSmA== X-Gm-Message-State: AOJu0Yz8OQvQRcpDLA7gqe3xDWcDfs2NwaynPGA39GmHCoBb0klREuSu +OL+LQUs4rSGXCJ6Mn2Cpq5MbrXB73CLsEIg6txbUBcAyBvPEl+0 X-Google-Smtp-Source: AGHT+IGyWxFaoFa1+IvC2ORjow3Na1wp3zhzntyvupejForDNfjnFm53ZyuyQXm9jJ+7DPPZog8puQ== X-Received: by 2002:a17:90a:2f26:b0:29c:7769:419b with SMTP id s35-20020a17090a2f2600b0029c7769419bmr8194248pjd.9.1711406024293; Mon, 25 Mar 2024 15:33:44 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id sx16-20020a17090b2cd000b002a053cdd4e5sm4356173pjb.9.2024.03.25.15.33.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 15:33:43 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, muchun.song@linux.dev, "Vishal Moola (Oracle)" Subject: [PATCH 2/5] hugetlb: Convert hugetlb_no_page() to use struct vm_fault Date: Mon, 25 Mar 2024 15:33:36 -0700 Message-ID: <20240325223339.169350-3-vishal.moola@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325223339.169350-1-vishal.moola@gmail.com> References: <20240325223339.169350-1-vishal.moola@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" hugetlb_no_page() can use the struct vm_fault passed in from hugetlb_fault(). This alleviates the stack by consolidating 7 variables into a single struct. Signed-off-by: Vishal Moola (Oracle) --- mm/hugetlb.c | 59 ++++++++++++++++++++++++++-------------------------- 1 file changed, 29 insertions(+), 30 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 81e8ade53b64..819a6d067985 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6096,9 +6096,7 @@ static bool hugetlb_pte_stable(struct hstate *h, stru= ct mm_struct *mm, =20 static vm_fault_t hugetlb_no_page(struct mm_struct *mm, struct vm_area_struct *vma, - struct address_space *mapping, pgoff_t idx, - unsigned long address, pte_t *ptep, - pte_t old_pte, unsigned int flags, + struct address_space *mapping, struct vm_fault *vmf) { struct hstate *h =3D hstate_vma(vma); @@ -6107,10 +6105,8 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *= mm, unsigned long size; struct folio *folio; pte_t new_pte; - spinlock_t *ptl; - unsigned long haddr =3D address & huge_page_mask(h); bool new_folio, new_pagecache_folio =3D false; - u32 hash =3D hugetlb_fault_mutex_hash(mapping, idx); + u32 hash =3D hugetlb_fault_mutex_hash(mapping, vmf->pgoff); =20 /* * Currently, we are forced to kill the process in the event the @@ -6129,10 +6125,10 @@ static vm_fault_t hugetlb_no_page(struct mm_struct = *mm, * before we get page_table_lock. */ new_folio =3D false; - folio =3D filemap_lock_hugetlb_folio(h, mapping, idx); + folio =3D filemap_lock_hugetlb_folio(h, mapping, vmf->pgoff); if (IS_ERR(folio)) { size =3D i_size_read(mapping->host) >> huge_page_shift(h); - if (idx >=3D size) + if (vmf->pgoff >=3D size) goto out; /* Check for page in userfault range */ if (userfaultfd_missing(vma)) { @@ -6153,7 +6149,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *m= m, * never happen on the page after UFFDIO_COPY has * correctly installed the page and returned. */ - if (!hugetlb_pte_stable(h, mm, ptep, old_pte)) { + if (!hugetlb_pte_stable(h, mm, vmf->pte, vmf->orig_pte)) { ret =3D 0; goto out; } @@ -6162,7 +6158,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *m= m, VM_UFFD_MISSING); } =20 - folio =3D alloc_hugetlb_folio(vma, haddr, 0); + folio =3D alloc_hugetlb_folio(vma, vmf->address, 0); if (IS_ERR(folio)) { /* * Returning error will result in faulting task being @@ -6176,18 +6172,20 @@ static vm_fault_t hugetlb_no_page(struct mm_struct = *mm, * here. Before returning error, get ptl and make * sure there really is no pte entry. */ - if (hugetlb_pte_stable(h, mm, ptep, old_pte)) + if (hugetlb_pte_stable(h, mm, vmf->pte, vmf->orig_pte)) ret =3D vmf_error(PTR_ERR(folio)); else ret =3D 0; goto out; } - clear_huge_page(&folio->page, address, pages_per_huge_page(h)); + clear_huge_page(&folio->page, vmf->real_address, + pages_per_huge_page(h)); __folio_mark_uptodate(folio); new_folio =3D true; =20 if (vma->vm_flags & VM_MAYSHARE) { - int err =3D hugetlb_add_to_page_cache(folio, mapping, idx); + int err =3D hugetlb_add_to_page_cache(folio, mapping, + vmf->pgoff); if (err) { /* * err can't be -EEXIST which implies someone @@ -6196,7 +6194,8 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *m= m, * to the page cache. So it's safe to call * restore_reserve_on_error() here. */ - restore_reserve_on_error(h, vma, haddr, folio); + restore_reserve_on_error(h, vma, vmf->address, + folio); folio_put(folio); goto out; } @@ -6226,7 +6225,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *m= m, folio_unlock(folio); folio_put(folio); /* See comment in userfaultfd_missing() block above */ - if (!hugetlb_pte_stable(h, mm, ptep, old_pte)) { + if (!hugetlb_pte_stable(h, mm, vmf->pte, vmf->orig_pte)) { ret =3D 0; goto out; } @@ -6241,23 +6240,23 @@ static vm_fault_t hugetlb_no_page(struct mm_struct = *mm, * any allocations necessary to record that reservation occur outside * the spinlock. */ - if ((flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) { - if (vma_needs_reservation(h, vma, haddr) < 0) { + if ((vmf->flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) { + if (vma_needs_reservation(h, vma, vmf->address) < 0) { ret =3D VM_FAULT_OOM; goto backout_unlocked; } /* Just decrements count, does not deallocate */ - vma_end_reservation(h, vma, haddr); + vma_end_reservation(h, vma, vmf->address); } =20 - ptl =3D huge_pte_lock(h, mm, ptep); + vmf->ptl =3D huge_pte_lock(h, mm, vmf->pte); ret =3D 0; /* If pte changed from under us, retry */ - if (!pte_same(huge_ptep_get(ptep), old_pte)) + if (!pte_same(huge_ptep_get(vmf->pte), vmf->orig_pte)) goto backout; =20 if (anon_rmap) - hugetlb_add_new_anon_rmap(folio, vma, haddr); + hugetlb_add_new_anon_rmap(folio, vma, vmf->address); else hugetlb_add_file_rmap(folio); new_pte =3D make_huge_pte(vma, &folio->page, ((vma->vm_flags & VM_WRITE) @@ -6266,17 +6265,18 @@ static vm_fault_t hugetlb_no_page(struct mm_struct = *mm, * If this pte was previously wr-protected, keep it wr-protected even * if populated. */ - if (unlikely(pte_marker_uffd_wp(old_pte))) + if (unlikely(pte_marker_uffd_wp(vmf->orig_pte))) new_pte =3D huge_pte_mkuffd_wp(new_pte); - set_huge_pte_at(mm, haddr, ptep, new_pte, huge_page_size(h)); + set_huge_pte_at(mm, vmf->address, vmf->pte, new_pte, huge_page_size(h)); =20 hugetlb_count_add(pages_per_huge_page(h), mm); - if ((flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) { + if ((vmf->flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) { /* Optimization, do the COW without a second fault */ - ret =3D hugetlb_wp(mm, vma, address, ptep, flags, folio, ptl, vmf); + ret =3D hugetlb_wp(mm, vma, vmf->real_address, vmf->pte, + vmf->flags, folio, vmf->ptl, vmf); } =20 - spin_unlock(ptl); + spin_unlock(vmf->ptl); =20 /* * Only set hugetlb_migratable in newly allocated pages. Existing pages @@ -6293,10 +6293,10 @@ static vm_fault_t hugetlb_no_page(struct mm_struct = *mm, return ret; =20 backout: - spin_unlock(ptl); + spin_unlock(vmf->ptl); backout_unlocked: if (new_folio && !new_pagecache_folio) - restore_reserve_on_error(h, vma, haddr, folio); + restore_reserve_on_error(h, vma, vmf->address, folio); =20 folio_unlock(folio); folio_put(folio); @@ -6392,8 +6392,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct= vm_area_struct *vma, * hugetlb_no_page will drop vma lock and hugetlb fault * mutex internally, which make us return immediately. */ - return hugetlb_no_page(mm, vma, mapping, vmf.pgoff, address, - vmf.pte, vmf.orig_pte, flags, &vmf); + return hugetlb_no_page(mm, vma, mapping, &vmf); } =20 ret =3D 0; --=20 2.43.0 From nobody Mon Feb 9 01:19:29 2026 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 38F6284D15 for ; Mon, 25 Mar 2024 22:33:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711406028; cv=none; b=jQGAZ2FBsKJ7taos+LfNsaJvLAapCsDieVzQ6px6RL6YgiX9IpW994k/HaJ7FXr5zpImu046Gzvlo94cazNzc2XFu2VmdWSVg0IHkQSaFAyeiomF3Bep08zgG/fr4aszvsBuhZNNr/+zhAU2vuNV6Z93/x7bvIQiOeG3kHOvUaU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711406028; c=relaxed/simple; bh=VrAow78uMHuAALAVFOGL+L/XvOFKIVUCc06t1rv74As=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rNpDbS2yEdplazYjpEhVHewT4lOUApdROl/lpvs8on6IXdx8mC0yuy0D5WSIeZspJOSTgotGZ8ke7SZQlhCH4kRROerjuXEGmXT7y+yZmLnfmcJYkkkFfM5DohOia43D+hcum1MdcBKi80XP7qJa70hk3JnR2B+rFBbvHP2utuE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lWlQfQCI; arc=none smtp.client-ip=209.85.215.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lWlQfQCI" Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-5e8470c1cb7so3379647a12.2 for ; Mon, 25 Mar 2024 15:33:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711406025; x=1712010825; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mVOzucr5mvu7KOVzAqxUx6FA7sdm3qrl6lnpegwv/As=; b=lWlQfQCI82Nml2GZUL+QeopzbIppMQEMr5ZNbFeyg9A8eOWPQ0Y0GE6zn24cXaMhzM rG8DReUfP46I3z+FOYrkjaZPkx3Ah4t9x5uqHf3Bm1FkH5SEEpFFUzHZdjH5a/5sRXWh z15MFbUYaNtcsMGJBAM5HyjaPC0yETtVvUx3dHGSOG17L8w9HSWyHRk5o2smH9u9ARKu rzTbCgHQyEV9IQjk8r8kmKmEVvAPl8RiII8Abzd23TSsd/Lr78OIoH+v5N0s2qC3ly2Y mjpF9BXEAoln1i5Xpt+YR85FLN8E5bQXJ0yqPqch2acebvKoKxNE9D7TW6hwGpU53XgL x0yg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711406025; x=1712010825; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mVOzucr5mvu7KOVzAqxUx6FA7sdm3qrl6lnpegwv/As=; b=dCjPyYXnfNeW9tMtmOULuENga9izITTft7L8HCw3oLohVPWJQn5tVIWCz6/44vIJUc G4gAQdmY6SnqCST1ATcrXFSjdM5pEkW8buYIZjo7l2TgS0pwgRWZf2xOWnYZP4AO5ZTL jbcJFO+55zl4rsyRj5N+tPt7fJCWzWm9U2DqbilQFrqBi9gHiHsCbgnr+n7Sha1dXGzJ B+0sqpXHNDgY00eERfhCpq7bhwaNeYIQqWs+Ogz8svyHKS19J6e1ZybEIJ7DFnQlW5c0 SZ7fqgV2ddJUOuFmk/otB+dpLu7G8IzP+TCq6YvV0pRGHj+5V563dVgMejgqIiGTImBU cY5Q== X-Gm-Message-State: AOJu0YxSxAds5ZJBvNRvNqVj5FIiKS2I0g59QmUGOnJacC19K8yj4/rR 8P2bMYIL3D6qop3Voab8OHoJu4OS1UNyccQDUel5R8xx7+5RRG7S X-Google-Smtp-Source: AGHT+IFDk0utxLL00lJGKaaeGgNK3Hrl4HLtx0LXzLecUbp3ONOONhamzCvdDjqJbMq5Hf8iz/uE5Q== X-Received: by 2002:a17:90a:8a03:b0:29c:6000:a12b with SMTP id w3-20020a17090a8a0300b0029c6000a12bmr1164130pjn.38.1711406025338; Mon, 25 Mar 2024 15:33:45 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id sx16-20020a17090b2cd000b002a053cdd4e5sm4356173pjb.9.2024.03.25.15.33.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 15:33:45 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, muchun.song@linux.dev, "Vishal Moola (Oracle)" Subject: [PATCH 3/5] hugetlb: Convert hugetlb_wp() to use struct vm_fault Date: Mon, 25 Mar 2024 15:33:37 -0700 Message-ID: <20240325223339.169350-4-vishal.moola@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325223339.169350-1-vishal.moola@gmail.com> References: <20240325223339.169350-1-vishal.moola@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" hugetlb_wp() can use the struct vm_fault passed in from hugetlb_fault(). This alleviates the stack by consolidating 5 variables into a single struct. Signed-off-by: Vishal Moola (Oracle) --- mm/hugetlb.c | 61 ++++++++++++++++++++++++++-------------------------- 1 file changed, 30 insertions(+), 31 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 819a6d067985..107b47329b9f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5825,18 +5825,16 @@ static void unmap_ref_private(struct mm_struct *mm,= struct vm_area_struct *vma, * Keep the pte_same checks anyway to make transition from the mutex easie= r. */ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *= vma, - unsigned long address, pte_t *ptep, unsigned int flags, - struct folio *pagecache_folio, spinlock_t *ptl, + struct folio *pagecache_folio, struct vm_fault *vmf) { - const bool unshare =3D flags & FAULT_FLAG_UNSHARE; - pte_t pte =3D huge_ptep_get(ptep); + const bool unshare =3D vmf->flags & FAULT_FLAG_UNSHARE; + pte_t pte =3D huge_ptep_get(vmf->pte); struct hstate *h =3D hstate_vma(vma); struct folio *old_folio; struct folio *new_folio; int outside_reserve =3D 0; vm_fault_t ret =3D 0; - unsigned long haddr =3D address & huge_page_mask(h); struct mmu_notifier_range range; =20 /* @@ -5859,7 +5857,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, st= ruct vm_area_struct *vma, =20 /* Let's take out MAP_SHARED mappings first. */ if (vma->vm_flags & VM_MAYSHARE) { - set_huge_ptep_writable(vma, haddr, ptep); + set_huge_ptep_writable(vma, vmf->address, vmf->pte); return 0; } =20 @@ -5878,7 +5876,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, st= ruct vm_area_struct *vma, SetPageAnonExclusive(&old_folio->page); } if (likely(!unshare)) - set_huge_ptep_writable(vma, haddr, ptep); + set_huge_ptep_writable(vma, vmf->address, vmf->pte); =20 delayacct_wpcopy_end(); return 0; @@ -5905,8 +5903,8 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, st= ruct vm_area_struct *vma, * Drop page table lock as buddy allocator may be called. It will * be acquired again before returning to the caller, as expected. */ - spin_unlock(ptl); - new_folio =3D alloc_hugetlb_folio(vma, haddr, outside_reserve); + spin_unlock(vmf->ptl); + new_folio =3D alloc_hugetlb_folio(vma, vmf->address, outside_reserve); =20 if (IS_ERR(new_folio)) { /* @@ -5931,19 +5929,21 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, = struct vm_area_struct *vma, * * Reacquire both after unmap operation. */ - idx =3D vma_hugecache_offset(h, vma, haddr); + idx =3D vma_hugecache_offset(h, vma, vmf->address); hash =3D hugetlb_fault_mutex_hash(mapping, idx); hugetlb_vma_unlock_read(vma); mutex_unlock(&hugetlb_fault_mutex_table[hash]); =20 - unmap_ref_private(mm, vma, &old_folio->page, haddr); + unmap_ref_private(mm, vma, &old_folio->page, + vmf->address); =20 mutex_lock(&hugetlb_fault_mutex_table[hash]); hugetlb_vma_lock_read(vma); - spin_lock(ptl); - ptep =3D hugetlb_walk(vma, haddr, huge_page_size(h)); - if (likely(ptep && - pte_same(huge_ptep_get(ptep), pte))) + spin_lock(vmf->ptl); + vmf->pte =3D hugetlb_walk(vma, vmf->address, + huge_page_size(h)); + if (likely(vmf->pte && + pte_same(huge_ptep_get(vmf->pte), pte))) goto retry_avoidcopy; /* * race occurs while re-acquiring page table @@ -5965,37 +5965,38 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, = struct vm_area_struct *vma, if (unlikely(ret)) goto out_release_all; =20 - if (copy_user_large_folio(new_folio, old_folio, address, vma)) { + if (copy_user_large_folio(new_folio, old_folio, vmf->real_address, vma)) { ret =3D VM_FAULT_HWPOISON_LARGE; goto out_release_all; } __folio_mark_uptodate(new_folio); =20 - mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, haddr, - haddr + huge_page_size(h)); + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, vmf->address, + vmf->address + huge_page_size(h)); mmu_notifier_invalidate_range_start(&range); =20 /* * Retake the page table lock to check for racing updates * before the page tables are altered */ - spin_lock(ptl); - ptep =3D hugetlb_walk(vma, haddr, huge_page_size(h)); - if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) { + spin_lock(vmf->ptl); + vmf->pte =3D hugetlb_walk(vma, vmf->address, huge_page_size(h)); + if (likely(vmf->pte && pte_same(huge_ptep_get(vmf->pte), pte))) { pte_t newpte =3D make_huge_pte(vma, &new_folio->page, !unshare); =20 /* Break COW or unshare */ - huge_ptep_clear_flush(vma, haddr, ptep); + huge_ptep_clear_flush(vma, vmf->address, vmf->pte); hugetlb_remove_rmap(old_folio); - hugetlb_add_new_anon_rmap(new_folio, vma, haddr); + hugetlb_add_new_anon_rmap(new_folio, vma, vmf->address); if (huge_pte_uffd_wp(pte)) newpte =3D huge_pte_mkuffd_wp(newpte); - set_huge_pte_at(mm, haddr, ptep, newpte, huge_page_size(h)); + set_huge_pte_at(mm, vmf->address, vmf->pte, newpte, + huge_page_size(h)); folio_set_hugetlb_migratable(new_folio); /* Make the old page be freed below */ new_folio =3D old_folio; } - spin_unlock(ptl); + spin_unlock(vmf->ptl); mmu_notifier_invalidate_range_end(&range); out_release_all: /* @@ -6003,12 +6004,12 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, = struct vm_area_struct *vma, * unshare) */ if (new_folio !=3D old_folio) - restore_reserve_on_error(h, vma, haddr, new_folio); + restore_reserve_on_error(h, vma, vmf->address, new_folio); folio_put(new_folio); out_release_old: folio_put(old_folio); =20 - spin_lock(ptl); /* Caller expects lock to be held */ + spin_lock(vmf->ptl); /* Caller expects lock to be held */ =20 delayacct_wpcopy_end(); return ret; @@ -6272,8 +6273,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *m= m, hugetlb_count_add(pages_per_huge_page(h), mm); if ((vmf->flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) { /* Optimization, do the COW without a second fault */ - ret =3D hugetlb_wp(mm, vma, vmf->real_address, vmf->pte, - vmf->flags, folio, vmf->ptl, vmf); + ret =3D hugetlb_wp(mm, vma, folio, vmf); } =20 spin_unlock(vmf->ptl); @@ -6486,8 +6486,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct= vm_area_struct *vma, =20 if (flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) { if (!huge_pte_write(vmf.orig_pte)) { - ret =3D hugetlb_wp(mm, vma, address, vmf.pte, flags, - pagecache_folio, vmf.ptl, &vmf); + ret =3D hugetlb_wp(mm, vma, pagecache_folio, &vmf); goto out_put_page; } else if (likely(flags & FAULT_FLAG_WRITE)) { vmf.orig_pte =3D huge_pte_mkdirty(vmf.orig_pte); --=20 2.43.0 From nobody Mon Feb 9 01:19:29 2026 Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com [209.85.215.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D725B84D08 for ; Mon, 25 Mar 2024 22:33:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711406028; cv=none; b=qmtLBdWJ158EU0Gpn/Wmkx/d9SXLzO+3drG0LNOnPnrnd4InXesYpWlt67DmOol/QTXCCE4xaofkS5ANKzbm3C6tWCWKMc6DKFl5R7xXD5+MXEOzSj7nhp2F5oCkjDh29QzbUU2Sw8V61+ZrzoLVcy06AuIW6dD3BvVmx4okt4w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711406028; c=relaxed/simple; bh=nWy26S5cj5CXdPcBf9dbCQGQIk5BbZMPt9JbLu2oKN4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pxUj4XGtZEi4R62ibl1q/sHdatU6T2/q2BN5QqEMjXYqchfQdCnXBkhKQn5LO2TDr4EQQx8ujCfet5zhftMjgL+muoTQ/GuEUjuOeiCDVzKzSbSTOsv9NmdyLCEd0nwRIWhcy9lm4OSztfnBwh6oAD8xYKdXNmaZGGdyTl0UIdM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=CKmhSCDR; arc=none smtp.client-ip=209.85.215.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CKmhSCDR" Received: by mail-pg1-f174.google.com with SMTP id 41be03b00d2f7-5cedfc32250so3042056a12.0 for ; Mon, 25 Mar 2024 15:33:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711406026; x=1712010826; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BlCoIx7jLqN5Is/HEKy53SFm13f31lm8FF5rHEKGzHA=; b=CKmhSCDR/SyjUms0IqsUSLSK/9OJ1cisFMRDb4zc2DgTEuXjvYjbE4P/lOfuadZ0cK yuBPw2CZmzSBKq9ev+NamjIR4O1UViu/KiNoCdYvD4hnO8BHY/o1rxgtrXrdkt6Hk4Vi nq0qM/5gk7Gxy2lGIEoVM6ZSX3GuG2pn6kWmCjHq52MfmYCZYp63tapnDO13l52bNM4a tLby7py23iRiIdvnCzL2zQyLp4rrPQStvx7UpclyA8phK7dxolerggkLa9UB+sMg/KaE eY9Z9GOTJoELwJ2WGAAUrGl1De388OgCPDJgCxlou9OFtrrSnNM2TG1CqrBVC1E74u2y pLwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711406026; x=1712010826; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BlCoIx7jLqN5Is/HEKy53SFm13f31lm8FF5rHEKGzHA=; b=gRaB6F2kXzQtHItzYBS/IuNSuaQ/5Q3UJEp9Vcq12OEQaEM8At9zJQOC0B//Irwzk4 3uKhpQX4pBVJZFLqX/gipnbqKSrWluqTg4GDATsIGPx3lXm114x5UzOEk79C7jh57N0l S6aZLTStvbZdtv7Aw9wRW8vYvxm7s19cEqjuR3zqBoYIdOE2KhXLqCzvIxjUpN0yve0V +9zajcD9eAsuL12N0DJI1tDMgdB7wQd/izTW4pM12FUjGEsnAsHqDI2GG8hf+r+VfkU3 S0+tvjOlcGiVRXaOL9fLgTZG1Nfhjie8CKi4eR3cZkKHmdRCamOX9ZdXmWyIxEK2f4ny SBhA== X-Gm-Message-State: AOJu0YxduqzRVgBpJnWow8FFA+NAEBjJfzJApHQ63YVX+SSveE4VnMeM d/6bPkXFrReDbeZIQbqPbP7llfIpKgXbqIVV0azgvdYRpgcx9PJ8 X-Google-Smtp-Source: AGHT+IHqkDjxiCBXirJRGAONJPRKYdtlto3CBeWqAPDu5xBIWL2dfiUZrx6HsrjVwyLsjkfUNcQ0aQ== X-Received: by 2002:a17:90b:f08:b0:29b:c2b3:2712 with SMTP id br8-20020a17090b0f0800b0029bc2b32712mr6441165pjb.26.1711406026161; Mon, 25 Mar 2024 15:33:46 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id sx16-20020a17090b2cd000b002a053cdd4e5sm4356173pjb.9.2024.03.25.15.33.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 15:33:45 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, muchun.song@linux.dev, "Vishal Moola (Oracle)" Subject: [PATCH 4/5] mm: Make pgoff non-const in struct vm_fault Date: Mon, 25 Mar 2024 15:33:38 -0700 Message-ID: <20240325223339.169350-5-vishal.moola@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325223339.169350-1-vishal.moola@gmail.com> References: <20240325223339.169350-1-vishal.moola@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Hugetlb calculates addresses and page offsets differently from the rest of mm. In order to pass struct vm_fault through the fault pathway we will let hugetlb_fault() and __handle_mm_fault() set those variables themselves instead. Signed-off-by: Vishal Moola (Oracle) --- include/linux/mm.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index f5a97dec5169..c6874aa7b7f0 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -507,10 +507,11 @@ struct vm_fault { const struct { struct vm_area_struct *vma; /* Target VMA */ gfp_t gfp_mask; /* gfp mask to be used for allocations */ - pgoff_t pgoff; /* Logical page offset based on vma */ - unsigned long address; /* Faulting virtual address - masked */ unsigned long real_address; /* Faulting virtual address - unmasked */ }; + unsigned long address; /* Faulting virtual address - masked */ + pgoff_t pgoff; /* Logical page offset based on vma */ + enum fault_flag flags; /* FAULT_FLAG_xxx flags * XXX: should really be 'const' */ pmd_t *pmd; /* Pointer to pmd entry matching --=20 2.43.0 From nobody Mon Feb 9 01:19:29 2026 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4BE2184D35 for ; Mon, 25 Mar 2024 22:33:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711406030; cv=none; b=R+vvmb4WXEyE/nOHsXzvDNTPveQ2dMghg1iLEJPcOOLakwIer1dhhF18ArDPXSw070TovUJ8sK2ZuuSEsomUPDCQiHXkAL43492xhn44CuBSSUUZuQuDpkgC3BwPtZ+KXt5fLSia+B+iwX99rlCbGTKZSYa897daKc/IagQ6FzY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711406030; c=relaxed/simple; bh=tBZdwXXPGB1xRNSiIH7furKYYODiVU7bBfowL2w3GfA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KLs4DV8Qgk3JXAAeYhm0XXvtfAfu31ktzNVPnHoiWaGzHtFDteGV42mbGZgWNLW2RGZ9FPTh35jg+LWod34bVEBj9/GQzFQcih9qpmGoCJx/mSCgsV8IK+r7xi2IYsYe1Y90nlGnBT1mPlIgjMz5xoYkLjN9O3q0EDadzDasyTQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Qp1KvW6N; arc=none smtp.client-ip=209.85.216.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Qp1KvW6N" Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-29fb12a22afso3264523a91.3 for ; Mon, 25 Mar 2024 15:33:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711406028; x=1712010828; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xa0qOW5rWdjbo+w+QnIgdTmY7x8+eguqE2o56xDAiCo=; b=Qp1KvW6NVBKkkqNk1dGgax1WAB0wLfthV6nfS5h3OLig2YtRvEwDqoi+QfWWgQewa4 ryGKl2suHY2PdiY/PJyNrK8CdP51FZ37tTuARQO1z63n0s+Zzk7tmFKeqr78NBsGAz+x GvACJtsXw/dx83drgdwKSGNH3XthmnLsKcJSJKrmYEuhP2VDPakscBBSeKIEb4HXNdMG YSioh5x+Plp8Uz1zMyxEesxpDzQml3MAMSoMm342s+AYXxbwyLnVkLCLV5E8qv3wYhdr OrMrq+tjhCbglybe414cBRE0d2IXPq//Hr1f7/5PTtJGJlhurbwyrWB9GTCmIc0SJ7BL 07IA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711406028; x=1712010828; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xa0qOW5rWdjbo+w+QnIgdTmY7x8+eguqE2o56xDAiCo=; b=SI2lkNdWoxswUG7mrmJG/3qMGaGQv1X8KVN8BlizUjbXV/1jx+SoBDSnBw2AjdNl78 N2u5h6nhj7u6moao4iSa3xQBiSXENfqzZOvpEU1iTcqq9w65x+BFxoZTsxuKW9Q4dZjV GmhLHhnGt+y2TQa5Y+wstcvdxmmLpPLkkBgjvduuWTcBkB8s3ydl0bzsmOcg7RES39L0 AofD2mMkNa9aqOOR6xPp5VyM4RSp2DWBDc7emTSa0EfhOyOvDRijAAcAvB1XRcDeLru/ 8grTTYjvLF+N0zpJCzokL9oTNg8cK4/kHGV0rCozUfv0g01FakVXI0r1g1kw8IyjbsJ2 sdVw== X-Gm-Message-State: AOJu0YxoBJKOwN3K7j2bU3h5i/THpJPWsvqjctUfBEBkGf1pKe4FZYKj nLF6TVTlDKwKQu9cK6vV5ZRxOsBU5Cp83OZ5MoEbtlr5IkG0hTc/f6RZ9GV9OEE= X-Google-Smtp-Source: AGHT+IFRm0SsVYiVz6wu55o5NNu1OirYSh8Wnv/Y+CNLuAFTeLpCWCaL50rUVsBgY/cv4dzw/fjRUw== X-Received: by 2002:a17:90a:16c2:b0:29a:3c70:1525 with SMTP id y2-20020a17090a16c200b0029a3c701525mr5308309pje.45.1711406027625; Mon, 25 Mar 2024 15:33:47 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id sx16-20020a17090b2cd000b002a053cdd4e5sm4356173pjb.9.2024.03.25.15.33.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 15:33:47 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, muchun.song@linux.dev, "Vishal Moola (Oracle)" Subject: [PATCH 5/5] memory: Define struct vm_fault in handle_mm_fault() Date: Mon, 25 Mar 2024 15:33:39 -0700 Message-ID: <20240325223339.169350-6-vishal.moola@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325223339.169350-1-vishal.moola@gmail.com> References: <20240325223339.169350-1-vishal.moola@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Define struct vm_fault in handle_mm_fault() to be passed throughout the rest of the fault pathway. Pass it through to hugetlb_fault() and __handle_mm_fault(), making any necessary trivial changes. Signed-off-by: Vishal Moola (Oracle) --- include/linux/hugetlb.h | 7 +-- mm/hugetlb.c | 106 +++++++++++++++++++--------------------- mm/memory.c | 87 +++++++++++++++++---------------- 3 files changed, 98 insertions(+), 102 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index c1ee640d87b1..0e0a93b4d9fc 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -147,8 +147,7 @@ void hugetlb_report_meminfo(struct seq_file *); int hugetlb_report_node_meminfo(char *buf, int len, int nid); void hugetlb_show_meminfo_node(int nid); unsigned long hugetlb_total_pages(void); -vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long address, unsigned int flags); +vm_fault_t hugetlb_fault(struct vm_fault *vmf); #ifdef CONFIG_USERFAULTFD int hugetlb_mfill_atomic_pte(pte_t *dst_pte, struct vm_area_struct *dst_vma, @@ -482,9 +481,7 @@ static inline void __unmap_hugepage_range(struct mmu_ga= ther *tlb, BUG(); } =20 -static inline vm_fault_t hugetlb_fault(struct mm_struct *mm, - struct vm_area_struct *vma, unsigned long address, - unsigned int flags) +static inline vm_fault_t hugetlb_fault(struct vm_fault *vmf) { BUG(); return 0; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 107b47329b9f..7ecc680f4681 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6327,30 +6327,24 @@ u32 hugetlb_fault_mutex_hash(struct address_space *= mapping, pgoff_t idx) } #endif =20 -vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long address, unsigned int flags) +vm_fault_t hugetlb_fault(struct vm_fault *vmf) { vm_fault_t ret; u32 hash; + struct vm_area_struct *vma =3D vmf->vma; + struct mm_struct *mm =3D vma->vm_mm; struct folio *folio =3D NULL; struct folio *pagecache_folio =3D NULL; struct hstate *h =3D hstate_vma(vma); struct address_space *mapping; int need_wait_lock =3D 0; - struct vm_fault vmf =3D { - .vma =3D vma, - .address =3D address & huge_page_mask(h), - .real_address =3D address, - .flags =3D flags, - .pgoff =3D vma_hugecache_offset(h, vma, - address & huge_page_mask(h)), - /* TODO: Track hugetlb faults using vm_fault */ - - /* - * Some fields may not be initialized, be careful as it may - * be hard to debug if called functions make assumptions - */ - }; + /* + * Some fields of vmf may not be initialized, be careful as it may + * be hard to debug if called functions make assumptions + */ + vmf->address =3D vmf->real_address & huge_page_mask(h); + vmf->pgoff =3D vma_hugecache_offset(h, vma, + vmf->address & huge_page_mask(h)); =20 /* * Serialize hugepage allocation and instantiation, so that we don't @@ -6358,27 +6352,27 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, stru= ct vm_area_struct *vma, * the same page in the page cache. */ mapping =3D vma->vm_file->f_mapping; - hash =3D hugetlb_fault_mutex_hash(mapping, vmf.pgoff); + hash =3D hugetlb_fault_mutex_hash(mapping, vmf->pgoff); mutex_lock(&hugetlb_fault_mutex_table[hash]); =20 /* * Acquire vma lock before calling huge_pte_alloc and hold - * until finished with vmf.pte. This prevents huge_pmd_unshare from - * being called elsewhere and making the vmf.pte no longer valid. + * until finished with vmf->pte. This prevents huge_pmd_unshare from + * being called elsewhere and making the vmf->pte no longer valid. */ hugetlb_vma_lock_read(vma); - vmf.pte =3D huge_pte_alloc(mm, vma, vmf.address, huge_page_size(h)); - if (!vmf.pte) { + vmf->pte =3D huge_pte_alloc(mm, vma, vmf->address, huge_page_size(h)); + if (!vmf->pte) { hugetlb_vma_unlock_read(vma); mutex_unlock(&hugetlb_fault_mutex_table[hash]); return VM_FAULT_OOM; } =20 - vmf.orig_pte =3D huge_ptep_get(vmf.pte); - if (huge_pte_none_mostly(vmf.orig_pte)) { - if (is_pte_marker(vmf.orig_pte)) { + vmf->orig_pte =3D huge_ptep_get(vmf->pte); + if (huge_pte_none_mostly(vmf->orig_pte)) { + if (is_pte_marker(vmf->orig_pte)) { pte_marker marker =3D - pte_marker_get(pte_to_swp_entry(vmf.orig_pte)); + pte_marker_get(pte_to_swp_entry(vmf->orig_pte)); =20 if (marker & PTE_MARKER_POISONED) { ret =3D VM_FAULT_HWPOISON_LARGE; @@ -6392,20 +6386,20 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, stru= ct vm_area_struct *vma, * hugetlb_no_page will drop vma lock and hugetlb fault * mutex internally, which make us return immediately. */ - return hugetlb_no_page(mm, vma, mapping, &vmf); + return hugetlb_no_page(mm, vma, mapping, vmf); } =20 ret =3D 0; =20 /* - * vmf.orig_pte could be a migration/hwpoison vmf.orig_pte at this + * vmf->orig_pte could be a migration/hwpoison vmf->orig_pte at this * point, so this check prevents the kernel from going below assuming * that we have an active hugepage in pagecache. This goto expects * the 2nd page fault, and is_hugetlb_entry_(migration|hwpoisoned) * check will properly handle it. */ - if (!pte_present(vmf.orig_pte)) { - if (unlikely(is_hugetlb_entry_migration(vmf.orig_pte))) { + if (!pte_present(vmf->orig_pte)) { + if (unlikely(is_hugetlb_entry_migration(vmf->orig_pte))) { /* * Release the hugetlb fault lock now, but retain * the vma lock, because it is needed to guard the @@ -6414,9 +6408,9 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct= vm_area_struct *vma, * be released there. */ mutex_unlock(&hugetlb_fault_mutex_table[hash]); - migration_entry_wait_huge(vma, vmf.pte); + migration_entry_wait_huge(vma, vmf->pte); return 0; - } else if (unlikely(is_hugetlb_entry_hwpoisoned(vmf.orig_pte))) + } else if (unlikely(is_hugetlb_entry_hwpoisoned(vmf->orig_pte))) ret =3D VM_FAULT_HWPOISON_LARGE | VM_FAULT_SET_HINDEX(hstate_index(h)); goto out_mutex; @@ -6429,53 +6423,53 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, stru= ct vm_area_struct *vma, * spinlock. Also lookup the pagecache page now as it is used to * determine if a reservation has been consumed. */ - if ((flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) && - !(vma->vm_flags & VM_MAYSHARE) && !huge_pte_write(vmf.orig_pte)) { - if (vma_needs_reservation(h, vma, vmf.address) < 0) { + if ((vmf->flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) && + !(vma->vm_flags & VM_MAYSHARE) && !huge_pte_write(vmf->orig_pte)) { + if (vma_needs_reservation(h, vma, vmf->address) < 0) { ret =3D VM_FAULT_OOM; goto out_mutex; } /* Just decrements count, does not deallocate */ - vma_end_reservation(h, vma, vmf.address); + vma_end_reservation(h, vma, vmf->address); =20 pagecache_folio =3D filemap_lock_hugetlb_folio(h, mapping, - vmf.pgoff); + vmf->pgoff); if (IS_ERR(pagecache_folio)) pagecache_folio =3D NULL; } =20 - vmf.ptl =3D huge_pte_lock(h, mm, vmf.pte); + vmf->ptl =3D huge_pte_lock(h, mm, vmf->pte); =20 /* Check for a racing update before calling hugetlb_wp() */ - if (unlikely(!pte_same(vmf.orig_pte, huge_ptep_get(vmf.pte)))) + if (unlikely(!pte_same(vmf->orig_pte, huge_ptep_get(vmf->pte)))) goto out_ptl; =20 /* Handle userfault-wp first, before trying to lock more pages */ - if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(vmf.pte)) && - (flags & FAULT_FLAG_WRITE) && !huge_pte_write(vmf.orig_pte)) { + if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(vmf->pte)) && + (vmf->flags & FAULT_FLAG_WRITE) && !huge_pte_write(vmf->orig_pte)) { if (!userfaultfd_wp_async(vma)) { - spin_unlock(vmf.ptl); + spin_unlock(vmf->ptl); if (pagecache_folio) { folio_unlock(pagecache_folio); folio_put(pagecache_folio); } hugetlb_vma_unlock_read(vma); mutex_unlock(&hugetlb_fault_mutex_table[hash]); - return handle_userfault(&vmf, VM_UFFD_WP); + return handle_userfault(vmf, VM_UFFD_WP); } =20 - vmf.orig_pte =3D huge_pte_clear_uffd_wp(vmf.orig_pte); - set_huge_pte_at(mm, vmf.address, vmf.pte, vmf.orig_pte, + vmf->orig_pte =3D huge_pte_clear_uffd_wp(vmf->orig_pte); + set_huge_pte_at(mm, vmf->address, vmf->pte, vmf->orig_pte, huge_page_size(hstate_vma(vma))); /* Fallthrough to CoW */ } =20 /* - * hugetlb_wp() requires page locks of pte_page(vmf.orig_pte) and + * hugetlb_wp() requires page locks of pte_page(vmf->orig_pte) and * pagecache_folio, so here we need take the former one * when folio !=3D pagecache_folio or !pagecache_folio. */ - folio =3D page_folio(pte_page(vmf.orig_pte)); + folio =3D page_folio(pte_page(vmf->orig_pte)); if (folio !=3D pagecache_folio) if (!folio_trylock(folio)) { need_wait_lock =3D 1; @@ -6484,24 +6478,24 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, stru= ct vm_area_struct *vma, =20 folio_get(folio); =20 - if (flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) { - if (!huge_pte_write(vmf.orig_pte)) { - ret =3D hugetlb_wp(mm, vma, pagecache_folio, &vmf); + if (vmf->flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) { + if (!huge_pte_write(vmf->orig_pte)) { + ret =3D hugetlb_wp(mm, vma, pagecache_folio, vmf); goto out_put_page; - } else if (likely(flags & FAULT_FLAG_WRITE)) { - vmf.orig_pte =3D huge_pte_mkdirty(vmf.orig_pte); + } else if (likely(vmf->flags & FAULT_FLAG_WRITE)) { + vmf->orig_pte =3D huge_pte_mkdirty(vmf->orig_pte); } } - vmf.orig_pte =3D pte_mkyoung(vmf.orig_pte); - if (huge_ptep_set_access_flags(vma, vmf.address, vmf.pte, vmf.orig_pte, - flags & FAULT_FLAG_WRITE)) - update_mmu_cache(vma, vmf.address, vmf.pte); + vmf->orig_pte =3D pte_mkyoung(vmf->orig_pte); + if (huge_ptep_set_access_flags(vma, vmf->address, vmf->pte, + vmf->orig_pte, vmf->flags & FAULT_FLAG_WRITE)) + update_mmu_cache(vma, vmf->address, vmf->pte); out_put_page: if (folio !=3D pagecache_folio) folio_unlock(folio); folio_put(folio); out_ptl: - spin_unlock(vmf.ptl); + spin_unlock(vmf->ptl); =20 if (pagecache_folio) { folio_unlock(pagecache_folio); diff --git a/mm/memory.c b/mm/memory.c index c93b058adfb2..a2fcb0322b11 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5191,39 +5191,35 @@ static vm_fault_t handle_pte_fault(struct vm_fault = *vmf) * the result, the mmap_lock is not held on exit. See filemap_fault() * and __folio_lock_or_retry(). */ -static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, - unsigned long address, unsigned int flags) +static vm_fault_t __handle_mm_fault(struct vm_fault *vmf) { - struct vm_fault vmf =3D { - .vma =3D vma, - .address =3D address & PAGE_MASK, - .real_address =3D address, - .flags =3D flags, - .pgoff =3D linear_page_index(vma, address), - .gfp_mask =3D __get_fault_gfp_mask(vma), - }; + struct vm_area_struct *vma =3D vmf->vma; struct mm_struct *mm =3D vma->vm_mm; unsigned long vm_flags =3D vma->vm_flags; + const unsigned long address =3D vmf->real_address; pgd_t *pgd; p4d_t *p4d; vm_fault_t ret; =20 + vmf->address =3D address & PAGE_MASK; + vmf->pgoff =3D linear_page_index(vma, address); pgd =3D pgd_offset(mm, address); p4d =3D p4d_alloc(mm, pgd, address); if (!p4d) return VM_FAULT_OOM; =20 - vmf.pud =3D pud_alloc(mm, p4d, address); - if (!vmf.pud) + vmf->pud =3D pud_alloc(mm, p4d, address); + if (!vmf->pud) return VM_FAULT_OOM; retry_pud: - if (pud_none(*vmf.pud) && - thp_vma_allowable_order(vma, vm_flags, false, true, true, PUD_ORDER))= { - ret =3D create_huge_pud(&vmf); + if (pud_none(*vmf->pud) && + thp_vma_allowable_order(vma, vm_flags, false, true, + true, PUD_ORDER)) { + ret =3D create_huge_pud(vmf); if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - pud_t orig_pud =3D *vmf.pud; + pud_t orig_pud =3D *vmf->pud; =20 barrier(); if (pud_trans_huge(orig_pud) || pud_devmap(orig_pud)) { @@ -5232,57 +5228,60 @@ static vm_fault_t __handle_mm_fault(struct vm_area_= struct *vma, * TODO once we support anonymous PUDs: NUMA case and * FAULT_FLAG_UNSHARE handling. */ - if ((flags & FAULT_FLAG_WRITE) && !pud_write(orig_pud)) { - ret =3D wp_huge_pud(&vmf, orig_pud); + if ((vmf->flags & FAULT_FLAG_WRITE) && + !pud_write(orig_pud)) { + ret =3D wp_huge_pud(vmf, orig_pud); if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - huge_pud_set_accessed(&vmf, orig_pud); + huge_pud_set_accessed(vmf, orig_pud); return 0; } } } =20 - vmf.pmd =3D pmd_alloc(mm, vmf.pud, address); - if (!vmf.pmd) + vmf->pmd =3D pmd_alloc(mm, vmf->pud, address); + if (!vmf->pmd) return VM_FAULT_OOM; =20 /* Huge pud page fault raced with pmd_alloc? */ - if (pud_trans_unstable(vmf.pud)) + if (pud_trans_unstable(vmf->pud)) goto retry_pud; =20 - if (pmd_none(*vmf.pmd) && - thp_vma_allowable_order(vma, vm_flags, false, true, true, PMD_ORDER))= { - ret =3D create_huge_pmd(&vmf); + if (pmd_none(*vmf->pmd) && + thp_vma_allowable_order(vma, vm_flags, false, true, + true, PMD_ORDER)) { + ret =3D create_huge_pmd(vmf); if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - vmf.orig_pmd =3D pmdp_get_lockless(vmf.pmd); + vmf->orig_pmd =3D pmdp_get_lockless(vmf->pmd); =20 - if (unlikely(is_swap_pmd(vmf.orig_pmd))) { + if (unlikely(is_swap_pmd(vmf->orig_pmd))) { VM_BUG_ON(thp_migration_supported() && - !is_pmd_migration_entry(vmf.orig_pmd)); - if (is_pmd_migration_entry(vmf.orig_pmd)) - pmd_migration_entry_wait(mm, vmf.pmd); + !is_pmd_migration_entry(vmf->orig_pmd)); + if (is_pmd_migration_entry(vmf->orig_pmd)) + pmd_migration_entry_wait(mm, vmf->pmd); return 0; } - if (pmd_trans_huge(vmf.orig_pmd) || pmd_devmap(vmf.orig_pmd)) { - if (pmd_protnone(vmf.orig_pmd) && vma_is_accessible(vma)) - return do_huge_pmd_numa_page(&vmf); - - if ((flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) && - !pmd_write(vmf.orig_pmd)) { - ret =3D wp_huge_pmd(&vmf); + if (pmd_trans_huge(vmf->orig_pmd) || + pmd_devmap(vmf->orig_pmd)) { + if (pmd_protnone(vmf->orig_pmd) && vma_is_accessible(vma)) + return do_huge_pmd_numa_page(vmf); + + if ((vmf->flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) + && !pmd_write(vmf->orig_pmd)) { + ret =3D wp_huge_pmd(vmf); if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - huge_pmd_set_accessed(&vmf); + huge_pmd_set_accessed(vmf); return 0; } } } =20 - return handle_pte_fault(&vmf); + return handle_pte_fault(vmf); } =20 /** @@ -5421,6 +5420,12 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vm= a, unsigned long address, /* If the fault handler drops the mmap_lock, vma may be freed */ struct mm_struct *mm =3D vma->vm_mm; vm_fault_t ret; + struct vm_fault vmf =3D { + .vma =3D vma, + .real_address =3D address, + .flags =3D flags, + .gfp_mask =3D __get_fault_gfp_mask(vma), + }; =20 __set_current_state(TASK_RUNNING); =20 @@ -5445,9 +5450,9 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma= , unsigned long address, lru_gen_enter_fault(vma); =20 if (unlikely(is_vm_hugetlb_page(vma))) - ret =3D hugetlb_fault(vma->vm_mm, vma, address, flags); + ret =3D hugetlb_fault(&vmf); else - ret =3D __handle_mm_fault(vma, address, flags); + ret =3D __handle_mm_fault(&vmf); =20 lru_gen_exit_fault(); =20 --=20 2.43.0