From nobody Sat Feb 7 21:15:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78633EB64D7 for ; Wed, 28 Jun 2023 21:54:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231828AbjF1VyN (ORCPT ); Wed, 28 Jun 2023 17:54:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37760 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231447AbjF1VyG (ORCPT ); Wed, 28 Jun 2023 17:54:06 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 36C62210E for ; Wed, 28 Jun 2023 14:53:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989199; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d8FL5BW3box1F48wj3b8rnHOt4NGgVkomS4C4Z085kw=; b=ic2L1rOpJ0/6YWQyVinnSV+wo4gaMSAydTYHE4XfVtt0UfnxJEfMyvkTsq0flqMxvHHk6R iC4lQAKDvClQj7YsyioEOcXpwcn4q13Og/q6pjNHhlDyHh6ivzPNwQNtP3KEFfECpyHsCh tgZ4CXkR+emPaYQdk7tc8UV0Fg3fX30= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-278-XGuUAjBEOgOhU_ifJYkfFw-1; Wed, 28 Jun 2023 17:53:14 -0400 X-MC-Unique: XGuUAjBEOgOhU_ifJYkfFw-1 Received: by mail-qv1-f69.google.com with SMTP id 6a1803df08f44-635f8ae5f88so39216d6.0 for ; Wed, 28 Jun 2023 14:53:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989194; x=1690581194; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d8FL5BW3box1F48wj3b8rnHOt4NGgVkomS4C4Z085kw=; b=aEv07ZggtSsJ5Lgec8AApn2cWsSXopBZ8N9XCAGr6UgIdsr4l0lc0hN6cyiVp1Y8Gx u79mdSSIFXN9Dl+hGK3DwC/Y6JcWsl50PuxCtb7pKYnY0ZBeWhpiMhMc9MfIq/0TBcjV uUdaWj9WOcHGF/HgpSpMuKn/S8m+mdDGw5Q7yHoedROgi7cQs2vDxmI/FyKcO6vM+91h B2JrCKujx7X70lIeP0L0t6kXIaQxzr+yxW3SqOfXJw1uaga5uERQ23pnvkLTst33dZpN FqKuresk64z02rT3hZET57Z6ouEeHsqydAt1S7XulTW0coDj1brEbTWmG11eRWWIbqpM bKOw== X-Gm-Message-State: AC+VfDwJK+BG+EJ9Njfoy3VqwBtZttpDjKH1Rito48ikzMM8WhclqsbN aNO23qYmqAwlsFY9qVQ7uPqqGlKywAJ5Hlpvxkrktd6yWYNL/GFWmKFzlke4CuniE+OsNI74wBc T55TS2KlZamMZWsW/ODqhKwqG X-Received: by 2002:a05:6214:401a:b0:62d:fdc4:1e8b with SMTP id kd26-20020a056214401a00b0062dfdc41e8bmr42375749qvb.2.1687989194251; Wed, 28 Jun 2023 14:53:14 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4MC34MgzekDp0X74lVrIdzH9S5zruaiTrJfdeBf4bPnhG07fp5o87gyAcA985czOX/TxVoqw== X-Received: by 2002:a05:6214:401a:b0:62d:fdc4:1e8b with SMTP id kd26-20020a056214401a00b0062dfdc41e8bmr42375738qvb.2.1687989194020; Wed, 28 Jun 2023 14:53:14 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:13 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 1/8] mm/hugetlb: Handle FOLL_DUMP well in follow_page_mask() Date: Wed, 28 Jun 2023 17:53:03 -0400 Message-ID: <20230628215310.73782-2-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Firstly, the no_page_table() is meaningless for hugetlb which is a no-op there, because a hugetlb page always satisfies: - vma_is_anonymous() =3D=3D false - vma->vm_ops->fault !=3D NULL So we can already safely remove it in hugetlb_follow_page_mask(), alongside with the page* variable. Meanwhile, what we do in follow_hugetlb_page() actually makes sense for a dump: we try to fault in the page only if the page cache is already allocated. Let's do the same here for follow_page_mask() on hugetlb. It should so far has zero effect on real dumps, because that still goes into follow_hugetlb_page(). But this may start to influence a bit on follow_page() users who mimics a "dump page" scenario, but hopefully in a good way. This also paves way for unifying the hugetlb gup-slow. Reviewed-by: Mike Kravetz Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu --- mm/gup.c | 9 ++------- mm/hugetlb.c | 9 +++++++++ 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index a8336b39d6b5..06bc2e31f209 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -767,7 +767,6 @@ static struct page *follow_page_mask(struct vm_area_str= uct *vma, struct follow_page_context *ctx) { pgd_t *pgd; - struct page *page; struct mm_struct *mm =3D vma->vm_mm; =20 ctx->page_mask =3D 0; @@ -780,12 +779,8 @@ static struct page *follow_page_mask(struct vm_area_st= ruct *vma, * hugetlb_follow_page_mask is only for follow_page() handling here. * Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ - if (is_vm_hugetlb_page(vma)) { - page =3D hugetlb_follow_page_mask(vma, address, flags); - if (!page) - page =3D no_page_table(vma, flags); - return page; - } + if (is_vm_hugetlb_page(vma)) + return hugetlb_follow_page_mask(vma, address, flags); =20 pgd =3D pgd_offset(mm, address); =20 diff --git a/mm/hugetlb.c b/mm/hugetlb.c index bce28cca73a1..d04ba5782fdd 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6498,6 +6498,15 @@ struct page *hugetlb_follow_page_mask(struct vm_area= _struct *vma, spin_unlock(ptl); out_unlock: hugetlb_vma_unlock_read(vma); + + /* + * Fixup retval for dump requests: if pagecache doesn't exist, + * don't try to allocate a new page but just skip it. + */ + if (!page && (flags & FOLL_DUMP) && + !hugetlbfs_pagecache_present(h, vma, address)) + page =3D ERR_PTR(-EFAULT); + return page; } =20 --=20 2.41.0 From nobody Sat Feb 7 21:15:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71AF3EB64D7 for ; Wed, 28 Jun 2023 21:54:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232117AbjF1VyY (ORCPT ); Wed, 28 Jun 2023 17:54:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37748 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231303AbjF1VyJ (ORCPT ); Wed, 28 Jun 2023 17:54:09 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B47E61FFB for ; Wed, 28 Jun 2023 14:53:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989197; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v3NUxws11iZimZs89ga1+mchuciCcK5t8cj7qgaLP/g=; b=OEl983S61xAqUc7taiXElasALKboGk6fgtZlBFgUd3JbyO63frx7h8JFAIq9/CMAz5m3Kd p6gr7hj052LFA+Milgu40T1D1LtvRkcc/GZnl9NwOl0bexv3vNv45p1tVaFCv/09zukdP2 aNVoUAp2Pqa47jUVLdeeflnZ8lC8zCg= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-558-JOaH0agJOf6-Ev5MPtQW0A-1; Wed, 28 Jun 2023 17:53:16 -0400 X-MC-Unique: JOaH0agJOf6-Ev5MPtQW0A-1 Received: by mail-qv1-f72.google.com with SMTP id 6a1803df08f44-62dd79f63e0so60436d6.0 for ; Wed, 28 Jun 2023 14:53:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989195; x=1690581195; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v3NUxws11iZimZs89ga1+mchuciCcK5t8cj7qgaLP/g=; b=KtfDYZi4WJPdTgeral5vIpmT5bcvhfjPh57yKrTRUDfCUY2MeHnlYGdrtM/OGnuhvJ /fDaeZWKQkcwcawYT1aF6FgeRb5O+WSDd3VdbLQuqsqOgWuINYxLyVf7GJUoD2bNvL0F g04zGBKT7QwQaKGv9nPVD+4Pv/eLPIu9krWLUMkqoSvSalhgYBouT9QpemVTpAPPAcPI 1NrXMxBJDZ5RMy1QY6C2+0OKM9r47A6N4USodc2c1w3VwZA9U+YEXcIUT8t9gjO7nBlp REuAtz9jzoedQH+yRBJCk9jy+cj0cdMtQ2FkxbZ78ywAnxhcNjnSuK8rQ+PPrvyPz5pO Ou+w== X-Gm-Message-State: AC+VfDyueb1hZSCRtAOP/g9plnRY+BjX1WCFtuQhPr7TK8D7rYrgdCZl qkQKtRASk85m7RGw6WxU2AlL4qkR2VfR1u0A4xWDT7h3jRruk9Cb0dt0zC9+5Y8DbtwazBZ8iBj 9nG0Dr/0is829bfVXlIOzJy2t X-Received: by 2002:a05:6214:f2f:b0:62b:6c6f:b3e3 with SMTP id iw15-20020a0562140f2f00b0062b6c6fb3e3mr45427363qvb.3.1687989195524; Wed, 28 Jun 2023 14:53:15 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6HBU77CP7xJIw4KayPpNq4pGfYosv7bGjCnvBLq1S/WNi+e+thj4Rrto2qPbX4dx7awzouIw== X-Received: by 2002:a05:6214:f2f:b0:62b:6c6f:b3e3 with SMTP id iw15-20020a0562140f2f00b0062b6c6fb3e3mr45427346qvb.3.1687989195254; Wed, 28 Jun 2023 14:53:15 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:15 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 2/8] mm/hugetlb: Prepare hugetlb_follow_page_mask() for FOLL_PIN Date: Wed, 28 Jun 2023 17:53:04 -0400 Message-ID: <20230628215310.73782-3-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" follow_page() doesn't use FOLL_PIN, meanwhile hugetlb seems to not be the target of FOLL_WRITE either. However add the checks. Namely, either the need to CoW due to missing write bit, or proper unsharing on !AnonExclusive pages over R/O pins to reject the follow page. That brings this function closer to follow_hugetlb_page(). So we don't care before, and also for now. But we'll care if we switch over slow-gup to use hugetlb_follow_page_mask(). We'll also care when to return -EMLINK properly, as that's the gup internal api to mean "we should unshare". Not really needed for follow page path, though. When at it, switching the try_grab_page() to use WARN_ON_ONCE(), to be clear that it just should never fail. When error happens, instead of setting page=3D=3DNULL, capture the errno instead. Reviewed-by: Mike Kravetz Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu --- mm/hugetlb.c | 33 ++++++++++++++++++++++----------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d04ba5782fdd..4410139cf890 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6462,13 +6462,7 @@ struct page *hugetlb_follow_page_mask(struct vm_area= _struct *vma, struct page *page =3D NULL; spinlock_t *ptl; pte_t *pte, entry; - - /* - * FOLL_PIN is not supported for follow_page(). Ordinary GUP goes via - * follow_hugetlb_page(). - */ - if (WARN_ON_ONCE(flags & FOLL_PIN)) - return NULL; + int ret; =20 hugetlb_vma_lock_read(vma); pte =3D hugetlb_walk(vma, haddr, huge_page_size(h)); @@ -6478,8 +6472,23 @@ struct page *hugetlb_follow_page_mask(struct vm_area= _struct *vma, ptl =3D huge_pte_lock(h, mm, pte); entry =3D huge_ptep_get(pte); if (pte_present(entry)) { - page =3D pte_page(entry) + - ((address & ~huge_page_mask(h)) >> PAGE_SHIFT); + page =3D pte_page(entry); + + if (!huge_pte_write(entry)) { + if (flags & FOLL_WRITE) { + page =3D NULL; + goto out; + } + + if (gup_must_unshare(vma, flags, page)) { + /* Tell the caller to do unsharing */ + page =3D ERR_PTR(-EMLINK); + goto out; + } + } + + page +=3D ((address & ~huge_page_mask(h)) >> PAGE_SHIFT); + /* * Note that page may be a sub-page, and with vmemmap * optimizations the page struct may be read only. @@ -6489,8 +6498,10 @@ struct page *hugetlb_follow_page_mask(struct vm_area= _struct *vma, * try_grab_page() should always be able to get the page here, * because we hold the ptl lock and have verified pte_present(). */ - if (try_grab_page(page, flags)) { - page =3D NULL; + ret =3D try_grab_page(page, flags); + + if (WARN_ON_ONCE(ret)) { + page =3D ERR_PTR(ret); goto out; } } --=20 2.41.0 From nobody Sat Feb 7 21:15:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11050EB64D7 for ; Wed, 28 Jun 2023 21:54:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232154AbjF1Vy1 (ORCPT ); Wed, 28 Jun 2023 17:54:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37770 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231637AbjF1VyJ (ORCPT ); Wed, 28 Jun 2023 17:54:09 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D2022111 for ; Wed, 28 Jun 2023 14:53:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989199; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IlLdQZDWL4abQusa0fWM05NgtC5yViIwxz1MKIy8bPA=; b=ZgkP7MNeWTBFLqPLJ1hIGcTOfs+K3Ubo7sNdgL/x80kV3dU1HNoTsPwBEq4a96EaqY8tCR dSH0Q+PDkk7ZC2VgZCPR26esbSbwrjRjegUEEBPSXKLuuO2j86/RRZmYT9ZzclvGpFd5oj GD+DnvXwRDWbSVbiwt6+SsnzjIEsrzk= Received: from mail-vk1-f198.google.com (mail-vk1-f198.google.com [209.85.221.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-335-lg8yFQ42PUCqedzxscCwcA-1; Wed, 28 Jun 2023 17:53:17 -0400 X-MC-Unique: lg8yFQ42PUCqedzxscCwcA-1 Received: by mail-vk1-f198.google.com with SMTP id 71dfb90a1353d-4716b1f55b9so2889e0c.0 for ; Wed, 28 Jun 2023 14:53:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989197; x=1690581197; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IlLdQZDWL4abQusa0fWM05NgtC5yViIwxz1MKIy8bPA=; b=d078f08flziWSVLAsr/H1JDvkV+WYJ46hM6vzWIwJloj+G7lSGzKI1J2HTSZK4fHia V/M8ZAP6b+682BLj/C1lRA1qeEWIpfvPGqLm0NVIgRfXBrv+LXjb4QLkNY4Nh0q/Ao2L a5ulKt0adNY5f/oiXNOPGPFaNpue8HhmxjTNoCAqPdBV9dBRLcB69jDT/A/CR4GOOjxk RP9aTSGqHA9wOBp/kUd4iNVPrAis3jzo8B84UTdJcLf0EUDwFy+K1qDm8UaXquqQmmXo zohUF01NgX/Wl4i5jQOYWLrPN9e5jcl/oqcoommB6KXn0VgvSZA3vyuWo7QcAmr1oM7H 5fkg== X-Gm-Message-State: AC+VfDwb/zm9HJ5L/ZQRPccNPt9ezGEQv3PVXGdswDbL8qQ2b5pNgXbn Hfsb6+/zvwQV1vH7RY7jCzM4UYbQw6DTSZSFQ/28u5NgyQzKKh//b3VdcTPv61/iZh4VsAq58SB GgZeBsGHx/1+H3umIHSZeLM51 X-Received: by 2002:a05:6102:358d:b0:440:b763:a69d with SMTP id h13-20020a056102358d00b00440b763a69dmr12045541vsu.2.1687989196707; Wed, 28 Jun 2023 14:53:16 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7xAhw0w/TD6i35B4jAPAsfvCfbDcR0SS33V5o7YLBIW4QDiUOfL0qQyToMemXcWuE4mGgWTQ== X-Received: by 2002:a05:6102:358d:b0:440:b763:a69d with SMTP id h13-20020a056102358d00b00440b763a69dmr12045528vsu.2.1687989196447; Wed, 28 Jun 2023 14:53:16 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:16 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 3/8] mm/hugetlb: Add page_mask for hugetlb_follow_page_mask() Date: Wed, 28 Jun 2023 17:53:05 -0400 Message-ID: <20230628215310.73782-4-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" follow_page() doesn't need it, but we'll start to need it when unifying gup for hugetlb. Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 8 +++++--- mm/gup.c | 3 ++- mm/hugetlb.c | 5 ++++- 3 files changed, 11 insertions(+), 5 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index ca3c8e10f24a..9f282f370d96 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -131,7 +131,8 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, int copy_hugetlb_page_range(struct mm_struct *, struct mm_struct *, struct vm_area_struct *, struct vm_area_struct *); struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags); + unsigned long address, unsigned int flags, + unsigned int *page_mask); long follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *, struct page **, unsigned long *, unsigned long *, long, unsigned int, int *); @@ -297,8 +298,9 @@ static inline void adjust_range_if_pmd_sharing_possible( { } =20 -static inline struct page *hugetlb_follow_page_mask(struct vm_area_struct = *vma, - unsigned long address, unsigned int flags) +static inline struct page *hugetlb_follow_page_mask( + struct vm_area_struct *vma, unsigned long address, unsigned int flags, + unsigned int *page_mask) { BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ } diff --git a/mm/gup.c b/mm/gup.c index 06bc2e31f209..1e2e23084f3c 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -780,7 +780,8 @@ static struct page *follow_page_mask(struct vm_area_str= uct *vma, * Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ if (is_vm_hugetlb_page(vma)) - return hugetlb_follow_page_mask(vma, address, flags); + return hugetlb_follow_page_mask(vma, address, flags, + &ctx->page_mask); =20 pgd =3D pgd_offset(mm, address); =20 diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4410139cf890..15e82a8a2b76 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6454,7 +6454,8 @@ static inline bool __follow_hugetlb_must_fault(struct= vm_area_struct *vma, } =20 struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags) + unsigned long address, unsigned int flags, + unsigned int *page_mask) { struct hstate *h =3D hstate_vma(vma); struct mm_struct *mm =3D vma->vm_mm; @@ -6504,6 +6505,8 @@ struct page *hugetlb_follow_page_mask(struct vm_area_= struct *vma, page =3D ERR_PTR(ret); goto out; } + + *page_mask =3D (1U << huge_page_order(h)) - 1; } out: spin_unlock(ptl); --=20 2.41.0 From nobody Sat Feb 7 21:15:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 178E1EB64D7 for ; Wed, 28 Jun 2023 21:54:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232192AbjF1Vyc (ORCPT ); Wed, 28 Jun 2023 17:54:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37764 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231636AbjF1VyJ (ORCPT ); Wed, 28 Jun 2023 17:54:09 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B4CC2110 for ; Wed, 28 Jun 2023 14:53:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989199; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=C1A4ivN2AFC1lD1f3DRS9KX47rVe/ku79YHFPXkOfKI=; b=Kt5ZAVDdr/jVhGx1pHDLciGBS5vMnFB4Ut2NcAbS5uYT0z1LRk4hQLNBXKmPIjBybkDoLm AKdANP0+1OwJzS32vBeWGapRVMeCNTZFe2u/R+xtmIllhpxxvhar9qT+8Yz29M2ZzGkLZD yNeKT/bQwUxyPULf8pU1+t2udNgVMzA= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-244-Repg9S1sOki2fFtIypOy6g-1; Wed, 28 Jun 2023 17:53:18 -0400 X-MC-Unique: Repg9S1sOki2fFtIypOy6g-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-635df023844so9736d6.1 for ; Wed, 28 Jun 2023 14:53:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989198; x=1690581198; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C1A4ivN2AFC1lD1f3DRS9KX47rVe/ku79YHFPXkOfKI=; b=HWx37WgvLrFAFvxRb62f5CdxN3t01hlsq6cxG3AnCyKUUEpok9TcaPw1vwHxZG0R0o 77DjUbORdmvzfWWI2We12w/1k8+s4RDFlnTKf2W8vGrlMwQjx62zem4XoyBJz5Tpwbv7 rn1uWdKAOzxaP1kWMH/tAuKPfi7zX2c1MCiPxJjoeOst2e3xFEgHRf1IcRsZw2O5wi3c sbZBFEubftghBFpdTo2oc4Ub2XZmiZjuYOEr/3oMo36dpn5Pgc1dya/hzVxUrVRr4H2V DTq2XqOxWXDJyM6rI/IcS0Ioz2dfA6JLq7EPnFaq3j/fqoQJNVEvGYMZEVRrjKFug3rJ ltCw== X-Gm-Message-State: AC+VfDyRO6WFc7FoUjC8DN5IyZSrXHH+GuGoKX3yPLZzYGrsNkw/rJXa Dqr0SpJ/O6sZIi9Gp0jjPi9GV6N/UE0ltlTEjT/FqIBgSYipXRhF65K+C1vrri452OcBKvYw1Oi e0Sir6apH8lgLdGocPyuIWGOP X-Received: by 2002:a05:6214:301a:b0:635:da19:a67f with SMTP id ke26-20020a056214301a00b00635da19a67fmr13098204qvb.1.1687989198001; Wed, 28 Jun 2023 14:53:18 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4bWthzbIpu8brAzYWFel0vezukPFHw5JtwGCHRd23ZlsKMqKd/fjN3mRP22Dyi1lIzaJO53Q== X-Received: by 2002:a05:6214:301a:b0:635:da19:a67f with SMTP id ke26-20020a056214301a00b00635da19a67fmr13098180qvb.1.1687989197705; Wed, 28 Jun 2023 14:53:17 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:17 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 4/8] mm/gup: Cleanup next_page handling Date: Wed, 28 Jun 2023 17:53:06 -0400 Message-ID: <20230628215310.73782-5-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The only path that doesn't use generic "**pages" handling is the gate vma. Make it use the same path, meanwhile tune the next_page label upper to cover "**pages" handling. This prepares for THP handling for "**pages". Reviewed-by: Lorenzo Stoakes Acked-by: David Hildenbrand Signed-off-by: Peter Xu --- mm/gup.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 1e2e23084f3c..5af1be81390b 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1124,7 +1124,7 @@ static long __get_user_pages(struct mm_struct *mm, if (!vma && in_gate_area(mm, start)) { ret =3D get_gate_page(mm, start & PAGE_MASK, gup_flags, &vma, - pages ? &pages[i] : NULL); + pages ? &page : NULL); if (ret) goto out; ctx.page_mask =3D 0; @@ -1194,19 +1194,18 @@ static long __get_user_pages(struct mm_struct *mm, ret =3D PTR_ERR(page); goto out; } - - goto next_page; } else if (IS_ERR(page)) { ret =3D PTR_ERR(page); goto out; } +next_page: if (pages) { pages[i] =3D page; flush_anon_page(vma, page, start); flush_dcache_page(page); ctx.page_mask =3D 0; } -next_page: + page_increm =3D 1 + (~(start >> PAGE_SHIFT) & ctx.page_mask); if (page_increm > nr_pages) page_increm =3D nr_pages; --=20 2.41.0 From nobody Sat Feb 7 21:15:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED1F4EB64D7 for ; Wed, 28 Jun 2023 21:55:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232295AbjF1Vze (ORCPT ); Wed, 28 Jun 2023 17:55:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232364AbjF1Vyv (ORCPT ); Wed, 28 Jun 2023 17:54:51 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4FD7F211D for ; Wed, 28 Jun 2023 14:53:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989203; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LErGcSwJladPWljtQDBZEDG1gs3DbaZNQ9dZknt9+SI=; b=bE1pmSAGRaHObDruWiFGWoU2a0fcU2xPfyl3oncN6oCiWwKj5aNjSaLYs8vu/QCXmkoDMU XuAbRgG1Lr38UfrRf6rgNWcBGL1AbhwSAH1unYIn/y0/mbsORqW2YK+7ZDVb4E4qN8MGlg zMRLGlppt+4IJVTOR4q7ncFFNSV0VEM= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-644-gJ9xuyYVOUOJFMofO_LFbA-1; Wed, 28 Jun 2023 17:53:19 -0400 X-MC-Unique: gJ9xuyYVOUOJFMofO_LFbA-1 Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-635e2618aaeso42496d6.0 for ; Wed, 28 Jun 2023 14:53:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989199; x=1690581199; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LErGcSwJladPWljtQDBZEDG1gs3DbaZNQ9dZknt9+SI=; b=cDV/hM+b43d+MbPRxVRdMMTBye2AcJ05aMtildRSxipnD1rl+9KaJ5FsFhFvcCdSY6 GzDYSzZ4t+RXBmpfyWLY9rdZfVz6khzhJLR1Ev37W70ypkU4/NKs1ZXJ3d7WQ6XAJ2PX M0wzC8gdZcyXz8t5uQHmZnbRZOkUcDkwhBlsnYHkJt8JliBae8zEGtMGgWELSB0ihEeb y2zIT3DPCV/+dUrftaamlS1OSOMfDZHQL/qta/H5gSUD+scs2ETWU28dn+n3vb4hwFpd aGM8XBddVWHuCNb142mlBhTWcOVL0gKxPk91xzvAvD/sd5IP88J5esECzn57l6ZTDxJp NAyQ== X-Gm-Message-State: AC+VfDwaJGSOGeCiqpuykHnkNht0NjWisg7doKMM8GBv2P9xuS1bZztL JYBbdiuL9gY1Vd/Wal7KrSAXPLgpqHJZ9qM/fYeJI+8dEbSP5zryfQxMrqLI1B/BQmnZApJ/WZF ZSGapHYUpetYVWYixoHgt9jak X-Received: by 2002:a05:6214:501d:b0:621:65de:f60c with SMTP id jo29-20020a056214501d00b0062165def60cmr6650320qvb.3.1687989199421; Wed, 28 Jun 2023 14:53:19 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4aH92Xekg+ZXFKVELiAnLIK82VapCJUdD+ZE2LzSIKvWv5j/RuoE6PplAHYjC4on1+uql7Bw== X-Received: by 2002:a05:6214:501d:b0:621:65de:f60c with SMTP id jo29-20020a056214501d00b0062165def60cmr6650303qvb.3.1687989199149; Wed, 28 Jun 2023 14:53:19 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:18 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 5/8] mm/gup: Accelerate thp gup even for "pages != NULL" Date: Wed, 28 Jun 2023 17:53:07 -0400 Message-ID: <20230628215310.73782-6-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The acceleration of THP was done with ctx.page_mask, however it'll be ignored if **pages is non-NULL. The old optimization was introduced in 2013 in 240aadeedc4a ("mm: accelerate mm_populate() treatment of THP pages"). It didn't explain why we can't optimize the **pages non-NULL case. It's possible that at that time the major goal was for mm_populate() which should be enough back then. Optimize thp for all cases, by properly looping over each subpage, doing cache flushes, and boost refcounts / pincounts where needed in one go. This can be verified using gup_test below: # chrt -f 1 ./gup_test -m 512 -t -L -n 1024 -r 10 Before: 13992.50 ( +-8.75%) After: 378.50 (+-69.62%) Reviewed-by: Lorenzo Stoakes Signed-off-by: Peter Xu --- mm/gup.c | 51 ++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 44 insertions(+), 7 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 5af1be81390b..0e2b0ff1143a 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1199,16 +1199,53 @@ static long __get_user_pages(struct mm_struct *mm, goto out; } next_page: - if (pages) { - pages[i] =3D page; - flush_anon_page(vma, page, start); - flush_dcache_page(page); - ctx.page_mask =3D 0; - } - page_increm =3D 1 + (~(start >> PAGE_SHIFT) & ctx.page_mask); if (page_increm > nr_pages) page_increm =3D nr_pages; + + if (pages) { + struct page *subpage; + unsigned int j; + + /* + * This must be a large folio (and doesn't need to + * be the whole folio; it can be part of it), do + * the refcount work for all the subpages too. + * + * NOTE: here the page may not be the head page + * e.g. when start addr is not thp-size aligned. + * try_grab_folio() should have taken care of tail + * pages. + */ + if (page_increm > 1) { + struct folio *folio; + + /* + * Since we already hold refcount on the + * large folio, this should never fail. + */ + folio =3D try_grab_folio(page, page_increm - 1, + foll_flags); + if (WARN_ON_ONCE(!folio)) { + /* + * Release the 1st page ref if the + * folio is problematic, fail hard. + */ + gup_put_folio(page_folio(page), 1, + foll_flags); + ret =3D -EFAULT; + goto out; + } + } + + for (j =3D 0; j < page_increm; j++) { + subpage =3D nth_page(page, j); + pages[i + j] =3D subpage; + flush_anon_page(vma, subpage, start + j * PAGE_SIZE); + flush_dcache_page(subpage); + } + } + i +=3D page_increm; start +=3D page_increm * PAGE_SIZE; nr_pages -=3D page_increm; --=20 2.41.0 From nobody Sat Feb 7 21:15:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEEC9EB64D7 for ; Wed, 28 Jun 2023 21:55:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231974AbjF1Vz0 (ORCPT ); Wed, 28 Jun 2023 17:55:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37838 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232357AbjF1Vyu (ORCPT ); Wed, 28 Jun 2023 17:54:50 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 786BF2129 for ; Wed, 28 Jun 2023 14:53:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989204; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f1uXyVFMcTV1oTRkqq3pr4O9sWiRe8mnSq6RPVSEFpA=; b=SIeCXVT0iCAceyn7bh1j3lTmzls/+92QkmMJgQyBUplE0AfwSmijPQ4Bdgo5emPgsVdkk2 Q97P4ABvMY24v6nSrm1PVaWjtk+Oo65HkhjHMOUNC/xvnA8dP+g32R9uUa3/uvFoWMuw7s hnEyrB4SBEqE7hINEO6x8va7TmO6Gn8= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-187-n002aZ1hMY60RK_efr0tPQ-1; Wed, 28 Jun 2023 17:53:21 -0400 X-MC-Unique: n002aZ1hMY60RK_efr0tPQ-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-635df023844so9866d6.1 for ; Wed, 28 Jun 2023 14:53:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989201; x=1690581201; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=f1uXyVFMcTV1oTRkqq3pr4O9sWiRe8mnSq6RPVSEFpA=; b=UKlDJYNga3hcmuBChk+RqkrZ+5oM2UD/jrmcbP73Z4ZHKyMrr99Ep8hZVnT/kPvlyf ce1Q9E2pkXH2ZldozUPddxqQuprg5DCmFD+2viP2K+WAapt0yAdFaH2sR7xlx9Eup70O w7n6mBhb36atakSjjMb8TfmXbj0jR7LjK63VRuQoIVim5ITKjB6+GzG05sEF63OB5V3T +X9oJPTswMDlb9WUvSrzdScmjBG+PGGigOWRgUifzjm9LuHNXHePur4f+hGyNObKsxPf RXncGoVksVjQyhxLsuK7XPN76Qc7QWefodsRwGBtU7ii6d30MifvAcfaIU70hOKnrfkC ZVkQ== X-Gm-Message-State: AC+VfDxQedFtMGe09Z95XEpvJUoyTgkIk3RUKGi5JI93mGo2AXGrzvyy r66IjQiWnFrlCzEvPonlCrn9J56M5O2Bu0s5Kc7E1tmv/uNx6s5zsaf3mBAi11J4LasJUa1PZ+h 7yjTyd/ZGq22CHi8k74XBRHMH X-Received: by 2002:a05:6214:c6c:b0:62f:f6ac:abf5 with SMTP id t12-20020a0562140c6c00b0062ff6acabf5mr43014705qvj.5.1687989200984; Wed, 28 Jun 2023 14:53:20 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6IPT/qUgAkouDwxzob72SaB3s+apQpeAgKJRetCHfNucooMDrCa2F7cSJb+eYJVhrci9iAmQ== X-Received: by 2002:a05:6214:c6c:b0:62f:f6ac:abf5 with SMTP id t12-20020a0562140c6c00b0062ff6acabf5mr43014693qvj.5.1687989200642; Wed, 28 Jun 2023 14:53:20 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:20 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 6/8] mm/gup: Retire follow_hugetlb_page() Date: Wed, 28 Jun 2023 17:53:08 -0400 Message-ID: <20230628215310.73782-7-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Now __get_user_pages() should be well prepared to handle thp completely, as long as hugetlb gup requests even without the hugetlb's special path. Time to retire follow_hugetlb_page(). Tweak misc comments to reflect reality of follow_hugetlb_page()'s removal. Acked-by: David Hildenbrand Signed-off-by: Peter Xu --- fs/userfaultfd.c | 2 +- include/linux/hugetlb.h | 12 --- mm/gup.c | 19 ---- mm/hugetlb.c | 224 ---------------------------------------- 4 files changed, 1 insertion(+), 256 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 7cecd49e078b..ae711f1d7a83 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -427,7 +427,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsig= ned long reason) * * We also don't do userfault handling during * coredumping. hugetlbfs has the special - * follow_hugetlb_page() to skip missing pages in the + * hugetlb_follow_page_mask() to skip missing pages in the * FOLL_DUMP case, anon memory also checks for FOLL_DUMP with * the no_page_table() helper in follow_page_mask(), but the * shmem_vm_ops->fault method is invoked even during diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 9f282f370d96..9bc3c2d71b71 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -133,9 +133,6 @@ int copy_hugetlb_page_range(struct mm_struct *, struct = mm_struct *, struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, unsigned long address, unsigned int flags, unsigned int *page_mask); -long follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *, - struct page **, unsigned long *, unsigned long *, - long, unsigned int, int *); void unmap_hugepage_range(struct vm_area_struct *, unsigned long, unsigned long, struct page *, zap_flags_t); @@ -305,15 +302,6 @@ static inline struct page *hugetlb_follow_page_mask( BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ } =20 -static inline long follow_hugetlb_page(struct mm_struct *mm, - struct vm_area_struct *vma, struct page **pages, - unsigned long *position, unsigned long *nr_pages, - long i, unsigned int flags, int *nonblocking) -{ - BUG(); - return 0; -} - static inline int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, struct vm_area_struct *dst_vma, diff --git a/mm/gup.c b/mm/gup.c index 0e2b0ff1143a..a7c294de6ae5 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -775,9 +775,6 @@ static struct page *follow_page_mask(struct vm_area_str= uct *vma, * Call hugetlb_follow_page_mask for hugetlb vmas as it will use * special hugetlb page table walking code. This eliminates the * need to check for hugetlb entries in the general walking code. - * - * hugetlb_follow_page_mask is only for follow_page() handling here. - * Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ if (is_vm_hugetlb_page(vma)) return hugetlb_follow_page_mask(vma, address, flags, @@ -1138,22 +1135,6 @@ static long __get_user_pages(struct mm_struct *mm, ret =3D check_vma_flags(vma, gup_flags); if (ret) goto out; - - if (is_vm_hugetlb_page(vma)) { - i =3D follow_hugetlb_page(mm, vma, pages, - &start, &nr_pages, i, - gup_flags, locked); - if (!*locked) { - /* - * We've got a VM_FAULT_RETRY - * and we've lost mmap_lock. - * We must stop here. - */ - BUG_ON(gup_flags & FOLL_NOWAIT); - goto out; - } - continue; - } } retry: /* diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 15e82a8a2b76..2f12da409a19 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5721,7 +5721,6 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, st= ruct vm_area_struct *vma, =20 /* * Return whether there is a pagecache page to back given address within V= MA. - * Caller follow_hugetlb_page() holds page_table_lock so we cannot lock_pa= ge. */ static bool hugetlbfs_pagecache_present(struct hstate *h, struct vm_area_struct *vma, unsigned long address) @@ -6422,37 +6421,6 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, } #endif /* CONFIG_USERFAULTFD */ =20 -static void record_subpages(struct page *page, struct vm_area_struct *vma, - int refs, struct page **pages) -{ - int nr; - - for (nr =3D 0; nr < refs; nr++) { - if (likely(pages)) - pages[nr] =3D nth_page(page, nr); - } -} - -static inline bool __follow_hugetlb_must_fault(struct vm_area_struct *vma, - unsigned int flags, pte_t *pte, - bool *unshare) -{ - pte_t pteval =3D huge_ptep_get(pte); - - *unshare =3D false; - if (is_swap_pte(pteval)) - return true; - if (huge_pte_write(pteval)) - return false; - if (flags & FOLL_WRITE) - return true; - if (gup_must_unshare(vma, flags, pte_page(pteval))) { - *unshare =3D true; - return true; - } - return false; -} - struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, unsigned long address, unsigned int flags, unsigned int *page_mask) @@ -6524,198 +6492,6 @@ struct page *hugetlb_follow_page_mask(struct vm_are= a_struct *vma, return page; } =20 -long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, - struct page **pages, unsigned long *position, - unsigned long *nr_pages, long i, unsigned int flags, - int *locked) -{ - unsigned long pfn_offset; - unsigned long vaddr =3D *position; - unsigned long remainder =3D *nr_pages; - struct hstate *h =3D hstate_vma(vma); - int err =3D -EFAULT, refs; - - while (vaddr < vma->vm_end && remainder) { - pte_t *pte; - spinlock_t *ptl =3D NULL; - bool unshare =3D false; - int absent; - struct page *page; - - /* - * If we have a pending SIGKILL, don't keep faulting pages and - * potentially allocating memory. - */ - if (fatal_signal_pending(current)) { - remainder =3D 0; - break; - } - - hugetlb_vma_lock_read(vma); - /* - * Some archs (sparc64, sh*) have multiple pte_ts to - * each hugepage. We have to make sure we get the - * first, for the page indexing below to work. - * - * Note that page table lock is not held when pte is null. - */ - pte =3D hugetlb_walk(vma, vaddr & huge_page_mask(h), - huge_page_size(h)); - if (pte) - ptl =3D huge_pte_lock(h, mm, pte); - absent =3D !pte || huge_pte_none(huge_ptep_get(pte)); - - /* - * When coredumping, it suits get_dump_page if we just return - * an error where there's an empty slot with no huge pagecache - * to back it. This way, we avoid allocating a hugepage, and - * the sparse dumpfile avoids allocating disk blocks, but its - * huge holes still show up with zeroes where they need to be. - */ - if (absent && (flags & FOLL_DUMP) && - !hugetlbfs_pagecache_present(h, vma, vaddr)) { - if (pte) - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - remainder =3D 0; - break; - } - - /* - * We need call hugetlb_fault for both hugepages under migration - * (in which case hugetlb_fault waits for the migration,) and - * hwpoisoned hugepages (in which case we need to prevent the - * caller from accessing to them.) In order to do this, we use - * here is_swap_pte instead of is_hugetlb_entry_migration and - * is_hugetlb_entry_hwpoisoned. This is because it simply covers - * both cases, and because we can't follow correct pages - * directly from any kind of swap entries. - */ - if (absent || - __follow_hugetlb_must_fault(vma, flags, pte, &unshare)) { - vm_fault_t ret; - unsigned int fault_flags =3D 0; - - if (pte) - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - - if (flags & FOLL_WRITE) - fault_flags |=3D FAULT_FLAG_WRITE; - else if (unshare) - fault_flags |=3D FAULT_FLAG_UNSHARE; - if (locked) { - fault_flags |=3D FAULT_FLAG_ALLOW_RETRY | - FAULT_FLAG_KILLABLE; - if (flags & FOLL_INTERRUPTIBLE) - fault_flags |=3D FAULT_FLAG_INTERRUPTIBLE; - } - if (flags & FOLL_NOWAIT) - fault_flags |=3D FAULT_FLAG_ALLOW_RETRY | - FAULT_FLAG_RETRY_NOWAIT; - if (flags & FOLL_TRIED) { - /* - * Note: FAULT_FLAG_ALLOW_RETRY and - * FAULT_FLAG_TRIED can co-exist - */ - fault_flags |=3D FAULT_FLAG_TRIED; - } - ret =3D hugetlb_fault(mm, vma, vaddr, fault_flags); - if (ret & VM_FAULT_ERROR) { - err =3D vm_fault_to_errno(ret, flags); - remainder =3D 0; - break; - } - if (ret & VM_FAULT_RETRY) { - if (locked && - !(fault_flags & FAULT_FLAG_RETRY_NOWAIT)) - *locked =3D 0; - *nr_pages =3D 0; - /* - * VM_FAULT_RETRY must not return an - * error, it will return zero - * instead. - * - * No need to update "position" as the - * caller will not check it after - * *nr_pages is set to 0. - */ - return i; - } - continue; - } - - pfn_offset =3D (vaddr & ~huge_page_mask(h)) >> PAGE_SHIFT; - page =3D pte_page(huge_ptep_get(pte)); - - VM_BUG_ON_PAGE((flags & FOLL_PIN) && PageAnon(page) && - !PageAnonExclusive(page), page); - - /* - * If subpage information not requested, update counters - * and skip the same_page loop below. - */ - if (!pages && !pfn_offset && - (vaddr + huge_page_size(h) < vma->vm_end) && - (remainder >=3D pages_per_huge_page(h))) { - vaddr +=3D huge_page_size(h); - remainder -=3D pages_per_huge_page(h); - i +=3D pages_per_huge_page(h); - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - continue; - } - - /* vaddr may not be aligned to PAGE_SIZE */ - refs =3D min3(pages_per_huge_page(h) - pfn_offset, remainder, - (vma->vm_end - ALIGN_DOWN(vaddr, PAGE_SIZE)) >> PAGE_SHIFT); - - if (pages) - record_subpages(nth_page(page, pfn_offset), - vma, refs, - likely(pages) ? pages + i : NULL); - - if (pages) { - /* - * try_grab_folio() should always succeed here, - * because: a) we hold the ptl lock, and b) we've just - * checked that the huge page is present in the page - * tables. If the huge page is present, then the tail - * pages must also be present. The ptl prevents the - * head page and tail pages from being rearranged in - * any way. As this is hugetlb, the pages will never - * be p2pdma or not longterm pinable. So this page - * must be available at this point, unless the page - * refcount overflowed: - */ - if (WARN_ON_ONCE(!try_grab_folio(pages[i], refs, - flags))) { - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - remainder =3D 0; - err =3D -ENOMEM; - break; - } - } - - vaddr +=3D (refs << PAGE_SHIFT); - remainder -=3D refs; - i +=3D refs; - - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - } - *nr_pages =3D remainder; - /* - * setting position is actually required only if remainder is - * not zero but it's faster not to add a "if (remainder)" - * branch. - */ - *position =3D vaddr; - - return i ? i : err; -} - long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags) --=20 2.41.0 From nobody Sat Feb 7 21:15:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F7CFEB64D7 for ; Wed, 28 Jun 2023 21:55:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232236AbjF1Vz3 (ORCPT ); Wed, 28 Jun 2023 17:55:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37842 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232369AbjF1Vyv (ORCPT ); Wed, 28 Jun 2023 17:54:51 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E97492127 for ; Wed, 28 Jun 2023 14:53:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989204; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8HfHdG/KQ5Uxfq5f0w/MDNpC44bP5ckORXtaAeg9+TI=; b=ZXrUgH2nVSv0WUOunmRgqKL8X0seYS0+KTaECkZc8pfdt2tM/4mkOI0VGWm33gBhmhEUBJ HxD5NBJ+wNy81U//o49gLi8MAIPDLc59BOPC9JRG958q3y6SsitFnueHSCrSbmmRY7AHNl Ze9ZGr81znxkjQO4FUyiFH1HvNY7+sg= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-584-kSXfzjzJNNu6lqB3GXRxFA-1; Wed, 28 Jun 2023 17:53:22 -0400 X-MC-Unique: kSXfzjzJNNu6lqB3GXRxFA-1 Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-635f8ae5f88so39436d6.0 for ; Wed, 28 Jun 2023 14:53:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989202; x=1690581202; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8HfHdG/KQ5Uxfq5f0w/MDNpC44bP5ckORXtaAeg9+TI=; b=GoDaY9wQRXOwQBbqFfNXAW8wQtt/fuanxzhap9543mwplBHrKLNQ5up4jvJXmhSyCG mgdQ+oWR2KBjdewXyXLVjjMZ1EcE7rk+vEHazpwD3dteLNZji/l4CWldQ1nsa5jbvqEB D4Zhh83r0FADAia/6qp4/X7Ei+hIZEfXkFYZUL0l++HGM/QPDmRcBIul1nRooEzhC3ma 0DxFScfvSCRveab7eZOVQwCoxINMii6j/+S9oZXoO+XzHd4ApbSB5vz++wpfrSQuMRAV fXHH1uBgEH2EjEzceqDAnp1c8BOXpdYazy6OudpMWAaw+SHs21qOFZhTRGBnk7a3HOGO xW3A== X-Gm-Message-State: AC+VfDxn4EqsJZ84td0o0bfpPXy652+wV5sEkLAqKi/aOwnNYv24wRAB Y1LxsoWazhZrq/TdXu2N4RaQzGuYKkPcV+I2J6w7bHZ2Yh3jJVkGrneP23HUq7tVOlhJjoW1dQt wLZrpyXImamUXtaiNNOAL1dVx X-Received: by 2002:a05:6214:260e:b0:626:2305:6073 with SMTP id gu14-20020a056214260e00b0062623056073mr6706614qvb.4.1687989202308; Wed, 28 Jun 2023 14:53:22 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4hmEwEfHLsDUveWKtt2eKqu3uz2U6qRFVCfBKaJKRVOW25O/2xvrvScvNnG1pcxxvjL3xB5Q== X-Received: by 2002:a05:6214:260e:b0:626:2305:6073 with SMTP id gu14-20020a056214260e00b0062623056073mr6706604qvb.4.1687989202071; Wed, 28 Jun 2023 14:53:22 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:21 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 7/8] selftests/mm: Add -a to run_vmtests.sh Date: Wed, 28 Jun 2023 17:53:09 -0400 Message-ID: <20230628215310.73782-8-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Allows to specify optional tests in run_vmtests.sh, where we can run time consuming test matrix only when user specified "-a". Acked-by: David Hildenbrand Signed-off-by: Peter Xu --- tools/testing/selftests/mm/run_vmtests.sh | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/self= tests/mm/run_vmtests.sh index 3f26f6e15b2a..824e651f62f4 100644 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -12,11 +12,14 @@ exitcode=3D0 =20 usage() { cat <"] +usage: ${BASH_SOURCE[0]:-$0} [ options ] + + -a: run all tests, including extra ones -t: specify specific categories to tests to run -h: display this message =20 -The default behavior is to run all tests. +The default behavior is to run required tests only. If -a is specified, +will run all tests. =20 Alternatively, specific groups tests can be run by passing a string to the -t argument containing one or more of the following categories @@ -60,9 +63,11 @@ EOF exit 0 } =20 +RUN_ALL=3Dfalse =20 -while getopts "ht:" OPT; do +while getopts "aht:" OPT; do case ${OPT} in + "a") RUN_ALL=3Dtrue ;; "h") usage ;; "t") VM_SELFTEST_ITEMS=3D${OPTARG} ;; esac --=20 2.41.0 From nobody Sat Feb 7 21:15:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28E8AEB64D7 for ; Wed, 28 Jun 2023 21:55:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231817AbjF1VzY (ORCPT ); Wed, 28 Jun 2023 17:55:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37840 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232356AbjF1Vyu (ORCPT ); Wed, 28 Jun 2023 17:54:50 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6BE772132 for ; Wed, 28 Jun 2023 14:53:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989205; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8+0w023mz5OfdmLI6Zdp3hWUHid6rdaL/9V50JZhDTo=; b=WAVyg2vOBMSZEI73gnmbY6XMoO98SXLZrWWEZIfILHHaF13dsnb2VkEfu2emXaQTBjyjLy e6ZnDWlGk+bECAub3yQAAQLtiqaYA2MZNs+klTvQlbMywmFNDNLzqgcZW/HaK8lQrt2LFk HCRRIRs0PcCtgDKFzPEqb5PqTh9BK7I= Received: from mail-vs1-f71.google.com (mail-vs1-f71.google.com [209.85.217.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-280-AIS6DH5mN5Gj7gPGGoNDYA-1; Wed, 28 Jun 2023 17:53:24 -0400 X-MC-Unique: AIS6DH5mN5Gj7gPGGoNDYA-1 Received: by mail-vs1-f71.google.com with SMTP id ada2fe7eead31-440bb217ba2so3077137.0 for ; Wed, 28 Jun 2023 14:53:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989203; x=1690581203; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8+0w023mz5OfdmLI6Zdp3hWUHid6rdaL/9V50JZhDTo=; b=WChpdnOkd86hXPmfOcGe22/C95IcPqfROSj/z5ENpEl6Kcc/tqfO6N7gbXfoLFH4wu 7pkpjrMbZx2mqn4oqltvbklOxE5dvZI5y32Cm2iprvrbHQt3JDyXQJCQL/xHM9N0hCAv FF1WIT5IwFiIdsasrBz9HHKRIyWUXvfAkYkR1gBhp0YnvAawXY997PTuBSkdTq6K1yxV c9nmL22WY9QT6if75Cxfsss8Hng3T6O7jrdaU/kxS8TVbTy4TCNQMVT8dgFOV2xQaRzd dcDseFer8PuQmLXkSwP4s6Pe4UKIjpbJWQMerSovw4AAgug6hVD5utq6wIjiIqO+zwjC X93Q== X-Gm-Message-State: AC+VfDxZcwiCvY1hGwC6Mv1pXJSG2+a+FoL9Ovbs7iPnuqhLeTnEhxzr sdjxDPdLt6L43hIzgKfdcUdMZGZ0S4ycHh1PGNeZXKJw9ETNTrsiO2+Yjq89ZL62PrY+6h62C4C S0STajzGgTLoq0WKHZ0yMUO/F X-Received: by 2002:a05:6102:3e16:b0:440:9794:fefa with SMTP id j22-20020a0561023e1600b004409794fefamr960328vsv.0.1687989203594; Wed, 28 Jun 2023 14:53:23 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ593pf/LYdqYwensAF9OSWrZV3jL3XksL0J6jNBIon+rk3QrdtClhhmqvv34L8EI9C3xofIng== X-Received: by 2002:a05:6102:3e16:b0:440:9794:fefa with SMTP id j22-20020a0561023e1600b004409794fefamr960312vsv.0.1687989203304; Wed, 28 Jun 2023 14:53:23 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:23 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 8/8] selftests/mm: Add gup test matrix in run_vmtests.sh Date: Wed, 28 Jun 2023 17:53:10 -0400 Message-ID: <20230628215310.73782-9-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Add a matrix for testing gup based on the current gup_test. Only run the matrix when -a is specified because it's a bit slow. It covers: - Different types of huge pages: thp, hugetlb, or no huge page - Permissions: Write / Read-only - Fast-gup, with/without - Types of the GUP: pin / gup / longterm pins - Shared / Private memories - GUP size: 1 / 512 / random page sizes Acked-by: David Hildenbrand Signed-off-by: Peter Xu --- tools/testing/selftests/mm/run_vmtests.sh | 37 ++++++++++++++++++++--- 1 file changed, 32 insertions(+), 5 deletions(-) diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/self= tests/mm/run_vmtests.sh index 824e651f62f4..9666c0c171ab 100644 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -90,6 +90,30 @@ test_selected() { fi } =20 +run_gup_matrix() { + # -t: thp=3Don, -T: thp=3Doff, -H: hugetlb=3Don + local hugetlb_mb=3D$(( needmem_KB / 1024 )) + + for huge in -t -T "-H -m $hugetlb_mb"; do + # -u: gup-fast, -U: gup-basic, -a: pin-fast, -b: pin-basic, -L: pi= n-longterm + for test_cmd in -u -U -a -b -L; do + # -w: write=3D1, -W: write=3D0 + for write in -w -W; do + # -S: shared + for share in -S " "; do + # -n: How many pages to fetch together? 512 is special + # because it's default thp size (or 2M on x86), 123 to + # just test partial gup when hit a huge in whatever fo= rm + for num in "-n 1" "-n 512" "-n 123"; do + CATEGORY=3D"gup_test" run_test ./gup_test \ + $huge $test_cmd $write $share $num + done + done + done + done + done +} + # get huge pagesize and freepages from /proc/meminfo while read -r name size unit; do if [ "$name" =3D "HugePages_Free:" ]; then @@ -194,13 +218,16 @@ fi =20 CATEGORY=3D"mmap" run_test ./map_fixed_noreplace =20 -# get_user_pages_fast() benchmark -CATEGORY=3D"gup_test" run_test ./gup_test -u -# pin_user_pages_fast() benchmark -CATEGORY=3D"gup_test" run_test ./gup_test -a +if $RUN_ALL; then + run_gup_matrix +else + # get_user_pages_fast() benchmark + CATEGORY=3D"gup_test" run_test ./gup_test -u + # pin_user_pages_fast() benchmark + CATEGORY=3D"gup_test" run_test ./gup_test -a +fi # Dump pages 0, 19, and 4096, using pin_user_pages: CATEGORY=3D"gup_test" run_test ./gup_test -ct -F 0x1 0 19 0x1000 - CATEGORY=3D"gup_test" run_test ./gup_longterm =20 CATEGORY=3D"userfaultfd" run_test ./uffd-unit-tests --=20 2.41.0