From nobody Tue Nov 26 15:24:30 2024 Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A2BF91DACA7 for ; Thu, 17 Oct 2024 09:47:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729158478; cv=none; b=DHMnp3moDiL/A29KCJcGIt1saKGvIi2j3jFOD/XSEAJkRDEWbi1ZwKdewffiYBJv33LPF0RJGV5TxmrRVxGfXzP+nBUfqq9kAC4/aWMppokSzdJD5rqFI045IbOF/d3lUH4FkQZVJGJ4cmGugrASPSLRyemsSjGKYT/CqHrJGok= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729158478; c=relaxed/simple; bh=xy3J47nYQIkyOxm6U50THk/J9Zo0uYz5kNGc+0/M+B8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=O2GW3rH+0nNFmWq/NWBWg3ObqPeKiXSiUaXpJ1WTnHOZpyg90do0phaBvpMwlIJ5Bb4ndFr/tyV/qkPkDggXt+Wc+738JCrjOPvONMgtA3gQn4Sl47msD//Wbx9M5XHRgm1kkEkvmMt8/sibYiAfHRbA9R/o8IVetuYo/6Pctdg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=LymoGsNQ; arc=none smtp.client-ip=209.85.215.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="LymoGsNQ" Received: by mail-pg1-f169.google.com with SMTP id 41be03b00d2f7-7ea16c7759cso499207a12.1 for ; Thu, 17 Oct 2024 02:47:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1729158475; x=1729763275; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+7VdqwStWQj5e7m9yAII5NEsGFIvs7nS0Am+cBAHNsQ=; b=LymoGsNQp2QXmmbwAL3cOXmeaGmYSJu3KKfXEaZchjE5RBd6YHFkonLroX5Tv9chJm BgboUAM4tUcKqKETnF66g1INCPp7poEM8DAD5g85r77c5if+9hKFxoBNGrNOvEkOE4T2 t5lI2DR2co8oWEeAugGBXVX0wiuJMWI2paWZsF0+4m8MjdDxb8pM922dch/KEewsIQz/ GMao+Bfb29Cu+gIfeF1s6hm3CQrgEtkZopqMEMwW7F3G6vR/bQIoMgS+D0xWgUqUpEpH 4dXMtLcYuBc5YRzdbW7/4hOcmPYkja5X3emFT8PT2PSp47yuMPJ+GxS5ahrqQ2WoNtTh X/WQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729158475; x=1729763275; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+7VdqwStWQj5e7m9yAII5NEsGFIvs7nS0Am+cBAHNsQ=; b=UgLnaS7q0rPeWzMSOv2hcEXUlWzlBkPdG5w5mDUEgLtEMJ7IhlOsvnvpCglgLkBzzO BL2/gTl0KIV/S7DGofEMNoVPq8+cYnFLzXxsrGJGntr4z3wDlIKN21+paQmkc0xaz7TK 48XAkfUmBFnzUAxmWQNVSFsw6t3mNPi1N2roIwe9z/TJS7T8RUz9J+jM0LwcvEfeNcqx oMfn5RCTdass4IZLa/jV7ebcp/qyajmMMOXkcGhEMpi+W5nc4O9qXhl7o2Ro7WA45SoL fZ/4nB8WEtXm33de/LAXIOXxHVYSjenlK6oce+OgHTU5ONCRpaZZBXiOYflp+Z7LXaly 67Pg== X-Forwarded-Encrypted: i=1; AJvYcCVx2C/eu/wNqV0vaIfzq0MghArBVWK3zvP1ZexZ+5Tac43CFqawZu2x7ZQ03Ps4ZfTqBP+s8erFRoRJHEY=@vger.kernel.org X-Gm-Message-State: AOJu0YwxKZpR1YdDDRUj1Qfy56/Hgbfk0x/Ytg/R/zRTMdenQmCUTJUT cexUh8I3jCdom4ROaigeza53lpwdN50HzRmq3beXzNBk4TwHCj2+DOiU3lXS0to= X-Google-Smtp-Source: AGHT+IHzuiiCIun9pa4mZiHfpa9E3frsNbxHgaXVoXJzu+vwbmm9rxbIGaUhA7zByM5nDjwu8FuQmQ== X-Received: by 2002:a05:6a21:e8a:b0:1d9:181f:e6d8 with SMTP id adf61e73a8af0-1d9181fe778mr4119688637.31.1729158474948; Thu, 17 Oct 2024 02:47:54 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71e774a4218sm4385365b3a.120.2024.10.17.02.47.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Oct 2024 02:47:54 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, mgorman@suse.de, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, zokeefe@google.com, rientjes@google.com, jannh@google.com, peterx@redhat.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, x86@kernel.org, Qi Zheng Subject: [PATCH v1 1/7] mm: khugepaged: retract_page_tables() use pte_offset_map_lock() Date: Thu, 17 Oct 2024 17:47:20 +0800 Message-Id: <258de4356bdcc01bce0ff1f6c29b2b64a4211494.1729157502.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In retract_page_tables(), we may modify the pmd entry after acquiring the pml and ptl, so we should also check whether the pmd entry is stable. Using pte_offset_map_lock() to do it, and then we can also remove the calling of the pte_lockptr(). Signed-off-by: Qi Zheng --- mm/khugepaged.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 94feb85ce996c..b4f49d323c8d9 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1721,6 +1721,7 @@ static void retract_page_tables(struct address_space = *mapping, pgoff_t pgoff) spinlock_t *pml; spinlock_t *ptl; bool skipped_uffd =3D false; + pte_t *pte; =20 /* * Check vma->anon_vma to exclude MAP_PRIVATE mappings that @@ -1757,9 +1758,15 @@ static void retract_page_tables(struct address_space= *mapping, pgoff_t pgoff) mmu_notifier_invalidate_range_start(&range); =20 pml =3D pmd_lock(mm, pmd); - ptl =3D pte_lockptr(mm, pmd); + pte =3D pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!pte) { + spin_unlock(pml); + mmu_notifier_invalidate_range_end(&range); + continue; + } if (ptl !=3D pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + pte_unmap(pte); =20 /* * Huge page lock is still held, so normally the page table --=20 2.20.1 From nobody Tue Nov 26 15:24:30 2024 Received: from mail-pg1-f175.google.com (mail-pg1-f175.google.com [209.85.215.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 70E671DB953 for ; Thu, 17 Oct 2024 09:48:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729158484; cv=none; b=j0QScizei6Fhrxl1vhRlnk34m0it21jUV4uZK6YLmS1Ou39sb7dalWNA3sN1omGv/fu1uNKWiEhBaCCznOM5H+r2Zpfgm4VlDmoANUGPeFMIl4TSi+EaULYlSgorT6hFMqK7mJhCO2PhykThJkxn8UQqRIkIAuApivqwNdT2VDQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729158484; c=relaxed/simple; bh=PuVSu6LbH9IyVae19F5YhUQxNt5tQRY/J+tLwJFocQ4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=S2ZS23OY1EGMJT4NiR/r81MfkWkKujFOGbu+vBB8EYBGICATj7DhHcq/MrRC9UoMbCSFFwzD1zEmLvGwhuAZGSSQdFcHyeKU/+1uzXCpYwtaU3i2ibRqmiGJImUtXWMM7vo1l5GtKoM2KI4j852QJ5UnZsNHUfdnAX0Q/BGo6HI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=hbUyKZKs; arc=none smtp.client-ip=209.85.215.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="hbUyKZKs" Received: by mail-pg1-f175.google.com with SMTP id 41be03b00d2f7-7db54269325so628091a12.2 for ; Thu, 17 Oct 2024 02:48:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1729158481; x=1729763281; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KKkeoST6I/bkefiDvyHuFZOvKUoGTogE4tqpFCMcSq8=; b=hbUyKZKs/kcqIUDmsmEHh7ZfkFWBmewxmJVbvMdpIbIMPyHm4IaVeZBSH+XD4DHFRI pLUbOV+utVwr8fqoK/IqmUXFnDb3d/VlFwS+cldikXu+6uiQydPSUFwfSO6cMpA3ITlt CDayg/pmV2UnXBO03UMIVV+S3sSpVzE9GfQxWV3tSe9942+AwBgHTuBadwr97Q78DYpB df/lplUYYfscQxUh4GtehdPEMJbbQhzSXkDxb7QZxlpVAYslP5D7Id4Qle7vKD4rFWtr OmiDJD60oyM6O6ZvtuEc9trDdGKsD0GF0offthl2O4aWYMO90/kvORxFpBSrkVw1UZMP 5xhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729158481; x=1729763281; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KKkeoST6I/bkefiDvyHuFZOvKUoGTogE4tqpFCMcSq8=; b=kTT292+ZFc6qDIKxsuC4Fv5midk8728heF1/pNNqGCHVdwaYX2/FWl1uJDCgO327xy kzhVS6p9urF1uEk6R2TTILJURB37xoEtqptkvKfk8DR0P+u2yQhEaa85vk8q+unrI9T1 Qu+O0d3M/fN+TEh5wTi46vxP4FY9LhmbPVGTWGLhrSUGuFe70EXZkwTuWNGx3MCiNM8Z bWZAxubsPbSlKjDf86ZAIEgLaSELEHTJCQGM1t2AH4KsI+7+jPIvfk+aWTiCv9J6jrZe bMrycvD6d1LZxIMSbEhoA9w9Un2IostDKaqVViQHapHDWkp3hm0wGmop1HLrOC670E0F jSow== X-Forwarded-Encrypted: i=1; AJvYcCU2b95i7dBrRdW1YzshsGixoa0CN9oiUu0yB5PhcrtivbbMgw7D6EhhjNNylcAH75ys7BdtqkkMtZkkKeA=@vger.kernel.org X-Gm-Message-State: AOJu0YwHTUHdJWQeyHrfAuIjK08OZF1c6t9c6jelIAuYrTQpqGHTeRIp MS5GGwgxOnjCOZRCkDbEoHoQTbB0lskp7INXL2UGBcbiE5Z362Y4YA5nkUfLIKM= X-Google-Smtp-Source: AGHT+IEI5E0ziycErtAHmWC08AA6d27vlvkOtexRBD8N1aJkqSz6qnPieAM9Lamrn4x3BOyjY5wnjQ== X-Received: by 2002:a05:6a21:3998:b0:1d9:21c7:5af7 with SMTP id adf61e73a8af0-1d921c75b3dmr1710778637.15.1729158480809; Thu, 17 Oct 2024 02:48:00 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71e774a4218sm4385365b3a.120.2024.10.17.02.47.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Oct 2024 02:48:00 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, mgorman@suse.de, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, zokeefe@google.com, rientjes@google.com, jannh@google.com, peterx@redhat.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, x86@kernel.org, Qi Zheng Subject: [PATCH v1 2/7] mm: make zap_pte_range() handle full within-PMD range Date: Thu, 17 Oct 2024 17:47:21 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In preparation for reclaiming empty PTE pages, this commit first makes zap_pte_range() to handle the full within-PMD range, so that we can more easily detect and free PTE pages in this function in subsequent commits. Signed-off-by: Qi Zheng --- mm/memory.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index caa6ed0a7fe5b..fd57c0f49fce2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1602,6 +1602,7 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, swp_entry_t entry; int nr; =20 +retry: tlb_change_page_size(tlb, PAGE_SIZE); init_rss_vec(rss); start_pte =3D pte =3D pte_offset_map_lock(mm, pmd, addr, &ptl); @@ -1706,6 +1707,11 @@ static unsigned long zap_pte_range(struct mmu_gather= *tlb, if (force_flush) tlb_flush_mmu(tlb); =20 + if (addr !=3D end) { + cond_resched(); + goto retry; + } + return addr; } =20 @@ -1744,8 +1750,6 @@ static inline unsigned long zap_pmd_range(struct mmu_= gather *tlb, continue; } addr =3D zap_pte_range(tlb, vma, pmd, addr, next, details); - if (addr !=3D next) - pmd--; } while (pmd++, cond_resched(), addr !=3D end); =20 return addr; --=20 2.20.1 From nobody Tue Nov 26 15:24:30 2024 Received: from mail-oa1-f47.google.com (mail-oa1-f47.google.com [209.85.160.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9932D1DB953 for ; Thu, 17 Oct 2024 09:48:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729158490; cv=none; b=gD+svAtAh/xl0dVe4kvR9a7Ce0+MjYQkHJolqm9JvdQbiqi7kpTahCnWeb1AyS8+eGeESCmMF+W48Z+dgfezSRukAdRZtI0tp3q5cNXJQh2ZSvGLAnXzy8FduWeTk2DPmWCMczFGfqMFH6qC5MJyPb1T5hThitBIHHD5Y/BWV1c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729158490; c=relaxed/simple; bh=XMqo9JC0uvRaenR0CNJgp9Ct19xGAvPL0Vj/MIAQrlc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rv/fG3BN1G/ujcAVNR4dKA7uEZWjX4DBkOU27smAwixqQnbXsRZSHkWYbhVJMiQnPaIPzgFWgXEtrweKSL9YyRc1HenkdjdM4Bk8blM7HVt7En+NG0/XWpR8yPMNtXdgEuCjZ4dS4lVYh0KPE8nL5z3FjN0l7sDoCzfO9r1IbvM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=OVjItaAx; arc=none smtp.client-ip=209.85.160.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="OVjItaAx" Received: by mail-oa1-f47.google.com with SMTP id 586e51a60fabf-28847f4207dso339781fac.0 for ; Thu, 17 Oct 2024 02:48:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1729158487; x=1729763287; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ns8LrZyEwJOJHznjnD7PlHQ9MHEMoW624Fd3QWpp04I=; b=OVjItaAxwftIJm0xQkZV3VPxaUEZliINdH8RZchGKyGTaGRE/eJ4ZgFiPBAPKXRpdW 6DSARKSykJNRZQ3MlpQ/5edBYzHWYxbQAO53bjvXBcCRvtNSYiPTViAF0p5fZx8mpKDj V4b9WHMzhoju5dNb1s0ApTVYJC1evTeKcoT7iH4Sa4WWZvDJVXFX1pgQj1Fe9CovVfEX vIwzCHQqpqNf9UjwpQ38XMJb+T0ILU1r7VJw4aCNMtsGGW1AXyZW8RVNHP+fLN5ceDoj 3u3gwoDFsPdXMeTqPLIsHLjOLL3LgZSvNkx5mKZyIowC71enG1ngV1SY9nb4efbNdsJ2 o/eQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729158487; x=1729763287; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ns8LrZyEwJOJHznjnD7PlHQ9MHEMoW624Fd3QWpp04I=; b=d7RG13VT30+xX6/m1COlllImPX+vLqKKt1TGeavXMPpxx/+P2EKm5BSKMlH9bHhz4S SQVfqcy8YiZFqU3fiIm5aFShXZ6OPoHOIcWKjPoPMQJjEZlKTJwOF8XJhOWWRC2Rj52W VACvjIz01ZxNTIrh5TeplBnXL4aaz4YaiDc0sP8v3J17iWiNTy/SeuogYVU7Gar9ElMk AOp1eWgLHaNXGwNIOmQnqpA+0CzW0OKke1UJMz2cpS4Afl8xi+TGapmkbHlLqh4lbULS ob+wcO1Z0+cfHZOund+0LJ8Ec3mISbcUJdHSNRNCFxl0FLKU2P7OHAms9inD8AANcXSg 4wYA== X-Forwarded-Encrypted: i=1; AJvYcCWw01b5jPlOdvL3z+I6GIiyz+U4z2EC5wXPGRQjpOO+pWeZufsHYtFeyc8CRjmPwXBn4hDJZlOxQwaCOsc=@vger.kernel.org X-Gm-Message-State: AOJu0YwtF7uSnc5Prm8ciHqiVxW0koUqkGczCaLmcmhHUFzNvcYatC0B IfF2Ij+LH2vo+c4Dg5EIfYfXAqWa0Seu14/fx1FTLpgSQOSKcAXKi0M7v+O78A0= X-Google-Smtp-Source: AGHT+IHvYEBhNwHNitO8wA559PhHl4cuzZKZHlF0Sd6OJ2vGi8dRLztGVbwRe2pbe9tQOZtPvKzwMw== X-Received: by 2002:a05:6870:798:b0:287:ce7:7c4a with SMTP id 586e51a60fabf-2886e00f920mr16405829fac.38.1729158486665; Thu, 17 Oct 2024 02:48:06 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71e774a4218sm4385365b3a.120.2024.10.17.02.48.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Oct 2024 02:48:06 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, mgorman@suse.de, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, zokeefe@google.com, rientjes@google.com, jannh@google.com, peterx@redhat.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, x86@kernel.org, Qi Zheng Subject: [PATCH v1 3/7] mm: zap_install_uffd_wp_if_needed: return whether uffd-wp pte has been re-installed Date: Thu, 17 Oct 2024 17:47:22 +0800 Message-Id: <37983544cb4fbd27c244b68d3e37d170ea51f444.1729157502.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In some cases, we'll replace the none pte with an uffd-wp swap special pte marker when necessary. Let's expose this information to the caller through the return value, so that subsequent commits can use this information to detect whether the PTE page is empty. Signed-off-by: Qi Zheng --- include/linux/mm_inline.h | 11 ++++++++--- mm/memory.c | 15 +++++++++++---- 2 files changed, 19 insertions(+), 7 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 6f801c7b36e2f..c3ba1da418caf 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -552,8 +552,10 @@ static inline pte_marker copy_pte_marker( * pte, and if they see it, they'll fault and serialize at the pgtable loc= k. * * This function is a no-op if PTE_MARKER_UFFD_WP is not enabled. + * + * Returns true if an uffd-wp pte was installed, false otherwise. */ -static inline void +static inline bool pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long ad= dr, pte_t *pte, pte_t pteval) { @@ -570,7 +572,7 @@ pte_install_uffd_wp_if_needed(struct vm_area_struct *vm= a, unsigned long addr, * with a swap pte. There's no way of leaking the bit. */ if (vma_is_anonymous(vma) || !userfaultfd_wp(vma)) - return; + return false; =20 /* A uffd-wp wr-protected normal pte */ if (unlikely(pte_present(pteval) && pte_uffd_wp(pteval))) @@ -583,10 +585,13 @@ pte_install_uffd_wp_if_needed(struct vm_area_struct *= vma, unsigned long addr, if (unlikely(pte_swp_uffd_wp_any(pteval))) arm_uffd_pte =3D true; =20 - if (unlikely(arm_uffd_pte)) + if (unlikely(arm_uffd_pte)) { set_pte_at(vma->vm_mm, addr, pte, make_pte_marker(PTE_MARKER_UFFD_WP)); + return true; + } #endif + return false; } =20 static inline bool vma_has_recency(struct vm_area_struct *vma) diff --git a/mm/memory.c b/mm/memory.c index fd57c0f49fce2..534d9d52b5ebe 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1467,27 +1467,34 @@ static inline bool zap_drop_file_uffd_wp(struct zap= _details *details) /* * This function makes sure that we'll replace the none pte with an uffd-wp * swap special pte marker when necessary. Must be with the pgtable lock h= eld. + * + * Returns true if uffd-wp ptes was installed, false otherwise. */ -static inline void +static inline bool zap_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr, pte_t *pte, int nr, struct zap_details *details, pte_t pteval) { + bool was_installed =3D false; + /* Zap on anonymous always means dropping everything */ if (vma_is_anonymous(vma)) - return; + return false; =20 if (zap_drop_file_uffd_wp(details)) - return; + return false; =20 for (;;) { /* the PFN in the PTE is irrelevant. */ - pte_install_uffd_wp_if_needed(vma, addr, pte, pteval); + if (pte_install_uffd_wp_if_needed(vma, addr, pte, pteval)) + was_installed =3D true; if (--nr =3D=3D 0) break; pte++; addr +=3D PAGE_SIZE; } + + return was_installed; } =20 static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb, --=20 2.20.1 From nobody Tue Nov 26 15:24:30 2024 Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 429161DC194 for ; Thu, 17 Oct 2024 09:48:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729158496; cv=none; b=fvmYs0nMSSFJImhFpfxAh7cyUIBH/SYHhZ6Fi41D9TNNC08pG5fz5BCsbxuaEFhRgCPn8zDxZCyHO06wZkcsD4HgLdgwi2KtvHhC5XUP/fKfOHY/COjWSkoX6YMSxXW+mpNbRUc2l6igbO8hZOoJqHgaJD5r+4IIhKpObVJeeyc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729158496; c=relaxed/simple; bh=H6+wTWP26uqZ4YBcgcKoTuVx8AqbeFe0Q57a4F7QhGU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=e5Cnt7YLUJww7eI7VCd0/bgSYB9h1RFAN4ippOrV7xmnAQPEhByhdyP8gloQLa/hMzjBEH5dkxbf2JSZgMehIdWBTL+fhcb8cYbI0tpHLGs+cn8FzjBjMc2OMJ0gT5pGupcLCuOJ3wpWYyNZiEwZF0i9m4Y1QBPo/gt4FLIdb9M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=er0o5HLf; arc=none smtp.client-ip=209.85.210.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="er0o5HLf" Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-71e5130832aso499846b3a.0 for ; Thu, 17 Oct 2024 02:48:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1729158492; x=1729763292; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Dj3ri9p71nmuCZf/HvkLHVU1fpbThKpmsfMSEQvNK9I=; b=er0o5HLf/f9slaOjlpuD4/dBrwP2oWdKRWK/0qMpXig0yQGFj9U8SgWW1aynnlqovf 8s1AOGVpF2wFfcQD1C+B2mMkT7I4h+xzHBWYSZlpZnIvDLpvyOoCQHLDZ+5H/RIrRjmy YdIjqhQSXOfkp9YFvRKIsJaWGIYMakeVQbmgXv1bCj9nZBi8FXCRGf5o7UzqrOXMLjHU p6yDTr3zHJ2fL20IxtYMVcLU4a/zSy3papYz5TNge5PP4vU1UHNslydr5DJDPZFYTOHw ziAvRSPNwByfvIzgj9G9SGp5qsdJ11eUDkWnO/T7Fnx8JD9pLd8QLcrXLFRn96sw3xJv Opfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729158492; x=1729763292; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Dj3ri9p71nmuCZf/HvkLHVU1fpbThKpmsfMSEQvNK9I=; b=jEb7JIyznwg+eWuNGeHb1HQOZfPR6F9stxGjBjDLF7sD9zyV/J+pSsTZbQGc3FDpxA 11Ck5Cr+9ODlFLyungl7PxyKM/8XDojCgNBxO0CK9d3uhe4ipe7BdKG1jFF9Q3SaN6Ci rMI9PwxvkZWsuNRp6W6ruLCF62ilgOHsCJIcL/gvEFlvhXtqFzB0ixO7TeUWsghXUU8X zwLEquu1iPbSdqskJHwZCftdtOhvCIw6pha/dXL9o7cvFuy5fFkiQ1alq9Uk4Y4uIMpA IXyPcQDEk7S5WUUIOpD49OLdigVhjpBvicyRdlCz+yb3e7KNGBEq9TE2psDfSFA1ZS3O 3K5A== X-Forwarded-Encrypted: i=1; AJvYcCVqwxfGFZXYH0+lBSxS87ksE/sF73RIQt+3LX1dqw2Xl1Qy/w+bOYQbzdBLXH3KKv4OGfkLOWjfOD/A8HI=@vger.kernel.org X-Gm-Message-State: AOJu0Yw+sBg596fkMsMAIAYBtCwBNvf2JybZf1+kFSSvvj9YN5LZ2dVF wHIqfrns+BTxHfwGNIL21hLb6OrPCMeOrf3Bxp0w5eqUZ2BEo38aROGx5rcRnXo= X-Google-Smtp-Source: AGHT+IEKEDRwguwxshyZXMicxDUmcRCwiWXS2djsKDNBwB/wtcHqk2whV9zra/Oag0PCIfiHyKtsxQ== X-Received: by 2002:a05:6a20:9f4a:b0:1d5:1729:35ec with SMTP id adf61e73a8af0-1d8c955c8ebmr29391532637.7.1729158492598; Thu, 17 Oct 2024 02:48:12 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71e774a4218sm4385365b3a.120.2024.10.17.02.48.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Oct 2024 02:48:12 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, mgorman@suse.de, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, zokeefe@google.com, rientjes@google.com, jannh@google.com, peterx@redhat.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, x86@kernel.org, Qi Zheng Subject: [PATCH v1 4/7] mm: zap_present_ptes: return whether the PTE page is unreclaimable Date: Thu, 17 Oct 2024 17:47:23 +0800 Message-Id: <84a9fddde9993e4a5108f188193fd9c8ff1c5c31.1729157502.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In the following two cases, the PTE page cannot be empty and cannot be reclaimed: 1. an uffd-wp pte was re-installed 2. should_zap_folio() return false Let's expose this information to the caller through is_pt_unreclaimable, so that subsequent commits can use this information in zap_pte_range() to detect whether the PTE page can be reclaimed. Signed-off-by: Qi Zheng --- mm/memory.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 534d9d52b5ebe..cc89ede8ce2ab 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1501,7 +1501,7 @@ static __always_inline void zap_present_folio_ptes(st= ruct mmu_gather *tlb, struct vm_area_struct *vma, struct folio *folio, struct page *page, pte_t *pte, pte_t ptent, unsigned int nr, unsigned long addr, struct zap_details *details, int *rss, - bool *force_flush, bool *force_break) + bool *force_flush, bool *force_break, bool *is_pt_unreclaimable) { struct mm_struct *mm =3D tlb->mm; bool delay_rmap =3D false; @@ -1527,8 +1527,8 @@ static __always_inline void zap_present_folio_ptes(st= ruct mmu_gather *tlb, arch_check_zapped_pte(vma, ptent); tlb_remove_tlb_entries(tlb, pte, nr, addr); if (unlikely(userfaultfd_pte_wp(vma, ptent))) - zap_install_uffd_wp_if_needed(vma, addr, pte, nr, details, - ptent); + *is_pt_unreclaimable =3D + zap_install_uffd_wp_if_needed(vma, addr, pte, nr, details, ptent); =20 if (!delay_rmap) { folio_remove_rmap_ptes(folio, page, nr, vma); @@ -1552,7 +1552,7 @@ static inline int zap_present_ptes(struct mmu_gather = *tlb, struct vm_area_struct *vma, pte_t *pte, pte_t ptent, unsigned int max_nr, unsigned long addr, struct zap_details *details, int *rss, bool *force_flush, - bool *force_break) + bool *force_break, bool *is_pt_unreclaimable) { const fpb_t fpb_flags =3D FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY; struct mm_struct *mm =3D tlb->mm; @@ -1567,15 +1567,18 @@ static inline int zap_present_ptes(struct mmu_gathe= r *tlb, arch_check_zapped_pte(vma, ptent); tlb_remove_tlb_entry(tlb, pte, addr); if (userfaultfd_pte_wp(vma, ptent)) - zap_install_uffd_wp_if_needed(vma, addr, pte, 1, - details, ptent); + *is_pt_unreclaimable =3D + zap_install_uffd_wp_if_needed(vma, addr, pte, 1, + details, ptent); ksm_might_unmap_zero_page(mm, ptent); return 1; } =20 folio =3D page_folio(page); - if (unlikely(!should_zap_folio(details, folio))) + if (unlikely(!should_zap_folio(details, folio))) { + *is_pt_unreclaimable =3D true; return 1; + } =20 /* * Make sure that the common "small folio" case is as fast as possible @@ -1587,11 +1590,12 @@ static inline int zap_present_ptes(struct mmu_gathe= r *tlb, =20 zap_present_folio_ptes(tlb, vma, folio, page, pte, ptent, nr, addr, details, rss, force_flush, - force_break); + force_break, is_pt_unreclaimable); return nr; } zap_present_folio_ptes(tlb, vma, folio, page, pte, ptent, 1, addr, - details, rss, force_flush, force_break); + details, rss, force_flush, force_break, + is_pt_unreclaimable); return 1; } =20 @@ -1622,6 +1626,7 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, pte_t ptent =3D ptep_get(pte); struct folio *folio; struct page *page; + bool is_pt_unreclaimable =3D false; int max_nr; =20 nr =3D 1; @@ -1635,7 +1640,7 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, max_nr =3D (end - addr) / PAGE_SIZE; nr =3D zap_present_ptes(tlb, vma, pte, ptent, max_nr, addr, details, rss, &force_flush, - &force_break); + &force_break, &is_pt_unreclaimable); if (unlikely(force_break)) { addr +=3D nr * PAGE_SIZE; break; --=20 2.20.1 From nobody Tue Nov 26 15:24:30 2024 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 07D631DC194 for ; Thu, 17 Oct 2024 09:48:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729158502; cv=none; b=ceKTTemDXA/P+quVdX3sNxYvTmwxU1gXXyScW99Wt9p+wbyu8o7FbgwCLMP2Bkl80MaE6Wk+WOZ2x74d3RBidaezyMxBjQF3nL8QWsTH+wwJzypd4LzmzZBiilUIoJtLKm5XHz+9tNUzf3QxK3sr592eUaBshRkFjmqq3QN0LZc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729158502; c=relaxed/simple; bh=78BqYEISZJiwjRYZkHzdiLqKPfoOhPjYtJX2SXEkwWU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WCY6yOXgyrdmhFqHmybFBMjDL43uSCnhLYmVK/NbVMhD3N0UZ/WfCaFr1G8H5KjeBptEL9panGhsosD+AI5TeyBCta2cdV0y3wvCDbn/ER8fbuk7HxeOib2gTGy5HG5l050gw6RXzHN9LePX19c9OWEyPolkB/amqHwVI4r/bXo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=HRhKvX7a; arc=none smtp.client-ip=209.85.215.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="HRhKvX7a" Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-7ea68af2f62so620958a12.3 for ; Thu, 17 Oct 2024 02:48:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1729158499; x=1729763299; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6MI+cLF01LX1jQYVjU1/jISHOw23BC0/0O3KrEoCWt0=; b=HRhKvX7anOVDTebT7vF0jYME88Ekx73JrnKE2bXhXgMYP1rKUOybIhscIdHR22eoP6 77pp23XKzAeVQ3CbXAJOdMMe9rnCRA3RD9ygyxnAoXB6AIyfVxwGqn1uOK2BLqGKRLLt G2WungFHIREJAG8T1o2taiVNbTf8WNO76Z8hKUfcYdjOfbjmXJ+Sg7dXfOe8PbuyRTuJ tIumJHY686RaLRRMaPyAWwnAhA+lWfMgkL9YZWnCv2mkkTeuFH0A1TZGf7zaB0oNMmAQ lVDQmtEwS+aaKkH3oyy1qwBGE3Zb4iLoBVlcxfPXLHt1V3dawyZ6m62zqlajlCv65lho jtbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729158499; x=1729763299; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6MI+cLF01LX1jQYVjU1/jISHOw23BC0/0O3KrEoCWt0=; b=ErpLzAkRuy2deKyzaj6RKXxutKNRR2JWC+UlebV0lIES/x633IDsu1hEA7/VTKsU5r IpOpLFRzn4ebCDyeSczrq2g80Tg3rCiLa0QnyczfeNLQsKlut2cH1gemDgm7qSjbpq07 P7vY5nJ35nug+1cgFxkC5wplZ/AjI2igXXHR0Oj1ApniCNnIHudbxQqHpSnQX0OvsOLN FGc8WyyjiDTqm72ubyu05mYiu+BcVz5Qw6uTPVz27Arqjn941v+l/xxN949jGp4VZaxW dQVhTGt7rCWyDZqltiVj1CsIfPEhMIJYlH0x7cx221AuiHMFGSGpLJYw0TcLgozWwmhl 7aOw== X-Forwarded-Encrypted: i=1; AJvYcCXXRBKYL4HGPcI73EDzfMe2D7FxDx1eLMK9HXVZYvFoDDyyEf4muWMIkelTDQ1Dq/XkvgZUmm323hnyOfY=@vger.kernel.org X-Gm-Message-State: AOJu0Yw04/P02PhEFeZxXQ68pOSVLJiBgwSfD6tdE0HedyyctXNODEPO hJD/Z+FNR5Svmr42R7BKua6KEINMEd5sekklqaDTDwEI9ZlPqoqbKCjOX9o3Go0= X-Google-Smtp-Source: AGHT+IG3h20lu4G0QNcRMk+sLLhl4rVPjvK7yVrB3CQ6qr+/Kwo1iZ1STapCyXTOLaWUcuJN2AVvmg== X-Received: by 2002:a05:6a21:3998:b0:1d9:21c7:5af7 with SMTP id adf61e73a8af0-1d921c75b3dmr1711810637.15.1729158499347; Thu, 17 Oct 2024 02:48:19 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71e774a4218sm4385365b3a.120.2024.10.17.02.48.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Oct 2024 02:48:18 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, mgorman@suse.de, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, zokeefe@google.com, rientjes@google.com, jannh@google.com, peterx@redhat.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, x86@kernel.org, Qi Zheng Subject: [PATCH v1 5/7] mm: pgtable: try to reclaim empty PTE page in madvise(MADV_DONTNEED) Date: Thu, 17 Oct 2024 17:47:24 +0800 Message-Id: <6c7fe15b0434a08a287c400869f9ba434e1a8fa3.1729157502.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now in order to pursue high performance, applications mostly use some high-performance user-mode memory allocators, such as jemalloc or tcmalloc. These memory allocators use madvise(MADV_DONTNEED or MADV_FREE) to release physical memory, but neither MADV_DONTNEED nor MADV_FREE will release page table memory, which may cause huge page table memory usage. The following are a memory usage snapshot of one process which actually happened on our server: VIRT: 55t RES: 590g VmPTE: 110g In this case, most of the page table entries are empty. For such a PTE page where all entries are empty, we can actually free it back to the system for others to use. As a first step, this commit aims to synchronously free the empty PTE pages in madvise(MADV_DONTNEED) case. We will detect and free empty PTE pages in zap_pte_range(), and will add zap_details.reclaim_pt to exclude cases other than madvise(MADV_DONTNEED). Once an empty PTE is detected, we first try to hold the pmd lock within the pte lock. If successful, we clear the pmd entry directly (fast path). Otherwise, we wait until the pte lock is released, then re-hold the pmd and pte locks and loop PTRS_PER_PTE times to check pte_none() to re-detect whether the PTE page is empty and free it (slow path). For other cases such as madvise(MADV_FREE), consider scanning and freeing empty PTE pages asynchronously in the future. The following code snippet can show the effect of optimization: mmap 50G while (1) { for (; i < 1024 * 25; i++) { touch 2M memory madvise MADV_DONTNEED 2M } } As we can see, the memory usage of VmPTE is reduced: before after VIRT 50.0 GB 50.0 GB RES 3.1 MB 3.1 MB VmPTE 102640 KB 240 KB Signed-off-by: Qi Zheng --- include/linux/mm.h | 1 + mm/Kconfig | 14 ++++++++++ mm/Makefile | 1 + mm/internal.h | 29 ++++++++++++++++++++ mm/madvise.c | 4 ++- mm/memory.c | 47 +++++++++++++++++++++++++++----- mm/pt_reclaim.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 156 insertions(+), 8 deletions(-) create mode 100644 mm/pt_reclaim.c diff --git a/include/linux/mm.h b/include/linux/mm.h index df0a5eac66b78..667a466bb4649 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2321,6 +2321,7 @@ extern void pagefault_out_of_memory(void); struct zap_details { struct folio *single_folio; /* Locked folio to be unmapped */ bool even_cows; /* Zap COWed private pages too? */ + bool reclaim_pt; zap_flags_t zap_flags; /* Extra flags for zapping */ }; =20 diff --git a/mm/Kconfig b/mm/Kconfig index 4b2a1ef9a161c..f5993b9cc2a9f 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1302,6 +1302,20 @@ config ARCH_HAS_USER_SHADOW_STACK The architecture has hardware support for userspace shadow call stacks (eg, x86 CET, arm64 GCS or RISC-V Zicfiss). =20 +config ARCH_SUPPORTS_PT_RECLAIM + def_bool n + +config PT_RECLAIM + bool "reclaim empty user page table pages" + default y + depends on ARCH_SUPPORTS_PT_RECLAIM && MMU && SMP + select MMU_GATHER_RCU_TABLE_FREE + help + Try to reclaim empty user page table pages in paths other that munmap + and exit_mmap path. + + Note: now only empty user PTE page table pages will be reclaimed. + source "mm/damon/Kconfig" =20 endmenu diff --git a/mm/Makefile b/mm/Makefile index d5639b0361663..9d816323d247a 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -145,3 +145,4 @@ obj-$(CONFIG_GENERIC_IOREMAP) +=3D ioremap.o obj-$(CONFIG_SHRINKER_DEBUG) +=3D shrinker_debug.o obj-$(CONFIG_EXECMEM) +=3D execmem.o obj-$(CONFIG_TMPFS_QUOTA) +=3D shmem_quota.o +obj-$(CONFIG_PT_RECLAIM) +=3D pt_reclaim.o diff --git a/mm/internal.h b/mm/internal.h index 906da6280c2df..4adaaea0917c0 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1445,4 +1445,33 @@ static inline void accept_page(struct page *page) } #endif /* CONFIG_UNACCEPTED_MEMORY */ =20 +#ifdef CONFIG_PT_RECLAIM +static inline void set_pt_unreclaimable(bool *can_reclaim_pt) +{ + *can_reclaim_pt =3D false; +} +bool try_get_and_clear_pmd(struct mm_struct *mm, pmd_t *pmd, pmd_t *pmdval= ); +void free_pte(struct mm_struct *mm, unsigned long addr, struct mmu_gather = *tlb, + pmd_t pmdval); +void try_to_free_pte(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, + struct mmu_gather *tlb); +#else +static inline void set_pt_unreclaimable(bool *can_reclaim_pt) +{ +} +static inline bool try_get_and_clear_pmd(struct mm_struct *mm, pmd_t *pmd, + pmd_t *pmdval) +{ + return false; +} +static inline void free_pte(struct mm_struct *mm, unsigned long addr, + struct mmu_gather *tlb, pmd_t pmdval) +{ +} +static inline void try_to_free_pte(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, struct mmu_gather *tlb) +{ +} +#endif /* CONFIG_PT_RECLAIM */ + #endif /* __MM_INTERNAL_H */ diff --git a/mm/madvise.c b/mm/madvise.c index e871a72a6c329..82a6d15429da7 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -843,7 +843,9 @@ static int madvise_free_single_vma(struct vm_area_struc= t *vma, static long madvise_dontneed_single_vma(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - zap_page_range_single(vma, start, end - start, NULL); + struct zap_details details =3D {.reclaim_pt =3D true,}; + + zap_page_range_single(vma, start, end - start, &details); return 0; } =20 diff --git a/mm/memory.c b/mm/memory.c index cc89ede8ce2ab..77774b34f2cde 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1437,7 +1437,7 @@ copy_page_range(struct vm_area_struct *dst_vma, struc= t vm_area_struct *src_vma) static inline bool should_zap_cows(struct zap_details *details) { /* By default, zap all pages */ - if (!details) + if (!details || details->reclaim_pt) return true; =20 /* Or, we zap COWed pages only if the caller wants to */ @@ -1611,8 +1611,18 @@ static unsigned long zap_pte_range(struct mmu_gather= *tlb, pte_t *start_pte; pte_t *pte; swp_entry_t entry; + pmd_t pmdval; + bool can_reclaim_pt =3D false; + bool direct_reclaim; + unsigned long start =3D addr; int nr; =20 + if (details && details->reclaim_pt) + can_reclaim_pt =3D true; + + if ((ALIGN_DOWN(end, PMD_SIZE)) - (ALIGN(start, PMD_SIZE)) < PMD_SIZE) + can_reclaim_pt =3D false; + retry: tlb_change_page_size(tlb, PAGE_SIZE); init_rss_vec(rss); @@ -1641,6 +1651,8 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, nr =3D zap_present_ptes(tlb, vma, pte, ptent, max_nr, addr, details, rss, &force_flush, &force_break, &is_pt_unreclaimable); + if (is_pt_unreclaimable) + set_pt_unreclaimable(&can_reclaim_pt); if (unlikely(force_break)) { addr +=3D nr * PAGE_SIZE; break; @@ -1653,8 +1665,10 @@ static unsigned long zap_pte_range(struct mmu_gather= *tlb, is_device_exclusive_entry(entry)) { page =3D pfn_swap_entry_to_page(entry); folio =3D page_folio(page); - if (unlikely(!should_zap_folio(details, folio))) + if (unlikely(!should_zap_folio(details, folio))) { + set_pt_unreclaimable(&can_reclaim_pt); continue; + } /* * Both device private/exclusive mappings should only * work with anonymous page so far, so we don't need to @@ -1670,14 +1684,18 @@ static unsigned long zap_pte_range(struct mmu_gathe= r *tlb, max_nr =3D (end - addr) / PAGE_SIZE; nr =3D swap_pte_batch(pte, max_nr, ptent); /* Genuine swap entries, hence a private anon pages */ - if (!should_zap_cows(details)) + if (!should_zap_cows(details)) { + set_pt_unreclaimable(&can_reclaim_pt); continue; + } rss[MM_SWAPENTS] -=3D nr; free_swap_and_cache_nr(entry, nr); } else if (is_migration_entry(entry)) { folio =3D pfn_swap_entry_folio(entry); - if (!should_zap_folio(details, folio)) + if (!should_zap_folio(details, folio)) { + set_pt_unreclaimable(&can_reclaim_pt); continue; + } rss[mm_counter(folio)]--; } else if (pte_marker_entry_uffd_wp(entry)) { /* @@ -1685,21 +1703,29 @@ static unsigned long zap_pte_range(struct mmu_gathe= r *tlb, * drop the marker if explicitly requested. */ if (!vma_is_anonymous(vma) && - !zap_drop_file_uffd_wp(details)) + !zap_drop_file_uffd_wp(details)) { + set_pt_unreclaimable(&can_reclaim_pt); continue; + } } else if (is_hwpoison_entry(entry) || is_poisoned_swp_entry(entry)) { - if (!should_zap_cows(details)) + if (!should_zap_cows(details)) { + set_pt_unreclaimable(&can_reclaim_pt); continue; + } } else { /* We should have covered all the swap entry types */ pr_alert("unrecognized swap entry 0x%lx\n", entry.val); WARN_ON_ONCE(1); } clear_not_present_full_ptes(mm, addr, pte, nr, tlb->fullmm); - zap_install_uffd_wp_if_needed(vma, addr, pte, nr, details, ptent); + if (zap_install_uffd_wp_if_needed(vma, addr, pte, nr, details, ptent)) + set_pt_unreclaimable(&can_reclaim_pt); } while (pte +=3D nr, addr +=3D PAGE_SIZE * nr, addr !=3D end); =20 + if (addr =3D=3D end && can_reclaim_pt) + direct_reclaim =3D try_get_and_clear_pmd(mm, pmd, &pmdval); + add_mm_rss_vec(mm, rss); arch_leave_lazy_mmu_mode(); =20 @@ -1724,6 +1750,13 @@ static unsigned long zap_pte_range(struct mmu_gather= *tlb, goto retry; } =20 + if (can_reclaim_pt) { + if (direct_reclaim) + free_pte(mm, start, tlb, pmdval); + else + try_to_free_pte(mm, pmd, start, tlb); + } + return addr; } =20 diff --git a/mm/pt_reclaim.c b/mm/pt_reclaim.c new file mode 100644 index 0000000000000..fc055da40b615 --- /dev/null +++ b/mm/pt_reclaim.c @@ -0,0 +1,68 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +#include "internal.h" + +bool try_get_and_clear_pmd(struct mm_struct *mm, pmd_t *pmd, pmd_t *pmdval) +{ + spinlock_t *pml =3D pmd_lockptr(mm, pmd); + + if (!spin_trylock(pml)) + return false; + + *pmdval =3D pmdp_get_lockless(pmd); + pmd_clear(pmd); + spin_unlock(pml); + + return true; +} + +void free_pte(struct mm_struct *mm, unsigned long addr, struct mmu_gather = *tlb, + pmd_t pmdval) +{ + pte_free_tlb(tlb, pmd_pgtable(pmdval), addr); + mm_dec_nr_ptes(mm); +} + +void try_to_free_pte(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, + struct mmu_gather *tlb) +{ + pmd_t pmdval; + spinlock_t *pml, *ptl; + pte_t *start_pte, *pte; + int i; + + start_pte =3D pte_offset_map_rw_nolock(mm, pmd, addr, &pmdval, &ptl); + if (!start_pte) + return; + + pml =3D pmd_lock(mm, pmd); + if (ptl !=3D pml) + spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) + goto out_ptl; + + /* Check if it is empty PTE page */ + for (i =3D 0, pte =3D start_pte; i < PTRS_PER_PTE; i++, pte++) { + if (!pte_none(ptep_get(pte))) + goto out_ptl; + } + pte_unmap(start_pte); + + pmd_clear(pmd); + + if (ptl !=3D pml) + spin_unlock(ptl); + spin_unlock(pml); + + free_pte(mm, addr, tlb, pmdval); + + return; +out_ptl: + pte_unmap_unlock(start_pte, ptl); + if (pml !=3D ptl) + spin_unlock(pml); +} --=20 2.20.1 From nobody Tue Nov 26 15:24:30 2024 Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 323051DCB2B for ; Thu, 17 Oct 2024 09:48:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729158511; cv=none; b=EZD8ZZHzffOWNSUfX0gTOpVUhEGHk/1C1723rQOVEqkBO3UbHtfHCX+phqy/pOYSrzWbAyt55lotVbIMbxFrkV9zpks5MiRRPUGNebEIn0Z1YzF+/MVtZ7qVhKBQ3OzmWQhjfwFXjfeTQWukOiZp3nJfEiGPTrG6ASFZCMQGkFk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729158511; c=relaxed/simple; bh=5pV2lLBjHA1VmB/XAuJiCUqSMDB+4PzLY1ZtruvLm/E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JfN++otg66dpX4YXMbVdPLdoxqfZZYlhy3I10dNTUSzCVNikfmDrX6rqqk6w/fziJI0BrLCCBiGnFxgZca8NAo6iFKjb0x0hkrWoYV6rP6OoETmxuS4modhhHmLrO3m8cdCr2nJz+nCuRj+mO+2paGvaANqXpWJDOr5wbNpBYgg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=BeAN8gOE; arc=none smtp.client-ip=209.85.215.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="BeAN8gOE" Received: by mail-pg1-f181.google.com with SMTP id 41be03b00d2f7-7ea8c4ce232so762839a12.0 for ; Thu, 17 Oct 2024 02:48:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1729158505; x=1729763305; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=96N82BICgFqmbQqgF/vu6V3zKuLzfmtjKmdmS7DODoY=; b=BeAN8gOESq95JnECUlA1FEisW1jD2oOsG2K8OwJtfuwDK8KoSC2aGOiGNYkLeTrnyY HsgIKkA+UgD0EZR52IqNWDdnQtRsHEkYTuh0l/uAMsEydTQFGXM8EnzxO+v29Td6ifvs BS1qncvbQpWYSFjyYRUjkVFuE6W7/AGzIf8AZeMMifopJm9c2+WB6cZaA0FS4D/NJAMa FEJPiu8U9QIGFSIDAjNvgQOQwLZg2UMi+NLiw7f0whbDmBI+ZLs3eCsT09uTZmROF5lg zLUF2GE6kggLZoiCHRIBSA4uFx3uZDRuUwDwMJEtZJcQA2Jk71dMgI1XzcuI3Qb6bQ84 47IQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729158505; x=1729763305; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=96N82BICgFqmbQqgF/vu6V3zKuLzfmtjKmdmS7DODoY=; b=l0zSEAQqR/S3vMaoDtoGc8Rnq6Ez488+a6r4UdtuTLAqOsbGoAmavTFi6ci9+QOTMD ql65Zp3I9HNHUpAZcWv5oeY9jj6AO8DZ4HXNWnR1fP789/9+SqBqxElyFcRMworVF0G9 eoiNxo3Ds82h8QNO6ARuoxTcDZz3K6h2W5JX+aaygS3iL1BKBcrm16Az/qcSq9KK0mFS ivdxMg89ciQj4F48G6veWbnu1gkuoUBl+bgm7eLdks0DbEtL/lND/JVK84GnSlXCpDX1 g3ObG1zqQZuHjwf16j77coKn1NF2AAWuQ27rOHtL6MDQxdGcHpsSxvdpKPXYh0EXRkmt mEFQ== X-Forwarded-Encrypted: i=1; AJvYcCUTdfjqEV7ra1PNCrNOaMZmBaAHn4aNznJZl490jKEeZcSULKGy4jug0ZNOZUxjJ+rPkmNJfCpgFGQ/mys=@vger.kernel.org X-Gm-Message-State: AOJu0YyUjY+TAMxoc2N7qinOyNzVnGuEBFbJglyy1V0vf5BGiQEtFjjt QI6IVSps9Z72ZvY6zN8r+wRelFPzNvPQ8qdxUdFuwyXyqw4ZWhn4eg+tncotB7w= X-Google-Smtp-Source: AGHT+IEf2y1GA9Rtz+/5nXvZXCaQlczdVC6XcSHY78uZicXgbVhL/Djw5qd37IxfTX2YDXEO2p/4DQ== X-Received: by 2002:a05:6a21:388a:b0:1d6:97f2:71b4 with SMTP id adf61e73a8af0-1d8c9577df8mr26874708637.1.1729158505469; Thu, 17 Oct 2024 02:48:25 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71e774a4218sm4385365b3a.120.2024.10.17.02.48.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Oct 2024 02:48:24 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, mgorman@suse.de, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, zokeefe@google.com, rientjes@google.com, jannh@google.com, peterx@redhat.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, x86@kernel.org, Qi Zheng Subject: [PATCH v1 6/7] x86: mm: free page table pages by RCU instead of semi RCU Date: Thu, 17 Oct 2024 17:47:25 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now, if CONFIG_MMU_GATHER_RCU_TABLE_FREE is selected, the page table pages will be freed by semi RCU, that is: - batch table freeing: asynchronous free by RCU - single table freeing: IPI + synchronous free In this way, the page table can be lockless traversed by disabling IRQ in paths such as fast GUP. But this is not enough to free the empty PTE page table pages in paths other that munmap and exit_mmap path, because IPI cannot be synchronized with rcu_read_lock() in pte_offset_map{_lock}(). In preparation for supporting empty PTE page table pages reclaimation, let single table also be freed by RCU like batch table freeing. Then we can also use pte_offset_map() etc to prevent PTE page from being freed. Like pte_free_defer(), we can also safely use ptdesc->pt_rcu_head to free the page table pages: - The pt_rcu_head is unioned with pt_list and pmd_huge_pte. - For pt_list, it is used to manage the PGD page in x86. Fortunately tlb_remove_table() will not be used for free PGD pages, so it is safe to use pt_rcu_head. - For pmd_huge_pte, we will do zap_deposited_table() before freeing the PMD page, so it is also safe. Signed-off-by: Qi Zheng --- arch/x86/include/asm/tlb.h | 19 +++++++++++++++++++ arch/x86/kernel/paravirt.c | 7 +++++++ arch/x86/mm/pgtable.c | 10 +++++++++- mm/mmu_gather.c | 9 ++++++++- 4 files changed, 43 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 580636cdc257b..e223b53a8b190 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -34,4 +34,23 @@ static inline void __tlb_remove_table(void *table) free_page_and_swap_cache(table); } =20 +#ifdef CONFIG_PT_RECLAIM +static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) +{ + struct page *page; + + page =3D container_of(head, struct page, rcu_head); + free_page_and_swap_cache(page); +} + +static inline void __tlb_remove_table_one(void *table) +{ + struct page *page; + + page =3D table; + call_rcu(&page->rcu_head, __tlb_remove_table_one_rcu); +} +#define __tlb_remove_table_one __tlb_remove_table_one +#endif /* CONFIG_PT_RECLAIM */ + #endif /* _ASM_X86_TLB_H */ diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index fec3815335558..89688921ea62e 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -59,10 +59,17 @@ void __init native_pv_lock_init(void) static_branch_enable(&virt_spin_lock_key); } =20 +#ifndef CONFIG_PT_RECLAIM static void native_tlb_remove_table(struct mmu_gather *tlb, void *table) { tlb_remove_page(tlb, table); } +#else +static void native_tlb_remove_table(struct mmu_gather *tlb, void *table) +{ + tlb_remove_table(tlb, table); +} +#endif =20 struct static_key paravirt_steal_enabled; struct static_key paravirt_steal_rq_enabled; diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 5745a354a241c..69a357b15974a 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -19,12 +19,20 @@ EXPORT_SYMBOL(physical_mask); #endif =20 #ifndef CONFIG_PARAVIRT +#ifndef CONFIG_PT_RECLAIM static inline void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table) { tlb_remove_page(tlb, table); } -#endif +#else +static inline +void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table) +{ + tlb_remove_table(tlb, table); +} +#endif /* !CONFIG_PT_RECLAIM */ +#endif /* !CONFIG_PARAVIRT */ =20 gfp_t __userpte_alloc_gfp =3D GFP_PGTABLE_USER | PGTABLE_HIGHMEM; =20 diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 99b3e9408aa0f..d948479ca09e6 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -311,10 +311,17 @@ static inline void tlb_table_invalidate(struct mmu_ga= ther *tlb) } } =20 +#ifndef __tlb_remove_table_one +static inline void __tlb_remove_table_one(void *table) +{ + __tlb_remove_table(table); +} +#endif + static void tlb_remove_table_one(void *table) { tlb_remove_table_sync_one(); - __tlb_remove_table(table); + __tlb_remove_table_one(table); } =20 static void tlb_table_flush(struct mmu_gather *tlb) --=20 2.20.1 From nobody Tue Nov 26 15:24:30 2024 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3E3E91DA63C for ; Thu, 17 Oct 2024 09:48:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729158515; cv=none; b=qzzxkeUtFL9c+P9Kug94/T5hK0bC+p9CVkbpocJh2gkltVJanxhRYgbrwS4ofYUJ6nisZj7uLDrfTBU5ZS5RWeKMPC7nPlN1GzrAaJbSnw2KkwHzTS7pK6mUFfAF4zL7jmMN2vZ9BmVrZepOQjyy/qtvrppEPs8tsaaRnc4ONh8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729158515; c=relaxed/simple; bh=qqQo/A6GgJalHO8lTKPH/NqobaJZhvja7PakqcVbblo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=TC9GjT8FNJjOcdhcL82QYK9q5/WRfOiWixcblWafUaPTZJCNMWDVnwh5DrzSkUlBzIdC3fCykme7a3IIPjyH+u6gTJtElP7wD3hDbF0E+R/Qza+uO30KqLtBLwC9Oz95v2SDLZJ9x/kuRgWguce8MFRM+L+RR+hm6SDzG7n0vxU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=aUBdWec/; arc=none smtp.client-ip=209.85.215.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="aUBdWec/" Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-7d4fa972cbeso619492a12.2 for ; Thu, 17 Oct 2024 02:48:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1729158511; x=1729763311; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rfwXxKs8PIaYIos0iWI11FlNBR0N0b0qINIDJ43u9VQ=; b=aUBdWec/yhF9Bpd1W0tMrBCqD4U+uwzsmvwKXazQzw8kHTUyNoYEatNit4kG9YNyaj j4tLcmK3QxN2neVuAZFX9oDokjOtHsJvxwh1CgERTcKHr6lMiReeZgqVJaOiqVhpdRu9 J2Oe33ZUmrVubMKVR7JS6HJLqa1NJ4ZP5tHIkjsghxh8kSjO/6ZMgu0iXJPmRwQO2qit Ee4bFooeRQN1OLHw+328LeUZIla28sAvbjJ64wdk9dQnk9OaKBF5NQsSw40iF4tKEIbq 0HOw4xRTlYu2ZkIcj04NT7WRVlv5VIbxKLyC3H4vrHnqHE046fXDTWSOrJ00/gYXtgLV M0UQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729158511; x=1729763311; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rfwXxKs8PIaYIos0iWI11FlNBR0N0b0qINIDJ43u9VQ=; b=b7yVOJepBBkm4YFuwheZ6uwZpiO0wQ4YVhbpyHR9NgpvDOc3cy4XaLjyvkZdYfMZEY l6ByU/fyj/crqOb4w282GeQ+NO6tTetF85bhU5Om3vG1oAuu4PClQbXh8iRQTcE79++q tbZhEX4fStbNhFtPJsp+9ProjBESwHnHeK91H/sUAKYEDXKeUM3vMSKfmm9w5Pqk7iC4 J+tPCfeOk6PZXigQxlpybzEKdoOrw9FOr3grdu2hRAbg9+RU1mxbemyCAFW0nppkFjza ijndnzWunAoHnIINFt0tlD1/66kSInYzKsibR1rOiZFmb8SKmM6w/jDYaZV8liRuW0qX VK4Q== X-Forwarded-Encrypted: i=1; AJvYcCVk9aJM/k3jYZ/AF1kIsTEP8qOuz4L21tST8zOEFxH8bQXrKmDa3LX5g/MP/MgSNrZj7Hduh65lwD04hnk=@vger.kernel.org X-Gm-Message-State: AOJu0YxW/v1uJrEvtlKTHNkiNagb5b/aO2U5lFqDusfT46MdSEyTBJcw 0j09SW2jkXOkr13QTdvAN7Ai69bLMoAR6PacK7Vfe03NQhGn+8yhhkk7qO1MiW4= X-Google-Smtp-Source: AGHT+IGzAl2OFA6S9xDWJuD2m0DVBLlwiMdwmaXRwCcTvTu0dK+Cc+T0fsktUtSx/wfralK9s8v26A== X-Received: by 2002:a05:6a20:6f91:b0:1cf:3677:1c4a with SMTP id adf61e73a8af0-1d905ecb8e9mr9815945637.16.1729158511599; Thu, 17 Oct 2024 02:48:31 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71e774a4218sm4385365b3a.120.2024.10.17.02.48.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Oct 2024 02:48:30 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, mgorman@suse.de, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, zokeefe@google.com, rientjes@google.com, jannh@google.com, peterx@redhat.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, x86@kernel.org, Qi Zheng Subject: [PATCH v1 7/7] x86: select ARCH_SUPPORTS_PT_RECLAIM if X86_64 Date: Thu, 17 Oct 2024 17:47:26 +0800 Message-Id: <0f6e7fb7fb21431710f28df60738f8be98fe9dd9.1729157502.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now, x86 has fully supported the CONFIG_PT_RECLAIM feature, and reclaiming PTE pages is profitable only on 64-bit systems, so select ARCH_SUPPORTS_PT_RECLAIM if X86_64. Signed-off-by: Qi Zheng --- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 1ea18662942c9..69a20cb9ddd81 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -319,6 +319,7 @@ config X86 select FUNCTION_ALIGNMENT_4B imply IMA_SECURE_AND_OR_TRUSTED_BOOT if EFI select HAVE_DYNAMIC_FTRACE_NO_PATCHABLE + select ARCH_SUPPORTS_PT_RECLAIM if X86_64 =20 config INSTRUCTION_DECODER def_bool y --=20 2.20.1