From nobody Thu Dec 18 08:12:16 2025 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 665B61D8A10 for ; Thu, 5 Dec 2024 10:37:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395065; cv=none; b=De1Jq5UQ2059kv5+qYPBLwsm7bdGw+6TbTCbJyfBFWKP+mernff9w3n2Y1/5nYqVKwyOdKRLmuTnpynrDikNjAy+pAK2Jcn7n6bkv3WAqqv44F3TIZebIo44z3bxaJi6igVS/4c9IIZYpR73cK+Vox+btNc7k86HI+1ukihxWJM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395065; c=relaxed/simple; bh=VRSm+emlI6QfoAPRbJgRjmp46JAY59uZbRFEaBrLwsg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=lyAxWDG1WfBRi0i0NguzgIH/2aRlCsW5zJ6Ir1EFPv+dMk6CTfFDrgwpuIKkmVl4q/6wjx7ayrOSJcQHlx7p0A+BQzW0dqsSCxhKFVdzOmjSKkMu5aGvBxOYLHt+2Ai3owynqzYm+KibcTKFuIt1/0jruubHIY8El84sGgfe+Hg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=PXVINP7Q; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="PXVINP7Q" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-7253bc4d25eso622037b3a.0 for ; Thu, 05 Dec 2024 02:37:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395063; x=1733999863; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KWkBqx6Qlg0tee9JR/yzxG5KwDs2r7O63kRKnpevvpY=; b=PXVINP7Qut5lCZhkp2VINt6enIOWgvXal2F9nHzo4vw3BGZTN5nzjlHTB3/6TLQqEX q413nYYF8pWXIkLBKeOHemN3mjn7tsUH1/6lHU49NRjwDfGopkX14TiRK7omHgLM2aHr 5sW2lAO9h5itmGudrmCjv+K0u/lVTX4b4ksvUKU4CxY0b2LLDU0oJwt/Uv13SxCzu7t+ 8wbtvSy8BiKZoe3ZteS1KhTpoJKxGXM7Nmnwk+TdzIkmVZHYzaU24JjNG2YjiC2/9vYZ sXqz1EDqod2zRl/Oj+Tqk+96Djibuelbg2mrF0i7LmTBWeIxlWUU1UnoBb+9YgSR/HD3 fy+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395063; x=1733999863; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KWkBqx6Qlg0tee9JR/yzxG5KwDs2r7O63kRKnpevvpY=; b=FtjN+ivlnj6ofey1WP5B5KKhMl5t2c54REeM3UexyRZf0nV1is+E6iUaAqqQh0Qnyx U6fzHeUm1o8/Lu5uq2zTOaRqTkUM25aCXBqhyiRM5s5fokmHfK1K+a0sgQOGR/oLaooi 2//qAiiOBA8znscEDPjReGDw6MWEPwnUhtHVX0EWBcapFxs4KHFRvuEWDH696VySnG2/ wD57cjNQXv/+JPBKSo5CC1qOSlS6/rPlLoTFAvIpRaGsPaXyZmrKhev4EybXS09tirOL LmRpxosa5lrQnrTNt0eZVQ+ftLI/hcITTn7cxJDHAgIC6am/xTrBM6aTATirebpTnI2I Fpuw== X-Forwarded-Encrypted: i=1; AJvYcCWfsCR4zbNKLbuconWnFsiWAKAttg7NxNzYEaHlS5OqdmTrzEYlgAAzeH9xsmvKvBwi7N38m94PTlzsP8g=@vger.kernel.org X-Gm-Message-State: AOJu0Yx63rY9KJ4WhnYx7b+neh5eqGouwrgjC5Y/1AgC3qxPHg0MvX2+ I38PG6TXoVNJ2xK9IsylK1W7Ixc4FBCCId2Jm7eSQisAKKWXiGau9tUXLLFxJ0o= X-Gm-Gg: ASbGncsTfKgjyP+kN61EPsHFRxpmejqWo5H295sClvZkRDr2SvLJxa1k4Ac1/kAVmSt +GHfDWK1gfZZ9aHdJGOv+sTwpYwTw3URb771HgBMfNtKDrgBVU9kv5f84O3ssfDlMJWNbuNVTmA +XvTzjvUuAWWN7T1tGo53rP/AAio6navOGAlXTMbtIdF63UKVgF2WJHYyfCMFtu0VoG9blhU2+9 fyIbaSSUUw8GkA+Dyk7V5dfe1tE+sCJwB+MOcs7fxzmWIpqQJFEMyJ9eHDtQMvALbJFm+/8YL2M /vMdks/SM7KE7Xn4WzAtUHA0tNQS0tul X-Google-Smtp-Source: AGHT+IEvp+5QW/iv1/V1tpDVHfeo0s8LNYsKCxdjWx85NP0irBg++mJLHUVM5oudhxdOyXEvXSVVbg== X-Received: by 2002:a17:902:e00c:b0:215:7136:f7e3 with SMTP id d9443c01a7336-215f3ce4f83mr33803375ad.19.1733395062764; Thu, 05 Dec 2024 02:37:42 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.37.39 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:37:42 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 01/21] riscv: mm: Distinguish hardware base page and software base page Date: Thu, 5 Dec 2024 18:37:09 +0800 Message-Id: <20241205103729.14798-2-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The key idea to implement larger base page based on MMU that only supports 4K page is to decouple the MMU page from the software page in view of kernel mm. In contrary to software page, we denote the MMU page as hardware page. To decouple these two kinds of pages, we should manage, allocate and map memory at a granularity of software page, which is exactly what existing mm code does. The page table operations, however, should configure page table entries at a granularity of hardware page, which is the responsibility of arch code. This commit introduces the concept of hardware base page for RISCV. Signed-off-by: Xu Lu --- arch/riscv/Kconfig | 10 ++++++++++ arch/riscv/include/asm/page.h | 7 +++++++ arch/riscv/include/asm/pgtable-32.h | 5 +++-- arch/riscv/include/asm/pgtable-64.h | 5 +++-- arch/riscv/include/asm/pgtable-bits.h | 3 ++- arch/riscv/include/asm/pgtable.h | 1 + 6 files changed, 26 insertions(+), 5 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index fa8f2da87a0a..2c0cb175a92a 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -289,6 +289,16 @@ config PAGE_OFFSET default 0xc0000000 if 32BIT default 0xff60000000000000 if 64BIT =20 +config RISCV_HW_PAGE_SHIFT + int + default 12 + +config RISCV_USE_SW_PAGE + bool + depends on 64BIT + depends on RISCV_HW_PAGE_SHIFT !=3D PAGE_SHIFT + default n + config KASAN_SHADOW_OFFSET hex depends on KASAN_GENERIC diff --git a/arch/riscv/include/asm/page.h b/arch/riscv/include/asm/page.h index 32d308a3355f..7c581a3e057b 100644 --- a/arch/riscv/include/asm/page.h +++ b/arch/riscv/include/asm/page.h @@ -12,6 +12,10 @@ #include #include =20 +#define HW_PAGE_SHIFT CONFIG_RISCV_HW_PAGE_SHIFT +#define HW_PAGE_SIZE (_AC(1, UL) << HW_PAGE_SHIFT) +#define HW_PAGE_MASK (~(HW_PAGE_SIZE - 1)) + #define PAGE_SHIFT CONFIG_PAGE_SHIFT #define PAGE_SIZE (_AC(1, UL) << PAGE_SHIFT) #define PAGE_MASK (~(PAGE_SIZE - 1)) @@ -185,6 +189,9 @@ extern phys_addr_t __phys_addr_symbol(unsigned long x); #define __pa(x) __virt_to_phys((unsigned long)(x)) #define __va(x) ((void *)__pa_to_va_nodebug((phys_addr_t)(x))) =20 +#define pfn_to_hwpfn(pfn) (pfn << (PAGE_SHIFT - HW_PAGE_SHIFT)) +#define hwpfn_to_pfn(hwpfn) (hwpfn >> (PAGE_SHIFT - HW_PAGE_SHIFT)) + #define phys_to_pfn(phys) (PFN_DOWN(phys)) #define pfn_to_phys(pfn) (PFN_PHYS(pfn)) =20 diff --git a/arch/riscv/include/asm/pgtable-32.h b/arch/riscv/include/asm/p= gtable-32.h index 00f3369570a8..159a668c3dd8 100644 --- a/arch/riscv/include/asm/pgtable-32.h +++ b/arch/riscv/include/asm/pgtable-32.h @@ -20,9 +20,10 @@ /* * rv32 PTE format: * | XLEN-1 10 | 9 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 - * PFN reserved for SW D A G U X W R V + * HW_PFN reserved for SW D A G U X W R V */ -#define _PAGE_PFN_MASK GENMASK(31, 10) +#define _PAGE_HW_PFN_MASK GENMASK(31, 10) +#define _PAGE_PFN_MASK GENMASK(31, (10 + PAGE_SHIFT - HW_PAGE_SHIFT)) =20 #define _PAGE_NOCACHE 0 #define _PAGE_IO 0 diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/p= gtable-64.h index 0897dd99ab8d..963aa4be9eed 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -72,9 +72,10 @@ typedef struct { /* * rv64 PTE format: * | 63 | 62 61 | 60 54 | 53 10 | 9 8 | 7 | 6 | 5 | 4 | 3 | 2= | 1 | 0 - * N MT RSV PFN reserved for SW D A G U X W= R V + * N MT RSV HW_PFN reserved for SW D A G U X W= R V */ -#define _PAGE_PFN_MASK GENMASK(53, 10) +#define _PAGE_HW_PFN_MASK GENMASK(53, 10) +#define _PAGE_PFN_MASK GENMASK(53, (10 + PAGE_SHIFT - HW_PAGE_SHIFT)) =20 /* * [63] Svnapot definitions: diff --git a/arch/riscv/include/asm/pgtable-bits.h b/arch/riscv/include/asm= /pgtable-bits.h index a8f5205cea54..e5bb6a805505 100644 --- a/arch/riscv/include/asm/pgtable-bits.h +++ b/arch/riscv/include/asm/pgtable-bits.h @@ -31,7 +31,8 @@ /* Used for swap PTEs only. */ #define _PAGE_SWP_EXCLUSIVE _PAGE_ACCESSED =20 -#define _PAGE_PFN_SHIFT 10 +#define _PAGE_HWPFN_SHIFT 10 +#define _PAGE_PFN_SHIFT (_PAGE_HWPFN_SHIFT + (PAGE_SHIFT - HW_PAGE_SHIFT)) =20 /* * when all of R/W/X are zero, the PTE is a pointer to the next level diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index e79f15293492..9d6d0ff86c76 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -114,6 +114,7 @@ #include #include =20 +#define __page_val_to_hwpfn(_val) (((_val) & _PAGE_HW_PFN_MASK) >> _PAGE_= HWPFN_SHIFT) #define __page_val_to_pfn(_val) (((_val) & _PAGE_PFN_MASK) >> _PAGE_PFN_S= HIFT) =20 #ifdef CONFIG_64BIT --=20 2.20.1 From nobody Thu Dec 18 08:12:16 2025 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7A3C71DB53A for ; Thu, 5 Dec 2024 10:37:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395070; cv=none; b=Yy3z3h0Fthuj8xCXHlGUQUN3e0UfoxR+9v3jz1laBkG8VNaHhjFIbgb7ys0AjDgVnU39qjc0GpF2iPM3ljVjZW7IS0KvkYxng149CT+bV9h52R4yeCHgeQWjXCJNf/zIH+O+dWV9WXZTu2lGNq1h0S479aJX0vAomLKrhsqATRI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395070; c=relaxed/simple; bh=bhoYuDIqZkOX9ko/sk3hGKAPbQU4geAd213pGOmQX5Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=a8xSAkKiTd6hZ37z02HlLypprW42fySAAbHh8W4ErwMlb0im8vS/xDSBEAZCa8oElX+5BV6jg1oaQEcat3XUcZWkFynOTyZGTrPREX1rKMqFsVGHjOLz88JwlSh81+cP6KnE2Zp7/bGCXowHMb+xJYE+7Ww7N8aR806EvVssY84= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=DjWmvOG+; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="DjWmvOG+" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-2155157c31fso5771815ad.1 for ; Thu, 05 Dec 2024 02:37:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395068; x=1733999868; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UNKH3fzWVL17kpFj96xVkU+IFapkCvyeY63L7HJogMk=; b=DjWmvOG+tAep76Y3SH3BzqLtsXnVMAt+XyQinfaA500SnFQESsFVXkoLDkmw6Ykxas xmqfbB9awhqzNATFO2oXtY3/NeHkQkkEZEN7bAfeN9koxOPkgltjR/S+jeTyvnfnBdPi NnaPzYGnWuDMoc/O76w/1fKwEIhsEJjK2r8kLzsJGlpCXVzK5ad167nwyeOc+Lr0cyaP vefec8xCftht+4fADwDuN/6/hk0sTL2QFjbh5X5WxvipD/cMZ4yupDIq7cgS9h0ETbs3 pQIvjtxX/FluV5FOLUVaOqae0KOCqKNeC4bVVtJSM6d4c3eE+wipIM03T4XFtCccpO2C UkNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395068; x=1733999868; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UNKH3fzWVL17kpFj96xVkU+IFapkCvyeY63L7HJogMk=; b=KQHN9hiTw/zNQ9cOkG3XEObbDa+H+DmCV2zH9yGC8hoV8HRVn/WzFxF/57FGMJcD2L mRvvfUv/axaBen6ZVm6DNzupYH3Mg1ytqOcsjH0giYquqCN8LYzS0cm7SQ0JCxVvEIY/ qq3+ABvJUHxd9YIh9ccmrwuG9TeQ1b676eANm3ugCBeGr9xLCxkuhBgdckmLPDHLTv0j YSeYdY0svuEpEsla7cbE3vaM8qIeVukqWuf36NriWVgH/5zJiqSMbU56Se6SP5peYOJH 79al7Lyrs34lQv9KZE5JuXeKLeaMYC09zYfW4vfdDpboFsC59sh4+VkK97vtsRm3wY8r A9hw== X-Forwarded-Encrypted: i=1; AJvYcCUi2OEx8nRz+jc4IIRqH4qJ/pkuDHmdA4OpTjt4PKyTtgsSFVYhuR02nhoj0GUSUAc2yCZf9isdHdX0zNE=@vger.kernel.org X-Gm-Message-State: AOJu0YxYnRkn1cY+SQm4c9AIDBQ7EYDxX6XZx3JIAyqC1DjZjRS5/GfC MtsKmSZguUqF3FFUxwXk/vVrCK8sMpZ087SNsb2yBIO6+q7tmM2ZHZRioGfc5Rs= X-Gm-Gg: ASbGncsEH4u+yl7j0oMyeqXnp6CyGp1EkpI4QRGEgN3D9Qep9Hj7DWP5Vb7TVnA4ytM /CjcsNPXj8s8NkmgDJCpHj9Y0DcEvDeDcQVeFjiww5TSSlECKhWs6i+HZ3rzka3RMLXI/f0Ednl FoYt2OuXY08hV+0iNwA+BMAc3bJyhegf/9G/4B3of96XhOylmVxW1jDKZIa1KKE9ayAppqVxUqi 6K8SzyKwxYVxvHrvdT3qHLRZv7p6jSK0kLBTWiP2GiS0NXH8hhhdSWhA+YfVdGtq4w0F4O+FHSJ 2HRCBa8F295A/wnTSSRp3CVg9FviK4rV X-Google-Smtp-Source: AGHT+IGkn+Ya+kK82LZNSNu9BskAONVJ6LJWivbKYKO/n1RCbEvXyllqnQCmoUfNjTenfPmHl113mQ== X-Received: by 2002:a17:902:d48a:b0:215:8112:2fb with SMTP id d9443c01a7336-215f3c7ec62mr46538845ad.13.1733395066881; Thu, 05 Dec 2024 02:37:46 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.37.43 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:37:46 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 02/21] riscv: mm: Configure satp with hw page pfn Date: Thu, 5 Dec 2024 18:37:10 +0800 Message-Id: <20241205103729.14798-3-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The control register CSR_SATP on RISC-V, which points to the root page table page, is used by MMU to translate va to pa when TLB miss happens. Thus it should be encoded at a granularity of hardware page, while existing code usually encodes it via software page frame number. This commit corrects encoding operations of CSR_SATP register. To get developers rid of the annoying encoding format of CSR_SATP and the conversion between sw pfn and hw pfn, we abstract the encoding operations of CSR_SATP into a specific function. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 14 ++++++++++++++ arch/riscv/kernel/head.S | 4 ++-- arch/riscv/kernel/hibernate.c | 3 ++- arch/riscv/mm/context.c | 7 +++---- arch/riscv/mm/fault.c | 2 +- arch/riscv/mm/init.c | 7 +++++-- arch/riscv/mm/kasan_init.c | 7 +++++-- 7 files changed, 32 insertions(+), 12 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index 9d6d0ff86c76..9d3947ec3523 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -206,6 +206,20 @@ extern pgd_t swapper_pg_dir[]; extern pgd_t trampoline_pg_dir[]; extern pgd_t early_pg_dir[]; =20 +static inline unsigned long make_satp(unsigned long pfn, + unsigned long asid, unsigned long satp_mode) +{ + return (pfn_to_hwpfn(pfn) | + ((asid & SATP_ASID_MASK) << SATP_ASID_SHIFT) | satp_mode); +} + +static inline unsigned long satp_pfn(unsigned long satp) +{ + unsigned long hwpfn =3D satp & SATP_PPN; + + return hwpfn_to_pfn(hwpfn); +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE static inline int pmd_present(pmd_t pmd) { diff --git a/arch/riscv/kernel/head.S b/arch/riscv/kernel/head.S index 356d5397b2a2..b8568e3ddefa 100644 --- a/arch/riscv/kernel/head.S +++ b/arch/riscv/kernel/head.S @@ -86,7 +86,7 @@ relocate_enable_mmu: csrw CSR_TVEC, a2 =20 /* Compute satp for kernel page tables, but don't load it yet */ - srl a2, a0, PAGE_SHIFT + srl a2, a0, HW_PAGE_SHIFT la a1, satp_mode XIP_FIXUP_OFFSET a1 REG_L a1, 0(a1) @@ -100,7 +100,7 @@ relocate_enable_mmu: */ la a0, trampoline_pg_dir XIP_FIXUP_OFFSET a0 - srl a0, a0, PAGE_SHIFT + srl a0, a0, HW_PAGE_SHIFT or a0, a0, a1 sfence.vma csrw CSR_SATP, a0 diff --git a/arch/riscv/kernel/hibernate.c b/arch/riscv/kernel/hibernate.c index 671b686c0158..155be6b1d32c 100644 --- a/arch/riscv/kernel/hibernate.c +++ b/arch/riscv/kernel/hibernate.c @@ -395,7 +395,8 @@ int swsusp_arch_resume(void) if (ret) return ret; =20 - hibernate_restore_image(resume_hdr.saved_satp, (PFN_DOWN(__pa(resume_pg_d= ir)) | satp_mode), + hibernate_restore_image(resume_hdr.saved_satp, + make_satp(PFN_DOWN(__pa(resume_pg_dir)), 0, satp_mode), resume_hdr.restore_cpu_addr); =20 return 0; diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 4abe3de23225..229c78d9ad3a 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -189,9 +189,8 @@ static void set_mm_asid(struct mm_struct *mm, unsigned = int cpu) raw_spin_unlock_irqrestore(&context_lock, flags); =20 switch_mm_fast: - csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | - (cntx2asid(cntx) << SATP_ASID_SHIFT) | - satp_mode); + csr_write(CSR_SATP, make_satp(virt_to_pfn(mm->pgd), cntx2asid(cntx), + satp_mode)); =20 if (need_flush_tlb) local_flush_tlb_all(); @@ -200,7 +199,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned = int cpu) static void set_mm_noasid(struct mm_struct *mm) { /* Switch the page table and blindly nuke entire local TLB */ - csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | satp_mode); + csr_write(CSR_SATP, make_satp(virt_to_pfn(mm->pgd), 0, satp_mode)); local_flush_tlb_all_asid(0); } =20 diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c index a9f2b4af8f3f..4772152be0f9 100644 --- a/arch/riscv/mm/fault.c +++ b/arch/riscv/mm/fault.c @@ -133,7 +133,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, = int code, unsigned long a * of a task switch. */ index =3D pgd_index(addr); - pfn =3D csr_read(CSR_SATP) & SATP_PPN; + pfn =3D satp_pfn(csr_read(CSR_SATP)); pgd =3D (pgd_t *)pfn_to_virt(pfn) + index; pgd_k =3D init_mm.pgd + index; =20 diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 0e8c20adcd98..f9334aab45a6 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -836,7 +836,7 @@ static __init void set_satp_mode(uintptr_t dtb_pa) (uintptr_t)early_p4d : (uintptr_t)early_pud, PGDIR_SIZE, PAGE_TABLE); =20 - identity_satp =3D PFN_DOWN((uintptr_t)&early_pg_dir) | satp_mode; + identity_satp =3D make_satp(PFN_DOWN((uintptr_t)&early_pg_dir), 0, satp_m= ode); =20 local_flush_tlb_all(); csr_write(CSR_SATP, identity_satp); @@ -1316,6 +1316,8 @@ static void __init create_linear_mapping_page_table(v= oid) =20 static void __init setup_vm_final(void) { + unsigned long satp; + /* Setup swapper PGD for fixmap */ #if !defined(CONFIG_64BIT) /* @@ -1349,7 +1351,8 @@ static void __init setup_vm_final(void) clear_fixmap(FIX_P4D); =20 /* Move to swapper page table */ - csr_write(CSR_SATP, PFN_DOWN(__pa_symbol(swapper_pg_dir)) | satp_mode); + satp =3D make_satp(PFN_DOWN(__pa_symbol(swapper_pg_dir)), 0, satp_mode); + csr_write(CSR_SATP, satp); local_flush_tlb_all(); =20 pt_ops_set_late(); diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c index c301c8d291d2..3eee1665358e 100644 --- a/arch/riscv/mm/kasan_init.c +++ b/arch/riscv/mm/kasan_init.c @@ -482,11 +482,13 @@ static void __init create_tmp_mapping(void) =20 void __init kasan_init(void) { + unsigned long satp; phys_addr_t p_start, p_end; u64 i; =20 create_tmp_mapping(); - csr_write(CSR_SATP, PFN_DOWN(__pa(tmp_pg_dir)) | satp_mode); + satp =3D make_satp(PFN_DOWN(__pa(tmp_pg_dir)), 0, satp_mode); + csr_write(CSR_SATP, satp); =20 kasan_early_clear_pgd(pgd_offset_k(KASAN_SHADOW_START), KASAN_SHADOW_START, KASAN_SHADOW_END); @@ -531,6 +533,7 @@ void __init kasan_init(void) memset(kasan_early_shadow_page, KASAN_SHADOW_INIT, PAGE_SIZE); init_task.kasan_depth =3D 0; =20 - csr_write(CSR_SATP, PFN_DOWN(__pa(swapper_pg_dir)) | satp_mode); + satp =3D make_satp(PFN_DOWN(__pa(swapper_pg_dir)), 0, satp_mode); + csr_write(CSR_SATP, satp); local_flush_tlb_all(); } --=20 2.20.1 From nobody Thu Dec 18 08:12:16 2025 Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E9141DC046 for ; Thu, 5 Dec 2024 10:37:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395075; cv=none; b=LJ3NbvUIzLgYEm27/jWq0iQxXGiiHuRORFeX4yvAjnAvLfX1gn4S9wnU3j3GQsdqjopFOUZjnKjboRsAMY7CoFi7TbIBbFz3KunnVKwps0u/USu0k1+d62yMOL4006iAxffh3R0mKqgs2hRPm1JlP1NM5Sbq27EOqGgAQ0TY8vk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395075; c=relaxed/simple; bh=3h/3p3d/jgDr8aXeHN9HrkJ5/RDVlpW+np4nhoQy9uo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=kdCqTWHZ53yVOSRUjX7UTfAlcHYOo2jtgYyF/rfMZDiPEvi2eR8wUfgdoT4OOpIMSn+DmWQs5IKBrnIxQUWAqK1xM9Ayvk7DHapEd62HxZnAhAUFEJJg2zOX5komm6OhTLm82zHT8uL/5mGbU5PunKIiQrzx9cWIpuLLhx/MSGc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=M9Gefwf0; arc=none smtp.client-ip=209.85.215.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="M9Gefwf0" Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-7ee11ff7210so653134a12.1 for ; Thu, 05 Dec 2024 02:37:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395072; x=1733999872; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=teCsoKxDGeiN3+DzdoxY3wVIl2UTRwagjcvIS6Z3bMs=; b=M9Gefwf0/McuRDvxTm1oJotlatXTQYL2viJY0xcsTTpdAXhIaMfWETVz7a4V08obGa fxK1JSHvW2AzeBvEyrjxjSXRHUaxy9h+7NF7xL5z3uYRIvsQX/54tnoX5wsC+KIeN1X1 ktnm9ctOqmRGufOlYhNSQfalp+1ZZ2WA4F7Gq56dzHKnSOiHu4uGnzndPPZnzLGefbOJ 6TL1Skl38TVn69asly6Km0INH5U5SjZPBA8sCyY9quCpHlgrB2FkDOQqryQoNfv2hzX1 /Ia6Z/+638vk4RvZINs0H0UA9LXtAq5oclat5VTADG+KD6YUBLSXlnlLFyYhdQEBogSg F4UA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395072; x=1733999872; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=teCsoKxDGeiN3+DzdoxY3wVIl2UTRwagjcvIS6Z3bMs=; b=a8JChBf5eo+2lJGx3niPMwus32Q68XqmEyNuHzWnh+czd+MF3DnjchFjDR8PHB06TG oWM0JZhIOYL1+SGFFz8lolGnplD1tciVNF5oXINe0m6V6MFXUNFHBD57OXx/Eg3U5YWZ qQb/XNBuXJ7j9z/1ODo/frFA1fHV/iJMosneVy/XkajvQAvNuGXxCa770L/l2k9410l0 684RWS6QxiyYwOpY0HEOmTNlk9cAfCDuCLALLwc4/+BdP4VKv+LFLrTbfTmTGEgsdjMV GHUlIwqi49qb/0OauUpyqN+sSA/1a2zmJ4HY4EIgeed3P6iTG1kW7qcYsSnELcTxB3xA 3wIg== X-Forwarded-Encrypted: i=1; AJvYcCWoFKIIJzyXcjkofCZZF8m0j/XvxAlYZz0R3/Oy/Q9DIK3wcFsZVi7uFucQExNP90Gwfg2ecaZML6u1Xa4=@vger.kernel.org X-Gm-Message-State: AOJu0YwqgmOWO4ox2Cy+OOXep8uEqItn5LbwowXN2mnzqNcMKjL0B+uO CF4tDMLY5g9aBqf2hXObIWCKhT2dfJivVXTX+Ua64RBXcWsEBipODNTKMeyFJDs= X-Gm-Gg: ASbGncsyBrL27gtFAzc8359MkNwOYbdGK8gw+8UUAoiMJ7SghApLcWxjpt3Bl0Ala4w PFnV1xfClftYQGcPESTSAlg0mZOT2EoshZ91XwqPXwUXWYfHj7pWzusDh1G34B+OX8y/Qk8lybN QOphxFuhGopQWSZgyWDFTQs/u0IMFBmrhh3khTMaBO4vrBRqvhAA7lZWJ6MCJ0qVttTWs+q99PW FAbfRlKA7vbR4Grj11Bq3fcaDMGYAcIhokEVDz/LbXJ/Cr8otrMtzNUIM98KKs1yusHp9txvK3b vFVkx6l6Tu0gEkMHz8gjVCC2wV+UAtl5 X-Google-Smtp-Source: AGHT+IHv/nBjYCvlcRUcvZSgl+N0//z6T9cTlerWiKA9ANKuf1lPaFN2gvQdQxXb4nAsIJJZSEYzng== X-Received: by 2002:a05:6a20:1d98:b0:1db:efab:390c with SMTP id adf61e73a8af0-1e1653f3f22mr17696126637.35.1733395071049; Thu, 05 Dec 2024 02:37:51 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.37.47 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:37:50 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 03/21] riscv: mm: Reimplement page table entry structures Date: Thu, 5 Dec 2024 18:37:11 +0800 Message-Id: <20241205103729.14798-4-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" After decoupling hardware base page and software base page, each software page can consists of several hardware base pages now. The pte struct should be turned to an array of mapping entires to map the software page. For example, in 64K Page Size kernel, each software page consists of 16 contiguous hardware pages. Thus the pte struct should contains 16 mapping entries to map 16 hardware pages. This commit reimplements pte structure. Signed-off-by: Xu Lu --- arch/riscv/include/asm/page.h | 43 +++++++++++++++++++++++---- arch/riscv/include/asm/pgtable-64.h | 41 +++++++++++++++++++++---- arch/riscv/include/asm/pgtable.h | 23 +++++++++++++-- arch/riscv/mm/pgtable.c | 46 +++++++++++++++++++++++++++++ 4 files changed, 141 insertions(+), 12 deletions(-) diff --git a/arch/riscv/include/asm/page.h b/arch/riscv/include/asm/page.h index 7c581a3e057b..9bc908d94c7a 100644 --- a/arch/riscv/include/asm/page.h +++ b/arch/riscv/include/asm/page.h @@ -63,6 +63,36 @@ void clear_page(void *page); * Use struct definitions to apply C type checking */ =20 +#ifdef CONFIG_RISCV_USE_SW_PAGE + +#define HW_PAGES_PER_PAGE (1 << (PAGE_SHIFT - HW_PAGE_SHIFT)) + +struct page_table_entry { + union { + unsigned long pgds[HW_PAGES_PER_PAGE]; + unsigned long p4ds[HW_PAGES_PER_PAGE]; + unsigned long puds[HW_PAGES_PER_PAGE]; + unsigned long pmds[HW_PAGES_PER_PAGE]; + unsigned long ptes[HW_PAGES_PER_PAGE]; + }; +}; + +/* Page Global Directory entry */ +typedef struct page_table_entry pgd_t; + +/* Page Table entry */ +typedef struct page_table_entry pte_t; + +#define pte_val(x) ((x).ptes[0]) +#define pgd_val(x) ((x).pgds[0]) + +pte_t __pte(unsigned long pteval); +pgd_t __pgd(unsigned long pgdval); +#define __pte __pte +#define __pgd __pgd + +#else /* CONFIG_RISCV_USE_SW_PAGE */ + /* Page Global Directory entry */ typedef struct { unsigned long pgd; @@ -73,18 +103,21 @@ typedef struct { unsigned long pte; } pte_t; =20 +#define pte_val(x) ((x).pte) +#define pgd_val(x) ((x).pgd) + +#define __pte(x) ((pte_t) { (x) }) +#define __pgd(x) ((pgd_t) { (x) }) + +#endif /* CONFIG_RISCV_USE_SW_PAGE */ + typedef struct { unsigned long pgprot; } pgprot_t; =20 typedef struct page *pgtable_t; =20 -#define pte_val(x) ((x).pte) -#define pgd_val(x) ((x).pgd) #define pgprot_val(x) ((x).pgprot) - -#define __pte(x) ((pte_t) { (x) }) -#define __pgd(x) ((pgd_t) { (x) }) #define __pgprot(x) ((pgprot_t) { (x) }) =20 #ifdef CONFIG_64BIT diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/p= gtable-64.h index 963aa4be9eed..e736873d7768 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -41,6 +41,35 @@ extern bool pgtable_l5_enabled; #define PMD_SIZE (_AC(1, UL) << PMD_SHIFT) #define PMD_MASK (~(PMD_SIZE - 1)) =20 +#ifdef CONFIG_RISCV_USE_SW_PAGE + +/* Page 4th Directory entry */ +typedef struct page_table_entry p4d_t; + +#define p4d_val(x) ((x).p4ds[0]) +p4d_t __p4d(unsigned long p4dval); +#define __p4d __p4d +#define PTRS_PER_P4D (PAGE_SIZE / sizeof(p4d_t)) + +/* Page Upper Directory entry */ +typedef struct page_table_entry pud_t; + +#define pud_val(x) ((x).puds[0]) +pud_t __pud(unsigned long pudval); +#define __pud __pud +#define PTRS_PER_PUD (PAGE_SIZE / sizeof(pud_t)) + +/* Page Middle Directory entry */ +typedef struct page_table_entry pmd_t; + +#define pmd_val(x) ((x).pmds[0]) +pmd_t __pmd(unsigned long pmdval); +#define __pmd __pmd + +#define PTRS_PER_PMD (PAGE_SIZE / sizeof(pmd_t)) + +#else /* CONFIG_RISCV_USE_SW_PAGE */ + /* Page 4th Directory entry */ typedef struct { unsigned long p4d; @@ -69,6 +98,8 @@ typedef struct { =20 #define PTRS_PER_PMD (PAGE_SIZE / sizeof(pmd_t)) =20 +#endif /* CONFIG_RISCV_USE_SW_PAGE */ + /* * rv64 PTE format: * | 63 | 62 61 | 60 54 | 53 10 | 9 8 | 7 | 6 | 5 | 4 | 3 | 2= | 1 | 0 @@ -98,7 +129,7 @@ enum napot_cont_order { #define for_each_napot_order_rev(order) \ for (order =3D NAPOT_ORDER_MAX - 1; \ order >=3D NAPOT_CONT_ORDER_BASE; order--) -#define napot_cont_order(val) (__builtin_ctzl((val.pte >> _PAGE_PFN_SHIFT)= << 1)) +#define napot_cont_order(val) (__builtin_ctzl((pte_val(val) >> _PAGE_PFN_S= HIFT) << 1)) =20 #define napot_cont_shift(order) ((order) + PAGE_SHIFT) #define napot_cont_size(order) BIT(napot_cont_shift(order)) @@ -279,7 +310,7 @@ static inline void set_p4d(p4d_t *p4dp, p4d_t p4d) if (pgtable_l4_enabled) WRITE_ONCE(*p4dp, p4d); else - set_pud((pud_t *)p4dp, (pud_t){ p4d_val(p4d) }); + set_pud((pud_t *)p4dp, __pud(p4d_val(p4d))); } =20 static inline int p4d_none(p4d_t p4d) @@ -327,7 +358,7 @@ static inline pud_t *p4d_pgtable(p4d_t p4d) if (pgtable_l4_enabled) return (pud_t *)pfn_to_virt(__page_val_to_pfn(p4d_val(p4d))); =20 - return (pud_t *)pud_pgtable((pud_t) { p4d_val(p4d) }); + return (pud_t *)pud_pgtable(__pud(p4d_val(p4d))); } #define p4d_page_vaddr(p4d) ((unsigned long)p4d_pgtable(p4d)) =20 @@ -346,7 +377,7 @@ static inline void set_pgd(pgd_t *pgdp, pgd_t pgd) if (pgtable_l5_enabled) WRITE_ONCE(*pgdp, pgd); else - set_p4d((p4d_t *)pgdp, (p4d_t){ pgd_val(pgd) }); + set_p4d((p4d_t *)pgdp, __p4d(pgd_val(pgd))); } =20 static inline int pgd_none(pgd_t pgd) @@ -384,7 +415,7 @@ static inline p4d_t *pgd_pgtable(pgd_t pgd) if (pgtable_l5_enabled) return (p4d_t *)pfn_to_virt(__page_val_to_pfn(pgd_val(pgd))); =20 - return (p4d_t *)p4d_pgtable((p4d_t) { pgd_val(pgd) }); + return (p4d_t *)p4d_pgtable(__p4d(pgd_val(pgd))); } #define pgd_page_vaddr(pgd) ((unsigned long)pgd_pgtable(pgd)) =20 diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index 9d3947ec3523..f9aed43809b3 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -574,6 +574,25 @@ static inline void __set_pte_at(struct mm_struct *mm, = pte_t *ptep, pte_t pteval) =20 #define PFN_PTE_SHIFT _PAGE_PFN_SHIFT =20 +#ifdef CONFIG_RISCV_USE_SW_PAGE +static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) +{ + unsigned int i; + + if (pte_present(pte) && !pte_napot(pte)) + for (i =3D 0; i < HW_PAGES_PER_PAGE; i++) + pte.ptes[i] +=3D nr << _PAGE_PFN_SHIFT; + + return pte; +} +#else +static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) +{ + return __pte(pte_val(pte) + (nr << _PAGE_PFN_SHIFT)); +} +#endif +#define pte_advance_pfn pte_advance_pfn + static inline void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval, unsigned int nr) { @@ -584,7 +603,7 @@ static inline void set_ptes(struct mm_struct *mm, unsig= ned long addr, if (--nr =3D=3D 0) break; ptep++; - pte_val(pteval) +=3D 1 << _PAGE_PFN_SHIFT; + pteval =3D pte_advance_pfn(pteval, 1); } } #define set_ptes set_ptes @@ -882,7 +901,7 @@ extern pmd_t pmdp_collapse_flush(struct vm_area_struct = *vma, ((offset) << __SWP_OFFSET_SHIFT) }) =20 #define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) }) -#define __swp_entry_to_pte(x) ((pte_t) { (x).val }) +#define __swp_entry_to_pte(x) (__pte((x).val)) =20 static inline int pte_swp_exclusive(pte_t pte) { diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index 4ae67324f992..0c6b2fc6be58 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -5,6 +5,52 @@ #include #include =20 +#ifdef CONFIG_RISCV_USE_SW_PAGE + +pte_t __pte(unsigned long pteval) +{ + pte_t pte; + + return pte; +} +EXPORT_SYMBOL(__pte); + +pgd_t __pgd(unsigned long pgdval) +{ + pgd_t pgd; + + return pgd; +} +EXPORT_SYMBOL(__pgd); + +#ifdef CONFIG_64BIT +p4d_t __p4d(unsigned long p4dval) +{ + p4d_t p4d; + + return p4d; +} +EXPORT_SYMBOL(__p4d); + +pud_t __pud(unsigned long pudval) +{ + pud_t pud; + + return pud; +} +EXPORT_SYMBOL(__pud); + +pmd_t __pmd(unsigned long pmdval) +{ + pmd_t pmd; + + return pmd; +} +EXPORT_SYMBOL(__pmd); +#endif /* CONFIG_64BIT */ + +#endif /* CONFIG_RISCV_USE_SW_PAGE */ + int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty) --=20 2.20.1 From nobody Thu Dec 18 08:12:16 2025 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CFC8A1DA605 for ; Thu, 5 Dec 2024 10:37:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395077; cv=none; b=iv1VgK3cQUxBMWkHKUdWxVOe/Algaw+/JwvIx7NoPQoxI4h+9meykItQjjexULTwyASSDg+GWnPeTn8MusEbSRbRPYg5TiaXus9Q3KgjkFTLhp3zpJfqX1N1+lNVdTut2Lwi/mwS13UIOq40ZFk/hqs/VoRO4PtMEJfsmjkKCYQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395077; c=relaxed/simple; bh=jUNY6BJZlto1cTtwp/89dNrWlebhXYA8qzsooGeN2uc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=GlA4PJE6bGd5NXbxZslFoRDMmWr34iasICPKkKunPpxPjCDShUlbFGJqbdPI17hsw24OB/PM8kJfH+/8iDXv2tiMZvv6gUAqKCiAclebeuOLvbY3UpvSGpcFTdrzn+LtyfkjA4bgtBLug4iAIpdJ5vvT4XAvj2aPPCm3+TouPr8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=avLg8tQ4; arc=none smtp.client-ip=209.85.215.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="avLg8tQ4" Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-7fcdbfd8090so606114a12.2 for ; Thu, 05 Dec 2024 02:37:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395075; x=1733999875; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cc/dZWEu2lOmGL5teCDDaBtpVgBp3/NGadgIOM//MYE=; b=avLg8tQ4gLQurRtwi2SIY+EgLV0sxOMmf4gm/Gl4AzF3W6Qfb5BPdICuyDTDCqiaWD DGCqd62rdV0bkSqjwyiddxjnRxHwvkJ7enTj+A28fFHnGlukQdv0EhkYVVpfV70bkQYK fKGUGsqzLoXtavQEEujqbVqwGeiv+3y2xLUXf7d3Z3hpbh2An4ApwQe/sSdGl7kCOHYZ e5nVrpuHuD9XYQdaGFyXS5hgpPV/mFicg1U+StAYtbhvNOM5JU4/UZ6wchseldltrWcf 4ez2fWc7hxSzm58ICXM4+9SDIN9hYKfTBh5JU7Xs87tFtGot+8n2BG0WAeJhvdsx3kIN dUNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395075; x=1733999875; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cc/dZWEu2lOmGL5teCDDaBtpVgBp3/NGadgIOM//MYE=; b=A42/zEfjadE+QtM0EOgfty5Oz8glSH43wb7+RJRi1sJCAuS4s4RNcHHFrCoi0rre7R m0uCuWNNh/PKr0stFc5kA1eoPLqYkNfOLzlRF6D8sMoGj6yGUs5rEkfJZBn1pJyHJX6R Bqyg+1NWojb/hFal+eNBkmYBd+Xu5JMXrByp5A93WT/+vgJQOz7eXHbAB53hXNzQvpkt 39t2mOPNcgOt8/YPH7lrCqs9MoHSc3XiDN+GLBMDVWZGeYEpZ5xjJnprMp6SKtjTNy/D je7+7IR7KnB91TKlOQG0wj7vgEpM+Lqo7ENuyiVaguH+YU7elKQ/Yi9bLQxawzlwBeOb DbqA== X-Forwarded-Encrypted: i=1; AJvYcCXz5aHVt8huo4CpfQA0bp5ueiYGP7NSxhhooQ4tLsGp/ayX2EJp61n83Oq2YzAwaj/8OxNnc1a4WltuNTE=@vger.kernel.org X-Gm-Message-State: AOJu0YxBhoOEpBbC6lYb3bxH8qUFklXMk07xT/RObhR6eFFDJLjq1k9D 6qoitBPMkRsHze8WwbxywN8mhxcawAcBkmfUlRQ3vn5cThGTGKgOKBPQxlq3ArQ= X-Gm-Gg: ASbGncv3lycykzfL8xeMJY1jO2ODE9QWk7nMbrkEWU6aWQ+K024Xbjwp/YXXzmEy2Ak TLspMjM+a+2BDt4nYaO3CX6KMDfpQVFw2Lz0YcY4/JxsbB9eC727TzkrHHFxfdCM135J5m4mpNu OS9ww1q9Djzb2EG70h+81sI5ZGmeI0cSkZBvki813gon6ybcOkMxOwYT1aLeqzQtMDn47lMde8E sJIHjdOvuiVESmv2jiL+FBXJW2RJfmE4SheCeg0I9X54xbVGwIkCWFS5cmkEb69F6Atwm54OVMq 2rS8wBhPNiXR8OsyRJVRgTLpZPLPjLbv X-Google-Smtp-Source: AGHT+IEA0IERrJMVRep0HIEJfScJ16JhQtj/WdB3oMlrbTyvrSoqe//NvEXdVU3Seu11zobAb4MPFg== X-Received: by 2002:a05:6a21:3989:b0:1db:e3f6:55f with SMTP id adf61e73a8af0-1e1653b992amr15407579637.18.1733395075109; Thu, 05 Dec 2024 02:37:55 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.37.51 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:37:54 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 04/21] riscv: mm: Reimplement page table entry constructor function Date: Thu, 5 Dec 2024 18:37:12 +0800 Message-Id: <20241205103729.14798-5-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit reimplements the page table entry constructor. As each page can contains several hardware pages now, the pte constructor need to initialize all mapping entries of these hardware pages. Note that the step path between mapping entries differs in different page table entry levels. For example, in PTE level, the step path between hardware mapping entries is hardware page size (aka 4K). In PMD level, the step path is (2 ^ 9) * hardware page size (aka 2M), etc. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable-32.h | 5 +++ arch/riscv/include/asm/pgtable-64.h | 41 +++++++++++++++++++--- arch/riscv/include/asm/pgtable.h | 54 ++++++++++++++++++++++++----- arch/riscv/mm/pgtable.c | 47 +++++++++++++++++++++++++ 4 files changed, 133 insertions(+), 14 deletions(-) diff --git a/arch/riscv/include/asm/pgtable-32.h b/arch/riscv/include/asm/p= gtable-32.h index 159a668c3dd8..2959ab72f926 100644 --- a/arch/riscv/include/asm/pgtable-32.h +++ b/arch/riscv/include/asm/pgtable-32.h @@ -37,4 +37,9 @@ static const __maybe_unused int pgtable_l4_enabled; static const __maybe_unused int pgtable_l5_enabled; =20 +static inline int __pgd_present(unsigned long pgdval) +{ + return pgdval & _PAGE_PRESENT; +} + #endif /* _ASM_RISCV_PGTABLE_32_H */ diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/p= gtable-64.h index e736873d7768..efcf63667f93 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -204,9 +204,14 @@ static inline u64 riscv_page_io(void) _PAGE_USER | _PAGE_GLOBAL | \ _PAGE_MTMASK)) =20 +static inline int __pud_present(unsigned long pudval) +{ + return pudval & _PAGE_PRESENT; +} + static inline int pud_present(pud_t pud) { - return (pud_val(pud) & _PAGE_PRESENT); + return __pud_present(pud_val(pud)); } =20 static inline int pud_none(pud_t pud) @@ -219,11 +224,16 @@ static inline int pud_bad(pud_t pud) return !pud_present(pud); } =20 -#define pud_leaf pud_leaf +static inline bool __pud_leaf(unsigned long pudval) +{ + return __pud_present(pudval) && (pudval & _PAGE_LEAF); +} + static inline bool pud_leaf(pud_t pud) { - return pud_present(pud) && (pud_val(pud) & _PAGE_LEAF); + return __pud_leaf(pud_val(pud)); } +#define pud_leaf pud_leaf =20 static inline int pud_user(pud_t pud) { @@ -321,14 +331,30 @@ static inline int p4d_none(p4d_t p4d) return 0; } =20 +static inline int __p4d_present(unsigned long p4dval) +{ + return p4dval & _PAGE_PRESENT; +} + static inline int p4d_present(p4d_t p4d) { if (pgtable_l4_enabled) - return (p4d_val(p4d) & _PAGE_PRESENT); + return __p4d_present(p4d_val(p4d)); =20 return 1; } =20 +static inline int __p4d_leaf(unsigned long p4dval) +{ + return 0; +} + +static inline int p4d_leaf(p4d_t p4d) +{ + return __p4d_leaf(p4d_val(p4d)); +} +#define p4d_leaf p4d_leaf + static inline int p4d_bad(p4d_t p4d) { if (pgtable_l4_enabled) @@ -388,10 +414,15 @@ static inline int pgd_none(pgd_t pgd) return 0; } =20 +static inline int __pgd_present(unsigned long pgdval) +{ + return pgdval & _PAGE_PRESENT; +} + static inline int pgd_present(pgd_t pgd) { if (pgtable_l5_enabled) - return (pgd_val(pgd) & _PAGE_PRESENT); + return __pgd_present(pgd_val(pgd)); =20 return 1; } diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index f9aed43809b3..1d5f533edbd5 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -220,8 +220,19 @@ static inline unsigned long satp_pfn(unsigned long sat= p) return hwpfn_to_pfn(hwpfn); } =20 +static inline int __pgd_leaf(unsigned long pgdval) +{ + return __pgd_present(pgdval) && (pgdval & _PAGE_LEAF); +} + +static inline int pgd_leaf(pgd_t pgd) +{ + return __pgd_leaf(pgd_val(pgd)); +} +#define pgd_leaf pgd_leaf + #ifdef CONFIG_TRANSPARENT_HUGEPAGE -static inline int pmd_present(pmd_t pmd) +static inline int __pmd_present(unsigned long pmdval) { /* * Checking for _PAGE_LEAF is needed too because: @@ -229,15 +240,20 @@ static inline int pmd_present(pmd_t pmd) * the present bit, in this situation, pmd_present() and * pmd_trans_huge() still needs to return true. */ - return (pmd_val(pmd) & (_PAGE_PRESENT | _PAGE_PROT_NONE | _PAGE_LEAF)); + return (pmdval & (_PAGE_PRESENT | _PAGE_PROT_NONE | _PAGE_LEAF)); } #else -static inline int pmd_present(pmd_t pmd) +static inline int __pmd_present(unsigned long pmdval) { - return (pmd_val(pmd) & (_PAGE_PRESENT | _PAGE_PROT_NONE)); + return (pmdval & (_PAGE_PRESENT | _PAGE_PROT_NONE)); } #endif =20 +static inline int pmd_present(pmd_t pmd) +{ + return __pmd_present(pmd_val(pmd)); +} + static inline int pmd_none(pmd_t pmd) { return (pmd_val(pmd) =3D=3D 0); @@ -248,11 +264,16 @@ static inline int pmd_bad(pmd_t pmd) return !pmd_present(pmd) || (pmd_val(pmd) & _PAGE_LEAF); } =20 -#define pmd_leaf pmd_leaf +static inline bool __pmd_leaf(unsigned long pmdval) +{ + return __pmd_present(pmdval) && (pmdval & _PAGE_LEAF); +} + static inline bool pmd_leaf(pmd_t pmd) { - return pmd_present(pmd) && (pmd_val(pmd) & _PAGE_LEAF); + return __pmd_leaf(pmd_val(pmd)); } +#define pmd_leaf pmd_leaf =20 static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) { @@ -306,9 +327,14 @@ static __always_inline bool has_svnapot(void) return riscv_has_extension_likely(RISCV_ISA_EXT_SVNAPOT); } =20 +static inline unsigned long __pte_napot(unsigned long val) +{ + return val & _PAGE_NAPOT; +} + static inline unsigned long pte_napot(pte_t pte) { - return pte_val(pte) & _PAGE_NAPOT; + return __pte_napot(pte_val(pte)); } =20 static inline pte_t pte_mknapot(pte_t pte, unsigned int order) @@ -324,11 +350,16 @@ static inline pte_t pte_mknapot(pte_t pte, unsigned i= nt order) =20 static __always_inline bool has_svnapot(void) { return false; } =20 -static inline unsigned long pte_napot(pte_t pte) +static inline unsigned long __pte_napot(unsigned long pteval) { return 0; } =20 +static inline unsigned long pte_napot(pte_t pte) +{ + return __pte_napot(pte_val(pte)); +} + #endif /* CONFIG_RISCV_ISA_SVNAPOT */ =20 /* Yields the page frame number (PFN) of a page table entry */ @@ -356,9 +387,14 @@ static inline pte_t pfn_pte(unsigned long pfn, pgprot_= t prot) =20 #define mk_pte(page, prot) pfn_pte(page_to_pfn(page), prot) =20 +static inline int __pte_present(unsigned long pteval) +{ + return (pteval & (_PAGE_PRESENT | _PAGE_PROT_NONE)); +} + static inline int pte_present(pte_t pte) { - return (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROT_NONE)); + return __pte_present(pte_val(pte)); } =20 #define pte_accessible pte_accessible diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index 0c6b2fc6be58..f57ada26a183 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -10,6 +10,13 @@ pte_t __pte(unsigned long pteval) { pte_t pte; + unsigned int i; + + for (i =3D 0; i < HW_PAGES_PER_PAGE; i++) { + pte.ptes[i] =3D pteval; + if (__pte_present(pteval) && !__pte_napot(pteval)) + pteval +=3D 1 << _PAGE_HWPFN_SHIFT; + } =20 return pte; } @@ -18,6 +25,16 @@ EXPORT_SYMBOL(__pte); pgd_t __pgd(unsigned long pgdval) { pgd_t pgd; + unsigned int i; + + for (i =3D 0; i < HW_PAGES_PER_PAGE; i++) { + pgd.pgds[i] =3D pgdval; + if (__pgd_leaf(pgdval)) + pgdval +=3D (1 << (PGDIR_SHIFT - PAGE_SHIFT)) << + _PAGE_HWPFN_SHIFT; + else if (__pgd_present(pgdval)) + pgdval +=3D 1 << _PAGE_HWPFN_SHIFT; + } =20 return pgd; } @@ -27,6 +44,16 @@ EXPORT_SYMBOL(__pgd); p4d_t __p4d(unsigned long p4dval) { p4d_t p4d; + unsigned int i; + + for (i =3D 0; i < HW_PAGES_PER_PAGE; i++) { + p4d.p4ds[i] =3D p4dval; + if (__p4d_leaf(p4dval)) + p4dval +=3D (1 << (P4D_SHIFT - PAGE_SHIFT)) << + _PAGE_HWPFN_SHIFT; + else if (__p4d_present(p4dval)) + p4dval +=3D 1 << _PAGE_HWPFN_SHIFT; + } =20 return p4d; } @@ -35,6 +62,16 @@ EXPORT_SYMBOL(__p4d); pud_t __pud(unsigned long pudval) { pud_t pud; + unsigned int i; + + for (i =3D 0; i < HW_PAGES_PER_PAGE; i++) { + pud.puds[i] =3D pudval; + if (__pud_leaf(pudval)) + pudval +=3D (1 << (PUD_SHIFT - PAGE_SHIFT)) << + _PAGE_HWPFN_SHIFT; + else if (__pud_present(pudval)) + pudval +=3D 1 << _PAGE_HWPFN_SHIFT; + } =20 return pud; } @@ -43,6 +80,16 @@ EXPORT_SYMBOL(__pud); pmd_t __pmd(unsigned long pmdval) { pmd_t pmd; + unsigned int i; + + for (i =3D 0; i < HW_PAGES_PER_PAGE; i++) { + pmd.pmds[i] =3D pmdval; + if (__pmd_leaf(pmdval)) + pmdval +=3D (1 << (PMD_SHIFT - PAGE_SHIFT)) << + _PAGE_HWPFN_SHIFT; + else if (__pmd_present(pmdval)) + pmdval +=3D 1 << _PAGE_HWPFN_SHIFT; + } =20 return pmd; } --=20 2.20.1 From nobody Thu Dec 18 08:12:16 2025 Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D0DE41DACB4 for ; Thu, 5 Dec 2024 10:37:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395081; cv=none; b=R7TioWbHX9pMBQKRDPdAvaPpsrmXD731GqvYzu2BiVbg9ic4nQ2eIjsce7Y7kwzt0Zj3T/0iIoCqnATP7h74h9S8hqTQU3RccLPTxoZ9QMPK1dP/RhfWW0sW4NGKOGY4U7oClxjxVFcRnvsnWXUH/q8bNqed4BV0N64sUg8loyw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395081; c=relaxed/simple; bh=Vm9Zkyv13t259pbyXmOKGdd1ewdyriURgjYtEXU6Jc0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gE9PMC5YDFIGjORDhqMpnRWM8i+gDMdr0GZo8iuMlM55BrqEYAURxu4HAxLM+EZJftL+7w2o0FqBbsqUpp91LI922kzlGwvQMa79NfMzurpH0E/ww04tUqOP0hg1WKwv9u7S0SpirZw9koDFbSVK/VhBqecCdEtGDZWC0dbdA3s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=OoWhgIVU; arc=none smtp.client-ip=209.85.215.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="OoWhgIVU" Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-7fc22a88bcbso609494a12.2 for ; Thu, 05 Dec 2024 02:37:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395079; x=1733999879; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7qR1jHqy2cDl06AgpV8oSKqeRVPpykz0hiLJxVMX/VM=; b=OoWhgIVUaX73GDZdtsN/xgz6J7uPp9UO3QkM5upoNXgphW2o/UlCzPyL8EVs4j6mdV cCtJDDdqM5QzNmi6yoNiPPByFSCJr0SzahyU0k90SWUoxQVT8Iymli7MnLU7cFrWnY8c 27Y8wmi/sZhxKypUCTL7GzzzXtGPEzxYhHWJ086JDX/iRXU3AN5Uoq9l+sQtkqnFloX4 EpTxT8ZBmTW0SboZEB6OAFAimcIQgIKsL5uZey2vgdXegsaLN/iK+vrgBkR09toux42u K4FjiHSJM8Az8pYssvupVL2LoGOLje3ReQ1EH6rg/9YYiskLdYenLZS/VwMUe3D2bbnp S1Fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395079; x=1733999879; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7qR1jHqy2cDl06AgpV8oSKqeRVPpykz0hiLJxVMX/VM=; b=bp3+3P2l+utIqN3N3oIuIa7VdMioTBXaxbbpTLOKbOsTipB7nOM5FtYVGMGxYeYcHl p3VhWZbYtsZxQA13BvvdzXvA29hMbYkYDvK3lCj81rW4DY4w2PuBnFbRNM2fie9X9Kaq f51K1zWzgc6zlIA7+yek9jOEWB/TPF7IOE4EZNVOkkELDC8ePg/N8j2UYlQ0hNdNMJdi 7mPV694P+ljMYTt6+Z85M/CU16clyTnIJCpjFIPY517Fi1X9N9VQK2vBDXxYvR8jQApR /7zhOk/2hymso9RhnKG5q5cTM3zmqIhv1Na+yjgazJRJZAouZwwUU+s5W85G72c2dd2D pH+Q== X-Forwarded-Encrypted: i=1; AJvYcCVcfHnfVv2FN3uhlBIYwe9epR13e07i1dBT8Jr350vaduVcMzinQyGVPi9W5l1acxfZLDcBgiz9Tq8zpaE=@vger.kernel.org X-Gm-Message-State: AOJu0YzShhkXhvuzfnnlf89bEnIL/neif8FQ1Y8/8mFmQs+cI1FTlBCj PJNbOSk/Fv7sAPjHeGgBHdNvNaDPv7YzqANSCQGuXDctRLldYGZEetXZKAz8dJc= X-Gm-Gg: ASbGncsipUcxCD+b2QtTjbCeK5z7jcMaql9Gs20C3KBkWnmbUwTN2810t76hDnC6hDP iLqdf5tPMQ3wI2Ku9R1pz086m84KIBPSNOcLrGybfbvvZwEi7BbYtpheg9wtF2UntFCEddacyYZ uI30qshI1YXO2aRRFkPgHSFkOqiDrzrPEO+SPmdxahXJyvDGGsoenKR7toQK+vsjGLSaOqCzUSF 5Pn+HPHdJf0lToBa5yjkn1fHNSD9iPTtuzM8Ned8Aufe+gis0FYLzfN42LpTzQP7rXLBHVtb2k0 hm3Bk+BEe18YfQLnPkAV9wdt4O4aGhDr X-Google-Smtp-Source: AGHT+IEUKSwiJ64FZ3Pe2Otsaz2xFzyoQCTrZnVdkwgamPJ8aR60BE+SnfSU6DdJI6PGfNndTdnNkA== X-Received: by 2002:a05:6a21:99a2:b0:1db:e358:48a8 with SMTP id adf61e73a8af0-1e1653b91d0mr15785669637.10.1733395079129; Thu, 05 Dec 2024 02:37:59 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.37.55 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:37:58 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 05/21] riscv: mm: Reimplement conversion functions between page table entry Date: Thu, 5 Dec 2024 18:37:13 +0800 Message-Id: <20241205103729.14798-6-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Some code will convert high level pte into low level pte to reuse pte functions. For example, pmd_dirty() will convert pmd struct into pte struct to check whether it is dirty using pte_dirty(). As pte struct at different level has different constructor now, we can not apply pte constructor during conversion. Thus, this commit converts ptes by directly converting structure type. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 81 ++++++++++++++++++++++++++++++-- 1 file changed, 76 insertions(+), 5 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index 1d5f533edbd5..f7b51c52b815 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -309,6 +309,50 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd) return (unsigned long)pfn_to_virt(__page_val_to_pfn(pmd_val(pmd))); } =20 +#ifdef CONFIG_RISCV_USE_SW_PAGE + +static inline pte_t pmd_pte(pmd_t pmd) +{ + return (pte_t)pmd; +} + +static inline pte_t pud_pte(pud_t pud) +{ + return (pte_t)pud; +} + +static inline pte_t p4d_pte(p4d_t p4d) +{ + return (pte_t)p4d; +} + +static inline pte_t pgd_pte(pgd_t pgd) +{ + return (pte_t)pgd; +} + +static inline pmd_t pte_pmd(pte_t pte) +{ + return (pmd_t)pte; +} + +static inline pud_t pte_pud(pte_t pte) +{ + return (pud_t)pte; +} + +static inline p4d_t pte_p4d(pte_t pte) +{ + return (p4d_t)pte; +} + +static inline pgd_t pte_pgd(pte_t pte) +{ + return (pgd_t)pte; +} + +#else /* CONFIG_RISCV_USE_SW_PAGE */ + static inline pte_t pmd_pte(pmd_t pmd) { return __pte(pmd_val(pmd)); @@ -319,6 +363,38 @@ static inline pte_t pud_pte(pud_t pud) return __pte(pud_val(pud)); } =20 +static inline pte_t p4d_pte(p4d_t p4d) +{ + return __pte(p4d_val(p4d)); +} + +static inline pte_t pgd_pte(pgd_t pgd) +{ + return __pte(pgd_val(pgd)); +} + +static inline pmd_t pte_pmd(pte_t pte) +{ + return __pmd(pte_val(pte)); +} + +static inline pud_t pte_pud(pte_t pte) +{ + return __pud(pte_val(pte)); +} + +static inline p4d_t pte_p4d(pte_t pte) +{ + return __p4d(pte_val(pte)); +} + +static inline pgd_t pte_pgd(pte_t pte) +{ + return __pgd(pte_val(pte)); +} + +#endif /* CONFIG_RISCV_USE_SW_PAGE */ + #ifdef CONFIG_RISCV_ISA_SVNAPOT #include =20 @@ -728,11 +804,6 @@ static inline pgprot_t pgprot_writecombine(pgprot_t _p= rot) /* * THP functions */ -static inline pmd_t pte_pmd(pte_t pte) -{ - return __pmd(pte_val(pte)); -} - static inline pmd_t pmd_mkhuge(pmd_t pmd) { return pmd; --=20 2.20.1 From nobody Thu Dec 18 08:12:16 2025 Received: from mail-ot1-f42.google.com (mail-ot1-f42.google.com [209.85.210.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2FB601DE2AC for ; Thu, 5 Dec 2024 10:38:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395085; cv=none; b=dFnC9YepkoD4w/MXtDw85XxSQCC5USKPBqji7q+OHKcZnl045JP0XcTHdQ21ph3zOFh4TXKUgVr5bvOVhB0yQwSk/kdvh2Da1I88l4RjMIOKTg28HRuZor43BiLJ/76mK6YoHTyyp4Sl6RBHppZaMdbRE7wk98UffEeKUSAAXRI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395085; c=relaxed/simple; bh=n5ObmEMSgZ6wjmUYK4IExflSlCXOnNpiwrBV7NJsDhs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=qRgutOiMYeAi4iFEnRHBtgfD2506+bHbt72cdtXWa1m2716rIguuQkGV5AKlyJ+tF5RrXtO9WMdFUhtw38WrWxAO029jXq/3NucjWQDgJ+Ji7Ar2qIRBZ8GD+E9ZrAnnK7+C5QPmH14YCq6oypeteUQZJM6JTOKTzFce4iUpxpI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=PMiUc5nt; arc=none smtp.client-ip=209.85.210.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="PMiUc5nt" Received: by mail-ot1-f42.google.com with SMTP id 46e09a7af769-71d46ace74aso371754a34.3 for ; Thu, 05 Dec 2024 02:38:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395083; x=1733999883; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XbXb8+gIbpPc0seecsAjhzRbpNpiwhPRJnpv+Yg3m9U=; b=PMiUc5ntK1X3HF5d6pDLQvBXxh4+o13sxAcg9wThcutXqhn6LImGc7Zyvx8wP3nZsp fbzfj9Q9lnzd/kGAiyL/mOczEZ9E3IR3SZ0nAVtQhYrJIrPMT/PMYjw9/yGpXt2X8XwR RE8e8gFoGk5+mamE+91Mma0A8PUhuYD0G6BRkJXe+9COBo32zrqZahkCJxfzlxKSBAoo p2/KQwuU7xboMFmYKyg7x+hCxI518cUvFIwWcOMTh6fOjxvLb1U1AUQnvZ1lWFEUPfgm 1dnu8QENuw58jbFy6qQCSHKY4r8JAUeosVkykc8p4oBZ7QxHaI9bZo2N7KWpt4uBPUXO p6TQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395083; x=1733999883; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XbXb8+gIbpPc0seecsAjhzRbpNpiwhPRJnpv+Yg3m9U=; b=tdvDrT1lMMm9e8mrhq/sW2ii1keBRiZKaK4405edLkrIWco5DShWQbZHvL2MVldJ3n ll4yhNUU+AikJPyFYOw3oYE6cUaQIwrW1MR46ynE/K8uGiCbq03SsZsYEiJTfl0w5yBO 6WKas+6WZ7PCTIG4tBDVOwNL2pKG6kWAqoseZ2cI+OgLP67otT0oJUVqQ/A4jgQUGMw8 2u+E+rMKQVGcXa33GQkFr/mm8yyoZHgpbycYCKvI1+G3R8UoRmTOi9CTndExFxmgn78F Xe5EbbfdzgcoWBBjFMFHwsDsPRWfvawdupFXvi+gmiD7cI28zNzaYuuss810/dDUXGK0 YDwg== X-Forwarded-Encrypted: i=1; AJvYcCWYd2xpWeeA7dvSQfs7Pv2m7dQyMZfyfgjbjKUdNMx4c7pbya+eLePdgNbSPyzgrGBrpSo869H+FV25tPg=@vger.kernel.org X-Gm-Message-State: AOJu0YwfHkTbrKZB97mQSSJat9s9av/xr1TImpahlI/lwNLUg0QTD1N0 OvJq5NGyhwqO82qTRyl4jFmwgSBk2v4rWg2OpmzRLU77ih1tvbHMloDlpa8L17Q= X-Gm-Gg: ASbGncuwCvCx6aEeSw1ztYp1tCZ/dwLEXKY6iFoeBpfvPkgueYfg5qkwsJB4Z9GsI/F EHtpmTpB6wtiL+kYhTpoccYHzAKCu5/lXrrchICaz4ux3VjX8Ny6P4A2j1bQgJZPczttWE69OAg kFgodd7S89ysvJOXqdi76282JuXcmu9sUcR2NJjyXp10wwo/RpyZRNt6QXPceKT8H8z82pxsSww LJSW4BfgiSbiVZoHSlPmh7wKDliCYkEJQPuXQKZrj0eU4Qsjwvy/ZyBPjYC0smvD0OTdKtL5ync pBJDMjtfrivD5OyPnrrZIKvdLBm229/y X-Google-Smtp-Source: AGHT+IEx3vS74gMj6raF1dIqh7j7ldd5iu4wb4DU/N4Y4QqLjAI1/NclGnjqcHYWmLe9eo91DygJAQ== X-Received: by 2002:a05:6830:d1a:b0:71a:6c7a:e23b with SMTP id 46e09a7af769-71dad652d39mr12397021a34.16.1733395083206; Thu, 05 Dec 2024 02:38:03 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.37.59 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:02 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 06/21] riscv: mm: Avoid pte constructor during pte conversion Date: Thu, 5 Dec 2024 18:37:14 +0800 Message-Id: <20241205103729.14798-7-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit converts ptes at different level via directly converting pte type instead of using pte constructor, as ptes from different levels has different constructors. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 52 +++++++++++++++++++++++++------- 1 file changed, 41 insertions(+), 11 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index f7b51c52b815..d3da8aee213c 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -351,6 +351,26 @@ static inline pgd_t pte_pgd(pte_t pte) return (pgd_t)pte; } =20 +static inline pte_t pte_set_flag(pte_t pte, unsigned long flag) +{ + unsigned int i; + + for (i =3D 0; i < HW_PAGES_PER_PAGE; i++) + pte.ptes[i] |=3D flag; + + return pte; +} + +static inline pte_t pte_clear_flag(pte_t pte, unsigned long flag) +{ + unsigned int i; + + for (i =3D 0; i < HW_PAGES_PER_PAGE; i++) + pte.ptes[i] &=3D (~flag); + + return pte; +} + #else /* CONFIG_RISCV_USE_SW_PAGE */ =20 static inline pte_t pmd_pte(pmd_t pmd) @@ -393,6 +413,16 @@ static inline pgd_t pte_pgd(pte_t pte) return __pgd(pte_val(pte)); } =20 +static inline pte_t pte_set_flag(pte_t pte, unsigned long flag) +{ + return __pte(pte_val(pte) | flag); +} + +static inline pte_t pte_clear_flag(pte_t pte, unsigned long flag) +{ + return __pte(pte_val(pte) & (~flag)); +} + #endif /* CONFIG_RISCV_USE_SW_PAGE */ =20 #ifdef CONFIG_RISCV_ISA_SVNAPOT @@ -537,46 +567,46 @@ static inline int pte_devmap(pte_t pte) =20 static inline pte_t pte_wrprotect(pte_t pte) { - return __pte(pte_val(pte) & ~(_PAGE_WRITE)); + return pte_clear_flag(pte, _PAGE_WRITE); } =20 /* static inline pte_t pte_mkread(pte_t pte) */ =20 static inline pte_t pte_mkwrite_novma(pte_t pte) { - return __pte(pte_val(pte) | _PAGE_WRITE); + return pte_set_flag(pte, _PAGE_WRITE); } =20 /* static inline pte_t pte_mkexec(pte_t pte) */ =20 static inline pte_t pte_mkdirty(pte_t pte) { - return __pte(pte_val(pte) | _PAGE_DIRTY); + return pte_set_flag(pte, _PAGE_DIRTY); } =20 static inline pte_t pte_mkclean(pte_t pte) { - return __pte(pte_val(pte) & ~(_PAGE_DIRTY)); + return pte_clear_flag(pte, _PAGE_DIRTY); } =20 static inline pte_t pte_mkyoung(pte_t pte) { - return __pte(pte_val(pte) | _PAGE_ACCESSED); + return pte_set_flag(pte, _PAGE_ACCESSED); } =20 static inline pte_t pte_mkold(pte_t pte) { - return __pte(pte_val(pte) & ~(_PAGE_ACCESSED)); + return pte_clear_flag(pte, _PAGE_ACCESSED); } =20 static inline pte_t pte_mkspecial(pte_t pte) { - return __pte(pte_val(pte) | _PAGE_SPECIAL); + return pte_set_flag(pte, _PAGE_SPECIAL); } =20 static inline pte_t pte_mkdevmap(pte_t pte) { - return __pte(pte_val(pte) | _PAGE_DEVMAP); + return pte_set_flag(pte, _PAGE_DEVMAP); } =20 static inline pte_t pte_mkhuge(pte_t pte) @@ -612,7 +642,7 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newp= rot) =20 ALT_THEAD_PMA(newprot_val); =20 - return __pte((pte_val(pte) & _PAGE_CHG_MASK) | newprot_val); + return pte_set_flag(pte_clear_flag(pte, ~_PAGE_CHG_MASK), newprot_val); } =20 #define pgd_ERROR(e) \ @@ -1017,12 +1047,12 @@ static inline int pte_swp_exclusive(pte_t pte) =20 static inline pte_t pte_swp_mkexclusive(pte_t pte) { - return __pte(pte_val(pte) | _PAGE_SWP_EXCLUSIVE); + return pte_set_flag(pte, _PAGE_SWP_EXCLUSIVE); } =20 static inline pte_t pte_swp_clear_exclusive(pte_t pte) { - return __pte(pte_val(pte) & ~_PAGE_SWP_EXCLUSIVE); + return pte_clear_flag(pte, _PAGE_SWP_EXCLUSIVE); } =20 #ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION --=20 2.20.1 From nobody Thu Dec 18 08:12:16 2025 Received: from mail-ot1-f43.google.com (mail-ot1-f43.google.com [209.85.210.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 154251DF748 for ; Thu, 5 Dec 2024 10:38:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395089; cv=none; b=ChrJU2N5Trt6tj9mS4LSu5QHEf7+zY8GLxeh69bqYFCWbdbliNIqEvrVEEIcMTQ2HNumS0m2rTdNZEllYn8i5gkA10YMHnyOrhJEBdKjneBatah2J8eSilcDvFMwnSZEyduiAY3vcWjVSCL8QaBraC8h067Oak2o52aVaE5l3zU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395089; c=relaxed/simple; bh=l5MuxZWDOeWyprgQyypSfLl0LW6lklDXjU9sKESU9IY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NnDQ5IL9aU0FSk11vWh4ZAjk7MLxnawSEWxDPVBCYiAFYCWdKZJZoXTqlWWJXgeK5sVcwkf7LSlQEyOgwUhuQ6l7thg1qBw3/xycmH0rKxnp+a9hH6N4q+bLpg6TjplDZqBYPyY78JiMKUOQhFZvysqblsNfsdibR1Zm87zOrDQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=NNp+Cndm; arc=none smtp.client-ip=209.85.210.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="NNp+Cndm" Received: by mail-ot1-f43.google.com with SMTP id 46e09a7af769-71d46ace74aso371775a34.3 for ; Thu, 05 Dec 2024 02:38:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395087; x=1733999887; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cw560nhmcCgMtoVMwQuAveKPR5/TXRAfvCK89A67aqo=; b=NNp+Cndm+KS57k21i/wUyQfcTJXavb+n9b/Oe9pbLLUA1pdYcvZqiM67UwpLL5LYnG m7hcHOQ2MR5zYm27rmuteX7qwHPtg5Asjxm4RooRJw8RVz/D7lvrWohm9atj10xZTOQm HZbIZCvfzSnpHpDnd0w02oGPZXWrdcD5oMWHtXmBh9bX9lTKc+UmUPMnmPdFa+st4TZa lrt3wL7gvG6iZ6lYHU50Iah9X5XGG6D8oXPvW8MXE3MAg6zo/UjdJIOh/TUWHAxH2Txn D2Q64rXQC1XlUcW4UEaDTN77qU5A1Y0ToaDNcR/B01W7sfUZgKZk7erA53ggQncC0lmb ZfQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395087; x=1733999887; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cw560nhmcCgMtoVMwQuAveKPR5/TXRAfvCK89A67aqo=; b=v8vo28rsvgPrjXXEfkJ2w+wIyNEXJ5PMC2za0i9ZKn7yrsIhpnOTU6O7ii0hbpspge //v6R7uLqjo4VtlPN5C1X7aucUmlsP0iuV0Gn5tzV9h3IdnrN0G8++hfva/wK6GtOrp6 /lvvmmalNhgsA/+K9aiumqK3HEFr4P15RhSUrZ/BUFR1mNKtZ1KgCGkvOsDsosqAwzbg y+qqDm6d8OF3s+xuK/LEgtJ/4ba7jBYOi/oOaa4p1245GT3VCXDwjldGXy6E0tDJrsBN M7Tf84AvSze9ZZe5ZRVne6mA0eSl70OKd/oe3BBFpm9cKSbCAWeqmn84eWHI9w5/pNV9 A7nw== X-Forwarded-Encrypted: i=1; AJvYcCWNZcg5NKgtyMhHGrpAvkDetAyVHJWvW8f43axNkeNowT+zlErm/S9dd6upLtvug3ho0VnOfsItyJOmJJw=@vger.kernel.org X-Gm-Message-State: AOJu0Yx4lPZy2y5KUUYBg6XFz2z4Bb7Iy1cW/a7xGsmxqyNOrD9YERHR PRr5mpuVm+5KWsPDeucRjImUQqyCQ/J5WD1hL8Jwu7QamQ3OreVyhr3MX3GO9CQ= X-Gm-Gg: ASbGncsiZdCGG4ENIHYV4FinZqteZosBftKYskamnP9Y/AYlDIlBwSljt7OhmXxyPH4 JR+LgC2mImWnCLBXQAn/UhNlC+URWeGsj9x52H7FqWoIGc55ZUkym3ka9B61Q1yjDWhL0/+d8HR drnDs+KfMEYIbb3eb+PlVt3OMuqVVsBftJFyw5qMt79WyLXYCPYc3kb3tX2OH4aL3QvTdEYfpaa r2e5ucHjrbfem2jlXDXn19jc+J/g2txoXr/+WD883rBZshqrgDn2SShniDwZRj8u4LOkfAql9Yp FHXT3PpmicPdNXWlVKDzwu6mKKjAU7xF X-Google-Smtp-Source: AGHT+IFgLX/MNdvVLEUe2bcmja2NfdcbWMv5BT39DW17XBNTzPIKVvGRlTw6+0l7AjGg6yRiky8Jng== X-Received: by 2002:a05:6830:6016:b0:71d:54fb:da56 with SMTP id 46e09a7af769-71dad5f33bbmr12858019a34.2.1733395087266; Thu, 05 Dec 2024 02:38:07 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.03 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:07 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 07/21] riscv: mm: Reimplement page table entry get function Date: Thu, 5 Dec 2024 18:37:15 +0800 Message-Id: <20241205103729.14798-8-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit reimplements ptep_get/pmdp_get/... functions. As pte structures now contains multiple mapping entries, we can not use READ_ONCE to fetch its value. Instead, we use traditional dereference way. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index d3da8aee213c..ba4a083b7210 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -704,6 +704,36 @@ static inline void set_pte(pte_t *ptep, pte_t pteval) WRITE_ONCE(*ptep, pteval); } =20 +static inline pte_t ptep_get(pte_t *ptep) +{ + return *ptep; +} +#define ptep_get ptep_get + +static inline pmd_t pmdp_get(pmd_t *pmdp) +{ + return *pmdp; +} +#define pmdp_get pmdp_get + +static inline pud_t pudp_get(pud_t *pudp) +{ + return *pudp; +} +#define pudp_get pudp_get + +static inline p4d_t p4dp_get(p4d_t *p4dp) +{ + return *p4dp; +} +#define p4dp_get p4dp_get + +static inline pgd_t pgdp_get(pgd_t *pgdp) +{ + return *pgdp; +} +#define pgdp_get pgdp_get + void flush_icache_pte(struct mm_struct *mm, pte_t pte); =20 static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t p= teval) --=20 2.20.1 From nobody Thu Dec 18 08:12:16 2025 Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 058AD1DF983 for ; Thu, 5 Dec 2024 10:38:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395093; cv=none; b=YmcsA5kK9OCutVMfc8Gm4YhZlH7yEzDolJDHXfkNS21i2YxYlPqOuH3Hh2aRwAZfJhbBIlYuAEGjsRuIZRDugCowHxdPxXt+BqdEawOBdZBY3VIRZPFtgqf3Pl5Pryd99l+WfBKIM4+rZuBlIcTy8FAX0pWQ9G0K+Jhj8kTfJag= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395093; c=relaxed/simple; bh=42gI9iWeOqYKXWG+UM1aKNB9ZHJ4WDj8WIzuLk8JrdU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JzIdNEms7mvafm8deF8GQB6bP5deO7CA/46x45jSptdhl75JGDwlsSXlPN5hgxQkK4dolgw4Imtazo/0MSKCHPTTZ1dX4UA6ZhPPOk8MVLq/3L5qab3uQcllxh7Cdkz6dPoJotOyxB9f/KTkI2B4GOSshCWaqjO6GNIhepzJzYI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=jiMUABAQ; arc=none smtp.client-ip=209.85.215.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="jiMUABAQ" Received: by mail-pg1-f169.google.com with SMTP id 41be03b00d2f7-7fcdbfd8090so606271a12.2 for ; Thu, 05 Dec 2024 02:38:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395091; x=1733999891; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O7hmUlID2KY6paYgoqiid5lfsbGFxDDCAN4DBvbC41c=; b=jiMUABAQJC8Sf5DBP7eHBzeDaeUTTYJKfm+Vw/cMUP8qq9Pqj/ygZbHwlzXIcub+60 2Llds1P0jcpKG2FVKh3DOdGesGw66SwsmdruR7fVK4ZPsagjZO6ud5LUJ0XQnfJP6Zv0 6BTO0ELm9OAQfKP6j0PEx508i+HwRNXcwQKJqPdN/rGPptldfFK5EUPWNkW95iP27j92 NSFoXvNUmE2epIwZaIFXN+LsU4/9E5coycoEMeKE6A55WKIoqw+kp+WhlH//VqwbQtFc BKATaDbUgTheLpHedz4zcYDbZpNcOgZidvFZkGVbR8HTfusgikZGN+YJwJlzqT78gX7n dxRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395091; x=1733999891; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O7hmUlID2KY6paYgoqiid5lfsbGFxDDCAN4DBvbC41c=; b=qonpwgsJ0C+BDiNy3g2uGWa8ma+YNJqvuh4XDTi97qDtCPKRVmCjMbub9D0LmSHINS EjKu3O5sCBKF3Te5L7BgEeeTHXD1SG6y+TcmVn8nBxzI8mA3TkFZ9Ls20XDq251RRXur n7aUmsm3KvZDqVWRk/+khL5SqQ5j7Xa65q5WKlwUIJTXP8C6CZFfeYDNlXJTkCwN3vr9 ChmhjZfetNA53Q89iSR1cUmz20O1Y6E1y9NrlzORcuxy+XPpj9xl5FIZmlTYF8E59wFq uOb1RSeoxDHMaGo4rPqTBabHE9zuvyM98UCPtBoQcaVYtOHb2EKe8+skDW6m7o5gr460 z77A== X-Forwarded-Encrypted: i=1; AJvYcCVD1gPiNvzeNKranbZqA5OExye7lbD70R93SDwJ8KQnGB/EJr0OPiXZL0WZde4/WzIVsuPpc4EJ8hOrIes=@vger.kernel.org X-Gm-Message-State: AOJu0YwykTUNqKToHrjZk4gU+/UPiETQJHScrijkhSPjZ7RilV26eIXq thGvgiez3o1WLbFfz9OD1KLfIVnBcBlRK4ph8pgOOw+8Bjs0DhItp/95BvLGEXE= X-Gm-Gg: ASbGnctYvcsyl9BzIZfaUnIT3G5eqUF4ZUy/RyzUsiGNVE8mK2Byl37pJStFEFaurQU XU7kYUwTETBWRACKDVzvRHFoPNfiw/scRmrEX0XhgY/FqSNf2L7j5Ad+k0d/L3WpCajYKuF8eVV L7ScAz2ZZmXPEEXwSPomlC9gUfmTYZEv4vD5N+hP46JopjO928kjx0lhCb4B1POFf4boBTmZLqV MpCJhENAYtWy8JowHOaoQBbz0rftgaAM+WS1z0aEXfC8JZ1SHQkwKXuxcOm13n2cBNnGcMzIaOl Srl6nWtB0R82EXlMXLlW3MwMPau+Tw+d X-Google-Smtp-Source: AGHT+IHbEOK4fVgmqFCn4T84ALhR/xBKYInyH1RV8Ugvahqt2YHHrGpIWPwGC34ww4f3ko0WEWYg3w== X-Received: by 2002:a05:6a20:cfa4:b0:1e0:d104:4dbd with SMTP id adf61e73a8af0-1e16542f238mr16645992637.46.1733395091287; Thu, 05 Dec 2024 02:38:11 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.07 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:11 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 08/21] riscv: mm: Reimplement page table entry atomic get function Date: Thu, 5 Dec 2024 18:37:16 +0800 Message-Id: <20241205103729.14798-9-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit implements lockless functions to atomically fetch pte's value. For each pte structure, we atomically fetch the first mapping entry, and then fetch the following entries and compare them with the first mappin entry plus certain step path in a loop. If we find any difference in their pfns or prots, then the pte structure has been modified and need to be reloaded. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 156 +++++++++++++++++++++++++++++++ include/linux/pgtable.h | 21 +++++ 2 files changed, 177 insertions(+) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index ba4a083b7210..fe42afb4441e 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -220,6 +220,18 @@ static inline unsigned long satp_pfn(unsigned long sat= p) return hwpfn_to_pfn(hwpfn); } =20 +static inline unsigned long __pte_pgprot(unsigned long pteval) +{ + unsigned long prot_mask =3D GENMASK(_PAGE_HWPFN_SHIFT - 1, 0); + + return pteval & prot_mask; +} + +static inline pgprot_t pte_pgprot(pte_t pte) +{ + return __pgprot(__pte_pgprot(pte_val(pte))); +} + static inline int __pgd_leaf(unsigned long pgdval) { return __pgd_present(pgdval) && (pgdval & _PAGE_LEAF); @@ -734,6 +746,150 @@ static inline pgd_t pgdp_get(pgd_t *pgdp) } #define pgdp_get pgdp_get =20 +#ifdef CONFIG_RISCV_USE_SW_PAGE +static inline pte_t ptep_get_lockless(pte_t *ptep) +{ + unsigned long pteval; + pte_t pte; + int i; + +retry: + pteval =3D READ_ONCE(ptep->ptes[0]); + pte =3D *ptep; + for (i =3D 0; i < HW_PAGES_PER_PAGE; i++) { + if (__page_val_to_pfn(pteval) !=3D + __page_val_to_pfn(pte.ptes[i])) + goto retry; + if ((__pte_pgprot(pteval) | _PAGE_DIRTY | _PAGE_ACCESSED) !=3D + (__pte_pgprot(pte.ptes[i]) | _PAGE_DIRTY | _PAGE_ACCESSED)) + goto retry; + + if (__pte_present(pteval) && !__pte_napot(pteval)) + pteval +=3D 1 << _PAGE_HWPFN_SHIFT; + } + + return pte; +} +#define ptep_get_lockless ptep_get_lockless + +static inline pmd_t pmdp_get_lockless(pmd_t *pmdp) +{ + unsigned long pmdval; + pmd_t pmd; + int i; + +retry: + pmdval =3D READ_ONCE(pmdp->pmds[0]); + pmd =3D *pmdp; + for (i =3D 0; i < HW_PAGES_PER_PAGE; i++) { + if (__page_val_to_pfn(pmdval) !=3D + __page_val_to_pfn(pmd.pmds[i])) + goto retry; + if ((__pte_pgprot(pmdval) | _PAGE_DIRTY | _PAGE_ACCESSED) !=3D + (__pte_pgprot(pmd.pmds[i]) | _PAGE_DIRTY | _PAGE_ACCESSED)) + goto retry; + + if (__pmd_leaf(pmdval)) + pmdval +=3D (1 << (PMD_SHIFT - PAGE_SHIFT)) << + _PAGE_HWPFN_SHIFT; + else if (__pmd_present(pmdval)) + pmdval +=3D 1 << _PAGE_HWPFN_SHIFT; + } + + return pmd; +} +#define pmdp_get_lockless pmdp_get_lockless + +static inline void pmdp_get_lockless_sync(void) +{ +} + +static inline pud_t pudp_get_lockless(pud_t *pudp) +{ + unsigned long pudval; + pud_t pud; + int i; + +retry: + pudval =3D READ_ONCE(pudp->puds[0]); + pud =3D *pudp; + for (i =3D 0; i < HW_PAGES_PER_PAGE; i++) { + if (__page_val_to_pfn(pudval) !=3D + __page_val_to_pfn(pud.puds[i])) + goto retry; + if ((__pte_pgprot(pudval) | _PAGE_DIRTY | _PAGE_ACCESSED) !=3D + (__pte_pgprot(pud.puds[i]) | _PAGE_DIRTY | _PAGE_ACCESSED)) + goto retry; + + if (__pud_leaf(pudval)) + pudval +=3D (1 << (PUD_SHIFT - PAGE_SHIFT)) << + _PAGE_HWPFN_SHIFT; + else if (__pud_present(pudval)) + pudval +=3D 1 << _PAGE_HWPFN_SHIFT; + } + + return pud; +} +#define pudp_get_lockless pudp_get_lockless + +static inline p4d_t p4dp_get_lockless(p4d_t *p4dp) +{ + unsigned long p4dval; + p4d_t p4d; + int i; + +retry: + p4dval =3D READ_ONCE(p4dp->p4ds[0]); + p4d =3D *p4dp; + for (i =3D 0; i < HW_PAGES_PER_PAGE; i++) { + if (__page_val_to_pfn(p4dval) !=3D + __page_val_to_pfn(p4d.p4ds[i])) + goto retry; + if ((__pte_pgprot(p4dval) | _PAGE_DIRTY | _PAGE_ACCESSED) !=3D + (__pte_pgprot(p4d.p4ds[i]) | _PAGE_DIRTY | _PAGE_ACCESSED)) + goto retry; + + if (__p4d_leaf(p4dval)) + p4dval +=3D (1 << (P4D_SHIFT - PAGE_SHIFT)) << + _PAGE_HWPFN_SHIFT; + else if (__p4d_present(p4dval)) + p4dval +=3D 1 << _PAGE_HWPFN_SHIFT; + } + + return p4d; +} +#define p4dp_get_lockless p4dp_get_lockless + +static inline pgd_t pgdp_get_lockless(pgd_t *pgdp) +{ + unsigned long pgdval; + pgd_t pgd; + int i; + +retry: + pgdval =3D READ_ONCE(pgdp->pgds[0]); + pgd =3D *pgdp; + for (i =3D 0; i < HW_PAGES_PER_PAGE; i++) { + if (__page_val_to_pfn(pgdval) !=3D + __page_val_to_pfn(pgd.pgds[i])) + goto retry; + if ((__pte_pgprot(pgdval) | _PAGE_DIRTY | _PAGE_ACCESSED) !=3D + (__pte_pgprot(pgd.pgds[i]) | _PAGE_DIRTY | _PAGE_ACCESSED)) + goto retry; + + if (__pgd_leaf(pgdval)) + pgdval +=3D (1 << (PGDIR_SHIFT - PAGE_SHIFT)) << + _PAGE_HWPFN_SHIFT; + else if (__pgd_present(pgdval)) + pgdval +=3D 1 << _PAGE_HWPFN_SHIFT; + } + + return pgd; +} +#define pgdp_get_lockless pgdp_get_lockless + +#endif /* CONFIG_RISCV_USE_SW_PAGE */ + void flush_icache_pte(struct mm_struct *mm, pte_t pte); =20 static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t p= teval) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e8b2ac6bd2ae..b629c48b980b 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -598,6 +598,27 @@ static inline void pmdp_get_lockless_sync(void) } #endif =20 +#ifndef pudp_get_lockless +static inline pud_t pudp_get_lockless(pud_t *pudp) +{ + return pudp_get(pudp); +} +#endif + +#ifndef p4dp_get_lockless +static inline p4d_t p4dp_get_lockless(p4d_t *p4dp) +{ + return p4dp_get(p4dp); +} +#endif + +#ifndef pgdp_get_lockless +static inline pgd_t pgdp_get_lockless(pgd_t *pgdp) +{ + return pgdp_get(pgdp); +} +#endif + #ifdef CONFIG_TRANSPARENT_HUGEPAGE #ifndef __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, --=20 2.20.1 From nobody Thu Dec 18 08:12:16 2025 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA69E1DB37F for ; Thu, 5 Dec 2024 10:38:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395097; cv=none; b=dmiAOPOOBe34ol4SNAD2a5Fq1aYINoK7bUHzukPyb9amwu6VqfgmY5/etfPgI9pxLFmUsKPyY3J3/GkBg/+t8IbmO49FYI0jv3CKPkWRPMvEWwUdVeCvDyf9Kte1FmXwRQjRwvDqYHY3tpkwEQ6sxtQu+JNVdPicjjcy2IUgg/E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395097; c=relaxed/simple; bh=Xrt6S2p9gVj8iFv//T0E/D73EoMBDaL86AIOToQrt78=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=lS7L60KmU8uJmGwo1y4xymtHM3OGmlA1kaBN8pMr2iUX4HUPMhyuM3Y4mt5rhd+MZdWYICjKC+E5FHm8iZ+jHbwZEtvQvQBTlR7i/45GqXRAaQJALLdv4oiXkIy6KBUbxiEOE7cOgyiXz/OQDqB6iROMoS7AyWDKyaA+wxhKmkA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=fXb1FGFf; arc=none smtp.client-ip=209.85.215.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="fXb1FGFf" Received: by mail-pg1-f171.google.com with SMTP id 41be03b00d2f7-7f46d5d1ad5so627107a12.3 for ; Thu, 05 Dec 2024 02:38:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395095; x=1733999895; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=83UO5ozW5NdISNbxrvGH5KVgxXxH6sNR5OZieItF5/g=; b=fXb1FGFfaKq22NMbAYCJOZZyZbLcjg6IZV/tLLJOFWbY2KtXmglvt/2H7B3GSy6P1l zcxzbDddYy7fK7LRT0fsBHymJ/ucMoFNuIGay6QV+9UnNQsuFjgtS7vUGwtJfrZe1wnn Sy3+pfsDGXrPHMsDo8fq9L7n+BMdKB9dpBh2qDzhOqM2DedEtlfJfGFAJ6Q7m9+0CRhc nxq7dNYLA7OHJZLOI/0kNA4Svh5aiv2GyijmzcFiwb2PYAWMrKvl1Q3pXMOXbqkcZfKA MzgPNLeHy1gvLsKlGhAjDZ378bVacfw4f3j+DxlCjUqJTttWo3NjUsw1pqaEKzAyEBI6 Eaug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395095; x=1733999895; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=83UO5ozW5NdISNbxrvGH5KVgxXxH6sNR5OZieItF5/g=; b=JQ2u5UcXnSOopI6+EkX1tPSSM+Y+j8250FZ/Xzzzvna6WywmmGy6i5IThVzKPQcoL2 ro57wAQ0XmPeFpCjPnpZPg69IBqEOyydRXoK1ZDo/nsfnC40waTlnH7jpkh1n7TC6f/P 9g6uzva65UKDpke5s8GJ9q0f1Ta+G9l/VI694/EEgT5dGlGAl1Ry6RsuFF0NWTqGca0h hheyyVvbhTrqE0wwwq3e6fi8PKx9/D5tGY7glqTd80ZfDjJsjDv6JYIhbQHiBo0zncDz JlS7jidRE8Orj8tRpOHqsbwfINcpylqAi/M+n/R3ecv5mxfQoFG7uyJcU3xpGgE+1Keh fcaw== X-Forwarded-Encrypted: i=1; AJvYcCU2P41aRxDSVhCzyfA7vO5WRRzprUJ6psdpBrBKVhWGRdjtxV5UGXItUDgeyiS+QbNfu7kmzh6xr19YQsU=@vger.kernel.org X-Gm-Message-State: AOJu0YxtB+LHGJzLwIiDfEkeFb8Go2JHjPqI+kUfpiPYsqNjK96MP8iK 2ZVPEawciXWXQ/ziVVihMq3GjnivBmltZbev+QS0fC/O0EpJlL+XHq+cm1pM0m8= X-Gm-Gg: ASbGncs7JC0E5yOzIlsb9dfsvtDLF2zji6TOzifPMNwuMUuuSJEL85P+E8gg6hnq6Py BynBXi2MMx6b439ar+Fmujc/5UIuO7BcIhCSsqYB4o0XLUG7meI/iy+E4Fvdw13uFgDA3uvrRdr lf5voS8SaSxg9QgzbjWL/li2TYKLkUGEqkVAcgJQfw2nvtDx9GlVgD2b1pR/V56oCUzewb0SCXi uiwOvEebiD5Xl/rcUCcHQM4CsECwDwkoWzQskPJfLBbLeUf5rDjYCWrLBWiGYXuOn9ZNxODGukg P/ol3vFuSo9fVY2iDZuCjX9QULyVUjBA X-Google-Smtp-Source: AGHT+IHonPj5or3FCVlbY151Zkh3FREpG17bX8AKYVEQ/Cqc6X6geX5o43T/JV8/DVhtAO5u57D41Q== X-Received: by 2002:a05:6a21:99a2:b0:1d9:c615:944f with SMTP id adf61e73a8af0-1e1653b9f1bmr15516117637.11.1733395095235; Thu, 05 Dec 2024 02:38:15 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.11 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:15 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 09/21] riscv: mm: Replace READ_ONCE with atomic pte get function Date: Thu, 5 Dec 2024 18:37:17 +0800 Message-Id: <20241205103729.14798-10-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" READ_ONCE can not be applied to pte structure with multipling mapping entries. This commit replaces READ_ONCE with atomic pte get function. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable-64.h | 6 +++--- arch/riscv/include/asm/pgtable.h | 21 +++++++++++++-------- arch/riscv/kernel/hibernate.c | 18 +++++++++--------- arch/riscv/mm/pgtable.c | 12 +++++++++--- kernel/events/core.c | 6 +++--- mm/debug_vm_pgtable.c | 4 ++-- mm/gup.c | 10 +++++----- mm/hmm.c | 2 +- mm/mapping_dirty_helpers.c | 2 +- mm/memory.c | 4 ++-- mm/mprotect.c | 2 +- mm/ptdump.c | 8 ++++---- mm/sparse-vmemmap.c | 2 +- mm/vmscan.c | 2 +- 14 files changed, 55 insertions(+), 44 deletions(-) diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/p= gtable-64.h index efcf63667f93..2649cc90b14e 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -242,7 +242,7 @@ static inline int pud_user(pud_t pud) =20 static inline void set_pud(pud_t *pudp, pud_t pud) { - WRITE_ONCE(*pudp, pud); + *pudp =3D pud; } =20 static inline void pud_clear(pud_t *pudp) @@ -318,7 +318,7 @@ static inline unsigned long _pmd_pfn(pmd_t pmd) static inline void set_p4d(p4d_t *p4dp, p4d_t p4d) { if (pgtable_l4_enabled) - WRITE_ONCE(*p4dp, p4d); + *p4dp =3D p4d; else set_pud((pud_t *)p4dp, __pud(p4d_val(p4d))); } @@ -401,7 +401,7 @@ pud_t *pud_offset(p4d_t *p4d, unsigned long address); static inline void set_pgd(pgd_t *pgdp, pgd_t pgd) { if (pgtable_l5_enabled) - WRITE_ONCE(*pgdp, pgd); + *pgdp =3D pgd; else set_p4d((p4d_t *)pgdp, __p4d(pgd_val(pgd))); } diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index fe42afb4441e..bf724d006236 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -289,7 +289,7 @@ static inline bool pmd_leaf(pmd_t pmd) =20 static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) { - WRITE_ONCE(*pmdp, pmd); + *pmdp =3D pmd; } =20 static inline void pmd_clear(pmd_t *pmdp) @@ -713,7 +713,7 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b) */ static inline void set_pte(pte_t *ptep, pte_t pteval) { - WRITE_ONCE(*ptep, pteval); + *ptep =3D pteval; } =20 static inline pte_t ptep_get(pte_t *ptep) @@ -953,10 +953,9 @@ extern int ptep_test_and_clear_young(struct vm_area_st= ruct *vma, unsigned long a static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address, pte_t *ptep) { - pte_t pte =3D __pte(atomic_long_xchg((atomic_long_t *)ptep, 0)); - + pte_t pte =3D ptep_get(ptep); + pte_clear(mm, address, ptep); page_table_check_pte_clear(mm, pte); - return pte; } =20 @@ -964,7 +963,8 @@ static inline pte_t ptep_get_and_clear(struct mm_struct= *mm, static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep) { - atomic_long_and(~(unsigned long)_PAGE_WRITE, (atomic_long_t *)ptep); + pte_t old_pte =3D ptep_get(ptep); + set_pte(ptep, pte_wrprotect(old_pte)); } =20 #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH @@ -1170,8 +1170,9 @@ static inline int pmdp_test_and_clear_young(struct vm= _area_struct *vma, static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long address, pmd_t *pmdp) { - pmd_t pmd =3D __pmd(atomic_long_xchg((atomic_long_t *)pmdp, 0)); + pmd_t pmd =3D pmdp_get(pmdp); =20 + pmd_clear(pmdp); page_table_check_pmd_clear(mm, pmd); =20 return pmd; @@ -1188,8 +1189,12 @@ static inline void pmdp_set_wrprotect(struct mm_stru= ct *mm, static inline pmd_t pmdp_establish(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t pmd) { + pmd_t old_pmd =3D pmdp_get(pmdp); + page_table_check_pmd_set(vma->vm_mm, pmdp, pmd); - return __pmd(atomic_long_xchg((atomic_long_t *)pmdp, pmd_val(pmd))); + set_pmd(pmdp, pmd); + + return old_pmd; } =20 #define pmdp_collapse_flush pmdp_collapse_flush diff --git a/arch/riscv/kernel/hibernate.c b/arch/riscv/kernel/hibernate.c index 155be6b1d32c..5018d38f5280 100644 --- a/arch/riscv/kernel/hibernate.c +++ b/arch/riscv/kernel/hibernate.c @@ -171,7 +171,7 @@ static int temp_pgtable_map_pte(pmd_t *dst_pmdp, pmd_t = *src_pmdp, unsigned long pte_t *src_ptep; pte_t *dst_ptep; =20 - if (pmd_none(READ_ONCE(*dst_pmdp))) { + if (pmd_none(pmdp_get_lockless(dst_pmdp))) { dst_ptep =3D (pte_t *)get_safe_page(GFP_ATOMIC); if (!dst_ptep) return -ENOMEM; @@ -183,7 +183,7 @@ static int temp_pgtable_map_pte(pmd_t *dst_pmdp, pmd_t = *src_pmdp, unsigned long src_ptep =3D pte_offset_kernel(src_pmdp, start); =20 do { - pte_t pte =3D READ_ONCE(*src_ptep); + pte_t pte =3D ptep_get_lockless(src_ptep); =20 if (pte_present(pte)) set_pte(dst_ptep, __pte(pte_val(pte) | pgprot_val(prot))); @@ -200,7 +200,7 @@ static int temp_pgtable_map_pmd(pud_t *dst_pudp, pud_t = *src_pudp, unsigned long pmd_t *src_pmdp; pmd_t *dst_pmdp; =20 - if (pud_none(READ_ONCE(*dst_pudp))) { + if (pud_none(pudp_get_lockless(dst_pudp))) { dst_pmdp =3D (pmd_t *)get_safe_page(GFP_ATOMIC); if (!dst_pmdp) return -ENOMEM; @@ -212,7 +212,7 @@ static int temp_pgtable_map_pmd(pud_t *dst_pudp, pud_t = *src_pudp, unsigned long src_pmdp =3D pmd_offset(src_pudp, start); =20 do { - pmd_t pmd =3D READ_ONCE(*src_pmdp); + pmd_t pmd =3D pmdp_get_lockless(src_pmdp); =20 next =3D pmd_addr_end(start, end); =20 @@ -239,7 +239,7 @@ static int temp_pgtable_map_pud(p4d_t *dst_p4dp, p4d_t = *src_p4dp, unsigned long pud_t *dst_pudp; pud_t *src_pudp; =20 - if (p4d_none(READ_ONCE(*dst_p4dp))) { + if (p4d_none(p4dp_get_lockless(dst_p4dp))) { dst_pudp =3D (pud_t *)get_safe_page(GFP_ATOMIC); if (!dst_pudp) return -ENOMEM; @@ -251,7 +251,7 @@ static int temp_pgtable_map_pud(p4d_t *dst_p4dp, p4d_t = *src_p4dp, unsigned long src_pudp =3D pud_offset(src_p4dp, start); =20 do { - pud_t pud =3D READ_ONCE(*src_pudp); + pud_t pud =3D pudp_get_lockless(src_pudp); =20 next =3D pud_addr_end(start, end); =20 @@ -278,7 +278,7 @@ static int temp_pgtable_map_p4d(pgd_t *dst_pgdp, pgd_t = *src_pgdp, unsigned long p4d_t *dst_p4dp; p4d_t *src_p4dp; =20 - if (pgd_none(READ_ONCE(*dst_pgdp))) { + if (pgd_none(pgdp_get_lockless(dst_pgdp))) { dst_p4dp =3D (p4d_t *)get_safe_page(GFP_ATOMIC); if (!dst_p4dp) return -ENOMEM; @@ -290,7 +290,7 @@ static int temp_pgtable_map_p4d(pgd_t *dst_pgdp, pgd_t = *src_pgdp, unsigned long src_p4dp =3D p4d_offset(src_pgdp, start); =20 do { - p4d_t p4d =3D READ_ONCE(*src_p4dp); + p4d_t p4d =3D p4dp_get_lockless(src_p4dp); =20 next =3D p4d_addr_end(start, end); =20 @@ -317,7 +317,7 @@ static int temp_pgtable_mapping(pgd_t *pgdp, unsigned l= ong start, unsigned long unsigned long ret; =20 do { - pgd_t pgd =3D READ_ONCE(*src_pgdp); + pgd_t pgd =3D pgdp_get_lockless(src_pgdp); =20 next =3D pgd_addr_end(start, end); =20 diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index f57ada26a183..150aea8e2d7a 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -128,9 +128,15 @@ int ptep_test_and_clear_young(struct vm_area_struct *v= ma, unsigned long address, pte_t *ptep) { - if (!pte_young(ptep_get(ptep))) - return 0; - return test_and_clear_bit(_PAGE_ACCESSED_OFFSET, &pte_val(*ptep)); + int r =3D 1; + pte_t pte =3D ptep_get(ptep); + + if (!pte_young(pte)) + r =3D 0; + else + set_pte(ptep, pte_mkold(pte)); + + return r; } EXPORT_SYMBOL_GPL(ptep_test_and_clear_young); =20 diff --git a/kernel/events/core.c b/kernel/events/core.c index df27d08a7232..84d49c60f55b 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -7709,7 +7709,7 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm= , unsigned long addr) pte_t *ptep, pte; =20 pgdp =3D pgd_offset(mm, addr); - pgd =3D READ_ONCE(*pgdp); + pgd =3D pgdp_get_lockless(pgdp); if (pgd_none(pgd)) return 0; =20 @@ -7717,7 +7717,7 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm= , unsigned long addr) return pgd_leaf_size(pgd); =20 p4dp =3D p4d_offset_lockless(pgdp, pgd, addr); - p4d =3D READ_ONCE(*p4dp); + p4d =3D p4dp_get_lockless(p4dp); if (!p4d_present(p4d)) return 0; =20 @@ -7725,7 +7725,7 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm= , unsigned long addr) return p4d_leaf_size(p4d); =20 pudp =3D pud_offset_lockless(p4dp, p4d, addr); - pud =3D READ_ONCE(*pudp); + pud =3D pudp_get_lockless(pudp); if (!pud_present(pud)) return 0; =20 diff --git a/mm/debug_vm_pgtable.c b/mm/debug_vm_pgtable.c index bc748f700a9e..1cec548cc6c7 100644 --- a/mm/debug_vm_pgtable.c +++ b/mm/debug_vm_pgtable.c @@ -438,7 +438,7 @@ static void __init pmd_huge_tests(struct pgtable_debug_= args *args) * X86 defined pmd_set_huge() verifies that the given * PMD is not a populated non-leaf entry. */ - WRITE_ONCE(*args->pmdp, __pmd(0)); + set_pmd(args->pmdp, __pmd(0)); WARN_ON(!pmd_set_huge(args->pmdp, __pfn_to_phys(args->fixed_pmd_pfn), arg= s->page_prot)); WARN_ON(!pmd_clear_huge(args->pmdp)); pmd =3D pmdp_get(args->pmdp); @@ -458,7 +458,7 @@ static void __init pud_huge_tests(struct pgtable_debug_= args *args) * X86 defined pud_set_huge() verifies that the given * PUD is not a populated non-leaf entry. */ - WRITE_ONCE(*args->pudp, __pud(0)); + set_pud(args->pudp, __pud(0)); WARN_ON(!pud_set_huge(args->pudp, __pfn_to_phys(args->fixed_pud_pfn), arg= s->page_prot)); WARN_ON(!pud_clear_huge(args->pudp)); pud =3D pudp_get(args->pudp); diff --git a/mm/gup.c b/mm/gup.c index ad0c8922dac3..db444d732028 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1004,7 +1004,7 @@ static struct page *follow_pud_mask(struct vm_area_st= ruct *vma, struct mm_struct *mm =3D vma->vm_mm; =20 pudp =3D pud_offset(p4dp, address); - pud =3D READ_ONCE(*pudp); + pud =3D pudp_get_lockless(pudp); if (!pud_present(pud)) return no_page_table(vma, flags, address); if (pud_leaf(pud)) { @@ -1029,7 +1029,7 @@ static struct page *follow_p4d_mask(struct vm_area_st= ruct *vma, p4d_t *p4dp, p4d; =20 p4dp =3D p4d_offset(pgdp, address); - p4d =3D READ_ONCE(*p4dp); + p4d =3D p4dp_get_lockless(p4dp); BUILD_BUG_ON(p4d_leaf(p4d)); =20 if (!p4d_present(p4d) || p4d_bad(p4d)) @@ -3259,7 +3259,7 @@ static int gup_fast_pud_range(p4d_t *p4dp, p4d_t p4d,= unsigned long addr, =20 pudp =3D pud_offset_lockless(p4dp, p4d, addr); do { - pud_t pud =3D READ_ONCE(*pudp); + pud_t pud =3D pudp_get_lockless(pudp); =20 next =3D pud_addr_end(addr, end); if (unlikely(!pud_present(pud))) @@ -3285,7 +3285,7 @@ static int gup_fast_p4d_range(pgd_t *pgdp, pgd_t pgd,= unsigned long addr, =20 p4dp =3D p4d_offset_lockless(pgdp, pgd, addr); do { - p4d_t p4d =3D READ_ONCE(*p4dp); + p4d_t p4d =3D p4dp_get_lockless(p4dp); =20 next =3D p4d_addr_end(addr, end); if (!p4d_present(p4d)) @@ -3307,7 +3307,7 @@ static void gup_fast_pgd_range(unsigned long addr, un= signed long end, =20 pgdp =3D pgd_offset(current->mm, addr); do { - pgd_t pgd =3D READ_ONCE(*pgdp); + pgd_t pgd =3D pgdp_get_lockless(pgdp); =20 next =3D pgd_addr_end(addr, end); if (pgd_none(pgd)) diff --git a/mm/hmm.c b/mm/hmm.c index 7e0229ae4a5a..fa56b735883e 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -423,7 +423,7 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long = start, unsigned long end, /* Normally we don't want to split the huge page */ walk->action =3D ACTION_CONTINUE; =20 - pud =3D READ_ONCE(*pudp); + pud =3D pudp_get_lockless(pudp); if (!pud_present(pud)) { spin_unlock(ptl); return hmm_vma_walk_hole(start, end, -1, walk); diff --git a/mm/mapping_dirty_helpers.c b/mm/mapping_dirty_helpers.c index 2f8829b3541a..8771432c3300 100644 --- a/mm/mapping_dirty_helpers.c +++ b/mm/mapping_dirty_helpers.c @@ -149,7 +149,7 @@ static int wp_clean_pud_entry(pud_t *pud, unsigned long= addr, unsigned long end, struct mm_walk *walk) { #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD - pud_t pudval =3D READ_ONCE(*pud); + pud_t pudval =3D pudp_get_lockless(pud); =20 /* Do not split a huge pud */ if (pud_trans_huge(pudval) || pud_devmap(pudval)) { diff --git a/mm/memory.c b/mm/memory.c index bdf77a3ec47b..03ee104cb009 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6428,12 +6428,12 @@ int follow_pfnmap_start(struct follow_pfnmap_args *= args) goto out; =20 p4dp =3D p4d_offset(pgdp, address); - p4d =3D READ_ONCE(*p4dp); + p4d =3D p4dp_get_lockless(p4dp); if (p4d_none(p4d) || unlikely(p4d_bad(p4d))) goto out; =20 pudp =3D pud_offset(p4dp, address); - pud =3D READ_ONCE(*pudp); + pud =3D pudp_get_lockless(pudp); if (pud_none(pud)) goto out; if (pud_leaf(pud)) { diff --git a/mm/mprotect.c b/mm/mprotect.c index 6f450af3252e..a165ab597a73 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -447,7 +447,7 @@ static inline long change_pud_range(struct mmu_gather *= tlb, break; } =20 - pud =3D READ_ONCE(*pudp); + pud =3D pudp_get_lockless(pudp); if (pud_none(pud)) continue; =20 diff --git a/mm/ptdump.c b/mm/ptdump.c index 106e1d66e9f9..b8a2ad43392f 100644 --- a/mm/ptdump.c +++ b/mm/ptdump.c @@ -30,7 +30,7 @@ static int ptdump_pgd_entry(pgd_t *pgd, unsigned long add= r, unsigned long next, struct mm_walk *walk) { struct ptdump_state *st =3D walk->private; - pgd_t val =3D READ_ONCE(*pgd); + pgd_t val =3D pgdp_get_lockless(pgd); =20 #if CONFIG_PGTABLE_LEVELS > 4 && \ (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) @@ -53,7 +53,7 @@ static int ptdump_p4d_entry(p4d_t *p4d, unsigned long add= r, unsigned long next, struct mm_walk *walk) { struct ptdump_state *st =3D walk->private; - p4d_t val =3D READ_ONCE(*p4d); + p4d_t val =3D p4dp_get_lockless(p4d); =20 #if CONFIG_PGTABLE_LEVELS > 3 && \ (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) @@ -76,7 +76,7 @@ static int ptdump_pud_entry(pud_t *pud, unsigned long add= r, unsigned long next, struct mm_walk *walk) { struct ptdump_state *st =3D walk->private; - pud_t val =3D READ_ONCE(*pud); + pud_t val =3D pudp_get_lockless(pud); =20 #if CONFIG_PGTABLE_LEVELS > 2 && \ (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) @@ -99,7 +99,7 @@ static int ptdump_pmd_entry(pmd_t *pmd, unsigned long add= r, unsigned long next, struct mm_walk *walk) { struct ptdump_state *st =3D walk->private; - pmd_t val =3D READ_ONCE(*pmd); + pmd_t val =3D pmdp_get_lockless(pmd); =20 #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) if (pmd_page(val) =3D=3D virt_to_page(lm_alias(kasan_early_shadow_pte))) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index c0388b2e959d..6621fb096fd0 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -337,7 +337,7 @@ int __meminit vmemmap_populate_hugepages(unsigned long = start, unsigned long end, return -ENOMEM; =20 pmd =3D pmd_offset(pud, addr); - if (pmd_none(READ_ONCE(*pmd))) { + if (pmd_none(pmdp_get_lockless(pmd))) { void *p; =20 p =3D vmemmap_alloc_block_buf(PMD_SIZE, node, altmap); diff --git a/mm/vmscan.c b/mm/vmscan.c index 28ba2b06fc7d..2bc78c339fd1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3608,7 +3608,7 @@ static int walk_pud_range(p4d_t *p4d, unsigned long s= tart, unsigned long end, pud =3D pud_offset(p4d, start & P4D_MASK); restart: for (i =3D pud_index(start), addr =3D start; addr !=3D end; i++, addr =3D= next) { - pud_t val =3D READ_ONCE(pud[i]); + pud_t val =3D pudp_get_lockless(&pud[i]); =20 next =3D pud_addr_end(addr, end); =20 --=20 2.20.1 From nobody Thu Dec 18 08:12:16 2025 Received: from mail-ot1-f51.google.com (mail-ot1-f51.google.com [209.85.210.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C7581DFE24 for ; Thu, 5 Dec 2024 10:38:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395101; cv=none; b=JIDnf1VdZYkP/4kbfPlbmTC+/qlklxJYluwcv9sbc/JCn2KsBAVL0fPCnhWx0rkf+VlXebx9iMFaWjvYtRH0rjOte01I4GTFdGMTOZJ0rObtdkKH6b9j20Jlz8bFP71aODDZHNgCIWaAyAtqN3O3zi/+Wej0PMRKt+/llyjhXHU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395101; c=relaxed/simple; bh=HyAJKSANO0n/ESTj5vAArdGzeHYyl5LydIiFKhzGbHE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gDNNUFbV47lDTcwnVtKyFxHbzGuppvyzKpfvuDMHt3HrmcSoTBKjNNrDFplrzUNC7urGKRxxlixB/yRJxaD7Cqu7EhqtAKUfNzrTBdC5cszUKEOPj5CUKfunvg2Tus6o7HtFMNZsm+w5orj/l9ynbNgff5y8BtgGrZZV5ynxBQE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=F7RsT8IU; arc=none smtp.client-ip=209.85.210.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="F7RsT8IU" Received: by mail-ot1-f51.google.com with SMTP id 46e09a7af769-71d54fc1944so432801a34.1 for ; Thu, 05 Dec 2024 02:38:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395099; x=1733999899; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hfWeT6m9rP2RvyJAYvuGAOSzcqpyDy27Avf8NzKYXNE=; b=F7RsT8IUo9hLeaDvF+0B8CgySYbOTwJB51dC3x1zW1iHJIE3EJoglS2KPXnNyHPyrz ROHvFOzZGbTVvirbGW0Nh/kjJJPlj2ZhmvO527AftzLVbLMudLm2RWEPOdvpbBBF5lGx MDsLp+2ZnlALpsfw3WR/OtM6C38+dRQ/oXK8SFcndln4oxItUI/tN1bavSdwLbTcmrn/ rsxFxPnQeH9ZCWygftsca94b7dXAWkh4YMviz4d22dDQI42i0ofNy79IZNEFvWh4/Q9D EAhJOazVYKB9IB9XrFAyvDIlak39Rib/Y706INxeAbj1Kov7sYglpKp4x2gSrjQI4HAs IpUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395099; x=1733999899; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hfWeT6m9rP2RvyJAYvuGAOSzcqpyDy27Avf8NzKYXNE=; b=T2W1NaD1fXJD2kY8uaURQP3Afmoe2XyeewJhUuJHjyC9jz/+EJupLA8cA5pAcvxKQO t2amc1czn1wqPn4idr3SVgHuSom5n1sKaumBcf2v/w0dcwKxgvQ9Ei288i/VRt3buKjU a4iow9l9oNzjiUI7al4U5+RIiipKjOXCw6TXgOHMJA24jwVX8FQe1IkOKeJINdRWSj3P 61pwSqPe9lluZsdEXZVRlfBybC0mtoVLvBzmq4RU80PijEVBtQKUfCBn/rm3TCJJqzF4 DoKu2x7wUvAwMUwZgagcZ5J6zoSG4o/10yUfXu18jAgBqYmgPwKpityRiu1lLHop4Uss yHmw== X-Forwarded-Encrypted: i=1; AJvYcCXIcKUDKVBU2p0iyvCAiyinOeDRw935t/O3IwLEkrum9Jnm/SqfBlG1Ft4qZChJt0fV9tO12POy8rULXaw=@vger.kernel.org X-Gm-Message-State: AOJu0Yw82GP5Dybci28owZaxPWaP4NcD9Rcqk2KKMzonyTcNCsqpRuSs bFrIO0vxqRKiqDz/n9YHBugObQVtmXtO0GTX8fJRpZCCPg9jln0+NeupDUncFvw= X-Gm-Gg: ASbGncvWCKzhwRHfq8AidzhHxVz/V3R2ftx4Lf6/o7Guao79RAtewWY26wIObQn9Cfx nUfwLVZBv5fYCz4z3zmvvejl3/omtxFQ/MDfcJ/OeeQgVKYA2NgCnmLx1iPYe9w0isVICtZvhMg bWevj3QFPnv2fP3frodCAITZwVUYDIvN45E6JH1gCQS8KP8z+ARwEmRzDugM+VV5gizNYE1F1sk fhGVRfL8qOPoEhczhQuK+FK3qGE+Mmf5oX5BLk/UrysgCx9AmwaNvjfZnZRMpzkU/GiKs9DugZH 8Aa6NeKkqTnFF6jaQAFJoxDuEXRaaIJk X-Google-Smtp-Source: AGHT+IHmo8dAIPLaAwW/lT+NI/PJnWblM6aaHV5WtIj+KNZJRyA2qtNhsMIENCAwyOOd5jQe2iKEOQ== X-Received: by 2002:a05:6830:2aa8:b0:71d:4488:19d9 with SMTP id 46e09a7af769-71dad652c1fmr12149732a34.17.1733395099595; Thu, 05 Dec 2024 02:38:19 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.15 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:19 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 10/21] riscv: mm: Reimplement PTE A/D bit check function Date: Thu, 5 Dec 2024 18:37:18 +0800 Message-Id: <20241205103729.14798-11-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" CPU that supports only 4K MMU usually updates access/dirty bit at 4K pte level. As each software page can contains multiple 4K hardware pages, we ne= ed to traverse all mapping entries to check whether corresponding 4K page is accessed or dirty during pte_dirty/pte_access functions. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index bf724d006236..c0f7442c8a9e 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -553,6 +553,29 @@ static inline int pte_huge(pte_t pte) return pte_present(pte) && (pte_val(pte) & _PAGE_LEAF); } =20 +#ifdef CONFIG_RISCV_USE_SW_PAGE +static inline int pte_dirty(pte_t pte) +{ + unsigned int i; + + for (i =3D 0; i < HW_PAGES_PER_PAGE; i++) + if (pte.ptes[i] & _PAGE_DIRTY) + return 1; + + return 0; +} + +static inline int pte_young(pte_t pte) +{ + unsigned int i; + + for (i =3D 0; i < HW_PAGES_PER_PAGE; i++) + if (pte.ptes[i] & _PAGE_ACCESSED) + return 1; + + return 0; +} +#else static inline int pte_dirty(pte_t pte) { return pte_val(pte) & _PAGE_DIRTY; @@ -562,6 +585,7 @@ static inline int pte_young(pte_t pte) { return pte_val(pte) & _PAGE_ACCESSED; } +#endif /* CONFIG_RISCV_USE_SW_PAGE */ =20 static inline int pte_special(pte_t pte) { --=20 2.20.1 From nobody Thu Dec 18 08:12:16 2025 Received: from mail-oa1-f45.google.com (mail-oa1-f45.google.com [209.85.160.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A62481F429B for ; Thu, 5 Dec 2024 10:38:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395106; cv=none; b=tgD14WWMK9vTPkeNH5X/wj+dW8wo+GFTdUfwCE5OvObu/YcQ2Ref6m9LtyBGaiRXtzMHevkMz5wvRt/bPTiZyzVzjHRPit+nZGfJLuij/y9q6CHQneMQnOAKzi0n2vl9omvYhvGVnGpmv1kfSvBCIOBo0Xoj96TeD6v0U1R3Y/0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395106; c=relaxed/simple; bh=gGgNHGAQnglp1EwvvnTVsa9Hle1YBo1yufwyIxTF2p8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=GdCL2c89J/CqbVurJ64j1qtRP8Dw7f8Rg/0crYqM8uP7PbzpclDm24FD4tssVjRTk7F2uqI5FLt2bhqTxUDSd5XQ1/b79N9xH9KQYiFNaXC72ipvw4+eWSMsKGgGlOtqjspUWY4qPFEz3U2h34lWa61EXHSVc0+6cAad8/cq+8Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=DXLh/qBo; arc=none smtp.client-ip=209.85.160.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="DXLh/qBo" Received: by mail-oa1-f45.google.com with SMTP id 586e51a60fabf-29e2a50586dso494572fac.0 for ; Thu, 05 Dec 2024 02:38:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395104; x=1733999904; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XIMWQ57ZznUQuD/aiNRYDz3W26VtFUEaOqg8e4FokQk=; b=DXLh/qBoWmGDI+SaL7Am02hvd+ZaDvBZUsJI9APctF4J8z3/9S6NBrIBj7RtZktLBi +FD4BFS891G5VODXJqFMrOSXbOQAMezUClFXvYpA05bjDtXwRuWtYU6iBWO/qe3YM9oR Vk7XWKKfuUy2Icikj78OXZoUkpGKwBsXQT+jq1QopN6yR8Xz44m0D7fu+K2mhRste9aN RmZe77KFtORid+0U9s9Udp8nBPhx+GXYCRBWEIR4o9x/UqrXGpzfjmwVvNcp9tBVa47F 1jf0e5oucD6BojArdtA9663v0AdqRhwV5ZMU9jjWw0GWnvhy1xj4UMOErxBA1TRXy8dM BJ5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395104; x=1733999904; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XIMWQ57ZznUQuD/aiNRYDz3W26VtFUEaOqg8e4FokQk=; b=cuX3xQAmZEoiUwRyhGtWxKbhYxk8kPE1gD4c6Ooiyn9yu8aPzp5PGxc1xm8UA/khVr 9XQMqt3YuNbv8a/Tk8o3a7buZ29d/DN3bnsL9h24x2ySzg/t/UcgSJeR+9iJ42BD6tyo 25ayqiaVnjqcZ+Y7U+XMWfStkXE0tZHw+xIm/Kztq9J1u58gaJK3zYNc4uAxnv71EWix q1zFk+WnSyfJ/0vhZEVkgBukYKKdF70Sn4mO1o4pkEpJrAw35a7qYtVSrixN2061TbkJ jI8HWwtdTo3gQEhAd797hGZrggOiKcMkPHhEsyLcmFzluSKXfybFamRgdr2900uZmnuY o0ZA== X-Forwarded-Encrypted: i=1; AJvYcCVN6Ow67FOzHFteY3HmR12URk8nmtfUuXCuck5+Gv9xEF7ki1ZF2+bTnXrM8AFmkfcAO7z4ao7Ib0HXCUI=@vger.kernel.org X-Gm-Message-State: AOJu0YzA+R6AsGYbeInT00Z3RzEaBOXvJJOzIG6M9qp0bur6+hp9zdi6 seoAfGxhnAHdDPhkKoXebD0tz0s+E6PzGLiJ7zjPo0alK9zrQYoCGgYnL5a9ZRk= X-Gm-Gg: ASbGnctolwQMzr+KLW/5OS+tGTPMnAkH1LLMY2yyCABPsjuHmS0MoBzPE4P9Xa7AEcm uVj3x3UaUOn3wR1ED5BUjl6kQt3RfOFOidPRvzlOPwShCCywAxzstBOznfM8Z1c9ts/Kseepq9m HN2OknsSjhBrFUEjTdxhjA/WFgJF05hyKowffamKqbxWNEFLpTpMr5M7x1KKiMdrUAnoGrlNHpd IOooKKPd3KbujjP08cYMDfTctPpvO0iJuNcPBXCtUdUHyW6WOOU9b2vGplRmyIG1rwkmn0OX1ov 1BjPF3yld8EKvm6r37pv78dOHgi04lnB X-Google-Smtp-Source: AGHT+IFGD682lN2KXS9mZyYiAgPgUddfTIrz2tftbpWLW1OwVDgKSjli0UbK/dW/c1XvobnxdhptFQ== X-Received: by 2002:a05:6870:4944:b0:29d:c5e8:e41f with SMTP id 586e51a60fabf-29e88576409mr11051212fac.5.1733395103681; Thu, 05 Dec 2024 02:38:23 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.19 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:23 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 11/21] riscv: mm: Reimplement mk_huge_pte function Date: Thu, 5 Dec 2024 18:37:19 +0800 Message-Id: <20241205103729.14798-12-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Huge pte can be pud, pmd, or svnapot pte. Huge ptes from different page table levels have different pte constructors. This commit reimplements mk_huge_pte function. We take vma struct as argument to check the target huge pte level and applying corresponding constructor. Signed-off-by: Xu Lu --- arch/riscv/include/asm/hugetlb.h | 5 +++++ arch/riscv/mm/hugetlbpage.c | 23 ++++++++++++++++++++++- arch/s390/include/asm/hugetlb.h | 2 +- include/asm-generic/hugetlb.h | 5 ++++- mm/debug_vm_pgtable.c | 2 +- mm/hugetlb.c | 4 ++-- 6 files changed, 35 insertions(+), 6 deletions(-) diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/huge= tlb.h index faf3624d8057..eafd00f4b74f 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -51,6 +51,11 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift= , vm_flags_t flags); =20 #endif /*CONFIG_RISCV_ISA_SVNAPOT*/ =20 +#ifdef CONFIG_RISCV_USE_SW_PAGE +#define __HAVE_ARCH_MK_HUGE_PTE +pte_t mk_huge_pte(struct vm_area_struct *vma, struct page *page, pgprot_t = pgprot); +#endif + #include =20 #endif /* _ASM_RISCV_HUGETLB_H */ diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 42314f093922..8896c28ec881 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -2,6 +2,27 @@ #include #include =20 +#ifdef CONFIG_RISCV_USE_SW_PAGE +pte_t mk_huge_pte(struct vm_area_struct *vma, struct page *page, pgprot_t = pgprot) +{ + pte_t pte; + unsigned int shift =3D huge_page_shift(hstate_vma(vma)); + + if (shift =3D=3D PGDIR_SHIFT) + pte =3D pgd_pte(pfn_pgd(page_to_pfn(page), pgprot)); + else if (shift =3D=3D P4D_SHIFT) + pte =3D p4d_pte(pfn_p4d(page_to_pfn(page), pgprot)); + else if (shift =3D=3D PUD_SHIFT) + pte =3D pud_pte(pfn_pud(page_to_pfn(page), pgprot)); + else if (shift =3D=3D PMD_SHIFT) + pte =3D pmd_pte(pfn_pmd(page_to_pfn(page), pgprot)); + else + pte =3D pfn_pte(page_to_pfn(page), pgprot); + + return pte; +} +#endif /* CONFIG_RISCV_USE_SW_PAGE */ + #ifdef CONFIG_RISCV_ISA_SVNAPOT pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { @@ -74,7 +95,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, =20 out: if (pte) { - pte_t pteval =3D ptep_get_lockless(pte); + pte_t pteval =3D ptep_get(pte); =20 WARN_ON_ONCE(pte_present(pteval) && !pte_huge(pteval)); } diff --git a/arch/s390/include/asm/hugetlb.h b/arch/s390/include/asm/hugetl= b.h index cf1b5d6fb1a6..cea9118d4bba 100644 --- a/arch/s390/include/asm/hugetlb.h +++ b/arch/s390/include/asm/hugetlb.h @@ -79,7 +79,7 @@ static inline void huge_ptep_set_wrprotect(struct mm_stru= ct *mm, __set_huge_pte_at(mm, addr, ptep, pte_wrprotect(pte)); } =20 -static inline pte_t mk_huge_pte(struct page *page, pgprot_t pgprot) +static inline pte_t mk_huge_pte(struct vm_area_struct *vma, struct page *p= age, pgprot_t pgprot) { return mk_pte(page, pgprot); } diff --git a/include/asm-generic/hugetlb.h b/include/asm-generic/hugetlb.h index 594d5905f615..90765bc03bba 100644 --- a/include/asm-generic/hugetlb.h +++ b/include/asm-generic/hugetlb.h @@ -5,10 +5,13 @@ #include #include =20 -static inline pte_t mk_huge_pte(struct page *page, pgprot_t pgprot) +#ifndef __HAVE_ARCH_MK_HUGE_PTE +static inline pte_t mk_huge_pte(struct vm_area_struct *vma, struct page *p= age, + pgprot_t pgprot) { return mk_pte(page, pgprot); } +#endif =20 static inline unsigned long huge_pte_write(pte_t pte) { diff --git a/mm/debug_vm_pgtable.c b/mm/debug_vm_pgtable.c index 1cec548cc6c7..24839883d513 100644 --- a/mm/debug_vm_pgtable.c +++ b/mm/debug_vm_pgtable.c @@ -919,7 +919,7 @@ static void __init hugetlb_basic_tests(struct pgtable_d= ebug_args *args) * as it was previously derived from a real kernel symbol. */ page =3D pfn_to_page(args->fixed_pmd_pfn); - pte =3D mk_huge_pte(page, args->page_prot); + pte =3D mk_huge_pte(args->vma, page, args->page_prot); =20 WARN_ON(!huge_pte_dirty(huge_pte_mkdirty(pte))); WARN_ON(!huge_pte_write(huge_pte_mkwrite(huge_pte_wrprotect(pte)))); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 190fa05635f4..2b33eb46408f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5140,10 +5140,10 @@ static pte_t make_huge_pte(struct vm_area_struct *v= ma, struct page *page, unsigned int shift =3D huge_page_shift(hstate_vma(vma)); =20 if (writable) { - entry =3D huge_pte_mkwrite(huge_pte_mkdirty(mk_huge_pte(page, + entry =3D huge_pte_mkwrite(huge_pte_mkdirty(mk_huge_pte(vma, page, vma->vm_page_prot))); } else { - entry =3D huge_pte_wrprotect(mk_huge_pte(page, + entry =3D huge_pte_wrprotect(mk_huge_pte(vma, page, vma->vm_page_prot)); } entry =3D pte_mkyoung(entry); --=20 2.20.1 From nobody Thu Dec 18 08:12:16 2025 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 559B21DB951 for ; Thu, 5 Dec 2024 10:38:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395110; cv=none; b=PDtOJpU19ONHPjI/iKqod7+tOVvAohEj4NUcPKFL2g98y3XCpYCdUbLA6SZ5BbaMXPBkara7v1R13W2ogxJCsHMzOLiBI4S8YCWCcoErA2+8tCXBaqfYAjuKpG9ALyqjiL5oZYuxaagt9n9An5PuokNpykO1cKKaS8UoUebcjCk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395110; c=relaxed/simple; bh=iaiXED/tqibRXcFpLigD3NvJz1CulAasWjUYazfFtPc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=fzrfH+FfmlHQP5Kgg/tQCZtVQuQF5e54DPZuVUPVBaGYv0XoU2Pd7YbXinrNEJB0PD1cUO4ed8sHgauEthpOkW/B5fgSgV3HKMdDzizTqTpefosLZ4WFKLpBzkd3TEAzuNxeUkb+d8flw+xa9ntCuBctRrixzAF9X+57ztyB6SY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=aTWOpJLQ; arc=none smtp.client-ip=209.85.215.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="aTWOpJLQ" Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-7fcfac22a30so832252a12.3 for ; Thu, 05 Dec 2024 02:38:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395108; x=1733999908; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HFQIXohFMC+aCLCj7cIVe0JD4G758F8Hwu3svQYABEY=; b=aTWOpJLQpSF75/OhWUsAWdmRJmhklV2DlaxSg/3IT7WwdLbkTHdh3/zGg+CL+tyQjn xY0pHJZBTS802R0y3Q78XglE5c8HjkTQaNP/rzOomPcs7q+PbLn4zYuuRQGs9hTRGLam jKLAYBxNNICIvWdLQRGq7XIihkVpNCvYdKLmMFKHguHJBmxJAo5B5BOBi7QrWzbH6b1k fh05d+UEDyDlpbowKIdZCYH37rmYwJ3wqbLQez6DlR2xWiqwct8RGA1r7alBQ/DjuUJ6 su+/eyaAMo6Py+M4jFYz/IBKSo/lsZ95pRcEWmKLai40P64EE9BNVSsrH4o/sHt4jNB5 N8GQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395108; x=1733999908; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HFQIXohFMC+aCLCj7cIVe0JD4G758F8Hwu3svQYABEY=; b=mBmzc93QZrMVlEe1TqqP4ic3OIjYiUZlAYInUCTETpuImheJTLPFwa2e0Y3BNlfDvZ fXMlzhcWaXNLekh4pnCMJvQXfMKqDMHbxMMpkZHhZljTb53fO6F6z+0wGpPf1J/V8zkz SdB8dzIBCQGxQqt9Vdr0fBTW1MyaHf4hlCwvem19zTZNz5NrZTQroQWeyWYjGnr862tA GHRZ9DCRLVaq40VkWtJJ82v2VXll2G4wgYuZIFPaD11HRsbFRw/7H9KjnlFZtN0yGQaQ eUMlUzyRGJs9CV6loCYjpxM7MhHSJn6F4EGiH/1dNS6WG60L5QfH985xhlcllDgIA8hm 2Ujg== X-Forwarded-Encrypted: i=1; AJvYcCUbo0oTx8M/PL10N7no69rjCgtsKdNF/ZrjwWoLKPcWycEtnhFuVX7fMdwXO8hPabczeC2HXcIxGveIPP4=@vger.kernel.org X-Gm-Message-State: AOJu0YzE0Ms6EZg8t9W2fXQXiWyg53H34YHGIDCV0vIkolRzMKjQUpN/ sxmGXKHBsEFFykwAqSYndflQ0LMLoxqSv8yAYczoZxeEt+UzK4Omc0iofarTG2Y= X-Gm-Gg: ASbGncsdfhXeqI3OBcH8vYzjQxwg+OGe8GX4p21V/sMLQRMk3JolY4RYpy22F+3uXnT 8JZxaq0mf3+x993VNdVvpRd6w5mt3a4+4jqXOM1b0XNxGRXdeBEn9hQ2djzWts6KCqWlRuIYpfY OpGz0zsDob9XkAGUChViW1RHHC0s0sdJQR84bS1iQKJXkRa61MU1x6Bnp6aWy/7mUi28K6a1tlt IiGr4lnrdTvgt1QEG/p3FYgN0Jp5gsvJYs7Ns4wOp5twZe9mjFNZ1IPIyqJ+oRypiiXPd1YvBH3 F/yvdHSFgn7HQOtMBeVvDIodEKey8xmY X-Google-Smtp-Source: AGHT+IEVSD+tgAqJx2WiLXY6iHldH8R0b3LHGoJ+robZaZ03Gbz5dxvb2ZG0GtuQ60x2xrNdVJqFOw== X-Received: by 2002:a05:6300:4041:b0:1e1:1659:82a4 with SMTP id adf61e73a8af0-1e16541341fmr16353259637.41.1733395107751; Thu, 05 Dec 2024 02:38:27 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.24 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:27 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 12/21] riscv: mm: Reimplement tlb flush function Date: Thu, 5 Dec 2024 18:37:20 +0800 Message-Id: <20241205103729.14798-13-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When tlb flushing a page correponding to a certain address, CPU actually only flushes tlb entries of the first 4K hardware page. This commit reimplements tlb flushing function to flush all tlb entries of hardware pag= es in the same software page. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 9 ++++++--- arch/riscv/include/asm/tlbflush.h | 26 ++++++++++++++++++++------ arch/riscv/mm/fault.c | 13 +++++++++---- arch/riscv/mm/init.c | 2 +- arch/riscv/mm/tlbflush.c | 31 +++++++++++++++++++++---------- 5 files changed, 57 insertions(+), 24 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index c0f7442c8a9e..9fa16c0c20aa 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -701,7 +701,7 @@ static inline void update_mmu_cache_range(struct vm_fau= lt *vmf, * the extra traps reduce performance. So, eagerly SFENCE.VMA. */ while (nr--) - local_flush_tlb_page(address + nr * PAGE_SIZE); + local_flush_tlb_page(address + nr * PAGE_SIZE, PAGE_SIZE); =20 svvptc:; /* @@ -719,9 +719,12 @@ svvptc:; static inline void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { - pte_t *ptep =3D (pte_t *)pmdp; + asm goto(ALTERNATIVE("nop", "j %l[svvptc]", 0, RISCV_ISA_EXT_SVVPTC, 1) + : : : : svvptc); =20 - update_mmu_cache(vma, address, ptep); + local_flush_tlb_page(address, PMD_SIZE); + +svvptc:; } =20 #define __HAVE_ARCH_PTE_SAME diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index 72e559934952..25cc39ab84d5 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -29,18 +29,32 @@ static inline void local_flush_tlb_all_asid(unsigned lo= ng asid) } =20 /* Flush one page from local TLB */ -static inline void local_flush_tlb_page(unsigned long addr) +static inline void local_flush_tlb_page(unsigned long addr, + unsigned long page_size) { - ALT_SFENCE_VMA_ADDR(addr); + unsigned int i; + unsigned long hw_page_num =3D 1 << (PAGE_SHIFT - HW_PAGE_SHIFT); + unsigned long hw_page_size =3D page_size >> (PAGE_SHIFT - HW_PAGE_SHIFT); + + for (i =3D 0; i < hw_page_num; i++, addr +=3D hw_page_size) + ALT_SFENCE_VMA_ADDR(addr); } =20 static inline void local_flush_tlb_page_asid(unsigned long addr, + unsigned long page_size, unsigned long asid) { - if (asid !=3D FLUSH_TLB_NO_ASID) - ALT_SFENCE_VMA_ADDR_ASID(addr, asid); - else - local_flush_tlb_page(addr); + unsigned int i; + unsigned long hw_page_num, hw_page_size; + + if (asid !=3D FLUSH_TLB_NO_ASID) { + hw_page_num =3D 1 << (PAGE_SHIFT - HW_PAGE_SHIFT); + hw_page_size =3D page_size >> (PAGE_SHIFT - HW_PAGE_SHIFT); + + for (i =3D 0; i < hw_page_num; i++, addr +=3D hw_page_size) + ALT_SFENCE_VMA_ADDR_ASID(addr, asid); + } else + local_flush_tlb_page(addr, page_size); } =20 void flush_tlb_all(void); diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c index 4772152be0f9..94524e5adc0b 100644 --- a/arch/riscv/mm/fault.c +++ b/arch/riscv/mm/fault.c @@ -118,7 +118,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, = int code, unsigned long a pmd_t *pmd_k; pte_t *pte_k; int index; - unsigned long pfn; + unsigned long pfn, page_size; =20 /* User mode accesses just cause a SIGSEGV */ if (user_mode(regs)) @@ -154,8 +154,10 @@ static inline void vmalloc_fault(struct pt_regs *regs,= int code, unsigned long a no_context(regs, addr); return; } - if (pud_leaf(pudp_get(pud_k))) + if (pud_leaf(pudp_get(pud_k))) { + page_size =3D PUD_SIZE; goto flush_tlb; + } =20 /* * Since the vmalloc area is global, it is unnecessary @@ -166,8 +168,10 @@ static inline void vmalloc_fault(struct pt_regs *regs,= int code, unsigned long a no_context(regs, addr); return; } - if (pmd_leaf(pmdp_get(pmd_k))) + if (pmd_leaf(pmdp_get(pmd_k))) { + page_size =3D PMD_SIZE; goto flush_tlb; + } =20 /* * Make sure the actual PTE exists as well to @@ -180,6 +184,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, = int code, unsigned long a no_context(regs, addr); return; } + page_size =3D PAGE_SIZE; =20 /* * The kernel assumes that TLBs don't cache invalid @@ -188,7 +193,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, = int code, unsigned long a * necessary even after writing invalid entries. */ flush_tlb: - local_flush_tlb_page(addr); + local_flush_tlb_page(addr, page_size); } =20 static inline bool access_error(unsigned long cause, struct vm_area_struct= *vma) diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index f9334aab45a6..678b892b4ed8 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -356,7 +356,7 @@ void __set_fixmap(enum fixed_addresses idx, phys_addr_t= phys, pgprot_t prot) set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot)); else pte_clear(&init_mm, addr, ptep); - local_flush_tlb_page(addr); + local_flush_tlb_page(addr, PAGE_SIZE); } =20 static inline pte_t *__init get_pte_virt_early(phys_addr_t pa) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 9b6e86ce3867..d5036f2a8244 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -27,7 +27,7 @@ static void local_flush_tlb_range_threshold_asid(unsigned= long start, } =20 for (i =3D 0; i < nr_ptes_in_range; ++i) { - local_flush_tlb_page_asid(start, asid); + local_flush_tlb_page_asid(start, stride, asid); start +=3D stride; } } @@ -36,7 +36,7 @@ static inline void local_flush_tlb_range_asid(unsigned lo= ng start, unsigned long size, unsigned long stride, unsigned long asid) { if (size <=3D stride) - local_flush_tlb_page_asid(start, asid); + local_flush_tlb_page_asid(start, stride, asid); else if (size =3D=3D FLUSH_TLB_MAX_SIZE) local_flush_tlb_all_asid(asid); else @@ -126,14 +126,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, start, end - start, page_size); } =20 -void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) -{ - __flush_tlb_range(mm_cpumask(vma->vm_mm), get_mm_asid(vma->vm_mm), - addr, PAGE_SIZE, PAGE_SIZE); -} - -void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, - unsigned long end) +static inline unsigned long local_flush_tlb_page_size(struct vm_area_struc= t *vma) { unsigned long stride_size; =20 @@ -161,6 +154,24 @@ void flush_tlb_range(struct vm_area_struct *vma, unsig= ned long start, } } =20 + return stride_size; +} + +void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) +{ + unsigned long page_size; + + page_size =3D local_flush_tlb_page_size(vma); + __flush_tlb_range(mm_cpumask(vma->vm_mm), get_mm_asid(vma->vm_mm), + addr, page_size, page_size); +} + +void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, + unsigned long end) +{ + unsigned long stride_size; + + stride_size =3D local_flush_tlb_page_size(vma); __flush_tlb_range(mm_cpumask(vma->vm_mm), get_mm_asid(vma->vm_mm), start, end - start, stride_size); } --=20 2.20.1 From nobody Thu Dec 18 08:12:16 2025 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7DB1E1F541D for ; Thu, 5 Dec 2024 10:38:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395114; cv=none; b=Q0a70FEd1XHHDonCJs9/fLIJewRzcudamtqtf9aPefNRRtJGnBCCK0mJoFP+XpZTUOz9t9S5iLQWynBGIXVCcO6kTvXPinExPzhWmPo/xJ/AsGGLK24Im77NqZdyeolVz80AnmnQ5tmE9agn270I/E1fPsVfWBOYu89CfM8EuQg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395114; c=relaxed/simple; bh=2xt5O246IYbEQQM1Fa2q3Rs7HZrYgfCnOBLIf+feb+A=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=ohGTU4atPXSwd9kVLjln/KJv9rr+2oLLP9SqegNQe6887klrPbhcg01sKpBdnorOCYvLTIzh/bB+pGWZKu564Mern7qjUGyZpJfCkn48i7CrXX1PbbVclszPs7rWZklRrt3cZlwgkRs4o6+crwQasvwnl/kRSf7+clNRC5KCna8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=JjMf19Ur; arc=none smtp.client-ip=209.85.210.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="JjMf19Ur" Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-724f74d6457so839676b3a.0 for ; Thu, 05 Dec 2024 02:38:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395112; x=1733999912; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=q07gCTIJJE8Xz9bnD//uUjqUF3bkJlTAdnYkbDObJk8=; b=JjMf19UrJpojzu/ra2B0GYngWlOpfaTnMM67JpAGGfyYtqAvD6eT8pRC/FRDFoHuw2 4UAuyc00soi1QgqJAGh62l7RckIBcxvUuwMpX2euwxwp+ma20raT/HtE7U9K6f915fbG dwQ/t2T5hxQyeyycJcDeESNEtpOvF8FUzEi3R8pFJ0bGjhAF0p81WiWmzZVlitJ+sj9I N7mLpvj/D4V3fcZ8EJbBLAlmBINP0iKtv+7JHryiP89PcPPWdD5DftDnEJfI9gPIOLlg zbh5vmBURW5tu7iWcjDBk2PCTU/ZLoezG1vWP7vhjU70dSRn6G0XtSmbwKUkGjP83l/F mBQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395112; x=1733999912; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=q07gCTIJJE8Xz9bnD//uUjqUF3bkJlTAdnYkbDObJk8=; b=PQ+Gh5Srqpjq1YXVdP6FNTEZQXMcGnrddBb6lbLBW/YrKr8sp9ry6NOI/LHKfad+q+ wbx/EGC2J9MyS4IfA7DzaugtaQxQmyoR9L/rD2vmxM6o/EeYxuFY4se+VdBaIMY8DN7+ RlKrEYYJqMLHDE+HkDItGTKlM/6E/jsOYVUW7Go+CkUWUldLP/EO2C+ExqZ/kTyCg+8Z B0KAOodixQPMEuRTMNZqOh1xM4b9jMbu9R9xwBNwak2sOwmCxxFhNjexBhUXWwgZVzPz BRtAFPkj5jIt4uZFR1nRPLc028B1FYe8ua5cXcvNSnpJ9dngOFhSEl7CcSv+IS8SJ4ya AiWQ== X-Forwarded-Encrypted: i=1; AJvYcCWK0AbrxUPihkbZb+Kxi9ZYixMOHsgfF6Ue+w91CCP8LwP06VVyWTwu4bB947r1AjrklHKvt42k3ODHEuU=@vger.kernel.org X-Gm-Message-State: AOJu0YzgeVy0Gi50qQUnEHsnJqCEJFE04cDuboAoeCQLrV0X3JOCS8WM kj8WpA0WkpLCjolxllKP5VVdQA6J3k2i2EJopNDae+pbrJ6wxGmKW9x1WBNhG9s= X-Gm-Gg: ASbGncvt/mp2iG0F9ENWuCRaMhtoqlT9d6sZeSFchJ5a0xETlO0Zpo8EgNVdjfTNpxU tsHRyqty9kTcElYVZ7y+hPKbP59StQmiVkXBSHAs/1nGrX5rABUTU/vYDDP5hqCcIQGr4hHdizN zTlTVzlzae7wHQKzLhBo2LTSlrwzreVQ10CHBept9vPBLD7Fdcyn8lMleLtGkpoNgFkLq7hlXbH xqb8pfLgXLl2KaUsXlty1wWovn9xxuKAW/7eQS70nwQt5dyS/PNpifha7CR/qq+fMfpGUKJUb8u dNr1leOLXzyH3Jcf0WlYxGclKuK8LsmO X-Google-Smtp-Source: AGHT+IG3G9tPg6qUeSO1QGI/Mdcn1uGTNUPe0Tsf/e+WSW8vxQ4wgkKlRQB0qkddigD/hXKfbzL+jQ== X-Received: by 2002:a05:6a00:138d:b0:71e:cb5:2219 with SMTP id d2e1a72fcca58-7257fa5e7d6mr10595111b3a.9.1733395111754; Thu, 05 Dec 2024 02:38:31 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.28 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:31 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 13/21] riscv: mm: Adjust PGDIR/P4D/PUD/PMD_SHIFT Date: Thu, 5 Dec 2024 18:37:21 +0800 Message-Id: <20241205103729.14798-14-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable This commit adjusts the SHIFT of pte index bits at each page table level. For example, in SV39, the traditional va behaves as: ---------------------------------------------- | pgd index | pmd index | pte index | offset | ---------------------------------------------- | 38 30 | 29 21 | 20 12 | 11 0 | ---------------------------------------------- When we choose 64K as basic software page, va now behaves as: ---------------------------------------------- | pgd index | pmd index | pte index | offset | ---------------------------------------------- | 38 34 | 33 25 | 24 16 | 15 0 | ---------------------------------------------- Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable-32.h | 2 +- arch/riscv/include/asm/pgtable-64.h | 16 ++++++++-------- arch/riscv/include/asm/pgtable.h | 19 +++++++++++++++++++ 3 files changed, 28 insertions(+), 9 deletions(-) diff --git a/arch/riscv/include/asm/pgtable-32.h b/arch/riscv/include/asm/p= gtable-32.h index 2959ab72f926..e0c5c62f88d9 100644 --- a/arch/riscv/include/asm/pgtable-32.h +++ b/arch/riscv/include/asm/pgtable-32.h @@ -11,7 +11,7 @@ #include =20 /* Size of region mapped by a page global directory */ -#define PGDIR_SHIFT 22 +#define PGDIR_SHIFT (10 + PAGE_SHIFT) #define PGDIR_SIZE (_AC(1, UL) << PGDIR_SHIFT) #define PGDIR_MASK (~(PGDIR_SIZE - 1)) =20 diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/p= gtable-64.h index 2649cc90b14e..26c13484e721 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -13,9 +13,9 @@ extern bool pgtable_l4_enabled; extern bool pgtable_l5_enabled; =20 -#define PGDIR_SHIFT_L3 30 -#define PGDIR_SHIFT_L4 39 -#define PGDIR_SHIFT_L5 48 +#define PGDIR_SHIFT_L3 (9 + 9 + PAGE_SHIFT) +#define PGDIR_SHIFT_L4 (9 + PGDIR_SHIFT_L3) +#define PGDIR_SHIFT_L5 (9 + PGDIR_SHIFT_L4) #define PGDIR_SHIFT (pgtable_l5_enabled ? PGDIR_SHIFT_L5 : \ (pgtable_l4_enabled ? PGDIR_SHIFT_L4 : PGDIR_SHIFT_L3)) /* Size of region mapped by a page global directory */ @@ -23,20 +23,20 @@ extern bool pgtable_l5_enabled; #define PGDIR_MASK (~(PGDIR_SIZE - 1)) =20 /* p4d is folded into pgd in case of 4-level page table */ -#define P4D_SHIFT_L3 30 -#define P4D_SHIFT_L4 39 -#define P4D_SHIFT_L5 39 +#define P4D_SHIFT_L3 (9 + 9 + PAGE_SHIFT) +#define P4D_SHIFT_L4 (9 + P4D_SHIFT_L3) +#define P4D_SHIFT_L5 (9 + P4D_SHIFT_L3) #define P4D_SHIFT (pgtable_l5_enabled ? P4D_SHIFT_L5 : \ (pgtable_l4_enabled ? P4D_SHIFT_L4 : P4D_SHIFT_L3)) #define P4D_SIZE (_AC(1, UL) << P4D_SHIFT) #define P4D_MASK (~(P4D_SIZE - 1)) =20 /* pud is folded into pgd in case of 3-level page table */ -#define PUD_SHIFT 30 +#define PUD_SHIFT (9 + 9 + PAGE_SHIFT) #define PUD_SIZE (_AC(1, UL) << PUD_SHIFT) #define PUD_MASK (~(PUD_SIZE - 1)) =20 -#define PMD_SHIFT 21 +#define PMD_SHIFT (9 + PAGE_SHIFT) /* Size of region mapped by a page middle directory */ #define PMD_SIZE (_AC(1, UL) << PMD_SHIFT) #define PMD_MASK (~(PMD_SIZE - 1)) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index 9fa16c0c20aa..0fd9bd4e0d13 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -30,12 +30,27 @@ /* Number of entries in the page table */ #define PTRS_PER_PTE (PAGE_SIZE / sizeof(pte_t)) =20 +#ifdef CONFIG_RISCV_USE_SW_PAGE + +/* + * PGDIR_SHIFT grows as PAGE_SIZE grows. To avoid va exceeds limitation, p= gd + * index bits should be cut. Thus we use HW_PAGE_SIZE instead. + */ +#define __PTRS_PER_PGD (HW_PAGE_SIZE / sizeof(pgd_t)) +#define pgd_index(a) (((a) >> PGDIR_SHIFT) & (__PTRS_PER_PGD - 1)) + +#define KERN_VIRT_SIZE ((__PTRS_PER_PGD / 2 * PGDIR_SIZE) / 2) + +#else + /* * Half of the kernel address space (1/4 of the entries of the page global * directory) is for the direct mapping. */ #define KERN_VIRT_SIZE ((PTRS_PER_PGD / 2 * PGDIR_SIZE) / 2) =20 +#endif /* CONFIG_RISCV_USE_SW_PAGE */ + #define VMALLOC_SIZE (KERN_VIRT_SIZE >> 1) #define VMALLOC_END PAGE_OFFSET #define VMALLOC_START (PAGE_OFFSET - VMALLOC_SIZE) @@ -1304,7 +1319,11 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pt= e) * Similarly for SV57, bits 63=E2=80=9357 must be equal to bit 56. */ #ifdef CONFIG_64BIT +#ifdef CONFIG_RISCV_USE_SW_PAGE +#define TASK_SIZE_64 (PGDIR_SIZE * __PTRS_PER_PGD / 2) +#else #define TASK_SIZE_64 (PGDIR_SIZE * PTRS_PER_PGD / 2) +#endif #define TASK_SIZE_MAX LONG_MAX =20 #ifdef CONFIG_COMPAT --=20 2.20.1 From nobody Thu Dec 18 08:12:16 2025 Received: from mail-oo1-f46.google.com (mail-oo1-f46.google.com [209.85.161.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ACA611FC7E7 for ; Thu, 5 Dec 2024 10:38:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395118; cv=none; b=uoDTVIlU6xlfmN+Jd2R4CU2EDBR2HCaDDY9DPuERkTygECrwjQtJr9RnPYpqeL8xgCiiAlc6pCmqroHZeovITRXwJ9B4XS7rH/dP4OFnxbFqMFIuyMAFonER5cvzUYvlPOYjM+umWypqaE3Y/cCTVQADFTSUryWA7kayUf6TxuE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395118; c=relaxed/simple; bh=4wlJhID0o3jWa77hDSW35GLVeQRNwf5/sMX6Ua0okEA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nfSUZ7VZfi8QaZL6cBhp8F3indYmxg/j3tQy9NIW9yxLmyZS839FFwoQpDKkbN5mgm587mcdwkUeQHAI32aR2vcQhAQvxX/3aoq0Y4uMw/sdG9QCe/e6U/FfYBRN3NoI5j8fRK58gZywmBqmYOm1KCXl2oECk7DaQ11Yy9dkk54= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Qn2kqWig; arc=none smtp.client-ip=209.85.161.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Qn2kqWig" Received: by mail-oo1-f46.google.com with SMTP id 006d021491bc7-5f1ed6ee4d6so399931eaf.2 for ; Thu, 05 Dec 2024 02:38:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395116; x=1733999916; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nxSp7kmSqkSnu4KEBfrma7Z74EMZDahAlv9AUZJtR6k=; b=Qn2kqWig/vHxSglw3enhYJ2yjOq/zqMtLd+oxdVe4uYKpwzzJCrMctsxm95STyUBtO k3qxhhTBcf/32F2KBKJy5YT9m64dfx2o5SrLnIi3J1l7L4nGsIlSFSmVQvQfqnNB3FS2 zzxpa4+zSDDY32v+Lch4rFLgIoW5qsT2TaV73BLXHkzkCAGwibuGT1/F6TrcD/eDwCEr MWrc2o7IayOTNCe1PML7PCuFu63DucwciX707KzwFxz7wN0GpHoV4dfpvK5xYP4XG9Je V7z2h+CkegE/PqVLfCUm9PPD7yBfVwVKv1L21p3TMNz7RrcyUwfTrAkwn6KvpoO91qEW zMZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395116; x=1733999916; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nxSp7kmSqkSnu4KEBfrma7Z74EMZDahAlv9AUZJtR6k=; b=G9JwWjiG/PqZxhc3Vfh6B/XY697Hh2ZBUHfutpyNq4Na8W4tkekumjIa7HnOpNeieU uzs8B47K15ikoTIRdGeiaKHJwUTJBzF8i/Dtulaoe+2Y6etj7yRPQ1xHPBpfjOnlQKeC epRhOLc7w7pSdDr6+n1OAYiEQlT4tWje8gROP2vc8p2z2PcKmHdWzDrFttGHw9f0ew6i XMEQWsmHVIhslcQf+TgfE72uCuoJ9RjtHez79RMPFTK6VAWJ4qlIvGTCIw+zkIHNFx0x I7WxWqlLBdxRlba2p8QOyb96Xv1S7tq6frNVKXxZdfaWvljaXbBnb2Ckv6AliBKYbAsI 06XA== X-Forwarded-Encrypted: i=1; AJvYcCVrH+GeO9+Mk2M6UOfs5ZgPqkqZBg7Zwp/ZvQv9ba03FT54WPgFR8V6j8ZB/Zh9nKc3CYsQFw1GHwrbkIk=@vger.kernel.org X-Gm-Message-State: AOJu0YwvcNbwBS150G8yidMKFJcIbg+BEO2pF4dU46B0PGY0ofkiNddg lSXNKFcyh22VEZFl9cGQsIFVnEQi5VQ84mdP1g58hiCP3Px3kF9dzBxVKy3Efz4= X-Gm-Gg: ASbGncsv3QTY1LomROOBpvvHlUq6og43FrR66Nw+7G1QPljpVa+suAP4TL+m7OYGhQt Pa2T6trGiBdi2q6zJ8cZsTNAp6iRAEKIC2gMZHJjCoIdErqEDfkVNObgVCIpoG3Di63mkoOI6t2 1Cd+xBdJxGG9gSAob7Bt8AGME2WAJQCF+5gjyOhhj5WAhQKHDWTvNqz8s6TRWrKh0EOnqG9oWPP iwqfMPIeaSyW/bCyAuUhHbdkqOG1GpjSLCI8WrwD2cQ3zRffcoirfTFLl/D3t5S1g/cgQAZBW1M Y6tA6mwZaEnu2NmyeihpZdk9Ovrho8OQ X-Google-Smtp-Source: AGHT+IGAja+xIu0LL2xTFT4mUl3KDqN8KlPjYMDJCXFuXRD/rOHFiqKg6itfbSNH2M80yhGpilt+dg== X-Received: by 2002:a05:6830:4882:b0:71d:6272:862f with SMTP id 46e09a7af769-71dad6d401bmr10590081a34.22.1733395115810; Thu, 05 Dec 2024 02:38:35 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.32 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:35 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 14/21] riscv: mm: Only apply svnapot region bigger than software page Date: Thu, 5 Dec 2024 18:37:22 +0800 Message-Id: <20241205103729.14798-15-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Usually, when it comes to napot pte order, we refer to the order of software page number. Thus, this commit updates the napot order calculation method. Also, we only apply svnapot pte as huge pte when its napot size is bigger than software page. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable-64.h | 21 +++++++++--- arch/riscv/include/asm/pgtable.h | 50 +++++++++++++++++++++++------ arch/riscv/mm/hugetlbpage.c | 7 ++-- 3 files changed, 61 insertions(+), 17 deletions(-) diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/p= gtable-64.h index 26c13484e721..fbdaad9a98dd 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -124,12 +124,23 @@ enum napot_cont_order { NAPOT_ORDER_MAX, }; =20 +#define NAPOT_PAGE_ORDER_BASE \ + ((NAPOT_CONT_ORDER_BASE >=3D (PAGE_SHIFT - HW_PAGE_SHIFT)) ? \ + (NAPOT_CONT_ORDER_BASE - (PAGE_SHIFT - HW_PAGE_SHIFT)) : 1) +#define NAPOT_PAGE_ORDER_MAX \ + ((NAPOT_ORDER_MAX > (PAGE_SHIFT - HW_PAGE_SHIFT)) ? \ + (NAPOT_ORDER_MAX - (PAGE_SHIFT - HW_PAGE_SHIFT)) : \ + NAPOT_PAGE_ORDER_BASE) + #define for_each_napot_order(order) \ - for (order =3D NAPOT_CONT_ORDER_BASE; order < NAPOT_ORDER_MAX; order++) + for (order =3D NAPOT_PAGE_ORDER_BASE; \ + order < NAPOT_PAGE_ORDER_MAX; order++) #define for_each_napot_order_rev(order) \ - for (order =3D NAPOT_ORDER_MAX - 1; \ - order >=3D NAPOT_CONT_ORDER_BASE; order--) -#define napot_cont_order(val) (__builtin_ctzl((pte_val(val) >> _PAGE_PFN_S= HIFT) << 1)) + for (order =3D NAPOT_PAGE_ORDER_MAX - 1; \ + order >=3D NAPOT_PAGE_ORDER_BASE; order--) +#define napot_cont_order(val) \ + (__builtin_ctzl((pte_val(val) >> _PAGE_HWPFN_SHIFT) << 1) - \ + (PAGE_SHIFT - HW_PAGE_SHIFT)) =20 #define napot_cont_shift(order) ((order) + PAGE_SHIFT) #define napot_cont_size(order) BIT(napot_cont_shift(order)) @@ -137,7 +148,7 @@ enum napot_cont_order { #define napot_pte_num(order) BIT(order) =20 #ifdef CONFIG_RISCV_ISA_SVNAPOT -#define HUGE_MAX_HSTATE (2 + (NAPOT_ORDER_MAX - NAPOT_CONT_ORDER_BASE)) +#define HUGE_MAX_HSTATE (2 + (NAPOT_ORDER_MAX - NAPOT_PAGE_ORDER_BASE)) #else #define HUGE_MAX_HSTATE 2 #endif diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index 0fd9bd4e0d13..07d557bc8b39 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -130,7 +130,7 @@ #include =20 #define __page_val_to_hwpfn(_val) (((_val) & _PAGE_HW_PFN_MASK) >> _PAGE_= HWPFN_SHIFT) -#define __page_val_to_pfn(_val) (((_val) & _PAGE_PFN_MASK) >> _PAGE_PFN_S= HIFT) +static inline unsigned long __page_val_to_pfn(unsigned long val); =20 #ifdef CONFIG_64BIT #include @@ -470,15 +470,42 @@ static inline unsigned long pte_napot(pte_t pte) return __pte_napot(pte_val(pte)); } =20 -static inline pte_t pte_mknapot(pte_t pte, unsigned int order) +static inline unsigned long __pte_mknapot(unsigned long pteval, + unsigned int order) { int pos =3D order - 1 + _PAGE_PFN_SHIFT; unsigned long napot_bit =3D BIT(pos); - unsigned long napot_mask =3D ~GENMASK(pos, _PAGE_PFN_SHIFT); + unsigned long napot_mask =3D ~GENMASK(pos, _PAGE_HWPFN_SHIFT); + + BUG_ON(__pte_napot(pteval)); + pteval =3D (pteval & napot_mask) | napot_bit | _PAGE_NAPOT; =20 - return __pte((pte_val(pte) & napot_mask) | napot_bit | _PAGE_NAPOT); + return pteval; } =20 +#ifdef CONFIG_RISCV_USE_SW_PAGE +static inline pte_t pte_mknapot(pte_t pte, unsigned int order) +{ + unsigned long pteval =3D pte_val(pte); + unsigned int i; + + pteval =3D __pte_mknapot(pteval, order); + for (i =3D 0; i < HW_PAGES_PER_PAGE; i++) + pte.ptes[i] =3D pteval; + + return pte; +} +#else +static inline pte_t pte_mknapot(pte_t pte, unsigned int order) +{ + unsigned long pteval =3D pte_val(pte); + + pte_val(pte) =3D __pte_mknapot(pteval, order); + + return pte; +} +#endif /* CONFIG_RISCV_USE_SW_PAGE */ + #else =20 static __always_inline bool has_svnapot(void) { return false; } @@ -495,15 +522,20 @@ static inline unsigned long pte_napot(pte_t pte) =20 #endif /* CONFIG_RISCV_ISA_SVNAPOT */ =20 -/* Yields the page frame number (PFN) of a page table entry */ -static inline unsigned long pte_pfn(pte_t pte) +static inline unsigned long __page_val_to_pfn(unsigned long pteval) { - unsigned long res =3D __page_val_to_pfn(pte_val(pte)); + unsigned long res =3D __page_val_to_hwpfn(pteval); =20 - if (has_svnapot() && pte_napot(pte)) + if (has_svnapot() && __pte_napot(pteval)) res =3D res & (res - 1UL); =20 - return res; + return hwpfn_to_pfn(res); +} + +/* Yields the page frame number (PFN) of a page table entry */ +static inline unsigned long pte_pfn(pte_t pte) +{ + return __page_val_to_pfn(pte_val(pte)); } =20 #define pte_page(x) pfn_to_page(pte_pfn(x)) diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 8896c28ec881..4286c7dea68d 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -212,7 +212,7 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shif= t, vm_flags_t flags) break; } } - if (order =3D=3D NAPOT_ORDER_MAX) + if (order =3D=3D NAPOT_PAGE_ORDER_MAX) entry =3D pte_mkhuge(entry); =20 return entry; @@ -405,7 +405,8 @@ static __init int napot_hugetlbpages_init(void) unsigned long order; =20 for_each_napot_order(order) - hugetlb_add_hstate(order); + if (napot_cont_shift(order) > PAGE_SHIFT) + hugetlb_add_hstate(order); } return 0; } @@ -426,7 +427,7 @@ static bool __hugetlb_valid_size(unsigned long size) return true; else if (IS_ENABLED(CONFIG_64BIT) && size =3D=3D PUD_SIZE) return true; - else if (is_napot_size(size)) + else if (is_napot_size(size) && size > PAGE_SIZE) return true; else return false; --=20 2.20.1 From nobody Thu Dec 18 08:12:16 2025 Received: from mail-il1-f170.google.com (mail-il1-f170.google.com [209.85.166.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E1E9E202C3E for ; Thu, 5 Dec 2024 10:38:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395122; cv=none; b=eMKKBDH2QpiAW+zgrmS2OpNxkeCu3+AQ1Ti5eyFtjHjk9R7o99CB5M5eyuT9lx4E2LIIlLzX+kmCBASrI215t97/q6Mz5qZA3MKy96BxehPtwTqy+Fd+uppc5C17OIyIdHaKV1I+For518peN++m+uaszbpPjH4+rkaJ6hLdyaI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395122; c=relaxed/simple; bh=LAwYaE87yw9Ykpi8K3m6cxKiTi2Y0B9KHX3J6XW9rFc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=c5aL+FPvNa/oVmAFcCBgLHNCRQPS9izs4dl12I5Qt5do72tan016BXQ90x/Dy53r/Ja4KjbWBULT75b+6hV+nK5RdfnCHGqNKKbXGn7mpZI22EuH0WnbyZ1e93N+6fAt0cB76rA+Abum2rU+Mi6gk0fTy0vUUudEtxM4zAVOxnI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=QOOfvl8Z; arc=none smtp.client-ip=209.85.166.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="QOOfvl8Z" Received: by mail-il1-f170.google.com with SMTP id e9e14a558f8ab-3a77c066a15so2819935ab.3 for ; Thu, 05 Dec 2024 02:38:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395120; x=1733999920; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uQ/0Scy9E/wZ7/MnA5BYlbNUXDy/8A/TpwTZAMtapEc=; b=QOOfvl8ZPBdvDomoOKLEGRU/7MBDZevZQtHL3cOsFNzmcpVLu3TSryug8jcApJVLqc RnwQITCdfx6HIBbkBiv7NdKGDgQcZ3LhkdwQJFPXYA2br6ycgPjPohdjz74/C4z5yvAx GCQ9KYg3ShOWSaicA+gdl9WWffo6G+1OqHI0arzvUmMxPduu5DjP+96HoML68AMdpO0W Zyn2wy+sjsbtAW1RSTKPYdpJ/G7ZBdwjis0qQNafSAMQwBQJQAuhBOZb3a65UOvcYueK z3zGMcJ5iS1nWAawJM77zKNyzcwIp9VD4AyV7u11sJLrbMIWAuIbj+8IFezrtGFBsIio U+cA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395120; x=1733999920; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uQ/0Scy9E/wZ7/MnA5BYlbNUXDy/8A/TpwTZAMtapEc=; b=UQzapTRQHKXlFkHfFwgslqm0lW81NpTN0pEr7MKSeDouMhnooAOLVhuAb9W5eif6vR rCCJBfrxAyLK8ufQDWz0mwoDAqcIMjc6RQAKICwEnCJx7ISjfxA84/x/tMNGSDY+tuhN f6KbeW/IMN3o3lw93HGnuJh6eD/vOuOsHPukzb0vb5TvylcVAPt3QAI0eMlL01jATsRQ vgQPElgCVQU+p7EiVmdJ+AVCJSdu88UhQ2Ze3ZMlILpjmRROKTCRrvDG4l+po31h5t4a qaDkH7+0Fcr83TqyKncxXiDUJJuS/NvRmjFd8t4o3kW0FvIF7YlkEyaJX5Lk7KLbLolS KFkQ== X-Forwarded-Encrypted: i=1; AJvYcCXcqzlAC6Uw7n50mzZCsu5dIVYAqGWbY6in9FBWpV6+lOc6SYe1Zgni6VMb5HGen9H9aYsskQeAWZzAuSI=@vger.kernel.org X-Gm-Message-State: AOJu0YzlZ3KZHVmLpiMtEQTZI3XeueCLYy4+A9VvU9fTpeYFDQ6vwqnS bj0tX7b2Yd8+OhvbgK1MKKeivILg8fBGFme/Ci2ZArCsjAl+BhkGv+ZmNt5hrNoY778AIxXZXWz gWa0= X-Gm-Gg: ASbGncu0hlTR70vWA6HjYenfN0/FGLSDXS6eGlmvyUq1Uo/m1u0py3rvK10pGJaaswu B5sw5zD/3bsCkSwU/4kwrsDir2Q1t3dtSiqp9Efpm+8CtYxHWgAkYxQqzYutUnPbnMD2jOWdT7h 0V8tWcQghaM6urgNOt6tOorNXuXyUnWl4dzTrAcxs50lLw5FZUy1cTQ7/oFFymwlt57CPF/wc31 eeAS8aD1oB9fs4vJ9gn4lxUMmPO9IBuTSypc3La7D54U2tzUEdz0719YFqwph3KZkC+eTwerj/6 lbvGdMDvjsxXSJ6aT/oDVazlCq4NDgRg X-Google-Smtp-Source: AGHT+IE8l6yfJ87eJad5FAeGJqDd+iC1OcZn198mK7ZYZDH6TLk8OTxSkqSKrMniNwvDITNae2Qz7w== X-Received: by 2002:a05:6e02:144c:b0:3a7:d792:d6ad with SMTP id e9e14a558f8ab-3a7f9ab0ff1mr115272635ab.22.1733395119846; Thu, 05 Dec 2024 02:38:39 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.36 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:39 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 15/21] riscv: mm: Adjust FIX_BTMAPS_SLOTS for variable PAGE_SIZE Date: Thu, 5 Dec 2024 18:37:23 +0800 Message-Id: <20241205103729.14798-16-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Xu Lu --- arch/riscv/include/asm/fixmap.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/fixmap.h b/arch/riscv/include/asm/fixma= p.h index 0a55099bb734..17bf31334bd5 100644 --- a/arch/riscv/include/asm/fixmap.h +++ b/arch/riscv/include/asm/fixmap.h @@ -44,7 +44,8 @@ enum fixed_addresses { * before ioremap() is functional. */ #define NR_FIX_BTMAPS (SZ_256K / PAGE_SIZE) -#define FIX_BTMAPS_SLOTS 7 +#define FIX_BTMAPS_SIZE (FIXADDR_SIZE - ((FIX_BTMAP_END + 1) << PAGE_SHIF= T)) +#define FIX_BTMAPS_SLOTS (FIX_BTMAPS_SIZE / SZ_256K) #define TOTAL_FIX_BTMAPS (NR_FIX_BTMAPS * FIX_BTMAPS_SLOTS) =20 FIX_BTMAP_END =3D __end_of_permanent_fixed_addresses, --=20 2.20.1 From nobody Thu Dec 18 08:12:16 2025 Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9C9EF204081 for ; Thu, 5 Dec 2024 10:38:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395126; cv=none; b=mBszGmbiptRwQnI8YDnq5Y4Tb4qlD2jmwn6BJV11mPlaXlKGyI38M0VbvRWQ7hUmXH/0VEkVMbF23x2EqBiy5nuPKB+SGwg4ngylGE/JNTsa+p7HHbfvMxBEbh3mEbx/5BQfhU/5NHD4wsvxFXs4Wq5RY+iAN3waRN6DgS7ltbo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395126; c=relaxed/simple; bh=xOye6TxZfYFU7+9BNB2Cblh+Egvr/wPlyq7m0ZZN5eI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=K7hZEZJQaXDKfxVQ1PIvat05lqYxTkvG3/7jcvzjdqzH9Lq8Le49AQLpszc8rQRSHpwwArJKQjuqL8CEPCUFOX9bhDziobrYEGtkstP+lr4BnzqqKptmTR9GwD+m7UcJLN+CwBk6hV1dIo5xr/Ad72B91gUpx2QtAGiBj1T0CKs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=HvWVmKQU; arc=none smtp.client-ip=209.85.215.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="HvWVmKQU" Received: by mail-pg1-f178.google.com with SMTP id 41be03b00d2f7-7fcf8406d8bso1533801a12.0 for ; Thu, 05 Dec 2024 02:38:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395124; x=1733999924; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wAxAI+FpEZPqeOtAYXtXd/fFddvtrgZvKhcRJUP9sTk=; b=HvWVmKQUJpYWhYTVansiNsXRmrZn4PiaXHWREuvBBNo3UY2XvRTNbS7OW3/jCdUGdS I4P4QG81O51gLnVVmivRc1P3w6YW5vZ6X92VY+12ZXyMc6r1xmqVFF9aql9sGmRhptC2 z17DOcRNNex5KqOR7xohjNqc3HVBiVdY+A/y6FGxdbClEkRkh94/qYNiw9qa+t+7Wyzy sOFMFlOhp8qcHbsiYStgNNCMN+X1V+LzAfFlFjYQLjkUrwmnP3K+CipO54+/t+U6iFy2 zflZXKVH7U09faBHDQCxXcjXVlnYMcVjUSO5DLsHSEYGFp4te7Pp+I9MGqR6q/KdbjFT xLIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395124; x=1733999924; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wAxAI+FpEZPqeOtAYXtXd/fFddvtrgZvKhcRJUP9sTk=; b=byWyaVFDm8OYo1jdE3DIcjnt/cR6Sqo0yZuFc0+jdSGUbUkhod6TAuXnLOJ3f792Sg 55+MRHXZtMvc/OTh37GYb/E6XrGxEMGTXKhIVCmj2p1oe9omtbAwwD9kVYFLLdj0dbD5 cIEf5KJ3r6SYVtYbxMGWHiInaYSz5SJpsbEhHScOh9W5Nc50ZMi5YLlZ696n+CJw1yfN yn0Dorp+yKSDAtZHfLj2zJCUvzizd3IRsZJNlAz/Mqt3OcWI2MNE48UPYBWKAS0hQA1S sRcUQKGkOZT6MgNqkMh4mUb+OFvl8BbxQ17cFwWautAhKqFopzxqQWQ2evvxUKKE5il/ gYQQ== X-Forwarded-Encrypted: i=1; AJvYcCXTqnzPP1JHNHDLX1OzUNhK1LRPYY/GCn/xJ25nSAzGqZKKMFdqUDSWHMCRI5uDaulgRoieuumzU9PQ7dw=@vger.kernel.org X-Gm-Message-State: AOJu0Yw67N/ehlT488G+uEPqWNXHYsPuCIvuM7QrgnenzMQCvsTFNz3I cUlR99XlIBxVtEG/TjfvJpsRfAt4gV5OUsr97KY0GCXqsu5G8Z8ckQDbf6NgJXU= X-Gm-Gg: ASbGnctBGbJ1H/2FHJTtSP2VX1LiWm6xJ0IQwxnMV+gaB/cm5TN28ZG94YvZUyVSkjL wxPr5cu4qlmb8eYn/MTGnv0ikISADl0GwUkyh2XDYMbOxKwBuUmsGW9VC3COhMRDB3xABV7xGd7 8Q1ikpTPdFiQtohdTzq7u51THzmM2w9UlYnptcy830qVt67ZjJnkSvjznrdyK9QLgPZluds8hXD Xle5+fUa7/P84SbxNvrdW5/5NFcs/7ToZ9ZNxyP1O0QAUm6uB7NQZ/XogHLq68WoPsk8oMXkuZc yFbTTO/jBUnvaS5dav0QZ4Zu7qKy0LLV X-Google-Smtp-Source: AGHT+IE8309GuNeyZxITDQ1e0WzOIa0uJDO0sUbGSJz89pLUWjkP1vS9Smp96TCW7bxn3tkgM30nOA== X-Received: by 2002:a05:6a20:a128:b0:1e0:ea70:7494 with SMTP id adf61e73a8af0-1e17d38c18dmr4437406637.9.1733395123986; Thu, 05 Dec 2024 02:38:43 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.40 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:43 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 16/21] riscv: mm: Adjust FIX_FDT_SIZE for variable PMD_SIZE Date: Thu, 5 Dec 2024 18:37:24 +0800 Message-Id: <20241205103729.14798-17-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index 07d557bc8b39..5b2ca92ad833 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -108,10 +108,10 @@ #define PCI_IO_END VMEMMAP_START #define PCI_IO_START (PCI_IO_END - PCI_IO_SIZE) =20 -#define FIXADDR_TOP PCI_IO_START +#define FIXADDR_TOP (PCI_IO_START & PMD_MASK) #ifdef CONFIG_64BIT #define MAX_FDT_SIZE PMD_SIZE -#define FIX_FDT_SIZE (MAX_FDT_SIZE + SZ_2M) +#define FIX_FDT_SIZE (MAX_FDT_SIZE + PMD_SIZE) #define FIXADDR_SIZE (PMD_SIZE + FIX_FDT_SIZE) #else #define MAX_FDT_SIZE PGDIR_SIZE --=20 2.20.1 From nobody Thu Dec 18 08:12:16 2025 Received: from mail-oi1-f171.google.com (mail-oi1-f171.google.com [209.85.167.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 082A82066EC for ; Thu, 5 Dec 2024 10:38:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395130; cv=none; b=SGLUAwQrGAZC3Igc6qpOsdnTdsu6W8nOArHOeealwHdGz+J3RpqK/h8hTgtZf2tBo5q81gSBXabmb128EVkowPs5VYM/2dh3a2y3P2aXQAMwi7+r874eCXm2fhjRrp21x6zFmlhSXOVHOSmg4YXcWdUBCgUYpIMlZWdmBEoOS44= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395130; c=relaxed/simple; bh=ppeWGXE3JwCtgI11oYFm67q3jViFEy7jG8L4BpO6njk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=P4WZzObGrUs/Dptzz9fVQM3J+ItqyvM6RZL0vl3SH2eqskKRZf4GzVVt+4Lq+pc/tlMFUFV7pLdtjyhXnbAtmC4KazKyQu9AacgvOtz9LV2RK3iCDLfSVNrQzLlKmAMyPrdwCb0LHAqD4oFejbdN+7qtwOVZUF8wJX44pXtb4pw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=LEKcgbwR; arc=none smtp.client-ip=209.85.167.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="LEKcgbwR" Received: by mail-oi1-f171.google.com with SMTP id 5614622812f47-3ea4c550a3fso414457b6e.2 for ; Thu, 05 Dec 2024 02:38:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395128; x=1733999928; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=x95dczg6H/IuNQfp1VlRQgQwszamKCGOevCsmbhARnA=; b=LEKcgbwRI36+p/0MwvKqPxQUH0FjNUZJq2m6iy5WcSIVAI7M97AIn7zxwSfQA5yqU9 FcEv6pf43g6R+Zkr0xMlySwRewCEADEzNwmYY+aScQaDKjn4XUUTwh7zsWsiMh/PSpfP DZLZeNnINn+TQhRihZJcvEEGGUejHX2U6tiiufewBtjPGGN2S6WPORXK1oiLJ6ezwnw0 jox0kvqeHFrVlJnb0kRw/dCNxUBh9I5/eXxDlYREsViQUvGk9QsvqITGjRkd4Ea1znWZ CYDNSa2BT9xCytSZGvMgHiRKB5L6mp+Z4IevZrwyCVkLd/B458e6AtWZW97hRvNy4NWC vFoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395128; x=1733999928; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=x95dczg6H/IuNQfp1VlRQgQwszamKCGOevCsmbhARnA=; b=optlT7ToTNAGF4ApFPQRCwbAnuKpRpdxjIrKmcRhOvqdz/gUmzB7AhuappSNLoUcsy 84ASbBIKfBw9S7y1J5itr3uY9HD3/UKK6vehzZGh41Prcjj8DqHYWn1dxU2EuRovYthL iQqJ+TzY6dSR8G50bi3Cwv6tsMHcuwLKKscyOgSsO82sVK0xGMtHzRclQjEsue+ExR6k UW1ZieBsqab8BRyXifoDPwRg2pDRoaCNmMr1+kw6NNAAk5mdmbKzfR43zTcpg6xZ+Oix V+Opa7ekXbsnUVIn2wY/ekANZQm2P7oQEWmgHYPhrlKj9sPmySR8eaFLaz9A/L3CL1JU W4rw== X-Forwarded-Encrypted: i=1; AJvYcCXohvqXn5Wfw9kE6OvzeTEsFCOkF4303x/lad8sVPqXFD5kTekjFYDRYX235FjCG6AbPxmvf25weLXcOgM=@vger.kernel.org X-Gm-Message-State: AOJu0YwLgtPM8u8wEN+zbNCWx3LLHQWtKrT/NsdS9JcCBp/60x85bP86 ezMUaCN6pS4oGOO85MZvLExKaquYG7+uWSUNb6lQP1zhlkOFkJOc/eCgwT+EQsA= X-Gm-Gg: ASbGncsuYdSePd5IQgWYytS4vmr39lm30KUSSo7oc6rnFDOB5Eebisci7Ml0NLBxmE3 SoeuiEpJwEYXx/bViRN+0qO9tpnzc3/aUpCm++skQf8c880Tma8RcpHG02AkivQhlXLwPxTuXa2 pAa37MdjmGbKhmMMWFEKHJYY9W14Unuln9KlqLm0wKNvXr09Y66b4zR9MPBesUlCvJFcLy/uxOr idZFreEcV3e/LNMwmYgHAnZiDqMkoKij/Bno0tgWZuLxn9GRn2v+9eKQMI7ZUArQ8kN0o3t5ujm umqLd3zXAvveo6F/KhHYUYFDOaLu6r0+ X-Google-Smtp-Source: AGHT+IFzuODYxPZb1b0Hsra5aByVGtGgW30pbNXl6TXKG3vo0iIQP7jLbPRNzxf3m6EfJtUU2ofomQ== X-Received: by 2002:a05:6808:bcd:b0:3ea:49a1:cba7 with SMTP id 5614622812f47-3eae50c5186mr8718922b6e.41.1733395128040; Thu, 05 Dec 2024 02:38:48 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.44 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:47 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 17/21] riscv: mm: Apply Svnapot for base page mapping if possible Date: Thu, 5 Dec 2024 18:37:25 +0800 Message-Id: <20241205103729.14798-18-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" All hardware pages in the same software page point to the same contiguous memory region (the region size is equal to the software page size) and has same prots. Thus this commit uses Svnapot extension to optimize the mapping to software page to reduce tlb pressure. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 16 +++++++++++++++- arch/riscv/mm/pgtable.c | 6 ++++++ 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index 5b2ca92ad833..9f347e5eefeb 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -483,13 +483,27 @@ static inline unsigned long __pte_mknapot(unsigned lo= ng pteval, return pteval; } =20 +static inline unsigned long __pte_denapot(unsigned long pteval) +{ + unsigned long prot_mask =3D ~(_PAGE_HW_PFN_MASK | _PAGE_NAPOT); + unsigned long res; + + if (!__pte_napot(pteval)) + return pteval; + res =3D __page_val_to_hwpfn(pteval); + res =3D res & (res - 1UL); + pteval =3D (res << _PAGE_HWPFN_SHIFT) | (pteval & prot_mask); + + return pteval; +} + #ifdef CONFIG_RISCV_USE_SW_PAGE static inline pte_t pte_mknapot(pte_t pte, unsigned int order) { unsigned long pteval =3D pte_val(pte); unsigned int i; =20 - pteval =3D __pte_mknapot(pteval, order); + pteval =3D __pte_denapot(pteval); for (i =3D 0; i < HW_PAGES_PER_PAGE; i++) pte.ptes[i] =3D pteval; =20 diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index 150aea8e2d7a..0bcaffe798d5 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -11,6 +11,12 @@ pte_t __pte(unsigned long pteval) { pte_t pte; unsigned int i; + unsigned int order; + + if (has_svnapot() && __pte_present(pteval) && !__pte_napot(pteval)) + for_each_napot_order(order) + if (napot_cont_shift(order) =3D=3D PAGE_SHIFT) + pteval =3D __pte_mknapot(pteval, order); =20 for (i =3D 0; i < HW_PAGES_PER_PAGE; i++) { pte.ptes[i] =3D pteval; --=20 2.20.1 From nobody Thu Dec 18 08:12:16 2025 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6ED9206F05 for ; Thu, 5 Dec 2024 10:38:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395134; cv=none; b=uftiwjToxNHL+KZ4igkM2t7oDah6bsZRh37rgor0yUUxeBnwE91pCWCL/VzJL38VXCO8QW/tDHT1iK4g8n/PONW57e8HOXdWYKBt/+bH5FjSgC+1nCWddOrm3dAD7X1JKWM9G8foy9fWGNibt1ynDIVCwj1/eO+hEGjA0dcT4eo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395134; c=relaxed/simple; bh=xZW3Hjbwaye/czccQX2iibB21IBSJjMq6pEPTJmFaVs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=SyE1NDZP7Un1FVRtBuEL4fVNLuLNpgYcV9cTUQFx07lDSoNCxj9jlRfLBdH15JW5ZXyYaVZvPw/ZOYBLRrKG3hPDEBy0dB4muMhywgnM4tknaRaMUczBaJ1Em4XKiDMCzcnvGH3rAoOuiKAHrPozZ6QRn6DDpLu0Gl3+JS/LEEM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=DfbvqQUq; arc=none smtp.client-ip=209.85.215.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="DfbvqQUq" Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-7fcc00285a9so834648a12.2 for ; Thu, 05 Dec 2024 02:38:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395132; x=1733999932; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6fWC9OQ8fDv8IwsAqd/wbk191tCVh4tSTxqplFbmaYk=; b=DfbvqQUqQ3XH9vHIclajmTrDUH/7gA8SQ2xtdMfxkhJm3BnyFUofn0UGaB5j+w4qNc lprY61uxSnGLHjMWwMrk07n/WepHHSopOf4KdSfCFdbcSQpRfidlZT9H0/WOUhVCXZ8l 2rCHUUhOmQPegiOLPOmytV/QfwNGcns7NdYkjYZctXF5r3Uj3EDKR5Bmjb5/gYv1CE5r begRMVUkYdtANvkHPpYOCu0IMyCcjC0LQS2/653/Jv8IQcEJqJMGZgzYJ/TV5hsmzM5U 99V62o+6ylbKez0qYwtON66Kdw9Q/II7aO35HttZonr+SlxXoZc0SQhxw4FoGlkIzAdv 28qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395132; x=1733999932; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6fWC9OQ8fDv8IwsAqd/wbk191tCVh4tSTxqplFbmaYk=; b=SOG0FtdM1rqtBZ5van4rkiw0yknGaK5BTz2jpdoc0vKCwMth1EMhhWejL33ZKztP8q R09ebO3StN1vH7ygAUMKlKaVMk5LqKr0+GFDMU3DsjuhBXy1+NsgbhEe3X4RbdCutIWE 9k3gERm8BcCYbjXmTb2/2JP82szX6/No28PQ7MWn83uDoWgtIsGlBTQ0kjZY1HSmMSvW zAKZHIYU7ARBvDCwV8ejlZMyVat2vIzphnUsBq/PqWQYjE/J7RrQa4SNvTSIED3Uokdl waxjcxHwslmE43tBpy0YwSRAXfuI/PSnexysveAXhHTW7DgMrcwp0w3/bAH7MxxxxqCM misg== X-Forwarded-Encrypted: i=1; AJvYcCWQjWpbvpOL0m9/tfEaez3IeewBFDevaZOPHN6SPuKhPrhbWADi+nA4g+sjwym2eJ+F6ynZA0WITgWxhOo=@vger.kernel.org X-Gm-Message-State: AOJu0Ywfk9ff6YXtihPuQcch42X70W2KX9VL0bXEaYqVz1IfvxU+Sabx yooLhAHJ8cjL2gi6B98b9wMizcZILZYfWNH/gndI1ikMd1C1bTnLv93GT9Fo7ZI= X-Gm-Gg: ASbGncsDXJE8JbrLOrhSUc1L6HakHlpOBd7wr5WFdwPuOAeBq3KlgZVaHNqW0vzc46p E+4AqE+YjcolNzGbyoA+0QXO7A8UTTED7cQRjOJq3l3KiKSILiPV/rX9NSHHNxaaEeaBrwFtb9H BUlWWzP0PyUoJpplxWoJZi+6KjiLjtyc7fSnnimhRNEHiF5CPCeJAPPLbFTPZETKIvFXpMRXNA5 xkLYlPndD0SrS+OHvl1Je4se7OKGUP1MpWK2wt1TUVpFiZ3L3nP+bf8b0zetH+sc6ONZ8hQmifQ tbqTo4TXA88ZFcnc+czQmvE5mVEGdyZn X-Google-Smtp-Source: AGHT+IEXD1d9VHmdEfHzDn4iIzGpKroyoIRBm9V8GpOCRUVEQI3tEWlF/wbQq82AYL+CFEewTt8KjA== X-Received: by 2002:a05:6a20:7350:b0:1db:ff9d:1560 with SMTP id adf61e73a8af0-1e1653bb992mr16055868637.18.1733395132079; Thu, 05 Dec 2024 02:38:52 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.48 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:51 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 18/21] riscv: Kconfig: Introduce 64K page size Date: Thu, 5 Dec 2024 18:37:26 +0800 Message-Id: <20241205103729.14798-19-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This patch introduces new config to control whether enabling the 64K base page feature on RISC-V. The 64K config will set software page size as 64K and automatically use svnapot to accelerate basic page mapping. Signed-off-by: Xu Lu --- arch/riscv/Kconfig | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 2c0cb175a92a..592eb5766029 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -167,7 +167,6 @@ config RISCV select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if !LD_IS_LLD select HAVE_MOVE_PMD select HAVE_MOVE_PUD - select HAVE_PAGE_SIZE_4KB select HAVE_PCI select HAVE_PERF_EVENTS select HAVE_PERF_REGS @@ -885,6 +884,28 @@ config RISCV_BOOT_SPINWAIT =20 If unsure what to do here, say N. =20 +choice + prompt "Page size" + default RISCV_4K_PAGES + help + Page size (translation granule) configuration. + +config RISCV_4K_PAGES + bool "4KB" + select HAVE_PAGE_SIZE_4KB + help + This feature enables 4KB pages support. + +config RISCV_64K_PAGES + bool "64KB" + depends on RISCV_ISA_SVNAPOT && 64BIT + select HAVE_PAGE_SIZE_64KB + select RISCV_USE_SW_PAGE + help + This feature enables 64KB pages support. + +endchoice + config ARCH_SUPPORTS_KEXEC def_bool y =20 --=20 2.20.1 From nobody Thu Dec 18 08:12:16 2025 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C0835206F18 for ; Thu, 5 Dec 2024 10:38:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395138; cv=none; b=dHMxPZtO3J4/YIoqk/tTDAeoAdSXpchambXkup1W75UjCFfa2Cb/CC02J9fEB9FaZt5E7DLgI4IV52iYoE0Q4V18/wBhCidduV1NbV6SrD0UzvXM7Xd75APOjfT3YPvkaYa6jzsmnuIS1wTneOPHdFOsLsIk+uQJ/7DN+MVcez8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395138; c=relaxed/simple; bh=fazeYO8J4gmrtAfZJ1PS4e4GzNzk2PD13l8YNImILBE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=baE2octEgf4b1Wss+rVU0S1UvES216dwaYNfzwHol0iyUQ11kKH1niWLX2+/r+E+IiLuDqCdL/haxwxn7Ht34yBWKe2JI9K6bxEbdqT2Hozd8LRGeVMOgoUqm3IvZsIVe4bd26m0J2bqk8IC27u6ZQbgCpwyde16U1Y4FYCjlUA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Prxs7ROf; arc=none smtp.client-ip=209.85.210.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Prxs7ROf" Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-725958d5ee0so745118b3a.1 for ; Thu, 05 Dec 2024 02:38:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395136; x=1733999936; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VfDjQEoTxdpd9iDlK1ps7bFltPFhkJicFLVtDzgiM9U=; b=Prxs7ROfiWPKBzfP6opIbB8o29k/FGsXBUZkXA7gB+CpA9TdE2LOAT4lYgpwmYgYdd ENoQ8TFHpsF78vw/uTDOwcI25b88ZO2X0DpNm4GQ1oDEFZ76bP4DPvlItHqxwnKZKFLX Gy8uWjUN3cJQ/zv+SoTMHTbXu7M5P85g4D/uhZZZ365eGdLYx0rVIMOS/r5KJ4+Es2gb J3l1/BhNzJx9eK24FUpFNVZXNZG+tfa7lzGRWNBYVcbliuegctKm6cP1nJyecT6O41aG CcHIIjJsvhxSGkfwT9LG6f/MFMD6cMRsLNNn5UxnRJ6fe3s1a2RsHU9pilm02FFmF/xg TVWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395136; x=1733999936; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VfDjQEoTxdpd9iDlK1ps7bFltPFhkJicFLVtDzgiM9U=; b=QOl8cEhmaE+7WbV1ONvSZz8/CpV9GwJf9BGJN/rRCQxrHsmKUFUVuqAFuPeW/fr7jd qHD0PO9EcuBbWpsCh6j9cnIdb3bLyYgLbRcDUx1k9JY5iBzv4pjrdbQnAwrArep5eNx1 ztNpTGsPCLAN8D/CvfYFo9Xvu2T6Yk933QJXQO+9E9sG7uBnsVI+G1NTnYTGp7bh5DK3 YU16aEv/EER82EJlV88Wi6TwLRi4CmcLJBbbnqEQ5M+ZKMD4XDma3CxltvAvSt160KKz 9HX+hD9KK5Eg49pIW18DUsZAKJ+tQM1uZjkvYjcgpfO3ia8Pq3q3QQyBTUVIoP22Ep95 yoGg== X-Forwarded-Encrypted: i=1; AJvYcCUJ59XU6qwEHjwnvJZLq0ivuBmb+UZHEXmFfyUVt+vL8swExN/j8rOYlyyHDFCrEeAMVxohfvsKEY+33TU=@vger.kernel.org X-Gm-Message-State: AOJu0Ywg9PzVuK88fw/hP2coVC84KXVJ6zbDPiFrMOWDL8Rcxs6rJJ35 hCFRyBs+6mj2mPJOBDLTt2bvpZq85m6orOy+lVKgP7z32h4peGciE8dOk7LwoZQ= X-Gm-Gg: ASbGnctMb8Klf4XgQWszGs+XCdMugf2958PPnUJC0hsr0ITowFMhdEHNxv2wn2g9nvz dVsosPnkc2ig6k1w48TBU/9/qOjCHg9z2S7IvIOkyO7sHpSrDwp61M4GtmSLw7s7F5qNgxn0Amj 4ofFWBGqJjz6XGbjB7KZhUQB6f6Zn64QeLzz8MbtM4FoGrO9SVSvuzhIV2rwlaymFt9y1SG+goD gIKP5eYku4KDOzQT4nioyMlACwsDqVN8pHCU+F9wMoiunHfDwXltLI/KAtj11rJePOtQOKRvZEF J/tnOXRT7KkWLTDLu7fLVFoYC2R/38X8 X-Google-Smtp-Source: AGHT+IHmLj1p53xhXYUvBEpMrjDW3Wg1vBwEcbP7EM0jf/sz5snayXz0eteUjpLhTi1nk0wWuAYigw== X-Received: by 2002:a05:6a00:1703:b0:725:3bd4:9b52 with SMTP id d2e1a72fcca58-7257fa5a308mr13643770b3a.3.1733395136132; Thu, 05 Dec 2024 02:38:56 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.52 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:55 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 19/21] riscv: Kconfig: Adjust mmap rnd bits for 64K Page Date: Thu, 5 Dec 2024 18:37:27 +0800 Message-Id: <20241205103729.14798-20-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Xu Lu --- arch/riscv/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 592eb5766029..9dfe95a12890 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -253,6 +253,7 @@ config ARCH_MMAP_RND_COMPAT_BITS_MIN # max bits determined by the following formula: # VA_BITS - PAGE_SHIFT - 3 config ARCH_MMAP_RND_BITS_MAX + default 20 if 64BIT && RISCV_64K_PAGES # SV39 based default 24 if 64BIT # SV39 based default 17 =20 --=20 2.20.1 From nobody Thu Dec 18 08:12:16 2025 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CA69B206F29 for ; Thu, 5 Dec 2024 10:39:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395142; cv=none; b=XvPj5C5lSdpnQDfY0VStXRA8/B4yM0qNRNb19M3r/x5FJUapVLo7yvDJDN8phuvC9dQw4Hxn6o4yTJeybvTTERA2Nvl3ytfqz59/2ClfRALPXojiuEXR2LjwbafuEqXGLYfNWpLJDJSbEcS3JxZqbR0qzqVeWy0wrA/4tp9A6Eo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395142; c=relaxed/simple; bh=DpO08fjIGRmuLFgPlCH9M2mTu8N/UKIsbU3j6nxBJ+U=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OJ0NjYBDJbspws1FcCkFaBCiVxePAo5AnGvWCt7iq+ecOgJHsv7RER71Scq5dB9+QzndzX7Yufli9jeJjqTuwZddMgMkDkSzhd1R8OFk7cLtrMMxJycn7BUjZsAO2D9gEeFTLWSOKwibx+gMJjkVTdGTulOItscNJ9EF7C6pUD4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Zto/FeYT; arc=none smtp.client-ip=209.85.215.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Zto/FeYT" Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-7fc22a88bcbso610032a12.2 for ; Thu, 05 Dec 2024 02:39:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395140; x=1733999940; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0HKGaUSlXgW0yAKeFS5JtDgO8P73Aga3Eke367yRUOM=; b=Zto/FeYTvrJ8EPutWjt/0l5LCn+K1XmeorPmPpQoI76MNks1abFQptonsxRVhnq53r XNvZtuQkHOsbO0cH9yzJVxIG429Zicw69pxCi5n+xMsXtQudCwJSrHtU/LEm57IxdzN1 qw2xLf5MhLRy+51pxHPysw417Q3R5+nT0Tl1aWkNowUx6sYIY+ghqtUFWEtlEH5F/gTv FUmTRht/fyFYp1yCPPIsDJLSRyPbduZMkJ75o27mYUzpHguNDI6OC7QCLFg72w+/ngnD LA2cD2QlBJ5hfMzoY520VbTCp53wderlRTAofICMdOkg6c1kGsmQrhJG+bBV9wxaV65T q+Lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395140; x=1733999940; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0HKGaUSlXgW0yAKeFS5JtDgO8P73Aga3Eke367yRUOM=; b=mqpWwgGOVaW09WoKEt/9mMR/YfpsqkX+uXtPqOHSDXRB3olOh2nZ/hxRxdPsHl46N0 EdzSap6tAt2L9fWnmwuQ0oh9Sz9h2yFCioC2U7ggV384qIo14yOcIM3uBvzzhFOHaWaO HgU1LscEdhWPsSm3EtZF6TSYbmVr+yTbicRyUgEYKCTdv+4QjaIPdu6lu1mcwcGl1EX7 2eDnPd0LrJ2TYxpLu/wyjHz3gq7euLkBOt9TWLV7zlXP1vygO11kJf7YGStqARGfmlNw Ala4cif1FjznG4ccosiihhSERS02JCX2xMP8HIVWh8F3LOTxGCcVDrU0B0hy0G1V8sac FwJg== X-Forwarded-Encrypted: i=1; AJvYcCXNrLeRz+M1W2wN+4K+O6696zNDtx+SjpYWmaP0Tw5iCT9G04VxTaZ8TPPVrh0+tlkbByRkeDG0ntxuSRs=@vger.kernel.org X-Gm-Message-State: AOJu0YyLSkfZFAKvDeOlsdtHD3+ixCRlAjO63qurD+IAVhxdKqZQF+IW CbG9idLMwaQVoKHS7EAKqrQo0MCzerQTw0pTj6I/MPibA95fkBPHqH47E5CNcgk= X-Gm-Gg: ASbGncsyq7uIVWk08eXXKgSezE7uePTBnzTgEcNSex3mbJnAnQR35AiD0jXfhXiSp+x UaNfUrHsbJojviG4lrkS7IRPAsIKpVC4UWOUsb0FJrhS98BpTzY9XJWvs4AMxv4lZbykBYanXIE EWxuzcQSrTuYYMiPe/aWemX3+M6lSJ7R2MCzuWGZJVv1VVzCKbNJwNzIJaENaCxX6HPQ4GPX44E Y9j6yksaPkWKnkry+DoY8YOEPq07tQmWYuQjGzWKpVT4ffVFXJmITn1XRaLXl3msQeLIQ+m+/Ej J7pGCfqLx53f+R/hbTP0NKGLQYxarMgI X-Google-Smtp-Source: AGHT+IENYN2wIy2LzaSrsQXEce8aC4H6FioafWLUrw0utp3p27SZ8r2YKo2I/DuxTOOKoa8IMLVh9Q== X-Received: by 2002:a05:6a21:78a6:b0:1e0:d60c:3f49 with SMTP id adf61e73a8af0-1e165413484mr13667504637.43.1733395140137; Thu, 05 Dec 2024 02:39:00 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.56 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:59 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 20/21] riscv: mm: Adjust address space layout and init page table for 64K Page Date: Thu, 5 Dec 2024 18:37:28 +0800 Message-Id: <20241205103729.14798-21-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Xu Lu --- arch/riscv/include/asm/page.h | 6 +++++- arch/riscv/include/asm/pgtable.h | 12 +++++++++++ arch/riscv/mm/init.c | 36 +++++++++++++++++++++++--------- 3 files changed, 43 insertions(+), 11 deletions(-) diff --git a/arch/riscv/include/asm/page.h b/arch/riscv/include/asm/page.h index 9bc908d94c7a..236b0106a1c9 100644 --- a/arch/riscv/include/asm/page.h +++ b/arch/riscv/include/asm/page.h @@ -40,8 +40,12 @@ * By default, CONFIG_PAGE_OFFSET value corresponds to SV57 address space = so * define the PAGE_OFFSET value for SV48 and SV39. */ +#ifdef CONFIG_RISCV_64K_PAGES +#define PAGE_OFFSET_L4 _AC(0xffffa80000000000, UL) +#else #define PAGE_OFFSET_L4 _AC(0xffffaf8000000000, UL) -#define PAGE_OFFSET_L3 _AC(0xffffffd600000000, UL) +#endif /* CONFIG_RISCV_64K_PAGES */ +#define PAGE_OFFSET_L3 _AC(0xffffffd800000000, UL) #else #define PAGE_OFFSET _AC(CONFIG_PAGE_OFFSET, UL) #endif /* CONFIG_64BIT */ diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index 9f347e5eefeb..fbc397c4e1c8 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -147,6 +147,18 @@ static inline unsigned long __page_val_to_pfn(unsigned= long val); #include #endif /* CONFIG_64BIT */ =20 +#define __PMD_SHIFT (PMD_SHIFT - (PAGE_SHIFT - HW_PAGE_SHIFT)) +#define __PMD_SIZE (_AC(1, UL) << __PMD_SHIFT) + +#define __PUD_SHIFT (PUD_SHIFT - (PAGE_SHIFT - HW_PAGE_SHIFT)) +#define __PUD_SIZE (_AC(1, UL) << __PUD_SHIFT) + +#define __P4D_SHIFT (P4D_SHIFT - (PAGE_SHIFT - HW_PAGE_SHIFT)) +#define __P4D_SIZE (_AC(1, UL) << __P4D_SHIFT) + +#define __PGD_SHIFT (PGD_SHIFT - (PAGE_SHIFT - HW_PAGE_SHIFT)) +#define __PGD_SIZE (_AC(1, UL) << __PGD_SHIFT) + #include =20 #ifdef CONFIG_XIP_KERNEL diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 678b892b4ed8..2c6b7ea33009 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -695,15 +695,15 @@ static uintptr_t __meminit best_map_size(phys_addr_t = pa, uintptr_t va, phys_addr return PAGE_SIZE; =20 if (pgtable_l5_enabled && - !(pa & (P4D_SIZE - 1)) && !(va & (P4D_SIZE - 1)) && size >=3D P4D_SIZ= E) + !(pa & (__P4D_SIZE - 1)) && !(va & (P4D_SIZE - 1)) && size >=3D P4D_S= IZE) return P4D_SIZE; =20 if (pgtable_l4_enabled && - !(pa & (PUD_SIZE - 1)) && !(va & (PUD_SIZE - 1)) && size >=3D PUD_SIZ= E) + !(pa & (__PUD_SIZE - 1)) && !(va & (PUD_SIZE - 1)) && size >=3D PUD_S= IZE) return PUD_SIZE; =20 if (IS_ENABLED(CONFIG_64BIT) && - !(pa & (PMD_SIZE - 1)) && !(va & (PMD_SIZE - 1)) && size >=3D PMD_SIZ= E) + !(pa & (__PMD_SIZE - 1)) && !(va & (PMD_SIZE - 1)) && size >=3D PMD_S= IZE) return PMD_SIZE; =20 return PAGE_SIZE; @@ -937,17 +937,33 @@ static void __init create_kernel_page_table(pgd_t *pg= dir, PMD_SIZE, PAGE_KERNEL); } #else + +#ifdef CONFIG_RISCV_64K_PAGES +/* TODO: better implementation */ +#define KERNEL_MAP_STEP PAGE_SIZE +#else +#define KERNEL_MAP_STEP PMD_SIZE +#endif + static void __init create_kernel_page_table(pgd_t *pgdir, bool early) { uintptr_t va, end_va; =20 end_va =3D kernel_map.virt_addr + kernel_map.size; - for (va =3D kernel_map.virt_addr; va < end_va; va +=3D PMD_SIZE) - create_pgd_mapping(pgdir, va, - kernel_map.phys_addr + (va - kernel_map.virt_addr), - PMD_SIZE, - early ? - PAGE_KERNEL_EXEC : pgprot_from_va(va)); + if (early) + for (va =3D kernel_map.virt_addr; va < end_va; va +=3D PMD_SIZE) + create_pgd_mapping(pgdir, va, + kernel_map.phys_addr + (va - kernel_map.virt_addr), + PMD_SIZE, + early ? + PAGE_KERNEL_EXEC : pgprot_from_va(va)); + else + for (va =3D kernel_map.virt_addr; va < end_va; va +=3D KERNEL_MAP_STEP) + create_pgd_mapping(pgdir, va, + kernel_map.phys_addr + (va - kernel_map.virt_addr), + KERNEL_MAP_STEP, + early ? + PAGE_KERNEL_EXEC : pgprot_from_va(va)); } #endif =20 @@ -1138,7 +1154,7 @@ asmlinkage void __init setup_vm(uintptr_t dtb_pa) =20 /* Sanity check alignment and size */ BUG_ON((PAGE_OFFSET % PGDIR_SIZE) !=3D 0); - BUG_ON((kernel_map.phys_addr % PMD_SIZE) !=3D 0); + BUG_ON((kernel_map.phys_addr % __PMD_SIZE) !=3D 0); =20 #ifdef CONFIG_64BIT /* --=20 2.20.1 From nobody Thu Dec 18 08:12:16 2025 Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com [209.85.215.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C17AB207641 for ; Thu, 5 Dec 2024 10:39:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395146; cv=none; b=lLy3VfUpX9mE7oKloP9il0qexSvvrPTK9zEy9uqu/QySPrBzYmthhwidp5bCL/MODlJ88XwC5dSdcnpK1XlhhiIvSBiA4cGTf7u8cwEZ7rjldzpgU3cuCE0nEHZJS3i/WwU28XvLHZIOf6RoL6LjmOWaDbX4AXG4TVtWe6aJdUs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395146; c=relaxed/simple; bh=U36m+IR6a76y9eKhACyf0YSUXrTJZS7C34HEKpBkfgU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=R4ngkSZMK+Wvjpbv0VQPvk7DsECyDRnvVNVl1S0wZN/IeUNwGyxQvnw3qb/gpi0qscvHDuYTeDwFLrF8uauEInVFqKXd+F/EjkjvX4Uk41jTRguvkUQlLSXFiOSthwJlRiM3xuzG0ibWYAtZAcDK5BoW1VaKFS2QNzB5KOxoojU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=GKbOJGor; arc=none smtp.client-ip=209.85.215.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="GKbOJGor" Received: by mail-pg1-f174.google.com with SMTP id 41be03b00d2f7-7ea8de14848so538363a12.2 for ; Thu, 05 Dec 2024 02:39:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395144; x=1733999944; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iDBjWSESYKBIcxm/6NEJ0fQX3zIek6Dl7/vbsKDZ5GU=; b=GKbOJGor1wcBGJZwL9QcDfKDLmhE2PvOUAbkSSqywM3Zs1mysvDAetn4+pyiuPskZ1 d8q/MOyq4ryj4OWGodv1iTfdfcq7A8c4z2KmTFKI2qSU/XjjRCgbcev4gqZwiP9LbUM1 Ls65SSMdRCY7vKa4YRKvgx6IUWod+WvnAHmvIjTTdiiidoRgZTMhlaOv8QXTRGGqq91r +2yK3a94JhXgXWsmsYO2gKdVCOeQROQGFTare3p3rx7ZlryQim3e6En7iBkQKR/qLvPN 6GPixlwBK7ioVKXRcDHJgGZIComZwvDpS7/Xrh2LZ4AleyriUH0CRrGXZqyCwBwuM1Hq ZUNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395144; x=1733999944; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iDBjWSESYKBIcxm/6NEJ0fQX3zIek6Dl7/vbsKDZ5GU=; b=F3OtxMp5YG3F8DiFWDDkSu+6NWtQJfQ0ztNvtbO/fsb7S/6HL5Bj3y5m7EOGegLlsh 1V1lURhbpNKKeMTqe70CqFBK/zGngSNXdE6xSulsFa6nsAUJKwYX3bbhUyRhay6MVSxF 7jwtHCF4OgnGK8DVmHf/J1fSsvywze/1Dk3y8cJTNLod450PwTMVDMwcDtVgTSB4pBzP g3FgpYdGFldLfQoLzcfxZKZ5R4ATdwlSwRYt32iOZlxPjj+pXTb+0R5LXvVrkmjjj0wc /6ZaogfobFOuTDkqhzCDKzSkLyk4fgMLP7PLRNFdYsqRLX5JOaddZYFQE7r4YYny06WE vOUw== X-Forwarded-Encrypted: i=1; AJvYcCXOwryA74KHPzkfTpbByrtzUofm6VzuVMImJTwg9SOHgGvFN/5mK3X0Q86bnaMJuxppoBjY/pm5jvDdNB4=@vger.kernel.org X-Gm-Message-State: AOJu0YzoabZ+xM9yiUMADcnXcnKXVsTprQKSBxEnMU3vUllvLE+9hgvO 5XcwX/M6VjzhsanD44elHSpQ3a5a8Xb/87mpkHoZ2woAO7UAlFelP8xIG36r/tI= X-Gm-Gg: ASbGncufbP20u+W6mGpZhQ6w0r5Iy5eZ4GiYV93h52JNAStP7jGktlITCUZL6LUJoxQ zxrJSruUEtktlZoOnAGnve6YhSMqoeay1qhWIfFCwR2SmgjFsrV9bi4ZCy4j1XJz1YKGnfeoO54 7G9OR9wFopSgU4qO2gWaGcyNhtts1WosjPxi+033un6X3etNH6pbsezpDwXqP0nKbW1/3j2WxJE q1Jz0RiIBW6cA5RFH2xps4/DmFq0YBZzqvbQFaLuA/Cya+LkGkb8He6fh2GAW09ivueCQKKqxUy mPOXeAFO4Gl/f6cl9/TpVaDI3pWkzXcA X-Google-Smtp-Source: AGHT+IHKuqD7zayvSRrPEZJa2a5jeQV4cpoc2Wz2fLw8d8qFILc/CQ/vzSdEu/3MiaJEA8awaTzYNQ== X-Received: by 2002:a05:6a20:3d87:b0:1db:d8a5:7c82 with SMTP id adf61e73a8af0-1e16539f381mr15601577637.5.1733395144181; Thu, 05 Dec 2024 02:39:04 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.39.00 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:39:03 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 21/21] riscv: mm: Update EXEC_PAGESIZE for 64K Page Date: Thu, 5 Dec 2024 18:37:29 +0800 Message-Id: <20241205103729.14798-22-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Xu Lu --- arch/riscv/include/uapi/asm/param.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 arch/riscv/include/uapi/asm/param.h diff --git a/arch/riscv/include/uapi/asm/param.h b/arch/riscv/include/uapi/= asm/param.h new file mode 100644 index 000000000000..1221e570a077 --- /dev/null +++ b/arch/riscv/include/uapi/asm/param.h @@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* + * Copyright (C) 2024 RISCV Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ +#ifndef __ASM_PARAM_H +#define __ASM_PARAM_H + +#define EXEC_PAGESIZE 65536 + +#include + +#endif --=20 2.20.1