From nobody Wed Dec 10 20:12:36 2025 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1BFC62F5318 for ; Thu, 13 Nov 2025 01:47:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762998452; cv=none; b=jn2s/8csM54GyzZsilJnNvtaJLO6+IyTR4363Nd3BRSqUcSZn0SvmSunSp4wW5mZuSMZtQRlWpAvzbf2m2avucNVOrK58XZXQOe0CGyPEwesW9kcWTXMrtgCT/VJtLhOXsc9xvLNhoRfa0uccObL8dSlBV7CXuSsFTsvdre7Xfk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762998452; c=relaxed/simple; bh=N6A8T4qywKGtmfh/1yC9rPFZpHNOQTVOB86CbX19NA4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qYds0OuaAvXnTBja5jicBJnlcmmwxrlffJV5VyTG2Qjg+wOn/H+wayrLeNSQGsRQr4E5E3JXboIV6K2CCupuDsUtqDQ5/3byDpXzKLrTa4tpdb9VCP/irSwBquwcvihN59a/3Vq78HWoHQNjVIj6vVsr5rjAidTBHXkmLoEQYUw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=SnkybLZr; arc=none smtp.client-ip=209.85.215.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="SnkybLZr" Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-b9a5b5b47bfso166937a12.1 for ; Wed, 12 Nov 2025 17:47:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1762998449; x=1763603249; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HOtRFITo02H4IQSZeZ1BamFbY19pkW2KaAlb2iNDjWw=; b=SnkybLZr4cuB3MzSHMTd8M7U8ScTjoh/thnAfEEai7BaphFjfR57o29d2lYiXnmBa+ GQk/MfRFBhUr7RXuYDMhR3o2+DK4RLRcprQI324CmQMuoTC3pIRQ3+bbj9ale2AZYKsJ w5mEPjYTL6aVJ0mOPtr5rT/Z2xW2Np+63Gc+0XJB8WMuIU2anIwgV5V5tv0C9Km1Pyl4 Rb/mp+y2AsGl39nGdatbaMcrT5RKAmbwgvJtOlw+y2v2Jz1KZuQJyDsH0VyfBXYduA2v iWhsbCStY2RTunU0uJVZCv/q1YNFlbaf3C7xYRp1PNG52APCBf5KrgcW9jHB21vY/2wX rVsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762998449; x=1763603249; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=HOtRFITo02H4IQSZeZ1BamFbY19pkW2KaAlb2iNDjWw=; b=ir6isaO5+S+9WVo2vl+m7MYRfG2/EPbGhhbiunj2gWa3txbYYF/Z4cGBUFypn4h6T6 +K66nJdPQbsB6BPZHTL6njr5SxXPurQbTwzpbGdIk3/ukUd0gdVlCHyVG6ikbbQJWw+i binHhiBcjJ63jng+y/bKnQARpzjrKDGADYEsJijmKQmNkTfm6utZVvlsGVCFt6hr0YGW NRUH2LYcpyM6O0g4q9mhIlrvrec7M5/0+bOQx+hHk5z5m1V7oFm4t0IE5HuVeH9FN/oW sucZrhzCUyL8luMJSbrwBvxDQEiefttEeNih5E3Z6QFYydRJmqrFaokm6nCYaoULhdMl a3NQ== X-Forwarded-Encrypted: i=1; AJvYcCVDbH+57s8vaCbv8nDYIBESHJh+w8oIlElmGXAdaFcfROiWJRPjuyXNkI/GPWGdom8VEeF7o3uV5+h9fKk=@vger.kernel.org X-Gm-Message-State: AOJu0YzTZOXVHBNyqJz22TeHzu4Lk2lXg7RvHZUdvxRysoqiFDT4ZBvX uGSwfFxaw2Rsql9HAXRxYtZ+4TQPTs+OysFpkiVcPTfnFyLSBU1OpNsmY2oquGrAhFA= X-Gm-Gg: ASbGncvaUxaf/5ZRPzoaNbhv2ic8TOG4cT46Ah5gYRdvZBntTN9nl+toy5HTZ3J69Lr uT8/6aayX53RiWLs723qKe4nt+UmpIT71f5VgJgNq9RktIIQ8K7+BSIBUKzDxLA0pZaBlQvR7vS CR0Go1TxlU4xSNdIXnQLUv0kxfNArVcBtz5VPKtxZhZMXfzgwMG1Mk5TbwYPUhh0zr9qXzOFkAV LcphUmFfo6JKiX9GMK3X7PrkIHmctutQh1HtuA819CMoK74vhWqc2d/mDM/ePJ9+jCVZmCPDVou 8QDGVDPafJKO+mGLESc9dPIv7x3stXKYEhlByzsUxN0JaQqn0hS3JeUkFyCGVQLwXtJfn27/fSo oHdImP8pOevbbE1INZrpji9ifqeyq0P5bmYCglE0gFkrabIDQTwzXmKVfSLt7wT23zwQMfwYpVO vgrI1fiAxhc/65VK96E4YXNw== X-Google-Smtp-Source: AGHT+IGSKyxeN0VSp7kzP7NfJ86/9I9ANN0lDslbNbt/QqR2hftPu3Zqd6k0IFT90ar8rvdWpbD7XQ== X-Received: by 2002:a17:902:f788:b0:298:49db:a9c5 with SMTP id d9443c01a7336-2984edd243amr57875185ad.43.1762998449290; Wed, 12 Nov 2025 17:47:29 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2985c2ccae8sm4986485ad.98.2025.11.12.17.47.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Nov 2025 17:47:29 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , Paul Walmsley , linux-riscv@lists.infradead.org, Andrew Morton , David Hildenbrand , linux-mm@kvack.org Cc: devicetree@vger.kernel.org, Suren Baghdasaryan , linux-kernel@vger.kernel.org, Mike Rapoport , Michal Hocko , Conor Dooley , Lorenzo Stoakes , Krzysztof Kozlowski , Alexandre Ghiti , Emil Renner Berthing , Rob Herring , Vlastimil Babka , "Liam R . Howlett" , Samuel Holland Subject: [PATCH v3 20/22] riscv: mm: Use physical memory aliases to apply PMAs Date: Wed, 12 Nov 2025 17:45:33 -0800 Message-ID: <20251113014656.2605447-21-samuel.holland@sifive.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20251113014656.2605447-1-samuel.holland@sifive.com> References: <20251113014656.2605447-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" On some RISC-V platforms, RAM is mapped simultaneously to multiple physical address ranges, with each alias having a different set of statically-determined Physical Memory Attributes (PMAs). Software alters the PMAs for a particular page at runtime by selecting a PFN from among the aliases of that page's physical address. Implement this by transforming the PFN when writing page tables. If the memory type field is nonzero, replace the PFN with the corresponding PFN from the noncached alias. Similarly, when reading from the page tables, if the PFN is found in a noncached alias, replace it with the PFN from the normal memory alias, and insert _PAGE_NOCACHE. The rest of the kernel sees only PFNs from normal memory and _PAGE_MTMASK values as if Svpbmt was implemented. Memory alias pairs are determined from the devicetree. A Linux custom ISA extension is added to trigger the alternative patching, as alternatives must be linked to an extension or a vendor erratum, and this behavior is not associated with any particular processor vendor. Signed-off-by: Samuel Holland --- Changes in v3: - Fix the logic to allow an alias to be paired with region entry 0 - Select DMA_DIRECT_REMAP Changes in v2: - Put new code behind a new Kconfig option RISCV_ISA_XLINUXMEMALIAS - Document the calling convention of riscv_fixup/unfix_memory_alias() - Do not transform !pte_present() (e.g. swap) PTEs - Export riscv_fixup/unfix_memory_alias() to fix module compilation arch/riscv/Kconfig | 17 ++++ arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/include/asm/pgtable-64.h | 44 +++++++-- arch/riscv/include/asm/pgtable-bits.h | 5 +- arch/riscv/include/asm/pgtable.h | 8 ++ arch/riscv/kernel/cpufeature.c | 6 ++ arch/riscv/kernel/setup.c | 1 + arch/riscv/mm/Makefile | 1 + arch/riscv/mm/memory-alias.S | 123 ++++++++++++++++++++++++++ arch/riscv/mm/pgtable.c | 91 +++++++++++++++++++ arch/riscv/mm/ptdump.c | 6 +- 11 files changed, 291 insertions(+), 12 deletions(-) create mode 100644 arch/riscv/mm/memory-alias.S diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index cf5a4b5cdcd4..21efa0d9bdd4 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -877,6 +877,23 @@ config TOOLCHAIN_NEEDS_OLD_ISA_SPEC versions of clang and GCC to be passed to GAS, which has the same result as passing zicsr and zifencei to -march. =20 +config RISCV_ISA_XLINUXMEMALIAS + bool "Use physical memory aliases to emulate page-based memory types" + depends on 64BIT && MMU + depends on RISCV_ALTERNATIVE + default y + select DMA_DIRECT_REMAP + help + Add support for the kernel to alter the Physical Memory Attributes + (PMAs) of a page at runtime by selecting from among the aliases of + that page in the physical address space. + + On systems where physical memory aliases are present, this option + is required in order to mark pages as non-cacheable for use with + non-coherent DMA devices. + + If you don't know what to do here, say Y. + config FPU bool "FPU support" default y diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index affd63e11b0a..6c6349fe15a7 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -107,6 +107,7 @@ #define RISCV_ISA_EXT_ZALRSC 98 #define RISCV_ISA_EXT_ZICBOP 99 =20 +#define RISCV_ISA_EXT_XLINUXMEMALIAS 126 #define RISCV_ISA_EXT_XLINUXENVCFG 127 =20 #define RISCV_ISA_EXT_MAX 128 diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/p= gtable-64.h index fa2c1dcb6f72..f1ecd022e3ee 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -97,7 +97,8 @@ enum napot_cont_order { #define HUGE_MAX_HSTATE 2 #endif =20 -#if defined(CONFIG_RISCV_ISA_SVPBMT) || defined(CONFIG_ERRATA_THEAD_MAE) +#if defined(CONFIG_RISCV_ISA_SVPBMT) || defined(CONFIG_RISCV_ISA_XLINUXMEM= ALIAS) || \ + defined(CONFIG_ERRATA_THEAD_MAE) =20 /* * ALT_FIXUP_MT @@ -107,6 +108,9 @@ enum napot_cont_order { * * On systems that support Svpbmt, the memory type bits are left alone. * + * On systems that support XLinuxMemalias, PTEs with a nonzero memory type= have + * the memory type bits cleared and the PFN replaced with the matching ali= as. + * * On systems that support XTheadMae, a Svpbmt memory type is transformed * into the corresponding XTheadMae memory type. * @@ -129,22 +133,35 @@ enum napot_cont_order { */ =20 #define ALT_FIXUP_MT(_val) \ - asm(ALTERNATIVE_2("addi t0, zero, 0x3\n\t" \ + asm(ALTERNATIVE_3("addi t0, zero, 0x3\n\t" \ "slli t0, t0, 61\n\t" \ "not t0, t0\n\t" \ "and %0, %0, t0\n\t" \ "nop\n\t" \ "nop\n\t" \ + "nop\n\t" \ "nop", \ - __nops(7), \ + __nops(8), \ 0, RISCV_ISA_EXT_SVPBMT, CONFIG_RISCV_ISA_SVPBMT, \ + "addi t0, zero, 0x3\n\t" \ + "slli t0, t0, 61\n\t" \ + "and t0, %0, t0\n\t" \ + "beqz t0, 2f\n\t" \ + "xor t1, %0, t0\n\t" \ + "1: auipc t0, %%pcrel_hi(riscv_fixup_memory_alias)\n\t" \ + "jalr t0, t0, %%pcrel_lo(1b)\n\t" \ + "mv %0, t1\n" \ + "2:", \ + 0, RISCV_ISA_EXT_XLINUXMEMALIAS, \ + CONFIG_RISCV_ISA_XLINUXMEMALIAS, \ "srli t0, %0, 59\n\t" \ "seqz t1, t0\n\t" \ "slli t1, t1, 1\n\t" \ "or t0, t0, t1\n\t" \ "xori t0, t0, 0x5\n\t" \ "slli t0, t0, 60\n\t" \ - "xor %0, %0, t0", \ + "xor %0, %0, t0\n\t" \ + "nop", \ THEAD_VENDOR_ID, ERRATA_THEAD_MAE, CONFIG_ERRATA_THEAD_MAE) \ : "+r" (_val) :: "t0", "t1") =20 @@ -152,9 +169,9 @@ enum napot_cont_order { =20 #define ALT_FIXUP_MT(_val) =20 -#endif /* CONFIG_RISCV_ISA_SVPBMT || CONFIG_ERRATA_THEAD_MAE */ +#endif /* CONFIG_RISCV_ISA_SVPBMT || CONFIG_RISCV_ISA_XLINUXMEMALIAS || CO= NFIG_ERRATA_THEAD_MAE */ =20 -#if defined(CONFIG_ERRATA_THEAD_MAE) +#if defined(CONFIG_RISCV_ISA_XLINUXMEMALIAS) || defined(CONFIG_ERRATA_THEA= D_MAE) =20 /* * ALT_UNFIX_MT @@ -162,6 +179,9 @@ enum napot_cont_order { * On systems that support Svpbmt, or do not support any form of page-based * memory type configuration, the memory type bits are left alone. * + * On systems that support XLinuxMemalias, PTEs with an aliased PFN have t= he + * matching memory type set and the PFN replaced with the normal memory al= ias. + * * On systems that support XTheadMae, the XTheadMae memory type (or zero) = is * transformed back into the corresponding Svpbmt memory type. * @@ -172,7 +192,15 @@ enum napot_cont_order { */ =20 #define ALT_UNFIX_MT(_val) \ - asm(ALTERNATIVE(__nops(6), \ + asm(ALTERNATIVE_2(__nops(6), \ + "mv t1, %0\n\t" \ + "1: auipc t0, %%pcrel_hi(riscv_unfix_memory_alias)\n\t" \ + "jalr t0, t0, %%pcrel_lo(1b)\n\t" \ + "mv %0, t1\n\t" \ + "nop\n\t" \ + "nop", \ + 0, RISCV_ISA_EXT_XLINUXMEMALIAS, \ + CONFIG_RISCV_ISA_XLINUXMEMALIAS, \ "srli t0, %0, 60\n\t" \ "andi t0, t0, 0xd\n\t" \ "srli t1, t0, 1\n\t" \ @@ -236,7 +264,7 @@ static inline pgd_t pgdp_get(pgd_t *pgdp) =20 #define ALT_UNFIX_MT(_val) =20 -#endif /* CONFIG_ERRATA_THEAD_MAE */ +#endif /* CONFIG_RISCV_ISA_XLINUXMEMALIAS || CONFIG_ERRATA_THEAD_MAE */ =20 static inline int pud_present(pud_t pud) { diff --git a/arch/riscv/include/asm/pgtable-bits.h b/arch/riscv/include/asm= /pgtable-bits.h index 18c50cbd78bf..4586917b2d98 100644 --- a/arch/riscv/include/asm/pgtable-bits.h +++ b/arch/riscv/include/asm/pgtable-bits.h @@ -38,7 +38,8 @@ #define _PAGE_PFN_MASK GENMASK(31, 10) #endif /* CONFIG_64BIT */ =20 -#if defined(CONFIG_RISCV_ISA_SVPBMT) || defined(CONFIG_ERRATA_THEAD_MAE) +#if defined(CONFIG_RISCV_ISA_SVPBMT) || defined(CONFIG_RISCV_ISA_XLINUXMEM= ALIAS) || \ + defined(CONFIG_ERRATA_THEAD_MAE) /* * [62:61] Svpbmt Memory Type definitions: * @@ -54,7 +55,7 @@ #define _PAGE_NOCACHE 0 #define _PAGE_IO 0 #define _PAGE_MTMASK 0 -#endif /* CONFIG_RISCV_ISA_SVPBMT || CONFIG_ERRATA_THEAD_MAE */ +#endif /* CONFIG_RISCV_ISA_SVPBMT || CONFIG_RISCV_ISA_XLINUXMEMALIAS || CO= NFIG_ERRATA_THEAD_MAE */ =20 #ifdef CONFIG_RISCV_ISA_SVNAPOT #define _PAGE_NAPOT_SHIFT 63 diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index 8b622f901707..27e8c20af0e2 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -1113,6 +1113,14 @@ extern u64 satp_mode; void paging_init(void); void misc_mem_init(void); =20 +#ifdef CONFIG_RISCV_ISA_XLINUXMEMALIAS +bool __init riscv_have_memory_alias(void); +void __init riscv_init_memory_alias(void); +#else +static inline bool riscv_have_memory_alias(void) { return false; } +static inline void riscv_init_memory_alias(void) {} +#endif /* CONFIG_RISCV_ISA_XLINUXMEMALIAS */ + /* * ZERO_PAGE is a global shared page that is always zero, * used for zero-mapped memory areas, etc. diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 72ca768f4e91..ee59b160e886 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -1093,6 +1093,12 @@ void __init riscv_fill_hwcap(void) riscv_v_setup_vsize(); } =20 + /* Vendor-independent alternatives require a bit in the ISA bitmap. */ + if (riscv_have_memory_alias()) { + set_bit(RISCV_ISA_EXT_XLINUXMEMALIAS, riscv_isa); + pr_info("Using physical memory alias for noncached mappings\n"); + } + memset(print_str, 0, sizeof(print_str)); for (i =3D 0, j =3D 0; i < NUM_ALPHA_EXTS; i++) if (riscv_isa[0] & BIT_MASK(i)) diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index b5bc5fc65cea..a6f821150101 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -357,6 +357,7 @@ void __init setup_arch(char **cmdline_p) } =20 riscv_init_cbo_blocksizes(); + riscv_init_memory_alias(); riscv_fill_hwcap(); apply_boot_alternatives(); init_rt_signal_env(); diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile index b916a68d324a..b4d757226efb 100644 --- a/arch/riscv/mm/Makefile +++ b/arch/riscv/mm/Makefile @@ -33,3 +33,4 @@ endif obj-$(CONFIG_DEBUG_VIRTUAL) +=3D physaddr.o obj-$(CONFIG_RISCV_DMA_NONCOHERENT) +=3D dma-noncoherent.o obj-$(CONFIG_RISCV_NONSTANDARD_CACHE_OPS) +=3D cache-ops.o +obj-$(CONFIG_RISCV_ISA_XLINUXMEMALIAS) +=3D memory-alias.o diff --git a/arch/riscv/mm/memory-alias.S b/arch/riscv/mm/memory-alias.S new file mode 100644 index 000000000000..e37b83d11591 --- /dev/null +++ b/arch/riscv/mm/memory-alias.S @@ -0,0 +1,123 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2024 SiFive + */ + +#include +#include +#include +#include + +#define CACHED_BASE_OFFSET (0 * RISCV_SZPTR) +#define NONCACHED_BASE_OFFSET (1 * RISCV_SZPTR) +#define SIZE_OFFSET (2 * RISCV_SZPTR) + +#define SIZEOF_PAIR (4 * RISCV_SZPTR) + +/* + * Called from ALT_FIXUP_MT with a non-standard calling convention: + * t0 =3D> return address + * t1 =3D> page table entry + * all other registers are callee-saved + */ +SYM_CODE_START(riscv_fixup_memory_alias) + addi sp, sp, -4 * SZREG + REG_S t2, (0 * SZREG)(sp) + REG_S t3, (1 * SZREG)(sp) + REG_S t4, (2 * SZREG)(sp) +#ifdef CONFIG_RISCV_ISA_SVNAPOT + REG_S t5, (3 * SZREG)(sp) + + /* Save and mask off _PAGE_NAPOT if present. */ + li t5, _PAGE_NAPOT + and t5, t1, t5 + xor t1, t1, t5 +#endif + + /* Ignore !pte_present() PTEs, including swap PTEs. */ + andi t2, t1, (_PAGE_PRESENT | _PAGE_PROT_NONE) + beqz t2, .Lfixup_end + + lla t2, memory_alias_pairs +.Lfixup_loop: + REG_L t3, SIZE_OFFSET(t2) + beqz t3, .Lfixup_end + REG_L t4, CACHED_BASE_OFFSET(t2) + sub t4, t1, t4 + bltu t4, t3, .Lfixup_found + addi t2, t2, SIZEOF_PAIR + j .Lfixup_loop + +.Lfixup_found: + REG_L t3, NONCACHED_BASE_OFFSET(t2) + add t1, t3, t4 + +.Lfixup_end: +#ifdef CONFIG_RISCV_ISA_SVNAPOT + xor t1, t1, t5 + + REG_L t5, (3 * SZREG)(sp) +#endif + REG_L t4, (2 * SZREG)(sp) + REG_L t3, (1 * SZREG)(sp) + REG_L t2, (0 * SZREG)(sp) + addi sp, sp, 4 * SZREG + jr t0 +SYM_CODE_END(riscv_fixup_memory_alias) +EXPORT_SYMBOL(riscv_fixup_memory_alias) + +/* + * Called from ALT_UNFIX_MT with a non-standard calling convention: + * t0 =3D> return address + * t1 =3D> page table entry + * all other registers are callee-saved + */ +SYM_CODE_START(riscv_unfix_memory_alias) + addi sp, sp, -4 * SZREG + REG_S t2, (0 * SZREG)(sp) + REG_S t3, (1 * SZREG)(sp) + REG_S t4, (2 * SZREG)(sp) +#ifdef CONFIG_RISCV_ISA_SVNAPOT + REG_S t5, (3 * SZREG)(sp) + + /* Save and mask off _PAGE_NAPOT if present. */ + li t5, _PAGE_NAPOT + and t5, t1, t5 + xor t1, t1, t5 +#endif + + /* Ignore !pte_present() PTEs, including swap PTEs. */ + andi t2, t1, (_PAGE_PRESENT | _PAGE_PROT_NONE) + beqz t2, .Lunfix_end + + lla t2, memory_alias_pairs +.Lunfix_loop: + REG_L t3, SIZE_OFFSET(t2) + beqz t3, .Lunfix_end + REG_L t4, NONCACHED_BASE_OFFSET(t2) + sub t4, t1, t4 + bltu t4, t3, .Lunfix_found + addi t2, t2, SIZEOF_PAIR + j .Lunfix_loop + +.Lunfix_found: + REG_L t3, CACHED_BASE_OFFSET(t2) + add t1, t3, t4 + + /* PFN was in the noncached alias, so mark it as such. */ + li t2, _PAGE_NOCACHE + or t1, t1, t2 + +.Lunfix_end: +#ifdef CONFIG_RISCV_ISA_SVNAPOT + xor t1, t1, t5 + + REG_L t5, (3 * SZREG)(sp) +#endif + REG_L t4, (2 * SZREG)(sp) + REG_L t3, (1 * SZREG)(sp) + REG_L t2, (0 * SZREG)(sp) + addi sp, sp, 4 * SZREG + jr t0 +SYM_CODE_END(riscv_unfix_memory_alias) +EXPORT_SYMBOL(riscv_unfix_memory_alias) diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index 604744d6924f..45f6a0ac22fa 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -1,8 +1,12 @@ // SPDX-License-Identifier: GPL-2.0 =20 #include +#include +#include #include #include +#include +#include #include =20 int ptep_set_access_flags(struct vm_area_struct *vma, @@ -160,3 +164,90 @@ pud_t pudp_invalidate(struct vm_area_struct *vma, unsi= gned long address, return old; } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + +#ifdef CONFIG_RISCV_ISA_XLINUXMEMALIAS +struct memory_alias_pair { + unsigned long cached_base; + unsigned long noncached_base; + unsigned long size; + int index; +} memory_alias_pairs[5]; + +bool __init riscv_have_memory_alias(void) +{ + return memory_alias_pairs[0].size; +} + +void __init riscv_init_memory_alias(void) +{ + int na =3D of_n_addr_cells(of_root); + int ns =3D of_n_size_cells(of_root); + int nc =3D na + ns + 2; + const __be32 *prop; + int pairs =3D 0; + int len; + + prop =3D of_get_property(of_root, "riscv,physical-memory-regions", &len); + if (!prop) + return; + + len /=3D sizeof(__be32); + for (int i =3D 0; len >=3D nc; i++, prop +=3D nc, len -=3D nc) { + unsigned long base =3D of_read_ulong(prop, na); + unsigned long size =3D of_read_ulong(prop + na, ns); + unsigned long flags =3D be32_to_cpup(prop + na + ns); + struct memory_alias_pair *pair; + + /* We only care about non-coherent memory. */ + if ((flags & PMA_ORDER_MASK) !=3D PMA_ORDER_MEMORY || (flags & PMA_COHER= ENT)) + continue; + + /* The cacheable alias must be usable memory. */ + if ((flags & PMA_CACHEABLE) && + !memblock_overlaps_region(&memblock.memory, base, size)) + continue; + + if (flags & PMR_IS_ALIAS) { + int alias =3D FIELD_GET(PMR_ALIAS_MASK, flags); + + pair =3D NULL; + for (int j =3D 0; j < pairs; j++) { + if (alias =3D=3D memory_alias_pairs[j].index) { + pair =3D &memory_alias_pairs[j]; + break; + } + } + if (!pair) + continue; + } else { + /* Leave room for the null sentinel. */ + if (pairs =3D=3D ARRAY_SIZE(memory_alias_pairs) - 1) + continue; + pair =3D &memory_alias_pairs[pairs++]; + pair->index =3D i; + } + + /* Align the address and size with the page table PFN field. */ + base >>=3D PAGE_SHIFT - _PAGE_PFN_SHIFT; + size >>=3D PAGE_SHIFT - _PAGE_PFN_SHIFT; + + if (flags & PMA_CACHEABLE) + pair->cached_base =3D base; + else + pair->noncached_base =3D base; + pair->size =3D min_not_zero(pair->size, size); + } + + /* Remove any unmatched pairs. */ + for (int i =3D 0; i < pairs; i++) { + struct memory_alias_pair *pair =3D &memory_alias_pairs[i]; + + if (pair->cached_base && pair->noncached_base && pair->size) + continue; + + for (int j =3D i + 1; j < pairs; j++) + memory_alias_pairs[j - 1] =3D memory_alias_pairs[j]; + memory_alias_pairs[--pairs].size =3D 0; + } +} +#endif /* CONFIG_RISCV_ISA_XLINUXMEMALIAS */ diff --git a/arch/riscv/mm/ptdump.c b/arch/riscv/mm/ptdump.c index 763ffde8ab5e..29a7be14cca5 100644 --- a/arch/riscv/mm/ptdump.c +++ b/arch/riscv/mm/ptdump.c @@ -140,7 +140,8 @@ static const struct prot_bits pte_bits[] =3D { .clear =3D ".", }, { #endif -#if defined(CONFIG_RISCV_ISA_SVPBMT) || defined(CONFIG_ERRATA_THEAD_MAE) +#if defined(CONFIG_RISCV_ISA_SVPBMT) || defined(CONFIG_RISCV_ISA_XLINUXMEM= ALIAS) || \ + defined(CONFIG_ERRATA_THEAD_MAE) .mask =3D _PAGE_MTMASK, .set =3D "MT(%s)", .clear =3D " .. ", @@ -216,7 +217,8 @@ static void dump_prot(struct pg_state *st) if (val) { if (pte_bits[i].mask =3D=3D _PAGE_SOFT) sprintf(s, pte_bits[i].set, val >> 8); -#if defined(CONFIG_RISCV_ISA_SVPBMT) || defined(CONFIG_ERRATA_THEAD_MAE) +#if defined(CONFIG_RISCV_ISA_SVPBMT) || defined(CONFIG_RISCV_ISA_XLINUXMEM= ALIAS) || \ + defined(CONFIG_ERRATA_THEAD_MAE) else if (pte_bits[i].mask =3D=3D _PAGE_MTMASK) { if (val =3D=3D _PAGE_NOCACHE) sprintf(s, pte_bits[i].set, "NC"); --=20 2.47.2