From nobody Sat Oct 4 00:32:11 2025 Received: from mail-qt1-f182.google.com (mail-qt1-f182.google.com [209.85.160.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7E24C299957 for ; Fri, 22 Aug 2025 17:47:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755884851; cv=none; b=IBd4oAe66vttuo3ZW/nDmNAmmYWgqf7RcjoPMwThLk7u948Dbq+OtocjhWsb1BXVvxo3yhGMoE7TBycYeZ7T7ZJ3vQ2GaEZehjWX8HzkUuD4XVqq+NjIQ4a1I66DkRCTD55ajGtCYFt4uh+zFgKAQPMolqGonh+BvRlrIXpYOBc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755884851; c=relaxed/simple; bh=IK2qrbCbw7QWIgjpGS8EycDQVc+7c6/YFfOZ+vU3rQA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=c1Ku70rx0sEprrJMbSg3/u0sQ9MmUqzH2JcLWFVfN6h0nRgwmdy8FzWU8smPVuwTrK0KatW3DnUrBuqimxvsUVt8htBbyF1gf9qASSQJyFseIJ+ALjSyKU7E5p2NYL+lPW6w3XnqdH7ysazGFeTQNbUKeFV69LdqwcsVTkx3maY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc.com header.i=@rivosinc.com header.b=IJTJ8YIN; arc=none smtp.client-ip=209.85.160.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc.com header.i=@rivosinc.com header.b="IJTJ8YIN" Received: by mail-qt1-f182.google.com with SMTP id d75a77b69052e-4b297962b20so21366711cf.2 for ; Fri, 22 Aug 2025 10:47:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc.com; s=google; t=1755884848; x=1756489648; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fOsch9BTYSqQ7yHbVB1d7sGG+vsty5tD44+7UPMzgWI=; b=IJTJ8YIN2C3eV4lZF1SLVjtZAlQ/SrMdWD+ztPy+l1nF04J1QZSeTBJBPzxDvfTv1B iekG2gshkHpwbHvWJ+f/kAwdy0/tGUTtuSzZAHlzGtdYe8NeAOohBqFaje6C5cnJlhoM kp++iIRpM4khpRXWOUQD6nLDmBhCuBXYLx7u1V+JI+v+hwAvilx7fHgdfAOnR60bCvvr phDd2UYYtoeRl9ZxovHhRA5m2KU1V2NR4Qu9uUnzPymXdEaj7LpA8730fwnZLuBaD3GZ EcD+Q2e8G5yDBAx8IU3QOqVwZDflVqphtweszwRFMgeMjGE28G/H7VMHPElytjyy/rSQ Zmgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755884848; x=1756489648; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fOsch9BTYSqQ7yHbVB1d7sGG+vsty5tD44+7UPMzgWI=; b=bofSJr+LyR1JIOZ72gTtfeeYM270zbaxxKVBFYIY+DHpKWJSfx8qBxuuOAIC1zYkMY v1OLqGUJ2g3o4VxB94GHZVdzI9pqlCLgB26vJ3etmFuVXx9rde1treKibraeh4hzBAdQ UC+XGGhXOuZQwpHEePKky4ik8q1TKNTWbnLJlDwGmrZI0lY0OzGwL/UHdtg5MlJujJHK EQvoaK1slFMNUh5RWaO77lFw2QXWsyXqkK+0jowVW3O8qmoAj/3dCS1Dh1DaOYT+0ykP zE5uivvhYfgTBy8Kf+SsRGT7KAU9KRW+oxn/+Hw5UNL0x6tyGbIZJQag6ZVkhhe9fNzm EKgw== X-Forwarded-Encrypted: i=1; AJvYcCWnEvrjakBl51xdMmDwuO18gaKOebFz9mQK+sLfg6AIbRVEFrAEGHITuUgSkPNhnMlFhSjPiewLtI10Gho=@vger.kernel.org X-Gm-Message-State: AOJu0Yxlu8C04y/CqP4s79l5mcXuGU0ZRjXO+4cOQMRepuD/3Cp39idF PuWynLUNjsh7+3Cc+KNgQgaa/qDv5LB1WcTysGthpNoZsPf6ZdDLTYxffMT6cJtwqLc= X-Gm-Gg: ASbGncsnR+M6v3DK6WAZjPWG98DK54RKcRPuu6ITRhYalcmyZh3q8CCgVDudvPe013U HZlHItKH6ggP6ifY39Xok5jGMq29uFI9QAsgG/PlpNg71Dl7E/t3EUmB73dBS/49Hhz5aeqKfIg 2J3Y/Kquk5md3BF2kQ65aYzZ0L6wp8foqTB6KEQIcxIh65WytdKnYYLtxht8M6+jeDlDqbsJ4r5 lCkK5y0ZvlPOWmGH2ESJ/epfvkPNf0gpQSvb3nJy7Muwnq/V9PILes0C4EAqVCGmopRYS80qb58 NLW1jM0sMN+Sc1SSbqOYYCg6CYBQD79b6CtEjIMTKDYurdHafw9noF5krWbvzGGC+/Z3abQJjcE tZg74SpUxq6L3Hu8qz4BrK8PUUTJtwBBRIwhoucjjfB3/I49sSSgbkFjJAsGJSDGTKLxMzUtFGR bLEo5DhQ== X-Google-Smtp-Source: AGHT+IHPnKj86bXuhSb+aVxY48yHR1TxfG62VZSUdYoVpxtQs3hsSZOq4XaawpO4CeN8AVMNRCbEEw== X-Received: by 2002:a05:622a:558d:b0:4b0:aa78:d770 with SMTP id d75a77b69052e-4b2aab3d156mr48000291cf.49.1755884847368; Fri, 22 Aug 2025 10:47:27 -0700 (PDT) Received: from jesse-lt.jtp-bos.lab (pool-108-26-215-125.bstnma.fios.verizon.net. [108.26.215.125]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4b2b8e6023asm3121361cf.53.2025.08.22.10.47.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Aug 2025 10:47:27 -0700 (PDT) From: Jesse Taube To: linux-riscv@lists.infradead.org Cc: Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexandre Ghiti , Oleg Nesterov , Kees Cook , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Namhyung Kim , Mark Rutland , Alexander Shishkin , Jiri Olsa , Ian Rogers , Adrian Hunter , "Liang Kan" , Shuah Khan , Jesse Taube , Himanshu Chauhan , Charlie Jenkins , Samuel Holland , Conor Dooley , Deepak Gupta , Andrew Jones , Atish Patra , Anup Patel , Mayuresh Chitale , Evan Green , WangYuli , Huacai Chen , Arnd Bergmann , Andrew Morton , Luis Chamberlain , "Mike Rapoport (Microsoft)" , Nam Cao , Yunhui Cui , Joel Granados , =?UTF-8?q?Cl=C3=A9ment=20L=C3=A9ger?= , Sebastian Andrzej Siewior , Celeste Liu , Chunyan Zhang , Nylon Chen , Thomas Gleixner , =?UTF-8?q?Thomas=20Wei=C3=9Fschuh?= , Vincenzo Frascino , Joey Gouly , Ravi Bangoria , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-perf-users@vger.kernel.org, linux-kselftest@vger.kernel.org, Joel Stanley Subject: [PATCH 4/8] riscv: Introduce support for hardware break/watchpoints Date: Fri, 22 Aug 2025 10:47:11 -0700 Message-ID: <20250822174715.1269138-5-jesse@rivosinc.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250822174715.1269138-1-jesse@rivosinc.com> References: <20250822174715.1269138-1-jesse@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Himanshu Chauhan RISC-V hardware breakpoint framework is built on top of perf subsystem and = uses SBI debug trigger extension to install/uninstall/update/enable/disable hard= ware triggers as specified in Sdtrig ISA extension. Signed-off-by: Himanshu Chauhan Signed-off-by: Jesse Taube --- RFC -> V1: - Add dbtr_mode to rv_init_mcontrol(6)_trigger - Add select HAVE_MIXED_BREAKPOINTS_REGS - Add TDATA1_MCTRL_SZ and TDATA1_MCTRL6_SZ - Capitalize F in Fallback comment - Fix in_callback code to allow multiple breakpoints - Move perf_bp_event above setup_singlestep to save the correct state - Use sbi_err_map_linux_errno for arch_smp_teardown/setup_sbi_shmem V1 -> V2: - No change --- arch/riscv/Kconfig | 2 + arch/riscv/include/asm/hw_breakpoint.h | 59 +++ arch/riscv/include/asm/kdebug.h | 3 +- arch/riscv/include/asm/sbi.h | 4 +- arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/hw_breakpoint.c | 614 +++++++++++++++++++++++++ arch/riscv/kernel/traps.c | 6 + 7 files changed, 687 insertions(+), 2 deletions(-) create mode 100644 arch/riscv/include/asm/hw_breakpoint.h create mode 100644 arch/riscv/kernel/hw_breakpoint.c diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index bbec87b79309..fd8b62cdc6f5 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -163,6 +163,7 @@ config RISCV select HAVE_FUNCTION_ERROR_INJECTION select HAVE_GCC_PLUGINS select HAVE_GENERIC_VDSO if MMU && 64BIT + select HAVE_HW_BREAKPOINT if PERF_EVENTS && RISCV_SBI select HAVE_IRQ_TIME_ACCOUNTING select HAVE_KERNEL_BZIP2 if !XIP_KERNEL && !EFI_ZBOOT select HAVE_KERNEL_GZIP if !XIP_KERNEL && !EFI_ZBOOT @@ -176,6 +177,7 @@ config RISCV select HAVE_KRETPROBES if !XIP_KERNEL # https://github.com/ClangBuiltLinux/linux/issues/1881 select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if !LD_IS_LLD + select HAVE_MIXED_BREAKPOINTS_REGS select HAVE_MOVE_PMD select HAVE_MOVE_PUD select HAVE_PAGE_SIZE_4KB diff --git a/arch/riscv/include/asm/hw_breakpoint.h b/arch/riscv/include/as= m/hw_breakpoint.h new file mode 100644 index 000000000000..cde6688b91d2 --- /dev/null +++ b/arch/riscv/include/asm/hw_breakpoint.h @@ -0,0 +1,59 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2024 Ventana Micro Systems Inc. + */ + +#ifndef __RISCV_HW_BREAKPOINT_H +#define __RISCV_HW_BREAKPOINT_H + +struct task_struct; + +#ifdef CONFIG_HAVE_HW_BREAKPOINT + +#include + +#if __riscv_xlen =3D=3D 64 +#define cpu_to_le cpu_to_le64 +#define le_to_cpu le64_to_cpu +#elif __riscv_xlen =3D=3D 32 +#define cpu_to_le cpu_to_le32 +#define le_to_cpu le32_to_cpu +#else +#error "Unexpected __riscv_xlen" +#endif + +struct arch_hw_breakpoint { + unsigned long address; + unsigned long len; + + /* Callback info */ + unsigned long next_addr; + bool in_callback; + + /* Trigger configuration data */ + unsigned long tdata1; + unsigned long tdata2; + unsigned long tdata3; +}; + +/* Maximum number of hardware breakpoints supported */ +#define RV_MAX_TRIGGERS 32 + +struct perf_event_attr; +struct notifier_block; +struct perf_event; +struct pt_regs; + +int hw_breakpoint_slots(int type); +int arch_check_bp_in_kernelspace(struct arch_hw_breakpoint *hw); +int hw_breakpoint_arch_parse(struct perf_event *bp, + const struct perf_event_attr *attr, + struct arch_hw_breakpoint *hw); +int hw_breakpoint_exceptions_notify(struct notifier_block *unused, + unsigned long val, void *data); +int arch_install_hw_breakpoint(struct perf_event *bp); +void arch_uninstall_hw_breakpoint(struct perf_event *bp); +void hw_breakpoint_pmu_read(struct perf_event *bp); + +#endif /* CONFIG_HAVE_HW_BREAKPOINT */ +#endif /* __RISCV_HW_BREAKPOINT_H */ diff --git a/arch/riscv/include/asm/kdebug.h b/arch/riscv/include/asm/kdebu= g.h index 85ac00411f6e..53e989781aa1 100644 --- a/arch/riscv/include/asm/kdebug.h +++ b/arch/riscv/include/asm/kdebug.h @@ -6,7 +6,8 @@ enum die_val { DIE_UNUSED, DIE_TRAP, - DIE_OOPS + DIE_OOPS, + DIE_DEBUG }; =20 #endif diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index be2ca8e8a49e..64fa7a82aa45 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -282,7 +282,9 @@ struct sbi_sta_struct { u8 pad[47]; } __packed; =20 -#define SBI_SHMEM_DISABLE -1 +#define SBI_SHMEM_DISABLE (-1UL) +#define SBI_SHMEM_LO(pa) ((unsigned long)lower_32_bits(pa)) +#define SBI_SHMEM_HI(pa) ((unsigned long)upper_32_bits(pa)) =20 enum sbi_ext_nacl_fid { SBI_EXT_NACL_PROBE_FEATURE =3D 0x0, diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index 4f719b09e5ad..3e72505734bd 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -99,6 +99,7 @@ obj-$(CONFIG_DYNAMIC_FTRACE) +=3D mcount-dyn.o =20 obj-$(CONFIG_PERF_EVENTS) +=3D perf_callchain.o obj-$(CONFIG_HAVE_PERF_REGS) +=3D perf_regs.o +obj-$(CONFIG_HAVE_HW_BREAKPOINT) +=3D hw_breakpoint.o obj-$(CONFIG_RISCV_SBI) +=3D sbi.o sbi_ecall.o ifeq ($(CONFIG_RISCV_SBI), y) obj-$(CONFIG_SMP) +=3D sbi-ipi.o diff --git a/arch/riscv/kernel/hw_breakpoint.c b/arch/riscv/kernel/hw_break= point.c new file mode 100644 index 000000000000..3f96e744a711 --- /dev/null +++ b/arch/riscv/kernel/hw_breakpoint.c @@ -0,0 +1,614 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2024 Ventana Micro Systems Inc. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#define DBTR_TDATA1_TYPE_SHIFT (__riscv_xlen - 4) +#define DBTR_TDATA1_DMODE BIT_UL(__riscv_xlen - 5) + +#define DBTR_TDATA1_TYPE_MCONTROL (2UL << DBTR_TDATA1_TYPE_SHIFT) +#define DBTR_TDATA1_TYPE_MCONTROL6 (6UL << DBTR_TDATA1_TYPE_SHIFT) + +#define DBTR_TDATA1_MCONTROL6_LOAD BIT(0) +#define DBTR_TDATA1_MCONTROL6_STORE BIT(1) +#define DBTR_TDATA1_MCONTROL6_EXECUTE BIT(2) +#define DBTR_TDATA1_MCONTROL6_U BIT(3) +#define DBTR_TDATA1_MCONTROL6_S BIT(4) +#define DBTR_TDATA1_MCONTROL6_M BIT(6) +#define DBTR_TDATA1_MCONTROL6_SIZE_FIELD GENMASK(18, 16) +#define DBTR_TDATA1_MCONTROL6_SELECT BIT(21) +#define DBTR_TDATA1_MCONTROL6_VU BIT(23) +#define DBTR_TDATA1_MCONTROL6_VS BIT(24) + +#define DBTR_TDATA1_MCONTROL6_SIZE_8BIT 1 +#define DBTR_TDATA1_MCONTROL6_SIZE_16BIT 2 +#define DBTR_TDATA1_MCONTROL6_SIZE_32BIT 3 +#define DBTR_TDATA1_MCONTROL6_SIZE_64BIT 5 + +#define TDATA1_MCTRL6_SZ(sz) \ + FIELD_PREP(DBTR_TDATA1_MCONTROL6_SIZE_FIELD, sz) + +#define DBTR_TDATA1_MCONTROL_LOAD BIT(0) +#define DBTR_TDATA1_MCONTROL_STORE BIT(1) +#define DBTR_TDATA1_MCONTROL_EXECUTE BIT(2) +#define DBTR_TDATA1_MCONTROL_U BIT(3) +#define DBTR_TDATA1_MCONTROL_S BIT(4) +#define DBTR_TDATA1_MCONTROL_M BIT(6) +#define DBTR_TDATA1_MCONTROL_SIZELO_FIELD GENMASK(17, 16) +#define DBTR_TDATA1_MCONTROL_SELECT BIT(19) +#define DBTR_TDATA1_MCONTROL_SIZEHI_FIELD GENMASK(22, 21) + +#define DBTR_TDATA1_MCONTROL_SIZELO_8BIT 1 +#define DBTR_TDATA1_MCONTROL_SIZELO_16BIT 2 +#define DBTR_TDATA1_MCONTROL_SIZELO_32BIT 3 +/* value of 5 split across HI and LO */ +#define DBTR_TDATA1_MCONTROL_SIZELO_64BIT 1 +#define DBTR_TDATA1_MCONTROL_SIZEHI_64BIT 1 + +#define TDATA1_MCTRL_SZ(lo, hi) \ + (FIELD_PREP(DBTR_TDATA1_MCONTROL_SIZELO_FIELD, lo) | \ + FIELD_PREP(DBTR_TDATA1_MCONTROL_SIZEHI_FIELD, hi)) + +enum dbtr_mode { + DBTR_MODE_U =3D 0, + DBTR_MODE_S, + DBTR_MODE_VS, + DBTR_MODE_VU, +}; + +/* Registered per-cpu bp/wp */ +static DEFINE_PER_CPU(struct perf_event *, pcpu_hw_bp_events[RV_MAX_TRIGGE= RS]); +static DEFINE_PER_CPU(unsigned long, ecall_lock_flags); +static DEFINE_PER_CPU(raw_spinlock_t, ecall_lock); + +/* Per-cpu shared memory between S and M mode */ +static DEFINE_PER_CPU(union sbi_dbtr_shmem_entry, sbi_dbtr_shmem); + +/* number of debug triggers on this cpu . */ +static int dbtr_total_num __ro_after_init; +static unsigned long dbtr_type __ro_after_init; +static unsigned long dbtr_init __ro_after_init; + +static int arch_smp_setup_sbi_shmem(unsigned int cpu) +{ + union sbi_dbtr_shmem_entry *dbtr_shmem; + unsigned long shmem_pa; + struct sbiret ret; + int rc; + + dbtr_shmem =3D per_cpu_ptr(&sbi_dbtr_shmem, cpu); + if (!dbtr_shmem) { + pr_err("Invalid per-cpu shared memory for debug triggers\n"); + return -ENODEV; + } + + shmem_pa =3D virt_to_phys(dbtr_shmem); + + ret =3D sbi_ecall(SBI_EXT_DBTR, SBI_EXT_DBTR_SETUP_SHMEM, + SBI_SHMEM_LO(shmem_pa), SBI_SHMEM_HI(shmem_pa), 0, 0, 0, 0); + if (ret.error) { + pr_warn("%s: failed to setup shared memory. error: %ld\n", __func__, ret= .error); + return sbi_err_map_linux_errno(ret.error); + } + + pr_debug("CPU %d: HW Breakpoint shared memory registered.\n", cpu); + + return rc; +} + +static int arch_smp_teardown_sbi_shmem(unsigned int cpu) +{ + struct sbiret ret; + + /* Disable shared memory */ + ret =3D sbi_ecall(SBI_EXT_DBTR, SBI_EXT_DBTR_SETUP_SHMEM, + SBI_SHMEM_DISABLE, SBI_SHMEM_DISABLE, 0, 0, 0, 0); + if (ret.error) { + pr_warn("%s: failed to teardown shared memory. error: %ld\n", __func__, = ret.error); + return sbi_err_map_linux_errno(ret.error); + } + + pr_debug("CPU %d: HW Breakpoint shared memory disabled.\n", cpu); + + return 0; +} + +static void init_sbi_dbtr(void) +{ + struct sbiret ret; + + /* + * Called by hw_breakpoint_slots and arch_hw_breakpoint_init. + * Only proceed if this is the first CPU to reach this code. + */ + if (test_and_set_bit(0, &dbtr_init)) + return; + + if (sbi_probe_extension(SBI_EXT_DBTR) <=3D 0) { + pr_debug("%s: SBI_EXT_DBTR is not supported\n", __func__); + dbtr_total_num =3D 0; + return; + } + + ret =3D sbi_ecall(SBI_EXT_DBTR, SBI_EXT_DBTR_NUM_TRIGGERS, + DBTR_TDATA1_TYPE_MCONTROL6, 0, 0, 0, 0, 0); + if (ret.error) { + pr_warn("%s: failed to detect mcontrol6 triggers. error: %ld.\n", + __func__, ret.error); + } else if (!ret.value) { + pr_warn("%s: No mcontrol6 triggers available.\n", __func__); + } else { + dbtr_total_num =3D ret.value; + dbtr_type =3D DBTR_TDATA1_TYPE_MCONTROL6; + return; + } + + /* Fallback to legacy mcontrol triggers if mcontrol6 is not available */ + ret =3D sbi_ecall(SBI_EXT_DBTR, SBI_EXT_DBTR_NUM_TRIGGERS, + DBTR_TDATA1_TYPE_MCONTROL, 0, 0, 0, 0, 0); + if (ret.error) { + pr_warn("%s: failed to detect mcontrol triggers. error: %ld.\n", + __func__, ret.error); + } else if (!ret.value) { + pr_err("%s: No mcontrol triggers available.\n", __func__); + dbtr_total_num =3D 0; + } else { + dbtr_total_num =3D ret.value; + dbtr_type =3D DBTR_TDATA1_TYPE_MCONTROL; + } +} + +int hw_breakpoint_slots(int type) +{ + /* + * We can be called early, so don't rely on + * static variables being initialised. + */ + init_sbi_dbtr(); + + return dbtr_total_num; +} + +int arch_check_bp_in_kernelspace(struct arch_hw_breakpoint *hw) +{ + unsigned int len; + unsigned long va; + + va =3D hw->address; + len =3D hw->len; + + return (va >=3D TASK_SIZE) && ((va + len - 1) >=3D TASK_SIZE); +} + +static int rv_init_mcontrol_trigger(const struct perf_event_attr *attr, + struct arch_hw_breakpoint *hw, enum dbtr_mode mode) +{ + unsigned long tdata1 =3D DBTR_TDATA1_TYPE_MCONTROL; + + switch (attr->bp_type) { + case HW_BREAKPOINT_X: + tdata1 |=3D DBTR_TDATA1_MCONTROL_EXECUTE; + break; + case HW_BREAKPOINT_R: + tdata1 |=3D DBTR_TDATA1_MCONTROL_LOAD; + break; + case HW_BREAKPOINT_W: + tdata1 |=3D DBTR_TDATA1_MCONTROL_STORE; + break; + case HW_BREAKPOINT_RW: + tdata1 |=3D DBTR_TDATA1_MCONTROL_STORE | DBTR_TDATA1_MCONTROL_LOAD; + break; + default: + return -EINVAL; + } + + switch (attr->bp_len) { + case HW_BREAKPOINT_LEN_1: + hw->len =3D 1; + tdata1 |=3D TDATA1_MCTRL_SZ(DBTR_TDATA1_MCONTROL_SIZELO_8BIT, 0); + break; + case HW_BREAKPOINT_LEN_2: + hw->len =3D 2; + tdata1 |=3D TDATA1_MCTRL_SZ(DBTR_TDATA1_MCONTROL_SIZELO_16BIT, 0); + break; + case HW_BREAKPOINT_LEN_4: + hw->len =3D 4; + tdata1 |=3D TDATA1_MCTRL_SZ(DBTR_TDATA1_MCONTROL_SIZELO_32BIT, 0); + break; +#if __riscv_xlen >=3D 64 + case HW_BREAKPOINT_LEN_8: + hw->len =3D 8; + tdata1 |=3D TDATA1_MCTRL_SZ(DBTR_TDATA1_MCONTROL_SIZELO_64BIT, + DBTR_TDATA1_MCONTROL_SIZEHI_64BIT); + break; +#endif + default: + return -EINVAL; + } + + switch (mode) { + case DBTR_MODE_U: + tdata1 |=3D DBTR_TDATA1_MCONTROL_U; + break; + case DBTR_MODE_S: + tdata1 |=3D DBTR_TDATA1_MCONTROL_S; + break; + default: + return -EINVAL; + } + + hw->tdata1 =3D tdata1; + + return 0; +} + +static int rv_init_mcontrol6_trigger(const struct perf_event_attr *attr, + struct arch_hw_breakpoint *hw, enum dbtr_mode mode) +{ + unsigned long tdata1 =3D DBTR_TDATA1_TYPE_MCONTROL; + + switch (attr->bp_type) { + case HW_BREAKPOINT_X: + tdata1 |=3D DBTR_TDATA1_MCONTROL6_EXECUTE; + break; + case HW_BREAKPOINT_R: + tdata1 |=3D DBTR_TDATA1_MCONTROL6_LOAD; + break; + case HW_BREAKPOINT_W: + tdata1 |=3D DBTR_TDATA1_MCONTROL6_STORE; + break; + case HW_BREAKPOINT_RW: + tdata1 |=3D DBTR_TDATA1_MCONTROL6_STORE | DBTR_TDATA1_MCONTROL6_LOAD; + break; + default: + return -EINVAL; + } + + switch (attr->bp_len) { + case HW_BREAKPOINT_LEN_1: + hw->len =3D 1; + tdata1 |=3D TDATA1_MCTRL6_SZ(DBTR_TDATA1_MCONTROL6_SIZE_8BIT); + break; + case HW_BREAKPOINT_LEN_2: + hw->len =3D 2; + tdata1 |=3D TDATA1_MCTRL6_SZ(DBTR_TDATA1_MCONTROL6_SIZE_16BIT); + break; + case HW_BREAKPOINT_LEN_4: + hw->len =3D 4; + tdata1 |=3D TDATA1_MCTRL6_SZ(DBTR_TDATA1_MCONTROL6_SIZE_32BIT); + break; + case HW_BREAKPOINT_LEN_8: + hw->len =3D 8; + tdata1 |=3D TDATA1_MCTRL6_SZ(DBTR_TDATA1_MCONTROL6_SIZE_64BIT); + break; + default: + return -EINVAL; + } + + switch (mode) { + case DBTR_MODE_U: + tdata1 |=3D DBTR_TDATA1_MCONTROL6_U; + break; + case DBTR_MODE_S: + tdata1 |=3D DBTR_TDATA1_MCONTROL6_S; + break; + case DBTR_MODE_VS: + tdata1 |=3D DBTR_TDATA1_MCONTROL6_VS; + break; + case DBTR_MODE_VU: + tdata1 |=3D DBTR_TDATA1_MCONTROL6_VU; + break; + default: + return -EINVAL; + } + + hw->tdata1 =3D tdata1; + + return 0; +} + +int hw_breakpoint_arch_parse(struct perf_event *bp, + const struct perf_event_attr *attr, + struct arch_hw_breakpoint *hw) +{ + int ret; + + /* Breakpoint address */ + hw->address =3D attr->bp_addr; + hw->tdata2 =3D attr->bp_addr; + hw->tdata3 =3D 0x0; + hw->next_addr =3D 0x0; + hw->in_callback =3D false; + + switch (dbtr_type) { + case DBTR_TDATA1_TYPE_MCONTROL: + ret =3D rv_init_mcontrol_trigger(attr, hw, DBTR_MODE_U); + break; + case DBTR_TDATA1_TYPE_MCONTROL6: + ret =3D rv_init_mcontrol6_trigger(attr, hw, DBTR_MODE_U); + break; + default: + pr_warn("Unsupported trigger type %lu.\n", dbtr_type >> DBTR_TDATA1_TYPE= _SHIFT); + ret =3D -EOPNOTSUPP; + break; + } + + return ret; +} + +/** + * setup_singlestep - Update breakpoint to next instruction after breakpoi= nt. + * @event: Perf event to change + * @regs: regs at breakpoint + * + * Update breakpoint to next instruction that would have + * executed after the current breakpoint. + * + * This allows for single-stepping the instruction being debugged. + * Then restoring the original breakpoint. + * + * Returns Zero on success, negative on failure + */ +static int setup_singlestep(struct perf_event *event, struct pt_regs *regs) +{ + struct arch_hw_breakpoint *bp =3D counter_arch_bp(event); + struct perf_event_attr bp_insn; + unsigned long insn, next_addr =3D 0; + int ret; + + /* Remove breakpoint even if return error as not to loop */ + arch_uninstall_hw_breakpoint(event); + + ret =3D get_insn_nofault(regs, regs->epc, &insn); + if (ret < 0) + return ret; + + next_addr =3D get_step_address(regs, insn); + + ret =3D get_insn_nofault(regs, next_addr, &insn); + if (ret < 0) + return ret; + + bp_insn.bp_type =3D HW_BREAKPOINT_X; + bp_insn.bp_addr =3D next_addr; + /* Get the size of the intruction */ + bp_insn.bp_len =3D GET_INSN_LENGTH(insn); + + ret =3D hw_breakpoint_arch_parse(NULL, &bp_insn, bp); + if (ret) + return ret; + + ret =3D arch_install_hw_breakpoint(event); + if (ret) + return ret; + + bp->in_callback =3D true; + bp->next_addr =3D next_addr; + return 0; +} + +/* + * HW Breakpoint/watchpoint handler + */ +static int hw_breakpoint_handler(struct pt_regs *regs) +{ + int i, ret =3D 0, bp_ret =3D NOTIFY_DONE; + bool expecting_callback =3D false; + struct arch_hw_breakpoint *bp; + struct perf_event *event; + + for (i =3D 0; i < dbtr_total_num; i++) { + event =3D this_cpu_read(pcpu_hw_bp_events[i]); + if (!event) + continue; + + bp =3D counter_arch_bp(event); + switch (event->attr.bp_type) { + /* Breakpoint */ + case HW_BREAKPOINT_X: + if (event->attr.bp_addr =3D=3D regs->epc) { + perf_bp_event(event, regs); + ret =3D setup_singlestep(event, regs); + if (ret < 0) { + pr_err("%s: setup_singlestep failed %d.\n", __func__, ret); + goto exit; + } + + bp_ret =3D NOTIFY_STOP; + goto exit; + } + break; + + /* Watchpoint */ + case HW_BREAKPOINT_W: + case HW_BREAKPOINT_R: + case HW_BREAKPOINT_RW: + /* Watchpoints will trigger on smaller loads than the given type. + * To allow for this, check if the load was within the size of + * the type. Cast badaddr to the type of bp_addr. + */ + if (abs_diff(event->attr.bp_addr, (__u64)regs->badaddr) < bp->len) { + perf_bp_event(event, regs); + ret =3D setup_singlestep(event, regs); + if (ret < 0) { + pr_err("%s: setup_singlestep failed %d.\n", __func__, ret); + goto exit; + } + + bp_ret =3D NOTIFY_STOP; + goto exit; + } + break; + + default: + pr_warn("%s: Unknown type: %u\n", __func__, event->attr.bp_type); + goto exit; + } + + if (bp->in_callback) { + expecting_callback =3D true; + if (regs->epc !=3D bp->next_addr) { + continue; + } + + arch_uninstall_hw_breakpoint(event); + /* Restore original breakpoint */ + if (hw_breakpoint_arch_parse(NULL, &event->attr, bp)) + goto exit; + if (arch_install_hw_breakpoint(event)) + goto exit; + + bp_ret =3D NOTIFY_STOP; + goto exit; + } + + } + + if (expecting_callback) { + pr_err("%s: in_callback was set, but epc (%lx) was not at next address(%= lx).\n", + __func__, regs->epc, bp->next_addr); + } +exit: + return bp_ret; + +} + +int hw_breakpoint_exceptions_notify(struct notifier_block *unused, + unsigned long val, void *data) +{ + struct die_args *args =3D data; + + if (val !=3D DIE_DEBUG) + return NOTIFY_DONE; + + return hw_breakpoint_handler(args->regs); +} + +/* atomic: counter->ctx->lock is held */ +int arch_install_hw_breakpoint(struct perf_event *event) +{ + struct arch_hw_breakpoint *bp =3D counter_arch_bp(event); + union sbi_dbtr_shmem_entry *shmem =3D this_cpu_ptr(&sbi_dbtr_shmem); + struct sbi_dbtr_data_msg *xmit; + struct sbi_dbtr_id_msg *recv; + struct perf_event **slot; + unsigned long idx; + struct sbiret ret; + int err =3D 0; + + raw_spin_lock_irqsave(this_cpu_ptr(&ecall_lock), + *this_cpu_ptr(&ecall_lock_flags)); + + xmit =3D &shmem->data; + recv =3D &shmem->id; + xmit->tdata1 =3D cpu_to_le(bp->tdata1); + xmit->tdata2 =3D cpu_to_le(bp->tdata2); + xmit->tdata3 =3D cpu_to_le(bp->tdata3); + + ret =3D sbi_ecall(SBI_EXT_DBTR, SBI_EXT_DBTR_TRIG_INSTALL, + 1, 0, 0, 0, 0, 0); + + if (ret.error) { + pr_warn("%s: failed to install trigger. error: %ld\n", __func__, ret.err= or); + err =3D sbi_err_map_linux_errno(ret.error); + goto done; + } + + idx =3D le_to_cpu(recv->idx); + if (idx >=3D dbtr_total_num) { + pr_warn("%s: invalid trigger index %lu\n", __func__, idx); + err =3D -EINVAL; + goto done; + } + + slot =3D this_cpu_ptr(&pcpu_hw_bp_events[idx]); + if (*slot) { + pr_warn("%s: slot %lu is in use\n", __func__, idx); + err =3D -EBUSY; + goto done; + } + + pr_debug("Trigger 0x%lx installed at index 0x%lx\n", bp->tdata2, idx); + + /* Save the event - to be looked up in handler */ + *slot =3D event; + +done: + raw_spin_unlock_irqrestore(this_cpu_ptr(&ecall_lock), + *this_cpu_ptr(&ecall_lock_flags)); + return err; +} + +void arch_uninstall_hw_breakpoint(struct perf_event *event) +{ + struct perf_event **slot; + struct sbiret ret; + int i; + + for (i =3D 0; i < dbtr_total_num; i++) { + slot =3D this_cpu_ptr(&pcpu_hw_bp_events[i]); + + if (*slot =3D=3D event) { + *slot =3D NULL; + break; + } + } + + if (i =3D=3D dbtr_total_num) { + pr_warn("%s: Breakpoint not installed.\n", __func__); + return; + } + + ret =3D sbi_ecall(SBI_EXT_DBTR, SBI_EXT_DBTR_TRIG_UNINSTALL, + i, 1, 0, 0, 0, 0); + if (ret.error) + pr_warn("%s: Failed to uninstall trigger %d. error: %ld\n", __func__, i,= ret.error); +} + +void flush_ptrace_hw_breakpoint(struct task_struct *tsk) { } + +void hw_breakpoint_pmu_read(struct perf_event *bp) { } + +static int __init arch_hw_breakpoint_init(void) +{ + unsigned int cpu; + int rc =3D 0; + + for_each_possible_cpu(cpu) + raw_spin_lock_init(&per_cpu(ecall_lock, cpu)); + + init_sbi_dbtr(); + + if (dbtr_total_num) { + pr_debug("%s: total number of type %lu triggers: %u\n", + __func__, dbtr_type >> DBTR_TDATA1_TYPE_SHIFT, dbtr_total_num); + } else { + pr_debug("%s: No hardware triggers available\n", __func__); + return rc; + } + + /* Hotplug handler to register/unregister shared memory with SBI */ + rc =3D cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, + "riscv/hw_breakpoint:prepare", + arch_smp_setup_sbi_shmem, + arch_smp_teardown_sbi_shmem); + + if (rc < 0) + pr_warn("%s: Failed to setup CPU hotplug state\n", __func__); + + return rc; +} +arch_initcall(arch_hw_breakpoint_init); diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c index 938a8b841f94..2ac471ec79a8 100644 --- a/arch/riscv/kernel/traps.c +++ b/arch/riscv/kernel/traps.c @@ -289,6 +289,12 @@ void handle_break(struct pt_regs *regs) if (probe_breakpoint_handler(regs)) return; =20 +#ifdef CONFIG_HAVE_HW_BREAKPOINT + if (notify_die(DIE_DEBUG, "EBREAK", regs, 0, regs->cause, SIGTRAP) + =3D=3D NOTIFY_STOP) + return; +#endif + current->thread.bad_cause =3D regs->cause; =20 if (user_mode(regs)) --=20 2.43.0