From nobody Mon Jun 15 18:06:37 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 74E492BF3F4 for ; Sun, 12 Apr 2026 17:46:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776015999; cv=none; b=us/KvtfHSRt2etUdJ3LszBHU90S/IEOBbUue0ZJ58RJlmcBhqv8dc2p2W1C/te+UBzYnTxQmF+uyPu4lnkD/txx1e+Ls//oOgvEV1JylAqj/tz6zDlTdhhbYFg58Os6DsOO679UxC6frm0IuMSdw3S3uEov2Jtzzh8oEqWy7eUk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776015999; c=relaxed/simple; bh=yPzOv+v/iy6xa8/ayFrVumlJR/ytdRy6e9zHi3lyxRs=; h=Date:From:To:Cc:Subject:References:Message-ID:Content-Type: MIME-Version; b=tJO4t4yad4W4IB+yVDFnGu7CEro3JH98GJ3TBhezG7r/pgqTDce891j3+FeS2y59a+zEKGF0nE/sSGK0rhkgT8x7RSUyLVONJpkem7mxbyztJnzj6TD/avkRnFp/b316Y6p35hYH+yK2LonDEBBdZqJSlbRTurreqPtTcvMNj2Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=f+pl92vP; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="f+pl92vP" Received: by smtp.kernel.org (Postfix) with ESMTPSA id EF562C19424; Sun, 12 Apr 2026 17:46:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1776015999; bh=yPzOv+v/iy6xa8/ayFrVumlJR/ytdRy6e9zHi3lyxRs=; h=Date:From:To:Cc:Subject:References:From; b=f+pl92vP+kOGdkREroSW6o3MvGUpuXnklhJpocxMbQW3RsDB28GhkO/8eWJDcoB+j ZNhjr343MLslFbcSe94FWDA816AcC5ZehSEyzulj6vPUTQndwNABejIFso3r5gt17X yPfuOjWUlxrng08hINP4S3c4CISuxPWJcQdRjO+PF9ckNueO0DDmavOylXLfAjA7cj u9mHsS8M9aLcY57Dof6pvMkfDuBK8NYhaeE5LGKAfRxlCOWX3ruQ+utr0+8cXf9Vty LB+6/wVTPKQ2wmRh0ZZo/KM9Dx0fSphWTeZHEJEIjyzvarVx1qDsUowCWBH9lJCQd7 xYGELc44iroXw== Date: Sun, 12 Apr 2026 19:46:34 +0200 From: Thomas Gleixner To: Linus Torvalds Cc: linux-kernel@vger.kernel.org, x86@kernel.org Subject: [GIT pull] timers/vdso for v7.1-rc1 References: <177601563477.7932.4081917600853246368.tglx@xen13> Message-ID: <177601564407.7932.9112032204169856652.tglx@xen13> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Linus, please pull the latest timers/vdso branch from: git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git timers-vdso-20= 26-04-12 up to: 7138a8698a39: timens: Use task_lock guard in timens_get*() Update to the VDSO subsystem: - Make the handling of compat functions consistent and more robust - Rework the underlying data store so that it is dynamically allocated, which allows the conversion of the last holdout SPARC64 to the generic VDSO implementation - Rework the SPARC64 VDSO to utilize the generic implementation - Mop up the left overs of the non-generic VDSO support in the core code. - Expand the VDSO selftest and make them more robust - Allow time namespaces to be enabled independently of the generic VDSO support, which was not possible before due to SPARC64 not using it. - Various cleanups and improvements in the related code. Thanks, tglx ------------------> Arnd Bergmann (1): clocksource: Remove ARCH_CLOCKSOURCE_DATA Randy Dunlap (1): vdso/datapage: Correct struct member kernel-doc Thomas Wei=C3=9Fschuh (49): x86/vdso: Use 32-bit CHECKFLAGS for compat vDSO sparc64: vdso: Use 32-bit CHECKFLAGS for compat vDSO s390: Add -m64 to KBUILD_CPPFLAGS powerpc/audit: Directly include unistd_32.h from compat_audit.c asm-generic/bitsperlong.h: Add sanity checks for __BITS_PER_LONG vdso/datastore: Reduce scope of some variables in vvar_fault() vdso/datastore: Drop inclusion of linux/mmap_lock.h vdso/datastore: Allocate data pages dynamically sparc64: vdso: Link with -z noexecstack sparc64: vdso: Remove obsolete "fake section table" reservation sparc64: vdso: Replace code patching with runtime conditional sparc64: vdso: Move hardware counter read into header sparc64: vdso: Move syscall fallbacks into header sparc64: vdso: Introduce vdso/processor.h sparc64: vdso: Switch to the generic vDSO library sparc64: vdso2c: Drop sym_vvar_start handling sparc64: vdso2c: Remove symbol handling sparc64: vdso: Implement clock_gettime64() vdso/gettimeofday: Drop a few usages of __maybe_unused vdso/gettimeofday: Add a helper to read the sequence lock of a time n= amespace aware clock vdso/gettimeofday: Add a helper to test if a clock is namespaced vdso/gettimeofday: Move the unlikely() into vdso_read_retry() arm64: vDSO: gettimeofday: Explicitly include vdso/clocksource.h arm64: vDSO: compat_gettimeofday: Add explicit includes ARM: vdso: gettimeofday: Add explicit includes powerpc/vdso/gettimeofday: Explicitly include vdso/time32.h powerpc/vdso: Explicitly include asm/cputable.h and asm/feature-fixup= s.h LoongArch: vDSO: Explicitly include asm/vdso/vdso.h MIPS: vdso: Add include guard to asm/vdso/vdso.h MIPS: vdso: Explicitly include asm/vdso/vdso.h random: vDSO: Add explicit includes vdso/gettimeofday: Add explicit includes vdso/helpers: Explicitly include vdso/processor.h vdso/datapage: Remove inclusion of gettimeofday.h vdso/datapage: Trim down unnecessary includes random: vDSO: Trim vDSO includes random: vDSO: Remove ifdeffery Revert "selftests: vDSO: parse_vdso: Use UAPI headers instead of libc= headers" selftests: vDSO: vdso_test_gettimeofday: Remove nolibc checks selftests: vDSO: vdso_test_correctness: Drop SYS_getcpu fallbacks selftests: vDSO: vdso_test_correctness: Handle different tv_usec types selftests: vDSO: vdso_test_correctness: Use facilities from parse_vds= o.c selftests: vDSO: vdso_test_correctness: Add a test for time() vdso/timens: Move functions to new file timens: Remove dependency on the vDSO timens: Add a __free() wrapper for put_time_ns() timens: Simplify some calls to put_time_ns() timens: Use mutex guard in proc_timens_set_offset() timens: Use task_lock guard in timens_get*() MAINTAINERS | 2 + arch/arm/include/asm/vdso/gettimeofday.h | 2 + arch/arm64/include/asm/vdso/compat_gettimeofday.h | 3 + arch/arm64/include/asm/vdso/gettimeofday.h | 2 + arch/loongarch/kernel/process.c | 1 + arch/loongarch/kernel/vdso.c | 1 + arch/mips/include/asm/vdso/vdso.h | 5 + arch/mips/kernel/vdso.c | 1 + arch/powerpc/include/asm/vdso/gettimeofday.h | 1 + arch/powerpc/include/asm/vdso/processor.h | 3 + arch/powerpc/kernel/compat_audit.c | 3 +- arch/s390/Makefile | 3 +- arch/sparc/Kconfig | 3 +- arch/sparc/include/asm/clocksource.h | 9 - arch/sparc/include/asm/processor.h | 3 + arch/sparc/include/asm/processor_32.h | 2 - arch/sparc/include/asm/processor_64.h | 25 -- arch/sparc/include/asm/vdso.h | 2 - arch/sparc/include/asm/vdso/clocksource.h | 10 + arch/sparc/include/asm/vdso/gettimeofday.h | 184 ++++++++++ arch/sparc/include/asm/vdso/processor.h | 41 +++ arch/sparc/include/asm/vdso/vsyscall.h | 10 + arch/sparc/include/asm/vvar.h | 75 ---- arch/sparc/kernel/Makefile | 1 - arch/sparc/kernel/time_64.c | 6 +- arch/sparc/kernel/vdso.c | 69 ---- arch/sparc/vdso/Makefile | 11 +- arch/sparc/vdso/vclock_gettime.c | 380 ++---------------= ---- arch/sparc/vdso/vdso-layout.lds.S | 26 +- arch/sparc/vdso/vdso.lds.S | 2 - arch/sparc/vdso/vdso2c.c | 24 -- arch/sparc/vdso/vdso2c.h | 45 +-- arch/sparc/vdso/vdso32/vdso32.lds.S | 4 +- arch/sparc/vdso/vma.c | 274 +-------------- arch/x86/entry/vdso/vdso32/Makefile | 4 + drivers/char/random.c | 16 +- include/asm-generic/bitsperlong.h | 9 + include/linux/clocksource.h | 6 +- include/linux/time_namespace.h | 39 ++- include/linux/vdso_datastore.h | 6 + include/vdso/datapage.h | 27 +- include/vdso/helpers.h | 31 +- init/Kconfig | 4 +- init/main.c | 2 + kernel/time/Kconfig | 4 - kernel/time/Makefile | 1 + kernel/time/namespace.c | 203 ++--------- kernel/time/namespace_internal.h | 28 ++ kernel/time/namespace_vdso.c | 160 +++++++++ lib/vdso/datastore.c | 122 +++---- lib/vdso/getrandom.c | 3 + lib/vdso/gettimeofday.c | 99 +++--- tools/testing/selftests/vDSO/Makefile | 6 +- tools/testing/selftests/vDSO/parse_vdso.c | 3 +- .../testing/selftests/vDSO/vdso_test_correctness.c | 112 ++++-- .../selftests/vDSO/vdso_test_gettimeofday.c | 2 - 56 files changed, 829 insertions(+), 1291 deletions(-) create mode 100644 arch/sparc/include/asm/vdso/clocksource.h create mode 100644 arch/sparc/include/asm/vdso/gettimeofday.h create mode 100644 arch/sparc/include/asm/vdso/processor.h create mode 100644 arch/sparc/include/asm/vdso/vsyscall.h delete mode 100644 arch/sparc/include/asm/vvar.h delete mode 100644 arch/sparc/kernel/vdso.c create mode 100644 kernel/time/namespace_internal.h create mode 100644 kernel/time/namespace_vdso.c diff --git a/MAINTAINERS b/MAINTAINERS index 77fdfcb55f06..6ad74a5196d1 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -10768,6 +10768,7 @@ S: Maintained T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git timers/vd= so F: include/asm-generic/vdso/vsyscall.h F: include/vdso/ +F: kernel/time/namespace_vdso.c F: kernel/time/vsyscall.c F: lib/vdso/ F: tools/testing/selftests/vDSO/ @@ -21000,6 +21001,7 @@ F: include/trace/events/timer* F: kernel/time/itimer.c F: kernel/time/posix-* F: kernel/time/namespace.c +F: kernel/time/namespace_vdso.c =20 POWER MANAGEMENT CORE M: "Rafael J. Wysocki" diff --git a/arch/arm/include/asm/vdso/gettimeofday.h b/arch/arm/include/as= m/vdso/gettimeofday.h index 1e9f81639c88..26da5d8621cc 100644 --- a/arch/arm/include/asm/vdso/gettimeofday.h +++ b/arch/arm/include/asm/vdso/gettimeofday.h @@ -11,6 +11,8 @@ #include #include #include +#include +#include #include =20 #define VDSO_HAS_CLOCK_GETRES 1 diff --git a/arch/arm64/include/asm/vdso/compat_gettimeofday.h b/arch/arm64= /include/asm/vdso/compat_gettimeofday.h index 0d513f924321..a03e34b572f1 100644 --- a/arch/arm64/include/asm/vdso/compat_gettimeofday.h +++ b/arch/arm64/include/asm/vdso/compat_gettimeofday.h @@ -7,6 +7,9 @@ =20 #ifndef __ASSEMBLER__ =20 +#include +#include + #include #include #include diff --git a/arch/arm64/include/asm/vdso/gettimeofday.h b/arch/arm64/includ= e/asm/vdso/gettimeofday.h index 3658a757e255..96d2eccd4995 100644 --- a/arch/arm64/include/asm/vdso/gettimeofday.h +++ b/arch/arm64/include/asm/vdso/gettimeofday.h @@ -9,6 +9,8 @@ =20 #ifndef __ASSEMBLER__ =20 +#include + #include #include #include diff --git a/arch/loongarch/kernel/process.c b/arch/loongarch/kernel/proces= s.c index 4ac1c3086152..ac3a0baa5d00 100644 --- a/arch/loongarch/kernel/process.c +++ b/arch/loongarch/kernel/process.c @@ -52,6 +52,7 @@ #include #include #include +#include =20 #ifdef CONFIG_STACKPROTECTOR #include diff --git a/arch/loongarch/kernel/vdso.c b/arch/loongarch/kernel/vdso.c index 0aa10cadb959..8ce8159c10b9 100644 --- a/arch/loongarch/kernel/vdso.c +++ b/arch/loongarch/kernel/vdso.c @@ -18,6 +18,7 @@ =20 #include #include +#include #include #include #include diff --git a/arch/mips/include/asm/vdso/vdso.h b/arch/mips/include/asm/vdso= /vdso.h index 6889e0f2e5db..ef50d33f3439 100644 --- a/arch/mips/include/asm/vdso/vdso.h +++ b/arch/mips/include/asm/vdso/vdso.h @@ -4,6 +4,9 @@ * Author: Alex Smith */ =20 +#ifndef __ASM_VDSO_VDSO_H +#define __ASM_VDSO_VDSO_H + #include #include =20 @@ -70,3 +73,5 @@ static inline void __iomem *get_gic(const struct vdso_tim= e_data *data) #endif /* CONFIG_CLKSRC_MIPS_GIC */ =20 #endif /* __ASSEMBLER__ */ + +#endif /* __ASM_VDSO_VDSO_H */ diff --git a/arch/mips/kernel/vdso.c b/arch/mips/kernel/vdso.c index de096777172f..2fa4df3e46e4 100644 --- a/arch/mips/kernel/vdso.c +++ b/arch/mips/kernel/vdso.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include =20 diff --git a/arch/powerpc/include/asm/vdso/gettimeofday.h b/arch/powerpc/in= clude/asm/vdso/gettimeofday.h index 8ea397e26ad0..a853f853da6c 100644 --- a/arch/powerpc/include/asm/vdso/gettimeofday.h +++ b/arch/powerpc/include/asm/vdso/gettimeofday.h @@ -8,6 +8,7 @@ #include #include #include +#include =20 #define VDSO_HAS_CLOCK_GETRES 1 =20 diff --git a/arch/powerpc/include/asm/vdso/processor.h b/arch/powerpc/inclu= de/asm/vdso/processor.h index c1f3d7aaf3ee..4c6802c3a580 100644 --- a/arch/powerpc/include/asm/vdso/processor.h +++ b/arch/powerpc/include/asm/vdso/processor.h @@ -4,6 +4,9 @@ =20 #ifndef __ASSEMBLER__ =20 +#include +#include + /* Macros for adjusting thread priority (hardware multi-threading) */ #ifdef CONFIG_PPC64 #define HMT_very_low() asm volatile("or 31, 31, 31 # very low priority") diff --git a/arch/powerpc/kernel/compat_audit.c b/arch/powerpc/kernel/compa= t_audit.c index 57b38c592b9f..b4d81a57b2d9 100644 --- a/arch/powerpc/kernel/compat_audit.c +++ b/arch/powerpc/kernel/compat_audit.c @@ -1,7 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 -#undef __powerpc64__ #include -#include +#include =20 #include "audit_32.h" =20 diff --git a/arch/s390/Makefile b/arch/s390/Makefile index d78ad6885ca2..02bc948a4a56 100644 --- a/arch/s390/Makefile +++ b/arch/s390/Makefile @@ -12,8 +12,7 @@ LD_BFD :=3D elf64-s390 KBUILD_LDFLAGS :=3D -m elf64_s390 KBUILD_AFLAGS_MODULE +=3D -fPIC KBUILD_CFLAGS_MODULE +=3D -fPIC -KBUILD_AFLAGS +=3D -m64 -KBUILD_CFLAGS +=3D -m64 +KBUILD_CPPFLAGS +=3D -m64 KBUILD_CFLAGS +=3D -fPIC LDFLAGS_vmlinux :=3D $(call ld-option,-no-pie) extra_tools :=3D relocs diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig index 8699be91fca9..a6b787efc2c4 100644 --- a/arch/sparc/Kconfig +++ b/arch/sparc/Kconfig @@ -104,7 +104,6 @@ config SPARC64 select ARCH_USE_QUEUED_RWLOCKS select ARCH_USE_QUEUED_SPINLOCKS select GENERIC_TIME_VSYSCALL - select ARCH_CLOCKSOURCE_DATA select ARCH_HAS_PTE_SPECIAL select PCI_DOMAINS if PCI select ARCH_HAS_GIGANTIC_PAGE @@ -115,6 +114,8 @@ config SPARC64 select ARCH_SUPPORTS_SCHED_SMT if SMP select ARCH_SUPPORTS_SCHED_MC if SMP select ARCH_HAS_LAZY_MMU_MODE + select HAVE_GENERIC_VDSO + select GENERIC_GETTIMEOFDAY =20 config ARCH_PROC_KCORE_TEXT def_bool y diff --git a/arch/sparc/include/asm/clocksource.h b/arch/sparc/include/asm/= clocksource.h index d63ef224befe..68303ad26eb2 100644 --- a/arch/sparc/include/asm/clocksource.h +++ b/arch/sparc/include/asm/clocksource.h @@ -5,13 +5,4 @@ #ifndef _ASM_SPARC_CLOCKSOURCE_H #define _ASM_SPARC_CLOCKSOURCE_H =20 -/* VDSO clocksources */ -#define VCLOCK_NONE 0 /* Nothing userspace can do. */ -#define VCLOCK_TICK 1 /* Use %tick. */ -#define VCLOCK_STICK 2 /* Use %stick. */ - -struct arch_clocksource_data { - int vclock_mode; -}; - #endif /* _ASM_SPARC_CLOCKSOURCE_H */ diff --git a/arch/sparc/include/asm/processor.h b/arch/sparc/include/asm/pr= ocessor.h index 18295ea625dd..e34de956519a 100644 --- a/arch/sparc/include/asm/processor.h +++ b/arch/sparc/include/asm/processor.h @@ -1,6 +1,9 @@ /* SPDX-License-Identifier: GPL-2.0 */ #ifndef ___ASM_SPARC_PROCESSOR_H #define ___ASM_SPARC_PROCESSOR_H + +#include + #if defined(__sparc__) && defined(__arch64__) #include #else diff --git a/arch/sparc/include/asm/processor_32.h b/arch/sparc/include/asm= /processor_32.h index ba8b70ffec08..a074d313f4f8 100644 --- a/arch/sparc/include/asm/processor_32.h +++ b/arch/sparc/include/asm/processor_32.h @@ -91,8 +91,6 @@ unsigned long __get_wchan(struct task_struct *); extern struct task_struct *last_task_used_math; int do_mathemu(struct pt_regs *regs, struct task_struct *fpt); =20 -#define cpu_relax() barrier() - extern void (*sparc_idle)(void); =20 #endif diff --git a/arch/sparc/include/asm/processor_64.h b/arch/sparc/include/asm= /processor_64.h index 321859454ca4..485070495263 100644 --- a/arch/sparc/include/asm/processor_64.h +++ b/arch/sparc/include/asm/processor_64.h @@ -182,31 +182,6 @@ unsigned long __get_wchan(struct task_struct *task); #define KSTK_EIP(tsk) (task_pt_regs(tsk)->tpc) #define KSTK_ESP(tsk) (task_pt_regs(tsk)->u_regs[UREG_FP]) =20 -/* Please see the commentary in asm/backoff.h for a description of - * what these instructions are doing and how they have been chosen. - * To make a long story short, we are trying to yield the current cpu - * strand during busy loops. - */ -#ifdef BUILD_VDSO -#define cpu_relax() asm volatile("\n99:\n\t" \ - "rd %%ccr, %%g0\n\t" \ - "rd %%ccr, %%g0\n\t" \ - "rd %%ccr, %%g0\n\t" \ - ::: "memory") -#else /* ! BUILD_VDSO */ -#define cpu_relax() asm volatile("\n99:\n\t" \ - "rd %%ccr, %%g0\n\t" \ - "rd %%ccr, %%g0\n\t" \ - "rd %%ccr, %%g0\n\t" \ - ".section .pause_3insn_patch,\"ax\"\n\t"\ - ".word 99b\n\t" \ - "wr %%g0, 128, %%asr27\n\t" \ - "nop\n\t" \ - "nop\n\t" \ - ".previous" \ - ::: "memory") -#endif - /* Prefetch support. This is tuned for UltraSPARC-III and later. * UltraSPARC-I will treat these as nops, and UltraSPARC-II has * a shallower prefetch queue than later chips. diff --git a/arch/sparc/include/asm/vdso.h b/arch/sparc/include/asm/vdso.h index 59e79d35cd73..f08562d10215 100644 --- a/arch/sparc/include/asm/vdso.h +++ b/arch/sparc/include/asm/vdso.h @@ -8,8 +8,6 @@ struct vdso_image { void *data; unsigned long size; /* Always a multiple of PAGE_SIZE */ - - long sym_vvar_start; /* Negative offset to the vvar area */ }; =20 #ifdef CONFIG_SPARC64 diff --git a/arch/sparc/include/asm/vdso/clocksource.h b/arch/sparc/include= /asm/vdso/clocksource.h new file mode 100644 index 000000000000..007aa8ceaf52 --- /dev/null +++ b/arch/sparc/include/asm/vdso/clocksource.h @@ -0,0 +1,10 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ASM_VDSO_CLOCKSOURCE_H +#define __ASM_VDSO_CLOCKSOURCE_H + +/* VDSO clocksources */ +#define VDSO_ARCH_CLOCKMODES \ + VDSO_CLOCKMODE_TICK, \ + VDSO_CLOCKMODE_STICK + +#endif /* __ASM_VDSO_CLOCKSOURCE_H */ diff --git a/arch/sparc/include/asm/vdso/gettimeofday.h b/arch/sparc/includ= e/asm/vdso/gettimeofday.h new file mode 100644 index 000000000000..b0c80c8a28bb --- /dev/null +++ b/arch/sparc/include/asm/vdso/gettimeofday.h @@ -0,0 +1,184 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright 2006 Andi Kleen, SUSE Labs. + */ + +#ifndef _ASM_SPARC_VDSO_GETTIMEOFDAY_H +#define _ASM_SPARC_VDSO_GETTIMEOFDAY_H + +#include +#include + +#include +#include +#include +#include + +#include + +#ifdef CONFIG_SPARC64 +static __always_inline u64 vread_tick(void) +{ + u64 ret; + + __asm__ __volatile__("rd %%tick, %0" : "=3Dr" (ret)); + return ret; +} + +static __always_inline u64 vread_tick_stick(void) +{ + u64 ret; + + __asm__ __volatile__("rd %%asr24, %0" : "=3Dr" (ret)); + return ret; +} +#else +static __always_inline u64 vdso_shift_ns(u64 val, u32 amt) +{ + u64 ret; + + __asm__ __volatile__("sllx %H1, 32, %%g1\n\t" + "srl %L1, 0, %L1\n\t" + "or %%g1, %L1, %%g1\n\t" + "srlx %%g1, %2, %L0\n\t" + "srlx %L0, 32, %H0" + : "=3Dr" (ret) + : "r" (val), "r" (amt) + : "g1"); + return ret; +} +#define vdso_shift_ns vdso_shift_ns + +static __always_inline u64 vread_tick(void) +{ + register unsigned long long ret asm("o4"); + + __asm__ __volatile__("rd %%tick, %L0\n\t" + "srlx %L0, 32, %H0" + : "=3Dr" (ret)); + return ret; +} + +static __always_inline u64 vread_tick_stick(void) +{ + register unsigned long long ret asm("o4"); + + __asm__ __volatile__("rd %%asr24, %L0\n\t" + "srlx %L0, 32, %H0" + : "=3Dr" (ret)); + return ret; +} +#endif + +static __always_inline u64 __arch_get_hw_counter(s32 clock_mode, const str= uct vdso_time_data *vd) +{ + if (likely(clock_mode =3D=3D VDSO_CLOCKMODE_STICK)) + return vread_tick_stick(); + else + return vread_tick(); +} + +#ifdef CONFIG_SPARC64 +#define SYSCALL_STRING \ + "ta 0x6d;" \ + "bcs,a 1f;" \ + " sub %%g0, %%o0, %%o0;" \ + "1:" +#else +#define SYSCALL_STRING \ + "ta 0x10;" \ + "bcs,a 1f;" \ + " sub %%g0, %%o0, %%o0;" \ + "1:" +#endif + +#define SYSCALL_CLOBBERS \ + "f0", "f1", "f2", "f3", "f4", "f5", "f6", "f7", \ + "f8", "f9", "f10", "f11", "f12", "f13", "f14", "f15", \ + "f16", "f17", "f18", "f19", "f20", "f21", "f22", "f23", \ + "f24", "f25", "f26", "f27", "f28", "f29", "f30", "f31", \ + "f32", "f34", "f36", "f38", "f40", "f42", "f44", "f46", \ + "f48", "f50", "f52", "f54", "f56", "f58", "f60", "f62", \ + "cc", "memory" + +#ifdef CONFIG_SPARC64 + +static __always_inline +long clock_gettime_fallback(clockid_t clock, struct __kernel_timespec *ts) +{ + register long num __asm__("g1") =3D __NR_clock_gettime; + register long o0 __asm__("o0") =3D clock; + register long o1 __asm__("o1") =3D (long) ts; + + __asm__ __volatile__(SYSCALL_STRING : "=3Dr" (o0) : "r" (num), + "0" (o0), "r" (o1) : SYSCALL_CLOBBERS); + return o0; +} + +#else /* !CONFIG_SPARC64 */ + +static __always_inline +long clock_gettime_fallback(clockid_t clock, struct __kernel_timespec *ts) +{ + register long num __asm__("g1") =3D __NR_clock_gettime64; + register long o0 __asm__("o0") =3D clock; + register long o1 __asm__("o1") =3D (long) ts; + + __asm__ __volatile__(SYSCALL_STRING : "=3Dr" (o0) : "r" (num), + "0" (o0), "r" (o1) : SYSCALL_CLOBBERS); + return o0; +} + +static __always_inline +long clock_gettime32_fallback(clockid_t clock, struct old_timespec32 *ts) +{ + register long num __asm__("g1") =3D __NR_clock_gettime; + register long o0 __asm__("o0") =3D clock; + register long o1 __asm__("o1") =3D (long) ts; + + __asm__ __volatile__(SYSCALL_STRING : "=3Dr" (o0) : "r" (num), + "0" (o0), "r" (o1) : SYSCALL_CLOBBERS); + return o0; +} + +#endif /* CONFIG_SPARC64 */ + +static __always_inline +long gettimeofday_fallback(struct __kernel_old_timeval *tv, struct timezon= e *tz) +{ + register long num __asm__("g1") =3D __NR_gettimeofday; + register long o0 __asm__("o0") =3D (long) tv; + register long o1 __asm__("o1") =3D (long) tz; + + __asm__ __volatile__(SYSCALL_STRING : "=3Dr" (o0) : "r" (num), + "0" (o0), "r" (o1) : SYSCALL_CLOBBERS); + return o0; +} + +static __always_inline const struct vdso_time_data *__arch_get_vdso_u_time= _data(void) +{ + unsigned long ret; + + /* + * SPARC does not support native PC-relative code relocations. + * Calculate the address manually, works for 32 and 64 bit code. + */ + __asm__ __volatile__( + "1:\n" + "call 3f\n" // Jump over the embedded data and set u= p %o7 + "nop\n" // Delay slot + "2:\n" + ".word vdso_u_time_data - .\n" // Embedded offset to external symbol + "3:\n" + "add %%o7, 2b - 1b, %%o7\n" // Point %o7 to the embedded offset + "ldsw [%%o7], %0\n" // Load the offset + "add %0, %%o7, %0\n" // Calculate the absolute address + : "=3Dr" (ret) + : + : "o7"); + + return (const struct vdso_time_data *)ret; +} +#define __arch_get_vdso_u_time_data __arch_get_vdso_u_time_data + +#endif /* _ASM_SPARC_VDSO_GETTIMEOFDAY_H */ diff --git a/arch/sparc/include/asm/vdso/processor.h b/arch/sparc/include/a= sm/vdso/processor.h new file mode 100644 index 000000000000..f7a9adc807f7 --- /dev/null +++ b/arch/sparc/include/asm/vdso/processor.h @@ -0,0 +1,41 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _ASM_SPARC_VDSO_PROCESSOR_H +#define _ASM_SPARC_VDSO_PROCESSOR_H + +#include + +#if defined(__arch64__) + +/* Please see the commentary in asm/backoff.h for a description of + * what these instructions are doing and how they have been chosen. + * To make a long story short, we are trying to yield the current cpu + * strand during busy loops. + */ +#ifdef BUILD_VDSO +#define cpu_relax() asm volatile("\n99:\n\t" \ + "rd %%ccr, %%g0\n\t" \ + "rd %%ccr, %%g0\n\t" \ + "rd %%ccr, %%g0\n\t" \ + ::: "memory") +#else /* ! BUILD_VDSO */ +#define cpu_relax() asm volatile("\n99:\n\t" \ + "rd %%ccr, %%g0\n\t" \ + "rd %%ccr, %%g0\n\t" \ + "rd %%ccr, %%g0\n\t" \ + ".section .pause_3insn_patch,\"ax\"\n\t"\ + ".word 99b\n\t" \ + "wr %%g0, 128, %%asr27\n\t" \ + "nop\n\t" \ + "nop\n\t" \ + ".previous" \ + ::: "memory") +#endif /* BUILD_VDSO */ + +#else /* ! __arch64__ */ + +#define cpu_relax() barrier() + +#endif /* __arch64__ */ + +#endif /* _ASM_SPARC_VDSO_PROCESSOR_H */ diff --git a/arch/sparc/include/asm/vdso/vsyscall.h b/arch/sparc/include/as= m/vdso/vsyscall.h new file mode 100644 index 000000000000..8bfe703fedc5 --- /dev/null +++ b/arch/sparc/include/asm/vdso/vsyscall.h @@ -0,0 +1,10 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _ASM_SPARC_VDSO_VSYSCALL_H +#define _ASM_SPARC_VDSO_VSYSCALL_H + +#define __VDSO_PAGES 4 + +#include + +#endif /* _ASM_SPARC_VDSO_VSYSCALL_H */ diff --git a/arch/sparc/include/asm/vvar.h b/arch/sparc/include/asm/vvar.h deleted file mode 100644 index 6eaf5cfcaae1..000000000000 --- a/arch/sparc/include/asm/vvar.h +++ /dev/null @@ -1,75 +0,0 @@ -/* - * Copyright (c) 2017 Oracle and/or its affiliates. All rights reserved. - */ - -#ifndef _ASM_SPARC_VVAR_DATA_H -#define _ASM_SPARC_VVAR_DATA_H - -#include -#include -#include -#include -#include - -struct vvar_data { - unsigned int seq; - - int vclock_mode; - struct { /* extract of a clocksource struct */ - u64 cycle_last; - u64 mask; - int mult; - int shift; - } clock; - /* open coded 'struct timespec' */ - u64 wall_time_sec; - u64 wall_time_snsec; - u64 monotonic_time_snsec; - u64 monotonic_time_sec; - u64 monotonic_time_coarse_sec; - u64 monotonic_time_coarse_nsec; - u64 wall_time_coarse_sec; - u64 wall_time_coarse_nsec; - - int tz_minuteswest; - int tz_dsttime; -}; - -extern struct vvar_data *vvar_data; -extern int vdso_fix_stick; - -static inline unsigned int vvar_read_begin(const struct vvar_data *s) -{ - unsigned int ret; - -repeat: - ret =3D READ_ONCE(s->seq); - if (unlikely(ret & 1)) { - cpu_relax(); - goto repeat; - } - smp_rmb(); /* Finish all reads before we return seq */ - return ret; -} - -static inline int vvar_read_retry(const struct vvar_data *s, - unsigned int start) -{ - smp_rmb(); /* Finish all reads before checking the value of seq */ - return unlikely(s->seq !=3D start); -} - -static inline void vvar_write_begin(struct vvar_data *s) -{ - ++s->seq; - smp_wmb(); /* Makes sure that increment of seq is reflected */ -} - -static inline void vvar_write_end(struct vvar_data *s) -{ - smp_wmb(); /* Makes the value of seq current before we increment */ - ++s->seq; -} - - -#endif /* _ASM_SPARC_VVAR_DATA_H */ diff --git a/arch/sparc/kernel/Makefile b/arch/sparc/kernel/Makefile index 22170d4f8e06..497b5714fa8f 100644 --- a/arch/sparc/kernel/Makefile +++ b/arch/sparc/kernel/Makefile @@ -41,7 +41,6 @@ obj-$(CONFIG_SPARC32) +=3D systbls_32.o obj-y +=3D time_$(BITS).o obj-$(CONFIG_SPARC32) +=3D windows.o obj-y +=3D cpu.o -obj-$(CONFIG_SPARC64) +=3D vdso.o obj-$(CONFIG_SPARC32) +=3D devices.o obj-y +=3D ptrace_$(BITS).o obj-y +=3D unaligned_$(BITS).o diff --git a/arch/sparc/kernel/time_64.c b/arch/sparc/kernel/time_64.c index b32f27f929d1..87b267043ccd 100644 --- a/arch/sparc/kernel/time_64.c +++ b/arch/sparc/kernel/time_64.c @@ -838,14 +838,14 @@ void __init time_init_early(void) if (tlb_type =3D=3D spitfire) { if (is_hummingbird()) { init_tick_ops(&hbtick_operations); - clocksource_tick.archdata.vclock_mode =3D VCLOCK_NONE; + clocksource_tick.vdso_clock_mode =3D VDSO_CLOCKMODE_NONE; } else { init_tick_ops(&tick_operations); - clocksource_tick.archdata.vclock_mode =3D VCLOCK_TICK; + clocksource_tick.vdso_clock_mode =3D VDSO_CLOCKMODE_TICK; } } else { init_tick_ops(&stick_operations); - clocksource_tick.archdata.vclock_mode =3D VCLOCK_STICK; + clocksource_tick.vdso_clock_mode =3D VDSO_CLOCKMODE_STICK; } } =20 diff --git a/arch/sparc/kernel/vdso.c b/arch/sparc/kernel/vdso.c deleted file mode 100644 index 0e27437eb97b..000000000000 --- a/arch/sparc/kernel/vdso.c +++ /dev/null @@ -1,69 +0,0 @@ -/* - * Copyright (C) 2001 Andrea Arcangeli SuSE - * Copyright 2003 Andi Kleen, SuSE Labs. - * - * Thanks to hpa@transmeta.com for some useful hint. - * Special thanks to Ingo Molnar for his early experience with - * a different vsyscall implementation for Linux/IA32 and for the name. - */ - -#include -#include - -#include - -void update_vsyscall_tz(void) -{ - if (unlikely(vvar_data =3D=3D NULL)) - return; - - vvar_data->tz_minuteswest =3D sys_tz.tz_minuteswest; - vvar_data->tz_dsttime =3D sys_tz.tz_dsttime; -} - -void update_vsyscall(struct timekeeper *tk) -{ - struct vvar_data *vdata =3D vvar_data; - - if (unlikely(vdata =3D=3D NULL)) - return; - - vvar_write_begin(vdata); - vdata->vclock_mode =3D tk->tkr_mono.clock->archdata.vclock_mode; - vdata->clock.cycle_last =3D tk->tkr_mono.cycle_last; - vdata->clock.mask =3D tk->tkr_mono.mask; - vdata->clock.mult =3D tk->tkr_mono.mult; - vdata->clock.shift =3D tk->tkr_mono.shift; - - vdata->wall_time_sec =3D tk->xtime_sec; - vdata->wall_time_snsec =3D tk->tkr_mono.xtime_nsec; - - vdata->monotonic_time_sec =3D tk->xtime_sec + - tk->wall_to_monotonic.tv_sec; - vdata->monotonic_time_snsec =3D tk->tkr_mono.xtime_nsec + - (tk->wall_to_monotonic.tv_nsec << - tk->tkr_mono.shift); - - while (vdata->monotonic_time_snsec >=3D - (((u64)NSEC_PER_SEC) << tk->tkr_mono.shift)) { - vdata->monotonic_time_snsec -=3D - ((u64)NSEC_PER_SEC) << tk->tkr_mono.shift; - vdata->monotonic_time_sec++; - } - - vdata->wall_time_coarse_sec =3D tk->xtime_sec; - vdata->wall_time_coarse_nsec =3D - (long)(tk->tkr_mono.xtime_nsec >> tk->tkr_mono.shift); - - vdata->monotonic_time_coarse_sec =3D - vdata->wall_time_coarse_sec + tk->wall_to_monotonic.tv_sec; - vdata->monotonic_time_coarse_nsec =3D - vdata->wall_time_coarse_nsec + tk->wall_to_monotonic.tv_nsec; - - while (vdata->monotonic_time_coarse_nsec >=3D NSEC_PER_SEC) { - vdata->monotonic_time_coarse_nsec -=3D NSEC_PER_SEC; - vdata->monotonic_time_coarse_sec++; - } - - vvar_write_end(vdata); -} diff --git a/arch/sparc/vdso/Makefile b/arch/sparc/vdso/Makefile index 683b2d408224..83fb2aca59cb 100644 --- a/arch/sparc/vdso/Makefile +++ b/arch/sparc/vdso/Makefile @@ -3,6 +3,9 @@ # Building vDSO images for sparc. # =20 +# Include the generic Makefile to check the built vDSO: +include $(srctree)/lib/vdso/Makefile.include + # files to link into the vdso vobjs-y :=3D vdso-note.o vclock_gettime.o =20 @@ -90,6 +93,9 @@ KBUILD_CFLAGS_32 +=3D -DDISABLE_BRANCH_PROFILING KBUILD_CFLAGS_32 +=3D -mv8plus $(obj)/vdso32.so.dbg: KBUILD_CFLAGS =3D $(KBUILD_CFLAGS_32) =20 +CHECKFLAGS_32 :=3D $(filter-out -m64 -D__sparc_v9__ -D__arch64__, $(CHECKF= LAGS)) -m32 +$(obj)/vdso32.so.dbg: CHECKFLAGS =3D $(CHECKFLAGS_32) + $(obj)/vdso32.so.dbg: FORCE \ $(obj)/vdso32/vdso32.lds \ $(obj)/vdso32/vclock_gettime.o \ @@ -102,6 +108,7 @@ $(obj)/vdso32.so.dbg: FORCE \ quiet_cmd_vdso =3D VDSO $@ cmd_vdso =3D $(LD) -nostdlib -o $@ \ $(VDSO_LDFLAGS) $(VDSO_LDFLAGS_$(filter %.lds,$(^F))) \ - -T $(filter %.lds,$^) $(filter %.o,$^) + -T $(filter %.lds,$^) $(filter %.o,$^); \ + $(cmd_vdso_check) =20 -VDSO_LDFLAGS =3D -shared --hash-style=3Dboth --build-id=3Dsha1 -Bsymbolic = --no-undefined +VDSO_LDFLAGS =3D -shared --hash-style=3Dboth --build-id=3Dsha1 -Bsymbolic = --no-undefined -z noexecstack diff --git a/arch/sparc/vdso/vclock_gettime.c b/arch/sparc/vdso/vclock_gett= ime.c index 79607804ea1b..1d9859392e4c 100644 --- a/arch/sparc/vdso/vclock_gettime.c +++ b/arch/sparc/vdso/vclock_gettime.c @@ -12,382 +12,48 @@ * Copyright (c) 2017 Oracle and/or its affiliates. All rights reserved. */ =20 -#include -#include -#include -#include -#include -#include -#include -#include +#include +#include =20 -#ifdef CONFIG_SPARC64 -#define SYSCALL_STRING \ - "ta 0x6d;" \ - "bcs,a 1f;" \ - " sub %%g0, %%o0, %%o0;" \ - "1:" -#else -#define SYSCALL_STRING \ - "ta 0x10;" \ - "bcs,a 1f;" \ - " sub %%g0, %%o0, %%o0;" \ - "1:" -#endif - -#define SYSCALL_CLOBBERS \ - "f0", "f1", "f2", "f3", "f4", "f5", "f6", "f7", \ - "f8", "f9", "f10", "f11", "f12", "f13", "f14", "f15", \ - "f16", "f17", "f18", "f19", "f20", "f21", "f22", "f23", \ - "f24", "f25", "f26", "f27", "f28", "f29", "f30", "f31", \ - "f32", "f34", "f36", "f38", "f40", "f42", "f44", "f46", \ - "f48", "f50", "f52", "f54", "f56", "f58", "f60", "f62", \ - "cc", "memory" - -/* - * Compute the vvar page's address in the process address space, and retur= n it - * as a pointer to the vvar_data. - */ -notrace static __always_inline struct vvar_data *get_vvar_data(void) -{ - unsigned long ret; - - /* - * vdso data page is the first vDSO page so grab the PC - * and move up a page to get to the data page. - */ - __asm__("rd %%pc, %0" : "=3Dr" (ret)); - ret &=3D ~(8192 - 1); - ret -=3D 8192; - - return (struct vvar_data *) ret; -} +#include =20 -notrace static long vdso_fallback_gettime(long clock, struct __kernel_old_= timespec *ts) -{ - register long num __asm__("g1") =3D __NR_clock_gettime; - register long o0 __asm__("o0") =3D clock; - register long o1 __asm__("o1") =3D (long) ts; +#include =20 - __asm__ __volatile__(SYSCALL_STRING : "=3Dr" (o0) : "r" (num), - "0" (o0), "r" (o1) : SYSCALL_CLOBBERS); - return o0; -} +#include "../../../../lib/vdso/gettimeofday.c" =20 -notrace static long vdso_fallback_gettimeofday(struct __kernel_old_timeval= *tv, struct timezone *tz) +int __vdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *= tz) { - register long num __asm__("g1") =3D __NR_gettimeofday; - register long o0 __asm__("o0") =3D (long) tv; - register long o1 __asm__("o1") =3D (long) tz; - - __asm__ __volatile__(SYSCALL_STRING : "=3Dr" (o0) : "r" (num), - "0" (o0), "r" (o1) : SYSCALL_CLOBBERS); - return o0; + return __cvdso_gettimeofday(tv, tz); } =20 -#ifdef CONFIG_SPARC64 -notrace static __always_inline u64 __shr64(u64 val, int amt) -{ - return val >> amt; -} +int gettimeofday(struct __kernel_old_timeval *, struct timezone *) + __weak __alias(__vdso_gettimeofday); =20 -notrace static __always_inline u64 vread_tick(void) +#if defined(CONFIG_SPARC64) +int __vdso_clock_gettime(clockid_t clock, struct __kernel_timespec *ts) { - u64 ret; - - __asm__ __volatile__("rd %%tick, %0" : "=3Dr" (ret)); - return ret; + return __cvdso_clock_gettime(clock, ts); } =20 -notrace static __always_inline u64 vread_tick_stick(void) -{ - u64 ret; +int clock_gettime(clockid_t, struct __kernel_timespec *) + __weak __alias(__vdso_clock_gettime); =20 - __asm__ __volatile__("rd %%asr24, %0" : "=3Dr" (ret)); - return ret; -} #else -notrace static __always_inline u64 __shr64(u64 val, int amt) -{ - u64 ret; - - __asm__ __volatile__("sllx %H1, 32, %%g1\n\t" - "srl %L1, 0, %L1\n\t" - "or %%g1, %L1, %%g1\n\t" - "srlx %%g1, %2, %L0\n\t" - "srlx %L0, 32, %H0" - : "=3Dr" (ret) - : "r" (val), "r" (amt) - : "g1"); - return ret; -} - -notrace static __always_inline u64 vread_tick(void) -{ - register unsigned long long ret asm("o4"); - - __asm__ __volatile__("rd %%tick, %L0\n\t" - "srlx %L0, 32, %H0" - : "=3Dr" (ret)); - return ret; -} - -notrace static __always_inline u64 vread_tick_stick(void) -{ - register unsigned long long ret asm("o4"); - - __asm__ __volatile__("rd %%asr24, %L0\n\t" - "srlx %L0, 32, %H0" - : "=3Dr" (ret)); - return ret; -} -#endif =20 -notrace static __always_inline u64 vgetsns(struct vvar_data *vvar) +int __vdso_clock_gettime(clockid_t clock, struct old_timespec32 *ts) { - u64 v; - u64 cycles; - - cycles =3D vread_tick(); - v =3D (cycles - vvar->clock.cycle_last) & vvar->clock.mask; - return v * vvar->clock.mult; + return __cvdso_clock_gettime32(clock, ts); } =20 -notrace static __always_inline u64 vgetsns_stick(struct vvar_data *vvar) -{ - u64 v; - u64 cycles; +int clock_gettime(clockid_t, struct old_timespec32 *) + __weak __alias(__vdso_clock_gettime); =20 - cycles =3D vread_tick_stick(); - v =3D (cycles - vvar->clock.cycle_last) & vvar->clock.mask; - return v * vvar->clock.mult; -} - -notrace static __always_inline int do_realtime(struct vvar_data *vvar, - struct __kernel_old_timespec *ts) +int __vdso_clock_gettime64(clockid_t clock, struct __kernel_timespec *ts) { - unsigned long seq; - u64 ns; - - do { - seq =3D vvar_read_begin(vvar); - ts->tv_sec =3D vvar->wall_time_sec; - ns =3D vvar->wall_time_snsec; - ns +=3D vgetsns(vvar); - ns =3D __shr64(ns, vvar->clock.shift); - } while (unlikely(vvar_read_retry(vvar, seq))); - - ts->tv_sec +=3D __iter_div_u64_rem(ns, NSEC_PER_SEC, &ns); - ts->tv_nsec =3D ns; - - return 0; + return __cvdso_clock_gettime(clock, ts); } =20 -notrace static __always_inline int do_realtime_stick(struct vvar_data *vva= r, - struct __kernel_old_timespec *ts) -{ - unsigned long seq; - u64 ns; - - do { - seq =3D vvar_read_begin(vvar); - ts->tv_sec =3D vvar->wall_time_sec; - ns =3D vvar->wall_time_snsec; - ns +=3D vgetsns_stick(vvar); - ns =3D __shr64(ns, vvar->clock.shift); - } while (unlikely(vvar_read_retry(vvar, seq))); - - ts->tv_sec +=3D __iter_div_u64_rem(ns, NSEC_PER_SEC, &ns); - ts->tv_nsec =3D ns; +int clock_gettime64(clockid_t, struct __kernel_timespec *) + __weak __alias(__vdso_clock_gettime64); =20 - return 0; -} - -notrace static __always_inline int do_monotonic(struct vvar_data *vvar, - struct __kernel_old_timespec *ts) -{ - unsigned long seq; - u64 ns; - - do { - seq =3D vvar_read_begin(vvar); - ts->tv_sec =3D vvar->monotonic_time_sec; - ns =3D vvar->monotonic_time_snsec; - ns +=3D vgetsns(vvar); - ns =3D __shr64(ns, vvar->clock.shift); - } while (unlikely(vvar_read_retry(vvar, seq))); - - ts->tv_sec +=3D __iter_div_u64_rem(ns, NSEC_PER_SEC, &ns); - ts->tv_nsec =3D ns; - - return 0; -} - -notrace static __always_inline int do_monotonic_stick(struct vvar_data *vv= ar, - struct __kernel_old_timespec *ts) -{ - unsigned long seq; - u64 ns; - - do { - seq =3D vvar_read_begin(vvar); - ts->tv_sec =3D vvar->monotonic_time_sec; - ns =3D vvar->monotonic_time_snsec; - ns +=3D vgetsns_stick(vvar); - ns =3D __shr64(ns, vvar->clock.shift); - } while (unlikely(vvar_read_retry(vvar, seq))); - - ts->tv_sec +=3D __iter_div_u64_rem(ns, NSEC_PER_SEC, &ns); - ts->tv_nsec =3D ns; - - return 0; -} - -notrace static int do_realtime_coarse(struct vvar_data *vvar, - struct __kernel_old_timespec *ts) -{ - unsigned long seq; - - do { - seq =3D vvar_read_begin(vvar); - ts->tv_sec =3D vvar->wall_time_coarse_sec; - ts->tv_nsec =3D vvar->wall_time_coarse_nsec; - } while (unlikely(vvar_read_retry(vvar, seq))); - return 0; -} - -notrace static int do_monotonic_coarse(struct vvar_data *vvar, - struct __kernel_old_timespec *ts) -{ - unsigned long seq; - - do { - seq =3D vvar_read_begin(vvar); - ts->tv_sec =3D vvar->monotonic_time_coarse_sec; - ts->tv_nsec =3D vvar->monotonic_time_coarse_nsec; - } while (unlikely(vvar_read_retry(vvar, seq))); - - return 0; -} - -notrace int -__vdso_clock_gettime(clockid_t clock, struct __kernel_old_timespec *ts) -{ - struct vvar_data *vvd =3D get_vvar_data(); - - switch (clock) { - case CLOCK_REALTIME: - if (unlikely(vvd->vclock_mode =3D=3D VCLOCK_NONE)) - break; - return do_realtime(vvd, ts); - case CLOCK_MONOTONIC: - if (unlikely(vvd->vclock_mode =3D=3D VCLOCK_NONE)) - break; - return do_monotonic(vvd, ts); - case CLOCK_REALTIME_COARSE: - return do_realtime_coarse(vvd, ts); - case CLOCK_MONOTONIC_COARSE: - return do_monotonic_coarse(vvd, ts); - } - /* - * Unknown clock ID ? Fall back to the syscall. - */ - return vdso_fallback_gettime(clock, ts); -} -int -clock_gettime(clockid_t, struct __kernel_old_timespec *) - __attribute__((weak, alias("__vdso_clock_gettime"))); - -notrace int -__vdso_clock_gettime_stick(clockid_t clock, struct __kernel_old_timespec *= ts) -{ - struct vvar_data *vvd =3D get_vvar_data(); - - switch (clock) { - case CLOCK_REALTIME: - if (unlikely(vvd->vclock_mode =3D=3D VCLOCK_NONE)) - break; - return do_realtime_stick(vvd, ts); - case CLOCK_MONOTONIC: - if (unlikely(vvd->vclock_mode =3D=3D VCLOCK_NONE)) - break; - return do_monotonic_stick(vvd, ts); - case CLOCK_REALTIME_COARSE: - return do_realtime_coarse(vvd, ts); - case CLOCK_MONOTONIC_COARSE: - return do_monotonic_coarse(vvd, ts); - } - /* - * Unknown clock ID ? Fall back to the syscall. - */ - return vdso_fallback_gettime(clock, ts); -} - -notrace int -__vdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) -{ - struct vvar_data *vvd =3D get_vvar_data(); - - if (likely(vvd->vclock_mode !=3D VCLOCK_NONE)) { - if (likely(tv !=3D NULL)) { - union tstv_t { - struct __kernel_old_timespec ts; - struct __kernel_old_timeval tv; - } *tstv =3D (union tstv_t *) tv; - do_realtime(vvd, &tstv->ts); - /* - * Assign before dividing to ensure that the division is - * done in the type of tv_usec, not tv_nsec. - * - * There cannot be > 1 billion usec in a second: - * do_realtime() has already distributed such overflow - * into tv_sec. So we can assign it to an int safely. - */ - tstv->tv.tv_usec =3D tstv->ts.tv_nsec; - tstv->tv.tv_usec /=3D 1000; - } - if (unlikely(tz !=3D NULL)) { - /* Avoid memcpy. Some old compilers fail to inline it */ - tz->tz_minuteswest =3D vvd->tz_minuteswest; - tz->tz_dsttime =3D vvd->tz_dsttime; - } - return 0; - } - return vdso_fallback_gettimeofday(tv, tz); -} -int -gettimeofday(struct __kernel_old_timeval *, struct timezone *) - __attribute__((weak, alias("__vdso_gettimeofday"))); - -notrace int -__vdso_gettimeofday_stick(struct __kernel_old_timeval *tv, struct timezone= *tz) -{ - struct vvar_data *vvd =3D get_vvar_data(); - - if (likely(vvd->vclock_mode !=3D VCLOCK_NONE)) { - if (likely(tv !=3D NULL)) { - union tstv_t { - struct __kernel_old_timespec ts; - struct __kernel_old_timeval tv; - } *tstv =3D (union tstv_t *) tv; - do_realtime_stick(vvd, &tstv->ts); - /* - * Assign before dividing to ensure that the division is - * done in the type of tv_usec, not tv_nsec. - * - * There cannot be > 1 billion usec in a second: - * do_realtime() has already distributed such overflow - * into tv_sec. So we can assign it to an int safely. - */ - tstv->tv.tv_usec =3D tstv->ts.tv_nsec; - tstv->tv.tv_usec /=3D 1000; - } - if (unlikely(tz !=3D NULL)) { - /* Avoid memcpy. Some old compilers fail to inline it */ - tz->tz_minuteswest =3D vvd->tz_minuteswest; - tz->tz_dsttime =3D vvd->tz_dsttime; - } - return 0; - } - return vdso_fallback_gettimeofday(tv, tz); -} +#endif diff --git a/arch/sparc/vdso/vdso-layout.lds.S b/arch/sparc/vdso/vdso-layou= t.lds.S index d31e57e8a3bb..180e5d0ee071 100644 --- a/arch/sparc/vdso/vdso-layout.lds.S +++ b/arch/sparc/vdso/vdso-layout.lds.S @@ -4,15 +4,9 @@ * This script controls its layout. */ =20 -#if defined(BUILD_VDSO64) -# define SHDR_SIZE 64 -#elif defined(BUILD_VDSO32) -# define SHDR_SIZE 40 -#else -# error unknown VDSO target -#endif - -#define NUM_FAKE_SHDRS 7 +#include +#include +#include =20 SECTIONS { @@ -23,8 +17,7 @@ SECTIONS * segment. Page size is 8192 for both 64-bit and 32-bit vdso binaries */ =20 - vvar_start =3D . -8192; - vvar_data =3D vvar_start; + VDSO_VVAR_SYMS =20 . =3D SIZEOF_HEADERS; =20 @@ -47,19 +40,8 @@ SECTIONS *(.bss*) *(.dynbss*) *(.gnu.linkonce.b.*) - - /* - * Ideally this would live in a C file: kept in here for - * compatibility with x86-64. - */ - VDSO_FAKE_SECTION_TABLE_START =3D .; - . =3D . + NUM_FAKE_SHDRS * SHDR_SIZE; - VDSO_FAKE_SECTION_TABLE_END =3D .; } :text =20 - .fake_shstrtab : { *(.fake_shstrtab) } :text - - .note : { *(.note.*) } :text :note =20 .eh_frame_hdr : { *(.eh_frame_hdr) } :text :eh_frame_hdr diff --git a/arch/sparc/vdso/vdso.lds.S b/arch/sparc/vdso/vdso.lds.S index 629ab6900df7..f3caa29a331c 100644 --- a/arch/sparc/vdso/vdso.lds.S +++ b/arch/sparc/vdso/vdso.lds.S @@ -18,10 +18,8 @@ VERSION { global: clock_gettime; __vdso_clock_gettime; - __vdso_clock_gettime_stick; gettimeofday; __vdso_gettimeofday; - __vdso_gettimeofday_stick; local: *; }; } diff --git a/arch/sparc/vdso/vdso2c.c b/arch/sparc/vdso/vdso2c.c index dc81240aab6f..e5c61214a0e2 100644 --- a/arch/sparc/vdso/vdso2c.c +++ b/arch/sparc/vdso/vdso2c.c @@ -58,28 +58,6 @@ =20 const char *outfilename; =20 -/* Symbols that we need in vdso2c. */ -enum { - sym_vvar_start, - sym_VDSO_FAKE_SECTION_TABLE_START, - sym_VDSO_FAKE_SECTION_TABLE_END, -}; - -struct vdso_sym { - const char *name; - int export; -}; - -struct vdso_sym required_syms[] =3D { - [sym_vvar_start] =3D {"vvar_start", 1}, - [sym_VDSO_FAKE_SECTION_TABLE_START] =3D { - "VDSO_FAKE_SECTION_TABLE_START", 0 - }, - [sym_VDSO_FAKE_SECTION_TABLE_END] =3D { - "VDSO_FAKE_SECTION_TABLE_END", 0 - }, -}; - __attribute__((format(printf, 1, 2))) __attribute__((noreturn)) static void fail(const char *format, ...) { @@ -119,8 +97,6 @@ static void fail(const char *format, ...) #define PUT_BE(x, val) \ PBE(x, val, 64, PBE(x, val, 32, PBE(x, val, 16, LAST_PBE(x, val)))) =20 -#define NSYMS ARRAY_SIZE(required_syms) - #define BITSFUNC3(name, bits, suffix) name##bits##suffix #define BITSFUNC2(name, bits, suffix) BITSFUNC3(name, bits, suffix) #define BITSFUNC(name) BITSFUNC2(name, ELF_BITS, ) diff --git a/arch/sparc/vdso/vdso2c.h b/arch/sparc/vdso/vdso2c.h index 60d69acc748f..bad6a0593f4c 100644 --- a/arch/sparc/vdso/vdso2c.h +++ b/arch/sparc/vdso/vdso2c.h @@ -17,11 +17,9 @@ static void BITSFUNC(go)(void *raw_addr, size_t raw_len, unsigned long mapping_size; int i; unsigned long j; - ELF(Shdr) *symtab_hdr =3D NULL, *strtab_hdr; + ELF(Shdr) *symtab_hdr =3D NULL; ELF(Ehdr) *hdr =3D (ELF(Ehdr) *)raw_addr; ELF(Dyn) *dyn =3D 0, *dyn_end =3D 0; - INT_BITS syms[NSYMS] =3D {}; - ELF(Phdr) *pt =3D (ELF(Phdr) *)(raw_addr + GET_BE(&hdr->e_phoff)); =20 /* Walk the segment table. */ @@ -72,42 +70,6 @@ static void BITSFUNC(go)(void *raw_addr, size_t raw_len, if (!symtab_hdr) fail("no symbol table\n"); =20 - strtab_hdr =3D raw_addr + GET_BE(&hdr->e_shoff) + - GET_BE(&hdr->e_shentsize) * GET_BE(&symtab_hdr->sh_link); - - /* Walk the symbol table */ - for (i =3D 0; - i < GET_BE(&symtab_hdr->sh_size) / GET_BE(&symtab_hdr->sh_entsize); - i++) { - int k; - - ELF(Sym) *sym =3D raw_addr + GET_BE(&symtab_hdr->sh_offset) + - GET_BE(&symtab_hdr->sh_entsize) * i; - const char *name =3D raw_addr + GET_BE(&strtab_hdr->sh_offset) + - GET_BE(&sym->st_name); - - for (k =3D 0; k < NSYMS; k++) { - if (!strcmp(name, required_syms[k].name)) { - if (syms[k]) { - fail("duplicate symbol %s\n", - required_syms[k].name); - } - - /* - * Careful: we use negative addresses, but - * st_value is unsigned, so we rely - * on syms[k] being a signed type of the - * correct width. - */ - syms[k] =3D GET_BE(&sym->st_value); - } - } - } - - /* Validate mapping addresses. */ - if (syms[sym_vvar_start] % 8192) - fail("vvar_begin must be a multiple of 8192\n"); - if (!name) { fwrite(stripped_addr, stripped_len, 1, outfile); return; @@ -133,10 +95,5 @@ static void BITSFUNC(go)(void *raw_addr, size_t raw_len, fprintf(outfile, "const struct vdso_image %s_builtin =3D {\n", name); fprintf(outfile, "\t.data =3D raw_data,\n"); fprintf(outfile, "\t.size =3D %lu,\n", mapping_size); - for (i =3D 0; i < NSYMS; i++) { - if (required_syms[i].export && syms[i]) - fprintf(outfile, "\t.sym_%s =3D %" PRIi64 ",\n", - required_syms[i].name, (int64_t)syms[i]); - } fprintf(outfile, "};\n"); } diff --git a/arch/sparc/vdso/vdso32/vdso32.lds.S b/arch/sparc/vdso/vdso32/v= dso32.lds.S index 218930fdff03..a14e4f77e6f2 100644 --- a/arch/sparc/vdso/vdso32/vdso32.lds.S +++ b/arch/sparc/vdso/vdso32/vdso32.lds.S @@ -17,10 +17,10 @@ VERSION { global: clock_gettime; __vdso_clock_gettime; - __vdso_clock_gettime_stick; + clock_gettime64; + __vdso_clock_gettime64; gettimeofday; __vdso_gettimeofday; - __vdso_gettimeofday_stick; local: *; }; } diff --git a/arch/sparc/vdso/vma.c b/arch/sparc/vdso/vma.c index c454689ce5fa..60029d60f4d3 100644 --- a/arch/sparc/vdso/vma.c +++ b/arch/sparc/vdso/vma.c @@ -16,17 +16,16 @@ #include #include #include +#include #include #include #include -#include #include =20 -unsigned int __read_mostly vdso_enabled =3D 1; +#include +#include =20 -static struct vm_special_mapping vvar_mapping =3D { - .name =3D "[vvar]" -}; +unsigned int __read_mostly vdso_enabled =3D 1; =20 #ifdef CONFIG_SPARC64 static struct vm_special_mapping vdso_mapping64 =3D { @@ -40,207 +39,8 @@ static struct vm_special_mapping vdso_mapping32 =3D { }; #endif =20 -struct vvar_data *vvar_data; - -struct vdso_elfinfo32 { - Elf32_Ehdr *hdr; - Elf32_Sym *dynsym; - unsigned long dynsymsize; - const char *dynstr; - unsigned long text; -}; - -struct vdso_elfinfo64 { - Elf64_Ehdr *hdr; - Elf64_Sym *dynsym; - unsigned long dynsymsize; - const char *dynstr; - unsigned long text; -}; - -struct vdso_elfinfo { - union { - struct vdso_elfinfo32 elf32; - struct vdso_elfinfo64 elf64; - } u; -}; - -static void *one_section64(struct vdso_elfinfo64 *e, const char *name, - unsigned long *size) -{ - const char *snames; - Elf64_Shdr *shdrs; - unsigned int i; - - shdrs =3D (void *)e->hdr + e->hdr->e_shoff; - snames =3D (void *)e->hdr + shdrs[e->hdr->e_shstrndx].sh_offset; - for (i =3D 1; i < e->hdr->e_shnum; i++) { - if (!strcmp(snames+shdrs[i].sh_name, name)) { - if (size) - *size =3D shdrs[i].sh_size; - return (void *)e->hdr + shdrs[i].sh_offset; - } - } - return NULL; -} - -static int find_sections64(const struct vdso_image *image, struct vdso_elf= info *_e) -{ - struct vdso_elfinfo64 *e =3D &_e->u.elf64; - - e->hdr =3D image->data; - e->dynsym =3D one_section64(e, ".dynsym", &e->dynsymsize); - e->dynstr =3D one_section64(e, ".dynstr", NULL); - - if (!e->dynsym || !e->dynstr) { - pr_err("VDSO64: Missing symbol sections.\n"); - return -ENODEV; - } - return 0; -} - -static Elf64_Sym *find_sym64(const struct vdso_elfinfo64 *e, const char *n= ame) -{ - unsigned int i; - - for (i =3D 0; i < (e->dynsymsize / sizeof(Elf64_Sym)); i++) { - Elf64_Sym *s =3D &e->dynsym[i]; - if (s->st_name =3D=3D 0) - continue; - if (!strcmp(e->dynstr + s->st_name, name)) - return s; - } - return NULL; -} - -static int patchsym64(struct vdso_elfinfo *_e, const char *orig, - const char *new) -{ - struct vdso_elfinfo64 *e =3D &_e->u.elf64; - Elf64_Sym *osym =3D find_sym64(e, orig); - Elf64_Sym *nsym =3D find_sym64(e, new); - - if (!nsym || !osym) { - pr_err("VDSO64: Missing symbols.\n"); - return -ENODEV; - } - osym->st_value =3D nsym->st_value; - osym->st_size =3D nsym->st_size; - osym->st_info =3D nsym->st_info; - osym->st_other =3D nsym->st_other; - osym->st_shndx =3D nsym->st_shndx; - - return 0; -} - -static void *one_section32(struct vdso_elfinfo32 *e, const char *name, - unsigned long *size) -{ - const char *snames; - Elf32_Shdr *shdrs; - unsigned int i; - - shdrs =3D (void *)e->hdr + e->hdr->e_shoff; - snames =3D (void *)e->hdr + shdrs[e->hdr->e_shstrndx].sh_offset; - for (i =3D 1; i < e->hdr->e_shnum; i++) { - if (!strcmp(snames+shdrs[i].sh_name, name)) { - if (size) - *size =3D shdrs[i].sh_size; - return (void *)e->hdr + shdrs[i].sh_offset; - } - } - return NULL; -} - -static int find_sections32(const struct vdso_image *image, struct vdso_elf= info *_e) -{ - struct vdso_elfinfo32 *e =3D &_e->u.elf32; - - e->hdr =3D image->data; - e->dynsym =3D one_section32(e, ".dynsym", &e->dynsymsize); - e->dynstr =3D one_section32(e, ".dynstr", NULL); - - if (!e->dynsym || !e->dynstr) { - pr_err("VDSO32: Missing symbol sections.\n"); - return -ENODEV; - } - return 0; -} - -static Elf32_Sym *find_sym32(const struct vdso_elfinfo32 *e, const char *n= ame) -{ - unsigned int i; - - for (i =3D 0; i < (e->dynsymsize / sizeof(Elf32_Sym)); i++) { - Elf32_Sym *s =3D &e->dynsym[i]; - if (s->st_name =3D=3D 0) - continue; - if (!strcmp(e->dynstr + s->st_name, name)) - return s; - } - return NULL; -} - -static int patchsym32(struct vdso_elfinfo *_e, const char *orig, - const char *new) -{ - struct vdso_elfinfo32 *e =3D &_e->u.elf32; - Elf32_Sym *osym =3D find_sym32(e, orig); - Elf32_Sym *nsym =3D find_sym32(e, new); - - if (!nsym || !osym) { - pr_err("VDSO32: Missing symbols.\n"); - return -ENODEV; - } - osym->st_value =3D nsym->st_value; - osym->st_size =3D nsym->st_size; - osym->st_info =3D nsym->st_info; - osym->st_other =3D nsym->st_other; - osym->st_shndx =3D nsym->st_shndx; - - return 0; -} - -static int find_sections(const struct vdso_image *image, struct vdso_elfin= fo *e, - bool elf64) -{ - if (elf64) - return find_sections64(image, e); - else - return find_sections32(image, e); -} - -static int patch_one_symbol(struct vdso_elfinfo *e, const char *orig, - const char *new_target, bool elf64) -{ - if (elf64) - return patchsym64(e, orig, new_target); - else - return patchsym32(e, orig, new_target); -} - -static int stick_patch(const struct vdso_image *image, struct vdso_elfinfo= *e, bool elf64) -{ - int err; - - err =3D find_sections(image, e, elf64); - if (err) - return err; - - err =3D patch_one_symbol(e, - "__vdso_gettimeofday", - "__vdso_gettimeofday_stick", elf64); - if (err) - return err; - - return patch_one_symbol(e, - "__vdso_clock_gettime", - "__vdso_clock_gettime_stick", elf64); - return 0; -} - /* - * Allocate pages for the vdso and vvar, and copy in the vdso text from the + * Allocate pages for the vdso and copy in the vdso text from the * kernel image. */ static int __init init_vdso_image(const struct vdso_image *image, @@ -248,16 +48,8 @@ static int __init init_vdso_image(const struct vdso_ima= ge *image, bool elf64) { int cnpages =3D (image->size) / PAGE_SIZE; - struct page *dp, **dpp =3D NULL; struct page *cp, **cpp =3D NULL; - struct vdso_elfinfo ei; - int i, dnpages =3D 0; - - if (tlb_type !=3D spitfire) { - int err =3D stick_patch(image, &ei, elf64); - if (err) - return err; - } + int i; =20 /* * First, the vdso text. This is initialied data, an integral number of @@ -280,31 +72,6 @@ static int __init init_vdso_image(const struct vdso_ima= ge *image, copy_page(page_address(cp), image->data + i * PAGE_SIZE); } =20 - /* - * Now the vvar page. This is uninitialized data. - */ - - if (vvar_data =3D=3D NULL) { - dnpages =3D (sizeof(struct vvar_data) / PAGE_SIZE) + 1; - if (WARN_ON(dnpages !=3D 1)) - goto oom; - dpp =3D kzalloc_objs(struct page *, dnpages); - vvar_mapping.pages =3D dpp; - - if (!dpp) - goto oom; - - dp =3D alloc_page(GFP_KERNEL); - if (!dp) - goto oom; - - dpp[0] =3D dp; - vvar_data =3D page_address(dp); - memset(vvar_data, 0, PAGE_SIZE); - - vvar_data->seq =3D 0; - } - return 0; oom: if (cpp !=3D NULL) { @@ -316,15 +83,6 @@ static int __init init_vdso_image(const struct vdso_ima= ge *image, vdso_mapping->pages =3D NULL; } =20 - if (dpp !=3D NULL) { - for (i =3D 0; i < dnpages; i++) { - if (dpp[i] !=3D NULL) - __free_page(dpp[i]); - } - kfree(dpp); - vvar_mapping.pages =3D NULL; - } - pr_warn("Cannot allocate vdso\n"); vdso_enabled =3D 0; return -ENOMEM; @@ -359,9 +117,12 @@ static unsigned long vdso_addr(unsigned long start, un= signed int len) return start + (offset << PAGE_SHIFT); } =20 +static_assert(VDSO_NR_PAGES =3D=3D __VDSO_PAGES); + static int map_vdso(const struct vdso_image *image, struct vm_special_mapping *vdso_mapping) { + const size_t area_size =3D image->size + VDSO_NR_PAGES * PAGE_SIZE; struct mm_struct *mm =3D current->mm; struct vm_area_struct *vma; unsigned long text_start, addr =3D 0; @@ -374,23 +135,20 @@ static int map_vdso(const struct vdso_image *image, * region is free. */ if (current->flags & PF_RANDOMIZE) { - addr =3D get_unmapped_area(NULL, 0, - image->size - image->sym_vvar_start, - 0, 0); + addr =3D get_unmapped_area(NULL, 0, area_size, 0, 0); if (IS_ERR_VALUE(addr)) { ret =3D addr; goto up_fail; } - addr =3D vdso_addr(addr, image->size - image->sym_vvar_start); + addr =3D vdso_addr(addr, area_size); } - addr =3D get_unmapped_area(NULL, addr, - image->size - image->sym_vvar_start, 0, 0); + addr =3D get_unmapped_area(NULL, addr, area_size, 0, 0); if (IS_ERR_VALUE(addr)) { ret =3D addr; goto up_fail; } =20 - text_start =3D addr - image->sym_vvar_start; + text_start =3D addr + VDSO_NR_PAGES * PAGE_SIZE; current->mm->context.vdso =3D (void __user *)text_start; =20 /* @@ -408,11 +166,7 @@ static int map_vdso(const struct vdso_image *image, goto up_fail; } =20 - vma =3D _install_special_mapping(mm, - addr, - -image->sym_vvar_start, - VM_READ|VM_MAYREAD, - &vvar_mapping); + vma =3D vdso_install_vvar_mapping(mm, addr); =20 if (IS_ERR(vma)) { ret =3D PTR_ERR(vma); diff --git a/arch/x86/entry/vdso/vdso32/Makefile b/arch/x86/entry/vdso/vdso= 32/Makefile index add6afb484ba..ded4fc6a48cd 100644 --- a/arch/x86/entry/vdso/vdso32/Makefile +++ b/arch/x86/entry/vdso/vdso32/Makefile @@ -15,6 +15,10 @@ flags-y :=3D -DBUILD_VDSO32 -m32 -mregparm=3D0 flags-$(CONFIG_X86_64) +=3D -include $(src)/fake_32bit_build.h flags-remove-y :=3D -m64 =20 +# Checker flags +CHECKFLAGS :=3D $(subst -m64,-m32,$(CHECKFLAGS)) +CHECKFLAGS :=3D $(subst -D__x86_64__,-D__i386__,$(CHECKFLAGS)) + # The location of this include matters! include $(src)/../common/Makefile.include =20 diff --git a/drivers/char/random.c b/drivers/char/random.c index 7ff4d29911fd..b4da1fb976c1 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -56,11 +56,7 @@ #include #include #include -#ifdef CONFIG_VDSO_GETRANDOM -#include #include -#include -#endif #include #include #include @@ -269,7 +265,7 @@ static void crng_reseed(struct work_struct *work) if (next_gen =3D=3D ULONG_MAX) ++next_gen; WRITE_ONCE(base_crng.generation, next_gen); -#ifdef CONFIG_VDSO_GETRANDOM + /* base_crng.generation's invalid value is ULONG_MAX, while * vdso_k_rng_data->generation's invalid value is 0, so add one to the * former to arrive at the latter. Use smp_store_release so that this @@ -283,8 +279,9 @@ static void crng_reseed(struct work_struct *work) * because the vDSO side only checks whether the value changed, without * actually using or interpreting the value. */ - smp_store_release((unsigned long *)&vdso_k_rng_data->generation, next_gen= + 1); -#endif + if (IS_ENABLED(CONFIG_VDSO_GETRANDOM)) + smp_store_release((unsigned long *)&vdso_k_rng_data->generation, next_ge= n + 1); + if (!static_branch_likely(&crng_is_ready)) crng_init =3D CRNG_READY; spin_unlock_irqrestore(&base_crng.lock, flags); @@ -734,9 +731,8 @@ static void __cold _credit_init_bits(size_t bits) if (system_dfl_wq) queue_work(system_dfl_wq, &set_ready); atomic_notifier_call_chain(&random_ready_notifier, 0, NULL); -#ifdef CONFIG_VDSO_GETRANDOM - WRITE_ONCE(vdso_k_rng_data->is_ready, true); -#endif + if (IS_ENABLED(CONFIG_VDSO_GETRANDOM)) + WRITE_ONCE(vdso_k_rng_data->is_ready, true); wake_up_interruptible(&crng_init_wait); kill_fasync(&fasync, SIGIO, POLL_IN); pr_notice("crng init done\n"); diff --git a/include/asm-generic/bitsperlong.h b/include/asm-generic/bitspe= rlong.h index 1023e2a4bd37..90e8aeebfd2f 100644 --- a/include/asm-generic/bitsperlong.h +++ b/include/asm-generic/bitsperlong.h @@ -19,6 +19,15 @@ #error Inconsistent word size. Check asm/bitsperlong.h #endif =20 +#if __CHAR_BIT__ * __SIZEOF_LONG__ !=3D __BITS_PER_LONG +#error Inconsistent word size. Check asm/bitsperlong.h +#endif + +#ifndef __ASSEMBLER__ +_Static_assert(sizeof(long) * 8 =3D=3D __BITS_PER_LONG, + "Inconsistent word size. Check asm/bitsperlong.h"); +#endif + #ifndef BITS_PER_LONG_LONG #define BITS_PER_LONG_LONG 64 #endif diff --git a/include/linux/clocksource.h b/include/linux/clocksource.h index 65b7c41471c3..12d853b18832 100644 --- a/include/linux/clocksource.h +++ b/include/linux/clocksource.h @@ -25,8 +25,7 @@ struct clocksource_base; struct clocksource; struct module; =20 -#if defined(CONFIG_ARCH_CLOCKSOURCE_DATA) || \ - defined(CONFIG_GENERIC_GETTIMEOFDAY) +#if defined(CONFIG_GENERIC_GETTIMEOFDAY) #include #endif =20 @@ -106,9 +105,6 @@ struct clocksource { u64 max_idle_ns; u32 maxadj; u32 uncertainty_margin; -#ifdef CONFIG_ARCH_CLOCKSOURCE_DATA - struct arch_clocksource_data archdata; -#endif u64 max_cycles; u64 max_raw_delta; const char *name; diff --git a/include/linux/time_namespace.h b/include/linux/time_namespace.h index c514d0e5a45c..58bd9728df58 100644 --- a/include/linux/time_namespace.h +++ b/include/linux/time_namespace.h @@ -8,6 +8,7 @@ #include #include #include +#include =20 struct user_namespace; extern struct user_namespace init_user_ns; @@ -25,7 +26,9 @@ struct time_namespace { struct ucounts *ucounts; struct ns_common ns; struct timens_offsets offsets; +#ifdef CONFIG_TIME_NS_VDSO struct page *vvar_page; +#endif /* If set prevents changing offsets after any task joined namespace. */ bool frozen_offsets; } __randomize_layout; @@ -38,9 +41,6 @@ static inline struct time_namespace *to_time_ns(struct ns= _common *ns) return container_of(ns, struct time_namespace, ns); } void __init time_ns_init(void); -extern int vdso_join_timens(struct task_struct *task, - struct time_namespace *ns); -extern void timens_commit(struct task_struct *tsk, struct time_namespace *= ns); =20 static inline struct time_namespace *get_time_ns(struct time_namespace *ns) { @@ -53,7 +53,6 @@ struct time_namespace *copy_time_ns(u64 flags, struct time_namespace *old_ns); void free_time_ns(struct time_namespace *ns); void timens_on_fork(struct nsproxy *nsproxy, struct task_struct *tsk); -struct page *find_timens_vvar_page(struct vm_area_struct *vma); =20 static inline void put_time_ns(struct time_namespace *ns) { @@ -117,17 +116,6 @@ static inline void __init time_ns_init(void) { } =20 -static inline int vdso_join_timens(struct task_struct *task, - struct time_namespace *ns) -{ - return 0; -} - -static inline void timens_commit(struct task_struct *tsk, - struct time_namespace *ns) -{ -} - static inline struct time_namespace *get_time_ns(struct time_namespace *ns) { return NULL; @@ -154,11 +142,6 @@ static inline void timens_on_fork(struct nsproxy *nspr= oxy, return; } =20 -static inline struct page *find_timens_vvar_page(struct vm_area_struct *vm= a) -{ - return NULL; -} - static inline void timens_add_monotonic(struct timespec64 *ts) { } static inline void timens_add_boottime(struct timespec64 *ts) { } =20 @@ -175,4 +158,20 @@ static inline ktime_t timens_ktime_to_host(clockid_t c= lockid, ktime_t tim) } #endif =20 +#ifdef CONFIG_TIME_NS_VDSO +extern void timens_commit(struct task_struct *tsk, struct time_namespace *= ns); +struct page *find_timens_vvar_page(struct vm_area_struct *vma); +#else /* !CONFIG_TIME_NS_VDSO */ +static inline void timens_commit(struct task_struct *tsk, struct time_name= space *ns) +{ +} + +static inline struct page *find_timens_vvar_page(struct vm_area_struct *vm= a) +{ + return NULL; +} +#endif /* CONFIG_TIME_NS_VDSO */ + +DEFINE_FREE(time_ns, struct time_namespace *, if (_T) put_time_ns(_T)) + #endif /* _LINUX_TIMENS_H */ diff --git a/include/linux/vdso_datastore.h b/include/linux/vdso_datastore.h index a91fa24b06e0..0b530428db71 100644 --- a/include/linux/vdso_datastore.h +++ b/include/linux/vdso_datastore.h @@ -2,9 +2,15 @@ #ifndef _LINUX_VDSO_DATASTORE_H #define _LINUX_VDSO_DATASTORE_H =20 +#ifdef CONFIG_HAVE_GENERIC_VDSO #include =20 extern const struct vm_special_mapping vdso_vvar_mapping; struct vm_area_struct *vdso_install_vvar_mapping(struct mm_struct *mm, uns= igned long addr); =20 +void __init vdso_setup_data_pages(void); +#else /* !CONFIG_HAVE_GENERIC_VDSO */ +static inline void vdso_setup_data_pages(void) { } +#endif /* CONFIG_HAVE_GENERIC_VDSO */ + #endif /* _LINUX_VDSO_DATASTORE_H */ diff --git a/include/vdso/datapage.h b/include/vdso/datapage.h index 23c39b96190f..5977723fb3b5 100644 --- a/include/vdso/datapage.h +++ b/include/vdso/datapage.h @@ -4,24 +4,16 @@ =20 #ifndef __ASSEMBLY__ =20 -#include +#include + #include #include -#include -#include =20 #include #include #include -#include -#include -#include -#include #include -#include #include -#include -#include =20 #ifdef CONFIG_ARCH_HAS_VDSO_TIME_DATA #include @@ -80,8 +72,8 @@ struct vdso_timestamp { * @mask: clocksource mask * @mult: clocksource multiplier * @shift: clocksource shift - * @basetime[clock_id]: basetime per clock_id - * @offset[clock_id]: time namespace offset per clock_id + * @basetime: basetime per clock_id + * @offset: time namespace offset per clock_id * * See also struct vdso_time_data for basic access and ordering informatio= n as * struct vdso_clock is used there. @@ -184,17 +176,6 @@ enum vdso_pages { VDSO_NR_PAGES }; =20 -/* - * The generic vDSO implementation requires that gettimeofday.h - * provides: - * - __arch_get_hw_counter(): to get the hw counter based on the - * clock_mode. - * - gettimeofday_fallback(): fallback for gettimeofday. - * - clock_gettime_fallback(): fallback for clock_gettime. - * - clock_getres_fallback(): fallback for clock_getres. - */ -#include - #else /* !__ASSEMBLY__ */ =20 #ifdef CONFIG_VDSO_GETRANDOM diff --git a/include/vdso/helpers.h b/include/vdso/helpers.h index 1a5ee9d9052c..a3bf4f1c0d37 100644 --- a/include/vdso/helpers.h +++ b/include/vdso/helpers.h @@ -6,6 +6,13 @@ =20 #include #include +#include +#include + +static __always_inline bool vdso_is_timens_clock(const struct vdso_clock *= vc) +{ + return IS_ENABLED(CONFIG_TIME_NS) && vc->clock_mode =3D=3D VDSO_CLOCKMODE= _TIMENS; +} =20 static __always_inline u32 vdso_read_begin(const struct vdso_clock *vc) { @@ -18,6 +25,28 @@ static __always_inline u32 vdso_read_begin(const struct = vdso_clock *vc) return seq; } =20 +/* + * Variant of vdso_read_begin() to handle VDSO_CLOCKMODE_TIMENS. + * + * Time namespace enabled tasks have a special VVAR page installed which h= as + * vc->seq set to 1 and vc->clock_mode set to VDSO_CLOCKMODE_TIMENS. For n= on + * time namespace affected tasks this does not affect performance because = if + * vc->seq is odd, i.e. a concurrent update is in progress the extra check= for + * vc->clock_mode is just a few extra instructions while spin waiting for + * vc->seq to become even again. + */ +static __always_inline bool vdso_read_begin_timens(const struct vdso_clock= *vc, u32 *seq) +{ + while (unlikely((*seq =3D READ_ONCE(vc->seq)) & 1)) { + if (vdso_is_timens_clock(vc)) + return true; + cpu_relax(); + } + smp_rmb(); + + return false; +} + static __always_inline u32 vdso_read_retry(const struct vdso_clock *vc, u32 start) { @@ -25,7 +54,7 @@ static __always_inline u32 vdso_read_retry(const struct v= dso_clock *vc, =20 smp_rmb(); seq =3D READ_ONCE(vc->seq); - return seq !=3D start; + return unlikely(seq !=3D start); } =20 static __always_inline void vdso_write_seq_begin(struct vdso_clock *vc) diff --git a/init/Kconfig b/init/Kconfig index 444ce811ea67..5e710b03a27a 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -1386,12 +1386,14 @@ config UTS_NS =20 config TIME_NS bool "TIME namespace" - depends on GENERIC_GETTIMEOFDAY default y help In this namespace boottime and monotonic clocks can be set. The time will keep going with the same pace. =20 +config TIME_NS_VDSO + def_bool TIME_NS && GENERIC_GETTIMEOFDAY + config IPC_NS bool "IPC namespace" depends on (SYSVIPC || POSIX_MQUEUE) diff --git a/init/main.c b/init/main.c index 1cb395dd94e4..de867b2693d2 100644 --- a/init/main.c +++ b/init/main.c @@ -105,6 +105,7 @@ #include #include #include +#include #include =20 #include @@ -1119,6 +1120,7 @@ void start_kernel(void) srcu_init(); hrtimers_init(); softirq_init(); + vdso_setup_data_pages(); timekeeping_init(); time_init(); =20 diff --git a/kernel/time/Kconfig b/kernel/time/Kconfig index 7c6a52f7836c..fe3311877097 100644 --- a/kernel/time/Kconfig +++ b/kernel/time/Kconfig @@ -9,10 +9,6 @@ config CLOCKSOURCE_WATCHDOG bool =20 -# Architecture has extra clocksource data -config ARCH_CLOCKSOURCE_DATA - bool - # Architecture has extra clocksource init called from registration config ARCH_CLOCKSOURCE_INIT bool diff --git a/kernel/time/Makefile b/kernel/time/Makefile index f7d52d9543cc..eaf290c972f9 100644 --- a/kernel/time/Makefile +++ b/kernel/time/Makefile @@ -30,5 +30,6 @@ obj-$(CONFIG_GENERIC_GETTIMEOFDAY) +=3D vsyscall.o obj-$(CONFIG_DEBUG_FS) +=3D timekeeping_debug.o obj-$(CONFIG_TEST_UDELAY) +=3D test_udelay.o obj-$(CONFIG_TIME_NS) +=3D namespace.o +obj-$(CONFIG_TIME_NS_VDSO) +=3D namespace_vdso.o obj-$(CONFIG_TEST_CLOCKSOURCE_WATCHDOG) +=3D clocksource-wdtest.o obj-$(CONFIG_TIME_KUNIT_TEST) +=3D time_test.o diff --git a/kernel/time/namespace.c b/kernel/time/namespace.c index 652744e00eb4..4bca3f78c8ea 100644 --- a/kernel/time/namespace.c +++ b/kernel/time/namespace.c @@ -18,8 +18,9 @@ #include #include #include +#include =20 -#include +#include "namespace_internal.h" =20 ktime_t do_timens_ktime_to_host(clockid_t clockid, ktime_t tim, struct timens_offsets *ns_offsets) @@ -93,8 +94,8 @@ static struct time_namespace *clone_time_ns(struct user_n= amespace *user_ns, if (!ns) goto fail_dec; =20 - ns->vvar_page =3D alloc_page(GFP_KERNEL_ACCOUNT | __GFP_ZERO); - if (!ns->vvar_page) + err =3D timens_vdso_alloc_vvar_page(ns); + if (err) goto fail_free; =20 err =3D ns_common_init(ns); @@ -109,7 +110,7 @@ static struct time_namespace *clone_time_ns(struct user= _namespace *user_ns, return ns; =20 fail_free_page: - __free_page(ns->vvar_page); + timens_vdso_free_vvar_page(ns); fail_free: kfree(ns); fail_dec: @@ -138,117 +139,7 @@ struct time_namespace *copy_time_ns(u64 flags, return clone_time_ns(user_ns, old_ns); } =20 -static struct timens_offset offset_from_ts(struct timespec64 off) -{ - struct timens_offset ret; - - ret.sec =3D off.tv_sec; - ret.nsec =3D off.tv_nsec; - - return ret; -} - -/* - * A time namespace VVAR page has the same layout as the VVAR page which - * contains the system wide VDSO data. - * - * For a normal task the VVAR pages are installed in the normal ordering: - * VVAR - * PVCLOCK - * HVCLOCK - * TIMENS <- Not really required - * - * Now for a timens task the pages are installed in the following order: - * TIMENS - * PVCLOCK - * HVCLOCK - * VVAR - * - * The check for vdso_clock->clock_mode is in the unlikely path of - * the seq begin magic. So for the non-timens case most of the time - * 'seq' is even, so the branch is not taken. - * - * If 'seq' is odd, i.e. a concurrent update is in progress, the extra che= ck - * for vdso_clock->clock_mode is a non-issue. The task is spin waiting for= the - * update to finish and for 'seq' to become even anyway. - * - * Timens page has vdso_clock->clock_mode set to VDSO_CLOCKMODE_TIMENS whi= ch - * enforces the time namespace handling path. - */ -static void timens_setup_vdso_clock_data(struct vdso_clock *vc, - struct time_namespace *ns) -{ - struct timens_offset *offset =3D vc->offset; - struct timens_offset monotonic =3D offset_from_ts(ns->offsets.monotonic); - struct timens_offset boottime =3D offset_from_ts(ns->offsets.boottime); - - vc->seq =3D 1; - vc->clock_mode =3D VDSO_CLOCKMODE_TIMENS; - offset[CLOCK_MONOTONIC] =3D monotonic; - offset[CLOCK_MONOTONIC_RAW] =3D monotonic; - offset[CLOCK_MONOTONIC_COARSE] =3D monotonic; - offset[CLOCK_BOOTTIME] =3D boottime; - offset[CLOCK_BOOTTIME_ALARM] =3D boottime; -} - -struct page *find_timens_vvar_page(struct vm_area_struct *vma) -{ - if (likely(vma->vm_mm =3D=3D current->mm)) - return current->nsproxy->time_ns->vvar_page; - - /* - * VM_PFNMAP | VM_IO protect .fault() handler from being called - * through interfaces like /proc/$pid/mem or - * process_vm_{readv,writev}() as long as there's no .access() - * in special_mapping_vmops(). - * For more details check_vma_flags() and __access_remote_vm() - */ - - WARN(1, "vvar_page accessed remotely"); - - return NULL; -} - -/* - * Protects possibly multiple offsets writers racing each other - * and tasks entering the namespace. - */ -static DEFINE_MUTEX(offset_lock); - -static void timens_set_vvar_page(struct task_struct *task, - struct time_namespace *ns) -{ - struct vdso_time_data *vdata; - struct vdso_clock *vc; - unsigned int i; - - if (ns =3D=3D &init_time_ns) - return; - - /* Fast-path, taken by every task in namespace except the first. */ - if (likely(ns->frozen_offsets)) - return; - - mutex_lock(&offset_lock); - /* Nothing to-do: vvar_page has been already initialized. */ - if (ns->frozen_offsets) - goto out; - - ns->frozen_offsets =3D true; - vdata =3D page_address(ns->vvar_page); - vc =3D vdata->clock_data; - - for (i =3D 0; i < CS_BASES; i++) - timens_setup_vdso_clock_data(&vc[i], ns); - - if (IS_ENABLED(CONFIG_POSIX_AUX_CLOCKS)) { - for (i =3D 0; i < ARRAY_SIZE(vdata->aux_clock_data); i++) - timens_setup_vdso_clock_data(&vdata->aux_clock_data[i], ns); - } - -out: - mutex_unlock(&offset_lock); -} +DEFINE_MUTEX(timens_offset_lock); =20 void free_time_ns(struct time_namespace *ns) { @@ -256,41 +147,39 @@ void free_time_ns(struct time_namespace *ns) dec_time_namespaces(ns->ucounts); put_user_ns(ns->user_ns); ns_common_free(ns); - __free_page(ns->vvar_page); + timens_vdso_free_vvar_page(ns); /* Concurrent nstree traversal depends on a grace period. */ kfree_rcu(ns, ns.ns_rcu); } =20 static struct ns_common *timens_get(struct task_struct *task) { - struct time_namespace *ns =3D NULL; + struct time_namespace *ns; struct nsproxy *nsproxy; =20 - task_lock(task); + guard(task_lock)(task); nsproxy =3D task->nsproxy; - if (nsproxy) { - ns =3D nsproxy->time_ns; - get_time_ns(ns); - } - task_unlock(task); + if (!nsproxy) + return NULL; =20 - return ns ? &ns->ns : NULL; + ns =3D nsproxy->time_ns; + get_time_ns(ns); + return &ns->ns; } =20 static struct ns_common *timens_for_children_get(struct task_struct *task) { - struct time_namespace *ns =3D NULL; + struct time_namespace *ns; struct nsproxy *nsproxy; =20 - task_lock(task); + guard(task_lock)(task); nsproxy =3D task->nsproxy; - if (nsproxy) { - ns =3D nsproxy->time_ns_for_children; - get_time_ns(ns); - } - task_unlock(task); + if (!nsproxy) + return NULL; =20 - return ns ? &ns->ns : NULL; + ns =3D nsproxy->time_ns_for_children; + get_time_ns(ns); + return &ns->ns; } =20 static void timens_put(struct ns_common *ns) @@ -298,12 +187,6 @@ static void timens_put(struct ns_common *ns) put_time_ns(to_time_ns(ns)); } =20 -void timens_commit(struct task_struct *tsk, struct time_namespace *ns) -{ - timens_set_vvar_page(tsk, ns); - vdso_join_timens(tsk, ns); -} - static int timens_install(struct nsset *nsset, struct ns_common *new) { struct nsproxy *nsproxy =3D nsset->nsproxy; @@ -367,36 +250,33 @@ static void show_offset(struct seq_file *m, int clock= id, struct timespec64 *ts) =20 void proc_timens_show_offsets(struct task_struct *p, struct seq_file *m) { - struct ns_common *ns; - struct time_namespace *time_ns; + struct time_namespace *time_ns __free(time_ns) =3D NULL; + struct ns_common *ns =3D timens_for_children_get(p); =20 - ns =3D timens_for_children_get(p); if (!ns) return; + time_ns =3D to_time_ns(ns); =20 show_offset(m, CLOCK_MONOTONIC, &time_ns->offsets.monotonic); show_offset(m, CLOCK_BOOTTIME, &time_ns->offsets.boottime); - put_time_ns(time_ns); } =20 int proc_timens_set_offset(struct file *file, struct task_struct *p, struct proc_timens_offset *offsets, int noffsets) { - struct ns_common *ns; - struct time_namespace *time_ns; + struct time_namespace *time_ns __free(time_ns) =3D NULL; + struct ns_common *ns =3D timens_for_children_get(p); struct timespec64 tp; - int i, err; + int i; =20 - ns =3D timens_for_children_get(p); if (!ns) return -ESRCH; + time_ns =3D to_time_ns(ns); =20 - if (!file_ns_capable(file, time_ns->user_ns, CAP_SYS_TIME)) { - put_time_ns(time_ns); + if (!file_ns_capable(file, time_ns->user_ns, CAP_SYS_TIME)) return -EPERM; - } =20 for (i =3D 0; i < noffsets; i++) { struct proc_timens_offset *off =3D &offsets[i]; @@ -409,15 +289,12 @@ int proc_timens_set_offset(struct file *file, struct = task_struct *p, ktime_get_boottime_ts64(&tp); break; default: - err =3D -EINVAL; - goto out; + return -EINVAL; } =20 - err =3D -ERANGE; - if (off->val.tv_sec > KTIME_SEC_MAX || off->val.tv_sec < -KTIME_SEC_MAX) - goto out; + return -ERANGE; =20 tp =3D timespec64_add(tp, off->val); /* @@ -425,16 +302,13 @@ int proc_timens_set_offset(struct file *file, struct = task_struct *p, * still unreachable. */ if (tp.tv_sec < 0 || tp.tv_sec > KTIME_SEC_MAX / 2) - goto out; + return -ERANGE; } =20 - mutex_lock(&offset_lock); - if (time_ns->frozen_offsets) { - err =3D -EACCES; - goto out_unlock; - } + guard(mutex)(&timens_offset_lock); + if (time_ns->frozen_offsets) + return -EACCES; =20 - err =3D 0; /* Don't report errors after this line */ for (i =3D 0; i < noffsets; i++) { struct proc_timens_offset *off =3D &offsets[i]; @@ -452,12 +326,7 @@ int proc_timens_set_offset(struct file *file, struct t= ask_struct *p, *offset =3D off->val; } =20 -out_unlock: - mutex_unlock(&offset_lock); -out: - put_time_ns(time_ns); - - return err; + return 0; } =20 const struct proc_ns_operations timens_operations =3D { diff --git a/kernel/time/namespace_internal.h b/kernel/time/namespace_inter= nal.h new file mode 100644 index 000000000000..b37ba179f43b --- /dev/null +++ b/kernel/time/namespace_internal.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _TIME_NAMESPACE_INTERNAL_H +#define _TIME_NAMESPACE_INTERNAL_H + +#include + +struct time_namespace; + +/* + * Protects possibly multiple offsets writers racing each other + * and tasks entering the namespace. + */ +extern struct mutex timens_offset_lock; + +#ifdef CONFIG_TIME_NS_VDSO +int timens_vdso_alloc_vvar_page(struct time_namespace *ns); +void timens_vdso_free_vvar_page(struct time_namespace *ns); +#else /* !CONFIG_TIME_NS_VDSO */ +static inline int timens_vdso_alloc_vvar_page(struct time_namespace *ns) +{ + return 0; +} +static inline void timens_vdso_free_vvar_page(struct time_namespace *ns) +{ +} +#endif /* CONFIG_TIME_NS_VDSO */ + +#endif /* _TIME_NAMESPACE_INTERNAL_H */ diff --git a/kernel/time/namespace_vdso.c b/kernel/time/namespace_vdso.c new file mode 100644 index 000000000000..88c075cd16a3 --- /dev/null +++ b/kernel/time/namespace_vdso.c @@ -0,0 +1,160 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Author: Andrei Vagin + * Author: Dmitry Safonov + */ + +#include +#include +#include +#include +#include + +#include +#include + +#include "namespace_internal.h" + +static struct timens_offset offset_from_ts(struct timespec64 off) +{ + struct timens_offset ret; + + ret.sec =3D off.tv_sec; + ret.nsec =3D off.tv_nsec; + + return ret; +} + +/* + * A time namespace VVAR page has the same layout as the VVAR page which + * contains the system wide VDSO data. + * + * For a normal task the VVAR pages are installed in the normal ordering: + * VVAR + * PVCLOCK + * HVCLOCK + * TIMENS <- Not really required + * + * Now for a timens task the pages are installed in the following order: + * TIMENS + * PVCLOCK + * HVCLOCK + * VVAR + * + * The check for vdso_clock->clock_mode is in the unlikely path of + * the seq begin magic. So for the non-timens case most of the time + * 'seq' is even, so the branch is not taken. + * + * If 'seq' is odd, i.e. a concurrent update is in progress, the extra che= ck + * for vdso_clock->clock_mode is a non-issue. The task is spin waiting for= the + * update to finish and for 'seq' to become even anyway. + * + * Timens page has vdso_clock->clock_mode set to VDSO_CLOCKMODE_TIMENS whi= ch + * enforces the time namespace handling path. + */ +static void timens_setup_vdso_clock_data(struct vdso_clock *vc, + struct time_namespace *ns) +{ + struct timens_offset *offset =3D vc->offset; + struct timens_offset monotonic =3D offset_from_ts(ns->offsets.monotonic); + struct timens_offset boottime =3D offset_from_ts(ns->offsets.boottime); + + vc->seq =3D 1; + vc->clock_mode =3D VDSO_CLOCKMODE_TIMENS; + offset[CLOCK_MONOTONIC] =3D monotonic; + offset[CLOCK_MONOTONIC_RAW] =3D monotonic; + offset[CLOCK_MONOTONIC_COARSE] =3D monotonic; + offset[CLOCK_BOOTTIME] =3D boottime; + offset[CLOCK_BOOTTIME_ALARM] =3D boottime; +} + +struct page *find_timens_vvar_page(struct vm_area_struct *vma) +{ + if (likely(vma->vm_mm =3D=3D current->mm)) + return current->nsproxy->time_ns->vvar_page; + + /* + * VM_PFNMAP | VM_IO protect .fault() handler from being called + * through interfaces like /proc/$pid/mem or + * process_vm_{readv,writev}() as long as there's no .access() + * in special_mapping_vmops(). + * For more details check_vma_flags() and __access_remote_vm() + */ + + WARN(1, "vvar_page accessed remotely"); + + return NULL; +} + +static void timens_set_vvar_page(struct task_struct *task, + struct time_namespace *ns) +{ + struct vdso_time_data *vdata; + struct vdso_clock *vc; + unsigned int i; + + if (ns =3D=3D &init_time_ns) + return; + + /* Fast-path, taken by every task in namespace except the first. */ + if (likely(ns->frozen_offsets)) + return; + + guard(mutex)(&timens_offset_lock); + /* Nothing to-do: vvar_page has been already initialized. */ + if (ns->frozen_offsets) + return; + + ns->frozen_offsets =3D true; + vdata =3D page_address(ns->vvar_page); + vc =3D vdata->clock_data; + + for (i =3D 0; i < CS_BASES; i++) + timens_setup_vdso_clock_data(&vc[i], ns); + + if (IS_ENABLED(CONFIG_POSIX_AUX_CLOCKS)) { + for (i =3D 0; i < ARRAY_SIZE(vdata->aux_clock_data); i++) + timens_setup_vdso_clock_data(&vdata->aux_clock_data[i], ns); + } +} + +/* + * The vvar page layout depends on whether a task belongs to the root or + * non-root time namespace. Whenever a task changes its namespace, the VVAR + * page tables are cleared and then they will be re-faulted with a + * corresponding layout. + * See also the comment near timens_setup_vdso_clock_data() for details. + */ +static int vdso_join_timens(struct task_struct *task, struct time_namespac= e *ns) +{ + struct mm_struct *mm =3D task->mm; + struct vm_area_struct *vma; + VMA_ITERATOR(vmi, mm, 0); + + guard(mmap_read_lock)(mm); + for_each_vma(vmi, vma) { + if (vma_is_special_mapping(vma, &vdso_vvar_mapping)) + zap_vma_pages(vma); + } + return 0; +} + +void timens_commit(struct task_struct *tsk, struct time_namespace *ns) +{ + timens_set_vvar_page(tsk, ns); + vdso_join_timens(tsk, ns); +} + +int timens_vdso_alloc_vvar_page(struct time_namespace *ns) +{ + ns->vvar_page =3D alloc_page(GFP_KERNEL_ACCOUNT | __GFP_ZERO); + if (!ns->vvar_page) + return -ENOMEM; + + return 0; +} + +void timens_vdso_free_vvar_page(struct time_namespace *ns) +{ + __free_page(ns->vvar_page); +} diff --git a/lib/vdso/datastore.c b/lib/vdso/datastore.c index a565c30c71a0..cf5d784a4a5a 100644 --- a/lib/vdso/datastore.c +++ b/lib/vdso/datastore.c @@ -1,64 +1,92 @@ // SPDX-License-Identifier: GPL-2.0-only =20 -#include -#include +#include +#include #include #include #include #include #include =20 -/* - * The vDSO data page. - */ +static u8 vdso_initdata[VDSO_NR_PAGES * PAGE_SIZE] __aligned(PAGE_SIZE) __= initdata =3D {}; + #ifdef CONFIG_GENERIC_GETTIMEOFDAY -static union { - struct vdso_time_data data; - u8 page[PAGE_SIZE]; -} vdso_time_data_store __page_aligned_data; -struct vdso_time_data *vdso_k_time_data =3D &vdso_time_data_store.data; -static_assert(sizeof(vdso_time_data_store) =3D=3D PAGE_SIZE); +struct vdso_time_data *vdso_k_time_data __refdata =3D + (void *)&vdso_initdata[VDSO_TIME_PAGE_OFFSET * PAGE_SIZE]; + +static_assert(sizeof(struct vdso_time_data) <=3D PAGE_SIZE); #endif /* CONFIG_GENERIC_GETTIMEOFDAY */ =20 #ifdef CONFIG_VDSO_GETRANDOM -static union { - struct vdso_rng_data data; - u8 page[PAGE_SIZE]; -} vdso_rng_data_store __page_aligned_data; -struct vdso_rng_data *vdso_k_rng_data =3D &vdso_rng_data_store.data; -static_assert(sizeof(vdso_rng_data_store) =3D=3D PAGE_SIZE); +struct vdso_rng_data *vdso_k_rng_data __refdata =3D + (void *)&vdso_initdata[VDSO_RNG_PAGE_OFFSET * PAGE_SIZE]; + +static_assert(sizeof(struct vdso_rng_data) <=3D PAGE_SIZE); #endif /* CONFIG_VDSO_GETRANDOM */ =20 #ifdef CONFIG_ARCH_HAS_VDSO_ARCH_DATA -static union { - struct vdso_arch_data data; - u8 page[VDSO_ARCH_DATA_SIZE]; -} vdso_arch_data_store __page_aligned_data; -struct vdso_arch_data *vdso_k_arch_data =3D &vdso_arch_data_store.data; +struct vdso_arch_data *vdso_k_arch_data __refdata =3D + (void *)&vdso_initdata[VDSO_ARCH_PAGES_START * PAGE_SIZE]; #endif /* CONFIG_ARCH_HAS_VDSO_ARCH_DATA */ =20 +void __init vdso_setup_data_pages(void) +{ + unsigned int order =3D get_order(VDSO_NR_PAGES * PAGE_SIZE); + struct page *pages; + + /* + * Allocate the data pages dynamically. SPARC does not support mapping + * static pages to be mapped into userspace. + * It is also a requirement for mlockall() support. + * + * Do not use folios. In time namespaces the pages are mapped in a differ= ent order + * to userspace, which is not handled by the folio optimizations in finis= h_fault(). + */ + pages =3D alloc_pages(GFP_KERNEL, order); + if (!pages) + panic("Unable to allocate VDSO storage pages"); + + /* The pages are mapped one-by-one into userspace and each one needs to b= e refcounted. */ + split_page(pages, order); + + /* Move the data already written by other subsystems to the new pages */ + memcpy(page_address(pages), vdso_initdata, VDSO_NR_PAGES * PAGE_SIZE); + + if (IS_ENABLED(CONFIG_GENERIC_GETTIMEOFDAY)) + vdso_k_time_data =3D page_address(pages + VDSO_TIME_PAGE_OFFSET); + + if (IS_ENABLED(CONFIG_VDSO_GETRANDOM)) + vdso_k_rng_data =3D page_address(pages + VDSO_RNG_PAGE_OFFSET); + + if (IS_ENABLED(CONFIG_ARCH_HAS_VDSO_ARCH_DATA)) + vdso_k_arch_data =3D page_address(pages + VDSO_ARCH_PAGES_START); +} + static vm_fault_t vvar_fault(const struct vm_special_mapping *sm, struct vm_area_struct *vma, struct vm_fault *vmf) { - struct page *timens_page =3D find_timens_vvar_page(vma); - unsigned long addr, pfn; - vm_fault_t err; + struct page *page, *timens_page; + + timens_page =3D find_timens_vvar_page(vma); =20 switch (vmf->pgoff) { case VDSO_TIME_PAGE_OFFSET: if (!IS_ENABLED(CONFIG_GENERIC_GETTIMEOFDAY)) return VM_FAULT_SIGBUS; - pfn =3D __phys_to_pfn(__pa_symbol(vdso_k_time_data)); + page =3D virt_to_page(vdso_k_time_data); if (timens_page) { /* * Fault in VVAR page too, since it will be accessed * to get clock data anyway. */ + unsigned long addr; + vm_fault_t err; + addr =3D vmf->address + VDSO_TIMENS_PAGE_OFFSET * PAGE_SIZE; - err =3D vmf_insert_pfn(vma, addr, pfn); + err =3D vmf_insert_page(vma, addr, page); if (unlikely(err & VM_FAULT_ERROR)) return err; - pfn =3D page_to_pfn(timens_page); + page =3D timens_page; } break; case VDSO_TIMENS_PAGE_OFFSET: @@ -71,24 +99,25 @@ static vm_fault_t vvar_fault(const struct vm_special_ma= pping *sm, */ if (!IS_ENABLED(CONFIG_TIME_NS) || !timens_page) return VM_FAULT_SIGBUS; - pfn =3D __phys_to_pfn(__pa_symbol(vdso_k_time_data)); + page =3D virt_to_page(vdso_k_time_data); break; case VDSO_RNG_PAGE_OFFSET: if (!IS_ENABLED(CONFIG_VDSO_GETRANDOM)) return VM_FAULT_SIGBUS; - pfn =3D __phys_to_pfn(__pa_symbol(vdso_k_rng_data)); + page =3D virt_to_page(vdso_k_rng_data); break; case VDSO_ARCH_PAGES_START ... VDSO_ARCH_PAGES_END: if (!IS_ENABLED(CONFIG_ARCH_HAS_VDSO_ARCH_DATA)) return VM_FAULT_SIGBUS; - pfn =3D __phys_to_pfn(__pa_symbol(vdso_k_arch_data)) + - vmf->pgoff - VDSO_ARCH_PAGES_START; + page =3D virt_to_page(vdso_k_arch_data) + vmf->pgoff - VDSO_ARCH_PAGES_S= TART; break; default: return VM_FAULT_SIGBUS; } =20 - return vmf_insert_pfn(vma, vmf->address, pfn); + get_page(page); + vmf->page =3D page; + return 0; } =20 const struct vm_special_mapping vdso_vvar_mapping =3D { @@ -100,31 +129,6 @@ struct vm_area_struct *vdso_install_vvar_mapping(struc= t mm_struct *mm, unsigned { return _install_special_mapping(mm, addr, VDSO_NR_PAGES * PAGE_SIZE, VM_READ | VM_MAYREAD | VM_IO | VM_DONTDUMP | - VM_PFNMAP | VM_SEALED_SYSMAP, + VM_MIXEDMAP | VM_SEALED_SYSMAP, &vdso_vvar_mapping); } - -#ifdef CONFIG_TIME_NS -/* - * The vvar page layout depends on whether a task belongs to the root or - * non-root time namespace. Whenever a task changes its namespace, the VVAR - * page tables are cleared and then they will be re-faulted with a - * corresponding layout. - * See also the comment near timens_setup_vdso_clock_data() for details. - */ -int vdso_join_timens(struct task_struct *task, struct time_namespace *ns) -{ - struct mm_struct *mm =3D task->mm; - struct vm_area_struct *vma; - VMA_ITERATOR(vmi, mm, 0); - - mmap_read_lock(mm); - for_each_vma(vmi, vma) { - if (vma_is_special_mapping(vma, &vdso_vvar_mapping)) - zap_vma_pages(vma); - } - mmap_read_unlock(mm); - - return 0; -} -#endif diff --git a/lib/vdso/getrandom.c b/lib/vdso/getrandom.c index 440f8a6203a6..7e29005aa208 100644 --- a/lib/vdso/getrandom.c +++ b/lib/vdso/getrandom.c @@ -7,8 +7,11 @@ #include #include #include +#include #include +#include #include +#include #include #include =20 diff --git a/lib/vdso/gettimeofday.c b/lib/vdso/gettimeofday.c index 4399e143d43a..a5798bd26d20 100644 --- a/lib/vdso/gettimeofday.c +++ b/lib/vdso/gettimeofday.c @@ -3,8 +3,25 @@ * Generic userspace implementations of gettimeofday() and similar. */ #include +#include #include #include +#include +#include +#include +#include +#include + +/* + * The generic vDSO implementation requires that gettimeofday.h + * provides: + * - __arch_get_hw_counter(): to get the hw counter based on the + * clock_mode. + * - gettimeofday_fallback(): fallback for gettimeofday. + * - clock_gettime_fallback(): fallback for clock_gettime. + * - clock_getres_fallback(): fallback for clock_getres. + */ +#include =20 /* Bring in default accessors */ #include @@ -135,7 +152,7 @@ bool do_hres_timens(const struct vdso_time_data *vdns, = const struct vdso_clock * =20 if (!vdso_get_timestamp(vd, vc, clk, &sec, &ns)) return false; - } while (unlikely(vdso_read_retry(vc, seq))); + } while (vdso_read_retry(vc, seq)); =20 /* Add the namespace offset */ sec +=3D offs->sec; @@ -158,28 +175,12 @@ bool do_hres(const struct vdso_time_data *vd, const s= truct vdso_clock *vc, return false; =20 do { - /* - * Open coded function vdso_read_begin() to handle - * VDSO_CLOCKMODE_TIMENS. Time namespace enabled tasks have a - * special VVAR page installed which has vc->seq set to 1 and - * vc->clock_mode set to VDSO_CLOCKMODE_TIMENS. For non time - * namespace affected tasks this does not affect performance - * because if vc->seq is odd, i.e. a concurrent update is in - * progress the extra check for vc->clock_mode is just a few - * extra instructions while spin waiting for vc->seq to become - * even again. - */ - while (unlikely((seq =3D READ_ONCE(vc->seq)) & 1)) { - if (IS_ENABLED(CONFIG_TIME_NS) && - vc->clock_mode =3D=3D VDSO_CLOCKMODE_TIMENS) - return do_hres_timens(vd, vc, clk, ts); - cpu_relax(); - } - smp_rmb(); + if (vdso_read_begin_timens(vc, &seq)) + return do_hres_timens(vd, vc, clk, ts); =20 if (!vdso_get_timestamp(vd, vc, clk, &sec, &ns)) return false; - } while (unlikely(vdso_read_retry(vc, seq))); + } while (vdso_read_retry(vc, seq)); =20 vdso_set_timespec(ts, sec, ns); =20 @@ -204,7 +205,7 @@ bool do_coarse_timens(const struct vdso_time_data *vdns= , const struct vdso_clock seq =3D vdso_read_begin(vc); sec =3D vdso_ts->sec; nsec =3D vdso_ts->nsec; - } while (unlikely(vdso_read_retry(vc, seq))); + } while (vdso_read_retry(vc, seq)); =20 /* Add the namespace offset */ sec +=3D offs->sec; @@ -223,21 +224,12 @@ bool do_coarse(const struct vdso_time_data *vd, const= struct vdso_clock *vc, u32 seq; =20 do { - /* - * Open coded function vdso_read_begin() to handle - * VDSO_CLOCK_TIMENS. See comment in do_hres(). - */ - while ((seq =3D READ_ONCE(vc->seq)) & 1) { - if (IS_ENABLED(CONFIG_TIME_NS) && - vc->clock_mode =3D=3D VDSO_CLOCKMODE_TIMENS) - return do_coarse_timens(vd, vc, clk, ts); - cpu_relax(); - } - smp_rmb(); + if (vdso_read_begin_timens(vc, &seq)) + return do_coarse_timens(vd, vc, clk, ts); =20 ts->tv_sec =3D vdso_ts->sec; ts->tv_nsec =3D vdso_ts->nsec; - } while (unlikely(vdso_read_retry(vc, seq))); + } while (vdso_read_retry(vc, seq)); =20 return true; } @@ -256,20 +248,12 @@ bool do_aux(const struct vdso_time_data *vd, clockid_= t clock, struct __kernel_ti vc =3D &vd->aux_clock_data[idx]; =20 do { - /* - * Open coded function vdso_read_begin() to handle - * VDSO_CLOCK_TIMENS. See comment in do_hres(). - */ - while ((seq =3D READ_ONCE(vc->seq)) & 1) { - if (IS_ENABLED(CONFIG_TIME_NS) && vc->clock_mode =3D=3D VDSO_CLOCKMODE_= TIMENS) { - vd =3D __arch_get_vdso_u_timens_data(vd); - vc =3D &vd->aux_clock_data[idx]; - /* Re-read from the real time data page */ - continue; - } - cpu_relax(); + if (vdso_read_begin_timens(vc, &seq)) { + vd =3D __arch_get_vdso_u_timens_data(vd); + vc =3D &vd->aux_clock_data[idx]; + /* Re-read from the real time data page */ + continue; } - smp_rmb(); =20 /* Auxclock disabled? */ if (vc->clock_mode =3D=3D VDSO_CLOCKMODE_NONE) @@ -277,7 +261,7 @@ bool do_aux(const struct vdso_time_data *vd, clockid_t = clock, struct __kernel_ti =20 if (!vdso_get_timestamp(vd, vc, VDSO_BASE_AUX, &sec, &ns)) return false; - } while (unlikely(vdso_read_retry(vc, seq))); + } while (vdso_read_retry(vc, seq)); =20 vdso_set_timespec(ts, sec, ns); =20 @@ -313,7 +297,7 @@ __cvdso_clock_gettime_common(const struct vdso_time_dat= a *vd, clockid_t clock, return do_hres(vd, vc, clock, ts); } =20 -static __maybe_unused int +static int __cvdso_clock_gettime_data(const struct vdso_time_data *vd, clockid_t cloc= k, struct __kernel_timespec *ts) { @@ -333,7 +317,7 @@ __cvdso_clock_gettime(clockid_t clock, struct __kernel_= timespec *ts) } =20 #ifdef BUILD_VDSO32 -static __maybe_unused int +static int __cvdso_clock_gettime32_data(const struct vdso_time_data *vd, clockid_t cl= ock, struct old_timespec32 *res) { @@ -359,7 +343,7 @@ __cvdso_clock_gettime32(clockid_t clock, struct old_tim= espec32 *res) } #endif /* BUILD_VDSO32 */ =20 -static __maybe_unused int +static int __cvdso_gettimeofday_data(const struct vdso_time_data *vd, struct __kernel_old_timeval *tv, struct timezone *tz) { @@ -376,8 +360,7 @@ __cvdso_gettimeofday_data(const struct vdso_time_data *= vd, } =20 if (unlikely(tz !=3D NULL)) { - if (IS_ENABLED(CONFIG_TIME_NS) && - vc->clock_mode =3D=3D VDSO_CLOCKMODE_TIMENS) + if (vdso_is_timens_clock(vc)) vd =3D __arch_get_vdso_u_timens_data(vd); =20 tz->tz_minuteswest =3D vd[CS_HRES_COARSE].tz_minuteswest; @@ -394,14 +377,13 @@ __cvdso_gettimeofday(struct __kernel_old_timeval *tv,= struct timezone *tz) } =20 #ifdef VDSO_HAS_TIME -static __maybe_unused __kernel_old_time_t +static __kernel_old_time_t __cvdso_time_data(const struct vdso_time_data *vd, __kernel_old_time_t *ti= me) { const struct vdso_clock *vc =3D vd->clock_data; __kernel_old_time_t t; =20 - if (IS_ENABLED(CONFIG_TIME_NS) && - vc->clock_mode =3D=3D VDSO_CLOCKMODE_TIMENS) { + if (vdso_is_timens_clock(vc)) { vd =3D __arch_get_vdso_u_timens_data(vd); vc =3D vd->clock_data; } @@ -432,8 +414,7 @@ bool __cvdso_clock_getres_common(const struct vdso_time= _data *vd, clockid_t cloc if (!vdso_clockid_valid(clock)) return false; =20 - if (IS_ENABLED(CONFIG_TIME_NS) && - vc->clock_mode =3D=3D VDSO_CLOCKMODE_TIMENS) + if (vdso_is_timens_clock(vc)) vd =3D __arch_get_vdso_u_timens_data(vd); =20 /* @@ -464,7 +445,7 @@ bool __cvdso_clock_getres_common(const struct vdso_time= _data *vd, clockid_t cloc return true; } =20 -static __maybe_unused +static int __cvdso_clock_getres_data(const struct vdso_time_data *vd, clockid_t c= lock, struct __kernel_timespec *res) { @@ -484,7 +465,7 @@ int __cvdso_clock_getres(clockid_t clock, struct __kern= el_timespec *res) } =20 #ifdef BUILD_VDSO32 -static __maybe_unused int +static int __cvdso_clock_getres_time32_data(const struct vdso_time_data *vd, clockid_= t clock, struct old_timespec32 *res) { diff --git a/tools/testing/selftests/vDSO/Makefile b/tools/testing/selftest= s/vDSO/Makefile index e361aca22a74..a61047bdcd57 100644 --- a/tools/testing/selftests/vDSO/Makefile +++ b/tools/testing/selftests/vDSO/Makefile @@ -19,8 +19,6 @@ endif =20 include ../lib.mk =20 -CFLAGS +=3D $(TOOLS_INCLUDES) - CFLAGS_NOLIBC :=3D -nostdlib -nostdinc -ffreestanding -fno-asynchronous-un= wind-tables \ -fno-stack-protector -include $(top_srcdir)/tools/include/nolibc/nolibc= .h \ -I$(top_srcdir)/tools/include/nolibc/ $(KHDR_INCLUDES) @@ -28,13 +26,11 @@ CFLAGS_NOLIBC :=3D -nostdlib -nostdinc -ffreestanding -= fno-asynchronous-unwind-tab $(OUTPUT)/vdso_test_gettimeofday: parse_vdso.c vdso_test_gettimeofday.c $(OUTPUT)/vdso_test_getcpu: parse_vdso.c vdso_test_getcpu.c $(OUTPUT)/vdso_test_abi: parse_vdso.c vdso_test_abi.c +$(OUTPUT)/vdso_test_correctness: parse_vdso.c vdso_test_correctness.c =20 $(OUTPUT)/vdso_standalone_test_x86: vdso_standalone_test_x86.c parse_vdso.= c | headers $(OUTPUT)/vdso_standalone_test_x86: CFLAGS:=3D$(CFLAGS_NOLIBC) $(CFLAGS) =20 -$(OUTPUT)/vdso_test_correctness: vdso_test_correctness.c -$(OUTPUT)/vdso_test_correctness: LDFLAGS +=3D -ldl - $(OUTPUT)/vdso_test_getrandom: parse_vdso.c $(OUTPUT)/vdso_test_getrandom: CFLAGS +=3D -isystem $(top_srcdir)/tools/in= clude \ $(KHDR_INCLUDES) \ diff --git a/tools/testing/selftests/vDSO/parse_vdso.c b/tools/testing/self= tests/vDSO/parse_vdso.c index 3ff00fb624a4..c6ff4413ea36 100644 --- a/tools/testing/selftests/vDSO/parse_vdso.c +++ b/tools/testing/selftests/vDSO/parse_vdso.c @@ -19,8 +19,7 @@ #include #include #include -#include -#include +#include =20 #include "parse_vdso.h" =20 diff --git a/tools/testing/selftests/vDSO/vdso_test_correctness.c b/tools/t= esting/selftests/vDSO/vdso_test_correctness.c index 055af95aa552..5c5a07dd1128 100644 --- a/tools/testing/selftests/vDSO/vdso_test_correctness.c +++ b/tools/testing/selftests/vDSO/vdso_test_correctness.c @@ -11,28 +11,22 @@ #include #include #include +#include #include -#include #include #include #include #include #include =20 +#include "parse_vdso.h" #include "vdso_config.h" #include "vdso_call.h" #include "kselftest.h" =20 +static const char *version; static const char **name; =20 -#ifndef SYS_getcpu -# ifdef __x86_64__ -# define SYS_getcpu 309 -# else -# define SYS_getcpu 318 -# endif -#endif - #ifndef __NR_clock_gettime64 #define __NR_clock_gettime64 403 #endif @@ -61,6 +55,10 @@ typedef long (*vgtod_t)(struct timeval *tv, struct timez= one *tz); =20 vgtod_t vdso_gettimeofday; =20 +typedef time_t (*vtime_t)(__kernel_time_t *tloc); + +vtime_t vdso_time; + typedef long (*getcpu_t)(unsigned *, unsigned *, void *); =20 getcpu_t vgetcpu; @@ -110,42 +108,39 @@ static void *vsyscall_getcpu(void) =20 static void fill_function_pointers(void) { - void *vdso =3D dlopen("linux-vdso.so.1", - RTLD_LAZY | RTLD_LOCAL | RTLD_NOLOAD); - if (!vdso) - vdso =3D dlopen("linux-gate.so.1", - RTLD_LAZY | RTLD_LOCAL | RTLD_NOLOAD); - if (!vdso) - vdso =3D dlopen("linux-vdso32.so.1", - RTLD_LAZY | RTLD_LOCAL | RTLD_NOLOAD); - if (!vdso) - vdso =3D dlopen("linux-vdso64.so.1", - RTLD_LAZY | RTLD_LOCAL | RTLD_NOLOAD); - if (!vdso) { + unsigned long sysinfo_ehdr =3D getauxval(AT_SYSINFO_EHDR); + + if (!sysinfo_ehdr) { printf("[WARN]\tfailed to find vDSO\n"); return; } =20 - vdso_getcpu =3D (getcpu_t)dlsym(vdso, name[4]); + vdso_init_from_sysinfo_ehdr(sysinfo_ehdr); + + vdso_getcpu =3D (getcpu_t)vdso_sym(version, name[4]); if (!vdso_getcpu) printf("Warning: failed to find getcpu in vDSO\n"); =20 vgetcpu =3D (getcpu_t) vsyscall_getcpu(); =20 - vdso_clock_gettime =3D (vgettime_t)dlsym(vdso, name[1]); + vdso_clock_gettime =3D (vgettime_t)vdso_sym(version, name[1]); if (!vdso_clock_gettime) printf("Warning: failed to find clock_gettime in vDSO\n"); =20 #if defined(VDSO_32BIT) - vdso_clock_gettime64 =3D (vgettime64_t)dlsym(vdso, name[5]); + vdso_clock_gettime64 =3D (vgettime64_t)vdso_sym(version, name[5]); if (!vdso_clock_gettime64) printf("Warning: failed to find clock_gettime64 in vDSO\n"); #endif =20 - vdso_gettimeofday =3D (vgtod_t)dlsym(vdso, name[0]); + vdso_gettimeofday =3D (vgtod_t)vdso_sym(version, name[0]); if (!vdso_gettimeofday) printf("Warning: failed to find gettimeofday in vDSO\n"); =20 + vdso_time =3D (vtime_t)vdso_sym(version, name[2]); + if (!vdso_time) + printf("Warning: failed to find time in vDSO\n"); + } =20 static long sys_getcpu(unsigned * cpu, unsigned * node, @@ -169,6 +164,16 @@ static inline int sys_gettimeofday(struct timeval *tv,= struct timezone *tz) return syscall(__NR_gettimeofday, tv, tz); } =20 +static inline __kernel_old_time_t sys_time(__kernel_old_time_t *tloc) +{ +#ifdef __NR_time + return syscall(__NR_time, tloc); +#else + errno =3D ENOSYS; + return -1; +#endif +} + static void test_getcpu(void) { printf("[RUN]\tTesting getcpu...\n"); @@ -412,10 +417,10 @@ static void test_gettimeofday(void) return; } =20 - printf("\t%llu.%06ld %llu.%06ld %llu.%06ld\n", - (unsigned long long)start.tv_sec, start.tv_usec, - (unsigned long long)vdso.tv_sec, vdso.tv_usec, - (unsigned long long)end.tv_sec, end.tv_usec); + printf("\t%llu.%06lld %llu.%06lld %llu.%06lld\n", + (unsigned long long)start.tv_sec, (long long)start.tv_usec, + (unsigned long long)vdso.tv_sec, (long long)vdso.tv_usec, + (unsigned long long)end.tv_sec, (long long)end.tv_usec); =20 if (!tv_leq(&start, &vdso) || !tv_leq(&vdso, &end)) { printf("[FAIL]\tTimes are out of sequence\n"); @@ -435,8 +440,56 @@ static void test_gettimeofday(void) VDSO_CALL(vdso_gettimeofday, 2, &vdso, NULL); } =20 +static void test_time(void) +{ + __kernel_old_time_t start, end, vdso_ret, vdso_param; + + if (!vdso_time) + return; + + printf("[RUN]\tTesting time...\n"); + + if (sys_time(&start) < 0) { + if (errno =3D=3D -ENOSYS) { + printf("[SKIP]\tNo time() support\n"); + } else { + printf("[FAIL]\tsys_time failed (%d)\n", errno); + nerrs++; + } + return; + } + + vdso_ret =3D VDSO_CALL(vdso_time, 1, &vdso_param); + end =3D sys_time(NULL); + + if (vdso_ret < 0 || end < 0) { + printf("[FAIL]\tvDSO returned %d, syscall errno=3D%d\n", + (int)vdso_ret, errno); + nerrs++; + return; + } + + printf("\t%lld %lld %lld\n", + (long long)start, + (long long)vdso_ret, + (long long)end); + + if (vdso_ret !=3D vdso_param) { + printf("[FAIL]\tinconsistent return values: %lld %lld\n", + (long long)vdso_ret, (long long)vdso_param); + nerrs++; + return; + } + + if (!(start <=3D vdso_ret) || !(vdso_ret <=3D end)) { + printf("[FAIL]\tTimes are out of sequence\n"); + nerrs++; + } +} + int main(int argc, char **argv) { + version =3D versions[VDSO_VERSION]; name =3D (const char **)&names[VDSO_NAMES]; =20 fill_function_pointers(); @@ -444,6 +497,7 @@ int main(int argc, char **argv) test_clock_gettime(); test_clock_gettime64(); test_gettimeofday(); + test_time(); =20 /* * Test getcpu() last so that, if something goes wrong setting affinity, diff --git a/tools/testing/selftests/vDSO/vdso_test_gettimeofday.c b/tools/= testing/selftests/vDSO/vdso_test_gettimeofday.c index 912edadad92c..990b29e0e272 100644 --- a/tools/testing/selftests/vDSO/vdso_test_gettimeofday.c +++ b/tools/testing/selftests/vDSO/vdso_test_gettimeofday.c @@ -11,10 +11,8 @@ */ =20 #include -#ifndef NOLIBC #include #include -#endif =20 #include "kselftest.h" #include "parse_vdso.h"