X86_FEATURE_CONSTANT_TSC is a Linux-defined, synthesized feature flag.
It is used across several vendors. Intel CPUs will set the feature when
the architectural CPUID.80000007.EDX[1] bit is set. There are also some
Intel CPUs that have the X86_FEATURE_CONSTANT_TSC behavior but don't
enumerate it with the architectural bit. Those currently have a model
range check.
Today, virtually all of the CPUs that have the CPUID bit *also* match
the "model >= 0x0e" check. This is confusing. Instead of an open-ended
check, pick some models (INTEL_IVYBRIDGE and P4_WILLAMETTE) as the end
of goofy CPUs that should enumerate the bit but don't. These models are
relatively arbitrary but conservative pick for this.
This makes it obvious that later CPUs (like Family 18+) no longer need
to synthesize X86_FEATURE_CONSTANT_TSC.
Signed-off-by: Sohil Mehta <sohil.mehta@intel.com>
---
v3: Make the non-architectural model checks more explicit.
Improve commit message.
v2: No change.
---
arch/x86/kernel/cpu/intel.c | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/arch/x86/kernel/cpu/intel.c b/arch/x86/kernel/cpu/intel.c
index fc68561d9f92..4fbc5465ca67 100644
--- a/arch/x86/kernel/cpu/intel.c
+++ b/arch/x86/kernel/cpu/intel.c
@@ -210,10 +210,6 @@ static void early_init_intel(struct cpuinfo_x86 *c)
{
u64 misc_enable;
- if ((c->x86 == 0xf && c->x86_model >= 0x03) ||
- (c->x86 == 0x6 && c->x86_model >= 0x0e))
- set_cpu_cap(c, X86_FEATURE_CONSTANT_TSC);
-
if (c->x86 >= 6 && !cpu_has(c, X86_FEATURE_IA64))
c->microcode = intel_get_microcode_revision();
@@ -266,10 +262,16 @@ static void early_init_intel(struct cpuinfo_x86 *c)
*
* It is also reliable across cores and sockets. (but not across
* cabinets - we turn it off in that case explicitly.)
+ *
+ * Use a model-specific check for some older CPUs that have invariant
+ * TSC but may not report it architecturally via 8000_0007.
*/
if (c->x86_power & (1 << 8)) {
set_cpu_cap(c, X86_FEATURE_CONSTANT_TSC);
set_cpu_cap(c, X86_FEATURE_NONSTOP_TSC);
+ } else if ((c->x86_vfm >= INTEL_P4_PRESCOTT && c->x86_vfm <= INTEL_P4_WILLAMETTE) ||
+ (c->x86_vfm >= INTEL_CORE_YONAH && c->x86_vfm <= INTEL_IVYBRIDGE)) {
+ set_cpu_cap(c, X86_FEATURE_CONSTANT_TSC);
}
/* Penwell and Cloverview have the TSC which doesn't sleep on S3 */
--
2.43.0On Wed, 2025-02-19 at 18:41 +0000, Sohil Mehta wrote:
> X86_FEATURE_CONSTANT_TSC is a Linux-defined, synthesized feature flag.
> It is used across several vendors. Intel CPUs will set the feature when
> the architectural CPUID.80000007.EDX[1] bit is set. There are also some
> Intel CPUs that have the X86_FEATURE_CONSTANT_TSC behavior but don't
> enumerate it with the architectural bit. Those currently have a model
> range check.
>
> Today, virtually all of the CPUs that have the CPUID bit *also* match
> the "model >= 0x0e" check. This is confusing. Instead of an open-ended
> check, pick some models (INTEL_IVYBRIDGE and P4_WILLAMETTE) as the end
> of goofy CPUs that should enumerate the bit but don't. These models are
> relatively arbitrary but conservative pick for this.
>
> This makes it obvious that later CPUs (like Family 18+) no longer need
> to synthesize X86_FEATURE_CONSTANT_TSC.
>
> Signed-off-by: Sohil Mehta <sohil.mehta@intel.com>
> ---
> v3: Make the non-architectural model checks more explicit.
> Improve commit message.
>
> v2: No change.
> ---
> arch/x86/kernel/cpu/intel.c | 10 ++++++----
> 1 file changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/arch/x86/kernel/cpu/intel.c b/arch/x86/kernel/cpu/intel.c
> index fc68561d9f92..4fbc5465ca67 100644
> --- a/arch/x86/kernel/cpu/intel.c
> +++ b/arch/x86/kernel/cpu/intel.c
> @@ -210,10 +210,6 @@ static void early_init_intel(struct cpuinfo_x86 *c)
> {
> u64 misc_enable;
>
> - if ((c->x86 == 0xf && c->x86_model >= 0x03) ||
> - (c->x86 == 0x6 && c->x86_model >= 0x0e))
> - set_cpu_cap(c, X86_FEATURE_CONSTANT_TSC);
> -
> if (c->x86 >= 6 && !cpu_has(c, X86_FEATURE_IA64))
> c->microcode = intel_get_microcode_revision();
>
> @@ -266,10 +262,16 @@ static void early_init_intel(struct cpuinfo_x86 *c)
> *
> * It is also reliable across cores and sockets. (but not across
> * cabinets - we turn it off in that case explicitly.)
> + *
> + * Use a model-specific check for some older CPUs that have invariant
> + * TSC but may not report it architecturally via 8000_0007.
> */
> if (c->x86_power & (1 << 8)) {
> set_cpu_cap(c, X86_FEATURE_CONSTANT_TSC);
> set_cpu_cap(c, X86_FEATURE_NONSTOP_TSC);
> + } else if ((c->x86_vfm >= INTEL_P4_PRESCOTT && c->x86_vfm <= INTEL_P4_WILLAMETTE) ||
> + (c->x86_vfm >= INTEL_CORE_YONAH && c->x86_vfm <= INTEL_IVYBRIDGE)) {
> + set_cpu_cap(c, X86_FEATURE_CONSTANT_TSC);
> }
>
> /* Penwell and Cloverview have the TSC which doesn't sleep on S3 */
Hm. My test host is INTEL_HASWELL_X (0x63f). For reasons which are
unclear to me, QEMU doesn't set bit 8 of 0x80000007 EDX unless I
explicitly append ',+invtsc' to the existing '-cpu host' on its command
line. So now my guest doesn't think it has X86_FEATURE_CONSTANT_TSC.
For reasons I also don't understand, for a Xen (in qemu/kvm) guest this
results in about a four-second delay when bringing up each vCPU.
Timestamps added to QEMU's stdout because the kernel's own timestamps
are lying...
1755781767: [ 0.489434] smp: Bringing up secondary CPUs ...
1755781767: [ 0.489434] installing Xen timer for CPU 1
1755781767: [ 0.489434] smpboot: x86: Booting SMP configuration:
1755781767: [ 0.489434] .... node #0, CPUs: #1
1755781767: [ 0.489434] installing Xen timer for CPU 2
1755781767: [ 0.489434] #2
1755781767: [ 0.489434] installing Xen timer for CPU 3
1755781767: [ 0.489434] #3
1755781771: [ 0.489434] cpu 1 spinlock event irq 45
1755781775: [ 0.544262] cpu 2 spinlock event irq 46
1755781779: [ 0.604306] cpu 3 spinlock event irq 47
1755781779: [ 0.604792] smp: Brought up 1 node, 4 CPUs
1755781779: [ 0.604792] smpboot: Total of 4 processors activated (1034118.38 BogoMIPS)
On 8/21/2025 6:15 AM, David Woodhouse wrote:
> Hm. My test host is INTEL_HASWELL_X (0x63f). For reasons which are
> unclear to me, QEMU doesn't set bit 8 of 0x80000007 EDX unless I
> explicitly append ',+invtsc' to the existing '-cpu host' on its command
> line. So now my guest doesn't think it has X86_FEATURE_CONSTANT_TSC.
>
Haswell should have X86_FEATURE_CONSTANT_TSC, so I would have expected
the guest bit to be set. Until now, X86_FEATURE_CONSTANT_TSC was set
based on the Family-model instead of the CPUID enumeration which may
have hid the issue.
From my initial look at the QEMU implementation, this seems intentional.
QEMU considers Invariant TSC as un-migratable which prevents it from
being exposed to migratable guests (default).
target/i386/cpu.c:
[FEAT_8000_0007_EDX]
.unmigratable_flags = CPUID_APM_INVTSC,
Can you please try '-cpu host,migratable=off'?
On 8/21/2025 12:34 PM, Sohil Mehta wrote: > On 8/21/2025 6:15 AM, David Woodhouse wrote: > >> Hm. My test host is INTEL_HASWELL_X (0x63f). For reasons which are >> unclear to me, QEMU doesn't set bit 8 of 0x80000007 EDX unless I >> explicitly append ',+invtsc' to the existing '-cpu host' on its command >> line. So now my guest doesn't think it has X86_FEATURE_CONSTANT_TSC. >> > > Haswell should have X86_FEATURE_CONSTANT_TSC, so I would have expected > the guest bit to be set. Until now, X86_FEATURE_CONSTANT_TSC was set > based on the Family-model instead of the CPUID enumeration which may > have hid the issue. > Correction: s/instead/as well as > From my initial look at the QEMU implementation, this seems intentional. > > QEMU considers Invariant TSC as un-migratable which prevents it from > being exposed to migratable guests (default). > target/i386/cpu.c: > [FEAT_8000_0007_EDX] > .unmigratable_flags = CPUID_APM_INVTSC, > > Can you please try '-cpu host,migratable=off'? This is mainly to verify. If confirmed, I am not sure what the long term solution should be.
On 8/22/2025 3:43 AM, Sohil Mehta wrote: > On 8/21/2025 12:34 PM, Sohil Mehta wrote: >> On 8/21/2025 6:15 AM, David Woodhouse wrote: >> >>> Hm. My test host is INTEL_HASWELL_X (0x63f). For reasons which are >>> unclear to me, QEMU doesn't set bit 8 of 0x80000007 EDX unless I >>> explicitly append ',+invtsc' to the existing '-cpu host' on its command >>> line. So now my guest doesn't think it has X86_FEATURE_CONSTANT_TSC. >>> >> >> Haswell should have X86_FEATURE_CONSTANT_TSC, so I would have expected >> the guest bit to be set. Until now, X86_FEATURE_CONSTANT_TSC was set >> based on the Family-model instead of the CPUID enumeration which may >> have hid the issue. >> > > Correction: > s/instead/as well as > >> From my initial look at the QEMU implementation, this seems intentional. >> >> QEMU considers Invariant TSC as un-migratable which prevents it from >> being exposed to migratable guests (default). >> target/i386/cpu.c: >> [FEAT_8000_0007_EDX] >> .unmigratable_flags = CPUID_APM_INVTSC, >> >> Can you please try '-cpu host,migratable=off'? > > This is mainly to verify. If confirmed, I am not sure what the long term > solution should be. yeah. It's the intentional behavior of QEMU. Invariant TSC is ummigratable unless users explicitly configures the TSC frequency, e.g., "-cpu host,tsc-frequency=xxx". Because the TSC frequency is by default the host's frequency if no "tsc-frequency" specified, and it will change when the VM is migrated to a host with a different TSC frequency. It's the specific behavior/rule of QEMU. We just need to keep it in mind. If we want to expose invariant TSC to the guest with QEMU's "-cpu host", we can either: 1) explicitly configure the "tsc-frequency", or 2) explicitly turn off "migratable"
On 8/21/25 21:46, Xiaoyao Li wrote: > On 8/22/2025 3:43 AM, Sohil Mehta wrote: >> On 8/21/2025 12:34 PM, Sohil Mehta wrote: >>> On 8/21/2025 6:15 AM, David Woodhouse wrote: >>> >>>> Hm. My test host is INTEL_HASWELL_X (0x63f). For reasons which are >>>> unclear to me, QEMU doesn't set bit 8 of 0x80000007 EDX unless I >>>> explicitly append ',+invtsc' to the existing '-cpu host' on its command >>>> line. So now my guest doesn't think it has X86_FEATURE_CONSTANT_TSC. >>>> >>> >>> Haswell should have X86_FEATURE_CONSTANT_TSC, so I would have expected >>> the guest bit to be set. Until now, X86_FEATURE_CONSTANT_TSC was set >>> based on the Family-model instead of the CPUID enumeration which may >>> have hid the issue. >>> >> >> Correction: >> s/instead/as well as >> >>> From my initial look at the QEMU implementation, this seems intentional. >>> >>> QEMU considers Invariant TSC as un-migratable which prevents it from >>> being exposed to migratable guests (default). >>> target/i386/cpu.c: >>> [FEAT_8000_0007_EDX] >>> .unmigratable_flags = CPUID_APM_INVTSC, >>> >>> Can you please try '-cpu host,migratable=off'? >> >> This is mainly to verify. If confirmed, I am not sure what the long term >> solution should be. > > yeah. It's the intentional behavior of QEMU. > > Invariant TSC is ummigratable unless users explicitly configures the TSC > frequency, e.g., "-cpu host,tsc-frequency=xxx". Because the TSC > frequency is by default the host's frequency if no "tsc-frequency" > specified, and it will change when the VM is migrated to a host with a > different TSC frequency. > > It's the specific behavior/rule of QEMU. We just need to keep it in > mind. If we want to expose invariant TSC to the guest with QEMU's "-cpu > host", we can either: > 1) explicitly configure the "tsc-frequency", or > 2) explicitly turn off "migratable" Could the TSC frequency be included in the migration stream? -- Sincerely, Demi Marie Obenour (she/her/hers)
On Thu, 2025-08-21 at 12:43 -0700, Sohil Mehta wrote:
> On 8/21/2025 12:34 PM, Sohil Mehta wrote:
> > On 8/21/2025 6:15 AM, David Woodhouse wrote:
> >
> > > Hm. My test host is INTEL_HASWELL_X (0x63f). For reasons which are
> > > unclear to me, QEMU doesn't set bit 8 of 0x80000007 EDX unless I
> > > explicitly append ',+invtsc' to the existing '-cpu host' on its command
> > > line. So now my guest doesn't think it has X86_FEATURE_CONSTANT_TSC.
> > >
> >
> > Haswell should have X86_FEATURE_CONSTANT_TSC, so I would have expected
> > the guest bit to be set. Until now, X86_FEATURE_CONSTANT_TSC was set
> > based on the Family-model instead of the CPUID enumeration which may
> > have hid the issue.
> >
>
> Correction:
> s/instead/as well as
>
> > From my initial look at the QEMU implementation, this seems intentional.
> >
> > QEMU considers Invariant TSC as un-migratable which prevents it from
> > being exposed to migratable guests (default).
> > target/i386/cpu.c:
> > [FEAT_8000_0007_EDX]
> > .unmigratable_flags = CPUID_APM_INVTSC,
> >
> > Can you please try '-cpu host,migratable=off'?
>
> This is mainly to verify. If confirmed, I am not sure what the long term
> solution should be.
Yes, explicitly turning it on with -cpu host,+invtsc does work.
I've been looking into why it takes a Xen guest four seconds per vCPU
in this case, but not a KVM guest.
When running as a KVM guest, Linux will infer the TSC frequency from
the KVM clock — or better still, from CPUID; see
https://lore.kernel.org/all/20250816101308.2594298-1-dwmw2@infradead.org
and/or
https://lore.kernel.org/all/20250227021855.3257188-36-seanjc@google.com
As a Xen guest though, Linux doesn't do that. This patch in the guest
should make it work without recalibrating the TSC for each vCPU...
--- a/arch/x86/xen/time.c
+++ b/arch/x86/xen/time.c
@@ -489,7 +489,15 @@ static void xen_setup_vsyscall_time_info(void)
*/
static int __init xen_tsc_safe_clocksource(void)
{
- u32 eax, ebx, ecx, edx;
+ u32 eax, ebx, ecx, edx;
+ u64 lpj;
+
+ /* Leaf 4, sub-leaf 0 (0x40000x03) */
+ cpuid_count(xen_cpuid_base() + 3, 0, &eax, &ebx, &ecx, &edx);
+
+ lpj = ((u64)ecx * 1000);
+ do_div(lpj, HZ);
+ preset_lpj = lpj;
if (!(boot_cpu_has(X86_FEATURE_CONSTANT_TSC)))
return 0;
@@ -500,9 +508,6 @@ static int __init xen_tsc_safe_clocksource(void)
if (check_tsc_unstable())
return 0;
- /* Leaf 4, sub-leaf 0 (0x40000x03) */
- cpuid_count(xen_cpuid_base() + 3, 0, &eax, &ebx, &ecx, &edx);
-
return ebx == XEN_CPUID_TSC_MODE_NEVER_EMULATE;
}
... but then I got slightly distracted by the question of why I was
getting *nonsense* in those values, and why KVM is 'correcting' EAX in
subleaf 2 which is supposed to be the *host* TSC, not ECX in subleaf
zero...
Under the Fedora 6.13.8-200 kernel I'm fairly sure the guest was seeing
values in subleaf 0 ECX/EDX that *should* have been in subleaf 1
ECX/EDX, and that problem went away when I rebooted the host into a
mainline kernel. Will have to go back and retest that part...
The following commit has been merged into the x86/core branch of tip:
Commit-ID: fadb6f569b10bf668677add876ed50586931b8f3
Gitweb: https://git.kernel.org/tip/fadb6f569b10bf668677add876ed50586931b8f3
Author: Sohil Mehta <sohil.mehta@intel.com>
AuthorDate: Wed, 19 Feb 2025 18:41:31
Committer: Ingo Molnar <mingo@kernel.org>
CommitterDate: Wed, 19 Mar 2025 11:19:56 +01:00
x86/cpu/intel: Limit the non-architectural constant_tsc model checks
X86_FEATURE_CONSTANT_TSC is a Linux-defined, synthesized feature flag.
It is used across several vendors. Intel CPUs will set the feature when
the architectural CPUID.80000007.EDX[1] bit is set. There are also some
Intel CPUs that have the X86_FEATURE_CONSTANT_TSC behavior but don't
enumerate it with the architectural bit. Those currently have a model
range check.
Today, virtually all of the CPUs that have the CPUID bit *also* match
the "model >= 0x0e" check. This is confusing. Instead of an open-ended
check, pick some models (INTEL_IVYBRIDGE and P4_WILLAMETTE) as the end
of goofy CPUs that should enumerate the bit but don't. These models are
relatively arbitrary but conservative pick for this.
This makes it obvious that later CPUs (like Family 18+) no longer need
to synthesize X86_FEATURE_CONSTANT_TSC.
Signed-off-by: Sohil Mehta <sohil.mehta@intel.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/r/20250219184133.816753-14-sohil.mehta@intel.com
---
arch/x86/kernel/cpu/intel.c | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/arch/x86/kernel/cpu/intel.c b/arch/x86/kernel/cpu/intel.c
index 2181304..4cbb2e6 100644
--- a/arch/x86/kernel/cpu/intel.c
+++ b/arch/x86/kernel/cpu/intel.c
@@ -201,10 +201,6 @@ static void early_init_intel(struct cpuinfo_x86 *c)
{
u64 misc_enable;
- if ((c->x86 == 0xf && c->x86_model >= 0x03) ||
- (c->x86 == 0x6 && c->x86_model >= 0x0e))
- set_cpu_cap(c, X86_FEATURE_CONSTANT_TSC);
-
if (c->x86 >= 6 && !cpu_has(c, X86_FEATURE_IA64))
c->microcode = intel_get_microcode_revision();
@@ -257,10 +253,16 @@ static void early_init_intel(struct cpuinfo_x86 *c)
*
* It is also reliable across cores and sockets. (but not across
* cabinets - we turn it off in that case explicitly.)
+ *
+ * Use a model-specific check for some older CPUs that have invariant
+ * TSC but may not report it architecturally via 8000_0007.
*/
if (c->x86_power & (1 << 8)) {
set_cpu_cap(c, X86_FEATURE_CONSTANT_TSC);
set_cpu_cap(c, X86_FEATURE_NONSTOP_TSC);
+ } else if ((c->x86_vfm >= INTEL_P4_PRESCOTT && c->x86_vfm <= INTEL_P4_WILLAMETTE) ||
+ (c->x86_vfm >= INTEL_CORE_YONAH && c->x86_vfm <= INTEL_IVYBRIDGE)) {
+ set_cpu_cap(c, X86_FEATURE_CONSTANT_TSC);
}
/* Penwell and Cloverview have the TSC which doesn't sleep on S3 */The following commit has been merged into the x86/cpu branch of tip:
Commit-ID: 08d9bb5b0d89826fedc5204c8bd2463220465996
Gitweb: https://git.kernel.org/tip/08d9bb5b0d89826fedc5204c8bd2463220465996
Author: Sohil Mehta <sohil.mehta@intel.com>
AuthorDate: Wed, 19 Feb 2025 18:41:31
Committer: Ingo Molnar <mingo@kernel.org>
CommitterDate: Tue, 18 Mar 2025 19:33:47 +01:00
x86/cpu/intel: Limit the non-architectural constant_tsc model checks
X86_FEATURE_CONSTANT_TSC is a Linux-defined, synthesized feature flag.
It is used across several vendors. Intel CPUs will set the feature when
the architectural CPUID.80000007.EDX[1] bit is set. There are also some
Intel CPUs that have the X86_FEATURE_CONSTANT_TSC behavior but don't
enumerate it with the architectural bit. Those currently have a model
range check.
Today, virtually all of the CPUs that have the CPUID bit *also* match
the "model >= 0x0e" check. This is confusing. Instead of an open-ended
check, pick some models (INTEL_IVYBRIDGE and P4_WILLAMETTE) as the end
of goofy CPUs that should enumerate the bit but don't. These models are
relatively arbitrary but conservative pick for this.
This makes it obvious that later CPUs (like Family 18+) no longer need
to synthesize X86_FEATURE_CONSTANT_TSC.
Signed-off-by: Sohil Mehta <sohil.mehta@intel.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/r/20250219184133.816753-14-sohil.mehta@intel.com
---
arch/x86/kernel/cpu/intel.c | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/arch/x86/kernel/cpu/intel.c b/arch/x86/kernel/cpu/intel.c
index 2181304..4cbb2e6 100644
--- a/arch/x86/kernel/cpu/intel.c
+++ b/arch/x86/kernel/cpu/intel.c
@@ -201,10 +201,6 @@ static void early_init_intel(struct cpuinfo_x86 *c)
{
u64 misc_enable;
- if ((c->x86 == 0xf && c->x86_model >= 0x03) ||
- (c->x86 == 0x6 && c->x86_model >= 0x0e))
- set_cpu_cap(c, X86_FEATURE_CONSTANT_TSC);
-
if (c->x86 >= 6 && !cpu_has(c, X86_FEATURE_IA64))
c->microcode = intel_get_microcode_revision();
@@ -257,10 +253,16 @@ static void early_init_intel(struct cpuinfo_x86 *c)
*
* It is also reliable across cores and sockets. (but not across
* cabinets - we turn it off in that case explicitly.)
+ *
+ * Use a model-specific check for some older CPUs that have invariant
+ * TSC but may not report it architecturally via 8000_0007.
*/
if (c->x86_power & (1 << 8)) {
set_cpu_cap(c, X86_FEATURE_CONSTANT_TSC);
set_cpu_cap(c, X86_FEATURE_NONSTOP_TSC);
+ } else if ((c->x86_vfm >= INTEL_P4_PRESCOTT && c->x86_vfm <= INTEL_P4_WILLAMETTE) ||
+ (c->x86_vfm >= INTEL_CORE_YONAH && c->x86_vfm <= INTEL_IVYBRIDGE)) {
+ set_cpu_cap(c, X86_FEATURE_CONSTANT_TSC);
}
/* Penwell and Cloverview have the TSC which doesn't sleep on S3 */© 2016 - 2025 Red Hat, Inc.