[PATCH] i386/xen: consistent locking around Xen singleshot timers

David Woodhouse posted 1 patch 11 months, 2 weeks ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/d61333c6394697d88dd08d48cc7fa730ff8e1987.camel@infradead.org
Maintainers: David Woodhouse <dwmw2@infradead.org>, Paul Durrant <paul@xen.org>, Paolo Bonzini <pbonzini@redhat.com>, Marcelo Tosatti <mtosatti@redhat.com>
There is a newer version of this series
target/i386/kvm/xen-emu.c | 15 +++++++++------
1 file changed, 9 insertions(+), 6 deletions(-)
[PATCH] i386/xen: consistent locking around Xen singleshot timers
Posted by David Woodhouse 11 months, 2 weeks ago
From: David Woodhouse <dwmw@amazon.co.uk>

Coverity points out (CID 1507534) that we sometimes access
env->xen_singleshot_timer_ns under the protection of
env->xen_timers_lock (eg in xen_vcpu_singleshot_timer_event()) and
sometimes not (the specific case Coverity complains about is in
do_vcpu_soft_reset()).

This isn't strictly an issue. There are two modes for the timers; if
the kernel supports the EVTCHN_SEND capability then it handles all the
timer hypercalls and delivery internally, and all we need to do is an
ioctl to get/set the next timer as part of the vCPU state. If the
kernel doesn't have that support, then we do all the emulation within
qemu, and *those* are the code paths where we actually care about the
locking.

But it doesn't hurt to be a little bit more consistent and avoid having
to explain *why* it's OK.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---

On Tue, 2023-05-09 at 15:55 +0100, Peter Maydell wrote:
> Hi; Coverity points out (CID 1507534) that we seem to sometimes
> access env->xen_singleshot_timer_ns under the protection of
> env->xen_timers_lock (eg in xen_vcpu_singleshot_timer_event())
> and sometimes not (the specific case Coverity complains about is
> in do_vcpu_soft_reset()). Is this a false positive, or is there

Thanks. As noted, I think it's harmless but it doesn't hurt to clean it
up a bit. I've pushed this to my tree at
https://git.infradead.org/users/dwmw2/qemu.git/shortlog/refs/heads/xenfv
on top of the other fixes that didn't make 8.0.

 target/i386/kvm/xen-emu.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/target/i386/kvm/xen-emu.c b/target/i386/kvm/xen-emu.c
index d7c7eb8d9c..36d2dd8d02 100644
--- a/target/i386/kvm/xen-emu.c
+++ b/target/i386/kvm/xen-emu.c
@@ -43,6 +43,7 @@
 
 static void xen_vcpu_singleshot_timer_event(void *opaque);
 static void xen_vcpu_periodic_timer_event(void *opaque);
+static int vcpuop_stop_singleshot_timer(CPUState *cs);
 
 #ifdef TARGET_X86_64
 #define hypercall_compat32(longmode) (!(longmode))
@@ -483,6 +484,7 @@ static int kvm_xen_set_vcpu_timer(CPUState *cs)
 
 static void do_set_vcpu_timer_virq(CPUState *cs, run_on_cpu_data data)
 {
+    QEMU_LOCK_GUARD(&X86_CPU(cs)->env.xen_timers_lock);
     kvm_xen_set_vcpu_timer(cs);
 }
 
@@ -545,7 +547,6 @@ static void do_vcpu_soft_reset(CPUState *cs, run_on_cpu_data data)
     env->xen_vcpu_time_info_gpa = INVALID_GPA;
     env->xen_vcpu_runstate_gpa = INVALID_GPA;
     env->xen_vcpu_callback_vector = 0;
-    env->xen_singleshot_timer_ns = 0;
     memset(env->xen_virq, 0, sizeof(env->xen_virq));
 
     set_vcpu_info(cs, INVALID_GPA);
@@ -555,8 +556,13 @@ static void do_vcpu_soft_reset(CPUState *cs, run_on_cpu_data data)
                           INVALID_GPA);
     if (kvm_xen_has_cap(EVTCHN_SEND)) {
         kvm_xen_set_vcpu_callback_vector(cs);
+
+        QEMU_LOCK_GUARD(&X86_CPU(cs)->env.xen_timers_lock);
+        env->xen_singleshot_timer_ns = 0;
         kvm_xen_set_vcpu_timer(cs);
-    }
+    } else {
+        vcpuop_stop_singleshot_timer(cs);
+    };
 
 }
 
@@ -1844,10 +1850,7 @@ int kvm_put_xen_state(CPUState *cs)
     }
 
     if (env->xen_virq[VIRQ_TIMER]) {
-        ret = kvm_xen_set_vcpu_timer(cs);
-        if (ret < 0) {
-            return ret;
-        }
+        do_set_vcpu_timer_virq(cs, RUN_ON_CPU_HOST_INT(env->xen_virq[VIRQ_TIMER]));
     }
     return 0;
 }
-- 
2.34.1


Re: [PATCH] i386/xen: consistent locking around Xen singleshot timers
Posted by Peter Maydell 11 months, 1 week ago
On Mon, 22 May 2023 at 19:52, David Woodhouse <dwmw2@infradead.org> wrote:
>
> From: David Woodhouse <dwmw@amazon.co.uk>
>
> Coverity points out (CID 1507534) that we sometimes access
> env->xen_singleshot_timer_ns under the protection of
> env->xen_timers_lock (eg in xen_vcpu_singleshot_timer_event()) and
> sometimes not (the specific case Coverity complains about is in
> do_vcpu_soft_reset()).
>
> This isn't strictly an issue. There are two modes for the timers; if
> the kernel supports the EVTCHN_SEND capability then it handles all the
> timer hypercalls and delivery internally, and all we need to do is an
> ioctl to get/set the next timer as part of the vCPU state. If the
> kernel doesn't have that support, then we do all the emulation within
> qemu, and *those* are the code paths where we actually care about the
> locking.
>
> But it doesn't hurt to be a little bit more consistent and avoid having
> to explain *why* it's OK.
>
> Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>

Looking a bit more closely at the Coverity lists, there's also
CID 1507968 which is for the access to env->xen_singleshot_timer_ns
in kvm_get_xen_state(), which this patch doesn't touch, I think ?

thanks
-- PMM
Re: [PATCH] i386/xen: consistent locking around Xen singleshot timers
Posted by David Woodhouse 10 months ago
On Fri, 2023-06-02 at 17:58 +0100, Peter Maydell wrote:
> On Mon, 22 May 2023 at 19:52, David Woodhouse <dwmw2@infradead.org> wrote:
> > 
> > From: David Woodhouse <dwmw@amazon.co.uk>
> > 
> > Coverity points out (CID 1507534) that we sometimes access
> > env->xen_singleshot_timer_ns under the protection of
> > env->xen_timers_lock (eg in xen_vcpu_singleshot_timer_event()) and
> > sometimes not (the specific case Coverity complains about is in
> > do_vcpu_soft_reset()).
> > 
> > This isn't strictly an issue. There are two modes for the timers; if
> > the kernel supports the EVTCHN_SEND capability then it handles all the
> > timer hypercalls and delivery internally, and all we need to do is an
> > ioctl to get/set the next timer as part of the vCPU state. If the
> > kernel doesn't have that support, then we do all the emulation within
> > qemu, and *those* are the code paths where we actually care about the
> > locking.
> > 
> > But it doesn't hurt to be a little bit more consistent and avoid having
> > to explain *why* it's OK.
> > 
> > Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
> 
> Looking a bit more closely at the Coverity lists, there's also
> CID 1507968 which is for the access to env->xen_singleshot_timer_ns
> in kvm_get_xen_state(), which this patch doesn't touch, I think ?

That one is basically a false positive since it's for the case where
the kernel is handling the timers and all we do is an ioctl to read the
state. Which is *inherently* racy if any other vCPU can be running and
modifying them at the time. And doesn't have anything to race *with* if
not :)

On the other hand, the one in kvm_put_xen_state() which calls
do_set_singleshot_timer() is for the userspace case, and *that* one
probably wants locking. Again in practice it's probably never going to
have anything to race with, but that's a poor excuse. New patch
follows.
[PATCH v2] i386/xen: consistent locking around Xen singleshot timers
Posted by David Woodhouse 10 months ago
From: David Woodhouse <dwmw@amazon.co.uk>

Coverity points out (CID 1507534, 1507968) that we sometimes access
env->xen_singleshot_timer_ns under the protection of
env->xen_timers_lock and sometimes not.

This isn't always an issue. There are two modes for the timers; if the
kernel supports the EVTCHN_SEND capability then it handles all the timer
hypercalls and delivery internally, and all we use the field for is to
get/set the timer as part of the vCPU state via an ioctl(). If the
kernel doesn't have that support, then we do all the emulation within
qemu, and *those* are the code paths where we actually care about the
locking.

But it doesn't hurt to be a little bit more consistent and avoid having
to explain *why* it's OK.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 target/i386/kvm/xen-emu.c | 36 ++++++++++++++++++++++++++----------
 1 file changed, 26 insertions(+), 10 deletions(-)

diff --git a/target/i386/kvm/xen-emu.c b/target/i386/kvm/xen-emu.c
index d7c7eb8d9c..9946ff0905 100644
--- a/target/i386/kvm/xen-emu.c
+++ b/target/i386/kvm/xen-emu.c
@@ -43,6 +43,7 @@
 
 static void xen_vcpu_singleshot_timer_event(void *opaque);
 static void xen_vcpu_periodic_timer_event(void *opaque);
+static int vcpuop_stop_singleshot_timer(CPUState *cs);
 
 #ifdef TARGET_X86_64
 #define hypercall_compat32(longmode) (!(longmode))
@@ -466,6 +467,7 @@ void kvm_xen_inject_vcpu_callback_vector(uint32_t vcpu_id, int type)
     }
 }
 
+/* Must always be called with xen_timers_lock held */
 static int kvm_xen_set_vcpu_timer(CPUState *cs)
 {
     X86CPU *cpu = X86_CPU(cs);
@@ -483,6 +485,7 @@ static int kvm_xen_set_vcpu_timer(CPUState *cs)
 
 static void do_set_vcpu_timer_virq(CPUState *cs, run_on_cpu_data data)
 {
+    QEMU_LOCK_GUARD(&X86_CPU(cs)->env.xen_timers_lock);
     kvm_xen_set_vcpu_timer(cs);
 }
 
@@ -545,7 +548,6 @@ static void do_vcpu_soft_reset(CPUState *cs, run_on_cpu_data data)
     env->xen_vcpu_time_info_gpa = INVALID_GPA;
     env->xen_vcpu_runstate_gpa = INVALID_GPA;
     env->xen_vcpu_callback_vector = 0;
-    env->xen_singleshot_timer_ns = 0;
     memset(env->xen_virq, 0, sizeof(env->xen_virq));
 
     set_vcpu_info(cs, INVALID_GPA);
@@ -555,8 +557,13 @@ static void do_vcpu_soft_reset(CPUState *cs, run_on_cpu_data data)
                           INVALID_GPA);
     if (kvm_xen_has_cap(EVTCHN_SEND)) {
         kvm_xen_set_vcpu_callback_vector(cs);
+
+        QEMU_LOCK_GUARD(&X86_CPU(cs)->env.xen_timers_lock);
+        env->xen_singleshot_timer_ns = 0;
         kvm_xen_set_vcpu_timer(cs);
-    }
+    } else {
+        vcpuop_stop_singleshot_timer(cs);
+    };
 
 }
 
@@ -1059,6 +1066,10 @@ static int vcpuop_stop_periodic_timer(CPUState *target)
     return 0;
 }
 
+/*
+ * Userspace handling of timer, for older kernels.
+ * Must always be called with xen_timers_lock held.
+ */
 static int do_set_singleshot_timer(CPUState *cs, uint64_t timeout_abs,
                                    bool future, bool linux_wa)
 {
@@ -1086,12 +1097,8 @@ static int do_set_singleshot_timer(CPUState *cs, uint64_t timeout_abs,
         timeout_abs = now + delta;
     }
 
-    qemu_mutex_lock(&env->xen_timers_lock);
-
     timer_mod_ns(env->xen_singleshot_timer, qemu_now + delta);
     env->xen_singleshot_timer_ns = now + delta;
-
-    qemu_mutex_unlock(&env->xen_timers_lock);
     return 0;
 }
 
@@ -1115,6 +1122,7 @@ static int vcpuop_set_singleshot_timer(CPUState *cs, uint64_t arg)
         return -EFAULT;
     }
 
+    QEMU_LOCK_GUARD(&X86_CPU(cs)->env.xen_timers_lock);
     return do_set_singleshot_timer(cs, sst.timeout_abs_ns,
                                    !!(sst.flags & VCPU_SSHOTTMR_future),
                                    false);
@@ -1141,6 +1149,7 @@ static bool kvm_xen_hcall_set_timer_op(struct kvm_xen_exit *exit, X86CPU *cpu,
     if (unlikely(timeout == 0)) {
         err = vcpuop_stop_singleshot_timer(CPU(cpu));
     } else {
+        QEMU_LOCK_GUARD(&X86_CPU(cpu)->env.xen_timers_lock);
         err = do_set_singleshot_timer(CPU(cpu), timeout, false, true);
     }
     exit->u.hcall.result = err;
@@ -1826,6 +1835,7 @@ int kvm_put_xen_state(CPUState *cs)
          * If the kernel has EVTCHN_SEND support then it handles timers too,
          * so the timer will be restored by kvm_xen_set_vcpu_timer() below.
          */
+        QEMU_LOCK_GUARD(&env->xen_timers_lock);
         if (env->xen_singleshot_timer_ns) {
             ret = do_set_singleshot_timer(cs, env->xen_singleshot_timer_ns,
                                     false, false);
@@ -1844,10 +1854,7 @@ int kvm_put_xen_state(CPUState *cs)
     }
 
     if (env->xen_virq[VIRQ_TIMER]) {
-        ret = kvm_xen_set_vcpu_timer(cs);
-        if (ret < 0) {
-            return ret;
-        }
+        do_set_vcpu_timer_virq(cs, RUN_ON_CPU_HOST_INT(env->xen_virq[VIRQ_TIMER]));
     }
     return 0;
 }
@@ -1896,6 +1903,15 @@ int kvm_get_xen_state(CPUState *cs)
         if (ret < 0) {
             return ret;
         }
+
+        /*
+         * This locking is fairly pointless, and is here to appease Coverity.
+         * There is an unavoidable race condition if a different vCPU sets a
+         * timer for this vCPU after the value has been read out. But that's
+         * OK in practice because *all* the vCPUs need to be stopped before
+         * we set about migrating their state.
+         */
+        QEMU_LOCK_GUARD(&X86_CPU(cs)->env.xen_timers_lock);
         env->xen_singleshot_timer_ns = va.u.timer.expires_ns;
     }
 
-- 
2.34.1


Re: [PATCH v2] i386/xen: consistent locking around Xen singleshot timers
Posted by Paul Durrant 10 months ago
On 04/07/2023 16:51, David Woodhouse wrote:
> From: David Woodhouse <dwmw@amazon.co.uk>
> 
> Coverity points out (CID 1507534, 1507968) that we sometimes access
> env->xen_singleshot_timer_ns under the protection of
> env->xen_timers_lock and sometimes not.
> 
> This isn't always an issue. There are two modes for the timers; if the
> kernel supports the EVTCHN_SEND capability then it handles all the timer
> hypercalls and delivery internally, and all we use the field for is to
> get/set the timer as part of the vCPU state via an ioctl(). If the
> kernel doesn't have that support, then we do all the emulation within
> qemu, and *those* are the code paths where we actually care about the
> locking.
> 
> But it doesn't hurt to be a little bit more consistent and avoid having
> to explain *why* it's OK.
> 
> Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
> ---
>   target/i386/kvm/xen-emu.c | 36 ++++++++++++++++++++++++++----------
>   1 file changed, 26 insertions(+), 10 deletions(-)
>

Reviewed-by: Paul Durrant <paul@xen.org>
Re: [PATCH] i386/xen: consistent locking around Xen singleshot timers
Posted by Paul Durrant 11 months, 2 weeks ago
On 22/05/2023 19:52, David Woodhouse wrote:
> From: David Woodhouse <dwmw@amazon.co.uk>
> 
> Coverity points out (CID 1507534) that we sometimes access
> env->xen_singleshot_timer_ns under the protection of
> env->xen_timers_lock (eg in xen_vcpu_singleshot_timer_event()) and
> sometimes not (the specific case Coverity complains about is in
> do_vcpu_soft_reset()).
> 
> This isn't strictly an issue. There are two modes for the timers; if
> the kernel supports the EVTCHN_SEND capability then it handles all the
> timer hypercalls and delivery internally, and all we need to do is an
> ioctl to get/set the next timer as part of the vCPU state. If the
> kernel doesn't have that support, then we do all the emulation within
> qemu, and *those* are the code paths where we actually care about the
> locking.
> 
> But it doesn't hurt to be a little bit more consistent and avoid having
> to explain *why* it's OK.
> 
> Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
> ---
> 
> On Tue, 2023-05-09 at 15:55 +0100, Peter Maydell wrote:
>> Hi; Coverity points out (CID 1507534) that we seem to sometimes
>> access env->xen_singleshot_timer_ns under the protection of
>> env->xen_timers_lock (eg in xen_vcpu_singleshot_timer_event())
>> and sometimes not (the specific case Coverity complains about is
>> in do_vcpu_soft_reset()). Is this a false positive, or is there
> 
> Thanks. As noted, I think it's harmless but it doesn't hurt to clean it
> up a bit. I've pushed this to my tree at
> https://git.infradead.org/users/dwmw2/qemu.git/shortlog/refs/heads/xenfv
> on top of the other fixes that didn't make 8.0.
> 
>   target/i386/kvm/xen-emu.c | 15 +++++++++------
>   1 file changed, 9 insertions(+), 6 deletions(-)
> 

Reviewed-by: Paul Durrant <paul@xen.org>