From nobody Thu Apr 25 15:56:10 2024 Delivered-To: importer@patchew.org Received-SPF: none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=oracle.com ARC-Seal: i=1; a=rsa-sha256; t=1557422829; cv=none; d=zoho.com; s=zohoarc; b=TUFZ0ZgDLmHxlZMAbbngviQ5YJ41FObQM1FUmAzeRDWRkOqAHkpfWcojNcw83TDfttZt2fc9kcNwgW3hpoZklaEZ5jtkpmM7cg9q6XxLCeZ6OH5R7wW3xE8DwKPyMDhxQOrmpgLH9d2GWffdHE4aX2MKseC0IXMZ/LYUgQXjHp8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1557422829; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=ttRkbYYvkP9yty/O/w23JuMisFzmkMzrOXIsAtgSX6w=; b=Nwg22j1fHMnqsw5bJPKyZBlxqPSXvJZaulPpinvFzy/ja2zeQzZPoTkeFzIwdbbQk9JSoGztJReLSlmM39hoAV7XCuAU3hAaTsxLJZlFffyLzNmJnYwoXXTkpK/YMWhm6pOz3rSYgrp++jM/BVHb03guBIYUDXEkXi7rFbIlEEA= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1557422829184513.3120392848123; Thu, 9 May 2019 10:27:09 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hOmnk-0000es-GI; Thu, 09 May 2019 17:25:48 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hOmnj-0000dx-16 for xen-devel@lists.xenproject.org; Thu, 09 May 2019 17:25:47 +0000 Received: from userp2120.oracle.com (unknown [156.151.31.85]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 7a3a413c-727f-11e9-864d-1302ff61f656; Thu, 09 May 2019 17:25:45 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x49HK1tB169783; Thu, 9 May 2019 17:25:42 GMT Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by userp2120.oracle.com with ESMTP id 2s94b14e31-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 09 May 2019 17:25:42 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x49HNxdO109630; Thu, 9 May 2019 17:25:42 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userp3030.oracle.com with ESMTP id 2sagyvcg4y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 09 May 2019 17:25:42 +0000 Received: from abhmp0008.oracle.com (abhmp0008.oracle.com [141.146.116.14]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x49HPfic011094; Thu, 9 May 2019 17:25:41 GMT Received: from aa1-ca-oracle-com.ca.oracle.com (/10.156.75.204) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 09 May 2019 10:25:41 -0700 X-Inumbo-ID: 7a3a413c-727f-11e9-864d-1302ff61f656 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2018-07-02; bh=5+AicXrLgHpXctVt8cnc/EdJ5KuJS7IwnasckkiqbqI=; b=ODdi+BeCjUmOTqRFTQ20OkerBYSOw3szijBMBMkJEdwfVLvYNhh0aDMaLTVBaZfQRbe5 gtx0vg55G24rR+MgEwDef/1Lm/h5drNye5oxe+xsVw19kWIVKAZzke889Lmc4PqMG3MN dYBAMRUBFtbnLCmuD7WnljTWsF35XAb77fgcAAT65B3V7OZTeEdl18MwZTKA0fTnifkM vnTqDL3lP8sx/g5jHeFAM9Cb2fEzQCaE6DuJGrzwi6IBC6Iw183ZUVaMQv8fTfOKmibe 5uFHlDHWpkOq+vhp9veLn8+64hkEyS1eTVZI5919HwDLOoRDpqV+EfobwK9LBQWrsetF QQ== From: Ankur Arora To: linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org Date: Thu, 9 May 2019 10:25:32 -0700 Message-Id: <20190509172540.12398-9-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190509172540.12398-1-ankur.a.arora@oracle.com> References: <20190509172540.12398-1-ankur.a.arora@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9252 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905090100 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9252 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905090100 Subject: [Xen-devel] [RFC PATCH 08/16] x86/xen: irq/upcall handling with multiple xenhosts X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: jgross@suse.com, sstabellini@kernel.org, konrad.wilk@oracle.com, ankur.a.arora@oracle.com, pbonzini@redhat.com, boris.ostrovsky@oracle.com, joao.m.martins@oracle.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) For configurations with multiple xenhosts, we need to handle events generated from multiple xenhosts. Having more than one upcall handler might be quite hairy, and it would be simpler if the callback from L0-Xen could be bounced via L1-Xen. This will also mean simpler pv_irq_ops code because now the IF flag maps onto the xh_default->vcpu_info->evtchn_upcall_mask. However, we still update the xh_remote->vcpu_info->evtchn_upcall_mask on a best effort basis to minimize unnecessary work in remote xenhost. TODO: - direct pv_ops.irq are disabled. Signed-off-by: Ankur Arora --- arch/x86/xen/Makefile | 2 +- arch/x86/xen/enlighten_pv.c | 4 ++- arch/x86/xen/irq.c | 69 +++++++++++++++++++++++++++++-------- arch/x86/xen/smp_pv.c | 11 ++++++ 4 files changed, 70 insertions(+), 16 deletions(-) diff --git a/arch/x86/xen/Makefile b/arch/x86/xen/Makefile index 564b4dddbc15..3c7056ad3520 100644 --- a/arch/x86/xen/Makefile +++ b/arch/x86/xen/Makefile @@ -34,7 +34,7 @@ obj-$(CONFIG_XEN_PV) +=3D enlighten_pv.o obj-$(CONFIG_XEN_PV) +=3D mmu_pv.o obj-$(CONFIG_XEN_PV) +=3D irq.o obj-$(CONFIG_XEN_PV) +=3D multicalls.o -obj-$(CONFIG_XEN_PV) +=3D xen-asm.o +obj-n +=3D xen-asm.o obj-$(CONFIG_XEN_PV) +=3D xen-asm_$(BITS).o =20 obj-$(CONFIG_XEN_PVH) +=3D enlighten_pvh.o diff --git a/arch/x86/xen/enlighten_pv.c b/arch/x86/xen/enlighten_pv.c index 5f6a1475ec0c..77b1a0d4aef2 100644 --- a/arch/x86/xen/enlighten_pv.c +++ b/arch/x86/xen/enlighten_pv.c @@ -996,8 +996,9 @@ void __init xen_setup_vcpu_info_placement(void) * xen_vcpu_setup managed to place the vcpu_info within the * percpu area for all cpus, so make use of it. */ +#if 0 + /* Disable direct access for now. */ if (xen_have_vcpu_info_placement && false) { - /* Disable direct access until we have proper pcpu data structures. */ pv_ops.irq.save_fl =3D __PV_IS_CALLEE_SAVE(xen_save_fl_direct); pv_ops.irq.restore_fl =3D __PV_IS_CALLEE_SAVE(xen_restore_fl_direct); @@ -1007,6 +1008,7 @@ void __init xen_setup_vcpu_info_placement(void) __PV_IS_CALLEE_SAVE(xen_irq_enable_direct); pv_ops.mmu.read_cr2 =3D xen_read_cr2_direct; } +#endif } =20 static const struct pv_info xen_info __initconst =3D { diff --git a/arch/x86/xen/irq.c b/arch/x86/xen/irq.c index 38ad1a1c4763..f760a6abfb1e 100644 --- a/arch/x86/xen/irq.c +++ b/arch/x86/xen/irq.c @@ -19,9 +19,9 @@ * callback mask. We do this in a very simple manner, by making a call * down into Xen. The pending flag will be checked by Xen on return. */ -void xen_force_evtchn_callback(void) +void xen_force_evtchn_callback(xenhost_t *xh) { - (void)HYPERVISOR_xen_version(0, NULL); + (void)hypervisor_xen_version(xh, 0, NULL); } =20 asmlinkage __visible unsigned long xen_save_fl(void) @@ -29,6 +29,21 @@ asmlinkage __visible unsigned long xen_save_fl(void) struct vcpu_info *vcpu; unsigned long flags; =20 + /* + * In scenarios with more than one xenhost, the primary xenhost + * is responsible for all the upcalls, with the remote xenhost + * bouncing its upcalls through it (see comment in + * cpu_initialize_context().) + * + * To minimize unnecessary upcalls, the remote xenhost still looks at + * the value of vcpu_info->evtchn_upcall_mask, so we still set and reset + * that. + * + * The fact that the upcall itself is gated by the default xenhost, + * also helps in simplifying the logic here because we don't have to + * worry about guaranteeing atomicity with updates to + * xh_remote->vcpu_info->evtchn_upcall_mask. + */ vcpu =3D xh_default->xen_vcpu[smp_processor_id()]; =20 /* flag has opposite sense of mask */ @@ -38,26 +53,34 @@ asmlinkage __visible unsigned long xen_save_fl(void) -0 -> 0x00000000 -1 -> 0xffffffff */ - return (-flags) & X86_EFLAGS_IF; + return ((-flags) & X86_EFLAGS_IF); } PV_CALLEE_SAVE_REGS_THUNK(xen_save_fl); =20 __visible void xen_restore_fl(unsigned long flags) { struct vcpu_info *vcpu; + xenhost_t **xh; =20 /* convert from IF type flag */ flags =3D !(flags & X86_EFLAGS_IF); =20 /* See xen_irq_enable() for why preemption must be disabled. */ preempt_disable(); - vcpu =3D xh_default->xen_vcpu[smp_processor_id()]; - vcpu->evtchn_upcall_mask =3D flags; + for_each_xenhost(xh) { + vcpu =3D (*xh)->xen_vcpu[smp_processor_id()]; + vcpu->evtchn_upcall_mask =3D flags; + } =20 if (flags =3D=3D 0) { barrier(); /* unmask then check (avoid races) */ - if (unlikely(vcpu->evtchn_upcall_pending)) - xen_force_evtchn_callback(); + for_each_xenhost(xh) { + /* Preemption is disabled so we should not have + * gotten moved to a different VCPU. */ + vcpu =3D (*xh)->xen_vcpu[smp_processor_id()]; + if (unlikely(vcpu->evtchn_upcall_pending)) + xen_force_evtchn_callback(*xh); + } preempt_enable(); } else preempt_enable_no_resched(); @@ -66,11 +89,19 @@ PV_CALLEE_SAVE_REGS_THUNK(xen_restore_fl); =20 asmlinkage __visible void xen_irq_disable(void) { + xenhost_t **xh; + /* There's a one instruction preempt window here. We need to make sure we're don't switch CPUs between getting the vcpu pointer and updating the mask. */ preempt_disable(); - xh_default->xen_vcpu[smp_processor_id()]->evtchn_upcall_mask =3D 1; + for_each_xenhost(xh) + /* + * Mask events on this CPU for both the xenhosts. As the + * comment above mentions, disabling preemption means we + * can safely do that. + */ + (*xh)->xen_vcpu[smp_processor_id()]->evtchn_upcall_mask =3D 1; preempt_enable_no_resched(); } PV_CALLEE_SAVE_REGS_THUNK(xen_irq_disable); @@ -78,6 +109,7 @@ PV_CALLEE_SAVE_REGS_THUNK(xen_irq_disable); asmlinkage __visible void xen_irq_enable(void) { struct vcpu_info *vcpu; + xenhost_t **xh; =20 /* * We may be preempted as soon as vcpu->evtchn_upcall_mask is @@ -86,16 +118,25 @@ asmlinkage __visible void xen_irq_enable(void) */ preempt_disable(); =20 - vcpu =3D xh_default->xen_vcpu[smp_processor_id()]; - vcpu->evtchn_upcall_mask =3D 0; + /* Given that the interrupts are generated from the default xenhost, + * we should do this in reverse order. + */ + for_each_xenhost(xh) { + vcpu =3D (*xh)->xen_vcpu[smp_processor_id()]; + vcpu->evtchn_upcall_mask =3D 0; =20 - /* Doesn't matter if we get preempted here, because any - pending event will get dealt with anyway. */ + /* We could get preempted by an incoming interrupt here with a + * half enabled irq (for the first xenhost.) + */ + } =20 barrier(); /* unmask then check (avoid races) */ - if (unlikely(vcpu->evtchn_upcall_pending)) - xen_force_evtchn_callback(); =20 + for_each_xenhost(xh) { + vcpu =3D (*xh)->xen_vcpu[smp_processor_id()]; + if (unlikely(vcpu->evtchn_upcall_pending)) + xen_force_evtchn_callback(*xh); + } preempt_enable(); } PV_CALLEE_SAVE_REGS_THUNK(xen_irq_enable); diff --git a/arch/x86/xen/smp_pv.c b/arch/x86/xen/smp_pv.c index 6d9c3e6611ef..f4ea9eac8b6a 100644 --- a/arch/x86/xen/smp_pv.c +++ b/arch/x86/xen/smp_pv.c @@ -343,6 +343,17 @@ cpu_initialize_context(unsigned int cpu, struct task_s= truct *idle) #else ctxt->gs_base_kernel =3D per_cpu_offset(cpu); #endif + /* + * We setup an upcall handler only for the default xenhost. The remote + * xenhost will generate evtchn events, but an additional callback would = be + * quite hairy, since we would have VCPU state initialised in multiple + * hypervisors and issues like re-entrancy of upcalls. + * + * It would be simpler if the callback from L0-Xen could be bounced + * bounced via L1-Xen. This also simplifies the pv_irq_ops code + * because now the CPU's IF processing only needs to happen on + * xh_default->vcpu_info. + */ ctxt->event_callback_eip =3D (unsigned long)xen_hypervisor_callback; ctxt->failsafe_callback_eip =3D --=20 2.20.1 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel