From nobody Mon Mar 10 02:56:22 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1741166001; cv=none; d=zohomail.com; s=zohoarc; b=hq6sZEsDcBq3qEkWfufe3Uv+w2MAO6GiHwmav9MpPLcxn2jr3khBzsmAWprhnPRpGTxbUTEb/tBAuOrGOfpRvhXzrB0/BfzMF5lS1EXX4K82HFRU0W88JBeW3PskOstl43rzkCOYjRwwuY8gQZTkj6DhU1gULLRBfTjcyRisDvY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741166001; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=7vRDesRQb/Nxhcf66JbLCYMstts9xs0Oaigt+qqhPzM=; b=FZaOw8KZaJ9QR+dlg1l5cZr1ymSnaOmaO/BKYt0Fp41Gl9wz0tcmThuLLubAOZPZB/U0TKMrUQ8dq1DHGjiAjiA9QeiA+8hRynumTRUOUJAugkTOGUW8IH1fxt3AaMxW7VcX8ZAVKHXMwfMVkxqDn4S20txys1x+QGhk+W2J0G0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1741166001558103.04660473337049; Wed, 5 Mar 2025 01:13:21 -0800 (PST) Received: from list by lists.xenproject.org with outflank-mailman.901883.1309897 (Exim 4.92) (envelope-from ) id 1tpknW-0001Hu-4j; Wed, 05 Mar 2025 09:12:14 +0000 Received: by outflank-mailman (output) from mailman id 901883.1309897; Wed, 05 Mar 2025 09:12:14 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tpknV-0001Gw-S2; Wed, 05 Mar 2025 09:12:13 +0000 Received: by outflank-mailman (input) for mailman id 901883; Wed, 05 Mar 2025 09:12:12 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tpknT-0006ev-Sd for xen-devel@lists.xenproject.org; Wed, 05 Mar 2025 09:12:11 +0000 Received: from mail-lf1-x131.google.com (mail-lf1-x131.google.com [2a00:1450:4864:20::131]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id ebabc915-f9a1-11ef-9ab4-95dc52dad729; Wed, 05 Mar 2025 10:12:11 +0100 (CET) Received: by mail-lf1-x131.google.com with SMTP id 2adb3069b0e04-549490e290dso5158117e87.2 for ; Wed, 05 Mar 2025 01:12:11 -0800 (PST) Received: from EPUAKYIW02F7.. ([95.67.15.120]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5495de028fasm1078264e87.139.2025.03.05.01.12.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Mar 2025 01:12:09 -0800 (PST) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: ebabc915-f9a1-11ef-9ab4-95dc52dad729 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741165930; x=1741770730; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7vRDesRQb/Nxhcf66JbLCYMstts9xs0Oaigt+qqhPzM=; b=ESjib+ijtuRtIkop52jBXTWP/vZdjuq1LbSSb1bzrDHb3gXFQvmlqL9guzkqGVBYzY kXmsuGEREScjkGdZzXQtstSaLfSTpsPPIOsaShYoB5cDsIJIejMfwPrSgTvjjenMQgxn okmqnTMvIJiDB30DhXA2ScJ1PFz/hWHT/NM8hsEvVIOnmL9yqzkfaiCv27IADyWiAxGX +4KVOspvBB5lgawJUJ86izoHwHQ9v6Rj2tOwI77qYT5H4sXwFUOXxfPRWRonq9ABPg38 pnDIJ8IdlOIMv4gntyNvcopOjaPJr2l3twfNzUI8Q35vlykHTvuRZRh0J86LON0mXj8R n7lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741165930; x=1741770730; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7vRDesRQb/Nxhcf66JbLCYMstts9xs0Oaigt+qqhPzM=; b=G1EdZXq4Gtqq0NaL8HczTcVS0TUT5RhtSeHQz6CVvXQtKpOyQAjDtK0DJtxGvxk5LT etGTEqjEBKwJ4q5q/aZj+qhGb6vdH75W8mBZ+Zyyrk1/lG3BvKK/YHHppGpQHks9RAEx Iy4thdXDOus/BxGEByueTs8y+uucdqMQdsHtD21vIx4RCbj864vsglzWoMI5MmjjkvWO m0rE/vLgpqcb+HlBFoKZUVn57A9Iz9tzjjpZd25JU47DZ7Caz90guGYtW4yE3M/yxmCA 0lq4gTcvliiQJZeeam3gIbOAXBBdjmbl/CMU7VBKi76Y21uZTa+ZrUO4Eh7+j4S5NztN 7MKg== X-Gm-Message-State: AOJu0Yzk+dHKtN38TaKQXyBCVoZgVGWaAJ9XKkvLJTSLEtfjhleQwocW PU2oFZBH7+MkkKY79r3SY/ZWqWr7UNZr5dbUAoCghrvnj6xQ5gRgML8Bi/oE X-Gm-Gg: ASbGncthnCbAeb7lEf/HqdFWt5PXIlfMzUobZJAjSbz1RkVV3Gb2skFwL646vvWNzDl S+L0nkGROS976xt8RpV30LBgpU7M3ZBRrwBsZsNVu6VPnYYWSsOt6UHIOzANGFOaYkaFHfA0DtH U59K6mR+hUuOM2g7+LcOa45AgwBqF0kaMlKFU2XC1IsQ/VykoFUzLp/PJZzYSM3m6A4Kpf1dAjL 1CauwSTKd5UO6xmLBhHTPE6jzZqDoAnvOLRYpdiyqJwqLAjr9JNapghFLF1tkU89n8a9oZ0BWVS bL1BYdNAn0oJu2Uhocp0Yn1gc8pC9n7ELD2yPImRfLV05PCTVA== X-Google-Smtp-Source: AGHT+IGgw7tZ/KupAA7aqLgKd7RgPJr1KdPD7ZGpud0YGWY6QTwxlrBZ7x7v9bHnysTCB2vsApcPRQ== X-Received: by 2002:a05:6512:3e1a:b0:545:1082:91a1 with SMTP id 2adb3069b0e04-5497d32f7bamr789411e87.7.1741165929631; Wed, 05 Mar 2025 01:12:09 -0800 (PST) From: Mykola Kvach To: xen-devel@lists.xenproject.org Cc: Mirela Simonovic , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , Volodymyr Babchuk , Saeed Nowshadi , Mykyta Poturai , Mykola Kvach Subject: [PATCH 12/16] xen/arm: Trigger Xen suspend when hardware domain completes suspend Date: Wed, 5 Mar 2025 11:11:20 +0200 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1741166003594019100 Content-Type: text/plain; charset="utf-8" From: Mirela Simonovic When hardware domain finalizes its suspend procedure the suspend of Xen is triggered by calling system_suspend(). Hardware domain finalizes the suspend from its boot core (VCPU#0), which could be mapped to any physical CPU, i.e. the system_suspend() function could be executed by any physical CPU. Since Xen suspend procedure has to be run by the boot CPU (non-boot CPUs will be disabled at some point in suspend procedure), system_suspend() execution has to continue on CPU#0. When the system_suspend() returns 0, it means that the system was suspended and it is coming out of the resume procedure. Regardless of the system_suspend() return value, after this function returns Xen is fully functional, and its state, including all devices and data structures, matches the state prior to calling system_suspend(). The status is returned by system_suspend() for debugging/logging purposes and function prototype compatibility. This patch also introduces some state changes in peripherals and CPUs during suspend/resume. Specifically, it: - disable/enable non-boot physical CPUs, freeze/thaw domains; - suspend/resume the timer, GIC, console, IOMMU, and hardware domain. Signed-off-by: Mirela Simonovic Signed-off-by: Saeed Nowshadi Signed-off-by: Mykyta Poturai Signed-off-by: Mykola Kvach --- Changes introduced in V3: Merged changes from other commits into this one (stashed changes): - disabled/enabled non-boot physical CPUs and froze/thawed domains; - suspended/resumed the timer, GIC, console, IOMMU, and hardware domain. --- xen/arch/arm/suspend.c | 233 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 233 insertions(+) diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c index 27fab8c999..fa81be5a4f 100644 --- a/xen/arch/arm/suspend.c +++ b/xen/arch/arm/suspend.c @@ -1,6 +1,9 @@ /* SPDX-License-Identifier: GPL-2.0-only */ =20 #include +#include +#include +#include #include #include #include @@ -8,6 +11,210 @@ #include #include =20 +/* Reset values of VCPU architecture specific registers */ +static void vcpu_arch_reset(struct vcpu *v) +{ + v->arch.ttbr0 =3D 0; + v->arch.ttbr1 =3D 0; + v->arch.ttbcr =3D 0; + + v->arch.csselr =3D 0; + v->arch.cpacr =3D 0; + v->arch.contextidr =3D 0; + v->arch.tpidr_el0 =3D 0; + v->arch.tpidrro_el0 =3D 0; + v->arch.tpidr_el1 =3D 0; + v->arch.vbar =3D 0; + v->arch.dacr =3D 0; + v->arch.par =3D 0; +#if defined(CONFIG_ARM_32) + v->arch.mair0 =3D 0; + v->arch.mair1 =3D 0; + v->arch.amair0 =3D 0; + v->arch.amair1 =3D 0; +#else + v->arch.mair =3D 0; + v->arch.amair =3D 0; +#endif + /* Fault Status */ +#if defined(CONFIG_ARM_32) + v->arch.dfar =3D 0; + v->arch.ifar =3D 0; + v->arch.dfsr =3D 0; +#elif defined(CONFIG_ARM_64) + v->arch.far =3D 0; + v->arch.esr =3D 0; +#endif + + v->arch.ifsr =3D 0; + v->arch.afsr0 =3D 0; + v->arch.afsr1 =3D 0; + +#ifdef CONFIG_ARM_32 + v->arch.joscr =3D 0; + v->arch.jmcr =3D 0; +#endif + + v->arch.teecr =3D 0; + v->arch.teehbr =3D 0; +} + +/* + * This function sets the context of current VCPU to the state which is ex= pected + * by the guest on resume. The expected VCPU state is: + * 1) pc to contain resume entry point (1st argument of PSCI SYSTEM_SUSPEN= D) + * 2) r0/x0 to contain context ID (2nd argument of PSCI SYSTEM_SUSPEND) + * 3) All other general purpose and system registers should have reset val= ues + */ +static void vcpu_resume(struct vcpu *v) +{ + + struct vcpu_guest_context ctxt; + + /* Make sure that VCPU guest regs are zeroed */ + memset(&ctxt, 0, sizeof(ctxt)); + + /* Set non-zero values to the registers prior to copying */ + ctxt.user_regs.pc64 =3D (u64)v->arch.suspend_ep; + + /* TODO: test changes on 32-bit domain */ + if ( is_32bit_domain(v->domain) ) + { + ctxt.user_regs.r0_usr =3D v->arch.suspend_cid; + ctxt.user_regs.cpsr =3D PSR_GUEST32_INIT; + + /* Thumb set is allowed only for 32-bit domain */ + if ( v->arch.suspend_ep & 1 ) + { + ctxt.user_regs.cpsr |=3D PSR_THUMB; + ctxt.user_regs.pc64 &=3D ~(u64)1; + } + } +#ifdef CONFIG_ARM_64 + else + { + ctxt.user_regs.x0 =3D v->arch.suspend_cid; + ctxt.user_regs.cpsr =3D PSR_GUEST64_INIT; + } +#endif + ctxt.sctlr =3D SCTLR_GUEST_INIT; + ctxt.flags =3D VGCF_online; + + /* Reset architecture specific registers */ + vcpu_arch_reset(v); + + /* Initialize VCPU registers */ + domain_lock(v->domain); + arch_set_info_guest(v, &ctxt); + domain_unlock(v->domain); + watchdog_domain_resume(v->domain); +} + +/* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */ +static long system_suspend(void *data) +{ + int status; + unsigned long flags; + + BUG_ON(system_state !=3D SYS_STATE_active); + + system_state =3D SYS_STATE_suspend; + freeze_domains(); + + /* + * Non-boot CPUs have to be disabled on suspend and enabled on resume + * (hotplug-based mechanism). Disabling non-boot CPUs will lead to PSCI + * CPU_OFF to be called by each non-boot CPU. Depending on the underly= ing + * platform capabilities, this may lead to the physical powering down = of + * CPUs. Tested on Xilinx Zynq Ultrascale+ MPSoC (including power down= of + * each non-boot CPU). + */ + status =3D disable_nonboot_cpus(); + if ( status ) + { + system_state =3D SYS_STATE_resume; + goto resume_nonboot_cpus; + } + + time_suspend(); + + local_irq_save(flags); + status =3D gic_suspend(); + if ( status ) + { + system_state =3D SYS_STATE_resume; + goto resume_irqs; + } + + printk("Xen suspending...\n"); + console_start_sync(); + + status =3D console_suspend(); + if ( status ) + { + dprintk(XENLOG_ERR, "Failed to suspend the console, err=3D%d\n", s= tatus); + system_state =3D SYS_STATE_resume; + goto resume_console; + } + + status =3D iommu_suspend(); + if ( status ) + { + system_state =3D SYS_STATE_resume; + goto resume_console; + } + + /* + * Enable identity mapping before entering suspend to simplify + * the resume path + */ + update_boot_mapping(true); + + system_state =3D SYS_STATE_resume; + update_boot_mapping(false); + + iommu_resume(); + +resume_console: + console_resume(); + + gic_resume(); + +resume_irqs: + local_irq_restore(flags); + + time_resume(); + +resume_nonboot_cpus: + /* + * The rcu_barrier() has to be added to ensure that the per cpu area is + * freed before a non-boot CPU tries to initialize it (_free_percpu_ar= ea() + * has to be called before the init_percpu_area()). This scenario occu= rs + * when non-boot CPUs are hot-unplugged on suspend and hotplugged on r= esume. + */ + rcu_barrier(); + enable_nonboot_cpus(); + thaw_domains(); + system_state =3D SYS_STATE_active; + + /* + * The hardware domain owns most of the devices and may be part of the + * suspend/resume path. Since the hardware domain suspend is tied to + * the host suspend, it makes sense to resume it at the same time, + * i.e. after host resumes. + */ + vcpu_resume(hardware_domain->vcpu[0]); + /* + * The resume of hardware domain should always follow Xen's resume. + * This is done by unblocking the first vCPU of Dom0. + */ + vcpu_unblock(hardware_domain->vcpu[0]); + + printk("Resume (status %d)\n", status); + + return status; +} + static void vcpu_suspend_prepare(register_t epoint, register_t cid) { struct vcpu *v =3D current; @@ -21,6 +228,7 @@ int32_t domain_suspend(register_t epoint, register_t cid) struct vcpu *v; struct domain *d =3D current->domain; bool is_thumb =3D epoint & 1; + int status; =20 dprintk(XENLOG_DEBUG, "Dom%d suspend: epoint=3D0x%"PRIregister", cid=3D0x%"PRIregist= er"\n", @@ -54,6 +262,31 @@ int32_t domain_suspend(register_t epoint, register_t ci= d) */ vcpu_block_unless_event_pending(current); =20 + /* If this was dom0 the whole system should suspend: trigger Xen suspe= nd */ + if ( is_hardware_domain(d) ) + { + /* + * system_suspend should be called when Dom0 finalizes the suspend + * procedure from its boot core (VCPU#0). However, Dom0's VCPU#0 c= ould + * be mapped to any PCPU (this function could be executed by any P= CPU). + * The suspend procedure has to be finalized by the PCPU#0 (non-bo= ot + * PCPUs will be disabled during the suspend). + */ + status =3D continue_hypercall_on_cpu(0, system_suspend, NULL); + /* + * If an error happened, there is nothing that needs to be done he= re + * because the system_suspend always returns in fully functional s= tate + * no matter what the outcome of suspend procedure is. If the syst= em + * suspended successfully the function will return 0 after the res= ume. + * Otherwise, if an error is returned it means Xen did not suspend= ed, + * but it is still in the same state as if the system_suspend was = never + * called. We dump a debug message in case of an error for debuggi= ng/ + * logging purpose. + */ + if ( status ) + dprintk(XENLOG_ERR, "Failed to suspend, errno=3D%d\n", status); + } + return PSCI_SUCCESS; } =20 --=20 2.43.0