From nobody Thu Oct 30 22:54:56 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1756764650; cv=none; d=zohomail.com; s=zohoarc; b=IUvfD0LWhlKzEk293E7lf6gK5QWnoMNw2+ldSj9IizdrRSVNF0/SnfOfMUd+epW/u1ChnHiqU6wLkCp2zqbtOlsdTA/q7tY25LjmEG9whrah6nEjHrfrd6W9fAmV7uPf8ZX+4ph27OrHG3HajlqS/5HXUf5oTOqbxiqUdGc49ys= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1756764650; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=uqPcozH5cY7WTdc/VNhtz0ZQbmCe6X/v3JhqZHPKizE=; b=EAQjZqQ4pkZWwcGlbrBR1O+O2JS5EvDcIZlRFJvT83tf/pPoKZx3VjEvIRbVBQzzs87GHFMejyxubV62yQAU8GePkrcTxyaGst4GitRXpke7TChOzexfuMwZfkW4zn+RgyRcw7r7UnJ3RG+Cbhv5cq9IYhx+3Kc+aQUJxxDeK1c= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1756764650896788.1941971611159; Mon, 1 Sep 2025 15:10:50 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1105398.1456325 (Exim 4.92) (envelope-from ) id 1utCjK-00059O-4Z; Mon, 01 Sep 2025 22:10:26 +0000 Received: by outflank-mailman (output) from mailman id 1105398.1456325; Mon, 01 Sep 2025 22:10:26 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjK-00059E-0I; Mon, 01 Sep 2025 22:10:26 +0000 Received: by outflank-mailman (input) for mailman id 1105398; Mon, 01 Sep 2025 22:10:25 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjJ-00055o-3U for xen-devel@lists.xenproject.org; Mon, 01 Sep 2025 22:10:25 +0000 Received: from mail-lf1-x136.google.com (mail-lf1-x136.google.com [2a00:1450:4864:20::136]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 7483f7db-8780-11f0-8dd7-1b34d833f44b; Tue, 02 Sep 2025 00:10:23 +0200 (CEST) Received: by mail-lf1-x136.google.com with SMTP id 2adb3069b0e04-55f69cf4b77so3061750e87.2 for ; Mon, 01 Sep 2025 15:10:23 -0700 (PDT) Received: from yp-VivoBook-ASUSLaptop-M1503QA-M1503QA.. ([95.67.15.120]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5608279307asm123038e87.75.2025.09.01.15.10.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 15:10:21 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 7483f7db-8780-11f0-8dd7-1b34d833f44b DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756764622; x=1757369422; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uqPcozH5cY7WTdc/VNhtz0ZQbmCe6X/v3JhqZHPKizE=; b=fvkpboOPpIPZqSscUPtg/Z76SMQfSVSgyWtvLFMB+e77wAGZNfS1+RTmUkI8Yh0/K0 j/8RJZPDSfXagf4U8b6zMiT8N98eyUnjQZoLSABwSd5BYkUx0lyYWPhVE6HObDMMiU/U DquxnS2pm+5MmI4DXXcx2NE068a1S7ccWaPmwwCuqMnXfGzL7g4RgAzG5r/JNEN+3XHb 3CMSWd6VVlHcEjOjkGbjnPYRFjgwUsNNVBkCLNrYodMqQaaTqjRPJiX+VdFHZcBRzHgn ne5TgdqAhnWGSRtuoQz46IP5nzYsZHHxlexX0kEgrCPG+pZe0qaI+jcV2sx80egOT3oG vSEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756764622; x=1757369422; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uqPcozH5cY7WTdc/VNhtz0ZQbmCe6X/v3JhqZHPKizE=; b=gKqHKcJgQ9rkGKHQxsJqkwwq044hAh4w6MYGhNELLa/Fki/JqckR/TE1rR8M52r9fF oGjtnuzhWwVmBkmELiJzr5gztMB/rthic3K271mRGv8uAswNlkWtwGmfUd5IDUxBHl6H TfR7BYK6r2wgkL7Xtt9XjC/p2m1NclM1y9tnwMwVVGMjadjedUAIOFY49YMyNtw6m74g b6iUYrlBWRCiMTcLjXskhE0Boz8ZoW5JJKyLtLJuwWk3twM/nz4sABs+hGJQEBsH2DKn 6qUuytNe6LgUyGTVV6MxU2MgtN43rnaZuof64mTDCs2Ct3jDR57m/fBwKuBab2wGxWvZ WdRg== X-Gm-Message-State: AOJu0YzMgqrsNEIFjvPTxqrordlfVR9y6/K/eFrC1x56DxibJT2zXhcR D9ms/bMNVfioVUF9/Yo72TVQHCEqstTJkfwpmK0cPcl17cW1sKIJN2oeqtd9EYEm X-Gm-Gg: ASbGncuATfkl59L6zA4qqJSStTS2rg+1iy44bkMROeGKlPHjAoskQus7uZB7vzV3+3h YaJ83Sk5duw7Uf8gikViYZ+8/uQBkCR80Y29GbBDHh3h4xVc48R6XSKofw9kcNl/M4KEZ4DAxbG VdSiuc0wRIUviIOO1oXXK/wP+hxOMXFP8Atk4NOPiS4ujacucEOqNn9WA19E4nm8tWq5AJkpHK5 WqyATTWSipjnlyWar3FRK76YZOOE4ivDYf3fJ1ltJtf6C9TsjKxpS/V0ImnMkNR069JFcysPxmV v7ykoF9IdU7bPhQM2KWK2HrAjmM3NAt1ExBcorlELQA4gW8w6Leea+y61ioU5yRw+D+86DRQR2z MnN6gtR7Los7wtQnv+xLJrvr4ccwTgsBKYPBR/77/9RE2EdFVHAxWQLZ1rG77y6iph2duIqeP X-Google-Smtp-Source: AGHT+IE0nct2I3ZXnzBAVzjS0A2t7RZ46uqQlrsxxAbmvZQR2mhhoN8TLctbHGFrjJ6lWGC/3qLOrw== X-Received: by 2002:a05:6512:1409:b0:55f:6831:6ee7 with SMTP id 2adb3069b0e04-55f708b60e7mr2174676e87.21.1756764622048; Mon, 01 Sep 2025 15:10:22 -0700 (PDT) From: Mykola Kvach To: xen-devel@lists.xenproject.org Cc: Mirela Simonovic , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , Volodymyr Babchuk , Saeed Nowshadi , Mykola Kvach Subject: [PATCH v6 01/13] xen/arm: Add suspend and resume timer helpers Date: Tue, 2 Sep 2025 01:10:05 +0300 Message-ID: <781c8e1b3a4d9b269bfde125072a84baae3f9bb3.1756763487.git.mykola_kvach@epam.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1756764653091124100 Content-Type: text/plain; charset="utf-8" From: Mirela Simonovic Timer interrupts must be disabled while the system is suspended to prevent spurious wake-ups. Suspending the timers involves disabling both the EL1 physical timer and the EL2 hypervisor timer. Resuming consists of raising the TIMER_SOFTIRQ, which prompts the generic timer code to reprogram the EL2 timer as needed. Re-enabling of the EL1 timer is left to the entity that uses it. Introduce a new helper, disable_physical_timers, to encapsulate disabling of the physical timers. Signed-off-by: Mirela Simonovic Signed-off-by: Saeed Nowshadi Signed-off-by: Mykola Kvach Reviewed-by: Volodymyr Babchuk --- Changes in V4: - Rephrased comment and commit message for better clarity - Created separate function for disabling physical timers Changes in V3: - time_suspend and time_resume are now conditionally compiled under CONFIG_SYSTEM_SUSPEND --- xen/arch/arm/include/asm/time.h | 5 +++++ xen/arch/arm/time.c | 38 +++++++++++++++++++++++++++------ 2 files changed, 37 insertions(+), 6 deletions(-) diff --git a/xen/arch/arm/include/asm/time.h b/xen/arch/arm/include/asm/tim= e.h index 49ad8c1a6d..f4fd0c6af5 100644 --- a/xen/arch/arm/include/asm/time.h +++ b/xen/arch/arm/include/asm/time.h @@ -108,6 +108,11 @@ void preinit_xen_time(void); =20 void force_update_vcpu_system_time(struct vcpu *v); =20 +#ifdef CONFIG_SYSTEM_SUSPEND +void time_suspend(void); +void time_resume(void); +#endif /* CONFIG_SYSTEM_SUSPEND */ + #endif /* __ARM_TIME_H__ */ /* * Local variables: diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c index e74d30d258..ad984fdfdd 100644 --- a/xen/arch/arm/time.c +++ b/xen/arch/arm/time.c @@ -303,6 +303,14 @@ static void check_timer_irq_cfg(unsigned int irq, cons= t char *which) "WARNING: %s-timer IRQ%u is not level triggered.\n", which, irq= ); } =20 +/* Disable physical timers for EL1 and EL2 on the current CPU */ +static inline void disable_physical_timers(void) +{ + WRITE_SYSREG(0, CNTP_CTL_EL0); /* Physical timer disabled */ + WRITE_SYSREG(0, CNTHP_CTL_EL2); /* Hypervisor's timer disabled */ + isb(); +} + /* Set up the timer interrupt on this CPU */ void init_timer_interrupt(void) { @@ -310,9 +318,7 @@ void init_timer_interrupt(void) WRITE_SYSREG64(0, CNTVOFF_EL2); /* No VM-specific offset */ /* Do not let the VMs program the physical timer, only read the physic= al counter */ WRITE_SYSREG(CNTHCTL_EL2_EL1PCTEN, CNTHCTL_EL2); - WRITE_SYSREG(0, CNTP_CTL_EL0); /* Physical timer disabled */ - WRITE_SYSREG(0, CNTHP_CTL_EL2); /* Hypervisor's timer disabled */ - isb(); + disable_physical_timers(); =20 request_irq(timer_irq[TIMER_HYP_PPI], 0, htimer_interrupt, "hyptimer", NULL); @@ -330,9 +336,7 @@ void init_timer_interrupt(void) */ static void deinit_timer_interrupt(void) { - WRITE_SYSREG(0, CNTP_CTL_EL0); /* Disable physical timer */ - WRITE_SYSREG(0, CNTHP_CTL_EL2); /* Disable hypervisor's timer */ - isb(); + disable_physical_timers(); =20 release_irq(timer_irq[TIMER_HYP_PPI], NULL); release_irq(timer_irq[TIMER_VIRT_PPI], NULL); @@ -372,6 +376,28 @@ void domain_set_time_offset(struct domain *d, int64_t = time_offset_seconds) /* XXX update guest visible wallclock time */ } =20 +#ifdef CONFIG_SYSTEM_SUSPEND + +void time_suspend(void) +{ + disable_physical_timers(); +} + +void time_resume(void) +{ + /* + * Raising the timer softirq triggers generic code to call reprogram_t= imer + * with the correct timeout (not known here). + * + * No further action is needed to restore timekeeping after power down, + * since the system counter is unaffected. See ARM DDI 0487 L.a, D12.1= .2 + * "The system counter must be implemented in an always-on power domai= n." + */ + raise_softirq(TIMER_SOFTIRQ); +} + +#endif /* CONFIG_SYSTEM_SUSPEND */ + static int cpu_time_callback(struct notifier_block *nfb, unsigned long action, void *hcpu) --=20 2.48.1 From nobody Thu Oct 30 22:54:56 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1756764654; cv=none; d=zohomail.com; s=zohoarc; b=cLKSILoR5DhALjeWHlz2ekBkR9OcVfcnIRZbov2wYwju3ZE4ATiwcxBxT8xUn3GPJUszQRF2fa0aiXZ1IZSvVA9eo7TvgIfTPPT4qLUuz0UkaOVgC9vTbzcbb0itR+x5Ce5KhP2OhCHYgLUwWOK0UGzy7Dqg7Rb6rNByem+bPSY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1756764654; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=knhQ+S9mPkiwhWSXwrIQ0d+Hly1TQbR8zGjL18AowhY=; b=Psq6Px4kMNxYM1TwTJdtHmqFGiyQ8AxOxNu0+DiLGpP4y6f8DNOSh1UMjlUi1tUgKiiGje5Dcqn7XB5UcbLUq7FxAOzKDUecfgDwDfEqf4rNazLcGwVj4RAX0STq3s1Omy3mvMGS/6JHG0fb05ngi+EjU+2+Jnf8NqAqX1Ldf8s= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1756764654667361.3200651410897; Mon, 1 Sep 2025 15:10:54 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1105405.1456394 (Exim 4.92) (envelope-from ) id 1utCjT-0006wl-5L; Mon, 01 Sep 2025 22:10:35 +0000 Received: by outflank-mailman (output) from mailman id 1105405.1456394; Mon, 01 Sep 2025 22:10:34 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjS-0006w4-QJ; Mon, 01 Sep 2025 22:10:34 +0000 Received: by outflank-mailman (input) for mailman id 1105405; Mon, 01 Sep 2025 22:10:33 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjR-0005XP-G1 for xen-devel@lists.xenproject.org; Mon, 01 Sep 2025 22:10:33 +0000 Received: from mail-lf1-x12b.google.com (mail-lf1-x12b.google.com [2a00:1450:4864:20::12b]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 75835772-8780-11f0-8adc-4578a1afcccb; Tue, 02 Sep 2025 00:10:24 +0200 (CEST) Received: by mail-lf1-x12b.google.com with SMTP id 2adb3069b0e04-55f78f32580so1211819e87.3 for ; Mon, 01 Sep 2025 15:10:24 -0700 (PDT) Received: from yp-VivoBook-ASUSLaptop-M1503QA-M1503QA.. ([95.67.15.120]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5608279307asm123038e87.75.2025.09.01.15.10.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 15:10:23 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 75835772-8780-11f0-8adc-4578a1afcccb DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756764624; x=1757369424; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=knhQ+S9mPkiwhWSXwrIQ0d+Hly1TQbR8zGjL18AowhY=; b=H+krl55qPWvuR2EzNnszNP6eR9gGVNB5M8OlNOFjTSHP3a96Nn8NB7izJdS3zhwguU 4tR9PmkYV2i7kuFhX4jwTnfGPSYnBPh7mc2iCz57lpMFAgydbaN9/kYEhO6G0CwhRhED jbRFDJh6AXCPvCQVfs7/1AvzKPljBoTUzop1Bo+qWtIJOiSvBxRCs8HzDcVnLfPRE0OP V8etXfmU6kFJElUsF6ez5/R5YeNFaNiUcjuwX8siEi84uWJqRgVnM7gGMWvozabHvadr mNSZ1ImqsgZb8Jt17Slf/eVeTb8FquH0f0NGs7x868S6gHIwYGRICIvlTn+ukJR1CDzR 606A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756764624; x=1757369424; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=knhQ+S9mPkiwhWSXwrIQ0d+Hly1TQbR8zGjL18AowhY=; b=RpLfgxOzHvSOLS+r7wOaby5+KFoB0iCpY2foJVJZpdXSjIM7XqWxkknpYIx7rmQMPW JHlefRpdU9/R5gWcBVsRLM7EE3S5J1wT2ek32mTHuBUt41fI/zEbOrZS1n8sSe2SSOz5 QJ03Q7ZC+qgM/JCC1/cEn4XugFQjod2Fy3KZB15d33MBEqNdNBGG1l0x7SRZxaok8sO4 0nY7v8UPMmWBSpBr75Pl3zOwJKwc8KH9CjbT9sl2zIIsZ9J1sGgkmw0cf92tZV9lwqYE iJHSgHbQDv5oqYVA1C5KbZ6+FiWajeACM/WZtuC7oZVuJuJwERCqOZ+c3ANfiHVvALBz 2a2w== X-Gm-Message-State: AOJu0Yxag8P1u1W80z7PL+xqf9UdlFzHhHUPApg8bf8bY4sJwSXZIosv GeBIxuZRT5uRYsMsNfVrMmgRf9+eO+JeWNOIoNsFITrO185IWRW5Jw0WAxhO2HYj X-Gm-Gg: ASbGncun9mSjhdDZMYGxA/HoIvlChtvayWwpPo5xqyi7EtSwUG2s8brq+a2AdiwU4ea 1SevuhNYnOMbcStP3mIKz6YuTh5QtSfH3bQL9iJ1NAmfrtaqPd9238o7A7hzXWqWuMIXDSKyvBK povstVeDgAS/N2Z6I5haoOMueAHbuQD2CKRKBC5WajL00ezofRU2gzUSIMcsDuKZmdSV8zHq+Ap Ne7usEPA/WBv9NULclJWsyjhkDucTm9cW0eUnvJp/eiEKy8qKmY2p5m3M9B1FayQk9BRtbYbwSd UueG/8nqx3eRf8X9zXbGANt2bFzYLlLdm/2yro1AX6iV/enyzYJjQjl5rAioXVjfYLjE1k+Cyht yB9sl6rojQ5Ty90PduZw4ClTkLRIAS8o8IQNOjzC6Owgzgcc8ShBB5uH4+AnBqv0n7iORDTfK8T L5gtWtnB4= X-Google-Smtp-Source: AGHT+IHx4s/XpnhU0YN9P6FuM23tHmNqjatW0aU1ecFV2aj3TF6onudCp4HF51qNc/4vWPZYRopttQ== X-Received: by 2002:a05:6512:4608:b0:55f:7050:9550 with SMTP id 2adb3069b0e04-55f70924a63mr2181909e87.38.1756764623544; Mon, 01 Sep 2025 15:10:23 -0700 (PDT) From: Mykola Kvach To: xen-devel@lists.xenproject.org Cc: Mirela Simonovic , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , Volodymyr Babchuk , Saeed Nowshadi , Mykyta Poturai , Mykola Kvach Subject: [PATCH v6 02/13] xen/arm: gic-v2: Implement GIC suspend/resume functions Date: Tue, 2 Sep 2025 01:10:06 +0300 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1756764657191124100 Content-Type: text/plain; charset="utf-8" From: Mirela Simonovic System suspend may lead to a state where GIC would be powered down. Therefore, Xen should save/restore the context of GIC on suspend/resume. Note that the context consists of states of registers which are controlled by the hypervisor. Other GIC registers which are accessible by guests are saved/restored on context switch. Signed-off-by: Mirela Simonovic Signed-off-by: Saeed Nowshadi Signed-off-by: Mykyta Poturai Signed-off-by: Mykola Kvach Reviewed-by: Volodymyr Babchuk --- Changes in v6: - drop extra func/line printing from dprintk - drop checking context allocation from resume handler - merge some loops where it is possible Changes in v4: - Add error logging for allocation failures Changes in v3: - Drop asserts and return error codes instead. - Wrap code with CONFIG_SYSTEM_SUSPEND. Changes in v2: - Minor fixes after review. --- xen/arch/arm/gic-v2.c | 143 +++++++++++++++++++++++++++++++++ xen/arch/arm/gic.c | 29 +++++++ xen/arch/arm/include/asm/gic.h | 12 +++ 3 files changed, 184 insertions(+) diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c index b23e72a3d0..6373599e69 100644 --- a/xen/arch/arm/gic-v2.c +++ b/xen/arch/arm/gic-v2.c @@ -1098,6 +1098,140 @@ static int gicv2_iomem_deny_access(struct domain *d) return iomem_deny_access(d, mfn, mfn + nr); } =20 +#ifdef CONFIG_SYSTEM_SUSPEND + +/* GICv2 registers to be saved/restored on system suspend/resume */ +struct gicv2_context { + /* GICC context */ + uint32_t gicc_ctlr; + uint32_t gicc_pmr; + uint32_t gicc_bpr; + /* GICD context */ + uint32_t gicd_ctlr; + uint32_t *gicd_isenabler; + uint32_t *gicd_isactiver; + uint32_t *gicd_ipriorityr; + uint32_t *gicd_itargetsr; + uint32_t *gicd_icfgr; +}; + +static struct gicv2_context gicv2_context; + +static int gicv2_suspend(void) +{ + unsigned int i; + + if ( !gicv2_context.gicd_isenabler ) + { + dprintk(XENLOG_WARNING, "GICv2 suspend context not allocated!\n"); + return -ENOMEM; + } + + /* Save GICC configuration */ + gicv2_context.gicc_ctlr =3D readl_gicc(GICC_CTLR); + gicv2_context.gicc_pmr =3D readl_gicc(GICC_PMR); + gicv2_context.gicc_bpr =3D readl_gicc(GICC_BPR); + + /* Save GICD configuration */ + gicv2_context.gicd_ctlr =3D readl_gicd(GICD_CTLR); + + for ( i =3D 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ ) + { + gicv2_context.gicd_isenabler[i] =3D readl_gicd(GICD_ISENABLER + i = * 4); + gicv2_context.gicd_isactiver[i] =3D readl_gicd(GICD_ISACTIVER + i = * 4); + } + + for ( i =3D 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ ) + { + gicv2_context.gicd_ipriorityr[i] =3D readl_gicd(GICD_IPRIORITYR + = i * 4); + gicv2_context.gicd_itargetsr[i] =3D readl_gicd(GICD_ITARGETSR + i = * 4); + } + + for ( i =3D 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ ) + gicv2_context.gicd_icfgr[i] =3D readl_gicd(GICD_ICFGR + i * 4); + + return 0; +} + +static void gicv2_resume(void) +{ + unsigned int i; + + gicv2_cpu_disable(); + /* Disable distributor */ + writel_gicd(0, GICD_CTLR); + + /* Restore GICD configuration */ + for ( i =3D 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ ) + { + writel_gicd(0xffffffff, GICD_ICENABLER + i * 4); + writel_gicd(gicv2_context.gicd_isenabler[i], GICD_ISENABLER + i * = 4); + + writel_gicd(0xffffffff, GICD_ICACTIVER + i * 4); + writel_gicd(gicv2_context.gicd_isactiver[i], GICD_ISACTIVER + i * = 4); + } + + for ( i =3D 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ ) + { + writel_gicd(gicv2_context.gicd_ipriorityr[i], GICD_IPRIORITYR + i = * 4); + writel_gicd(gicv2_context.gicd_itargetsr[i], GICD_ITARGETSR + i * = 4); + } + + for ( i =3D 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ ) + writel_gicd(gicv2_context.gicd_icfgr[i], GICD_ICFGR + i * 4); + + /* Make sure all registers are restored and enable distributor */ + writel_gicd(gicv2_context.gicd_ctlr | GICD_CTL_ENABLE, GICD_CTLR); + + /* Restore GIC CPU interface configuration */ + writel_gicc(gicv2_context.gicc_pmr, GICC_PMR); + writel_gicc(gicv2_context.gicc_bpr, GICC_BPR); + + /* Enable GIC CPU interface */ + writel_gicc(gicv2_context.gicc_ctlr | GICC_CTL_ENABLE | GICC_CTL_EOI, + GICC_CTLR); +} + +static void gicv2_alloc_context(struct gicv2_context *gc) +{ + uint32_t n =3D gicv2_info.nr_lines; + + gc->gicd_isenabler =3D xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32)); + if ( !gc->gicd_isenabler ) + goto err_free; + + gc->gicd_isactiver =3D xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32)); + if ( !gc->gicd_isactiver ) + goto err_free; + + gc->gicd_itargetsr =3D xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4)); + if ( !gc->gicd_itargetsr ) + goto err_free; + + gc->gicd_ipriorityr =3D xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4)); + if ( !gc->gicd_ipriorityr ) + goto err_free; + + gc->gicd_icfgr =3D xzalloc_array(uint32_t, DIV_ROUND_UP(n, 16)); + if ( !gc->gicd_icfgr ) + goto err_free; + + return; + + err_free: + printk(XENLOG_ERR "Failed to allocate memory for GICv2 suspend context= \n"); + + xfree(gc->gicd_icfgr); + xfree(gc->gicd_ipriorityr); + xfree(gc->gicd_itargetsr); + xfree(gc->gicd_isactiver); + xfree(gc->gicd_isenabler); + + memset(gc, 0, sizeof(*gc)); +} + +#endif /* CONFIG_SYSTEM_SUSPEND */ + #ifdef CONFIG_ACPI static unsigned long gicv2_get_hwdom_extra_madt_size(const struct domain *= d) { @@ -1302,6 +1436,11 @@ static int __init gicv2_init(void) =20 spin_unlock(&gicv2.lock); =20 +#ifdef CONFIG_SYSTEM_SUSPEND + /* Allocate memory to be used for saving GIC context during the suspen= d */ + gicv2_alloc_context(&gicv2_context); +#endif /* CONFIG_SYSTEM_SUSPEND */ + return 0; } =20 @@ -1345,6 +1484,10 @@ static const struct gic_hw_operations gicv2_ops =3D { .map_hwdom_extra_mappings =3D gicv2_map_hwdom_extra_mappings, .iomem_deny_access =3D gicv2_iomem_deny_access, .do_LPI =3D gicv2_do_LPI, +#ifdef CONFIG_SYSTEM_SUSPEND + .suspend =3D gicv2_suspend, + .resume =3D gicv2_resume, +#endif /* CONFIG_SYSTEM_SUSPEND */ }; =20 /* Set up the GIC */ diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c index e80fe0ca24..a018bd7715 100644 --- a/xen/arch/arm/gic.c +++ b/xen/arch/arm/gic.c @@ -425,6 +425,35 @@ int gic_iomem_deny_access(struct domain *d) return gic_hw_ops->iomem_deny_access(d); } =20 +#ifdef CONFIG_SYSTEM_SUSPEND + +int gic_suspend(void) +{ + /* Must be called by boot CPU#0 with interrupts disabled */ + ASSERT(!local_irq_is_enabled()); + ASSERT(!smp_processor_id()); + + if ( !gic_hw_ops->suspend || !gic_hw_ops->resume ) + return -ENOSYS; + + return gic_hw_ops->suspend(); +} + +void gic_resume(void) +{ + /* + * Must be called by boot CPU#0 with interrupts disabled after gic_sus= pend + * has returned successfully. + */ + ASSERT(!local_irq_is_enabled()); + ASSERT(!smp_processor_id()); + ASSERT(gic_hw_ops->resume); + + gic_hw_ops->resume(); +} + +#endif /* CONFIG_SYSTEM_SUSPEND */ + static int cpu_gic_callback(struct notifier_block *nfb, unsigned long action, void *hcpu) diff --git a/xen/arch/arm/include/asm/gic.h b/xen/arch/arm/include/asm/gic.h index 541f0eeb80..a706303008 100644 --- a/xen/arch/arm/include/asm/gic.h +++ b/xen/arch/arm/include/asm/gic.h @@ -280,6 +280,12 @@ extern int gicv_setup(struct domain *d); extern void gic_save_state(struct vcpu *v); extern void gic_restore_state(struct vcpu *v); =20 +#ifdef CONFIG_SYSTEM_SUSPEND +/* Suspend/resume */ +extern int gic_suspend(void); +extern void gic_resume(void); +#endif /* CONFIG_SYSTEM_SUSPEND */ + /* SGI (AKA IPIs) */ enum gic_sgi { GIC_SGI_EVENT_CHECK, @@ -395,6 +401,12 @@ struct gic_hw_operations { int (*iomem_deny_access)(struct domain *d); /* Handle LPIs, which require special handling */ void (*do_LPI)(unsigned int lpi); +#ifdef CONFIG_SYSTEM_SUSPEND + /* Save GIC configuration due to the system suspend */ + int (*suspend)(void); + /* Restore GIC configuration due to the system resume */ + void (*resume)(void); +#endif /* CONFIG_SYSTEM_SUSPEND */ }; =20 extern const struct gic_hw_operations *gic_hw_ops; --=20 2.48.1 From nobody Thu Oct 30 22:54:56 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1756764651; cv=none; d=zohomail.com; s=zohoarc; b=iH4wEbGr+/DdH1AJ5Ef9AFqJGwMI9EJznM3UkbZpKizjp/mocxCA8HrgLRhQKsNj2iFW0/zdAPt50CoD7bZXy8jZyFMUPxhcUPvN/4X5CBH8+zVptICMAYlqG+RjgP5Kv7psGTLUTPqkVIxTYySLGRLTpkKEvS35qw4Nmnv7WZw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1756764651; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=GL9EWkohjo8a/Dn92rOcdmsDJxHtlTsb7OJVk0QSUJM=; b=F8990InxG1NvFfmBvFFHurLklDuzL5XuPXCMY6d7NA0EKL2KjvcZaxiHaJY6XWKzo9b4bHEA64RNIws4JTvKxfBh6K54/buuxtVYPYYN9IAwt0lcfqO6sWVIcbkM9nsI7N4KCo8jG2BWlfVxtFkqLjUCS4RGvGLqt3Kc8OXqHa0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1756764651627270.14825648143665; Mon, 1 Sep 2025 15:10:51 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1105399.1456339 (Exim 4.92) (envelope-from ) id 1utCjN-0005Ys-DK; Mon, 01 Sep 2025 22:10:29 +0000 Received: by outflank-mailman (output) from mailman id 1105399.1456339; Mon, 01 Sep 2025 22:10:29 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjN-0005Yj-AX; Mon, 01 Sep 2025 22:10:29 +0000 Received: by outflank-mailman (input) for mailman id 1105399; Mon, 01 Sep 2025 22:10:27 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjL-00055o-KS for xen-devel@lists.xenproject.org; Mon, 01 Sep 2025 22:10:27 +0000 Received: from mail-lf1-x12a.google.com (mail-lf1-x12a.google.com [2a00:1450:4864:20::12a]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 7647d278-8780-11f0-8dd7-1b34d833f44b; Tue, 02 Sep 2025 00:10:25 +0200 (CEST) Received: by mail-lf1-x12a.google.com with SMTP id 2adb3069b0e04-55f6186cc17so3747152e87.2 for ; Mon, 01 Sep 2025 15:10:25 -0700 (PDT) Received: from yp-VivoBook-ASUSLaptop-M1503QA-M1503QA.. ([95.67.15.120]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5608279307asm123038e87.75.2025.09.01.15.10.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 15:10:24 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 7647d278-8780-11f0-8dd7-1b34d833f44b DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756764625; x=1757369425; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GL9EWkohjo8a/Dn92rOcdmsDJxHtlTsb7OJVk0QSUJM=; b=DuO4koIctpb21IVOdjzP5KU4+SUc96C9z7oZb+Yd6UASzoYMx38LPEb6LGRmO9Rc+r qtRxihrKEthqPb1Z+gHeFN4TGv0xKsqVcBHTPOjKge9LAn49BEl7HQBmfXHf+8L7fIWb LT7vtUIJl4R0c/POeFxMIFamgohmyuL4UJLjH2c+pndAJbfM3OAtTAlUaTgI+18pnSZQ VYxhhCcRVuhM3rSvqYNsATa7WKg1es8Jwqa82jvGlw5YsLXvakZw153TRRDdZosD9nEl VTQ8/0rQHBrj54rOU+gqM0uUZkIlUtyUAceVrH/KAhImhXfa22FAGyTD640pRHhqJ7xL P8/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756764625; x=1757369425; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GL9EWkohjo8a/Dn92rOcdmsDJxHtlTsb7OJVk0QSUJM=; b=aD4YICYeLw2AzxsdloPLu0USEFit6vhd2XkVV/J5mRThHJqKfRFpD61Rziwz5CA8GW 3dycvD/gd485I/hvKciO+qQWHDYb8HTC1AJPFjOdb2UBiw01OlK6gwYXxwZ/xTb2GaYC Gzr++xialMPFXy1FC+zcCp4wyy2AIGxtXkGmeO1SaACPMG6gMlYh6rXw/xiPCHHt4hLb UTouSqDfErWVGnyhvifdCU7sEEJK+oWxvP9+SuJgrZ8sa4WiNMz6hwT/E2MqbAGPcTYe WiyIRBNqE0NZctQzOFFihh6Ovp0ml/XsLDDwMTdIda13odjvc9m0wMdd1noZFkqJSKyo IHIg== X-Gm-Message-State: AOJu0YwfU8H1Z5YxHVrxNvAXv5gJxs06zyypp7M9wtKC/yZPXUn+/g/o C0JWH/JmZTvnr2F+4ThFuIhNHtvwSmhRgNRUWdUMn+FRJniKBVybAGHlkuYjqsP8 X-Gm-Gg: ASbGncsPEIXA6oKR75+8PM0bPqL1jHtjuHDexZbFdTsIeHezKPeuydGZoBovoGVURfb fAhN77Ru7ZBB0bBhlTpiSzrgdXRoueiWqB8gpToczsc40pO0JNTf3PbjGVtW++ayiTj84x8gHbI U8UJma2ZroFnm5a2ftWs20jXsyKrABgM0cgb3a8QGBWzzEyC63Gt926oIJz4UdiLWpd2VUUylJ3 ollMI+EDFbxwpBbGzWTnpaTuGXoJfbZRUwp5tkrqv1GJx0/+MWCblb4O/PdeT2/32aT1XCY2MEw 8aIuhW2Cefw8d6UyeFpJIgHdoneXnTZL0xuxfOJCuNgW1pHmn3OadXTzbp+U8ObAPuVmaoKp2DZ zMJS5F++kYm9S3u7G3Q/Yy+RWH4tpd85IIiTJq4AvvVBNAwIxU70aFTKlG6mEKMFTTgqkcbKu X-Google-Smtp-Source: AGHT+IGHO6vPuMNSviLACKBC72hu/IH8PKVE6FIUQK3U79KxMfeWob9GsNWWKppkqNO3yp5F/sT2uA== X-Received: by 2002:a05:6512:4608:b0:55f:3996:4f82 with SMTP id 2adb3069b0e04-55f708a2cb5mr2329558e87.1.1756764624937; Mon, 01 Sep 2025 15:10:24 -0700 (PDT) From: Mykola Kvach To: xen-devel@lists.xenproject.org Cc: Mykola Kvach , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , Volodymyr Babchuk Subject: [PATCH v6 03/13] xen/arm: gic-v3: Implement GICv3 suspend/resume functions Date: Tue, 2 Sep 2025 01:10:07 +0300 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1756764653143124100 Content-Type: text/plain; charset="utf-8" From: Mykola Kvach System suspend may lead to a state where GIC would be powered down. Therefore, Xen should save/restore the context of GIC on suspend/resume. Note that the context consists of states of registers which are controlled by the hypervisor. Other GIC registers which are accessible by guests are saved/restored on context switch. Signed-off-by: Mykola Kvach --- Changes in V6: - Drop gicv3_save/restore_state since it is already handled during vCPU context switch. - The comment about systems without SPIs is clarified for readability. - Error and warning messages related to suspend context allocation are unif= ied and now use printk() with XENLOG_ERR for consistency. - The check for suspend context allocation in gicv3_resume() is removed, as it is handled earlier in the suspend path. - The loop for saving and restoring PPI/SGI priorities is corrected to use the proper increment. - The gicv3_suspend() function now prints an explicit error if ITS suspend support is not implemented, and returns ENOSYS in this case. - The GICD_CTLR_DS bit definition is added to gic_v3_defs.h. - The comment for GICR_WAKER access is expanded to reference the relevant ARM specification section and clarify the RAZ/WI behavior for Non-secure accesses. - Cleanup active and enable registers before restoring. --- xen/arch/arm/gic-v3-lpi.c | 3 + xen/arch/arm/gic-v3.c | 235 +++++++++++++++++++++++++ xen/arch/arm/include/asm/gic_v3_defs.h | 1 + 3 files changed, 239 insertions(+) diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c index de5052e5cf..61a6e18303 100644 --- a/xen/arch/arm/gic-v3-lpi.c +++ b/xen/arch/arm/gic-v3-lpi.c @@ -391,6 +391,9 @@ static int cpu_callback(struct notifier_block *nfb, uns= igned long action, switch ( action ) { case CPU_UP_PREPARE: + if ( system_state =3D=3D SYS_STATE_resume ) + break; + rc =3D gicv3_lpi_allocate_pendtable(cpu); if ( rc ) printk(XENLOG_ERR "Unable to allocate the pendtable for CPU%lu= \n", diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c index cd3e1acf79..9f1be7e905 100644 --- a/xen/arch/arm/gic-v3.c +++ b/xen/arch/arm/gic-v3.c @@ -1776,6 +1776,233 @@ static bool gic_dist_supports_lpis(void) return (readl_relaxed(GICD + GICD_TYPER) & GICD_TYPE_LPIS); } =20 +#ifdef CONFIG_SYSTEM_SUSPEND + +/* GICv3 registers to be saved/restored on system suspend/resume */ +struct gicv3_ctx { + struct dist_ctx { + uint32_t ctlr; + /* + * This struct represent block of 32 IRQs + * TODO: store extended SPI configuration (GICv3.1+) + */ + struct irq_regs { + uint32_t icfgr[2]; + uint32_t ipriorityr[8]; + uint64_t irouter[32]; + uint32_t isactiver; + uint32_t isenabler; + } *irqs; + } dist; + + /* have only one rdist structure for last running CPU during suspend */ + struct redist_ctx { + uint32_t ctlr; + /* TODO: handle case when we have more than 16 PPIs (GICv3.1+) */ + uint32_t icfgr[2]; + uint32_t igroupr; + uint32_t ipriorityr[8]; + uint32_t isactiver; + uint32_t isenabler; + } rdist; + + struct cpu_ctx { + uint32_t ctlr; + uint32_t pmr; + uint32_t bpr; + uint32_t sre_el2; + uint32_t grpen; + } cpu; +}; + +static struct gicv3_ctx gicv3_ctx; + +static void __init gicv3_alloc_context(void) +{ + uint32_t blocks =3D DIV_ROUND_UP(gicv3_info.nr_lines, 32); + + /* We don't have ITS support for suspend */ + if ( gicv3_its_host_has_its() ) + return; + + /* The spec allows for systems without any SPIs */ + if ( blocks > 1 ) + { + gicv3_ctx.dist.irqs =3D xzalloc_array(typeof(*gicv3_ctx.dist.irqs), + blocks - 1); + if ( !gicv3_ctx.dist.irqs ) + printk(XENLOG_ERR "Failed to allocate memory for GICv3 suspend= context\n"); + } +} + +static void gicv3_disable_redist(void) +{ + void __iomem* waker =3D GICD_RDIST_BASE + GICR_WAKER; + + /* + * Avoid infinite loop if Non-secure does not have access to GICR_WAKE= R. + * See Arm IHI 0069H.b, 12.11.42 GICR_WAKER: + * When GICD_CTLR.DS =3D=3D 0 and an access is Non-secure accesses= to this + * register are RAZ/WI. + */ + if ( !(readl_relaxed(GICD + GICD_CTLR) & GICD_CTLR_DS) ) + return; + + writel_relaxed(readl_relaxed(waker) | GICR_WAKER_ProcessorSleep, waker= ); + while ( (readl_relaxed(waker) & GICR_WAKER_ChildrenAsleep) =3D=3D 0 ); +} + +static int gicv3_suspend(void) +{ + unsigned int i; + void __iomem *base; + typeof(gicv3_ctx.rdist)* rdist =3D &gicv3_ctx.rdist; + + /* TODO: implement support for ITS */ + if ( gicv3_its_host_has_its() ) + { + printk(XENLOG_ERR "GICv3: ITS suspend support is not implemented\n= "); + return -ENOSYS; + } + + if ( !gicv3_ctx.dist.irqs && gicv3_info.nr_lines > NR_GIC_LOCAL_IRQS ) + { + printk(XENLOG_ERR "GICv3: suspend context is not allocated!\n"); + return -ENOMEM; + } + + /* Save GICC configuration */ + gicv3_ctx.cpu.ctlr =3D READ_SYSREG(ICC_CTLR_EL1); + gicv3_ctx.cpu.pmr =3D READ_SYSREG(ICC_PMR_EL1); + gicv3_ctx.cpu.bpr =3D READ_SYSREG(ICC_BPR1_EL1); + gicv3_ctx.cpu.sre_el2 =3D READ_SYSREG(ICC_SRE_EL2); + gicv3_ctx.cpu.grpen =3D READ_SYSREG(ICC_IGRPEN1_EL1); + + gicv3_disable_interface(); + gicv3_disable_redist(); + + /* Save GICR configuration */ + gicv3_redist_wait_for_rwp(); + + base =3D GICD_RDIST_SGI_BASE; + + rdist->ctlr =3D readl_relaxed(base + GICR_CTLR); + + /* Save priority on PPI and SGI interrupts */ + for ( i =3D 0; i < NR_GIC_LOCAL_IRQS / 4; i++ ) + rdist->ipriorityr[i] =3D readl_relaxed(base + GICR_IPRIORITYR0 + 4= * i); + + rdist->isactiver =3D readl_relaxed(base + GICR_ISACTIVER0); + rdist->isenabler =3D readl_relaxed(base + GICR_ISENABLER0); + rdist->igroupr =3D readl_relaxed(base + GICR_IGROUPR0); + rdist->icfgr[0] =3D readl_relaxed(base + GICR_ICFGR0); + rdist->icfgr[1] =3D readl_relaxed(base + GICR_ICFGR1); + + /* Save GICD configuration */ + gicv3_dist_wait_for_rwp(); + gicv3_ctx.dist.ctlr =3D readl_relaxed(GICD + GICD_CTLR); + + for ( i =3D 1; i < DIV_ROUND_UP(gicv3_info.nr_lines, 32); i++ ) + { + typeof(gicv3_ctx.dist.irqs) irqs =3D gicv3_ctx.dist.irqs + i - 1; + unsigned int irq; + + base =3D GICD + GICD_ICFGR + 8 * i; + irqs->icfgr[0] =3D readl_relaxed(base); + irqs->icfgr[1] =3D readl_relaxed(base + 4); + + base =3D GICD + GICD_IPRIORITYR + 32 * i; + for ( irq =3D 0; irq < 8; irq++ ) + irqs->ipriorityr[irq] =3D readl_relaxed(base + 4 * irq); + + base =3D GICD + GICD_IROUTER + 32 * i; + for ( irq =3D 0; irq < 32; irq++ ) + irqs->irouter[irq] =3D readq_relaxed_non_atomic(base + 8 * irq= ); + + irqs->isactiver =3D readl_relaxed(GICD + GICD_ISACTIVER + 4 * i); + irqs->isenabler =3D readl_relaxed(GICD + GICD_ISENABLER + 4 * i); + } + + return 0; +} + +static void gicv3_resume(void) +{ + unsigned int i; + void __iomem *base; + typeof(gicv3_ctx.rdist)* rdist =3D &gicv3_ctx.rdist; + + writel_relaxed(0, GICD + GICD_CTLR); + + for ( i =3D NR_GIC_LOCAL_IRQS; i < gicv3_info.nr_lines; i +=3D 32 ) + writel_relaxed(GENMASK(31, 0), GICD + GICD_IGROUPR + (i / 32) * 4); + + for ( i =3D 1; i < DIV_ROUND_UP(gicv3_info.nr_lines, 32); i++ ) + { + typeof(gicv3_ctx.dist.irqs) irqs =3D gicv3_ctx.dist.irqs + i - 1; + unsigned int irq; + + base =3D GICD + GICD_ICFGR + 8 * i; + writel_relaxed(irqs->icfgr[0], base); + writel_relaxed(irqs->icfgr[1], base + 4); + + base =3D GICD + GICD_IPRIORITYR + 32 * i; + for ( irq =3D 0; irq < 8; irq++ ) + writel_relaxed(irqs->ipriorityr[irq], base + 4 * irq); + + base =3D GICD + GICD_IROUTER + 32 * i; + for ( irq =3D 0; irq < 32; irq++ ) + writeq_relaxed_non_atomic(irqs->irouter[irq], base + 8 * irq); + + writel_relaxed(GENMASK(31, 0), GICD + GICD_ICENABLER + i * 4); + writel_relaxed(irqs->isenabler, GICD + GICD_ISENABLER + i * 4); + + writel_relaxed(GENMASK(31, 0), GICD + GICD_ICACTIVER + i * 4); + writel_relaxed(irqs->isactiver, GICD + GICD_ISACTIVER + i * 4); + } + + writel_relaxed(gicv3_ctx.dist.ctlr, GICD + GICD_CTLR); + gicv3_dist_wait_for_rwp(); + + /* Restore GICR (Redistributor) configuration */ + gicv3_enable_redist(); + + base =3D GICD_RDIST_SGI_BASE; + + writel_relaxed(0xffffffff, base + GICR_ICENABLER0); + gicv3_redist_wait_for_rwp(); + + for ( i =3D 0; i < NR_GIC_LOCAL_IRQS / 4; i++ ) + writel_relaxed(rdist->ipriorityr[i], base + GICR_IPRIORITYR0 + i *= 4); + + writel_relaxed(rdist->isactiver, base + GICR_ISACTIVER0); + + writel_relaxed(rdist->igroupr, base + GICR_IGROUPR0); + writel_relaxed(rdist->icfgr[0], base + GICR_ICFGR0); + writel_relaxed(rdist->icfgr[1], base + GICR_ICFGR1); + + gicv3_redist_wait_for_rwp(); + + writel_relaxed(rdist->isenabler, base + GICR_ISENABLER0); + writel_relaxed(rdist->ctlr, GICD_RDIST_BASE + GICR_CTLR); + + gicv3_redist_wait_for_rwp(); + + WRITE_SYSREG(gicv3_ctx.cpu.sre_el2, ICC_SRE_EL2); + isb(); + + /* Restore CPU interface (System registers) */ + WRITE_SYSREG(gicv3_ctx.cpu.pmr, ICC_PMR_EL1); + WRITE_SYSREG(gicv3_ctx.cpu.bpr, ICC_BPR1_EL1); + WRITE_SYSREG(gicv3_ctx.cpu.ctlr, ICC_CTLR_EL1); + WRITE_SYSREG(gicv3_ctx.cpu.grpen, ICC_IGRPEN1_EL1); + isb(); + + gicv3_hyp_init(); +} + +#endif /* CONFIG_SYSTEM_SUSPEND */ + /* Set up the GIC */ static int __init gicv3_init(void) { @@ -1850,6 +2077,10 @@ static int __init gicv3_init(void) =20 gicv3_hyp_init(); =20 +#ifdef CONFIG_SYSTEM_SUSPEND + gicv3_alloc_context(); +#endif + out: spin_unlock(&gicv3.lock); =20 @@ -1889,6 +2120,10 @@ static const struct gic_hw_operations gicv3_ops =3D { #endif .iomem_deny_access =3D gicv3_iomem_deny_access, .do_LPI =3D gicv3_do_LPI, +#ifdef CONFIG_SYSTEM_SUSPEND + .suspend =3D gicv3_suspend, + .resume =3D gicv3_resume, +#endif }; =20 static int __init gicv3_dt_preinit(struct dt_device_node *node, const void= *data) diff --git a/xen/arch/arm/include/asm/gic_v3_defs.h b/xen/arch/arm/include/= asm/gic_v3_defs.h index 2af093e774..7e86309acb 100644 --- a/xen/arch/arm/include/asm/gic_v3_defs.h +++ b/xen/arch/arm/include/asm/gic_v3_defs.h @@ -56,6 +56,7 @@ #define GICD_TYPE_LPIS (1U << 17) =20 #define GICD_CTLR_RWP (1UL << 31) +#define GICD_CTLR_DS (1U << 6) #define GICD_CTLR_ARE_NS (1U << 4) #define GICD_CTLR_ENABLE_G1A (1U << 1) #define GICD_CTLR_ENABLE_G1 (1U << 0) --=20 2.48.1 From nobody Thu Oct 30 22:54:56 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1756764643; cv=none; d=zohomail.com; s=zohoarc; b=I/iVV8wbnm8OVTH27xmmQ972Q655APVmEezfe9856dOYU7uav2iNGDxkydJrz6Bf3A3rxXvA29UNfqRPrkqpVayVkOUtIP+sXhX0QkKd+UO6VnN9yqiUn4ZFWFyVUpGraV29+hlegp+53spAnX9EcJtx6Cg6zqhXY7nlDCTmpk4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1756764643; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=claAnoGUSFMYFARQ+oTnePrqKOC6QAQ+IRoEf91ZFiM=; b=YJohQtqm7RVcLdpliDrG9TWKxperR7vCdewpGFJd6dBmaaKtk8cbTlfydMxGI3JKjzag9dLRRqXOVbxvCpp+MLoZ9PMsJ5hOcmpctopfHBjEU27g9mG1OzvAUOAt9kVZPIC9uVbveCIJenTM3kuiXhH6Az0EhB9VI3CpduXyN9U= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1756764643675971.7897932335776; Mon, 1 Sep 2025 15:10:43 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1105400.1456344 (Exim 4.92) (envelope-from ) id 1utCjN-0005bs-O3; Mon, 01 Sep 2025 22:10:29 +0000 Received: by outflank-mailman (output) from mailman id 1105400.1456344; Mon, 01 Sep 2025 22:10:29 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjN-0005bH-IT; Mon, 01 Sep 2025 22:10:29 +0000 Received: by outflank-mailman (input) for mailman id 1105400; Mon, 01 Sep 2025 22:10:28 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjM-0005XP-2Y for xen-devel@lists.xenproject.org; Mon, 01 Sep 2025 22:10:28 +0000 Received: from mail-lf1-x136.google.com (mail-lf1-x136.google.com [2a00:1450:4864:20::136]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 76fca5fb-8780-11f0-8adc-4578a1afcccb; Tue, 02 Sep 2025 00:10:27 +0200 (CEST) Received: by mail-lf1-x136.google.com with SMTP id 2adb3069b0e04-55f7039aa1eso2190788e87.1 for ; Mon, 01 Sep 2025 15:10:27 -0700 (PDT) Received: from yp-VivoBook-ASUSLaptop-M1503QA-M1503QA.. ([95.67.15.120]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5608279307asm123038e87.75.2025.09.01.15.10.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 15:10:25 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 76fca5fb-8780-11f0-8adc-4578a1afcccb DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756764626; x=1757369426; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=claAnoGUSFMYFARQ+oTnePrqKOC6QAQ+IRoEf91ZFiM=; b=bl/q8yyy9+exPnPGrmq279pz/L584xBgnnBm8a78VPJ1aEh2Ht5A2y2BgaMus3+YJy f5TAcQmYaJ6Xae7tU2lsfgMsh7uoTlNCrV4mRKB0Ub41nmV34sJnghKiYyd6DutS+Q1p Ldh6Zswb/RV+dqo/pu+e7Stqp5NcIItGa/2pej4fMkRfV8oixb+GwHfTCerT0TC8SRTD b+a3R/mnkootOl5E6WPrlE0fsRltHS67Ig+GpjAeQAsmOZwARH+5keKzIcJAKJR/6DRg kJvGvN6GSO3b803ED2mpSyuyqdnTdFHi6NZSYauTrwiFt4AkmgwELx4oUHY5FHLrOgX4 UVGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756764626; x=1757369426; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=claAnoGUSFMYFARQ+oTnePrqKOC6QAQ+IRoEf91ZFiM=; b=MURouoznv9hN/Bi09PJTb8xQ3znN325wa8WrL6iSac7XCdi1ZR7fGnj0N6XKd8YRRf zK8qJzRZSqZEpJ8IfIfo/U6KtasIzhznGgJWz3bPLINHHcwnix82tnlYCrsIMohe1eGw Lz9ujVk0UuFE3Lmle7SbxElIxHNolSsH6KQ5cbf35emRBHatwOlbquICnpQ9FVbeLd5u ucD9WA1Km614993lwY5vwwYLHVYGO1ODY2q2j/SB2MAgI/UkupcO90e4Z5A7oPAWMGER qhVPftmMG917DUCI3wTYBBBMNenXXmJ1iz4jD2F8jGF4qTYSxjOxMhXCipkGlAba+nPl bS0g== X-Gm-Message-State: AOJu0YwGpID7Hd0p0G6ma7BrNGj9Uu3VfXxPT37HmROZgrdL8JYabKnv p+h/gNx8iP7Iok9pRZ9/EFyBQSi50vbjObqaR18l4JM/9MRnstd76Ra0fFrFQYj6 X-Gm-Gg: ASbGnctmsCvSdQakgvH4Jc6Q8D8LN3NaOC4muNg968DKCK96mKJFwbyooymT5ABHOfo SbksPtsD+M+f0z1TZcsSj/V7u2iAvP5W/yvAGC3JD9cQzTgTu6Kw3oNRQGS2Ivm6adw3Giehj4y 3MoCPMYvqaBoyaRq8EwQhICaHcqkk6ceNGdbnvVvRTrnQjxAkHJ1byPosRNLS3Fh5sGUEU/7FL7 3xq23yZP0uRFNtTVmBxd3fpyueglHvbgNVsQAwgZV4uRAWskjJC4HwaeSLpRZkTFHQNHIcX6O25 N84+jk+zu/5TxFTcLt2+ff3CldmUGdn2OKinbR+vz1CSMl76IRg7Mu5LNvFAKdzfEcr1Y2sNCxJ 4/S1rgd4Qmz8/aVu6RyddlUb6xU6udGdKHUbk6WPKxcuI3bNw565x2+88np9PQRUZ5GRuAbkA X-Google-Smtp-Source: AGHT+IHlRsA6VSEbFS7n1dHyaPDUYiGqNPk98LgH9nIWFqTNJk8rBvrkVuVXOyy9dbnfiE6iP+ROeQ== X-Received: by 2002:a19:6b18:0:b0:560:827f:9ff6 with SMTP id 2adb3069b0e04-560827fa7cfmr125919e87.57.1756764626205; Mon, 01 Sep 2025 15:10:26 -0700 (PDT) From: Mykola Kvach To: xen-devel@lists.xenproject.org Cc: Mykola Kvach , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , Volodymyr Babchuk Subject: [PATCH v6 04/13] xen/arm: Don't release IRQs on suspend Date: Tue, 2 Sep 2025 01:10:08 +0300 Message-ID: <293acbb653b5f4d5bf71dc459f9de3e729bff3e1.1756763487.git.mykola_kvach@epam.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1756764647155124100 Content-Type: text/plain; charset="utf-8" From: Mykola Kvach If we call disable_nonboot_cpus on ARM64 with system_state set to SYS_STATE_suspend, the following assertion will be triggered: ``` (XEN) [ 25.582712] Disabling non-boot CPUs ... (XEN) [ 25.587032] Assertion '!in_irq() && (local_irq_is_enabled() || num= _online_cpus() <=3D 1)' failed at common/xmalloc_tlsf.c:714 [...] (XEN) [ 25.975069] Xen call trace: (XEN) [ 25.978353] [<00000a000022e098>] xfree+0x130/0x1a4 (PC) (XEN) [ 25.984314] [<00000a000022e08c>] xfree+0x124/0x1a4 (LR) (XEN) [ 25.990276] [<00000a00002747d4>] release_irq+0xe4/0xe8 (XEN) [ 25.996152] [<00000a0000278588>] time.c#cpu_time_callback+0x44/= 0x60 (XEN) [ 26.003150] [<00000a000021d678>] notifier_call_chain+0x7c/0xa0 (XEN) [ 26.009717] [<00000a00002018e0>] cpu.c#cpu_notifier_call_chain+= 0x24/0x48 (XEN) [ 26.017148] [<00000a000020192c>] cpu.c#_take_cpu_down+0x28/0x34 (XEN) [ 26.023801] [<00000a0000201944>] cpu.c#take_cpu_down+0xc/0x18 (XEN) [ 26.030281] [<00000a0000225c5c>] stop_machine.c#stopmachine_act= ion+0xbc/0xe4 (XEN) [ 26.038057] [<00000a00002264bc>] tasklet.c#do_tasklet_work+0xb8= /0x100 (XEN) [ 26.045229] [<00000a00002268a4>] do_tasklet+0x68/0xb0 (XEN) [ 26.051018] [<00000a000026e120>] domain.c#idle_loop+0x7c/0x194 (XEN) [ 26.057585] [<00000a0000277e30>] start_secondary+0x21c/0x220 (XEN) [ 26.063978] [<00000a0000361258>] 00000a0000361258 ``` This happens because before invoking take_cpu_down via the stop_machine_run function on the target CPU, stop_machine_run requests the STOPMACHINE_DISABLE_IRQ state on that CPU. Releasing memory in the release_irq function then triggers the assertion: /* * Heap allocations may need TLB flushes which may require IRQs to be * enabled (except when only 1 PCPU is online). */ This patch adds system state checks to guard calls to request_irq and release_irq. These calls are now skipped when system_state is SYS_STATE_{resume,suspend}, preventing unsafe operations during suspend/resume handling. Signed-off-by: Mykola Kvach Reviewed-by: Volodymyr Babchuk --- Changes in V6: - skipping of IRQ release during system suspend is now handled inside release_irq(). Changes in V4: - removed the prior tasklet-based workaround in favor of a more straightforward and safer solution - reworked the approach by adding explicit system state checks around request_irq and release_irq calls, skips these calls during suspend and resume states to avoid unsafe memory operations when IRQs are disabled --- xen/arch/arm/gic.c | 3 +++ xen/arch/arm/irq.c | 3 +++ xen/arch/arm/tee/ffa_notif.c | 2 +- xen/arch/arm/time.c | 11 +++++++---- 4 files changed, 14 insertions(+), 5 deletions(-) diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c index a018bd7715..c64481faa7 100644 --- a/xen/arch/arm/gic.c +++ b/xen/arch/arm/gic.c @@ -388,6 +388,9 @@ void gic_dump_info(struct vcpu *v) =20 void init_maintenance_interrupt(void) { + if ( system_state =3D=3D SYS_STATE_resume ) + return; + request_irq(gic_hw_ops->info->maintenance_irq, 0, maintenance_interrup= t, "irq-maintenance", NULL); } diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c index 02ca82c089..361496a6d0 100644 --- a/xen/arch/arm/irq.c +++ b/xen/arch/arm/irq.c @@ -300,6 +300,9 @@ void release_irq(unsigned int irq, const void *dev_id) unsigned long flags; struct irqaction *action, **action_ptr; =20 + if ( system_state =3D=3D SYS_STATE_suspend ) + return; + desc =3D irq_to_desc(irq); =20 spin_lock_irqsave(&desc->lock,flags); diff --git a/xen/arch/arm/tee/ffa_notif.c b/xen/arch/arm/tee/ffa_notif.c index 86bef6b3b2..4835e25619 100644 --- a/xen/arch/arm/tee/ffa_notif.c +++ b/xen/arch/arm/tee/ffa_notif.c @@ -363,7 +363,7 @@ void ffa_notif_init_interrupt(void) { int ret; =20 - if ( fw_notif_enabled && notif_sri_irq < NR_GIC_SGI ) + if ( fw_notif_enabled && notif_sri_irq < NR_GIC_SGI && system_state != =3D SYS_STATE_resume ) { /* * An error here is unlikely since the primary CPU has already diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c index ad984fdfdd..8267fa5191 100644 --- a/xen/arch/arm/time.c +++ b/xen/arch/arm/time.c @@ -320,10 +320,13 @@ void init_timer_interrupt(void) WRITE_SYSREG(CNTHCTL_EL2_EL1PCTEN, CNTHCTL_EL2); disable_physical_timers(); =20 - request_irq(timer_irq[TIMER_HYP_PPI], 0, htimer_interrupt, - "hyptimer", NULL); - request_irq(timer_irq[TIMER_VIRT_PPI], 0, vtimer_interrupt, - "virtimer", NULL); + if ( system_state !=3D SYS_STATE_resume ) + { + request_irq(timer_irq[TIMER_HYP_PPI], 0, htimer_interrupt, + "hyptimer", NULL); + request_irq(timer_irq[TIMER_VIRT_PPI], 0, vtimer_interrupt, + "virtimer", NULL); + } =20 check_timer_irq_cfg(timer_irq[TIMER_HYP_PPI], "hypervisor"); check_timer_irq_cfg(timer_irq[TIMER_VIRT_PPI], "virtual"); --=20 2.48.1 From nobody Thu Oct 30 22:54:56 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1756764654; cv=none; d=zohomail.com; s=zohoarc; b=Rn8xgda55k8TERavWgoEKhOJA+mBN9QdKB6rBTInV8nQkXY13H72NpEOce1pJRPkuXHpLtH7CS6tlGR1aY+3INAcv2m9Ol9n/H4ZzzMu2wDphfpLPu95CczRCSJT67QioKkD4017B7/FxbxLH0RO6oEBMZux51YmTFRaMrSAyKc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1756764654; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Imh9qPLbn5GldmbHLssYKpBvOdimEk2erOhkD1ZB1X8=; b=Skye+5K80iXVEYCav5Xn6QNhs7q05ba3D1N156Dhfpa+6l4/3dMq66MRF689o2CVpxoF+ReTzmMOZGU3ETVibItYtx63++T2dQiSO3qbUOFe9m+/KcasSicTOCRCAP748dX8uXLZ2pw72pND6zZT1woNVXlpv0y5ez36BGAcd6s= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 175676465445922.4345083824835; Mon, 1 Sep 2025 15:10:54 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1105401.1456350 (Exim 4.92) (envelope-from ) id 1utCjO-0005kG-6c; Mon, 01 Sep 2025 22:10:30 +0000 Received: by outflank-mailman (output) from mailman id 1105401.1456350; Mon, 01 Sep 2025 22:10:30 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjO-0005iZ-1E; Mon, 01 Sep 2025 22:10:30 +0000 Received: by outflank-mailman (input) for mailman id 1105401; Mon, 01 Sep 2025 22:10:28 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjM-0005XP-O4 for xen-devel@lists.xenproject.org; Mon, 01 Sep 2025 22:10:28 +0000 Received: from mail-lf1-x12f.google.com (mail-lf1-x12f.google.com [2a00:1450:4864:20::12f]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 77a58396-8780-11f0-8adc-4578a1afcccb; Tue, 02 Sep 2025 00:10:28 +0200 (CEST) Received: by mail-lf1-x12f.google.com with SMTP id 2adb3069b0e04-55f78f32580so1211845e87.3 for ; Mon, 01 Sep 2025 15:10:28 -0700 (PDT) Received: from yp-VivoBook-ASUSLaptop-M1503QA-M1503QA.. ([95.67.15.120]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5608279307asm123038e87.75.2025.09.01.15.10.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 15:10:26 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 77a58396-8780-11f0-8adc-4578a1afcccb DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756764627; x=1757369427; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Imh9qPLbn5GldmbHLssYKpBvOdimEk2erOhkD1ZB1X8=; b=la9/h4wyXouvgPLBUiIb+PFQZaA45R6xOBbtN4m5II8VC1xAlw/sWb3NjgQtnkD5pD 90aisZLNF9fZnxtDuD5wtofyTWfP9UaJKrHSstT4ja/gVDIl0tLTmwozNQSbfcDg2xRv /MJDalRCcA2Qq/cMxkL1FAV/EmCA+QneMsPIMrOrXHvOcrYmVfPVijrAbcEi9r/bFs5q OR1DJXr7f9W7CHs2kt30rJ1payZaEOsoM1jyZetqqUrFTnxgStnWRUvxu1D15DSXcy+c R+oxUOt8NHtem+mtWuHF5zbzi0XDXDXurm4qOpEzIbmBZJJhyyKmA8KvqHsWMqPanoEt mrog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756764627; x=1757369427; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Imh9qPLbn5GldmbHLssYKpBvOdimEk2erOhkD1ZB1X8=; b=DTuGhUIXLcToMy6Qgo5XdlwYJkmXTf8XtOuYlG4wiFw5810G5pbGeBy2NjEbSK1XZF LFzrPf7wur0RF/BVWiUsE+2g5mMOuLFf9hci5gjdJt5asmoyvYTxlISovSoWnJa7GmbU 2axd8kn38uSyz2Aqsy1icKuexqacB4KxqBZU9xtkl8z8T6frcuAcQ0iaLka6y7N7xlZ+ w09F3bzcTwZ6JPOB4cUWr/CPpZv4msSaB4SZ+f+IzYBrMvGM/Z5Oq9SZ+S0FT/7yERP4 M+3f1K5o6FCTGteQz5jl9+ykAv3969sWQvOeDBWwPIeVr+ViDi/g7/db/H2wAOZEbybH 4E4w== X-Gm-Message-State: AOJu0Ywd3M5ROClHBIx89fCVlJ6l614G1cUNbwHDCUn6OdwRwvmoIFPE LQRSlSRO/I/aer3jDfnzUL1i0dsC/dJhz8sfM/UymTp4/AGq0jiwh3BKu+0Rum0l X-Gm-Gg: ASbGncuWAQtCPgMwZLfrpEm2fzX9E5YPH9bI5tg4WhEzEhPKS1VH03t5roP6Rw8YDaL EFLo2D1uQ3MpkdrYLdVxMs8QIlEA6skAndZrVJArhiodhh0XKWwZmJ0Pty5vSZOh7P5XJFziasL MHoZAyGCsmqDSzwo4V4NrE1lF6FvgIrsqcXKVX7J978T7MkfaQbvyN7az+dM0AofSBtrTvfB3oa wgD8dWpAFDmmkZAIZ0yYQ5FTmfTyB2KyFH/v81OZJeIrYvzw2FbD5yYBAn+n+hB9wHWZlYi/ECA uUanhG8EA2bd45IIvFkQOHkm5pCRtzR1U5KlZpjVeap5ELlmPmcwxFAPJnyQsrHN/mJlG/pdrgi qFE89X00YVz8tmeoEqWSx3QVGB1rTEbg9Ct6U3oSDi1aqHJknj7Qw1zqBOZJj+878PcRlp0vC X-Google-Smtp-Source: AGHT+IGv4DeyaoILJLzD1OzBQ513bzoPPJ1uc4pXtpi4SvI8+NvOPK8grxlS2JGjHTs+MQgDr7xIIw== X-Received: by 2002:a05:6512:2248:b0:55f:5195:9251 with SMTP id 2adb3069b0e04-55f708ec42bmr2709277e87.28.1756764627392; Mon, 01 Sep 2025 15:10:27 -0700 (PDT) From: Mykola Kvach To: xen-devel@lists.xenproject.org Cc: Mykola Kvach , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , Volodymyr Babchuk , Volodymyr Babchuk Subject: [PATCH v6 05/13] xen/arm: irq: avoid local IRQ descriptors reinit on system resume Date: Tue, 2 Sep 2025 01:10:09 +0300 Message-ID: <84acac884fa1df0ea64eef2253e75918d4b9245f.1756763487.git.mykola_kvach@epam.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1756764657060124100 Content-Type: text/plain; charset="utf-8" From: Mykola Kvach On ARM, during system resume, CPUs are brought online again. This normally triggers init_local_irq_data, which reinitializes IRQ descriptors for banked interrupts (SGIs and PPIs). These descriptors are statically allocated per CPU and retain valid state across suspend/resume cycles. Re-initializing them on resume is unnecessary and may result in loss of interrupt configuration or restored state. This patch skips init_local_irq_data when system_state is set to SYS_STATE_resume to preserve banked IRQ descs state during resume. Signed-off-by: Mykola Kvach Reviewed-by: Volodymyr Babchuk --- xen/arch/arm/irq.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c index 361496a6d0..6c899347ca 100644 --- a/xen/arch/arm/irq.c +++ b/xen/arch/arm/irq.c @@ -125,6 +125,10 @@ static int cpu_callback(struct notifier_block *nfb, un= signed long action, switch ( action ) { case CPU_UP_PREPARE: + /* Skip local IRQ cleanup on resume */ + if ( system_state =3D=3D SYS_STATE_resume ) + break; + rc =3D init_local_irq_data(cpu); if ( rc ) printk(XENLOG_ERR "Unable to allocate local IRQ for CPU%u\n", --=20 2.48.1 From nobody Thu Oct 30 22:54:56 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1756764647; cv=none; d=zohomail.com; s=zohoarc; b=Lawpho56WkBP0nqlOCmJsk8pUcMIIcS5CkkEUBj16nTOAhpPFtC+s8Kz1F8nifae3dCOD2yh9fItK+avFF0GV3paJSFCPlC5KPfWLVNUCtTvV5kHVUNddjK3KTc9/iVSI9z1QQ4fCS90Yj4DrIFgRGBPlkIVdQ90yxGWaf/ILKA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1756764647; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=h+LNbMBKwYezbM5EFVtZ7DgCmMZXO6MyvOXCUpxJxe0=; b=N5q1QoTNgp5RSksmE0FsrzySvAFPP+wZSOPL8hKnA55b3TL/B/mEzK38Je0NPeYWKIJKr+v18MKyGmMEDn5KYMjlyd8QPWWX1hn+otdMZeQvCaJLtVzUkx5eTMLrrkCPxdfEXNdZgLmGB8iGP54M1+gsjCGO5BK5i4++FjaCgEw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1756764646932714.8361981707388; Mon, 1 Sep 2025 15:10:46 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1105402.1456369 (Exim 4.92) (envelope-from ) id 1utCjQ-0006Ju-GV; Mon, 01 Sep 2025 22:10:32 +0000 Received: by outflank-mailman (output) from mailman id 1105402.1456369; Mon, 01 Sep 2025 22:10:32 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjQ-0006Jl-CJ; Mon, 01 Sep 2025 22:10:32 +0000 Received: by outflank-mailman (input) for mailman id 1105402; Mon, 01 Sep 2025 22:10:31 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjP-00055o-5I for xen-devel@lists.xenproject.org; Mon, 01 Sep 2025 22:10:31 +0000 Received: from mail-lf1-x12b.google.com (mail-lf1-x12b.google.com [2a00:1450:4864:20::12b]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 787b9eec-8780-11f0-8dd7-1b34d833f44b; Tue, 02 Sep 2025 00:10:29 +0200 (CEST) Received: by mail-lf1-x12b.google.com with SMTP id 2adb3069b0e04-55f720ffe34so2422009e87.1 for ; Mon, 01 Sep 2025 15:10:29 -0700 (PDT) Received: from yp-VivoBook-ASUSLaptop-M1503QA-M1503QA.. ([95.67.15.120]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5608279307asm123038e87.75.2025.09.01.15.10.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 15:10:27 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 787b9eec-8780-11f0-8dd7-1b34d833f44b DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756764629; x=1757369429; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=h+LNbMBKwYezbM5EFVtZ7DgCmMZXO6MyvOXCUpxJxe0=; b=bCJownRWF8EZOLTdvoJVuxGdEXwx3yuoi+/VNSQxmVMKJfJrmeNXqFU7OYEolzIZRX EQwvlwdu0UBku8xSb63HPpD5hrl77TKs63OHnZDjBI0WvbALfM4IYOBNALZOc1YVCdxu fXG+zl/fMhEDyYel4+d8E09S0V0nvFekHkgKvfWguEI1v2GXXHaNGrkKkt3fUyEZzHmb fL48eTzwjAIK384Bu0v6ej5uE4nZlk5nsCSICqtbAu4hc7QUd2ZR839EUiLtnKjv7SKM 5v5uLtqpRUCL8y1+WS5P3REYHoEeq8isE5wfKhalXk8tteGwKmU3mMrccfc+Ire3UP+v BEag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756764629; x=1757369429; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=h+LNbMBKwYezbM5EFVtZ7DgCmMZXO6MyvOXCUpxJxe0=; b=YYDUWaKos6KSJJMqtP7ktLNif3UiDsHeMQMCC15zBJ9znowBZSXQYtKovCwHJgPpfe d2jVLDTU3p2oNSwg6kx1ugc9tIcpSpODKhO9L8TwBym5K3yuV2MMRYbREdM0gmeCIyRN BOYg0AlTt3969JgzmbSgrHc4U7TcyBRRpo1l12utrdoJxvwy6Jmi7xC8fuCwnhPrO0KB yEBTvIVs+wxJW6SdgonjfMu6i1pU7jCRS/t3j2cTRGQBwXODKXN9UeHH1kmpVHiG91Vp JWNpsWFOj70g02qCHHQc6Ulc7EqwwVAAvGqC4AjyrCpcbEy04ihhN7RgleauDWESeemv HjDA== X-Gm-Message-State: AOJu0YzyNsbX4ZFM0X1y7xdZpYBxsZi0wfJLe0pW/ELoM45zG/0yW3RN DSHvpFv0OtOhyQnk0V4KY3X8PlgwtS/MAs9lx0qI86dym0c/yXSp4d0nZ6PGFGPF X-Gm-Gg: ASbGnctnHwkRWqsMmslDb1PCSHDiUBY+yn31zweN4qCkBeAfsmN27RbplFwfmbPnZaQ 1IPlJukAKC9nB8DLY2HiA+G5tiVEC4BfK0mT6JPPRGdY0tZG4C8r9bxnEEbsHcTxutZb/cnybNx 3ghrR5SLK9C/LTnpGdafWf99Q1ojccX/LSaef3GItazVHpL5OHCKqzwd21xghCdfrM2Ca7z83/R GwWFqp/RHC8uCCsf5BL1BP6aQ3Xljh9lWiNBH6yYry8Y8K6OAeaqzInPI1JtRXC3LCHLI8Fl/6E VXmZ0gKUggoKSQ70BJId0TETYYyaVOWUkh0tiDHL2Kg3nqsZ/TI2NNYw0VL7JyuvtTJEqX7oHeW JnYhHh/IQ9Vdbf0QSGMDDd87LH3mS6KeTjIMSMLI2vp4PJCEkyEHy00jJrPDH/cxH7+6L1SbXd1 fs7zvq3Bc= X-Google-Smtp-Source: AGHT+IEa5RjJ2GSGquDO+Z4AQEB5nVu4GpDaAJNmK102svb69HNEnk3KnmcyEzQrF2S0OM2mun/7QA== X-Received: by 2002:ac2:58e9:0:b0:55f:67c6:be48 with SMTP id 2adb3069b0e04-55f6f6ab662mr1948372e87.1.1756764628474; Mon, 01 Sep 2025 15:10:28 -0700 (PDT) From: Mykola Kvach To: xen-devel@lists.xenproject.org Cc: Mykola Kvach , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , Volodymyr Babchuk Subject: [PATCH v6 06/13] xen/arm: irq: Restore state of local IRQs during system resume Date: Tue, 2 Sep 2025 01:10:10 +0300 Message-ID: <18c51957660441c945d51b02be965fbcc19c7c2b.1756763487.git.mykola_kvach@epam.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1756764648507116600 Content-Type: text/plain; charset="utf-8" From: Mykola Kvach On ARM, the first 32 interrupts (SGIs and PPIs) are banked per-CPU and not restored by gic_resume (for secondary cpus). This patch introduces restore_local_irqs_on_resume, a function that restores the state of local interrupts on the target CPU during system resume. It iterates over all local IRQs and re-enables those that were not disabled, reprogramming their routing and affinity accordingly. The function is invoked from start_secondary, ensuring that local IRQ state is restored early during CPU bring-up after suspend. Signed-off-by: Mykola Kvach --- Changes in V6: - Call handler->disable() instead of just setting the _IRQ_DISABLED flag - Move the system state check outside of restore_local_irqs_on_resume() --- xen/arch/arm/irq.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c index 6c899347ca..ddd2940554 100644 --- a/xen/arch/arm/irq.c +++ b/xen/arch/arm/irq.c @@ -116,6 +116,41 @@ static int init_local_irq_data(unsigned int cpu) return 0; } =20 +/* + * The first 32 interrupts (PPIs and SGIs) are per-CPU, + * so call this function on the target CPU to restore them. + * + * SPIs are restored via gic_resume. + */ +static void restore_local_irqs_on_resume(void) +{ + int irq; + + spin_lock(&local_irqs_type_lock); + + for ( irq =3D 0; irq < NR_LOCAL_IRQS; irq++ ) + { + struct irq_desc *desc =3D irq_to_desc(irq); + + spin_lock(&desc->lock); + + if ( test_bit(_IRQ_DISABLED, &desc->status) ) + { + spin_unlock(&desc->lock); + continue; + } + + /* Disable the IRQ to avoid assertions in the following calls */ + desc->handler->disable(desc); + gic_route_irq_to_xen(desc, GIC_PRI_IRQ); + desc->handler->startup(desc); + + spin_unlock(&desc->lock); + } + + spin_unlock(&local_irqs_type_lock); +} + static int cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu) { @@ -134,6 +169,10 @@ static int cpu_callback(struct notifier_block *nfb, un= signed long action, printk(XENLOG_ERR "Unable to allocate local IRQ for CPU%u\n", cpu); break; + case CPU_STARTING: + if ( system_state =3D=3D SYS_STATE_resume ) + restore_local_irqs_on_resume(); + break; } =20 return notifier_from_errno(rc); --=20 2.48.1 From nobody Thu Oct 30 22:54:56 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1756764651; cv=none; d=zohomail.com; s=zohoarc; b=QjkVS0WODV5Wu+GGzwAL3fJLZZwVR4JIKsK4Tof5e2iEdRDvYf8UVbyuoOQEAdg5Q+dlBbr3XLRZp4qWhnwhAFee5X8QU3SC36SCeelDPSdvL07xR9tHjVmR3fHpbUHWmMkPBW2ftJjSQ7HGKY7/y4NqKzjwJsQdq4j419PZKfs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1756764651; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=RY9tX/c5T2T5//S4piNjlgD25sf5vU9/GmLRb02Wl1w=; b=eivOPrUE7E+OJVR0KG3L1MnJ7WNVpi4GvJwZYCJ5DhDbGabDa9fM/BZk4g9XVSIBAyEnWX1E9JP8lAmSzCw0tmrRFOfBPBetI1az5LUlvX/20plXQAOzYD1x5nsws+LQ79rlPKlOQ8SS/nMUAaN1YDzz8D/MYmWxsyuNr6VAZ4g= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 175676465136671.86257811040923; Mon, 1 Sep 2025 15:10:51 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1105403.1456373 (Exim 4.92) (envelope-from ) id 1utCjQ-0006Nn-Tv; Mon, 01 Sep 2025 22:10:32 +0000 Received: by outflank-mailman (output) from mailman id 1105403.1456373; Mon, 01 Sep 2025 22:10:32 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjQ-0006MQ-OF; Mon, 01 Sep 2025 22:10:32 +0000 Received: by outflank-mailman (input) for mailman id 1105403; Mon, 01 Sep 2025 22:10:31 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjP-0005XP-Gv for xen-devel@lists.xenproject.org; Mon, 01 Sep 2025 22:10:31 +0000 Received: from mail-lf1-x129.google.com (mail-lf1-x129.google.com [2a00:1450:4864:20::129]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 7927e6f3-8780-11f0-8adc-4578a1afcccb; Tue, 02 Sep 2025 00:10:30 +0200 (CEST) Received: by mail-lf1-x129.google.com with SMTP id 2adb3069b0e04-55f7b6e4145so1330704e87.1 for ; Mon, 01 Sep 2025 15:10:30 -0700 (PDT) Received: from yp-VivoBook-ASUSLaptop-M1503QA-M1503QA.. ([95.67.15.120]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5608279307asm123038e87.75.2025.09.01.15.10.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 15:10:28 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 7927e6f3-8780-11f0-8adc-4578a1afcccb DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756764630; x=1757369430; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RY9tX/c5T2T5//S4piNjlgD25sf5vU9/GmLRb02Wl1w=; b=XVAO+pdutTd1AVWu2A18pCKhyfR6/QhArEcEJ7X3+UTFtlM/oJ9Cy/qe6lmWvBLSnU W8rfxGND1yHwimQgs/cMVsJMad1BPYGQN9y0Cl3dsFh5OvIkYTmwEJIlNhcbrIDm4UM+ aKw3bTlXu3zlKLUwLYXXgiQaZFhtqrHQ7pDjfKLt7bdnKGCdSncNslurzFXY9dAMXF1i C6jvYlxu8fF1yR32WsIu6GzUgywRGya6P3IkLAfjFll0qjaI+n9/Xrnz1oPKkifMvrwU bn/gXOcj2zsp+NPlUU1Fv8Wif8nU9DcLVpLxXB7atTrJ5qSkKCDH1XymDV5X6pezRbBK gEtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756764630; x=1757369430; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RY9tX/c5T2T5//S4piNjlgD25sf5vU9/GmLRb02Wl1w=; b=dwpBemIhreX5Y7mic3+KIwzrMMNjTYbu63IQNz+EW8dYauFOEXBpLuwCKxSXqeWpPy Jj8m0hEm5ubw4YVMU8DkYQpd8rzxg31AnuKSq2FgCQjyhA+tGeA84N6cH+Vd+BIbBbZd dNlQyhGwg7jIXv6vKfYhrjNl4riMym6DP1IfU4tX0NPnB+JxVu/nfkydB/kM1KAer1pD h82njPNp2bvsIBa3YZdqrm8jSWLjk5SzJ/d27MisnlUbDnRAQ94mvlwFgOJLMHDXjdfz BNbCOcyzMiet/nBtzDMj5zhSHzMM0g1Ns9OLRgeQ5Ndy6Q5wTutgg3YsHBixkaKFEkZ3 jmEg== X-Gm-Message-State: AOJu0YxcAJX7JeNY3sM6JxdWw/Gd15b4CUWFrNfg3wt0Wo6aATpxJSJb va5DLK7WRlAhcpKRKtc6JjAgs63yif4EC83eJO7+8Rs6bQMMAkpe3DE2hJfUV9fu X-Gm-Gg: ASbGnctGeLmsAG2+gOEXpO+u7BkNVmQyya/6toY8/EqzdwtGEWokrzmUYleOTQUVRsZ JA0TzUy6NIyGQNn9eGA70TiB/85d+XWEDVct6eidFI5NPHhRADVugqEY3fDL8axr2PcrfMw7xwQ SYzm+5plEIX0NnHmOCcO9mhtm90FneLtJ3bfN1GTbs41+E1uAIwKyxpIZb3SuKWnhcjm+7TTy0i n4XaUlulaWlTRO0r/sRYk7+RvxXMZIp8N2PkvA4Z8wHbzegQU18nGExiWgxlus9ocFFilUqjDIb SbPbBv2mUn4uNRngECfZKk3HEsVhAJLESm2QBIipbfSO8ZOB2GUAiFv6XelQ3pzChIJ3RkJF+JX caK9ollVKydjbOMX2XJZ0s/j6Eo9ccJN62fp/ReNaM8xUgM/ZGB6qvFAY5SmAnlFJ0i1GMq2K X-Google-Smtp-Source: AGHT+IEkwBm4cpJTQnKBoX8tD+SaJ/XRbp9ktMXnYGjhR+0nJcJbopep6TTT4iDCLZRyqgZKntvLtQ== X-Received: by 2002:a05:6512:2610:b0:55f:3f03:946d with SMTP id 2adb3069b0e04-55f708b60dcmr2388583e87.23.1756764629748; Mon, 01 Sep 2025 15:10:29 -0700 (PDT) From: Mykola Kvach To: xen-devel@lists.xenproject.org Cc: Oleksandr Tyshchenko , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , Volodymyr Babchuk , Mykola Kvach Subject: [PATCH v6 07/13] iommu/ipmmu-vmsa: Implement suspend/resume callbacks Date: Tue, 2 Sep 2025 01:10:11 +0300 Message-ID: <3a05d0f188943173703690981a7590fd12fabd4c.1756763487.git.mykola_kvach@epam.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1756764653173124100 Content-Type: text/plain; charset="utf-8" From: Oleksandr Tyshchenko Store and restore active context and micro-TLB registers. Tested on R-Car H3 Starter Kit. Signed-off-by: Oleksandr Tyshchenko Signed-off-by: Mykola Kvach Reviewed-by: Volodymyr Babchuk --- Changes in V6: - refactor code related to hw_register struct, from now it's called ipmmu_reg_ctx --- xen/drivers/passthrough/arm/ipmmu-vmsa.c | 257 +++++++++++++++++++++++ 1 file changed, 257 insertions(+) diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c b/xen/drivers/passthr= ough/arm/ipmmu-vmsa.c index ea9fa9ddf3..0973559861 100644 --- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c +++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c @@ -71,6 +71,8 @@ }) #endif =20 +#define dev_dbg(dev, fmt, ...) \ + dev_print(dev, XENLOG_DEBUG, fmt, ## __VA_ARGS__) #define dev_info(dev, fmt, ...) \ dev_print(dev, XENLOG_INFO, fmt, ## __VA_ARGS__) #define dev_warn(dev, fmt, ...) \ @@ -130,6 +132,24 @@ struct ipmmu_features { unsigned int imuctr_ttsel_mask; }; =20 +#ifdef CONFIG_SYSTEM_SUSPEND + +struct ipmmu_reg_ctx { + unsigned int imttlbr0; + unsigned int imttubr0; + unsigned int imttbcr; + unsigned int imctr; +}; + +struct ipmmu_vmsa_backup { + struct device *dev; + unsigned int *utlbs_val; + unsigned int *asids_val; + struct list_head list; +}; + +#endif + /* Root/Cache IPMMU device's information */ struct ipmmu_vmsa_device { struct device *dev; @@ -142,6 +162,9 @@ struct ipmmu_vmsa_device { struct ipmmu_vmsa_domain *domains[IPMMU_CTX_MAX]; unsigned int utlb_refcount[IPMMU_UTLB_MAX]; const struct ipmmu_features *features; +#ifdef CONFIG_SYSTEM_SUSPEND + struct ipmmu_reg_ctx *reg_backup[IPMMU_CTX_MAX]; +#endif }; =20 /* @@ -547,6 +570,222 @@ static void ipmmu_domain_free_context(struct ipmmu_vm= sa_device *mmu, spin_unlock_irqrestore(&mmu->lock, flags); } =20 +#ifdef CONFIG_SYSTEM_SUSPEND + +static DEFINE_SPINLOCK(ipmmu_devices_backup_lock); +static LIST_HEAD(ipmmu_devices_backup); + +static struct ipmmu_reg_ctx root_pgtable[IPMMU_CTX_MAX]; + +static uint32_t ipmmu_imuasid_read(struct ipmmu_vmsa_device *mmu, + unsigned int utlb) +{ + return ipmmu_read(mmu, ipmmu_utlb_reg(mmu, IMUASID(utlb))); +} + +static void ipmmu_utlbs_backup(struct ipmmu_vmsa_device *mmu) +{ + struct ipmmu_vmsa_backup *backup_data; + + dev_dbg(mmu->dev, "Handle micro-TLBs backup\n"); + + spin_lock(&ipmmu_devices_backup_lock); + + list_for_each_entry( backup_data, &ipmmu_devices_backup, list ) + { + struct iommu_fwspec *fwspec =3D dev_iommu_fwspec_get(backup_data->= dev); + unsigned int i; + + if ( to_ipmmu(backup_data->dev) !=3D mmu ) + continue; + + for ( i =3D 0; i < fwspec->num_ids; i++ ) + { + unsigned int utlb =3D fwspec->ids[i]; + + backup_data->asids_val[i] =3D ipmmu_imuasid_read(mmu, utlb); + backup_data->utlbs_val[i] =3D ipmmu_imuctr_read(mmu, utlb); + } + } + + spin_unlock(&ipmmu_devices_backup_lock); +} + +static void ipmmu_utlbs_restore(struct ipmmu_vmsa_device *mmu) +{ + struct ipmmu_vmsa_backup *backup_data; + + dev_dbg(mmu->dev, "Handle micro-TLBs restore\n"); + + spin_lock(&ipmmu_devices_backup_lock); + + list_for_each_entry( backup_data, &ipmmu_devices_backup, list ) + { + struct iommu_fwspec *fwspec =3D dev_iommu_fwspec_get(backup_data->= dev); + unsigned int i; + + if ( to_ipmmu(backup_data->dev) !=3D mmu ) + continue; + + for ( i =3D 0; i < fwspec->num_ids; i++ ) + { + unsigned int utlb =3D fwspec->ids[i]; + + ipmmu_imuasid_write(mmu, utlb, backup_data->asids_val[i]); + ipmmu_imuctr_write(mmu, utlb, backup_data->utlbs_val[i]); + } + } + + spin_unlock(&ipmmu_devices_backup_lock); +} + +static void ipmmu_domain_backup_context(struct ipmmu_vmsa_domain *domain) +{ + struct ipmmu_vmsa_device *mmu =3D domain->mmu->root; + struct ipmmu_reg_ctx *regs =3D mmu->reg_backup[domain->context_id]; + + dev_dbg(mmu->dev, "Handle domain context %u backup\n", domain->context= _id); + + regs->imttlbr0 =3D ipmmu_ctx_read_root(domain, IMTTLBR0); + regs->imttubr0 =3D ipmmu_ctx_read_root(domain, IMTTUBR0); + regs->imttbcr =3D ipmmu_ctx_read_root(domain, IMTTBCR); + regs->imctr =3D ipmmu_ctx_read_root(domain, IMCTR); +} + +static void ipmmu_domain_restore_context(struct ipmmu_vmsa_domain *domain) +{ + struct ipmmu_vmsa_device *mmu =3D domain->mmu->root; + struct ipmmu_reg_ctx *regs =3D mmu->reg_backup[domain->context_id]; + + dev_dbg(mmu->dev, "Handle domain context %u restore\n", domain->contex= t_id); + + ipmmu_ctx_write_root(domain, IMTTLBR0, regs->imttlbr0); + ipmmu_ctx_write_root(domain, IMTTUBR0, regs->imttubr0); + ipmmu_ctx_write_root(domain, IMTTBCR, regs->imttbcr); + ipmmu_ctx_write_all(domain, IMCTR, regs->imctr | IMCTR_FLUSH); +} + +/* + * Xen: Unlike Linux implementation, Xen uses a single driver instance + * for handling all IPMMUs. There is no framework for ipmmu_suspend/resume + * callbacks to be invoked for each IPMMU device. So, we need to iterate + * through all registered IPMMUs performing required actions. + * + * Also take care of restoring special settings, such as translation + * table format, etc. + */ +static int __must_check ipmmu_suspend(void) +{ + struct ipmmu_vmsa_device *mmu; + + if ( !iommu_enabled ) + return 0; + + printk(XENLOG_DEBUG "ipmmu: Suspending ...\n"); + + spin_lock(&ipmmu_devices_lock); + + list_for_each_entry( mmu, &ipmmu_devices, list ) + { + if ( ipmmu_is_root(mmu) ) + { + unsigned int i; + + for ( i =3D 0; i < mmu->num_ctx; i++ ) + { + if ( !mmu->domains[i] ) + continue; + ipmmu_domain_backup_context(mmu->domains[i]); + } + } + else + ipmmu_utlbs_backup(mmu); + } + + spin_unlock(&ipmmu_devices_lock); + + return 0; +} + +static void ipmmu_resume(void) +{ + struct ipmmu_vmsa_device *mmu; + + if ( !iommu_enabled ) + return; + + printk(XENLOG_DEBUG "ipmmu: Resuming ...\n"); + + spin_lock(&ipmmu_devices_lock); + + list_for_each_entry( mmu, &ipmmu_devices, list ) + { + uint32_t reg; + + /* Do not use security group function */ + reg =3D IMSCTLR + mmu->features->control_offset_base; + ipmmu_write(mmu, reg, ipmmu_read(mmu, reg) & ~IMSCTLR_USE_SECGRP); + + if ( ipmmu_is_root(mmu) ) + { + unsigned int i; + + /* Use stage 2 translation table format */ + reg =3D IMSAUXCTLR + mmu->features->control_offset_base; + ipmmu_write(mmu, reg, ipmmu_read(mmu, reg) | IMSAUXCTLR_S2PTE); + + for ( i =3D 0; i < mmu->num_ctx; i++ ) + { + if ( !mmu->domains[i] ) + continue; + ipmmu_domain_restore_context(mmu->domains[i]); + } + } + else + ipmmu_utlbs_restore(mmu); + } + + spin_unlock(&ipmmu_devices_lock); +} + +static int ipmmu_alloc_ctx_suspend(struct device *dev) +{ + struct ipmmu_vmsa_backup *backup_data; + unsigned int *utlbs_val, *asids_val; + struct iommu_fwspec *fwspec =3D dev_iommu_fwspec_get(dev); + + utlbs_val =3D xzalloc_array(unsigned int, fwspec->num_ids); + if ( !utlbs_val ) + return -ENOMEM; + + asids_val =3D xzalloc_array(unsigned int, fwspec->num_ids); + if ( !asids_val ) + { + xfree(utlbs_val); + return -ENOMEM; + } + + backup_data =3D xzalloc(struct ipmmu_vmsa_backup); + if ( !backup_data ) + { + xfree(utlbs_val); + xfree(asids_val); + return -ENOMEM; + } + + backup_data->dev =3D dev; + backup_data->utlbs_val =3D utlbs_val; + backup_data->asids_val =3D asids_val; + + spin_lock(&ipmmu_devices_backup_lock); + list_add(&backup_data->list, &ipmmu_devices_backup); + spin_unlock(&ipmmu_devices_backup_lock); + + return 0; +} + +#endif /* CONFIG_SYSTEM_SUSPEND */ + static int ipmmu_domain_init_context(struct ipmmu_vmsa_domain *domain) { uint64_t ttbr; @@ -559,6 +798,9 @@ static int ipmmu_domain_init_context(struct ipmmu_vmsa_= domain *domain) return ret; =20 domain->context_id =3D ret; +#ifdef CONFIG_SYSTEM_SUSPEND + domain->mmu->root->reg_backup[ret] =3D &root_pgtable[ret]; +#endif =20 /* * TTBR0 @@ -615,6 +857,9 @@ static void ipmmu_domain_destroy_context(struct ipmmu_v= msa_domain *domain) ipmmu_ctx_write_root(domain, IMCTR, IMCTR_FLUSH); ipmmu_tlb_sync(domain); =20 +#ifdef CONFIG_SYSTEM_SUSPEND + domain->mmu->root->reg_backup[domain->context_id] =3D NULL; +#endif ipmmu_domain_free_context(domain->mmu->root, domain->context_id); } =20 @@ -1427,6 +1672,14 @@ static int ipmmu_add_device(u8 devfn, struct device = *dev) } #endif =20 +#ifdef CONFIG_SYSTEM_SUSPEND + if ( ipmmu_alloc_ctx_suspend(dev) ) + { + dev_err(dev, "Failed to allocate context for suspend\n"); + return -ENOMEM; + } +#endif + dev_info(dev, "Added master device (IPMMU %s micro-TLBs %u)\n", dev_name(fwspec->iommu_dev), fwspec->num_ids); =20 @@ -1492,6 +1745,10 @@ static const struct iommu_ops ipmmu_iommu_ops =3D .unmap_page =3D arm_iommu_unmap_page, .dt_xlate =3D ipmmu_dt_xlate, .add_device =3D ipmmu_add_device, +#ifdef CONFIG_SYSTEM_SUSPEND + .suspend =3D ipmmu_suspend, + .resume =3D ipmmu_resume, +#endif }; =20 static __init int ipmmu_init(struct dt_device_node *node, const void *data) --=20 2.48.1 From nobody Thu Oct 30 22:54:56 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1756764646; cv=none; d=zohomail.com; s=zohoarc; b=euEpWYfUl2ZkPzPjvuz0QdHqqZfG/d/An8WZgPaKemK6QMM9QWVg6DWZ0pN4JqPjKijdmyM7aUdeeBdUU/qHiof0TdtQUnCuvp/b2X4HHLrbhnyFcltbfWsaiOcZCv/JrmG1jStX6IX0dAHuHB6gQx8mLi/f02kdsXrOO7bgJJo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1756764646; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=pux8zQR0d0n5/4MKkffDUfDraLZ/szsPvbUCV13VP9s=; b=bMYoRM1KMI9eTizK7gUH0aW5KVHqwCPUIim5K74TXspWyx1AZB7EfFPwOpqxP8LUJPmeUQ5rYaZTH77h6IfYN/AIvkOZk8dVmTkC1X6Fm4yRzHVvebGxKRq5rx1iDqj35jOVYWTI2qY7Rh5PL6Q5Dkxq0/ttPXldb1LB1C1NDMc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1756764646172802.6286177186835; Mon, 1 Sep 2025 15:10:46 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1105404.1456387 (Exim 4.92) (envelope-from ) id 1utCjS-0006lP-Ax; Mon, 01 Sep 2025 22:10:34 +0000 Received: by outflank-mailman (output) from mailman id 1105404.1456387; Mon, 01 Sep 2025 22:10:34 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjS-0006kP-5G; Mon, 01 Sep 2025 22:10:34 +0000 Received: by outflank-mailman (input) for mailman id 1105404; Mon, 01 Sep 2025 22:10:32 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjQ-0005XP-Gp for xen-devel@lists.xenproject.org; Mon, 01 Sep 2025 22:10:32 +0000 Received: from mail-lf1-x134.google.com (mail-lf1-x134.google.com [2a00:1450:4864:20::134]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 79e20492-8780-11f0-8adc-4578a1afcccb; Tue, 02 Sep 2025 00:10:31 +0200 (CEST) Received: by mail-lf1-x134.google.com with SMTP id 2adb3069b0e04-55f7ad815ceso1676658e87.3 for ; Mon, 01 Sep 2025 15:10:31 -0700 (PDT) Received: from yp-VivoBook-ASUSLaptop-M1503QA-M1503QA.. ([95.67.15.120]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5608279307asm123038e87.75.2025.09.01.15.10.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 15:10:30 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 79e20492-8780-11f0-8adc-4578a1afcccb DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756764631; x=1757369431; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pux8zQR0d0n5/4MKkffDUfDraLZ/szsPvbUCV13VP9s=; b=PGGhcZ/hfRBGET+34FZXXy8e9qfhTeoJKvb0IdwX+Ak+g0G97wyBa26UM7q0xp43jI gq/fsySj86GTvTPOPUVlgZTyZicgN9pMkqlr85sGakaQeYQdmh9RQBjnJ4nvUUbEB3h0 xaicmoCEs7a8vEPdS/2PG9twkX76E1FY6mSau1bgjeLXHZVjNgqQBeY9GO2gFQB0iLwN pxwUdc2pAQc0qMhab4xCzgs8wmdWf2+OQyKNrc55WRj1JMzriure58cY1J9eEKFZfu8N +YnrwHNmG0lA63ccmbrXAbCoUEhESIXDN4S2HeMYTQ6x4sFKLLzepyfuVIWvcZvEXjU9 5tXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756764631; x=1757369431; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pux8zQR0d0n5/4MKkffDUfDraLZ/szsPvbUCV13VP9s=; b=uQsueeDGTNzsWjuH/K+YRLhIGDR+jjCoIml4wkolRxMpHMNvGLX3k+C5aXone2kFUu +UqIWI4dfgeEmuau/BnjWUKxTyFYcPigvRe0+dQDBQ2oAPeXKQ/nlCkkl9fWGGMCbUk9 C549kJeWLovTEyee0a6Bpsqs4HkzUaHVo4qzSqusE/4F6qOoVpoSl+9wXSSKeowjTYzb tbhFDxJwNZaWojYrd6vwgVoHEmAd2hJ2BxMrYknyHGZ+ntofIGImnkMNoXIp3i/p7Gfj 9QJQQBLp0tKu2jBiyKJyXzKvrCYaJ06kLx9KEYeAr8KOBeHEOOZfsdyQKlXds7iGN1ts mZWg== X-Gm-Message-State: AOJu0YwIaqPx7/xzKFkBzl5kO6Gd937/V3oa6KsUlwZKnDpqTnZE/fB3 8AfeoVGjfRCLJlBQs0832tvLz39NfrMjQ3h4Fr0vw9a+pvCv5dSKnCJ/iDSzHbDm X-Gm-Gg: ASbGncteqzkOOGFq9iF6sSGTRZ6kyYuDVoLndEqiQ7gyXNsoO5h5vdy1ys7t8GgYqDC TtNZi7CIduHgzSNDVRgbOei47yjXW3JtzqpMvg+mwuTio/BxNkribCwhzMTGMKObpYTFquNmeZJ 3GbpW98AURgh4zRD82lsgc4PHzl0dfCLYZjM8KuNT7O0ijJ4QaYtqnePyNtw80G0Otv33KoJYD3 qxSkH2z6ctMRHunXMRhvLNEx0ohtTizevPLLrd1hRYypGeyahkYK32opul8GUUJUcqbS53Usi6F Ul/G2mqORPHdRUkYuzh/J0TNroyPMkvONugHsaN7GR+pE8QzoGz7c6pX/RfpXy4kbfPMJeXZkl5 uAD8Lh3/YR0zyyTk5yBcr6o1EN/2+0/6LbsclHZEheaa+ZNwlJT0FH7iicqklaV0H0S485yC4 X-Google-Smtp-Source: AGHT+IHac5NYtAevecrJdPsnvQsSQwBeZX1PPUz7eIaje6LakHkn37It/dfRC3WdGHf3MYfBkW7PZQ== X-Received: by 2002:a05:6512:2c94:b0:55f:5b65:a3e3 with SMTP id 2adb3069b0e04-55f7093e96amr2529872e87.35.1756764631025; Mon, 01 Sep 2025 15:10:31 -0700 (PDT) From: Mykola Kvach To: xen-devel@lists.xenproject.org Cc: Mirela Simonovic , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , Volodymyr Babchuk , Saeed Nowshadi , Mykyta Poturai , Mykola Kvach Subject: [PATCH v6 08/13] xen/arm: Implement PSCI SYSTEM_SUSPEND call (host interface) Date: Tue, 2 Sep 2025 01:10:12 +0300 Message-ID: <20b6774cafc2d4d6c92c220d3aa94a666d55c930.1756763487.git.mykola_kvach@epam.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1756764649053124100 Content-Type: text/plain; charset="utf-8" From: Mirela Simonovic Invoke PSCI SYSTEM_SUSPEND to finalize Xen's suspend sequence on ARM64 plat= forms. Pass the resume entry point (hyp_resume) as the first argument to EL3. The = resume handler is currently a stub and will be implemented later in assembly. Igno= re the context ID argument, as is done in Linux. Only enable this path when CONFIG_SYSTEM_SUSPEND is set and PSCI version is >=3D 1.0. Signed-off-by: Mirela Simonovic Signed-off-by: Saeed Nowshadi Signed-off-by: Mykyta Poturai Signed-off-by: Mykola Kvach --- Changes in v6: - move calling of call_psci_system_suspend to commit that implements system_suspend call Changes in v4: - select the appropriate PSCI SYSTEM_SUSPEND function ID based on platform - update comments and commit message to reflect recent changes Changes in v3: - return PSCI_NOT_SUPPORTED instead of a hardcoded 1 on ARM32 - check PSCI version before invoking SYSTEM_SUSPEND in call_psci_system_s= uspend --- xen/arch/arm/arm64/head.S | 8 ++++++++ xen/arch/arm/include/asm/psci.h | 1 + xen/arch/arm/include/asm/suspend.h | 22 ++++++++++++++++++++++ xen/arch/arm/psci.c | 23 ++++++++++++++++++++++- 4 files changed, 53 insertions(+), 1 deletion(-) create mode 100644 xen/arch/arm/include/asm/suspend.h diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S index 72c7b24498..3522c497c5 100644 --- a/xen/arch/arm/arm64/head.S +++ b/xen/arch/arm/arm64/head.S @@ -561,6 +561,14 @@ END(efi_xen_start) =20 #endif /* CONFIG_ARM_EFI */ =20 +#ifdef CONFIG_SYSTEM_SUSPEND + +FUNC(hyp_resume) + b . +END(hyp_resume) + +#endif /* CONFIG_SYSTEM_SUSPEND */ + /* * Local variables: * mode: ASM diff --git a/xen/arch/arm/include/asm/psci.h b/xen/arch/arm/include/asm/psc= i.h index 48a93e6b79..bb3c73496e 100644 --- a/xen/arch/arm/include/asm/psci.h +++ b/xen/arch/arm/include/asm/psci.h @@ -23,6 +23,7 @@ int call_psci_cpu_on(int cpu); void call_psci_cpu_off(void); void call_psci_system_off(void); void call_psci_system_reset(void); +int call_psci_system_suspend(void); =20 /* Range of allocated PSCI function numbers */ #define PSCI_FNUM_MIN_VALUE _AC(0,U) diff --git a/xen/arch/arm/include/asm/suspend.h b/xen/arch/arm/include/asm/= suspend.h new file mode 100644 index 0000000000..7e04c6e915 --- /dev/null +++ b/xen/arch/arm/include/asm/suspend.h @@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#ifndef __ASM_ARM_SUSPEND_H__ +#define __ASM_ARM_SUSPEND_H__ + +#ifdef CONFIG_SYSTEM_SUSPEND + +void hyp_resume(void); + +#endif /* CONFIG_SYSTEM_SUSPEND */ + +#endif /* __ASM_ARM_SUSPEND_H__ */ + + /* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/xen/arch/arm/psci.c b/xen/arch/arm/psci.c index b6860a7760..c9d126b195 100644 --- a/xen/arch/arm/psci.c +++ b/xen/arch/arm/psci.c @@ -17,17 +17,20 @@ #include #include #include +#include =20 /* * While a 64-bit OS can make calls with SMC32 calling conventions, for * some calls it is necessary to use SMC64 to pass or return 64-bit values. - * For such calls PSCI_0_2_FN_NATIVE(x) will choose the appropriate + * For such calls PSCI_*_FN_NATIVE(x) will choose the appropriate * (native-width) function ID. */ #ifdef CONFIG_ARM_64 #define PSCI_0_2_FN_NATIVE(name) PSCI_0_2_FN64_##name +#define PSCI_1_0_FN_NATIVE(name) PSCI_1_0_FN64_##name #else #define PSCI_0_2_FN_NATIVE(name) PSCI_0_2_FN32_##name +#define PSCI_1_0_FN_NATIVE(name) PSCI_1_0_FN32_##name #endif =20 uint32_t psci_ver; @@ -60,6 +63,24 @@ void call_psci_cpu_off(void) } } =20 +int call_psci_system_suspend(void) +{ +#ifdef CONFIG_SYSTEM_SUSPEND + struct arm_smccc_res res; + + if ( psci_ver < PSCI_VERSION(1, 0) ) + return PSCI_NOT_SUPPORTED; + + /* 2nd argument (context ID) is not used */ + arm_smccc_smc(PSCI_1_0_FN_NATIVE(SYSTEM_SUSPEND), __pa(hyp_resume), &r= es); + return PSCI_RET(res); +#else + dprintk(XENLOG_WARNING, + "SYSTEM_SUSPEND not supported (CONFIG_SYSTEM_SUSPEND disabled)= \n"); + return PSCI_NOT_SUPPORTED; +#endif +} + void call_psci_system_off(void) { if ( psci_ver > PSCI_VERSION(0, 1) ) --=20 2.48.1 From nobody Thu Oct 30 22:54:56 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1756764655; cv=none; d=zohomail.com; s=zohoarc; b=VACmOMMyAyeA28nG9N5W2ahMZtm5LNylI1X37EdKulJ3+V8ESrH9PTyFbLXzyvlUXW2SbSeb5U29/jzdsUXiXJ9evvIl+vv3bawSi1W9wESIYj/E9+7lE0C2hDtwhz7Y1tTA6ITHzM7sjyDfLROkp2tC8Zs523ObKVuwzzO/DDE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1756764655; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Seeyes3vY54fd1iXGBX0pVpgCs7BHM5X3gEx1tAo94Y=; b=NHo7D2TibSS4WAzEsPaBLNuPTUJmSenAMKEIR8ETxv3r7OzFzZVhL38zJQBVvGBdLAeL+onBJpB3iA+1xNJIH8Tzoy7PSZLO01PTVyMZ7ra0semiOaOT+z2Kc2a6oRoq9Eh3ifpcFkyBG57hzprE1fclarSe7wWy35Dr2I6KiVw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1756764655882511.30129887327143; Mon, 1 Sep 2025 15:10:55 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1105407.1456404 (Exim 4.92) (envelope-from ) id 1utCjU-0007Go-Ay; Mon, 01 Sep 2025 22:10:36 +0000 Received: by outflank-mailman (output) from mailman id 1105407.1456404; Mon, 01 Sep 2025 22:10:36 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjU-0007FB-3r; Mon, 01 Sep 2025 22:10:36 +0000 Received: by outflank-mailman (input) for mailman id 1105407; Mon, 01 Sep 2025 22:10:34 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjS-00055o-NV for xen-devel@lists.xenproject.org; Mon, 01 Sep 2025 22:10:34 +0000 Received: from mail-lf1-x134.google.com (mail-lf1-x134.google.com [2a00:1450:4864:20::134]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 7a98470c-8780-11f0-8dd7-1b34d833f44b; Tue, 02 Sep 2025 00:10:33 +0200 (CEST) Received: by mail-lf1-x134.google.com with SMTP id 2adb3069b0e04-55f7a34fb35so1270886e87.1 for ; Mon, 01 Sep 2025 15:10:33 -0700 (PDT) Received: from yp-VivoBook-ASUSLaptop-M1503QA-M1503QA.. ([95.67.15.120]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5608279307asm123038e87.75.2025.09.01.15.10.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 15:10:31 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 7a98470c-8780-11f0-8dd7-1b34d833f44b DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756764632; x=1757369432; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Seeyes3vY54fd1iXGBX0pVpgCs7BHM5X3gEx1tAo94Y=; b=BK6ECKdm0QAJkSxTS9j1iOvns+W8+lf78lpJaGTdVEsmKOhtKnQcqUfC9vpK/znma6 f2e/+0xZtY+g3ixFop3ISP6P30XFNziscIDRemlk2sW2vIK8VBDUC4vOwNl6uknzHw1y /pJ3y6Q1MfDRqPDR5ZvB9vS6wXw1BfRpmDrz/HkrH5Qf2JdIlXGJrLxN1KooSjwqF2u5 D1BqWJxKkUAVXhzyHsEta9d/WapEfkWYCDvRho01qr48nia3kKFqIFCtaQ5peWrCg75h wJTWs2UKCCS3z3sMoCVhNkKuA4OHWI/HQUc8gKm4Ey+G4TlNWXRM5fWbEARi48UpZ2nq n2mQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756764632; x=1757369432; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Seeyes3vY54fd1iXGBX0pVpgCs7BHM5X3gEx1tAo94Y=; b=mkPNlY7UVqcraRGSq9VmjqswVpZNWZpRWMfNDGDoBactoGcXYPrLVSf7TIQv6+9rMX puhEe4M8QvsHH2abmvQ7lMWtdUCvO920DWXzW+bgmYo8YjtaV9YxQM7CwgOOdlS4gryy YuMs+U+9XRz/3Zqw8WoGMxsk6wYvoJpMSB7/ypKll5cWFrUtD0lbGkyTllNRMlSt3ERr afIh0NqXWfK1k7taR+YE4FzG08QyISd1qg9v1WTBYC0xBXBQZuyOSCCkEnfw79uzJ5mn wSusw6RkZKwGjyxRF69tQSGRGOH8gb2Ic7YgFR3KzDsSkq1Aos4mjylJq7aCOf0tUKV2 Cw0Q== X-Gm-Message-State: AOJu0YyKyLzdhP1zTt++TWi4CDyVrp2NmEnh91ggPQSSDCWx5El8rRor 2mMbRDKSqLYZ3ePjo7/B+M0AuRTk5Aqs5bbgJOuWikqx9ZRS+mzifPI3gjHFUhqa X-Gm-Gg: ASbGncuYWVxgRzsQQYEJsB1yljoJ2IIoNydfHSCCA+eQ8NBT5xR3taAZX1aUAwqPLjA NYAbBQMrpNDvE0Bg85vBaFGARIyOWAJAKps/o2Pa4DmrsC1BkL7H+0IuaYlzju5V9yNs/SPAn6e ARQT27PIWwoDAfezLR7tYlQSIdKDo4qLn9sxwOSiHvY36CwxthGh79vQ4tZbUZZNISUAKCykNKq CCeRnCSOrvwRFs0yhqgOVjlLuQXHKa+12GYNZfs9Hov56l2w6yR0v7ULGhuCAk3YlUF3BKP1/sw TWhR3iE0vlUy6zP/YQzvuggyHtSt5w/Tn4whPMHVt0PsSZsIsKfp1GZSCXvvJ4j42FjhjK2+Ei5 mUzEfE5iA8okyvaxf49AxpzqAhj+w7+mH488CtYAQkYrippCr1hvQJmUzGVJ/n9TlvdO61P33 X-Google-Smtp-Source: AGHT+IH5uEVLs+KWjRNFUSLQfbCaJILBg0f9f4JMQd0Q0IH2Ub2fAbgyI59Wkme8wRbZraFL+Nofsw== X-Received: by 2002:a05:6512:6512:b0:55f:391b:54df with SMTP id 2adb3069b0e04-55f70955022mr2152500e87.47.1756764632215; Mon, 01 Sep 2025 15:10:32 -0700 (PDT) From: Mykola Kvach To: xen-devel@lists.xenproject.org Cc: Mirela Simonovic , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , Volodymyr Babchuk , Saeed Nowshadi , Mykyta Poturai , Mykola Kvach Subject: [PATCH v6 09/13] xen/arm: Resume memory management on Xen resume Date: Tue, 2 Sep 2025 01:10:13 +0300 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1756764656438116600 Content-Type: text/plain; charset="utf-8" From: Mirela Simonovic The MMU must be enabled during the resume path before restoring context, as virtual addresses are used to access the saved context data. This patch adds MMU setup during resume by reusing the existing enable_secondary_cpu_mm function, which enables data cache and the MMU. Before the MMU is enabled, the content of TTBR0_EL2 is changed to point to init_ttbr (page tables used at runtime). Signed-off-by: Mirela Simonovic Signed-off-by: Saeed Nowshadi Signed-off-by: Mykyta Poturai Signed-off-by: Mykola Kvach --- Changes in v6: - moved changes related to set_init_ttbr to commit that implements system_suspend call Changes in v4: - Drop unnecessary DAIF masking; interrupts are already masked on resume - Remove leftover TLB flush instructions; flushing is done in enable_mmu - Avoid setting x19 in hyp_resume; not needed - Replace prepare_secondary_mm with set_init_ttbr; call it from system_susp= end Changes in v3: - Update commit message for clarity - Replace create_page_tables, enable_mmu, and mmu_init_secondary_cpu with enable_secondary_cpu_mm - Move prepare_secondary_mm to start_xen to avoid crash - Add early UART init during resume Changes in v2: - Move hyp_resume to head.S to keep resume logic together - Simplify hyp_resume using existing helpers: check_cpu_mode, cpu_init, create_page_tables, enable_mmu --- xen/arch/arm/arm64/head.S | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S index 3522c497c5..596e960152 100644 --- a/xen/arch/arm/arm64/head.S +++ b/xen/arch/arm/arm64/head.S @@ -564,6 +564,22 @@ END(efi_xen_start) #ifdef CONFIG_SYSTEM_SUSPEND =20 FUNC(hyp_resume) + /* Initialize the UART if earlyprintk has been enabled. */ +#ifdef CONFIG_EARLY_PRINTK + bl init_uart +#endif + PRINT_ID("- Xen resuming -\r\n") + + bl check_cpu_mode + bl cpu_init + + ldr x0, =3Dstart + adr x20, start /* x20 :=3D paddr (start) */ + sub x20, x20, x0 /* x20 :=3D phys-offset */ + ldr lr, =3Dmmu_resumed + b enable_secondary_cpu_mm + +mmu_resumed: b . END(hyp_resume) =20 --=20 2.48.1 From nobody Thu Oct 30 22:54:56 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1756764659; cv=none; d=zohomail.com; s=zohoarc; b=Ktk3nhzHcmu3ue7IYM0KOLpy3O2vJXAkkhVpKXnsy6NM6oh3YBLGx1LLyFGjfrH2xwlMGdJHVpJ3iyCG4PIo3Xf8DNFSwPtqTajXgOlQU/ghl7PMPza7ubEJnbvJnwnBUjDnxyLTZ6NDKkm4umPMT58kdl6pRx5lLTtsEAl2RWc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1756764659; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=RUm3eao0nv7lQDT5rWWvwgkJGYkE0FYcno3r5As+ock=; b=OwAN058Oa2JIAgkziYdc5s5MB8SVFtiDaQm4WRQovPuV0XwwTCNpmi9fpFAE3Dn8IPlhgkFilQekv5rBAYyCQjXEabS2AkRTXf/q+GcfSHddFDR8b4UxNVl+TlauyGxRO21E/0LqqiaMFkoaodgOQl/fGw/Nn5SMeheNqNuFD38= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1756764659875817.6577779715594; Mon, 1 Sep 2025 15:10:59 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1105408.1456410 (Exim 4.92) (envelope-from ) id 1utCjU-0007Nl-U5; Mon, 01 Sep 2025 22:10:36 +0000 Received: by outflank-mailman (output) from mailman id 1105408.1456410; Mon, 01 Sep 2025 22:10:36 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjU-0007M9-OW; Mon, 01 Sep 2025 22:10:36 +0000 Received: by outflank-mailman (input) for mailman id 1105408; Mon, 01 Sep 2025 22:10:35 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjS-0005XP-Vp for xen-devel@lists.xenproject.org; Mon, 01 Sep 2025 22:10:34 +0000 Received: from mail-lf1-x136.google.com (mail-lf1-x136.google.com [2a00:1450:4864:20::136]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 7b522cba-8780-11f0-8adc-4578a1afcccb; Tue, 02 Sep 2025 00:10:34 +0200 (CEST) Received: by mail-lf1-x136.google.com with SMTP id 2adb3069b0e04-55f69cf4b77so3061874e87.2 for ; Mon, 01 Sep 2025 15:10:34 -0700 (PDT) Received: from yp-VivoBook-ASUSLaptop-M1503QA-M1503QA.. ([95.67.15.120]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5608279307asm123038e87.75.2025.09.01.15.10.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 15:10:32 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 7b522cba-8780-11f0-8adc-4578a1afcccb DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756764634; x=1757369434; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RUm3eao0nv7lQDT5rWWvwgkJGYkE0FYcno3r5As+ock=; b=gMC/XyEF3f85vCpaLUurs0tmK7pd5exHkoE5EKQwKE2lGnHCkfJLvkGn9S6vQlBoqU ty9E01XQsuqUc3UnruMF/dYVUVX/BE+Q58ArXY6MPHYXRuf9Vm9EERGD+sCbonW8EafB jfx3aMVaPq7zPPHIt84JlznJ9SuedQzpfhw9AXc2ji6b/RDg8/jivCWQO7cLz4s3z1HI E7Y1W4A8fgCs035XNPLoxV9lM9ClG+11FqlNH4A+r7WIXDDDPdqmipsp88Ek65g+T2NK 1zffexZunfGprfoYyE01iBlEEzZ8scxO/PTwyiDGiQ2Emg4/byV7zrMGiRYdeGKAXwmi mrqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756764634; x=1757369434; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RUm3eao0nv7lQDT5rWWvwgkJGYkE0FYcno3r5As+ock=; b=riQ+hJsV8GHqJVzZF48YEaWn/4MzprJ3lAl/JX5U4uMs7WeFlC7Y3wYojFfMDyWE7A K/zCnZ9Fw60P8T3WOl6oSBXTDM+tyk0zjNGGoWl2iQwBfP7tOeFqyovoaJT/oKQqZzz3 UUR8r3W3eI+u4WA8E1O73VCTbl00JMtBWo3Ol5MtGWIXRSXBzOqHF3WzmMk3+CbqGflJ IQS7/Tg8tmPsYOVaksadNHOK2wE6Exs8oe5AfNIt/x1r5KbXxL/wH5twh3rZVOTQYfjN VqvJJs27NAopjmhguDv5JKL0crH1oxB4PbR02fV+LFq8uk4c+54A9l6nlg6mFbT0l2F+ u3yw== X-Gm-Message-State: AOJu0Yz2CRpcDQ1ROc+jyIezzseLAsh79HcSa9hKcYTIaQOHnb8a1XHf 2c+n7PJ7mAGGv1yPDmxiTT+GXpm650HROYYYSFnB3gv9FSx2wtV3KscBJoDE2Op4 X-Gm-Gg: ASbGncs9lGJ+pSUa9kLWkxVrDOrzraYXMF0lxZP1BfmfTbNaVBJKJoVTyxe/LmabJyl Rvob/a/BuECbt49f9H2bzF5nC5PAbpP7yMQuN/XYgm1rIBXPoYGOdqu+LtXCf5KWU7/jPC48h2C npOJMXs5jgWP1IgSgoFacqpvXru7n8l3IH6rRP2AuDb7Ujy1dbaByuhqpcX2y6cxG8yY1yuvt2w agArvQsi5oTNE7qaC8BZFbjVug5yjrXAl7HCfgiak7Eg4ukwfZzj80+gKo4xhrNtENjQU/UtEvZ aAcOMY7I3uXE0Ws2vOy6LQXNRPRqLU6xmf5W8yHqYPUNkaYITy4OZ6Wr68E/QEZhaCVGd75CH4J 6cGv/D/2NsHqiWa8wAbA/G11cteJd1mRVoxyi6yLmfYptFDdTfL+kwoIcgchZKAT9b+WC2VvD X-Google-Smtp-Source: AGHT+IH5UsKj5cGBhn+itOcgkLe6OXnp3JP492VPrs/+mbJzdvslmcWUzDDr1Sm7tPeTMH53B2gN1A== X-Received: by 2002:a05:6512:2212:b0:55f:6649:45c5 with SMTP id 2adb3069b0e04-55f7089c32bmr2494605e87.11.1756764633457; Mon, 01 Sep 2025 15:10:33 -0700 (PDT) From: Mykola Kvach To: xen-devel@lists.xenproject.org Cc: Mirela Simonovic , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , Volodymyr Babchuk , Saeed Nowshadi , Mykyta Poturai , Mykola Kvach Subject: [PATCH v6 10/13] xen/arm: Save/restore context on suspend/resume Date: Tue, 2 Sep 2025 01:10:14 +0300 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1756764661185124100 Content-Type: text/plain; charset="utf-8" From: Mirela Simonovic The context of CPU general purpose and system control registers must be saved on suspend and restored on resume. This is implemented in prepare_resume_ctx and before the return from the hyp_resume function. The prepare_resume_ctx must be invoked just before the PSCI system suspend call is issued to the ATF. The prepare_resume_ctx must return a non-zero value so that the calling 'if' statement evaluates to true, causing the system suspend to be invoked. Upon resume, the context saved on suspend will be restored, including the link register. Therefore, after restoring the context, the control flow will return to the address pointed to by the saved link register, which is the place from which prepare_resume_ctx was called. To ensure that the calling 'if' statement does not again evaluate to true and initiate system suspend, hyp_resume must return a zero value after restoring the context. Note that the order of saving register context into cpu_context structure must match the order of restoring. Support for ARM32 is not implemented. Instead, compilation fails with a build-time error if suspend is enabled for ARM32. Signed-off-by: Mirela Simonovic Signed-off-by: Saeed Nowshadi Signed-off-by: Mykyta Poturai Signed-off-by: Mykola Kvach --- Changes in v6: - Rename hyp_suspend to prepare_resume_ctx - moved invocation of prepare_resume_ctx to commit which implements system_suspend call Changes in v4: - Produce build-time error for ARM32 when CONFIG_SYSTEM_SUSPEND is enabled - Use register_t instead of uint64_t in cpu_context structure --- xen/arch/arm/Makefile | 1 + xen/arch/arm/arm64/head.S | 90 +++++++++++++++++++++++++++++- xen/arch/arm/include/asm/suspend.h | 22 ++++++++ xen/arch/arm/suspend.c | 14 +++++ 4 files changed, 126 insertions(+), 1 deletion(-) create mode 100644 xen/arch/arm/suspend.c diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile index f833cdf207..3f6247adee 100644 --- a/xen/arch/arm/Makefile +++ b/xen/arch/arm/Makefile @@ -51,6 +51,7 @@ obj-y +=3D setup.o obj-y +=3D shutdown.o obj-y +=3D smp.o obj-y +=3D smpboot.o +obj-$(CONFIG_SYSTEM_SUSPEND) +=3D suspend.o obj-$(CONFIG_SYSCTL) +=3D sysctl.o obj-y +=3D time.o obj-y +=3D traps.o diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S index 596e960152..c6594c0bdd 100644 --- a/xen/arch/arm/arm64/head.S +++ b/xen/arch/arm/arm64/head.S @@ -562,6 +562,52 @@ END(efi_xen_start) #endif /* CONFIG_ARM_EFI */ =20 #ifdef CONFIG_SYSTEM_SUSPEND +/* + * int prepare_resume_ctx(struct cpu_context *ptr) + * + * x0 - pointer to the storage where callee's context will be saved + * + * CPU context saved here will be restored on resume in hyp_resume functio= n. + * prepare_resume_ctx shall return a non-zero value. Upon restoring context + * hyp_resume shall return value zero instead. From C code that invokes + * prepare_resume_ctx, the return value is interpreted to determine whether + * the context is saved (prepare_resume_ctx) or restored (hyp_resume). + */ +FUNC(prepare_resume_ctx) + /* Store callee-saved registers */ + stp x19, x20, [x0], #16 + stp x21, x22, [x0], #16 + stp x23, x24, [x0], #16 + stp x25, x26, [x0], #16 + stp x27, x28, [x0], #16 + stp x29, lr, [x0], #16 + + /* Store stack-pointer */ + mov x2, sp + str x2, [x0], #8 + + /* Store system control registers */ + mrs x2, VBAR_EL2 + str x2, [x0], #8 + mrs x2, VTCR_EL2 + str x2, [x0], #8 + mrs x2, VTTBR_EL2 + str x2, [x0], #8 + mrs x2, TPIDR_EL2 + str x2, [x0], #8 + mrs x2, MDCR_EL2 + str x2, [x0], #8 + mrs x2, HSTR_EL2 + str x2, [x0], #8 + mrs x2, CPTR_EL2 + str x2, [x0], #8 + mrs x2, HCR_EL2 + str x2, [x0], #8 + + /* prepare_resume_ctx must return a non-zero value */ + mov x0, #1 + ret +END(prepare_resume_ctx) =20 FUNC(hyp_resume) /* Initialize the UART if earlyprintk has been enabled. */ @@ -580,7 +626,49 @@ FUNC(hyp_resume) b enable_secondary_cpu_mm =20 mmu_resumed: - b . + /* Now we can access the cpu_context, so restore the context here = */ + ldr x0, =3Dcpu_context + + /* Restore callee-saved registers */ + ldp x19, x20, [x0], #16 + ldp x21, x22, [x0], #16 + ldp x23, x24, [x0], #16 + ldp x25, x26, [x0], #16 + ldp x27, x28, [x0], #16 + ldp x29, lr, [x0], #16 + + /* Restore stack pointer */ + ldr x2, [x0], #8 + mov sp, x2 + + /* Restore system control registers */ + ldr x2, [x0], #8 + msr VBAR_EL2, x2 + ldr x2, [x0], #8 + msr VTCR_EL2, x2 + ldr x2, [x0], #8 + msr VTTBR_EL2, x2 + ldr x2, [x0], #8 + msr TPIDR_EL2, x2 + ldr x2, [x0], #8 + msr MDCR_EL2, x2 + ldr x2, [x0], #8 + msr HSTR_EL2, x2 + ldr x2, [x0], #8 + msr CPTR_EL2, x2 + ldr x2, [x0], #8 + msr HCR_EL2, x2 + isb + + /* + * Since context is restored return from this function will appear + * as return from prepare_resume_ctx. To distinguish a return from + * prepare_resume_ctx which is called upon finalizing the suspend, + * as opposed to return from this function which executes on resum= e, + * we need to return zero value here. + */ + mov x0, #0 + ret END(hyp_resume) =20 #endif /* CONFIG_SYSTEM_SUSPEND */ diff --git a/xen/arch/arm/include/asm/suspend.h b/xen/arch/arm/include/asm/= suspend.h index 7e04c6e915..29eed4ee7f 100644 --- a/xen/arch/arm/include/asm/suspend.h +++ b/xen/arch/arm/include/asm/suspend.h @@ -3,9 +3,31 @@ #ifndef __ASM_ARM_SUSPEND_H__ #define __ASM_ARM_SUSPEND_H__ =20 +#include + #ifdef CONFIG_SYSTEM_SUSPEND =20 +#ifdef CONFIG_ARM_64 +struct cpu_context { + register_t callee_regs[12]; + register_t sp; + register_t vbar_el2; + register_t vtcr_el2; + register_t vttbr_el2; + register_t tpidr_el2; + register_t mdcr_el2; + register_t hstr_el2; + register_t cptr_el2; + register_t hcr_el2; +} __aligned(16); +#else +#error "Define cpu_context structure for arm32" +#endif + +extern struct cpu_context cpu_context; + void hyp_resume(void); +int prepare_resume_ctx(struct cpu_context *ptr); =20 #endif /* CONFIG_SYSTEM_SUSPEND */ =20 diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c new file mode 100644 index 0000000000..5093f1bf3d --- /dev/null +++ b/xen/arch/arm/suspend.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#include + +struct cpu_context cpu_context; + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * indent-tabs-mode: nil + * End: + */ --=20 2.48.1 From nobody Thu Oct 30 22:54:56 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1756764657; cv=none; d=zohomail.com; s=zohoarc; b=NFt1xYsRgcx43pAdNW22tmmod1TDnZCIMlnYCVvzfLLJdaHPHVjsJAopWZjDXNaNt0OMnOJzyOLb2B+TGpZa5x9LzL9A78x3FHSLpm2QMQh9YkrP9lo6tLymUhclS65u7Q2gKO6h4atH7cLjdL1PrR9zo+Hh7NOM8wRNh8zjrRU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1756764657; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=iGzUVANbBHVQm2yt+EaiKz11N+mMUBJTMXRpUM7rLq4=; b=j+pluraYznezhntQCv1owEORN3CCTk0X7TxEIoAnMT9B/8Tg5SorQeLIqfA42aoKyrwrx9Yf2Q+yz7DcRgdy04+S6wWd2wWDD0W3ZpgodNRCI7tQzcksivJWha38x12SE8trh1CzV8qw9YAZgsUUB5NqnMoVVIzq6WbizhMjo60= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1756764657805386.7126283841443; Mon, 1 Sep 2025 15:10:57 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1105410.1456426 (Exim 4.92) (envelope-from ) id 1utCjX-0007wE-3C; Mon, 01 Sep 2025 22:10:39 +0000 Received: by outflank-mailman (output) from mailman id 1105410.1456426; Mon, 01 Sep 2025 22:10:39 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjW-0007ui-Rq; Mon, 01 Sep 2025 22:10:38 +0000 Received: by outflank-mailman (input) for mailman id 1105410; Mon, 01 Sep 2025 22:10:38 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjV-00055o-Sr for xen-devel@lists.xenproject.org; Mon, 01 Sep 2025 22:10:37 +0000 Received: from mail-lf1-x12f.google.com (mail-lf1-x12f.google.com [2a00:1450:4864:20::12f]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 7c5a7763-8780-11f0-8dd7-1b34d833f44b; Tue, 02 Sep 2025 00:10:36 +0200 (CEST) Received: by mail-lf1-x12f.google.com with SMTP id 2adb3069b0e04-55f7cd8ec2cso1473228e87.2 for ; Mon, 01 Sep 2025 15:10:36 -0700 (PDT) Received: from yp-VivoBook-ASUSLaptop-M1503QA-M1503QA.. ([95.67.15.120]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5608279307asm123038e87.75.2025.09.01.15.10.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 15:10:34 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 7c5a7763-8780-11f0-8dd7-1b34d833f44b DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756764635; x=1757369435; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iGzUVANbBHVQm2yt+EaiKz11N+mMUBJTMXRpUM7rLq4=; b=buVqwQTpFdnxff4F5Wq6sWoIKb28FFtDIx5qMbh2rU2EdfrDLp5TzhE6F2TYzEsZNR Rzw3EC0Qg5WZiOqskgeiSWbCGAUqJyjuxGKSfSKCNRLjLssAUPPzkrvUZugnViljDoA5 jiiP8iV+xJPBs5wuzkhpu0HkgI2CW2OhrIjlqiPJ3+HyZwpokDZYxGIk4a3LgF+GzrZ0 ieuuKLqn32tDylrUEL0SZP5RqqNQiYVNXBH6KAe2cSJY+vp6V+8KunSzgdRehhXfd03f QNstpsq+zMc9vNgAFeEoOJTBZxDs8tg90rhNaGC1h0A9D0HEDmD0om7F299EVCE0kvkI s6Fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756764635; x=1757369435; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iGzUVANbBHVQm2yt+EaiKz11N+mMUBJTMXRpUM7rLq4=; b=f85O8GxyC3Z/pZQOuT4fhbXX4qv6WQBi/HapHy3WjQDSvjktUB+uJVs1JSifIKcHdV OS/2xM6lL6EiYFJlo2Sdmqc++0U2td+uh3EFxYhKmWqEh1iJUo/uPUTs0U/ZT1twsxgf bfHoqfJyTQw6vMwSVFmjIDUNANBvPlCe9b+XTtThUd3KMh5AXWnAIz7IH7iyl+4TJ30R V13FxAlsqnfShfVlnFeeFrvYDiA+T7x1YWivDEAh6y0SlWWe0EM5iJAeWq6dIgEfWgD4 Uyi5vsc2rO3+itSJywHwVIfI/x+yd3AoquPBQJAlE1cuH8M+ZzLPqxzRf4klWRFLnt+i bL/w== X-Gm-Message-State: AOJu0YzsP9GCbabLjgyU2+pUAgCUOqrPC4De3uS8cUdlPwBTU+/noK6I XN51rKdB6T53BrQweltl7gdbNdK5Z/KtCdIzaJ/4Zkzyeyh5NbTTx06IceT7ZbcL X-Gm-Gg: ASbGncvvFoycXGyKoyAqStFofUUkvTQ96b5+6c08JrUZ3XjuP29jQKp7DD4y8kNL43F CGaVELZoX4ujh/3JzBlTYms5GIH+I/W/Qz7ZLlvijiuVwU4+oLlkyvExt7EweQKgRpGPGv15CWm wbXJO2XcySsaI36FFgS06mwgHulB+yIO1jZFGlrBtbHiWEfCU1l2+8oVE5AjmQnDOtsqkb998Vw ojIm1ZZsjZGIfAm8nontl05uwgbghfr6slrdixIoPMVCcYcZe3D3Xk3hwsmo75bH/0h3FcVrVZR SmYaTKr/Wg/9IMmjLNvyZ1uZ91oPcVNuCvhf0h9t9T7wvu7b5BZ39syUTeRA8tM5wMmg2qRnSn7 r/gDzPkOEKJGsnOXGVa2ddSaJVkRyqpDo1mJNG5Jvwg/SULNCa53yKxNwByYKCg== X-Google-Smtp-Source: AGHT+IH+YIb5/EiRCiek3MEhwqufDEz2NkLHmu/RcFSNtgnxC2gM0m7/gUw3Q4lC0sULlGwUgJZ6ng== X-Received: by 2002:a05:6512:691:b0:55f:56db:7648 with SMTP id 2adb3069b0e04-55f708b6be9mr2634797e87.19.1756764635113; Mon, 01 Sep 2025 15:10:35 -0700 (PDT) From: Mykola Kvach To: xen-devel@lists.xenproject.org Cc: Mirela Simonovic , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , Volodymyr Babchuk , Andrew Cooper , Anthony PERARD , Jan Beulich , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Saeed Nowshadi , Mykyta Poturai , Mykola Kvach Subject: [PATCH v6 11/13] xen/arm: Add support for system suspend triggered by hardware domain Date: Tue, 2 Sep 2025 01:10:15 +0300 Message-ID: <547196292a007ec2bbedd52036e8f8a0cc69c4ea.1756763487.git.mykola_kvach@epam.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1756764659357124100 Content-Type: text/plain; charset="utf-8" From: Mirela Simonovic Trigger Xen suspend when the hardware domain initiates suspend via SHUTDOWN_suspend. Redirect system suspend to CPU#0 to ensure the suspend logic runs on the boot CPU, as required. Introduce full suspend/resume infrastructure gated by CONFIG_SYSTEM_SUSPEND, including logic to: - disable and enable non-boot physical CPUs - freeze and thaw domains - suspend and resume the GIC, timer, iommu and console - maintain system state before and after suspend On boot, init_ttbr is normally initialized during secondary CPU hotplug. On uniprocessor systems, this would leave init_ttbr uninitialized, causing resume to fail. To address this, the boot CPU now sets init_ttbr during suspend. Remove the restriction in the vPSCI interface preventing suspend from the hardware domain. Select HAS_SYSTEM_SUSPEND for ARM_64. Signed-off-by: Mirela Simonovic Signed-off-by: Saeed Nowshadi Signed-off-by: Mykyta Poturai Signed-off-by: Mykola Kvach --- Changes in v6: - Minor changes in comments. - The implementation now uses own tasklet instead of continue_hypercall_on_= cpu, as the latter rewrites user registers and would tie system suspend status to guest suspend status. - The order of calls when suspending devices has been updated. Changes in v5: - select HAS_SYSTEM_SUSPEND in ARM_64 instead of ARM - check llc_coloring_enabled instead of LLC_COLORING during the selection of HAS_SYSTEM_SUSPEND config - call host_system_suspend from guest PSCI system suspend instead of arch_domain_shutdown, reducing the complexity of the new code - update some comments Changes introduced in V4: - drop code for saving and restoring VCPU context (for more information refer part 1 of patch series) - remove IOMMU suspend and resume calls until they will be implemented - move system suspend logic to arch_domain_shutdown, invoked from domain_shutdown - apply minor fixes such as renaming functions, updating comments, and modifying the commit message to reflect these changes - add console_end_sync to resume path after system suspend Changes introduced in V3: - merge changes from other commits into this patch (previously stashed): 1) disable/enable non-boot physical CPUs and freeze/thaw domains during suspend/resume 2) suspend/resume the timer, GIC, console, IOMMU, and hardware domain --- xen/arch/arm/Kconfig | 1 + xen/arch/arm/include/asm/mm.h | 2 + xen/arch/arm/include/asm/suspend.h | 2 + xen/arch/arm/mmu/smpboot.c | 2 +- xen/arch/arm/suspend.c | 150 +++++++++++++++++++++++++++++ xen/arch/arm/vpsci.c | 9 +- xen/common/domain.c | 4 + 7 files changed, 168 insertions(+), 2 deletions(-) diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig index 5355534f3d..fdad53fd68 100644 --- a/xen/arch/arm/Kconfig +++ b/xen/arch/arm/Kconfig @@ -8,6 +8,7 @@ config ARM_64 depends on !ARM_32 select 64BIT select HAS_FAST_MULTIPLY + select HAS_SYSTEM_SUSPEND if UNSUPPORTED select HAS_VPCI_GUEST_SUPPORT if PCI_PASSTHROUGH =20 config ARM diff --git a/xen/arch/arm/include/asm/mm.h b/xen/arch/arm/include/asm/mm.h index 7a93dad2ed..61e211d087 100644 --- a/xen/arch/arm/include/asm/mm.h +++ b/xen/arch/arm/include/asm/mm.h @@ -365,6 +365,8 @@ static inline void page_set_xenheap_gfn(struct page_inf= o *p, gfn_t gfn) } while ( (y =3D cmpxchg(&p->u.inuse.type_info, x, nx)) !=3D x ); } =20 +void set_init_ttbr(lpae_t *root); + #endif /* __ARCH_ARM_MM__ */ /* * Local variables: diff --git a/xen/arch/arm/include/asm/suspend.h b/xen/arch/arm/include/asm/= suspend.h index 29eed4ee7f..8d30b01b7c 100644 --- a/xen/arch/arm/include/asm/suspend.h +++ b/xen/arch/arm/include/asm/suspend.h @@ -29,6 +29,8 @@ extern struct cpu_context cpu_context; void hyp_resume(void); int prepare_resume_ctx(struct cpu_context *ptr); =20 +void host_system_suspend(void); + #endif /* CONFIG_SYSTEM_SUSPEND */ =20 #endif /* __ASM_ARM_SUSPEND_H__ */ diff --git a/xen/arch/arm/mmu/smpboot.c b/xen/arch/arm/mmu/smpboot.c index 37e91d72b7..ff508ecf40 100644 --- a/xen/arch/arm/mmu/smpboot.c +++ b/xen/arch/arm/mmu/smpboot.c @@ -72,7 +72,7 @@ static void clear_boot_pagetables(void) clear_table(boot_third); } =20 -static void set_init_ttbr(lpae_t *root) +void set_init_ttbr(lpae_t *root) { /* * init_ttbr is part of the identity mapping which is read-only. So diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c index 5093f1bf3d..35b20581f1 100644 --- a/xen/arch/arm/suspend.c +++ b/xen/arch/arm/suspend.c @@ -1,9 +1,159 @@ /* SPDX-License-Identifier: GPL-2.0-only */ =20 +#include #include =20 +#include +#include +#include +#include +#include + +/* + * TODO list: + * - Decide which domain will trigger system suspend ctl or hw ? + * - Test system suspend with LLC_COLORING enabled and verify functionali= ty + * - Implement IOMMU suspend/resume handlers and integrate them + * into the suspend/resume path (SMMU) + * - Enable "xl suspend" support on ARM architecture + * - Properly disable Xen timer watchdog from relevant services (init.d l= eft) + * - Add suspend/resume CI test for ARM (QEMU if feasible) + * - Investigate feasibility and need for implementing system suspend on = ARM32 + */ + struct cpu_context cpu_context; =20 +/* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */ +static void cf_check system_suspend(void *data) +{ + int status; + unsigned long flags; + /* TODO: drop check after verification that features can work together= */ + if ( llc_coloring_enabled ) + { + dprintk(XENLOG_ERR, + "System suspend is not supported with LLC_COLORING enabled= \n"); + status =3D -ENOSYS; + goto dom_resume; + } + + BUG_ON(system_state !=3D SYS_STATE_active); + + system_state =3D SYS_STATE_suspend; + + printk("Xen suspending...\n"); + + freeze_domains(); + scheduler_disable(); + + /* + * Non-boot CPUs have to be disabled on suspend and enabled on resume + * (hotplug-based mechanism). Disabling non-boot CPUs will lead to PSCI + * CPU_OFF to be called by each non-boot CPU. Depending on the underly= ing + * platform capabilities, this may lead to the physical powering down = of + * CPUs. + */ + status =3D disable_nonboot_cpus(); + if ( status ) + { + system_state =3D SYS_STATE_resume; + goto resume_nonboot_cpus; + } + + time_suspend(); + + console_start_sync(); + status =3D console_suspend(); + if ( status ) + { + dprintk(XENLOG_ERR, "Failed to suspend the console, err=3D%d\n", s= tatus); + system_state =3D SYS_STATE_resume; + goto resume_console; + } + + local_irq_save(flags); + status =3D gic_suspend(); + if ( status ) + { + system_state =3D SYS_STATE_resume; + goto resume_irqs; + } + + set_init_ttbr(xen_pgtable); + + /* + * Enable identity mapping before entering suspend to simplify + * the resume path + */ + update_boot_mapping(true); + + if ( prepare_resume_ctx(&cpu_context) ) + { + status =3D call_psci_system_suspend(); + /* + * If suspend is finalized properly by above system suspend PSCI c= all, + * the code below in this 'if' branch will never execute. Execution + * will continue from hyp_resume which is the hypervisor's resume = point. + * In hyp_resume CPU context will be restored and since link-regis= ter is + * restored as well, it will appear to return from prepare_resume_= ctx. + * The difference in returning from prepare_resume_ctx on system s= uspend + * versus resume is in function's return value: on suspend, the re= turn + * value is a non-zero value, on resume it is zero. That is why the + * control flow will not re-enter this 'if' branch on resume. + */ + if ( status ) + dprintk(XENLOG_WARNING, "PSCI system suspend failed, err=3D%d\= n", + status); + } + + system_state =3D SYS_STATE_resume; + update_boot_mapping(false); + + gic_resume(); + + resume_irqs: + local_irq_restore(flags); + + resume_console: + console_resume(); + console_end_sync(); + + time_resume(); + + resume_nonboot_cpus: + /* + * The rcu_barrier() has to be added to ensure that the per cpu area is + * freed before a non-boot CPU tries to initialize it (_free_percpu_ar= ea() + * has to be called before the init_percpu_area()). This scenario occu= rs + * when non-boot CPUs are hot-unplugged on suspend and hotplugged on r= esume. + */ + rcu_barrier(); + enable_nonboot_cpus(); + scheduler_enable(); + thaw_domains(); + + system_state =3D SYS_STATE_active; + + printk("Resume (status %d)\n", status); + + dom_resume: + /* The resume of hardware domain should always follow Xen's resume. */ + domain_resume(hardware_domain); +} + +static DECLARE_TASKLET(system_suspend_tasklet, system_suspend, NULL); + +void host_system_suspend(void) +{ + /* + * system_suspend should be called when hardware domain finalizes the + * suspend procedure from its boot core (VCPU#0). However, Dom0's vCPU= #0 + * could be mapped to any pCPU. The suspend procedure has to be finali= zed + * by the pCPU#0 (non-boot pCPUs will be disabled during the suspend). + */ + tasklet_schedule_on_cpu(&system_suspend_tasklet, 0); +} + /* * Local variables: * mode: C diff --git a/xen/arch/arm/vpsci.c b/xen/arch/arm/vpsci.c index 22c3a5f544..2f52aba48d 100644 --- a/xen/arch/arm/vpsci.c +++ b/xen/arch/arm/vpsci.c @@ -4,6 +4,7 @@ #include =20 #include +#include #include #include #include @@ -221,9 +222,10 @@ static int32_t do_psci_1_0_system_suspend(register_t e= point, register_t cid) if ( is_64bit_domain(d) && is_thumb ) return PSCI_INVALID_ADDRESS; =20 - /* SYSTEM_SUSPEND is not supported for the hardware domain yet */ +#ifndef CONFIG_SYSTEM_SUSPEND if ( is_hardware_domain(d) ) return PSCI_NOT_SUPPORTED; +#endif =20 /* Ensure that all CPUs other than the calling one are offline */ domain_lock(d); @@ -249,6 +251,11 @@ static int32_t do_psci_1_0_system_suspend(register_t e= point, register_t cid) "SYSTEM_SUSPEND requested, epoint=3D0x%"PRIregister", cid=3D0x= %"PRIregister"\n", epoint, cid); =20 +#ifdef CONFIG_SYSTEM_SUSPEND + if ( is_hardware_domain(d) ) + host_system_suspend(); +#endif + return rc; } =20 diff --git a/xen/common/domain.c b/xen/common/domain.c index 667017c5e1..5e224740d3 100644 --- a/xen/common/domain.c +++ b/xen/common/domain.c @@ -1317,7 +1317,11 @@ int domain_shutdown(struct domain *d, u8 reason) d->shutdown_code =3D reason; reason =3D d->shutdown_code; =20 +#if defined(CONFIG_SYSTEM_SUSPEND) && defined(CONFIG_ARM) + if ( reason !=3D SHUTDOWN_suspend && is_hardware_domain(d) ) +#else if ( is_hardware_domain(d) ) +#endif hwdom_shutdown(reason); =20 if ( d->is_shutting_down ) --=20 2.48.1 From nobody Thu Oct 30 22:54:56 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1756764662; cv=none; d=zohomail.com; s=zohoarc; b=ZUNihgyP/PkKnXhGwFcyLAIq+vv2ASd3mVDUzL8kFILFsAuqHW4BDxOsAHFEEemID9Lkb+6smoUhdiECPfrZJhbAQxTeZ9cTmAmneV3oYvvPVDORkiwx4ab4Mei3YKtClxmRkqYutY06TOgvG1QVOm9xCLaAY8aeWk/4NE6ZgrM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1756764662; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=7jLnGK/W9IzxOZ6CLKOO5BPor3o3fyjNQvRa1agLv/o=; b=MsFZY4A6XpaKE2QCG9qtiJizMiYXRFxNL6g76yl6IWQ0mwxOirnOIDCNMzQBocwjy6Dl5yFW/XNEPa4qNMpu4BBAcVDWKoOVA2s1QhTslJ8TAZR9wjzO9wVIsDY4YAAGx+ciI3aMmqVWsPL3SHejbSDXd56+ZDP5SiiWk9gn2Cs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1756764662559437.7836381035529; Mon, 1 Sep 2025 15:11:02 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1105412.1456437 (Exim 4.92) (envelope-from ) id 1utCjZ-0008Qe-Gc; Mon, 01 Sep 2025 22:10:41 +0000 Received: by outflank-mailman (output) from mailman id 1105412.1456437; Mon, 01 Sep 2025 22:10:41 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjZ-0008Pb-9K; Mon, 01 Sep 2025 22:10:41 +0000 Received: by outflank-mailman (input) for mailman id 1105412; Mon, 01 Sep 2025 22:10:39 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjX-00055o-TK for xen-devel@lists.xenproject.org; Mon, 01 Sep 2025 22:10:39 +0000 Received: from mail-lf1-x12b.google.com (mail-lf1-x12b.google.com [2a00:1450:4864:20::12b]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 7d232868-8780-11f0-8dd7-1b34d833f44b; Tue, 02 Sep 2025 00:10:37 +0200 (CEST) Received: by mail-lf1-x12b.google.com with SMTP id 2adb3069b0e04-55f69cf4b77so3061906e87.2 for ; Mon, 01 Sep 2025 15:10:37 -0700 (PDT) Received: from yp-VivoBook-ASUSLaptop-M1503QA-M1503QA.. ([95.67.15.120]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5608279307asm123038e87.75.2025.09.01.15.10.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 15:10:36 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 7d232868-8780-11f0-8dd7-1b34d833f44b DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756764637; x=1757369437; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7jLnGK/W9IzxOZ6CLKOO5BPor3o3fyjNQvRa1agLv/o=; b=iTP2GJ/agVaLRIl9CbyxtbmKv9vK0Kmfc0vQvD9UJJ+VsWHnwm0C91teejA5069jbj lvFqrgdbERgifdjCWjURa5l9Bu8c9emdO4e8DqxZj/ebimOAlGs4BZfd5yFcFhvquTeq 3WEwZw9jewRdjaDmWlnViXn/8J7ukMTjdNha7uVsH2IPYGmmWmbyfOIJmcaLwd3ZFrBu D/00UtgdFwaF+rNmxT0mDBvPftWN4BWm4PH+wJadGkCpR+z2S3Jk+1j7lpNbf43tAw5D oTMCxYXZacmJUr+LstIXKxlSy7FlwXJczRaazLnvNJdP04Y/B1SLm12N05aetx+kk1Ul fL2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756764637; x=1757369437; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7jLnGK/W9IzxOZ6CLKOO5BPor3o3fyjNQvRa1agLv/o=; b=Kl6XnjXUI5L8r90NrRQ6pL1FYqA9VjslaIcnFvcd+wC8o9gGLZEe/FJ0cHe/za5YHG +O864eobkRAnUpfO1ym45/ZAuwi0/FaYxXtFK+I+mkrztqMa3AL8iAV7VMNt35OSnMPp Ry/YbM7saD1Ns1eW3j4lMLJnxnw9Y8Ji6Tyf33pHCAqlEEz/uOeGV4XvY+KKp6vB3TeR Dd6e6vf1OJtgVUGIVGgmWS7ZMVMuE53iw4O2KTUIzEG/I7B+Iaozo4tYgSIZWVXnE2Vo CcBl3a1d8j4d8hnAvL7/SScYfFGOSYQgaNSsUYBIEhTB3bYgPeJHCtzL1REO22f4WBhb uFhg== X-Gm-Message-State: AOJu0Yx7K6ZRTJAE2/O1XtvM83f05XC1xVpsDmedUEbd0E0wWaorTb3a pGesJ2r/BX7HGZ0n1i6LQdF9Gfkh0mGmuH5wwfycrQKKgN4gZz4FArsxL7nnyiTS X-Gm-Gg: ASbGnct64cGiOEmevkj+8JPF/y1Zve+X5FLAzDZ2VjPvBHf8PxHlAKXg/uoJtNBuHCf Z28SAmUyv5sDYQjdNzuJOaWqnJnSGkfDOgCcPo8QWEBXx/dCdsm9eGMgkZJD9z+OTunwz4Qmv8i Ew1GmlKramjC2vUOe2hMo9n5pCHVhxnMvfaZ6egEettEf48Jt+9I+JaAXyj+fjnMtroKISWA5si WdKWQMznb/eJAs0vV9mYyOez/H6NWAnC2ZWoqD4JvnqenRSwiTXXB1O12camonfEZ3Hiec7n/Y2 MppBAalrq4m39+HD0PlQmDuGPnJKzGU7daSBslhCnvQlglIxr/yZoAyd8gEgV0+D1+oj9xAnvBg NSZJkkmlotKRwVF5qEoOKGv6mQ5n4vNoXeEKcHAw8RX8k6PvDQ0YtVWjkzaLZnQ== X-Google-Smtp-Source: AGHT+IGXdEdoSAzn5JEBY3rqDuehu56DiyGTfp49x/4lqJkOcHbolBJuNKUoq29Yojmc/eQDmAjVSQ== X-Received: by 2002:a05:6512:3c97:b0:55f:6a49:6e71 with SMTP id 2adb3069b0e04-55f708ecdadmr2606498e87.29.1756764636522; Mon, 01 Sep 2025 15:10:36 -0700 (PDT) From: Mykola Kvach To: xen-devel@lists.xenproject.org Cc: Oleksandr Tyshchenko , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , Volodymyr Babchuk , Rahul Singh , Mykola Kvach Subject: [PATCH v6 12/13] xen/arm: Suspend/resume IOMMU on Xen suspend/resume Date: Tue, 2 Sep 2025 01:10:16 +0300 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1756764664679116600 Content-Type: text/plain; charset="utf-8" From: Oleksandr Tyshchenko This is done using generic iommu_suspend/resume functions that cause IOMMU driver specific suspend/resume handlers to be called for enabled IOMMU (if one has suspend/resume driver handlers implemented). Signed-off-by: Oleksandr Tyshchenko Signed-off-by: Mykola Kvach Reviewed-by: Volodymyr Babchuk --- Changes in V6: - Drop iommu_enabled check from host system suspend. --- xen/arch/arm/suspend.c | 11 +++++++++++ xen/drivers/passthrough/arm/smmu-v3.c | 10 ++++++++++ xen/drivers/passthrough/arm/smmu.c | 10 ++++++++++ 3 files changed, 31 insertions(+) diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c index 35b20581f1..f3a3b831c5 100644 --- a/xen/arch/arm/suspend.c +++ b/xen/arch/arm/suspend.c @@ -5,6 +5,7 @@ =20 #include #include +#include #include #include #include @@ -62,6 +63,13 @@ static void cf_check system_suspend(void *data) =20 time_suspend(); =20 + status =3D iommu_suspend(); + if ( status ) + { + system_state =3D SYS_STATE_resume; + goto resume_time; + } + console_start_sync(); status =3D console_suspend(); if ( status ) @@ -118,6 +126,9 @@ static void cf_check system_suspend(void *data) console_resume(); console_end_sync(); =20 + iommu_resume(); + + resume_time: time_resume(); =20 resume_nonboot_cpus: diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthroug= h/arm/smmu-v3.c index 81071f4018..f887faf7dc 100644 --- a/xen/drivers/passthrough/arm/smmu-v3.c +++ b/xen/drivers/passthrough/arm/smmu-v3.c @@ -2854,6 +2854,13 @@ static void arm_smmu_iommu_xen_domain_teardown(struc= t domain *d) xfree(xen_domain); } =20 +#ifdef CONFIG_SYSTEM_SUSPEND +static int arm_smmu_suspend(void) +{ + return -ENOSYS; +} +#endif + static const struct iommu_ops arm_smmu_iommu_ops =3D { .page_sizes =3D PAGE_SIZE_4K, .init =3D arm_smmu_iommu_xen_domain_init, @@ -2866,6 +2873,9 @@ static const struct iommu_ops arm_smmu_iommu_ops =3D { .unmap_page =3D arm_iommu_unmap_page, .dt_xlate =3D arm_smmu_dt_xlate, .add_device =3D arm_smmu_add_device, +#ifdef CONFIG_SYSTEM_SUSPEND + .suspend =3D arm_smmu_suspend, +#endif }; =20 static __init int arm_smmu_dt_init(struct dt_device_node *dev, diff --git a/xen/drivers/passthrough/arm/smmu.c b/xen/drivers/passthrough/a= rm/smmu.c index 22d306d0cb..45f29ef8ec 100644 --- a/xen/drivers/passthrough/arm/smmu.c +++ b/xen/drivers/passthrough/arm/smmu.c @@ -2947,6 +2947,13 @@ static void arm_smmu_iommu_domain_teardown(struct do= main *d) xfree(xen_domain); } =20 +#ifdef CONFIG_SYSTEM_SUSPEND +static int arm_smmu_suspend(void) +{ + return -ENOSYS; +} +#endif + static const struct iommu_ops arm_smmu_iommu_ops =3D { .page_sizes =3D PAGE_SIZE_4K, .init =3D arm_smmu_iommu_domain_init, @@ -2960,6 +2967,9 @@ static const struct iommu_ops arm_smmu_iommu_ops =3D { .map_page =3D arm_iommu_map_page, .unmap_page =3D arm_iommu_unmap_page, .dt_xlate =3D arm_smmu_dt_xlate_generic, +#ifdef CONFIG_SYSTEM_SUSPEND + .suspend =3D arm_smmu_suspend, +#endif }; =20 static struct arm_smmu_device *find_smmu(const struct device *dev) --=20 2.48.1 From nobody Thu Oct 30 22:54:56 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1756764669; cv=none; d=zohomail.com; s=zohoarc; b=NV6kO7PAQOR1mhiT2p9tEnbW5xs3ztEuw2nITYZpU8UGnyvJUnYL09p3QbFT/KVczm9mNdJxb8KVaswflGIaZGSUb7x7LW6kuC8RwwJSkwdbYJv2cyuYWPyHM/KifXZoosfR7k3HJ3h9q4fKrgibGgGH1G3V/Ll4PTzhhjnzZH0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1756764669; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=BjizghnjyTk2+1djq5Y5MywqYNIMRO2uUzS6ohxl4Eg=; b=JcGY+3GqgnrCqMhNe0H/W2uGv+xtkeGAemIHkZA9Nb7nCgaDG63561OY+161EF4CySwyLBtlBpUyEW+GeOFCxHAh4CWu/88nNY2Z4rDhRKnDZAIZnbf2UFgHnIHcxiywY5TY0abuBdWvhH3pF6cZDeQkXu/50YWtW1aP6FoMb80= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1756764669694265.0385248372588; Mon, 1 Sep 2025 15:11:09 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1105417.1456449 (Exim 4.92) (envelope-from ) id 1utCjc-0000Vw-1g; Mon, 01 Sep 2025 22:10:44 +0000 Received: by outflank-mailman (output) from mailman id 1105417.1456449; Mon, 01 Sep 2025 22:10:43 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjb-0000Vb-O0; Mon, 01 Sep 2025 22:10:43 +0000 Received: by outflank-mailman (input) for mailman id 1105417; Mon, 01 Sep 2025 22:10:42 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1utCjZ-00055o-Tq for xen-devel@lists.xenproject.org; Mon, 01 Sep 2025 22:10:41 +0000 Received: from mail-lf1-x12c.google.com (mail-lf1-x12c.google.com [2a00:1450:4864:20::12c]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 7ddf5df5-8780-11f0-8dd7-1b34d833f44b; Tue, 02 Sep 2025 00:10:38 +0200 (CEST) Received: by mail-lf1-x12c.google.com with SMTP id 2adb3069b0e04-55f687fd3bdso4496391e87.1 for ; Mon, 01 Sep 2025 15:10:38 -0700 (PDT) Received: from yp-VivoBook-ASUSLaptop-M1503QA-M1503QA.. ([95.67.15.120]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5608279307asm123038e87.75.2025.09.01.15.10.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 15:10:37 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 7ddf5df5-8780-11f0-8dd7-1b34d833f44b DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756764638; x=1757369438; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BjizghnjyTk2+1djq5Y5MywqYNIMRO2uUzS6ohxl4Eg=; b=iMnv7d1krAOipiWYOMc5/CdfPYAp+xruycgxo6Z5turoVv+X/bUoVJIWVqqbz5r2MQ YfQyIYzY1Rl0/sszIviPZcFJcrAC7OGzDFnfsUaTft4iwi7iis5F1HRT2wd19cyEbH6i oIXwSMuTY7SF3EbSiN2nOhVX6zJdlSX9cw1OYoKBf5rxvvzy17fwAxaKmJNADvuHVaF4 M636tMvq4Qayln8HU5aNVQPFqBFy6a6cL837OLOpTK6Pwl3EAzslH3JELnnk6E9ZOMCp bmTUKJmuu5lIvwRxqBmKaUSrWIAo5G+dynzmPIjgGPfc/UCm6itQl5jmNT6E5ZRSSTop wegw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756764638; x=1757369438; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BjizghnjyTk2+1djq5Y5MywqYNIMRO2uUzS6ohxl4Eg=; b=XEyVF/3UMcWRFni44Mh81eM3xlSIVA3AKZyLVZPq5G0SWhb6bss9aIxcppuUCIa6cC NVoRZV9XA+Uwcr+hKezaV/nkWx/kDMX8Stnztz3JSFooCQGt95Sus8ADegMstQ0SWncB EItTwiZtNr/ouNLa42MctZqLLb3+rQKaR4uMvSn3ytKUf1fOufYVYPnHabSvLHOaEmmh wf3I4NOmGM5kozKgkNOydpqMSBJzxJ8aAvdhutVI86BeAX2hXje4H6p7QZDke+UJpcJv 5ivV06Z5lSTOMLY72YwVHNQRRraZKsZ6EUFn9m81Fv1jxvyNTspc2g0tpH+MvS9PO3c5 /6Sg== X-Gm-Message-State: AOJu0YxMHrmcfuWNUsg0Hi8iEUp//1fgY/h0n9sjJy6f3mYrjoPGQX1+ v8tQyb8GK06XCDmn8zYLJj21pgM2OckYfBmZDJRHcpVw/IY3lu8he2xcEM3dE6nU X-Gm-Gg: ASbGncvxONl76xFllWIjApp9rBvonkZK+TvkMxoOnrlvNYPSMV8Qqlx0iThIJcifI3V FMyNPTvos0sEUrmTuLitdq6bfFLZxm5DGKC1ZYGAmqt2rRhgAUCFTshj/qnyM48OzkyCqa6tN8u qaHs43LLRqT+yMx2HqPoS8cLyCrZR34t0g/1ahVYV4CR0VUKHuZzkIvBtAb+byqL3hZ6hU2xvXo UxLKM1672aTdRQGzDwwGEt1gUAmVAhqzC/Fb/0DZ6D+r6dZFX/dbpyTreMATPXPI8/VHpQT/ksZ JCux90fO8hvM3qfR0StG/RCN0A3q7qQR2qmtn7+XZ2xnR8Y5lQzn44RqOKB45uQpVCl3eFvWg6z Pv22SE1Tf07DlvmWnNQ5/0pu5d9fffPhz4Hhg8nktmsxzOVmuIt4m2VMSyXiuLw== X-Google-Smtp-Source: AGHT+IEn9+xcjXygXejhEa0wyNG2hkVDUF3pRbujhPECoHvmyQVKBSuBe7768CCtbszL4fNE5HZPEA== X-Received: by 2002:a05:6512:3ca4:b0:55f:501e:7bf7 with SMTP id 2adb3069b0e04-55f709bdaa0mr2975906e87.57.1756764637835; Mon, 01 Sep 2025 15:10:37 -0700 (PDT) From: Mykola Kvach To: xen-devel@lists.xenproject.org Cc: Mykola Kvach , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , Volodymyr Babchuk Subject: [PATCH v6 13/13] xen/arm: gic-v3: Add suspend/resume support for eSPI registers Date: Tue, 2 Sep 2025 01:10:17 +0300 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1756764671280124100 Content-Type: text/plain; charset="utf-8" From: Mykola Kvach Add suspend/resume handling for GICv3 eSPI registers. Signed-off-by: Mykola Kvach --- Note: The main eSPI patch series is still under review. This commit is intended to be applied after the main eSPI series: [PATCH v5 00/12] Introduce eSPI support https://patchew.org/Xen/cover.1756481577.git.leonid._5Fkomarianskyi@epam.co= m/ --- xen/arch/arm/gic-v3.c | 141 +++++++++++++++++++++++++++++------------- 1 file changed, 97 insertions(+), 44 deletions(-) diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c index 9f1be7e905..57403c82a8 100644 --- a/xen/arch/arm/gic-v3.c +++ b/xen/arch/arm/gic-v3.c @@ -1782,17 +1782,14 @@ static bool gic_dist_supports_lpis(void) struct gicv3_ctx { struct dist_ctx { uint32_t ctlr; - /* - * This struct represent block of 32 IRQs - * TODO: store extended SPI configuration (GICv3.1+) - */ + /* This struct represent block of 32 IRQs */ struct irq_regs { uint32_t icfgr[2]; uint32_t ipriorityr[8]; uint64_t irouter[32]; uint32_t isactiver; uint32_t isenabler; - } *irqs; + } *irqs, *espi_irqs; } dist; =20 /* have only one rdist structure for last running CPU during suspend */ @@ -1831,8 +1828,26 @@ static void __init gicv3_alloc_context(void) gicv3_ctx.dist.irqs =3D xzalloc_array(typeof(*gicv3_ctx.dist.irqs), blocks - 1); if ( !gicv3_ctx.dist.irqs ) + { printk(XENLOG_ERR "Failed to allocate memory for GICv3 suspend= context\n"); + return; + } } + +#ifdef CONFIG_GICV3_ESPI + if ( !gicv3_info.nr_espi ) + return; + + gicv3_ctx.dist.espi_irqs =3D xzalloc_array(typeof(*gicv3_ctx.dist.espi= _irqs), + gicv3_info.nr_espi / 32); + if ( !gicv3_ctx.dist.espi_irqs ) + { + xfree(gicv3_ctx.dist.irqs); + gicv3_ctx.dist.irqs =3D NULL; + + printk(XENLOG_ERR "Failed to allocate memory for GICv3 eSPI suspen= d context\n"); + } +#endif } =20 static void gicv3_disable_redist(void) @@ -1852,6 +1867,65 @@ static void gicv3_disable_redist(void) while ( (readl_relaxed(waker) & GICR_WAKER_ChildrenAsleep) =3D=3D 0 ); } =20 +#define GET_SPI_REG_OFFSET(name, is_espi) \ + ((is_espi) ? GICD_##name##nE : GICD_##name) + +static void gicv3_store_spi_irq_block(typeof(gicv3_ctx.dist.irqs) irqs, + unsigned int i, bool is_espi) +{ + void __iomem *base; + unsigned int irq; + + base =3D GICD + GET_SPI_REG_OFFSET(ICFGR, is_espi) + 8 * i; + irqs->icfgr[0] =3D readl_relaxed(base); + irqs->icfgr[1] =3D readl_relaxed(base + 4); + + base =3D GICD + GET_SPI_REG_OFFSET(IPRIORITYR, is_espi) + 32 * i; + for ( irq =3D 0; irq < 8; irq++ ) + irqs->ipriorityr[irq] =3D readl_relaxed(base + 4 * irq); + + base =3D GICD + GET_SPI_REG_OFFSET(IROUTER, is_espi) + 32 * i; + for ( irq =3D 0; irq < 32; irq++ ) + irqs->irouter[irq] =3D readq_relaxed_non_atomic(base + 8 * irq); + + base =3D GICD + GET_SPI_REG_OFFSET(ISACTIVER, is_espi) + 4 * i; + irqs->isactiver =3D readl_relaxed(base); + + base =3D GICD + GET_SPI_REG_OFFSET(ISENABLER, is_espi) + 4 * i; + irqs->isenabler =3D readl_relaxed(base); +} + +static void gicv3_restore_spi_irq_block(typeof(gicv3_ctx.dist.irqs) irqs, + unsigned int i, bool is_espi) +{ + void __iomem *base; + unsigned int irq; + + base =3D GICD + GET_SPI_REG_OFFSET(ICFGR, is_espi) + 8 * i; + writel_relaxed(irqs->icfgr[0], base); + writel_relaxed(irqs->icfgr[1], base + 4); + + base =3D GICD + GET_SPI_REG_OFFSET(IPRIORITYR, is_espi) + 32 * i; + for ( irq =3D 0; irq < 8; irq++ ) + writel_relaxed(irqs->ipriorityr[irq], base + 4 * irq); + + base =3D GICD + GET_SPI_REG_OFFSET(IROUTER, is_espi) + 32 * i; + for ( irq =3D 0; irq < 32; irq++ ) + writeq_relaxed_non_atomic(irqs->irouter[irq], base + 8 * irq); + + base =3D GICD + GET_SPI_REG_OFFSET(ICENABLER, is_espi) + i * 4; + writel_relaxed(GENMASK(31, 0), base); + + base =3D GICD + GET_SPI_REG_OFFSET(ISENABLER, is_espi) + i * 4; + writel_relaxed(irqs->isenabler, base); + + base =3D GICD + GET_SPI_REG_OFFSET(ICACTIVER, is_espi) + i * 4; + writel_relaxed(GENMASK(31, 0), base); + + base =3D GICD + GET_SPI_REG_OFFSET(ISACTIVER, is_espi) + i * 4; + writel_relaxed(irqs->isactiver, base); +} + static int gicv3_suspend(void) { unsigned int i; @@ -1871,6 +1945,14 @@ static int gicv3_suspend(void) return -ENOMEM; } =20 +#ifdef CONFIG_GICV3_ESPI + if ( gicv3_info.nr_espi && !gicv3_ctx.dist.espi_irqs ) + { + printk(XENLOG_ERR "GICv3: eSPI suspend context is not allocated!\n= "); + return -ENOMEM; + } +#endif + /* Save GICC configuration */ gicv3_ctx.cpu.ctlr =3D READ_SYSREG(ICC_CTLR_EL1); gicv3_ctx.cpu.pmr =3D READ_SYSREG(ICC_PMR_EL1); @@ -1903,25 +1985,12 @@ static int gicv3_suspend(void) gicv3_ctx.dist.ctlr =3D readl_relaxed(GICD + GICD_CTLR); =20 for ( i =3D 1; i < DIV_ROUND_UP(gicv3_info.nr_lines, 32); i++ ) - { - typeof(gicv3_ctx.dist.irqs) irqs =3D gicv3_ctx.dist.irqs + i - 1; - unsigned int irq; + gicv3_store_spi_irq_block(gicv3_ctx.dist.irqs + i - 1, i, false); =20 - base =3D GICD + GICD_ICFGR + 8 * i; - irqs->icfgr[0] =3D readl_relaxed(base); - irqs->icfgr[1] =3D readl_relaxed(base + 4); - - base =3D GICD + GICD_IPRIORITYR + 32 * i; - for ( irq =3D 0; irq < 8; irq++ ) - irqs->ipriorityr[irq] =3D readl_relaxed(base + 4 * irq); - - base =3D GICD + GICD_IROUTER + 32 * i; - for ( irq =3D 0; irq < 32; irq++ ) - irqs->irouter[irq] =3D readq_relaxed_non_atomic(base + 8 * irq= ); - - irqs->isactiver =3D readl_relaxed(GICD + GICD_ISACTIVER + 4 * i); - irqs->isenabler =3D readl_relaxed(GICD + GICD_ISENABLER + 4 * i); - } +#ifdef CONFIG_GICV3_ESPI + for ( i =3D 0; i < gicv3_info.nr_espi / 32; i++ ) + gicv3_store_spi_irq_block(gicv3_ctx.dist.espi_irqs + i, i, true); +#endif =20 return 0; } @@ -1938,28 +2007,12 @@ static void gicv3_resume(void) writel_relaxed(GENMASK(31, 0), GICD + GICD_IGROUPR + (i / 32) * 4); =20 for ( i =3D 1; i < DIV_ROUND_UP(gicv3_info.nr_lines, 32); i++ ) - { - typeof(gicv3_ctx.dist.irqs) irqs =3D gicv3_ctx.dist.irqs + i - 1; - unsigned int irq; + gicv3_restore_spi_irq_block(gicv3_ctx.dist.irqs + i - 1, i, false); =20 - base =3D GICD + GICD_ICFGR + 8 * i; - writel_relaxed(irqs->icfgr[0], base); - writel_relaxed(irqs->icfgr[1], base + 4); - - base =3D GICD + GICD_IPRIORITYR + 32 * i; - for ( irq =3D 0; irq < 8; irq++ ) - writel_relaxed(irqs->ipriorityr[irq], base + 4 * irq); - - base =3D GICD + GICD_IROUTER + 32 * i; - for ( irq =3D 0; irq < 32; irq++ ) - writeq_relaxed_non_atomic(irqs->irouter[irq], base + 8 * irq); - - writel_relaxed(GENMASK(31, 0), GICD + GICD_ICENABLER + i * 4); - writel_relaxed(irqs->isenabler, GICD + GICD_ISENABLER + i * 4); - - writel_relaxed(GENMASK(31, 0), GICD + GICD_ICACTIVER + i * 4); - writel_relaxed(irqs->isactiver, GICD + GICD_ISACTIVER + i * 4); - } +#ifdef CONFIG_GICV3_ESPI + for ( i =3D 0; i < gicv3_info.nr_espi / 32; i++ ) + gicv3_restore_spi_irq_block(gicv3_ctx.dist.espi_irqs + i, i, true); +#endif =20 writel_relaxed(gicv3_ctx.dist.ctlr, GICD + GICD_CTLR); gicv3_dist_wait_for_rwp(); --=20 2.48.1