From nobody Tue Sep 29 09:09:02 2026 Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E5B913B47D9 for ; Mon, 10 Aug 2026 10:19:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786357172; cv=none; b=cUhQXGkkY1j1o02t8Cn3Iy8BBQ5WgfRJsgXUN/GBNHPyUQVferAYhAuos9gSCz9sLHvK3pF6q47I0dB0D8aM60ZXSMkmoALC2GAfXHDHicJfCfzuy2srPgLsV9wJwV8oG5yt8zdUFuWeyrbOxLtmsoFcoC+T6Z5IXNBzQDEPyTQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786357172; c=relaxed/simple; bh=/VqRg1FiiWK5ygK/J2KLrf5GhjMkP/GTVRwzNgg/XzA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:To:Cc; b=k9C+ooF/lEe2evpZVzM08/vDYKamz1OvQGVx9ctfGetDUbtu0XPp/k39Be8epn6MufATwfP0aR3o2EDiiczC9Ze+V+JGbaKcn9g4Q3wWDmOW8wIwKni7olh9tM7DwOlKihSY1cg5e9TDtMhKwfH/wYUQEk7cx7/gTUQ5bKgS73U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=DusIShwu; arc=none smtp.client-ip=209.85.215.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="DusIShwu" Received: by mail-pg1-f176.google.com with SMTP id 41be03b00d2f7-c981c2c37cbso1177259a12.0 for ; Mon, 10 Aug 2026 03:19:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1786357170; x=1786961970; darn=vger.kernel.org; h=cc:to:message-id:content-transfer-encoding:content-type :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to:content-type; bh=ThHcrvznI2+QSAI5cW+DWFGODRhB/qhrAmcxC4O4jLs=; b=DusIShwuxKZWozYxV5jAV2zv1vdSxnfq+dPE3zi5Puc/jbaRc/ds4UYuzCqDk0uvKx 9lJ23XXiUh9pxHxKZ6g88iwMwp+FLFp51RZNJyTEmEHoJyVSd6BgTLhkR4TsGYzH0Y87 RPiGdS4aZVCqSHbFAGmJdfIYTwu++nGbMtGjGnrj9qp19j3q3C6pLVvlAxH7eCe585WB KBx/dZdRyNZwQ/92asFHosU3JeZnA/xVElZRl+gN4dAV6UKcWoLnK3mvWUuDNZfV6iKF iwh1m7YFXaPe8A+qu6tYzeEJRhL9xfCtZSLuKkPG13W4ks1b/iHqB5QLNgRAAhOjvey2 bB+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786357170; x=1786961970; h=cc:to:message-id:content-transfer-encoding:content-type :mime-version:subject:date:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=ThHcrvznI2+QSAI5cW+DWFGODRhB/qhrAmcxC4O4jLs=; b=py31n/sQI5A9+6qX2PV2jE+9hZvWZkzR9SQQn2J6+542slkvZV6RROljUQKu/hsF35 uanLUqog3g05h8Itv2WCmrnHaEUy/j1yzYYLUvCQlOVhlm1Qso6E/LqrwpvF8/LX9ZFS gkc9UYBmsz0iC5t0x7JDWoPZpyTaneOa8pQyldNoWWW5m7ZuLdnqnBYEfSxzvRAD9uWb o5JHauj68b6WxdmyulB91YcqOz1fLd1GfUnl/VxNWAGH0LpEnwLAagcM1cnGNcyVBV4D Cepz9AddXMLoBAVaF+DnAFEo+RYp9TKfnGM3IE9RynB8m2W0fEzFh46a8VMR2ajOyxx/ aPfA== X-Forwarded-Encrypted: i=1; AHgh+RrqJPzsBoq+v7IRcgcUVJ5V6j9h55Rb90hrRUOx9KUmJmyUXq8oqW6xvIHey5XDho+vWyRVEwTbpBQCv9g=@vger.kernel.org X-Gm-Message-State: AOJu0YzAHNYwOVQz7wROnD/vyFOWS2/NszEZWN5V6kASyryAv/sYAng+ XFlsGrytNam+LlzZHsV0O98cgrebmt2ayz+G1GjmXoDNqstF5VQannpgn6G01jYZ1CU= X-Gm-Gg: AR+sD11AAR7qA1g3fIaIso42QGx5FvfO/YpMssCetzosMGcxfR53XbQk1ULd1EKEXuQ HwwLPnya66uGnim71X6IIsakdUMWuQztb4LWbuX6AvbtO8LgmLmms2gow3Kdkte2pCQcQlk5WzX l/XxyzC6OYuwgLENni8NzIO1cIecBWmpbB7GFbAH/mQn+Bs1dlzPjAkqTJufsqQSYejlvNaMeW1 s+M62lTHBNyeZcapWlDw9mSsvter0MnxlFd67to5VWrRZlrecvGToFtW8HcTV6NsYkeF4BFbf1L l3le8SMKHFrtKuccFO3hNHZS7RhFyFKFrR85nWfCr00iskwgc4jPGQN/T5nQXCKCxDKB6ONmbBE hbDmqGVMzJOkCPzIYxHqnCZxeuJF9U6prsa6KsLHx2WJBNJU1HNNvGnh4c9dZVKGgIpE/PH9/Te QIxqEvs5wH3CGAzO8Dn/NTfxMf8wcgVReYQjFD/F4nP9rVYu+6ZuK5KiOC2Dmg4eZX9ZBjSoPNI LbM8+v5tBk+NU4jqs4= X-Received: by 2002:a05:6a20:cf8d:b0:3c3:b5db:cd07 with SMTP id adf61e73a8af0-3cb85e61a8emr53401373637.15.1786357170215; Mon, 10 Aug 2026 03:19:30 -0700 (PDT) Received: from sw07.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-315be86fc7bsm41861803eec.1.2026.08.10.03.19.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Aug 2026 03:19:29 -0700 (PDT) From: Yong-Xuan Wang Date: Mon, 10 Aug 2026 03:18:50 -0700 Subject: [PATCH v5] KVM: RISC-V: Add CPU PM notifier for non-retention idle states Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260810-kvm-cpu-pm-v5-1-6c9e4c95caf5@sifive.com> X-B4-Tracking: v=1; b=H4sIAImleWoC/3XNTW7CMBCG4asgr5nKM4zzw4p7oC5sZ1xcFBLFY LVCuXsdumgj2uUnzfPOXSWZoiS139zVJDmmOFzKMNuN8id7eROIXdmKNFW6IoZz7sGPNxh7aBk ZrfjOBKsKGCcJ8eMRO75+73Rz7+KvS2G5OMV0HabPx7eMy92f4YyAYEg3jSDV1phDiiFmefFDr 5Zypt+2WlkCDVw3YiQ0nW/dk939b3fFOtEGrQ3SOvtk+cfWhCvLxSJXLrRMhutuZed5/gK0Urq mawEAAA== X-Change-ID: 20260624-kvm-cpu-pm-94141aecd5fa To: greentime.hu@sifive.com, vincent.chen@sifive.com, zong.li@sifive.com Cc: Yong-Xuan Wang , Anup Patel , Atish Patra , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexandre Ghiti , =?utf-8?q?Radim_Kr=C4=8Dm=C3=A1=C5=99?= , Andrew Jones , Nutty Liu , Jinyu Tang , kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1786357169; l=9484; i=yongxuan.wang@sifive.com; s=20260424; h=from:subject:message-id; bh=/VqRg1FiiWK5ygK/J2KLrf5GhjMkP/GTVRwzNgg/XzA=; b=iAYM+gAFzpYb4R/Mq3dv0mA5Ov5+pwu7A2n2aOT5kR1LiY/G6elW/hWelTbdQUKOXFfzd6hXG /hKKgxqd6X0C1UVKkhtWIO60/zJ8pJCYMpicxju11FH2TJeM9H98+kM X-Developer-Key: i=yongxuan.wang@sifive.com; a=ed25519; pk=+8NCHB1ZJvZthQAmZspOAaqjo+/snaW8mFSiDx45HxY= Register a CPU_PM notifier to restore hypervisor CSR content during CPU non-retention idle states. When a CPU enters a deep idle state that powers off the CPU domain, hypervisor CSRs and VS CSRs lose their state and must be saved before entry and restored after exit. This completes KVM's power management coverage for RISC-V: - CPU hotplug: handled by kvm_online_cpu/kvm_offline_cpu (cpuhp callbacks) - System suspend: handled by kvm_suspend/kvm_resume (syscore ops) - CPU idle (retention): no action needed, CSRs are retained - CPU idle (non-retention): handled by this CPU_PM notifier Signed-off-by: Yong-Xuan Wang Reviewed-by: Anup Patel --- Changes in v5: - Remove patch1 as it was merged. - Add per-CPU virtualization state tracking (sashiko) - Rename CSR helper functions (Anup) - Introduce dedicated PM functions for AIA (Anup) - Fix module exit sequence (Anup) - Link to v4: https://patch.msgid.link/20260721-kvm-cpu-pm-v4-0-146bf942547= d@sifive.com Changes in v4: - Remove the system_state checking (sashiko) - Link to v3: https://patch.msgid.link/20260626-kvm-cpu-pm-v3-0-be051aafe9b= a@sifive.com Changes in v3: - Remove the error kvm_riscv_nacl_disable() call in kvm_riscv_csr_disable() (sashiko) - Unregister CPU PM notifier first in riscv_kvm_exit() (sashiko) - Link to v2: https://patch.msgid.link/20260626-kvm-cpu-pm-v2-0-478e5ef8dc9= b@sifive.com Changes in v2: - Add patch 1 force restore VCPU after power state transitions (sashiko) - Added CPU_PM_ENTER_FAILED event handling (sashiko) - Optimized HGEIE save/restore with saved_value field (sashiko) - Fixed HIE CSR restoration in lightweight idle resume path (sashiko) - Link to v1: https://patch.msgid.link/20260624-kvm-cpu-pm-v1-1-52088e127a5= 5@sifive.com --- To: Anup Patel To: Atish Patra To: Paul Walmsley To: Palmer Dabbelt To: Albert Ou To: Alexandre Ghiti To: Radim Kr=C4=8Dm=C3=A1=C5=99 To: Andrew Jones To: Nutty Liu To: Jinyu Tang Cc: greentime.hu@sifive.com Cc: vincent.chen@sifive.com Cc: zong.li@sifive.com Cc: kvm@vger.kernel.org Cc: kvm-riscv@lists.infradead.org Cc: linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org --- arch/riscv/include/asm/kvm_aia.h | 3 ++ arch/riscv/kvm/aia.c | 42 ++++++++++++++++ arch/riscv/kvm/main.c | 106 ++++++++++++++++++++++++++++++++---= ---- 3 files changed, 133 insertions(+), 18 deletions(-) diff --git a/arch/riscv/include/asm/kvm_aia.h b/arch/riscv/include/asm/kvm_= aia.h index c67ec5ac0a14..ab03f853e507 100644 --- a/arch/riscv/include/asm/kvm_aia.h +++ b/arch/riscv/include/asm/kvm_aia.h @@ -165,6 +165,9 @@ int kvm_riscv_aia_alloc_hgei(int cpu, struct kvm_vcpu *= owner, void __iomem **hgei_va, phys_addr_t *hgei_pa); void kvm_riscv_aia_free_hgei(int cpu, int hgei); =20 +void kvm_riscv_aia_pm_exit(void); +void kvm_riscv_aia_pm_enter(void); + void kvm_riscv_aia_enable(void); void kvm_riscv_aia_disable(void); int kvm_riscv_aia_init(void); diff --git a/arch/riscv/kvm/aia.c b/arch/riscv/kvm/aia.c index 9a653b4ad40a..0de9b8852ed2 100644 --- a/arch/riscv/kvm/aia.c +++ b/arch/riscv/kvm/aia.c @@ -25,6 +25,7 @@ struct aia_hgei_control { unsigned long free_bitmap; struct kvm_vcpu *owners[BITS_PER_LONG]; unsigned int nr_hgei; + unsigned long saved_value; }; static DEFINE_PER_CPU(struct aia_hgei_control, aia_hgei); static int hgei_parent_irq; @@ -553,6 +554,47 @@ static void aia_hgei_exit(void) free_percpu_irq(hgei_parent_irq, &aia_hgei); } =20 +void kvm_riscv_aia_pm_exit(void) +{ + struct aia_hgei_control *hgctrl; + + if (!kvm_riscv_aia_available()) + return; + + hgctrl =3D this_cpu_ptr(&aia_hgei); + csr_write(CSR_HGEIE, hgctrl->saved_value); + + csr_write(CSR_HVICTL, aia_hvictl_value(false)); + csr_write(CSR_HVIPRIO1, 0x0); + csr_write(CSR_HVIPRIO2, 0x0); +#ifdef CONFIG_32BIT + csr_write(CSR_HVIPH, 0x0); + csr_write(CSR_HIDELEGH, 0x0); + csr_write(CSR_HVIPRIO1H, 0x0); + csr_write(CSR_HVIPRIO2H, 0x0); +#endif + csr_set(CSR_HIE, BIT(IRQ_S_GEXT)); + /* Enable IRQ filtering for overflow interrupt only if sscofpmf is presen= t */ + if (__riscv_isa_extension_available(NULL, RISCV_ISA_EXT_SSCOFPMF)) + csr_set(CSR_HVIEN, BIT(IRQ_PMU_OVF)); +} + +void kvm_riscv_aia_pm_enter(void) +{ + struct aia_hgei_control *hgctrl; + + if (!kvm_riscv_aia_available()) + return; + + if (__riscv_isa_extension_available(NULL, RISCV_ISA_EXT_SSCOFPMF)) + csr_clear(CSR_HVIEN, BIT(IRQ_PMU_OVF)); + + csr_write(CSR_HVICTL, aia_hvictl_value(false)); + + hgctrl =3D this_cpu_ptr(&aia_hgei); + hgctrl->saved_value =3D csr_read(CSR_HGEIE); +} + void kvm_riscv_aia_enable(void) { const struct imsic_global_config *gc; diff --git a/arch/riscv/kvm/main.c b/arch/riscv/kvm/main.c index 350e4f097d6e..20b516e84e60 100644 --- a/arch/riscv/kvm/main.c +++ b/arch/riscv/kvm/main.c @@ -10,11 +10,14 @@ #include #include #include +#include #include #include #include #include =20 +static DEFINE_PER_CPU(bool, kvm_riscv_virtualization_enabled); + DEFINE_STATIC_KEY_FALSE(kvm_riscv_vsstage_tlb_no_gpa); =20 static void kvm_riscv_setup_vendor_features(void) @@ -33,14 +36,9 @@ long kvm_arch_dev_ioctl(struct file *filp, return -EINVAL; } =20 -int kvm_arch_enable_virtualization_cpu(void) +/* Initialize hypervisor CSRs - called during CPU online and non-retention= idle resume */ +static void kvm_riscv_csr_init(void) { - int rc; - - rc =3D kvm_riscv_nacl_enable(); - if (rc) - return rc; - csr_write(CSR_HEDELEG, 0); csr_write(CSR_HIDELEG, 0); =20 @@ -48,16 +46,11 @@ int kvm_arch_enable_virtualization_cpu(void) csr_write(CSR_HCOUNTEREN, 0x02); =20 csr_write(CSR_HVIP, 0); - - kvm_riscv_aia_enable(); - - return 0; } =20 -void kvm_arch_disable_virtualization_cpu(void) +/* Clear hypervisor CSRs - called during CPU offline and non-retention idl= e entry */ +static void kvm_riscv_csr_cleanup(void) { - kvm_riscv_aia_disable(); - /* * After clearing the hideleg CSR, the host kernel will receive * spurious interrupts if hvip CSR has pending interrupts and the @@ -70,10 +63,69 @@ void kvm_arch_disable_virtualization_cpu(void) csr_write(CSR_HIDELEG, 0); =20 kvm_riscv_clear_former_vcpu(); +} + +int kvm_arch_enable_virtualization_cpu(void) +{ + int rc; + + rc =3D kvm_riscv_nacl_enable(); + if (rc) + return rc; + + kvm_riscv_csr_init(); + kvm_riscv_aia_enable(); + + __this_cpu_write(kvm_riscv_virtualization_enabled, true); =20 + return 0; +} + +void kvm_arch_disable_virtualization_cpu(void) +{ + kvm_riscv_aia_disable(); + kvm_riscv_csr_cleanup(); kvm_riscv_nacl_disable(); + + __this_cpu_write(kvm_riscv_virtualization_enabled, false); +} + +static int kvm_riscv_cpu_pm_notifier(struct notifier_block *self, unsigned= long cmd, void *v) +{ + switch (cmd) { + case CPU_PM_EXIT: + case CPU_PM_ENTER_FAILED: + /* + * Only restore hypervisor state if KVM virtualization is + * enabled on this CPU. This prevents unintentional re-enabling + * of virtualization after it has been explicitly disabled. + */ + if (__this_cpu_read(kvm_riscv_virtualization_enabled)) { + kvm_riscv_csr_init(); + kvm_riscv_aia_pm_exit(); + } + return NOTIFY_OK; + case CPU_PM_ENTER: + /* + * Only save and clear hypervisor state if KVM virtualization + * is enabled on this CPU. + */ + if (__this_cpu_read(kvm_riscv_virtualization_enabled)) { + kvm_riscv_aia_pm_enter(); + kvm_riscv_csr_cleanup(); + } + return NOTIFY_OK; + default: + break; + } + + return NOTIFY_DONE; } =20 +static struct notifier_block kvm_riscv_cpu_pm_nb =3D { + .notifier_call =3D kvm_riscv_cpu_pm_notifier, +}; + static void kvm_riscv_teardown(void) { kvm_riscv_aia_exit(); @@ -174,17 +226,31 @@ static int __init riscv_kvm_init(void) =20 kvm_register_perf_callbacks(); =20 - rc =3D kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE); - if (rc) { - kvm_riscv_teardown(); - return rc; + /* Register CPU PM notifier for CPU idle non-retention states */ + if (IS_ENABLED(CONFIG_CPU_PM)) { + rc =3D cpu_pm_register_notifier(&kvm_riscv_cpu_pm_nb); + if (rc) { + kvm_err("Failed to register CPU PM notifier: %d\n", rc); + goto err_teardown; + } } =20 + rc =3D kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE); + if (rc) + goto err_unregister_cpu_pm; + if (kvm_riscv_aia_available()) kvm_info("AIA available with %d guest external interrupts\n", atomic_read(&kvm_riscv_aia_nr_hgei)); =20 return 0; + +err_unregister_cpu_pm: + if (IS_ENABLED(CONFIG_CPU_PM)) + cpu_pm_unregister_notifier(&kvm_riscv_cpu_pm_nb); +err_teardown: + kvm_riscv_teardown(); + return rc; } module_init(riscv_kvm_init); =20 @@ -192,6 +258,10 @@ static void __exit riscv_kvm_exit(void) { kvm_exit(); =20 + /* Unregister CPU PM notifier */ + if (IS_ENABLED(CONFIG_CPU_PM)) + cpu_pm_unregister_notifier(&kvm_riscv_cpu_pm_nb); + kvm_riscv_teardown(); } module_exit(riscv_kvm_exit); --- base-commit: dfdf1374fdeccb5b7e3d35186228e01ec5ea5f01 change-id: 20260624-kvm-cpu-pm-94141aecd5fa