From nobody Fri Sep 25 21:05:34 2026 Received: from mail-pj2-f13.google.com (mail-pj2-f13.google.com [74.125.227.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 618FF489FBA for ; Mon, 21 Sep 2026 11:14:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.141 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789989262; cv=none; b=cNZn2Yh4SN84tE8sybXbW/bffifBH0HNP/6OElbiazbZORLZ53QxLffKRUbL1r/jBuzG2FhbOxU/3zZUhOXNC7B59Uov1dR8T3jHf9p97nBqjtrza03eSf4GrZiw2FYXELYaI5wTivyklYz/98NEa+tfIR1pl+cMVAxYZjvE4k8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789989262; c=relaxed/simple; bh=7P7BJ6r23yYgS++1l6kZcvA8y1BN9I+co7T6D4BrvHc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mWyr7M1uW7sq0xJmf0PdSxqcSwDvwiy86EJVTVXV7w+iLde7a6lOS3U++xikBqW5mtd/WdxZ4/qdFNbP6JlEaZ7phplKsdvJNCYkDB3owLPMoqlzUgVzfns3+3sW/p6cyqrxtGwF0MDb8FkK+IN4DUdSvQkoPQFdBsVQ9uRWBN0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=d/9P10Se; arc=none smtp.client-ip=74.125.227.141 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="d/9P10Se" Received: by mail-pj2-f13.google.com with SMTP id d9443c01a7336-2d747ed6d6eso27472205ad.2 for ; Mon, 21 Sep 2026 04:14:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789989260; x=1790594060; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=01ra1AO1mSvOrLFVgo6C3hu4ILWJ3jo+KfsYZ5db6+s=; b=d/9P10Se2nOuZrbxgw2stTg3T1B0DVkDqRQsVULRZs+Vkh5kA2iAFXLWaXe25wqYsr Sbp31rWip/KTyGfnHpoKsldelmiG8lfxSVvudPl2NhkwmqrQZ3Nkp1Zm0y/eeEvwD+mF Ow81zSbc008prfcbabE+HeMpcEvwxS0P0EYnHGIuOc0nedJOpxjk/eNdS3NsCYNegm9z kIVLgt8nbWuRbndxDMQ0pxCjgiyuEu7DaV9raJwM48VBt1CavPFa1/GD182G1LMR2naq uH7drKSiZ0hL2U56RMkaiWAMo975sR9nQMzZ6VUrg1bEhdY6OxvvQn7zY6Y/VRgEaDAd Dr9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789989261; x=1790594061; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=01ra1AO1mSvOrLFVgo6C3hu4ILWJ3jo+KfsYZ5db6+s=; b=Y4LdHtH7jYmnoMZWmANNqQtFJ9AESFoaZmSVeP+zuuyX5B0DRqEx9JohC6UuDgiVvk ibZd5/UXz3lK/3c8BjEd2lf6/l6beVNLO2gHylvXCUgwSLQDvC7CFT07BX3+9/JyaeGi 4NtstokkKXINmuwGQwKSdUT3rRJg2SwvFLkGan111gxBk7We1RgqD9HbKeVIttkn1mGt iq73G7/6NtGlwyDR2xzy1RBTKz979b64Ap8gkeDP2at56JYf0B00Z1x+RbHcFRZHvewp 9bBZrnlcI/ClyTzJasUNgGMDjMDsrfkZ04EO+b5JWLjfhHo6gMHTQZWIMJukKCia0JAd 9exw== X-Forwarded-Encrypted: i=1; AKwUvBxzciOmKJp/e4m17Hv0pckxAvWlEK1Gl9yQHc/d1HQgqAT87ESV35Hnuiiw6iI09JbnI+fITxOr15RwROY=@vger.kernel.org X-Gm-Message-State: AFuF++mvhJ2o2ZnY4D4ZnwHDcTs8Ev8X0zWRYdw0xWGKgkWVyL0rxB6c UIcdKLMjb+UWIFat+DiJg7UA10Nvc0+39GYiV/Ui5UHXzhnTyPco0IFQ X-Gm-Gg: AYBFou2pVBEpNCjzIV11NxCfMguz6sWM/ad+7pDJCR+Gf3H8aSg5wHztI+PbhkTe1Hd WpoPA/yWw/we7dBoJLutwgIEHxLEp/psB//layBPoPJOA4HQNGX3wJV223+RPm9Zrk+MYhZPabi wFyLjy/AD3nBJt7sLn6UoJidHNGnXOsJUS1+pAMfKo7mF1I+88GO0D1zya8vtlpP13wKrlHBRUC WyGMN4u70QS6N9e7oxZxj0sHX43MHsDSjt0gUNMVi8Qra4hoE9Fk9VApnQzTVjpl4qfQDPafJyu mLRBe22dpv8gHByu26iWc6Xoecnht5itqgMIQzOsL4W7sPnH11MI6hRXErJBqZV2p8opT3o3waP Ukvf3k+l8XeJzdvqWSKxTlwyGVAYgVfsS21TNjfWCUR/9904VQD8g/a+SO2P+9m0EZGgx1HhJDw 2avfyRM1kYNF2XstEKdzdlEQkWzguYEbSUOEb/kXgn8Svr4368/E3734uD/6ES80ml9fBWysGyu Lq8jd4TB6QPa5n+ivybYCfXrvgfH409 X-Received: by 2002:a17:90b:54c4:b0:39e:4c7f:7308 with SMTP id 98e67ed59e1d1-39e55071981mr15415502a91.31.1789989260551; Mon, 21 Sep 2026 04:14:20 -0700 (PDT) Received: from fedora ([61.74.238.173]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39e6c6dd585sm13438703a91.0.2026.09.21.04.14.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 04:14:20 -0700 (PDT) From: SeungJu Cheon To: Anup Patel Cc: Atish Patra , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexandre Ghiti , Paolo Bonzini , Andrew Jones , Jinyu Tang , Wang Yechao , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, SeungJu Cheon Subject: [PATCH v1 1/5] KVM: riscv: Age all G-stage PTEs in a GFN range Date: Mon, 21 Sep 2026 20:13:58 +0900 Message-ID: <20260921111402.120911-2-suunj1331@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260921111402.120911-1-suunj1331@gmail.com> References: <20260921111402.120911-1-suunj1331@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" kvm_age_gfn() and kvm_test_age_gfn() only operate on the G-stage leaf covering range->start. They also warn unless the range size is exactly 4K, 2M, or 1G. The MM can pass larger ranges when batching the PTEs of a large folio. For example, a 64K mTHP range triggers the warning and only the first 4K leaf is aged, leaving the Accessed bits of the remaining leaves unchanged. This can be reproduced on an rv64 host with Svadu by disabling 2M THP, enabling 64K mTHP, and running: access_tracking_perf_test -s anonymous_thp This results in: WARNING: arch/riscv/kvm/mmu.c:362 at kvm_age_gfn+0x210/0x370 ... [] kvm_age_gfn+0x210/0x370 [] kvm_mmu_notifier_clear_young+0x250/0x478 [] __mmu_notifier_clear_young+0xde/0x1a0 [] walk_pud_range+0x86e/0x1688 [] walk_pgd_range+0x518/0x17b0 [] __walk_page_range+0xea/0x580 [] walk_page_range+0x3a/0x80 [] try_to_inc_max_seq+0x520/0xe68 [] lru_gen_seq_write+0xd08/0x1310 Walk all G-stage leaves in the requested range. Advance by the leaf size when a mapping is found, or by the size of the non-present region at the level where the walk stopped. When testing the age, return as soon as a young leaf is found. Remove the range-size warning since arbitrary GFN ranges are valid. Fixes: 9955371cc014 ("RISC-V: KVM: Implement MMU notifiers") Signed-off-by: SeungJu Cheon --- arch/riscv/include/asm/kvm_gstage.h | 2 ++ arch/riscv/kvm/gstage.c | 34 +++++++++++++++++++++++++++++ arch/riscv/kvm/mmu.c | 22 ++++--------------- 3 files changed, 40 insertions(+), 18 deletions(-) diff --git a/arch/riscv/include/asm/kvm_gstage.h b/arch/riscv/include/asm/k= vm_gstage.h index aaf080ba1b77..a03db1a10095 100644 --- a/arch/riscv/include/asm/kvm_gstage.h +++ b/arch/riscv/include/asm/kvm_gstage.h @@ -81,6 +81,8 @@ bool kvm_riscv_gstage_unmap_range(struct kvm_gstage *gsta= ge, gpa_t start, gpa_t size, bool may_block); =20 bool kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa= _t end); +bool kvm_riscv_gstage_age_range(struct kvm_gstage *gstage, gpa_t start, + gpa_t end, bool test_only); =20 bool kvm_riscv_gstage_wp_pt_masked(struct kvm_gstage *gstage, gfn_t base_g= fn, unsigned long mask); diff --git a/arch/riscv/kvm/gstage.c b/arch/riscv/kvm/gstage.c index e5002cb9cbef..f61ba434093c 100644 --- a/arch/riscv/kvm/gstage.c +++ b/arch/riscv/kvm/gstage.c @@ -485,6 +485,40 @@ bool kvm_riscv_gstage_wp_range(struct kvm_gstage *gsta= ge, gpa_t start, gpa_t end return flush; } =20 +bool kvm_riscv_gstage_age_range(struct kvm_gstage *gstage, gpa_t start, + gpa_t end, bool test_only) +{ + unsigned long page_size; + bool young =3D false; + gpa_t addr =3D start; + pte_t *ptep; + u32 level; + bool found; + + while (addr < end) { + found =3D kvm_riscv_gstage_get_leaf(gstage, addr, &ptep, &level); + if (gstage_level_to_page_size(gstage, level, &page_size)) + break; + + if (found) { + if (test_only) { + if (pte_young(ptep_get(ptep))) + return true; + } else { + young |=3D ptep_test_and_clear_young(NULL, 0, ptep); + } + } + + /* + * Advance past this leaf, or past the non-present region at + * the level where the walk stopped. + */ + addr =3D ALIGN_DOWN(addr, page_size) + page_size; + } + + return young; +} + static inline void clear_huge_mask(unsigned long *mask, unsigned long page= _size, gfn_t base_gfn, gpa_t addr) { diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index 3e955d808743..342f606399e5 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -351,42 +351,28 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_= gfn_range *range) =20 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { - pte_t *ptep; - u32 ptep_level =3D 0; - u64 size =3D (range->end - range->start) << PAGE_SHIFT; struct kvm_gstage gstage; =20 if (!kvm->arch.pgd) return false; =20 - WARN_ON(size !=3D PAGE_SIZE && size !=3D PMD_SIZE && size !=3D PUD_SIZE); - kvm_riscv_gstage_init(&gstage, kvm); - if (!kvm_riscv_gstage_get_leaf(&gstage, range->start << PAGE_SHIFT, - &ptep, &ptep_level)) - return false; =20 - return ptep_test_and_clear_young(NULL, 0, ptep); + return kvm_riscv_gstage_age_range(&gstage, range->start << PAGE_SHIFT, + range->end << PAGE_SHIFT, false); } =20 bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { - pte_t *ptep; - u32 ptep_level =3D 0; - u64 size =3D (range->end - range->start) << PAGE_SHIFT; struct kvm_gstage gstage; =20 if (!kvm->arch.pgd) return false; =20 - WARN_ON(size !=3D PAGE_SIZE && size !=3D PMD_SIZE && size !=3D PUD_SIZE); - kvm_riscv_gstage_init(&gstage, kvm); - if (!kvm_riscv_gstage_get_leaf(&gstage, range->start << PAGE_SHIFT, - &ptep, &ptep_level)) - return false; =20 - return pte_young(ptep_get(ptep)); + return kvm_riscv_gstage_age_range(&gstage, range->start << PAGE_SHIFT, + range->end << PAGE_SHIFT, true); } =20 static bool fault_supports_gstage_huge_mapping(struct kvm_memory_slot *mem= slot, --=20 2.52.0 From nobody Fri Sep 25 21:05:34 2026 Received: from mail-pj2-f12.google.com (mail-pj2-f12.google.com [74.125.227.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7F005484238 for ; Mon, 21 Sep 2026 11:14:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789989268; cv=none; b=e2yJ6TE/OyNgNgzpbEciBpI+fr7e7gVJ6cVbQyyz0K/osWl9Aet8Y9i4Vpg8qPQN0VsFJDIosd+NHbiYSwhZ8De8c2If3T9Vd28J7strSiFDc+YbVdZHJouXS4SJO254OvpXdhg0PXPK+ShFHxV+qSreGKfGtvCOHEkcpkKgNck= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789989268; c=relaxed/simple; bh=1jg1XxWMFzFAotXiHpvxbD8wsFp6YWqm9Uxeq0aUeV8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fU/n+0u0s/r/Th2DmEPZNGHzOhHV56uJ74GE0f5UsxYhEaRllmZabi3UhNoQcHY6SiWNI6ELBr3mD0vG2xHoYZKCQl7zF/rRrtnd5m5gBlooVAfn2dNUBES9n6KwA6zFGdUuVKT7KIoTOLYiXf8uCwPDSGBuhaLq0LtrKQlzB4E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=FBzRnrvp; arc=none smtp.client-ip=74.125.227.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FBzRnrvp" Received: by mail-pj2-f12.google.com with SMTP id d9443c01a7336-2d91ede8035so33621205ad.3 for ; Mon, 21 Sep 2026 04:14:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789989264; x=1790594064; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=TdOHOI1E8fGa5CU5V13c9pAIdSK9plZJHC6KFQ2NxiI=; b=FBzRnrvp0gBU6Yet5j1nepLE+K8P+GMBuDeoX/06fyykYHBs7XemjVfaELpoRkpv9X CpEmddG4LkZvUz6fBa64Z6Q/GpZeDg/c/+m4wILvIuoaytVxv0RsIpod/lGV9EQqMggn 1+eAKjABo/KgpNKOP173d2j9H2V282NjsxehrnBCKGaS02EvBPnVOYnjoYUgepqrIp4i yfmX8BAnMZ/Zm0Jo7CAe+HdT2gphdeQ17s5b/m1Zuv8fvuahfWtVJYk7Dq812gnfayQm jpVmfVU6jgCWvdedZRQNUJYgTH/7+F98jPwiRXKseifZOGnWVIDw02/YUZm7tM6u987Z gqBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789989264; x=1790594064; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=TdOHOI1E8fGa5CU5V13c9pAIdSK9plZJHC6KFQ2NxiI=; b=hpbMBl2ESsCFTytQQIXQvp5hjYFMKuoUhWaVkNV7wzq1ItZVxiw8eF35wR9CFbMO0P PhsN4OT/xrMS1VSU+iHku2pYgAjHmm7Xoi37QzyF79N7ulDcsxssz85PKIG9295VpC8T KgGY4FnvaAG//E51kimSxGiUoWxlzlL4yp33wwX0rXrHhL/eBh/iJgr7kbzApRJPADxt l4nHJwL50jK2tbggqtgP6JKNvBE2ktrGp0gFnJOj516MqlGwLQ37hGpjH2bUw4g64ezO 88qaB9OqbvUfjP6KxyKIBQ5Cs0apsAHQHbZwOqjqQc40UNVJejlzmmAEm+giq1lpoLjk NaWw== X-Forwarded-Encrypted: i=1; AKwUvBwM58CHvPax/XWeSZHqh/PBUyrEi8zWrACTPel0lsKAv5ltyj4Mdgq2MXUDmkzXkNitvguu75TOgSpL8Kc=@vger.kernel.org X-Gm-Message-State: AFuF++mJK8YOwe92ysTi1mm7a1bjjvsL9WCqk+40KWT6K6QKe811ip7R pfVNTikTVqUMW7oemZXjJQWIPQX/tNeNE1AFCogduyOLeJXImBDMkWpv X-Gm-Gg: AYBFou10gX00MTrJr753h/1hHiSOl0TIXgb+rEZjuzj1Or+gnvaCyqj9ERuJQe/YguG W7FwCqI1hnCgXJ20UmrXvE2IiJkdjEASC1fwuBGttvaKaxEuH8mbBZ4qmotbgHVvN/F+ifKK79Z WqD62JPucoWfiL64rJq3l/3FraJl4mvgEHRvwWm4IZRbbTw28elnwa+gyCY9ssJWG430ZrARz4M NDY0rdsm20ttGzr6scmS4vy4DVC/oybCY09DggcB7fOlhfpTlZEoWXLfI7O+mNKrBrOn1MZ3baF oAcYEjLN9qiFgQmonUz0HehKqwTviy7e7dt4g7NOr+aSKgSCI5q2rUF1bKfe7zWDLJDwQvmsIpi 27c6AF4kc7/AXIDYdWJwvSUMGlzQ34oWSXKheLzvk8bDiVm+9Z79ybvmZzv8tYxnMbGefZpqrtI lKABBUoHzkzgmAkhj6g/nYZW6wKLPWT+anZuV7Xo2IuBZBQCnKI6645bR16i9FQ3f4pdvfZOfOq zH6OpAFAUjZv9rzCXSZveJgAka7uzAz X-Received: by 2002:a17:90b:35c9:b0:39e:6c6a:656c with SMTP id 98e67ed59e1d1-39e6c6a674bmr9093523a91.47.1789989264562; Mon, 21 Sep 2026 04:14:24 -0700 (PDT) Received: from fedora ([61.74.238.173]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39e6c6dd585sm13438703a91.0.2026.09.21.04.14.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 04:14:24 -0700 (PDT) From: SeungJu Cheon To: Anup Patel Cc: Atish Patra , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexandre Ghiti , Paolo Bonzini , Andrew Jones , Jinyu Tang , Wang Yechao , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, SeungJu Cheon Subject: [PATCH v1 2/5] KVM: riscv: Read G-stage PTEs once when walking page tables Date: Mon, 21 Sep 2026 20:13:59 +0900 Message-ID: <20260921111402.120911-3-suunj1331@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260921111402.120911-1-suunj1331@gmail.com> References: <20260921111402.120911-1-suunj1331@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" gstage_pte_leaf() dereferences the PTE pointer, causing walkers to read the same entry multiple times. This is safe under mmu_lock, but a lockless walker could observe different values when deciding whether an entry is a leaf and which child table to follow. Make gstage_pte_leaf() take a PTE value and use a single snapshot for presence, leaf and child-table checks. Also remove a redundant leaf check in kvm_riscv_gstage_set_pte(), where mmu_lock prevents the entry from changing. Signed-off-by: SeungJu Cheon --- arch/riscv/kvm/gstage.c | 48 +++++++++++++++++++++++------------------ 1 file changed, 27 insertions(+), 21 deletions(-) diff --git a/arch/riscv/kvm/gstage.c b/arch/riscv/kvm/gstage.c index f61ba434093c..944fa4c95aea 100644 --- a/arch/riscv/kvm/gstage.c +++ b/arch/riscv/kvm/gstage.c @@ -19,8 +19,8 @@ unsigned long kvm_riscv_gstage_max_pgd_levels __ro_after_= init =3D 3; unsigned long kvm_riscv_gstage_max_pgd_levels __ro_after_init =3D 2; #endif =20 -#define gstage_pte_leaf(__ptep) \ - (pte_val(*(__ptep)) & (_PAGE_READ | _PAGE_WRITE | _PAGE_EXEC)) +#define gstage_pte_leaf(__pte) \ + (pte_val(__pte) & (_PAGE_READ | _PAGE_WRITE | _PAGE_EXEC)) =20 static inline unsigned long gstage_pte_index(struct kvm_gstage *gstage, gpa_t addr, u32 level) @@ -84,14 +84,18 @@ static int gstage_level_to_page_size(struct kvm_gstage = *gstage, u32 level, bool kvm_riscv_gstage_get_leaf(struct kvm_gstage *gstage, gpa_t addr, pte_t **ptepp, u32 *ptep_level) { - pte_t *ptep; + pte_t *ptep, pte; u32 current_level =3D gstage->pgd_levels - 1; =20 *ptep_level =3D current_level; ptep =3D (pte_t *)gstage->pgd; ptep =3D &ptep[gstage_pte_index(gstage, addr, current_level)]; - while (ptep && pte_val(ptep_get(ptep))) { - if (gstage_pte_leaf(ptep)) { + while (ptep) { + pte =3D ptep_get(ptep); + if (!pte_val(pte)) + break; + + if (gstage_pte_leaf(pte)) { *ptep_level =3D current_level; *ptepp =3D ptep; return true; @@ -100,7 +104,7 @@ bool kvm_riscv_gstage_get_leaf(struct kvm_gstage *gstag= e, gpa_t addr, if (current_level) { current_level--; *ptep_level =3D current_level; - ptep =3D (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); + ptep =3D (pte_t *)gstage_pte_page_vaddr(pte); ptep =3D &ptep[gstage_pte_index(gstage, addr, current_level)]; } else { ptep =3D NULL; @@ -151,10 +155,12 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstag= e, return -EINVAL; =20 while (current_level !=3D map->level) { - if (gstage_pte_leaf(ptep)) + pte_t pte =3D ptep_get(ptep); + + if (gstage_pte_leaf(pte)) return -EEXIST; =20 - if (!pte_val(ptep_get(ptep))) { + if (!pte_val(pte)) { if (!pcache) return -ENOMEM; next_ptep =3D kvm_mmu_memory_cache_alloc(pcache); @@ -163,9 +169,7 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage, set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)), __pgprot(_PAGE_TABLE))); } else { - if (gstage_pte_leaf(ptep)) - return -EEXIST; - next_ptep =3D (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); + next_ptep =3D (pte_t *)gstage_pte_page_vaddr(pte); } =20 current_level--; @@ -175,7 +179,7 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage, if (pte_val(*ptep) !=3D pte_val(map->pte)) { bool was_invalid =3D !pte_val(*ptep); set_pte(ptep, map->pte); - if (gstage_pte_leaf(ptep) && + if (gstage_pte_leaf(map->pte) && !(was_invalid && riscv_has_extension_unlikely(RISCV_ISA_EXT_SVVPTC))) gstage_tlb_flush(gstage, current_level, map->addr); } @@ -316,7 +320,7 @@ bool kvm_riscv_gstage_split_huge(struct kvm_gstage *gst= age, unsigned long huge_pte, child_pte; unsigned long child_page_size; bool need_flush =3D false; - pte_t *ptep; + pte_t *ptep, pte; int i, ret; =20 if (!pcache) @@ -325,16 +329,17 @@ bool kvm_riscv_gstage_split_huge(struct kvm_gstage *g= stage, while(current_level > target_level) { ptep =3D (pte_t *)&next_ptep[gstage_pte_index(gstage, addr, current_leve= l)]; =20 - if (!pte_val(ptep_get(ptep))) + pte =3D ptep_get(ptep); + if (!pte_val(pte)) break; =20 - if (!gstage_pte_leaf(ptep)) { - next_ptep =3D (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); + if (!gstage_pte_leaf(pte)) { + next_ptep =3D (pte_t *)gstage_pte_page_vaddr(pte); current_level--; continue; } =20 - huge_pte =3D pte_val(ptep_get(ptep)); + huge_pte =3D pte_val(pte); =20 ret =3D gstage_level_to_page_size(gstage, current_level - 1, &child_page= _size); if (ret) @@ -373,7 +378,7 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage,= gpa_t addr, pte_t *ptep, u32 ptep_level, enum kvm_riscv_gstage_op op) { int i, ret; - pte_t old_pte, *next_ptep; + pte_t old_pte, pte, *next_ptep; u32 next_ptep_level; unsigned long next_page_size, page_size; bool flush =3D false; @@ -384,11 +389,12 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstag= e, gpa_t addr, =20 WARN_ON(addr & (page_size - 1)); =20 - if (!pte_val(ptep_get(ptep))) + pte =3D ptep_get(ptep); + if (!pte_val(pte)) return false; =20 - if (ptep_level && !gstage_pte_leaf(ptep)) { - next_ptep =3D (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); + if (ptep_level && !gstage_pte_leaf(pte)) { + next_ptep =3D (pte_t *)gstage_pte_page_vaddr(pte); next_ptep_level =3D ptep_level - 1; ret =3D gstage_level_to_page_size(gstage, next_ptep_level, &next_page_si= ze); if (ret) --=20 2.52.0 From nobody Fri Sep 25 21:05:34 2026 Received: from mail-pj2-f13.google.com (mail-pj2-f13.google.com [74.125.227.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7A9E648166B for ; Mon, 21 Sep 2026 11:14:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.141 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789989272; cv=none; b=d4EXeISgfSzmmLbCICTPkGJ//mhsWGSB1+4ougjzaA/oTit8PMUdGzpYAcLrRJhNMxk2m8r2LTflH3kQnZhz3UcV52rLLSoKjfWi8CkfPdI/Afd14RUKlEC4nn2wQDvgxPsTJd0rUdOJJ3cwikxplQMSWrvcND97mNobavrmATc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789989272; c=relaxed/simple; bh=7Kae4217hH0Tw1s8Lang6k2eDbR1kYC9KLQLEwPXcls=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RilywflfUzEcKq5FNprqzm5ty5LI3xmiqFgqDhXjv+r2vGAZLQpcGNu4gSDOIn9jXKA8OCUfVPRyz10SZprqgn7mhzH/+jaL8Cs/saDRUzBtUCdqub93AjJQXCmh4kbCV05ZTF9Ev69/9LP19pFBF9fxh8PVUxHpexCLFnfm+C4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=akv8dRDA; arc=none smtp.client-ip=74.125.227.141 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="akv8dRDA" Received: by mail-pj2-f13.google.com with SMTP id d9443c01a7336-2d8fbef5018so35119185ad.0 for ; Mon, 21 Sep 2026 04:14:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789989269; x=1790594069; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=077YoXqYGKuusFGEo62X7PJkVmKJetb2TRAJK+g/S80=; b=akv8dRDAowJJaGtSE2+iStxSucNPJR2EobwMNJlRn5kFYUqf304XU8CgS2vO+V70Gi mPt1LFTAhbnvz2qSbwAcb0q9Px18DmpgdMkDZi7KiFkyFu8RIcHZgeDpLLGQmIlJmqoc jFUTsRrMCtZxq86tf0goLgFyqPNxicnouR4ss+edX4Fjdz2NOR0F5Ehap002/c0yfzf9 /2tHD1razm0kYGIM3JQPZ4VHJxpLM0CaFHhgDpGjs4R4bmisi3UcL9LmeDHPwezc64T5 jngsFWsnV5PJr9XM6q/EjSp431z3YLDMY8TQKIyQpymaT9OfPf8dEOBZEwjO18XDrkst 8YBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789989269; x=1790594069; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=077YoXqYGKuusFGEo62X7PJkVmKJetb2TRAJK+g/S80=; b=GFAvjE+Y54jVVi/m1JIfaG2YT56w67Hk5N5JpDxc4dWEFsxV1Hc/yRuNvA623rjoEC 2yXDo9OIBm+M2J9Z3vU3yEZr/pB6wzMlZ1Fqwf/+aMOh5RDh2dGNDeSVGLJEShAmvHxw kpXX+XKFY4PxHg5e0pjtsqLC4FyiS/usaAQNpulxdNQhGl/YBmOi820ZmkxIq8/i+sYT E1tE+kJCI+vyFrq3rqwvCY8jeJou43R/adlZi8fMU8uIB/tJnTceeY487i9mEwaXkwYx +ZdXW72SUv5HLb900KRsIbll7dTRFfb2WYZ01P1v/MS7Zs+sILW4TkLPiDrlL+paQO1x kyjQ== X-Forwarded-Encrypted: i=1; AKwUvBzalfMI3FBL32TBcNp7fOWGSpBQkZvUzWY/JxHtkSl5ZPuxQneccQWgRyI4E06nExj3O+8z8rVbs8zLcQU=@vger.kernel.org X-Gm-Message-State: AFuF++lseNotc9NLuSCcERb72trMFXvWId/VWaOKUWOHTJjCEXXvtiIj IYBTkY6ht1ho88VipaJETe6wl+yf8y1BbYc/H4770+hhr3wGKr1vl+Wi+daCzg== X-Gm-Gg: AYBFou2OqjY/6BonYZ3ZamGbr3xqk3hC9cEnoVXb+pKoon4FVMN850QxfhDpFUnfgHk PwTvRUefTAUF819tpb8pNO/fRE7TCmq8VIno6TJkmw38TNyvb0rB8tTGSzqS8WmRzI53WH+UvUS OuxJ/WmLcj+6WGCqiIha6DLu/wpGeizFFDJg7dmSzBFnod4OfQOzjYucQs8uiOuQGWFMY5/Wy0/ zYkXNEiH4E1+5YP6+BKRPr4IPhejCnq427/jCT6ZfAaoucmGGckXkTnyhNBPdxIw+cGkK5+0Cmh WDUMzeH4F/ct7HswmKycCXFvMlH9oWpZQ+7VM+uBttWgNkfMWjnDswsZohZQ8TI5lhPf89acKIB 5FTmsHvYRuQTGDTAs3ssjGO9/McXKmeSOFtIj8t9UuH9QwfHos4pvwU+Ag+XNTukG1F+PQr6DfS xv7cxStUEJ991U8/+E3JMttF+upHkElmvwEza8i40ICdJLU3mSyQ/vxxFvCx39EWAbnNoQnIk7s 2iedRVvBLKV/Vw3e9M/tg== X-Received: by 2002:a17:90b:1cc8:b0:39e:1329:7816 with SMTP id 98e67ed59e1d1-39e54d703dbmr15998993a91.15.1789989268553; Mon, 21 Sep 2026 04:14:28 -0700 (PDT) Received: from fedora ([61.74.238.173]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39e6c6dd585sm13438703a91.0.2026.09.21.04.14.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 04:14:28 -0700 (PDT) From: SeungJu Cheon To: Anup Patel Cc: Atish Patra , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexandre Ghiti , Paolo Bonzini , Andrew Jones , Jinyu Tang , Wang Yechao , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, SeungJu Cheon Subject: [PATCH v1 3/5] KVM: riscv: Write-protect G-stage PTEs atomically Date: Mon, 21 Sep 2026 20:14:00 +0900 Message-ID: <20260921111402.120911-4-suunj1331@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260921111402.120911-1-suunj1331@gmail.com> References: <20260921111402.120911-1-suunj1331@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" GSTAGE_OP_WP clears the W bit with a read-modify-write of the entire PTE. Lockless aging can concurrently clear the Accessed bit, allowing write-protection to overwrite the update with a stale PTE value. Clear W atomically with test_and_clear_bit() so that concurrent Accessed-bit updates are preserved. Its return value also preserves the existing behavior of requesting a TLB flush only when W was actually cleared. Signed-off-by: SeungJu Cheon --- arch/riscv/kvm/gstage.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/arch/riscv/kvm/gstage.c b/arch/riscv/kvm/gstage.c index 944fa4c95aea..f7e4756ce15b 100644 --- a/arch/riscv/kvm/gstage.c +++ b/arch/riscv/kvm/gstage.c @@ -378,7 +378,7 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage,= gpa_t addr, pte_t *ptep, u32 ptep_level, enum kvm_riscv_gstage_op op) { int i, ret; - pte_t old_pte, pte, *next_ptep; + pte_t pte, *next_ptep; u32 next_ptep_level; unsigned long next_page_size, page_size; bool flush =3D false; @@ -408,13 +408,16 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstag= e, gpa_t addr, if (op =3D=3D GSTAGE_OP_CLEAR) put_page(virt_to_page(next_ptep)); } else { - old_pte =3D *ptep; - if (op =3D=3D GSTAGE_OP_CLEAR) + if (op =3D=3D GSTAGE_OP_CLEAR) { set_pte(ptep, __pte(0)); - else if (op =3D=3D GSTAGE_OP_WP) - set_pte(ptep, __pte(pte_val(ptep_get(ptep)) & ~_PAGE_WRITE)); - if (pte_val(*ptep) !=3D pte_val(old_pte)) flush =3D true; + } else if (op =3D=3D GSTAGE_OP_WP) { + /* + * Clear W atomically to avoid clobbering a concurrent + * Accessed-bit update by lockless aging. + */ + flush =3D test_and_clear_bit(__ffs(_PAGE_WRITE), &ptep->pte); + } } =20 return flush; --=20 2.52.0 From nobody Fri Sep 25 21:05:34 2026 Received: from mail-pj2-f12.google.com (mail-pj2-f12.google.com [74.125.227.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9403D48EBE4 for ; Mon, 21 Sep 2026 11:14:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789989278; cv=none; b=ueT+YVdVQlY26+D8QUzivmtrUT9mWuMwney660yZEi0IH0Eyk86c7OkxdtY13klzTA7jOGOEMtsbRnXVmJopXtgHnb2rxl0A1HARhWVE0vTUYFlfT2cbZEGMjKmE+hwNYXqGTwkSr+jBbiOc/fX6q4Qz3gLweR0+GKuf/k00Ejk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789989278; c=relaxed/simple; bh=JcBuvM3CVeR3tUQ0SEU79f7AefJbf/1Y11q/0iXbWM4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TwsjaUVAtiO7SezM/0bZgQLr4oLj/vi5icfxDg5tLNTGZCgXnZ9R/eauFNdrCRiE6YwFdA5hANKdiX5i93bO248SHrlRp8OzSnnsZFe5gC6PgAY968PSvODNChEm/tjY182xpsJL1+Ecnve1FAxu+ec5kCccoP6IAOeV2QcGLzc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YLxUfYhs; arc=none smtp.client-ip=74.125.227.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YLxUfYhs" Received: by mail-pj2-f12.google.com with SMTP id d9443c01a7336-2d747ed1368so33801135ad.1 for ; Mon, 21 Sep 2026 04:14:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789989273; x=1790594073; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=6ef3phNMqqlA8Z4yqCL8AgS9AIUhshDlKUByQ4fToEc=; b=YLxUfYhsrMlNPeXWO0F8mcHAQzu9dds3x6IdHH//Lba5mk2MKySIAQo8p8HY7cKVvm SwxfBFE0367uswsJNCSlUX1nVmN86YxleKpWf8ACOHUFLjBVbznYWpV2YPBHkyq6300z uwi86vWfVtlN0CqHtpvhr2eMWVx/DRNGaic84bS+Pwp2xgmputYOdAhL51Hhe7HUWf8L 6ieRd4TRWU1XJClbk8judMp1JkP8Q9NDJbVghud3KR5uIWp93sM4AukMnMvH+JjzHlgV Sz1duqw2rrhzJg0MYXTPlaDhlJbMO8+orcqKrAR6/HEHsoEI337oJPND5W9CZOnwf2+g xP6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789989273; x=1790594073; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=6ef3phNMqqlA8Z4yqCL8AgS9AIUhshDlKUByQ4fToEc=; b=OKtkawU3JAU5GKZ8p4fZhH/NeH7nY0xIp872uK6JwZSHmj6c+5s636XLrKwvlbGMaF yKXD6YbXW+7wLHg5+9ctS5i99sOHziN2JCcI5gIQQmB9+jk3qfoHeLRH6dYxhTBVeo57 FEpeQqtr2QTmjma+tIjD3E0dx8WcKsiZoCSvE3SqwTPaQXcLYWmN+B+8B+dKpd0I6oD+ plzPLCt3HdLary0oNkG+yjXeThMcTZhTlQ0Bhpxcb+6HvIlrVACEb9TuZDgICqbn9sVq Z2snt4arzFxIh+O5j4KFKq+ZneDMnmlFSQ828nLojlHSr6CSDxcwzkUvJuY2b4M2jg9l l1wg== X-Forwarded-Encrypted: i=1; AKwUvBz2XQ9EkvEbH5eUFUn8Bgq72Uo4ZMZ9JXAWoJMhRHlV2lM2R9MoInO7zboeTtyasq8PSNtKEZ3v7ELQxSQ=@vger.kernel.org X-Gm-Message-State: AFuF++lycdelI3DbbG6gE6ouNFNjtQdiH55mbRb2Iutp9D+rw0+i8+JT W+ZRx9zskdiY30z/gZOJP/rt8WRB9c4V7srnCsd2IhHLQ5MA6DidurJ6 X-Gm-Gg: AYBFou3pPe/KnYdyZLUb5iLia1oQpXexvDWvCJSsf7V60BrpzPK2I9Gwa3p0qgqQlv2 ZIjyZ7VNquhbKDvMVt9xl5R6lggijSrCoEAa6nYJ2/QdzylR93r20AvMRnjZCtB/2HfB/f9f5sl KGipUG68ez/nzeIl8GOZp0Erz3VuplmbRh1kSqmo8o84fCi6CtwHOPgbsZLIVUEfUKH7wOLBeml tndEd4psskw4t8akq5pjFV9W+9IyPOOO3W8ND32lZPgzPS8TgKLftOm4mkAsaKt/SkF3VPWvKTR UVEWL4fseGTq2QI2fEMAY201gN8v7Th2Dcpqg7LtUhFKuRxHUbK/FTiAhfko6WKxoRFZRe6byYH Qt4JFU9p6AV8+PK48keVd5IJcMFBgTJ20qz7VQI1F5M8YhqxLg5XEFS6gtz0eOdjbfKQhyrP3rd b59e13i5qAQYOWWhVY9Z01NausdrODQRaRfDDgWXQu68n8xzBsLdLomJ7AbaTqeD9rRw36FC/Ge LgENr4oX9y33fL0GuaDcQ== X-Received: by 2002:a17:90a:e18f:b0:39d:f5ef:b1d5 with SMTP id 98e67ed59e1d1-39e54aa347dmr15111936a91.7.1789989272818; Mon, 21 Sep 2026 04:14:32 -0700 (PDT) Received: from fedora ([61.74.238.173]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39e6c6dd585sm13438703a91.0.2026.09.21.04.14.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 04:14:32 -0700 (PDT) From: SeungJu Cheon To: Anup Patel Cc: Atish Patra , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexandre Ghiti , Paolo Bonzini , Andrew Jones , Jinyu Tang , Wang Yechao , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, SeungJu Cheon Subject: [PATCH v1 4/5] KVM: riscv: Free G-stage page tables after an RCU grace period Date: Mon, 21 Sep 2026 20:14:01 +0900 Message-ID: <20260921111402.120911-5-suunj1331@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260921111402.120911-1-suunj1331@gmail.com> References: <20260921111402.120911-1-suunj1331@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" G-stage page tables are currently freed immediately after being unlinked under mmu_lock. This is safe while all walkers hold mmu_lock, as a table cannot be freed while a walker is using it. A subsequent change will allow aging walks without mmu_lock, where a walker may obtain a child table pointer just before a concurrent unmap unlinks and frees the table. Defer freeing unlinked page-table pages with call_rcu(). The parent entry is cleared before the child table is retired, so new walkers cannot acquire it while existing RCU-protected walkers can safely finish using it. Apply the same lifetime rule to the root PGD. Unpublish the root with WRITE_ONCE() and defer its free with call_rcu(). Keep pgd_levels unchanged so a walker that observed the old root continues to use matching page-table metadata. Wait for pending G-stage page-table callbacks with rcu_barrier() when the RISC-V KVM module exits so that they complete before the module is unloaded. Signed-off-by: SeungJu Cheon --- arch/riscv/include/asm/kvm_gstage.h | 2 +- arch/riscv/kvm/gstage.c | 16 +++++++++++++++- arch/riscv/kvm/main.c | 3 +++ arch/riscv/kvm/mmu.c | 16 +++++++++++++--- 4 files changed, 32 insertions(+), 5 deletions(-) diff --git a/arch/riscv/include/asm/kvm_gstage.h b/arch/riscv/include/asm/k= vm_gstage.h index a03db1a10095..caeed6de6dbe 100644 --- a/arch/riscv/include/asm/kvm_gstage.h +++ b/arch/riscv/include/asm/kvm_gstage.h @@ -111,7 +111,7 @@ static inline void kvm_riscv_gstage_init(struct kvm_gst= age *gstage, struct kvm * gstage->kvm =3D kvm; gstage->flags =3D 0; gstage->vmid =3D READ_ONCE(kvm->arch.vmid.vmid); - gstage->pgd =3D kvm->arch.pgd; + gstage->pgd =3D READ_ONCE(kvm->arch.pgd); gstage->pgd_levels =3D kvm->arch.pgd_levels; } =20 diff --git a/arch/riscv/kvm/gstage.c b/arch/riscv/kvm/gstage.c index f7e4756ce15b..fc39d188b20a 100644 --- a/arch/riscv/kvm/gstage.c +++ b/arch/riscv/kvm/gstage.c @@ -22,6 +22,20 @@ unsigned long kvm_riscv_gstage_max_pgd_levels __ro_after= _init =3D 2; #define gstage_pte_leaf(__pte) \ (pte_val(__pte) & (_PAGE_READ | _PAGE_WRITE | _PAGE_EXEC)) =20 +static void gstage_free_page_table_rcu(struct rcu_head *head) +{ + put_page(container_of(head, struct page, rcu_head)); +} + +/* + * Defer freeing an unlinked page table until lockless walkers + * that may have observed it have exited. + */ +static void gstage_free_page_table(pte_t *table) +{ + call_rcu(&virt_to_page(table)->rcu_head, gstage_free_page_table_rcu); +} + static inline unsigned long gstage_pte_index(struct kvm_gstage *gstage, gpa_t addr, u32 level) { @@ -406,7 +420,7 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage,= gpa_t addr, flush |=3D kvm_riscv_gstage_op_pte(gstage, addr + i * next_page_size, &next_ptep[i], next_ptep_level, op); if (op =3D=3D GSTAGE_OP_CLEAR) - put_page(virt_to_page(next_ptep)); + gstage_free_page_table(next_ptep); } else { if (op =3D=3D GSTAGE_OP_CLEAR) { set_pte(ptep, __pte(0)); diff --git a/arch/riscv/kvm/main.c b/arch/riscv/kvm/main.c index 89568ccce01d..5fcf425f7150 100644 --- a/arch/riscv/kvm/main.c +++ b/arch/riscv/kvm/main.c @@ -262,6 +262,9 @@ static void __exit riscv_kvm_exit(void) { kvm_exit(); =20 + /* Wait for pending G-stage page-table RCU callbacks. */ + rcu_barrier(); + /* Unregister CPU PM notifier */ if (IS_ENABLED(CONFIG_CPU_PM)) cpu_pm_unregister_notifier(&kvm_riscv_cpu_pm_nb); diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index 342f606399e5..8aed69abf814 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -761,6 +761,13 @@ int kvm_riscv_mmu_alloc_pgd(struct kvm *kvm) return 0; } =20 +static void kvm_riscv_mmu_free_pgd_rcu(struct rcu_head *head) +{ + struct page *page =3D container_of(head, struct page, rcu_head); + + __free_pages(page, get_order(kvm_riscv_gstage_pgd_size)); +} + void kvm_riscv_mmu_free_pgd(struct kvm *kvm) { struct kvm_gstage gstage; @@ -773,9 +780,12 @@ void kvm_riscv_mmu_free_pgd(struct kvm *kvm) flush =3D kvm_riscv_gstage_unmap_range(&gstage, 0UL, kvm_riscv_gstage_gpa_size(kvm->arch.pgd_levels), false); pgd =3D READ_ONCE(kvm->arch.pgd); - kvm->arch.pgd =3D NULL; + /* + * Keep pgd_levels unchanged for lockless walkers that already + * observed the old root. + */ + WRITE_ONCE(kvm->arch.pgd, NULL); kvm->arch.pgd_phys =3D 0; - kvm->arch.pgd_levels =3D 0; } write_unlock(&kvm->mmu_lock); =20 @@ -783,7 +793,7 @@ void kvm_riscv_mmu_free_pgd(struct kvm *kvm) kvm_flush_remote_tlbs(kvm); =20 if (pgd) - free_pages((unsigned long)pgd, get_order(kvm_riscv_gstage_pgd_size)); + call_rcu(&virt_to_page(pgd)->rcu_head, kvm_riscv_mmu_free_pgd_rcu); =20 kvm_mmu_free_memory_cache(&kvm->arch.pgd_split_page_cache); } --=20 2.52.0 From nobody Fri Sep 25 21:05:34 2026 Received: from mail-pj2-f12.google.com (mail-pj2-f12.google.com [74.125.227.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA845484238 for ; Mon, 21 Sep 2026 11:14:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789989281; cv=none; b=p6c+MtOQ/qfvxRK+28wiK+gi4qKTGpcbt1nu93x6patttifEUQsPGpSVPc8/3SZxR22mm1tZNCKxTkLWf+L+/IsK52pHqg+hPxDDtWTZfFrPU1q5+YNj1IR8JjprLvi24jUbX3t81eezcIGd3hImLx4H+iEVLhaBr0z2fMWbo48= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789989281; c=relaxed/simple; bh=I/vrSTMKABeo2I5WHFzvnttJqh0/4r7sFNJ5F/4nOHE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nTz5D4YX9qKTpbKXHeRAkH28ulAH5gmIezhusc+SDdvINo6xfGEoe3W2zdAS0PAi2zLoHAaVSiwGufR8kVQeiWcHPnhg94K1lqxG3clJgwZT8y0j+OPGiJwjj/ZkIr+9LaYoaxFTFPKAkWPLo0jo+10STMCG9UQF/WbZIlujJEM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ltJ+4EU9; arc=none smtp.client-ip=74.125.227.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ltJ+4EU9" Received: by mail-pj2-f12.google.com with SMTP id 98e67ed59e1d1-39b5b07ec78so1902481a91.3 for ; Mon, 21 Sep 2026 04:14:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789989277; x=1790594077; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=4xMVi+cD5sodYKE++E9vQWOcZxlG7jO/nwtdaiSfROQ=; b=ltJ+4EU9zK+DVRa1CFbt692UfimWgcwW0z2YRIWDpG2UDx0ROLHTnOcRKTSzuKXPOM 6wwhIt0MAGs9I9h0neyRVhZFHVJmjP/j8pYElPO9x/8RqcCUiOfbyDCa459EAgY62Tj9 ekqoYGxf4CKnzOTFQrAXUYnPlk9ruI9OtehnLiMFwCQjD9aXhQFpTKLIp+zltAogRm1z OqAoeJhu4kl+ciwTTTr0AZNxm2NVDWs7rqzUOR8Pso5lt+qIKT/XviqbsA0s4yZMEMkr 6kSfEjZS3ujMmDUivRGzD7JmO9u2AyEtp+158b/HkgVuwkV2W1Y5IFaX78TBPDbwQOrI w9ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789989277; x=1790594077; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=4xMVi+cD5sodYKE++E9vQWOcZxlG7jO/nwtdaiSfROQ=; b=FFzMxzJFFVxOZoOBtH1+exth/blVAeJmbGPO3EMfW1Dd7Dt6124Spoc9TN00KpILuE EJLKFlyCAMi8/TnBC4eFavaaytY/s8ZtHJWMxOoW5r9MULixNN6O9W8TIaV93AbAghyH hva+6SY3aQHQ51btFPHrlVnE+P68HqTIHjv1JkIQmioZHP3G6BinETv3vtHq7lEYQZkr h6C3PJ3CwfzmEwtuLSluT3dbw0uwtysAMNVeJSDCsCIzVDeb7mzSnDINaYULZzDRIc1o /LcKbhJX9Zpc/XD9eQfwmowyidRKZXKWHz/Dh0n2bNkgPVPHF1900ETvBce22K1y/Ftj rBAA== X-Forwarded-Encrypted: i=1; AKwUvBwZjutZJ+8723nEzBe9wOPNE0w0kCkk6mL9zind8uv3D2BB4J7oU0I1D9NBhxXUGCLNL3ViTcpJZBbHOhI=@vger.kernel.org X-Gm-Message-State: AFuF++m0YZ3wneVRtjTLMJ46vwvNeYT27luhYUbjy2W5Owzwq58gjSU4 NKmzSAxQaFgeLAbjzjAVZGxfT2Vlv3AkAqZoPcemTnlszu6ArcQmo67w X-Gm-Gg: AYBFou2LOjZ907ulmyjk0w58wUTNL+SA2/LHeliiRP8BhFYhZ287Uj0+fQLQJ0hQdKi Ud08nsTiQlDWNV4WYXMMNjv+j/n/b/NtskE+O9Dczt3WvrVOh+cy7j1PTVV388vwZkBxgqm3Uu6 aHbjcDRewbWm9yYSa4CFxWKCf8YLHN3sJzbzsKu/gSjIe5KPgIWC7qXoPFZrOaBk1WwweBziEjm q79Fv5Qbchh9rdY69yhNM00INAD89Jr7wWbtXYwllZGAVMTHMU7FgSnoOJ8Md6BB1IVra0sYi0m b4k5LfK2GsAQi66ciVlsI7Kqc4QW3y1bzkiT7C0qveYH6W5xwVIVXdIFMJfOxEzy+QmaNIyUzad WSc+lwA/XLF0b20vu09AIz5eP3sUKJNROfiKDR6Ct4uWcEpLxi/gnwjEKi47fyKH0VL/2H4+LvX tamhb8Oa1k9b2Cd0wnmk+6n0ipYQ+ruPjuzFZaJx8QyennZOyD840kxxzPsziLNUIK9Njn095VF HdQagNaffBQtSgYq4Lc8w== X-Received: by 2002:a17:90b:17cc:b0:39e:135e:130e with SMTP id 98e67ed59e1d1-39e54af6a0emr16005396a91.3.1789989276896; Mon, 21 Sep 2026 04:14:36 -0700 (PDT) Received: from fedora ([61.74.238.173]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39e6c6dd585sm13438703a91.0.2026.09.21.04.14.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 04:14:36 -0700 (PDT) From: SeungJu Cheon To: Anup Patel Cc: Atish Patra , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexandre Ghiti , Paolo Bonzini , Andrew Jones , Jinyu Tang , Wang Yechao , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, SeungJu Cheon Subject: [PATCH v1 5/5] KVM: riscv: Age G-stage PTEs locklessly Date: Mon, 21 Sep 2026 20:14:02 +0900 Message-ID: <20260921111402.120911-6-suunj1331@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260921111402.120911-1-suunj1331@gmail.com> References: <20260921111402.120911-1-suunj1331@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Aging G-stage PTEs currently runs with mmu_lock held for write, taken by the common MMU notifier code. When MGLRU or kswapd ages a large range, every vCPU taking a G-stage fault blocks on the lock, and the dirty-logging read-side fast path blocks behind aging as well. The preceding patches prepare G-stage page-table walks for lockless aging by using consistent PTE snapshots, preserving concurrent Accessed-bit updates, and deferring page-table frees with RCU. Select KVM_MMU_LOCKLESS_AGING so the common code no longer takes mmu_lock for aging, and protect the G-stage walk with an RCU read-side critical section. Read the root inside the RCU read-side critical section and check the resulting snapshot instead of checking kvm->arch.pgd separately before initializing the G-stage context. This ensures that the root used by the walk remains protected until the walk completes. On QEMU TCG with 4 vCPUs, running dirty_log_perf_test -v 3 -b 256M -i 3 with MGLRU aging of the VM's cgroup forced every 100ms (86 passes): before after mmu_lock write wait, total 44.4 s 0.15 s mmu_lock write contentions 1,002,092 30,092 mmu_lock read contentions 351,937 0 guest dirty-memory time 18.6 s 16.7 s Before, 93% of write-lock waiters were kvm_mmu_notifier_clear_young(). With no aging, no meaningful difference was observed between the two kernels. Signed-off-by: SeungJu Cheon --- arch/riscv/kvm/Kconfig | 1 + arch/riscv/kvm/mmu.c | 26 +++++++++++++------------- 2 files changed, 14 insertions(+), 13 deletions(-) diff --git a/arch/riscv/kvm/Kconfig b/arch/riscv/kvm/Kconfig index ec2cee0a39e0..77898d58ff9a 100644 --- a/arch/riscv/kvm/Kconfig +++ b/arch/riscv/kvm/Kconfig @@ -32,6 +32,7 @@ config KVM select VIRT_XFER_TO_GUEST_WORK select SCHED_INFO select GUEST_PERF_EVENTS if PERF_EVENTS + select KVM_MMU_LOCKLESS_AGING help Support hosting virtualized guest machines. =20 diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index 8aed69abf814..ff282bdbe492 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -349,30 +349,30 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_= gfn_range *range) return false; } =20 -bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) +static bool kvm_riscv_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range, + bool test_only) { struct kvm_gstage gstage; =20 - if (!kvm->arch.pgd) - return false; + guard(rcu)(); + lockdep_assert_not_held(&kvm->mmu_lock); =20 kvm_riscv_gstage_init(&gstage, kvm); + if (!gstage.pgd) + return false; =20 return kvm_riscv_gstage_age_range(&gstage, range->start << PAGE_SHIFT, - range->end << PAGE_SHIFT, false); + range->end << PAGE_SHIFT, test_only); } =20 -bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) +bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { - struct kvm_gstage gstage; - - if (!kvm->arch.pgd) - return false; - - kvm_riscv_gstage_init(&gstage, kvm); + return kvm_riscv_age_gfn(kvm, range, false); +} =20 - return kvm_riscv_gstage_age_range(&gstage, range->start << PAGE_SHIFT, - range->end << PAGE_SHIFT, true); +bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) +{ + return kvm_riscv_age_gfn(kvm, range, true); } =20 static bool fault_supports_gstage_huge_mapping(struct kvm_memory_slot *mem= slot, --=20 2.52.0