From nobody Sat Feb 7 10:16:38 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0697F15F40A for ; Fri, 9 Aug 2024 19:43:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232623; cv=none; b=fv5WcoiWCdaOh9IJ0XW9v/00M++hjT8jaWnNA23x3sF7Nqyn/Bv5G9sWQsP+aGIw1NMuuwBdmPUlbHI2Q8KqH6iV6lXrvQvM8+6+9vugydbGGAypZ89iiWatjIMpsyaM/wTMshjJ7QHPqzZF7tHW07MZBIvrKz52K+6MpoHta50= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232623; c=relaxed/simple; bh=UxnvgXHSiR2kQsYdw03waRMGPjo2zH4fYnT3IGJL6c0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=eYV2UnWhyePaFlMet3CjGO9TybyukeMMb9fsEkCIZNnGW9FbInnWV32jUZqK9yI2s7SNMI9uOAwy3jouF4HZd6OT2mYe4SNm3OdETdx16IMLZgQ+YRO3WmKnoj05ZdN1ap284gJ/nzTcr8G19Z2CPW/L61UIpiO42QdQB1gek40= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=LCaId1Mh; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="LCaId1Mh" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-1fc60ef3076so25729975ad.1 for ; Fri, 09 Aug 2024 12:43:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232621; x=1723837421; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=7EQAoYr/rc8nw+x53hvnHC7R0vSXW8+m/U14Qwxr4YU=; b=LCaId1Mh/cfaZ1A3flIXbMN83yBaKr3eYyJEIzeWrB9ivmSxDZc+VvMaDztmkpgr+q IdtIZrzYDAjomWisZ/GQgXsnJ2RxFlGQp400ic94KFn3V1GkU8cMXITkZFSi1CYETvNO eslSjnmvX3Gj8rLGgXN7ugHWhNACs36OiQo2pTzCcg095YFxQQClYPbo9USpdyVBHTmS L9b+E2cFQ6RB4w5N+v2/Jln6IXLEsoKEinkMSazH2Kn2OQWqYfiKyLAVrGkpivH1bZQ9 dIBIQGc3YW+kRLmcI2UXk01/cFwcNdb4N5S+kbP6KKvJsGuWn0OnUhnlunOXCMJ/n7ZR uuWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232621; x=1723837421; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=7EQAoYr/rc8nw+x53hvnHC7R0vSXW8+m/U14Qwxr4YU=; b=sRyz4KyaRc7iTW9caRfSIEGgmOIjRqHsW3w3fOvkslQMg8cdMX1Js3W/GpIdEV3c4r prT2zYPwry6S9Yh7mCRgwJHGyviGV7G8opC0GfoLHhgtmFURIXgzBZnY4i1w6xNUsHDs hu8BDmV9LXYjJiyqPWNCTkyiUAKAqP729LlKlhoqDhQUyoZ/mNyGZs0qFWS6Agf0lteD 7HD0pcLCDAc/bf1TSpcdE+5031ZOMtFQlwfWFqonJQXARyn6KGzcfWxSMo9e0N2+hzi8 mB24BtMPxFhePShKJ91fq0zfeGA+48c7nRBOaKvY9b3gV3U1MDmL7zq37/0fbslB5hDT 9WEw== X-Forwarded-Encrypted: i=1; AJvYcCUT6JhU2n/fWcVfP7mjawMxc4AD9ANgWyrxOprfbptYkF0STHdyidW7PIDMyUl96dFtCpx7aAy0Z86eBotWFt0w+Gy6eg7EwSbrbvIU X-Gm-Message-State: AOJu0Yyv0dw/TIJpsAjy2EGyeVczS36Z8P07NAQfq60kGn/W1YZRXqcs Uoa/44k+tXD5p5n0wqjWOUhIZwTGLDpiB1YY4/AE+sn/S2lwChKTE351WFm+J7CrCDs8hU6qTkT g+w== X-Google-Smtp-Source: AGHT+IFZbM8cukPN8xrEPWCmEvmIVa0hspDzfqR9/uQbMDXcIueCj/k5JgcFqB326LDruTDoIrGeiyLQyKs= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:903:2342:b0:1fb:56c8:f273 with SMTP id d9443c01a7336-200ae4d640bmr1732425ad.1.1723232621226; Fri, 09 Aug 2024 12:43:41 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:13 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-2-seanjc@google.com> Subject: [PATCH 01/22] KVM: selftests: Check for a potential unhandled exception iff KVM_RUN succeeded From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Don't check for an unhandled exception if KVM_RUN failed, e.g. if it returned errno=3DEFAULT, as reporting unhandled exceptions is done via a ucall, i.e. requires KVM_RUN to exit cleanly. Theoretically, checking for a ucall on a failed KVM_RUN could get a false positive, e.g. if there were stale data in vcpu->run from a previous exit. Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/lib/kvm_util.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/sel= ftests/kvm/lib/kvm_util.c index 56b170b725b3..0e25011d9b51 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -1719,7 +1719,8 @@ int _vcpu_run(struct kvm_vcpu *vcpu) rc =3D __vcpu_run(vcpu); } while (rc =3D=3D -1 && errno =3D=3D EINTR); =20 - assert_on_unhandled_exception(vcpu); + if (!rc) + assert_on_unhandled_exception(vcpu); =20 return rc; } --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 15AD41662FE for ; Fri, 9 Aug 2024 19:43:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232625; cv=none; b=p1A7RzGzYSxObY0B/sBooEtDKc1a+AWl0Qz4nJzEUA/n8ByRmlqGtlYKvtNTshTc6qw6zlNGo27DzQuyGMKTu8PNrsq1Piy9gRm0WCIBH/T7k8ua9idqbE9INKY4VjJQYteq+BhA7HWOxncXAPjxzSp2lPsac5om33kXq7FVBCo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232625; c=relaxed/simple; bh=ht4r0fbuo5n7i8loKFQk+Ce1717qM4Ak8JQynsJUkR0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=g7zRfYaaJL2mPkMu5f8BeE9GvR29kmOiEUbJa6c4lXX+wOyXqdE5LjaYiHOMRLaHUJOnIJ5T0NJJTecE8VEoYAnCuZb8s5awyNKFMP4uRViUwF6SW0qTaMkIf7Dn2qrZA722zxr3eczSzBSfWAjIqNspBczM3Om9qzOK0sxbRTQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=voYt1Ial; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="voYt1Ial" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e0be470f76bso4061835276.3 for ; Fri, 09 Aug 2024 12:43:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232623; x=1723837423; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=D+L1OtzzgTK4mUsnsbZ07IxjRBCo4OaeWvC+DT5ZIuI=; b=voYt1Ialpznoltp+hT0r4u1JwjHZY1y6xSPgveyofeQ5rOrSTAn9PM/SZUcxvf3mXa OwXnZBTpRwL9oVbaed1gH8oLH6gs1OmqFrgbu3PXhfYmedF5wsmXxZxRivSHNJwhg4mu 4aFkpsfbQwmVBD3B1orVO6Bfu83R8oe3vGrxeggaz58ewESOpmAObo9dL1Sh5bpw1Rv0 aYN0SBcOHO/oyROXa9T4r95ujEyIV69d3UeGRvvnx0kLqaaBh5626FSYDX8YTv7UGFCX 0XWFUOAOTycrBthgxv/mHQJgwWfvi9b/u9Daw7Cp3hvxj+3EU4wZpcNaWd6HnHxWfjSY T7Dw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232623; x=1723837423; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=D+L1OtzzgTK4mUsnsbZ07IxjRBCo4OaeWvC+DT5ZIuI=; b=cqIYtM+1YleFPvHBYwSJhn5eBaPEIwwU26Bo2mlsp+vkJhrqA9Xh3eXbZFPm2buTOl 2CRrnDEYIyHMqA2VQd1PFhiXcxvZ3BI+VA/KJeCWG/lM5Noo9JcinFL0I7ozRwH/Nl24 n7SXHZmbSxuvMXhpib79G5V97JeS1mmUxFasJlblX/kFiOJVdm2nS4gUyUp6pZjL97aG uylZG8QTgDcZyFIQbK8WtjxaSTcuIVH8psUJcKdHlm0PKLSGm06N+6fgt5hnK2q3zsg4 aJUmptQB1FCCj+NDtyR3xcDHDDeTEjSK1FInZ+FC5zw55cOb6mvI8kilY3egkg0lnQgB 1vLw== X-Forwarded-Encrypted: i=1; AJvYcCVN5dzThQdhC3Wly/xW0rA5TnNesVYDRTv397q9bkmFDf9T20xCvTpUmlKzOEhbA0mHUQRy1KzXr8tP+BZTGgkdA63wais+p1WPNu5D X-Gm-Message-State: AOJu0YwbDrNmaFA63dAKMsYsA6sw5PbY4ynZ8oEWUjRkJP6ZFLT6xra/ xuEyMWgAjI9cXtkIqY/zPAKR5eoFgizLQlPaLktmWQwabRtKavw0jUhvxSjghtU+KtAf3MvSVVY /lg== X-Google-Smtp-Source: AGHT+IEKeKIaPW8YLP4xWw2FBfo2QHuCebzeo0dHE5fX5nmyiW5fDNnzekf775+ZUqeRP6mPt7GOOOJZQOw= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6902:f0a:b0:e08:6c33:7334 with SMTP id 3f1490d57ef6-e0eb9a10ac7mr19255276.8.1723232622954; Fri, 09 Aug 2024 12:43:42 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:14 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-3-seanjc@google.com> Subject: [PATCH 02/22] KVM: selftests: Rename max_guest_memory_test to mmu_stress_test From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Rename max_guest_memory_test to mmu_stress_test so that the name isn't horribly misleading when future changes extend the test to verify things like mprotect() interactions, and because the test is useful even when its configured to populate far less than the maximum amount of guest memory. Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/Makefile | 2 +- .../kvm/{max_guest_memory_test.c =3D> mmu_stress_test.c} | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename tools/testing/selftests/kvm/{max_guest_memory_test.c =3D> mmu_stres= s_test.c} (100%) diff --git a/tools/testing/selftests/kvm/Makefile b/tools/testing/selftests= /kvm/Makefile index b084ba2262a0..cc25f19b62da 100644 --- a/tools/testing/selftests/kvm/Makefile +++ b/tools/testing/selftests/kvm/Makefile @@ -138,7 +138,7 @@ TEST_GEN_PROGS_x86_64 +=3D guest_print_test TEST_GEN_PROGS_x86_64 +=3D hardware_disable_test TEST_GEN_PROGS_x86_64 +=3D kvm_create_max_vcpus TEST_GEN_PROGS_x86_64 +=3D kvm_page_table_test -TEST_GEN_PROGS_x86_64 +=3D max_guest_memory_test +TEST_GEN_PROGS_x86_64 +=3D mmu_stress_test TEST_GEN_PROGS_x86_64 +=3D memslot_modification_stress_test TEST_GEN_PROGS_x86_64 +=3D memslot_perf_test TEST_GEN_PROGS_x86_64 +=3D rseq_test diff --git a/tools/testing/selftests/kvm/max_guest_memory_test.c b/tools/te= sting/selftests/kvm/mmu_stress_test.c similarity index 100% rename from tools/testing/selftests/kvm/max_guest_memory_test.c rename to tools/testing/selftests/kvm/mmu_stress_test.c --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 19A831684BA for ; Fri, 9 Aug 2024 19:43:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232627; cv=none; b=glufCrXkQ9pAu/9pT0VZJUBNfmjoaZvDhpHDIZZgz8JCqI388A3KPY8zokrFMdJFABlkisZojFV7HEJGQZ/5kg+eCczQQ5nPvtXitmXixpbQElJZ85qN+tpc+Pb5bJAqgwT+MdV9abAma0OC6qDtc8cxIJQiREgNruN/nnhyWyw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232627; c=relaxed/simple; bh=i8/Z9P+b877HKYf3foOn3WZsd81L9lIIe5UnyEoK280=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=VQf0laUYwgYsdjABYtFiPzSZ1+GffaathB+U2nhHpcScY9UTpvuFac9CMmNIBryZOMZxl17+dbogTldvhEXEuoEWAIXxu82CC4T2GJ1SAPKQ7xMVaZisprnP/H43ryULUYldHMHE6XixX1nl153aNPTnBwR45voSaSfFF0MSn1c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=2+mwTNZ+; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="2+mwTNZ+" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2cb5594a9f3so2735499a91.1 for ; Fri, 09 Aug 2024 12:43:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232625; x=1723837425; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=ZhbtdXG7GxIZJlUwxv6/PwIOZfF88Cf7tN+DbOKZy1o=; b=2+mwTNZ+PaQ22lVNLKw4Jx9yQ9nbPlrzD0GvqMLNbAJH+jPnGWDPgdyTRr3GNDIHUt vjAmVRB3ZE9KMFwZ81pKpHzIodwGcckOLyja9jacLw+Mq94dED4USn1ZNHz1uY8DpOZo 79Gm5nrinALn9DsjNSu+G+2/5y2jfixyD2z+ACQzrJwWL1VpDEFdhVRFjTUlVhDv9Jdi FCFDX4jVuPhDDwtZWG36dgr6WV9tDkbZTKczflJBH3QVFlvXbRQFkHW9P7a5Jjq5Kp08 mJoihL6PEwLRpNWFKMI6QfKqkyz8JjrYtc3typ3EN1hW5HrIWQ8u2y0wCi84JuItigKh G1XA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232625; x=1723837425; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ZhbtdXG7GxIZJlUwxv6/PwIOZfF88Cf7tN+DbOKZy1o=; b=RsN76qCjQ9EYyPULcVNf721J7vxTWVOqtWf6WmxL97mYaxXIJfahOZfRdw6ENU2ph2 9GAmk3UvkC4kDjYji5uMW4H8xddS1XYJkLsIH8DWAAfg+QEXnEgVEFZwq9SsXtsh7kqP 1ymxFpWHh2SBA5OrNuSin7CwiYySgJKzvX+lWrd9JjfHuZv/CmoMlfHdgA4ST+PSJ24N E8U+Yv8WvdVz8ogKNhCxqveQqtvh22CIQvONuOvm7u21W1sWaZcmq6xaJdm1kWuyHWgf mfxh0m48Rts/Rs90qqbsPBsZWie9PgSbZclDYQjdj0L9Tiov7iR8ZZSPbrIhGIFXGtHP oaUA== X-Forwarded-Encrypted: i=1; AJvYcCVDOpRSVTfJ+uLoC5Vb7IjLttitArzTr0zpHBaLQQIxbLHcKH08aWu3hPbUPrqv2nigk3w7lBLUPwe+m4R4dWe94PK10/GdX6bODi/a X-Gm-Message-State: AOJu0YxDvsdyVH/Tw0AKMXjVikcfdNHh55MNLa3kl2btUa1aYuCgIb8v 5eni2d/tXd0xc6WFvDylMTcpKgPSp0fqd6cVKWwgZOtOlvIrSTDot3JtiTRWCLs+RlVJtMPPZrF Nlw== X-Google-Smtp-Source: AGHT+IEJSGntFoEairVQfIondbRS/wPjqzJ9gWC9vEesMP0+8PhPR1ut4xtIPVXbgE0J+qJ9SIwm32dkTPA= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:90b:f81:b0:2c9:81a2:e8e4 with SMTP id 98e67ed59e1d1-2d1e80605e9mr5545a91.5.1723232625089; Fri, 09 Aug 2024 12:43:45 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:15 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-4-seanjc@google.com> Subject: [PATCH 03/22] KVM: selftests: Only muck with SREGS on x86 in mmu_stress_test From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Try to get/set SREGS in mmu_stress_test only when running on x86, as the ioctls are supported only by x86 and PPC, and the latter doesn't yet support KVM selftests. Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/mmu_stress_test.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/kvm/mmu_stress_test.c b/tools/testing/= selftests/kvm/mmu_stress_test.c index 0b9678858b6d..847da23ec1b1 100644 --- a/tools/testing/selftests/kvm/mmu_stress_test.c +++ b/tools/testing/selftests/kvm/mmu_stress_test.c @@ -59,10 +59,10 @@ static void run_vcpu(struct kvm_vcpu *vcpu) =20 static void *vcpu_worker(void *data) { + struct kvm_sregs __maybe_unused sregs; struct vcpu_info *info =3D data; struct kvm_vcpu *vcpu =3D info->vcpu; struct kvm_vm *vm =3D vcpu->vm; - struct kvm_sregs sregs; =20 vcpu_args_set(vcpu, 3, info->start_gpa, info->end_gpa, vm->page_size); =20 @@ -70,12 +70,12 @@ static void *vcpu_worker(void *data) =20 run_vcpu(vcpu); rendezvous_with_boss(); +#ifdef __x86_64__ vcpu_sregs_get(vcpu, &sregs); -#ifdef __x86_64__ /* Toggle CR0.WP to trigger a MMU context reset. */ sregs.cr0 ^=3D X86_CR0_WP; -#endif vcpu_sregs_set(vcpu, &sregs); +#endif rendezvous_with_boss(); =20 run_vcpu(vcpu); --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7BAA316A95A for ; Fri, 9 Aug 2024 19:43:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232628; cv=none; b=S0Qif9KKTOYpgnu/isdRBMVDVhxJ9+3sGcYB12d+U2mT9XB6hmk76rck8jmXx3AQrAHjWGiWYWEfPYiEUZUIOrPKBWN7SwPhClS1cKeN9ZNtAvBXx02p8BSzsHEsiUocGLbCYbksvwaENvQUyNVvoXfxk9L1w+mbVhttp+PbLnM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232628; c=relaxed/simple; bh=2ubMLB5DYODq+qnXHnLMuf0I7PP+FeHp7xjE64E2dt8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=IMzy3cgsWpbxB5RAse6aSLQeQoWdOvdmq9LgoEqkKwekbiHKP3iyPui0rFfuVRAF8+ju0bp+jQQ22BalazNRwssb3nyd3KJqYf4j8K8diLHbtF7TT03I6OF/EO/6urMPRmtUeTm/djy/v40zeaIV11CQPl6CCCQNzGo8S4oCaX0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=uho2lfO3; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="uho2lfO3" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-7106fcb5543so2040728b3a.2 for ; Fri, 09 Aug 2024 12:43:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232627; x=1723837427; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=RH8NStSlBVyK9g+UsmyXuI40nq9wzYPRU0gcFL0VkzM=; b=uho2lfO3JEKa3BYwurP6bC28fXllFmtwsLtKqkYwXKuVhG9iGLmF2aaPR5Odju1Ebl Z9NS24VN5EgBnwC8jBTgeHCFw2AsVyYNmazjHtqug5NluFgb9zdsyV3lRidSK7IbdXzO SCBUzgu7UERbgaLMluS8YD20m2GpAIJeCkYKVxXWoViB3+m1ymd9pJ3TY9AQlWsLi3mG ukBmfSIkISSc7wZvKIuD2A98IXIKxXMdV0Wo3kFxs+Vue5h2yajrz24/2pvUgYvrWhhP qXu6rPM4PbVbmOA+Q0UCHXc5EYZRTr0eYAybkGBmS0e7gFq+rVkpau1v/mDziY9M4tpz B86g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232627; x=1723837427; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=RH8NStSlBVyK9g+UsmyXuI40nq9wzYPRU0gcFL0VkzM=; b=WMn6mb/ECl1OZGfxHXAdlHG4lQ3EQZKl+HmQlYqYo7Oez7FH9+hFpGsb0CJhS1gUky BfAsGTn3z9wSgkx0hmqJydWfHLgq2k7+Uxf0rVnb8EaeQIvxw4moCAyao3FWFPfVytBn bDn69W4G/qetMIV6Zq0YlldM9fyX/veSn7aQHeh2oVarheIiN/RkSY3OHaMoafFT7qsk vkjORLdtLkDcNP1AounxZGtDq+NSfwARPDWjVsitzpSgvMm88jmErpP+HwD/4P1DmDTw H7LR51EGm5i9wEyienHJrcvy9gmM239FlQTVXYPW/vAx6Txd3zSzkUz6gg6F/MEJRTXU nGJw== X-Forwarded-Encrypted: i=1; AJvYcCW+/fIG0GNKe6ZJg9W1syf7k1desAxOZaYcZNkwlzOS1JV6IKO+7Ii+jj7t7+MsNYwU605z4hYYXTzQ0CiCry6pimLfakERL1G75KDj X-Gm-Message-State: AOJu0YzTsWS5Y8sxIAYKALPzn2zzRW674th49gWt82rhUrBaDJ6VrA8W 2fSKzPfma5WYJb1nGveddA+yRfpw9hHwOoiFKgoRycW/ZVL+Y1Nj5VYKWEN5eL2hqz8FiJpIKdn U6w== X-Google-Smtp-Source: AGHT+IFF/ikXFYmI7zWUXbZoAyAwoVgAHirCcT99AraR5wR32CSYiy6TQQF1iEJF/4Gcjt4kmzWk21e9a/Y= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6a00:6f12:b0:70d:30a8:abaa with SMTP id d2e1a72fcca58-710dcb31a13mr28290b3a.5.1723232626683; Fri, 09 Aug 2024 12:43:46 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:16 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-5-seanjc@google.com> Subject: [PATCH 04/22] KVM: selftests: Compute number of extra pages needed in mmu_stress_test From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Create mmu_stress_tests's VM with the correct number of extra pages needed to map all of memory in the guest. The bug hasn't been noticed before as the test currently runs only on x86, which maps guest memory with 1GiB pages, i.e. doesn't need much memory in the guest for page tables. Signed-off-by: Sean Christopherson Reviewed-by: James Houghton --- tools/testing/selftests/kvm/mmu_stress_test.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/kvm/mmu_stress_test.c b/tools/testing/= selftests/kvm/mmu_stress_test.c index 847da23ec1b1..5467b12f5903 100644 --- a/tools/testing/selftests/kvm/mmu_stress_test.c +++ b/tools/testing/selftests/kvm/mmu_stress_test.c @@ -209,7 +209,13 @@ int main(int argc, char *argv[]) vcpus =3D malloc(nr_vcpus * sizeof(*vcpus)); TEST_ASSERT(vcpus, "Failed to allocate vCPU array"); =20 - vm =3D vm_create_with_vcpus(nr_vcpus, guest_code, vcpus); + vm =3D __vm_create_with_vcpus(VM_SHAPE_DEFAULT, nr_vcpus, +#ifdef __x86_64__ + max_mem / SZ_1G, +#else + max_mem / vm_guest_mode_params[VM_MODE_DEFAULT].page_size, +#endif + guest_code, vcpus); =20 max_gpa =3D vm->max_gfn << vm->page_shift; TEST_ASSERT(max_gpa > (4 * slot_size), "MAXPHYADDR <4gb "); --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8AF1616D4F8 for ; Fri, 9 Aug 2024 19:43:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232630; cv=none; b=Kei3t4kXi5Kb7jaUpOu1MAQU1Q20C+fdQgIBVKMF8tXlUCt5GDduip4M53dA0iAfUKrHntOc/ZSb+jN/JizBKmJrq7bywYvFcbFRDW3qqTo+/CVZhuiWZjpev1REnTeqXYMAzDcbgX3azAUnMQdeaDn/aE1aS7BAsoOVkOmHZnI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232630; c=relaxed/simple; bh=LVFq/UF8yA/HGcn66G7X9qPCu7tUDBlRE4IGgs3dGtA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=abkEfyapnOwTL2UZ9ecf2p51u+US9Vsxq8iBvkq0EElKc2mpfIOY22pXTPcyPFWJymsEwXBEs53+AxHKfWQciKGJcLnljkWprfcV+PHoc3ujt4jzvQCL2gz17oJMti2B7pvUxD6hs7xDAkwMzq84k/CPnofLd4XI0BUMNd4/cx0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=4NU4x8JE; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="4NU4x8JE" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-7c3da1ac936so144068a12.2 for ; Fri, 09 Aug 2024 12:43:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232629; x=1723837429; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=SWkGT60Gokzaw7JUaOakLjtXVj6aQ6BddU74vUMlOq4=; b=4NU4x8JERO+C70hLXuiNZ493SWVQUvaPJSy0P/sZKZUWmW+DkZeJWFA/nX7Axu2bxF F6KJs7VmmEJUAwxuJavd4YOQii+X7qWhRew3ld0Xewk/0NUlXi/8snlu1UI2SgbVneOz 6ezwWRyJjGuDw4Tn7wR+q9FgaHAiOtP5UxW72NWDIfr//rLW6QA4Q+NjOUbmq9NQrgC6 dHSePuhHidAyoeNkAxw1wybz/fA/Q+zJoUGQeqNX/nYkgGm+acNKooqQAegVc4e1zTy4 LOH60kpWare2laFB7anvkw++H3Y1AX+sYhIVUA68QGxPOpoWGdTOKYQ5q/KgDtv7nr0d 0M2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232629; x=1723837429; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=SWkGT60Gokzaw7JUaOakLjtXVj6aQ6BddU74vUMlOq4=; b=caS+YGTqpOZUWpVlHGtoBDjejGKsZT8YoU84tl2tLHSWWK4DjvRdEUpxV8W/8VDiLb /an0v3BOSh8b5hCn51jUKvgZRuQDs3G4Ik8pQK8Ivrj1g4ktX8mZTH/o0KVHiLOsmd9s j5osgVMNWu7Qu1cFwmKWMsNILTGTHgtO2ey7Ya+cJs3Rn21KpK35mJLj3oSv9fwvbOah uYLOfxUmbnWIzeXlbHi/X9NnQYZ4yeiDLP3KA9VGNCyisikRpvhEpzCjJyB0nH6x5xmo eHUOY5Wto0P/UmoNnd7sL4TXLwwvOr/47ZA4HlZbZmapJQ1EyOwG1vxO1kPaIiT1LgmF EptQ== X-Forwarded-Encrypted: i=1; AJvYcCXBInDLUhKBtuNhe6QTxwLBkvGRhE7q8PJ7cdTREMfTSd/IVgv1A+Aj/rxN/ZIotPYQ86gsMUAnyrXCpV0=@vger.kernel.org X-Gm-Message-State: AOJu0YxqeN4zkOiDnpbxjEgOPi2bna3SI6NHsHSbC44h4sZYs/xV+YCn wOAJxuOTxjkqjj3wHe6efJhw5L7EHo3+pFYTkhfDhLN6OeJqaWCg4jSBzwpPxDUIzTiaHDh7Joj ZxQ== X-Google-Smtp-Source: AGHT+IFmZCXNvXRh9QF9VYGRkhlcjb88/DI8HKkyVC2+5y45yUvR2nURAEut79pR9l0FdvfAx89nEJfuzPI= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:d482:b0:200:62bb:e22c with SMTP id d9443c01a7336-200ae5d8874mr822625ad.12.1723232628645; Fri, 09 Aug 2024 12:43:48 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:17 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-6-seanjc@google.com> Subject: [PATCH 05/22] KVM: selftests: Enable mmu_stress_test on arm64 From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Enable the mmu_stress_test on arm64. The intent was to enable the test across all architectures when it was first added, but a few goofs made it unrunnable on !x86. Now that those goofs are fixed, at least for arm64, enable the test. Cc: Oliver Upton Cc: Marc Zyngier Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/Makefile | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/testing/selftests/kvm/Makefile b/tools/testing/selftests= /kvm/Makefile index cc25f19b62da..4198df29503f 100644 --- a/tools/testing/selftests/kvm/Makefile +++ b/tools/testing/selftests/kvm/Makefile @@ -174,6 +174,7 @@ TEST_GEN_PROGS_aarch64 +=3D kvm_create_max_vcpus TEST_GEN_PROGS_aarch64 +=3D kvm_page_table_test TEST_GEN_PROGS_aarch64 +=3D memslot_modification_stress_test TEST_GEN_PROGS_aarch64 +=3D memslot_perf_test +TEST_GEN_PROGS_aarch64 +=3D mmu_stress_test TEST_GEN_PROGS_aarch64 +=3D rseq_test TEST_GEN_PROGS_aarch64 +=3D set_memory_region_test TEST_GEN_PROGS_aarch64 +=3D steal_time --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3539D16D9DF for ; Fri, 9 Aug 2024 19:43:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232632; cv=none; b=ZsjsEVGDM78oknZ/Tn7K9YRTbDmTz+ddJQ1whfqG46UgQSE4irSHK/S2+tXRaZ599QhlAWlbpa3XNgFtLe0CG22ufYMwduyXIYsd5lWpn4yzB5+nj42bL3INKq07KLL03g/2fajf7uo/cT3dsJl0j9Zo8r3np93sySfRDtkCB9Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232632; c=relaxed/simple; bh=WEKb1xlIGs0k4iOZ5Yx5UQ9S4hkU1zI6zoLoK8JwUP8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=j4SEkU2EXqiIURcNgxbx8OXoaKI4IjW/ZIo9P65b/wc0ttbFowRIorbZF81K+kF1Nxv0cEPf0QW0sgwxCWKjZqCYyaIy9C02PJO0yl5kD63RZ27+DyqE4GUCHWdKIO1ug8FyG0j2To8IwNg8wYkOwFC+qFRyyXGycILcpBNcG8Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=4cGYP35V; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="4cGYP35V" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-79028eac001so2560129a12.0 for ; Fri, 09 Aug 2024 12:43:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232630; x=1723837430; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=qOWEA+gf18ibEg+1IgBuUDOlia9YtX+EopmbGWf1wCE=; b=4cGYP35V9J9edTKAVY6ReehrY14PvgOdHA6GXtdUdpGI4d4RmtIWAj6+uHnGIZnNl4 7WlH1DZd57Ia6pMgOrgJsgmvjsDijx6MwJR33tcIT98ZwOz9W0qkao+m2DuwYoAQdMwu WgpJe9kKQMm9Kmcz2gfjfXPRXJsgZ1Gk7K5LWurw9AQHRck7waZVmWxw4/JuQQWVYhec XdLoc/hIYfcPNQkrHjks3Rrgsp1suuE2ez8Lhxvd2bddXJumBvVSeFS8FzQtFb5S3LaP P6kPeeZ2eGN0Ten0EpFhF/PqrPiubwOg73pyOPANOwCkuh6otJXQOUoKzhu/QMWm+nh4 oW4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232630; x=1723837430; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=qOWEA+gf18ibEg+1IgBuUDOlia9YtX+EopmbGWf1wCE=; b=Ho9po2GhnO4zKWSOI4GFTuy2pKG2egMJewg+kpQ73mIxOg1JFH6qb+wdns0rs8iflP /dCPqh0JE0Sm3JfH6UZ/Xkgh14mjvs6TKcgyH2GBrWntQ0jjzbPpsedm3B23O4lTfHeV 1wMPCAGcyCnskgOINxyRHUVx0a+deCUnSkdnyj3bBM5p0hFmEIOmhi6YFSNLN/B6yUA+ O+gcQDMooqyF8HY2cROtRG9gVhO296MpFyq4ukPDJupTXltkPBJsw7JUlEQXPHHZQv3x 9i+gAwu3JpHlPQRPTVLlG1sXQ1u1MBtZ8bHba4q7yY+i3ZlaFyZBnX71oiKAMu10mZw/ XUkQ== X-Forwarded-Encrypted: i=1; AJvYcCVIOSdDePhpHGfWlZrqPNjxOYiZ9aj45wyADADTl8FH5cSqvbJ4MFr9xmM5/Fb2wPtkf9Xp0FXLBR85ssUF0EmY1uEj+PP79fZX9AXs X-Gm-Message-State: AOJu0YwUUU/f4S28a6zaIUtNfiBOdbY1q7paHoewOnbkb/E6Mwev/N2R SQ2buTb6iPCunwFKFde/zvbxBI+9iGqK86z0MUu3rW3VjmSzI+NOGKa0+PFettvuNiKqqKgBbD8 ncw== X-Google-Smtp-Source: AGHT+IEN/4gRYyT6fFtEw5/PSlX3VvEJMGy4wvRZ57TDQ5LWVk26M6iloQiU8f0vV2+Ypg0y5mNrr4U+8Bc= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a63:6f49:0:b0:6e6:a607:19d4 with SMTP id 41be03b00d2f7-7c3d2c7fa08mr4698a12.9.1723232630436; Fri, 09 Aug 2024 12:43:50 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:18 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-7-seanjc@google.com> Subject: [PATCH 06/22] KVM: selftests: Use vcpu_arch_put_guest() in mmu_stress_test From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use vcpu_arch_put_guest() to write memory from the guest in mmu_stress_test as an easy way to provide a bit of extra coverage. Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/mmu_stress_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/kvm/mmu_stress_test.c b/tools/testing/= selftests/kvm/mmu_stress_test.c index 5467b12f5903..80863e8290db 100644 --- a/tools/testing/selftests/kvm/mmu_stress_test.c +++ b/tools/testing/selftests/kvm/mmu_stress_test.c @@ -22,7 +22,7 @@ static void guest_code(uint64_t start_gpa, uint64_t end_g= pa, uint64_t stride) =20 for (;;) { for (gpa =3D start_gpa; gpa < end_gpa; gpa +=3D stride) - *((volatile uint64_t *)gpa) =3D gpa; + vcpu_arch_put_guest(*((volatile uint64_t *)gpa), gpa); GUEST_SYNC(0); } } --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F38416DED3 for ; Fri, 9 Aug 2024 19:43:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232634; cv=none; b=thhR/1SZf5BRKTv6YZYbWjvYRkazY7DJqRi1+24m8n24g5qeCfSWqvk4NDaKefbEEKluusbs5h5+AwW47RPrBrQpMvR9q0IQEKF0M1U/JUnGVEn4xPv5xtudV1A2cHiqH30pnyDBTdMF+MCJRbeFxNyo73idEhy3n4HV2EPc7aI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232634; c=relaxed/simple; bh=oqLsHduvmYrRBvhT430Njd0wP4bVpgHOh7c0T70VHPc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=kHOlmGERieW0UnYTwDO6IGChEbJY1a1TgYKPEDjc9aGGkQNbbKCNFtD7CvUJ/xQ225hCnRrXCIfqpt31xKJ4F0Lgsy3Gl1qAmVxvtsOrZDJn0ZQGGHeSTGMcI5H0ioP+ODyKKJfmeTuOiaxQQqLW6s8PM63R1MDMJKXySHgvqPU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=3G3qEOen; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="3G3qEOen" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-1fc5651e888so22885125ad.0 for ; Fri, 09 Aug 2024 12:43:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232632; x=1723837432; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=Cvey1ZPxH+FTyjcmf9WeApEHMp7j68Ugrd4yPwlOS4s=; b=3G3qEOen1ah2pii+2eNgPqGvrKcUEnJVE0wQ8m3axxVsmLNGatC5zxOV4DBk5fmf7z POhd0EXVYHSm7I+1KvHCGagkHkJVBRHvv8e6bWDdfGbeJEq3KnYmLbLyZLJhPwkmG73P fILFf06DLCT0ZpGVfv7bRjclIpMwhKGFx5hE5jRx6PSBkA5mccvyVz7FjV85qNrbVO35 woRAHl/MFwTCmjDlMDRB0r/sXUMYoVrR0ie6/CQzZjJt8lj/TA81RFsB+8LSDNd2Sfdu +hGF4odoebHk8n6hCDpJB+zZQOUZuZbiw2iQAAi3PChVP/mTmy6gGmDTY/PHiXZKkCnd rJuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232632; x=1723837432; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Cvey1ZPxH+FTyjcmf9WeApEHMp7j68Ugrd4yPwlOS4s=; b=OlIt0ew6vaiZcIrQTUZXewCAtvGWNNOL0AIRx9tKInLIEL6bDXJ3RV4iuCc275JKcW IjQOCr4EZdOpDVvOfibDu8iX2aBGhnGKvXsrzB1VQwsDOtyMzkAidGbZcObyiRV0Z98c R7ntbLogOgYpXdklDFGkQF+EoIwIr22xwScACThNbaSewi9NrpaVm4z4HFHt5NbYYgqW gByGLF8I+KK1SRR8tJR6DHBxU+zFJ0EGJUGoXVYzzIgmFC+Zxk5FBJ0CpAJ5zEIorUFV NeBqkuEQO1Nuq8lZB2MxbDZ7M2miJi3pyT3MfnubMQ3MdIX/+GzuJrvM7c5IKkkzIBmw Zpog== X-Forwarded-Encrypted: i=1; AJvYcCXKgd5msnP+2cd7TbCpWo1GINDxi5EWdLQjx+x66eiWcjmgcwYNoJ28FcZp93hTls2smvaXB6wXNNdBELM6gy0TB7+odfMZnWenb6rF X-Gm-Message-State: AOJu0Yzuv/b+i47ns59cE2UZrIBdnkNH9QCuL8ENIWEKgXqzGB0WXFmv 2OUDLjAREbYgLdBF9gdil4vXRVvzUzddl2qnRimDFTiGvUyAkMG6q3CwjYia0J/L+FjIEQZXowO cFg== X-Google-Smtp-Source: AGHT+IH+toPWCiPDFAxpzyjZiz0XzGzpUA6+VQs+q838rMwjQmsoPXGyEJ4VZpEN+6XxrI30IFrHifQqDGo= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:903:2352:b0:1f7:3bb3:abe2 with SMTP id d9443c01a7336-200ae5eee52mr1182405ad.12.1723232632391; Fri, 09 Aug 2024 12:43:52 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:19 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-8-seanjc@google.com> Subject: [PATCH 07/22] KVM: selftests: Precisely limit the number of guest loops in mmu_stress_test From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Run the exact number of guest loops required in mmu_stress_test instead of looping indefinitely in anticipation of adding more stages that run different code (e.g. reads instead of writes). Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/mmu_stress_test.c | 25 ++++++++++++++----- 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/tools/testing/selftests/kvm/mmu_stress_test.c b/tools/testing/= selftests/kvm/mmu_stress_test.c index 80863e8290db..9573ed0e696d 100644 --- a/tools/testing/selftests/kvm/mmu_stress_test.c +++ b/tools/testing/selftests/kvm/mmu_stress_test.c @@ -19,12 +19,15 @@ static void guest_code(uint64_t start_gpa, uint64_t end_gpa, uint64_t stri= de) { uint64_t gpa; + int i; =20 - for (;;) { + for (i =3D 0; i < 2; i++) { for (gpa =3D start_gpa; gpa < end_gpa; gpa +=3D stride) vcpu_arch_put_guest(*((volatile uint64_t *)gpa), gpa); - GUEST_SYNC(0); + GUEST_SYNC(i); } + + GUEST_ASSERT(0); } =20 struct vcpu_info { @@ -51,10 +54,18 @@ static void rendezvous_with_boss(void) } } =20 -static void run_vcpu(struct kvm_vcpu *vcpu) +static void assert_sync_stage(struct kvm_vcpu *vcpu, int stage) +{ + struct ucall uc; + + TEST_ASSERT_EQ(get_ucall(vcpu, &uc), UCALL_SYNC); + TEST_ASSERT_EQ(uc.args[1], stage); +} + +static void run_vcpu(struct kvm_vcpu *vcpu, int stage) { vcpu_run(vcpu); - TEST_ASSERT_EQ(get_ucall(vcpu, NULL), UCALL_SYNC); + assert_sync_stage(vcpu, stage); } =20 static void *vcpu_worker(void *data) @@ -68,7 +79,8 @@ static void *vcpu_worker(void *data) =20 rendezvous_with_boss(); =20 - run_vcpu(vcpu); + /* Stage 0, write all of guest memory. */ + run_vcpu(vcpu, 0); rendezvous_with_boss(); #ifdef __x86_64__ vcpu_sregs_get(vcpu, &sregs); @@ -78,7 +90,8 @@ static void *vcpu_worker(void *data) #endif rendezvous_with_boss(); =20 - run_vcpu(vcpu); + /* Stage 1, re-write all of guest memory. */ + run_vcpu(vcpu, 1); rendezvous_with_boss(); =20 return NULL; --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3E88216F0FE for ; Fri, 9 Aug 2024 19:43:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232636; cv=none; b=Ao0H949oydYnaNiub3Zh1U+sqwwOo46+g5XZXUh+nKE/3X08UrRWmf0zUbRbp1+fqvSknMe8RdaFJyfqUWXMD7NOc5MU55Y+/s2L+uv9XqKNqVEili5n3WxSkOrZ0SNHps8+icUitWnTU5uRiz2Zwek5uQuzMNIiqhhhgAI0gCo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232636; c=relaxed/simple; bh=JMLriyucuH1C15GtvvibdoIfA8+6so79QMDBwKK8/UQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=HFPNlBpSGeHyBOpxwYG53wYlNrBmfnxpwfBd+mmkyi0BQa2I72oUzUtD/+oDQWcsKG/8/a4VYr4+Wl23R/MMfvdkymBqS7LTWAaeOC4gG0/MZ0IZ3hQiYtM9H09wu2JJ2oo0yDq0uy5f+cGfyxI32fqgkXLR10k3gdhxUUUga7g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=yJzVP8fp; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="yJzVP8fp" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2cd2c7904dcso3031347a91.0 for ; Fri, 09 Aug 2024 12:43:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232634; x=1723837434; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=KtOd9aj4hqeGD1AC+o3mLs+Sgxz7FerwNOXhm7lsLkU=; b=yJzVP8fpammFpUwlRKxnR30bVqCTcPr6UQpxuw18PmiM5fSSjMw+yv4gKxL4pi3sqF zhHLJAzpPap6QdBRQO6yPeYO+2Sdloc0CW5kdXRlnbNToXyyQgxA2yHm9Tv9beKhM3fA njYEpKrjTfqEZUZ2Cbu2moRRTsH1NRhAsExnPh27+dmmEEoUYkhcD6gr/fuVSkaoroAP aejQhHfhgbOH+AcKZNqGr4lHfca35Vk0WCVTq/mVCtgEX1NOplJGnEoJJ0jTwC9pgu4O +1i5/m0EGXGqbldIFrQgbVih2h0K3ZEWMEt7SZoQKVxn2QpgeiZDZAT2Hff9oVJ/eUXh UsXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232634; x=1723837434; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=KtOd9aj4hqeGD1AC+o3mLs+Sgxz7FerwNOXhm7lsLkU=; b=ZGt6tHe/NjZHm1RdH+P9XQELSPf2YkzIjt8RI1x0KJNd4+tX+oB78cjo0dbxDvqQ+D tWQcB6IJKoXn0GLR1m5U84s59+rbBvLrhiyodwz20uDWCBOM9UInr/54Cp/q4J2Hyk1p jPbx94Xb0W+E5BXp3fqa4NthPaVIdRdwURK9BX3HJctTTuMkGDypHpMEfF0TNqzNg1lW SMJkb5/wYE7q+c8zhtVK8nVBIjEPKUVyk5cM2k97oNNGdj41N5rhS0ibfBajbHq+PU62 YoviHlxzvLMNNpGAg4cNHE2sJHeJ8xQEAJh0BNDGllHPDnYIBeKPAESwBs4jJvyLaD11 EzBw== X-Forwarded-Encrypted: i=1; AJvYcCUuUTbDXUFFOoE7SmGIBEk7GfPJkwsw3yToDL2Dg7TVnYBKb7QuCGqiN6EIsj2tZqD+zB3HZ2nZOcH7SalkuHZ2h4seRF8kqktSc1Yl X-Gm-Message-State: AOJu0YwTWlmBNRwL1BJrhNtRBfj7knmXUat3d48I4bagFBC1Yctz6oG/ Vj1Rh5FbVgVqryVM1eykkxssnGhXETY3X0F9+4RZkDulIZqFjD/MdsHLFlOthg6MsAPTnyfS1mR YTw== X-Google-Smtp-Source: AGHT+IGvAmg8+tqrGrCIZoGsSdnG8ZSFhwx3qgY9wt9WSt5Z7jq3JEgK8XnEDH316ti3bUud1crZuHLg05M= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:90a:a88:b0:2c3:1985:e9c3 with SMTP id 98e67ed59e1d1-2d1e80b80femr12946a91.3.1723232634383; Fri, 09 Aug 2024 12:43:54 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:20 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-9-seanjc@google.com> Subject: [PATCH 08/22] KVM: selftests: Add a read-only mprotect() phase to mmu_stress_test From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a third phase of mmu_stress_test to verify that mprotect()ing guest memory to make it read-only doesn't cause explosions, e.g. to verify KVM correctly handles the resulting mmu_notifier invalidations. Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/mmu_stress_test.c | 22 +++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/kvm/mmu_stress_test.c b/tools/testing/= selftests/kvm/mmu_stress_test.c index 9573ed0e696d..50c3a17418c4 100644 --- a/tools/testing/selftests/kvm/mmu_stress_test.c +++ b/tools/testing/selftests/kvm/mmu_stress_test.c @@ -27,6 +27,10 @@ static void guest_code(uint64_t start_gpa, uint64_t end_= gpa, uint64_t stride) GUEST_SYNC(i); } =20 + for (gpa =3D start_gpa; gpa < end_gpa; gpa +=3D stride) + *((volatile uint64_t *)gpa); + GUEST_SYNC(2); + GUEST_ASSERT(0); } =20 @@ -94,6 +98,10 @@ static void *vcpu_worker(void *data) run_vcpu(vcpu, 1); rendezvous_with_boss(); =20 + /* Stage 2, read all of guest memory, which is now read-only. */ + run_vcpu(vcpu, 2); + rendezvous_with_boss(); + return NULL; } =20 @@ -174,7 +182,7 @@ int main(int argc, char *argv[]) const uint64_t start_gpa =3D SZ_4G; const int first_slot =3D 1; =20 - struct timespec time_start, time_run1, time_reset, time_run2; + struct timespec time_start, time_run1, time_reset, time_run2, time_ro; uint64_t max_gpa, gpa, slot_size, max_mem, i; int max_slots, slot, opt, fd; bool hugepages =3D false; @@ -278,14 +286,20 @@ int main(int argc, char *argv[]) rendezvous_with_vcpus(&time_reset, "reset"); rendezvous_with_vcpus(&time_run2, "run 2"); =20 + mprotect(mem, slot_size, PROT_READ); + rendezvous_with_vcpus(&time_ro, "mprotect RO"); + + time_ro =3D timespec_sub(time_ro, time_run2); time_run2 =3D timespec_sub(time_run2, time_reset); - time_reset =3D timespec_sub(time_reset, time_run1); + time_reset =3D timespec_sub(time_reset, time_run1); time_run1 =3D timespec_sub(time_run1, time_start); =20 - pr_info("run1 =3D %ld.%.9lds, reset =3D %ld.%.9lds, run2 =3D %ld.%.9lds\= n", + pr_info("run1 =3D %ld.%.9lds, reset =3D %ld.%.9lds, run2 =3D %ld.%.9lds, " + "ro =3D %ld.%.9lds\n", time_run1.tv_sec, time_run1.tv_nsec, time_reset.tv_sec, time_reset.tv_nsec, - time_run2.tv_sec, time_run2.tv_nsec); + time_run2.tv_sec, time_run2.tv_nsec, + time_ro.tv_sec, time_ro.tv_nsec); =20 /* * Delete even numbered slots (arbitrary) and unmap the first half of --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 67A7A193069 for ; Fri, 9 Aug 2024 19:43:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232639; cv=none; b=OycwKFXedDVk6yYnkDd/pthYBWA8T7WESKz6PN6razokj7C50u3yjar/cexLM/dlRv8W03AM27EYk1PlAFNnW3g4UtMwbad5VaeeKSAy0sHSYy+pLVKQbjw+pSMRjhEWGxbwQtqx7sW3eKgjbnVU0nhyTlkBBKieiqc8sWc+pjI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232639; c=relaxed/simple; bh=WbIw5XKazA4cY8kYGA63IriSB3FyI1oMjErI9b0li8o=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=UxSKzQy+1u/cEn5nL71PmJtwp5Exqn85SwZP/z/h4m0JTOTIErUbRCrvshxZsRwMi2s9uJoasXz1QY3z9cUXpN6IZryrsmOofJBpnU5lzu9ttd5InYMvbg3msawMz6OysYNh17KLgfcYNHgndSnmEGhGbAxblpGx4jj0gpWYaAg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=gU6jeuVi; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="gU6jeuVi" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6902dc6d3ffso59920447b3.2 for ; Fri, 09 Aug 2024 12:43:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232636; x=1723837436; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=LA7LUE9c83FojgXGi3HkbRYYkaUFQd5wswdwV1F5jBI=; b=gU6jeuViOpEtFfoHDpTbdPptIImL9ZovcOCdszkOZsjx27gvFGSMuOaL1Nk4YRFdaW TBPjZo61b9A+tImg6G/19nlAI4Y1aXROj4xy04zWKcd7oKoqDVNUgVZ91wHiuLaoX3CX SET3qiE3+UqPd2JHDY8NxwAX2PEW2H8NjsqjPt8UWDCtXyL6EYu7EDlxQDL6UL2dWi7F ahB22lv6mIat8iSud9GoKZjbEsJEqBOfVhiY71XVAICbCw+u0u2Mh5vo8qkvNwPM0YdM lMcUBkQTSmFmBN6Toq8hGr2VFtCdcToPGgOwr4KQ5HiS9lRRHGsnlsISb1W/HihqFq2z c/9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232636; x=1723837436; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=LA7LUE9c83FojgXGi3HkbRYYkaUFQd5wswdwV1F5jBI=; b=pKSo77ltlBTT7fnkoGeYeM+5hsjaLFvovu1oQHO55xMebTLfEMKzvZaloTahvJ8sej of+0GrShghq3gnyRccjfPmkWVQAaOW8JjG7Nx88RKc2Yz6OrGRaK1MyTQZG2R0X1anha 2sdogYbQpD9O9tC5pF0NNLv2R5/HUuVwhhCiKW2XI0BAH/vfhSFMZiB/MYJLjtxTQ/Xi xNWhbs7dRN3S1yO9AC8XuywpXhtxqg4mR/Ah4qZYWM2pN6ZAX8MRQarCnu0gyhvnG/nr SS2L39x5ha72FcKLWoAAyvw2c4jQ/X/hFYFcF/h4UUtdNxD9vq/TWZpksbq7N/CxIfL3 hbGw== X-Forwarded-Encrypted: i=1; AJvYcCUCtynn7ycdViqNUfdXQt7KSXo9qq3mnl3TtnKGxuyIlWHR87tHypee0xDjwDj8H7zPWZjMGZIbx49QBFrASk2Mvm/Xy7zUoosw7PGR X-Gm-Message-State: AOJu0YwjbbNN8EoCbYsyuhr+SeFEEz7W2vK0VU8M1u7yi9wk2k589ZIB eZCNW8rQPR8KMj5Nh6JGU2atavm2ptxC32Z0KMhW7p0x0F4bRyLvGSBjDBCiKW42hvyKGlLHrCf IsQ== X-Google-Smtp-Source: AGHT+IHdm1tAiwdHoiER1evk1B2BVo/sVRdO+cs/X5rKSY/fp4P6+azgk/JjvaFk5hFSNW8l0gSAdoLXOSY= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a0d:f084:0:b0:697:9aae:1490 with SMTP id 00721157ae682-69ec4a1937dmr228707b3.1.1723232636462; Fri, 09 Aug 2024 12:43:56 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:21 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-10-seanjc@google.com> Subject: [PATCH 09/22] KVM: selftests: Verify KVM correctly handles mprotect(PROT_READ) From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add two phases to mmu_stress_test to verify that KVM correctly handles guest memory that was writable, and then made read-only in the primary MMU, and then made writable again. Add bonus coverage for x86 to verify that all of guest memory was marked read-only. Making forward progress (without making memory writable) requires arch specific code to skip over the faulting instruction, but the test can at least verify each vCPU's starting page was made read-only. Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/mmu_stress_test.c | 87 ++++++++++++++++++- 1 file changed, 84 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/kvm/mmu_stress_test.c b/tools/testing/= selftests/kvm/mmu_stress_test.c index 50c3a17418c4..98f9a4660269 100644 --- a/tools/testing/selftests/kvm/mmu_stress_test.c +++ b/tools/testing/selftests/kvm/mmu_stress_test.c @@ -16,6 +16,8 @@ #include "guest_modes.h" #include "processor.h" =20 +static bool mprotect_ro_done; + static void guest_code(uint64_t start_gpa, uint64_t end_gpa, uint64_t stri= de) { uint64_t gpa; @@ -31,6 +33,33 @@ static void guest_code(uint64_t start_gpa, uint64_t end_= gpa, uint64_t stride) *((volatile uint64_t *)gpa); GUEST_SYNC(2); =20 + /* + * Write to the region while mprotect(PROT_READ) is underway. Keep + * looping until the memory is guaranteed to be read-only, otherwise + * vCPUs may complete their writes and advance to the next stage + * prematurely. + */ + do { + for (gpa =3D start_gpa; gpa < end_gpa; gpa +=3D stride) +#ifdef __x86_64__ + asm volatile(".byte 0xc6,0x40,0x0,0x0" :: "a" (gpa) : "memory"); +#else + vcpu_arch_put_guest(*((volatile uint64_t *)gpa), gpa); +#endif + } while (!READ_ONCE(mprotect_ro_done)); + + /* + * Only x86 can explicitly sync, as other architectures will be stuck + * on the write fault. + */ +#ifdef __x86_64__ + GUEST_SYNC(3); +#endif + + for (gpa =3D start_gpa; gpa < end_gpa; gpa +=3D stride) + vcpu_arch_put_guest(*((volatile uint64_t *)gpa), gpa); + GUEST_SYNC(4); + GUEST_ASSERT(0); } =20 @@ -78,6 +107,7 @@ static void *vcpu_worker(void *data) struct vcpu_info *info =3D data; struct kvm_vcpu *vcpu =3D info->vcpu; struct kvm_vm *vm =3D vcpu->vm; + int r; =20 vcpu_args_set(vcpu, 3, info->start_gpa, info->end_gpa, vm->page_size); =20 @@ -100,6 +130,49 @@ static void *vcpu_worker(void *data) =20 /* Stage 2, read all of guest memory, which is now read-only. */ run_vcpu(vcpu, 2); + + /* + * Stage 3, write guest memory and verify KVM returns -EFAULT for once + * the mprotect(PROT_READ) lands. Only architectures that support + * validating *all* of guest memory sync for this stage, as vCPUs will + * be stuck on the faulting instruction for other architectures. Go to + * stage 3 without a rendezvous + */ + do { + r =3D _vcpu_run(vcpu); + } while (!r); + TEST_ASSERT(r =3D=3D -1 && errno =3D=3D EFAULT, + "Expected EFAULT on write to RO memory, got r =3D %d, errno =3D %d",= r, errno); + +#ifdef __x86_64__ + /* + * Verify *all* writes from the guest hit EFAULT due to the VMA now + * being read-only. x86-only at this time as skipping the instruction + * that hits the EFAULT requires advancing the program counter, which + * is arch specific and currently relies on hand-coded assembly. + */ + vcpu->run->kvm_valid_regs =3D KVM_SYNC_X86_REGS; + for (;;) { + r =3D _vcpu_run(vcpu); + if (!r) + break; + TEST_ASSERT_EQ(errno, EFAULT); + WRITE_ONCE(vcpu->run->kvm_dirty_regs, KVM_SYNC_X86_REGS); + vcpu->run->s.regs.regs.rip +=3D 4; + } + assert_sync_stage(vcpu, 3); +#endif + rendezvous_with_boss(); + + /* + * Stage 4. Run to completion, waiting for mprotect(PROT_WRITE) to + * make the memory writable again. + */ + do { + r =3D _vcpu_run(vcpu); + } while (r && errno =3D=3D EFAULT); + TEST_ASSERT_EQ(r, 0); + assert_sync_stage(vcpu, 4); rendezvous_with_boss(); =20 return NULL; @@ -182,7 +255,7 @@ int main(int argc, char *argv[]) const uint64_t start_gpa =3D SZ_4G; const int first_slot =3D 1; =20 - struct timespec time_start, time_run1, time_reset, time_run2, time_ro; + struct timespec time_start, time_run1, time_reset, time_run2, time_ro, ti= me_rw; uint64_t max_gpa, gpa, slot_size, max_mem, i; int max_slots, slot, opt, fd; bool hugepages =3D false; @@ -287,19 +360,27 @@ int main(int argc, char *argv[]) rendezvous_with_vcpus(&time_run2, "run 2"); =20 mprotect(mem, slot_size, PROT_READ); + usleep(10); + mprotect_ro_done =3D true; + sync_global_to_guest(vm, mprotect_ro_done); + rendezvous_with_vcpus(&time_ro, "mprotect RO"); + mprotect(mem, slot_size, PROT_READ | PROT_WRITE); + rendezvous_with_vcpus(&time_rw, "mprotect RW"); =20 + time_rw =3D timespec_sub(time_rw, time_ro); time_ro =3D timespec_sub(time_ro, time_run2); time_run2 =3D timespec_sub(time_run2, time_reset); time_reset =3D timespec_sub(time_reset, time_run1); time_run1 =3D timespec_sub(time_run1, time_start); =20 pr_info("run1 =3D %ld.%.9lds, reset =3D %ld.%.9lds, run2 =3D %ld.%.9lds, " - "ro =3D %ld.%.9lds\n", + "ro =3D %ld.%.9lds, rw =3D %ld.%.9lds\n", time_run1.tv_sec, time_run1.tv_nsec, time_reset.tv_sec, time_reset.tv_nsec, time_run2.tv_sec, time_run2.tv_nsec, - time_ro.tv_sec, time_ro.tv_nsec); + time_ro.tv_sec, time_ro.tv_nsec, + time_rw.tv_sec, time_rw.tv_nsec); =20 /* * Delete even numbered slots (arbitrary) and unmap the first half of --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E4A2E194C65 for ; Fri, 9 Aug 2024 19:43:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232640; cv=none; b=rtID3MF54IKQw7ZmrtDogw3vsq2KJ1/8bciPKamWZoqA8o0ey1dz7dEw+ZujCF/jz1SIv6Se5kQOAcW0hWGYsEbNOQaErJmuUp8G/0ZnJefgPdRw6KCDdedkyGNSGd/57+f7hPVcWdDLX1iX5WaayYPcNbyosWyTMqwJZ3SSvpY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232640; c=relaxed/simple; bh=+GylkI/uIWrUOvqPBl8m0ev0LDK+4fi2QwLYgo7xKSQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=FX7QN8GZpbz2EZ+w5pGpzqaJpOLS552CFQE8quPIveotnI4nCTzEVCI8lcOkosGg8r2EwNt/x/FioHMdrlxW8qmr+g6tiNEOmpIH36Tif/Zf56kUhGceEa+3+enzGU/Mj91H+16CIktZKWZLAdsWJHrQdWPmeB0pfUyvApQ3Lpg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=pDHcY69F; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="pDHcY69F" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-1fc47634e3dso22959755ad.0 for ; Fri, 09 Aug 2024 12:43:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232638; x=1723837438; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=FfIoOq8ezL3lF0COKJQzO4R/UUZXUE0fWXr809zSI60=; b=pDHcY69FQBoDOpmgX9GU7Ao38U8UCZoRyRFL1OayPd0huY0pcxvBJuTkt0CslTH2wY O1EGGgK1qMUm2uoa5VPUydKHIg7jVq9xVdG4BjoWrhNkaG9IHT2Fr2s0PVUoZCrLYv5o W0ZUuBLoFPVsUuP6WBNrILPxBc3StUJN8q6ionxxWAqrbWsgHLO+kZHZEkK+e9kSgyd1 G/wn9NGJu/WUI0SGZ6qYxbueDpz35n5JbqwrL+Uy+hANhKxZxQvsNH3qVasxHkvcYAC8 46lsucWkoI1INjt10wLVyn//vGBcH9+sIlmMQ1wPuJq7spFdhw7d1qtQxnvmmpb+YL1X vF6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232638; x=1723837438; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=FfIoOq8ezL3lF0COKJQzO4R/UUZXUE0fWXr809zSI60=; b=VDZKYhFtIaQoYfT83R3wTJ99kh4+52P6584V8RFMIafkjggoBwpTdWlGGbzM7aAtp0 gfkR8o0vPoKx+vZy+oD935iPVgbSZUFBnPovGj453RQBUHDelK+oy02Fk+eM20FYrbtX pJpvspMsoWc4nrJG8cu7ccgDx7XQ1gDoIVrOtyhXBm1IvnNNLubio0Z70DDfXJpwj7Pe RgdGIXcNUPNt2EKQZ+gvczGPgUEO7i6jQV6SsxEAY60OOx+vbnj0YqRevV4k/7UqQvMK PT0y0VVREt8VevvdLnz9/GYxVDTPv+fo8S4r3Kbq9HZUtExFTDUq0LeCS33WVgcvFpQ+ YICw== X-Forwarded-Encrypted: i=1; AJvYcCXo3E9A5uy918NgccBnNRha9cvlszF0VfKWiwzeAj5v8e4in13y3XlFYn4QFknahgN64jd4RgCm/OxFZ3nM9E3A6TFQ+/6d3cMrMKQ4 X-Gm-Message-State: AOJu0YwRcRrirnLrcFS+NiQcHf2BscbDywu4kpYXwJ82j6syCA8TCgmF S3NkAOs+piYRPZ79WIrPzHiauNjvInkcum9FqbUXTCQzG6aTFwps3cojt1CC+vpfNVuDrEqj/kG 6Fg== X-Google-Smtp-Source: AGHT+IHI2Sj/+82ITfhQRaMFsQWLje/k3ROR9iGp71RmEKU0cgVARHxIz4EvnVRZ5brISpMtlPzaid6nWq4= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:e80b:b0:1fa:f9ac:8b4a with SMTP id d9443c01a7336-200ae4c9f0amr1621115ad.3.1723232638087; Fri, 09 Aug 2024 12:43:58 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:22 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-11-seanjc@google.com> Subject: [PATCH 10/22] KVM: x86/mmu: Move walk_slot_rmaps() up near for_each_slot_rmap_range() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move walk_slot_rmaps() and friends up near for_each_slot_rmap_range() so that the walkers can be used to handle mmu_notifier invalidations, and so that similar function has some amount of locality in code. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 107 +++++++++++++++++++++-------------------- 1 file changed, 54 insertions(+), 53 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 901be9e420a4..676cb7dfcbf9 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1534,6 +1534,60 @@ static void slot_rmap_walk_next(struct slot_rmap_wal= k_iterator *iterator) slot_rmap_walk_okay(_iter_); \ slot_rmap_walk_next(_iter_)) =20 + +/* The return value indicates if tlb flush on all vcpus is needed. */ +typedef bool (*slot_rmaps_handler) (struct kvm *kvm, + struct kvm_rmap_head *rmap_head, + const struct kvm_memory_slot *slot); + +static __always_inline bool __walk_slot_rmaps(struct kvm *kvm, + const struct kvm_memory_slot *slot, + slot_rmaps_handler fn, + int start_level, int end_level, + gfn_t start_gfn, gfn_t end_gfn, + bool flush_on_yield, bool flush) +{ + struct slot_rmap_walk_iterator iterator; + + lockdep_assert_held_write(&kvm->mmu_lock); + + for_each_slot_rmap_range(slot, start_level, end_level, start_gfn, + end_gfn, &iterator) { + if (iterator.rmap) + flush |=3D fn(kvm, iterator.rmap, slot); + + if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) { + if (flush && flush_on_yield) { + kvm_flush_remote_tlbs_range(kvm, start_gfn, + iterator.gfn - start_gfn + 1); + flush =3D false; + } + cond_resched_rwlock_write(&kvm->mmu_lock); + } + } + + return flush; +} + +static __always_inline bool walk_slot_rmaps(struct kvm *kvm, + const struct kvm_memory_slot *slot, + slot_rmaps_handler fn, + int start_level, int end_level, + bool flush_on_yield) +{ + return __walk_slot_rmaps(kvm, slot, fn, start_level, end_level, + slot->base_gfn, slot->base_gfn + slot->npages - 1, + flush_on_yield, false); +} + +static __always_inline bool walk_slot_rmaps_4k(struct kvm *kvm, + const struct kvm_memory_slot *slot, + slot_rmaps_handler fn, + bool flush_on_yield) +{ + return walk_slot_rmaps(kvm, slot, fn, PG_LEVEL_4K, PG_LEVEL_4K, flush_on_= yield); +} + typedef bool (*rmap_handler_t)(struct kvm *kvm, struct kvm_rmap_head *rmap= _head, struct kvm_memory_slot *slot, gfn_t gfn, int level); @@ -6199,59 +6253,6 @@ void kvm_configure_mmu(bool enable_tdp, int tdp_forc= ed_root_level, } EXPORT_SYMBOL_GPL(kvm_configure_mmu); =20 -/* The return value indicates if tlb flush on all vcpus is needed. */ -typedef bool (*slot_rmaps_handler) (struct kvm *kvm, - struct kvm_rmap_head *rmap_head, - const struct kvm_memory_slot *slot); - -static __always_inline bool __walk_slot_rmaps(struct kvm *kvm, - const struct kvm_memory_slot *slot, - slot_rmaps_handler fn, - int start_level, int end_level, - gfn_t start_gfn, gfn_t end_gfn, - bool flush_on_yield, bool flush) -{ - struct slot_rmap_walk_iterator iterator; - - lockdep_assert_held_write(&kvm->mmu_lock); - - for_each_slot_rmap_range(slot, start_level, end_level, start_gfn, - end_gfn, &iterator) { - if (iterator.rmap) - flush |=3D fn(kvm, iterator.rmap, slot); - - if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) { - if (flush && flush_on_yield) { - kvm_flush_remote_tlbs_range(kvm, start_gfn, - iterator.gfn - start_gfn + 1); - flush =3D false; - } - cond_resched_rwlock_write(&kvm->mmu_lock); - } - } - - return flush; -} - -static __always_inline bool walk_slot_rmaps(struct kvm *kvm, - const struct kvm_memory_slot *slot, - slot_rmaps_handler fn, - int start_level, int end_level, - bool flush_on_yield) -{ - return __walk_slot_rmaps(kvm, slot, fn, start_level, end_level, - slot->base_gfn, slot->base_gfn + slot->npages - 1, - flush_on_yield, false); -} - -static __always_inline bool walk_slot_rmaps_4k(struct kvm *kvm, - const struct kvm_memory_slot *slot, - slot_rmaps_handler fn, - bool flush_on_yield) -{ - return walk_slot_rmaps(kvm, slot, fn, PG_LEVEL_4K, PG_LEVEL_4K, flush_on_= yield); -} - static void free_mmu_pages(struct kvm_mmu *mmu) { if (!tdp_enabled && mmu->pae_root) --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9F0751990AA for ; Fri, 9 Aug 2024 19:44:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232643; cv=none; b=CkFEu/i4lPBLVev1tLluyWJ+YNbH7vQVTj2fdZxMruRtBW4vIiLyLy19TgiwUkKZQAtxSHrJvHcTG603cBnwngQBIlNz8c0d8i0KulP3n6/jVAkoiJialc3dJSMxd1PVjlJwWb3WrenSq2Xst3UJT/S1COWUUpZxYWI8XmAIl+w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232643; c=relaxed/simple; bh=Q4/0S9zjurXdMR/Xc6/7FhGHVIB+yXDEQA05A1JT85k=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=YTHnXO3VApeHEa0gHQTLEItqNj/Rs3FxDrgqTEzp0L57l9/u12txDA9EoNKhOIr8HkIxl+OUUdwZcly3Z7DtpwGmhn95FdP3wjzddYmwmWxGyBepqCxWqKJ+PwQnj06JeM/cbuN7YEXz2pdaQ+2t0eL44Acwdf/nc97AWIRr0CQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=yOPfdfbk; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="yOPfdfbk" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2cb6b642c49so3319676a91.1 for ; Fri, 09 Aug 2024 12:44:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232641; x=1723837441; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=RZnIwpm2DX6oLiBTd7LSzopcPIvohhjKZPunqiEaNiE=; b=yOPfdfbkTACJtGhMaGp8/glp73j/d1XutUuiT1/hOrk0OHG2Co/fiHKEI+gpRxzY7P HI7Ng6xM24W+fzHZk14tpEPSlhZM1IDhxlrj7p6VBLQhllP4gAh9ZQsBmRW1d83yMiv4 E3V4t34SmWHKgTNXoKZq6qoDyotqBwVv0cpRq5Z0bxkoZ8mmVhctJP4O/ljfqaCamKt4 fQzCBe610zoObazsxBFS8IxbwVgoURES985c6oS0jIas/xkEsZHYXqAa3/625u/xaJLq becO5cYeFiS+KjddHSpsu8gGzOnHu8l4J/IkG5fSzEehedxR7GiqWSrDaKI8Etsn55sn 9GyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232641; x=1723837441; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=RZnIwpm2DX6oLiBTd7LSzopcPIvohhjKZPunqiEaNiE=; b=fu1vsatqg4XdX1tp/e+GFfFoyKGKIr5KOcZ5fk2q3QCp/IgIaMhCZzRtlCnVB8rm0s 5hsdqNgksw5ve6i+56pgvZTD8AtMk8ulaww0XtGHW1kx8MucvZ9BEYihyo4iN/rh5Oip xczU1tJpAOSYPW5SZ5qMtxy4Oyna00brPEgFgJwGGuSwIwS5rVzRs7TUnmQajsNpgGe1 Smq/pL/Nf3nF6NBqaegUp/HEDBTRtLCudBBNsQMFztx/AyfZJYHKQCSIWcJd9SiuDpl5 tI3PNdeeQvQEqVnmLNPXAYbwa8ZVXI+bmWxxlNkg271e1xQPUT1UU3so4lX+R7W3aqt/ b6bQ== X-Forwarded-Encrypted: i=1; AJvYcCW6r7q34puE2ngXT+9g6fObzsGuKit3JfrIrpHAcp3cgH42FFjWkiZ/jq7wR9oirHnYWJ6aJFaQIC2YDS8I89g+4VTAdX4ftdAoX2Ul X-Gm-Message-State: AOJu0YxwFt/KtxLTLolHKt1Ybk+/pdvXlD9Jvo8XoaPhT7OPNogR9Wm+ 1keZVjNmh1sGznqwAo8wMEX3rUimSs4IGQkn5iSsJVv+UCctM/7uh2+vzhbxyyICrjCWzCXOlAZ s3g== X-Google-Smtp-Source: AGHT+IEVMPLIN0gtxbuCx2PJeNj0X98BYvziqTLoHJHpzW4rfW380DSvpsuYxUtuP8Md6WisaeszFgzwm+A= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:90a:68c4:b0:2ca:7f5a:db6c with SMTP id 98e67ed59e1d1-2d1c4c45398mr42369a91.3.1723232640835; Fri, 09 Aug 2024 12:44:00 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:23 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-12-seanjc@google.com> Subject: [PATCH 11/22] KVM: x86/mmu: Plumb a @can_yield parameter into __walk_slot_rmaps() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a @can_yield param to __walk_slot_rmaps() to control whether or not dropping mmu_lock and conditionally rescheduling is allowed. This will allow using __walk_slot_rmaps() and thus cond_resched() to handle mmu_notifier invalidations, which usually allow blocking/yielding, but not when invoked by the OOM killer. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 676cb7dfcbf9..a5a7e476f5bb 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1545,7 +1545,8 @@ static __always_inline bool __walk_slot_rmaps(struct = kvm *kvm, slot_rmaps_handler fn, int start_level, int end_level, gfn_t start_gfn, gfn_t end_gfn, - bool flush_on_yield, bool flush) + bool can_yield, bool flush_on_yield, + bool flush) { struct slot_rmap_walk_iterator iterator; =20 @@ -1556,6 +1557,9 @@ static __always_inline bool __walk_slot_rmaps(struct = kvm *kvm, if (iterator.rmap) flush |=3D fn(kvm, iterator.rmap, slot); =20 + if (!can_yield) + continue; + if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) { if (flush && flush_on_yield) { kvm_flush_remote_tlbs_range(kvm, start_gfn, @@ -1577,7 +1581,7 @@ static __always_inline bool walk_slot_rmaps(struct kv= m *kvm, { return __walk_slot_rmaps(kvm, slot, fn, start_level, end_level, slot->base_gfn, slot->base_gfn + slot->npages - 1, - flush_on_yield, false); + true, flush_on_yield, false); } =20 static __always_inline bool walk_slot_rmaps_4k(struct kvm *kvm, @@ -6528,7 +6532,7 @@ static bool kvm_rmap_zap_gfn_range(struct kvm *kvm, g= fn_t gfn_start, gfn_t gfn_e =20 flush =3D __walk_slot_rmaps(kvm, memslot, __kvm_zap_rmap, PG_LEVEL_4K, KVM_MAX_HUGEPAGE_LEVEL, - start, end - 1, true, flush); + start, end - 1, true, true, flush); } } =20 @@ -6816,7 +6820,7 @@ static void kvm_shadow_mmu_try_split_huge_pages(struc= t kvm *kvm, */ for (level =3D KVM_MAX_HUGEPAGE_LEVEL; level > target_level; level--) __walk_slot_rmaps(kvm, slot, shadow_mmu_try_split_huge_pages, - level, level, start, end - 1, true, false); + level, level, start, end - 1, true, true, false); } =20 /* Must be called with the mmu_lock held in write-mode. */ --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6EF9A1991AE for ; Fri, 9 Aug 2024 19:44:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232644; cv=none; b=DYhi2iYrGS+6k8FKd0HikDWHMxRn6BAtIiV/uGG/MAs74RN31hQMCk3MFcCqu8Vwq8nsPHS+fo5tUwaWhfVmQe/et/jb2WKgFGL/hjOEHPFZIqWqBx0UddpliyPXgfLGvUUYLCx4h25V6YnFnC510ezkAP1qV8E4GJmauU5y10I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232644; c=relaxed/simple; bh=Ky1N4NEPPXYQ68tuOtFA6BSNneI8rYbZCaa3EnVYquw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=CTHpx/6BNTShMCprSBiXHNUX4xXC7j1rvcwTnyKl5GHI2ZQ/i0uq3ID1RLxqKfX5RSXAz7FiALbPMGzEJjxXzAqZ/yfamHJkJwUCb9PLY0ok6avPmO3YcVlo/qe5R/J+Ww+wB0dyE4w6ZDF7cniNtx0u4QJTks9byYa1kq6sgtU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=EIVw8Rlu; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="EIVw8Rlu" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-70d1df50db2so2316791b3a.0 for ; Fri, 09 Aug 2024 12:44:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232643; x=1723837443; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=KngE1DTnwbCcWR9aNh2KG9Wx5wVMsPDCUrAPTcj/AmI=; b=EIVw8RluigmxDMVw8j0b2Ek+6tEzRzap1dFnmb+4h3VXdMwG9dCQXL32e14jOFjvGq TcFb2itAsoBKoowacw1mYdnagzEUOnCjwVwOQhe1bzOZVFE341gyKyp0mmv7NVmfQVj9 qXcgXymrr4xrro76/E+7OWbsfhBXynxxJmc5JMTc95mvodngkABgAy2c8QrgeEGvDe2Q nJJ7FZD5Djp3HzAPlEoLWWovTtfnYyNTzl/CdC9ggrO3be9KFyFFT35hmaiGbuDW4dXd d1SH/3cGzB+KqD0CRd4uebTb7oY6fYtB1S5SnO7MIis0GZR+pFNNHRVNsUsVzXdn1MqC +kfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232643; x=1723837443; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=KngE1DTnwbCcWR9aNh2KG9Wx5wVMsPDCUrAPTcj/AmI=; b=OKb2BgoE+1OWO5boNIwVR6bcluLMvaRW3gAyVc4nqMUWmQK64ZjvnF81NCu7s6rE0I xUYMS1gKOMT5entNCflT9GpeDIYRDisTqEuVnRJxmJvwQrIwZdxsQcx+UHaUdXOwc3hX /Vn+cHT5QwAJj2dbSsCNeX8jNcuGy977hhktebd5dRG0Zz4x2cQ5VyrgN7MLrmhfw82q 31DeAm7pl+ij8g+WifoeBU50aTQ1nxrTC16uL9Yu8Aw7uGb94UvkIwC15mkhJGl1Cx11 /SYZKm7NPGLfHTrBU0FnDwhi5aFyIsI/9IhW10nWCAijw0xalNgmmNnmenQnnvFCurwo DtWA== X-Forwarded-Encrypted: i=1; AJvYcCVSKuHs5E+LbQs8k4BY0UOZpYmk+zIWzNA7ipcWDv2dytfOfQZDkttmpJG1yeynKyq85k09kZIh3gMM224tEisIzoS9oxF7GvPJ8hJ9 X-Gm-Message-State: AOJu0YyoUJadbeXalNAfT7a0w/LttyhrgHB1iXfZ5XOOVgiSCN16yKzn i1CkXQp1jB08XA73EfqvKtJhvHRL2U1Pc+Jsb4gKKAaVb7BkRUl/MXoLfQ+bRLfmTzDHONpGC0y fkQ== X-Google-Smtp-Source: AGHT+IG7ko0vuFPPXt3xcZarFz9eU+f5poN4SmWQJRgKUX/WSEyph3BvOPDm1891EeOCuHmelP3kV0TthVI= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6a00:91e4:b0:710:4e4c:a4ad with SMTP id d2e1a72fcca58-710cc32523dmr145683b3a.0.1723232642725; Fri, 09 Aug 2024 12:44:02 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:24 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-13-seanjc@google.com> Subject: [PATCH 12/22] KVM: x86/mmu: Add a helper to walk and zap rmaps for a memslot From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a dedicated helper to walk and zap rmaps for a given memslot so that the code can be shared between KVM-initiated zaps and mmu_notifier invalidations. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a5a7e476f5bb..4016f63d03e8 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1592,6 +1592,16 @@ static __always_inline bool walk_slot_rmaps_4k(struc= t kvm *kvm, return walk_slot_rmaps(kvm, slot, fn, PG_LEVEL_4K, PG_LEVEL_4K, flush_on_= yield); } =20 +static bool __kvm_rmap_zap_gfn_range(struct kvm *kvm, + const struct kvm_memory_slot *slot, + gfn_t start, gfn_t end, bool can_yield, + bool flush) +{ + return __walk_slot_rmaps(kvm, slot, __kvm_zap_rmap, + PG_LEVEL_4K, KVM_MAX_HUGEPAGE_LEVEL, + start, end - 1, can_yield, true, flush); +} + typedef bool (*rmap_handler_t)(struct kvm *kvm, struct kvm_rmap_head *rmap= _head, struct kvm_memory_slot *slot, gfn_t gfn, int level); @@ -6530,9 +6540,8 @@ static bool kvm_rmap_zap_gfn_range(struct kvm *kvm, g= fn_t gfn_start, gfn_t gfn_e if (WARN_ON_ONCE(start >=3D end)) continue; =20 - flush =3D __walk_slot_rmaps(kvm, memslot, __kvm_zap_rmap, - PG_LEVEL_4K, KVM_MAX_HUGEPAGE_LEVEL, - start, end - 1, true, true, flush); + flush =3D __kvm_rmap_zap_gfn_range(kvm, memslot, start, + end, true, flush); } } =20 --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8DF5F19922C for ; Fri, 9 Aug 2024 19:44:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232646; cv=none; b=swAxS4vIOO1Gc474g36y6KkdpxydvDMxH+70iwSs1lsSFfOe0OTW0jVmigOhj7/GOeZ46c/hknCIFQBqcVkbqzwdBqOJWqSlOsQJKtJfedlEqplAIe903k5cMwkjAbhVddicBrelU4mttO2pAobXzGPKwopN4l8AV/pNwaUnrtY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232646; c=relaxed/simple; bh=3xSCN07is/d7+9qa5tb05xLdmkaS8Uh8BYF+HVNaYMo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=c4XwSZvIHam9sBFthCD9m4B/6u2B6Fkk+X537S7py7zSdIwXh6PxNUIWbEP9pWZj1tgDsQkE5hqa+pckYeV94knTIq0EXBymI/65NLJF11m9DfNk6QV1QO2fBCdmFspvU8TpYNDAMYgdn73c7eCr+vt1Ib5Qq/i7DUAPgxgXk5k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=zJKcxA9G; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="zJKcxA9G" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-1fc5651e888so22886055ad.0 for ; Fri, 09 Aug 2024 12:44:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232645; x=1723837445; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:from:to:cc:subject:date :message-id:reply-to; bh=C/Kjdpw4GZGWgN0ZxVg8kngRskxTTrucp/2D4JfCR5g=; b=zJKcxA9G0HILjOPS7dzZMv3QTb+R5BJNGYIwuowxdDZ91Sl7KCJMNREMOEGyz2zh2G agDsp++3cPC+IO/vGRG3tKdAE/roBrNSOIeeGCufasezPUgrNNu1Dpj/rI5+rDxdneEB 6u7uWgY1focTsh+dE7EFpyZJ74N36oNJwtqmZpoHPrFYmIyKQgssZl8K3LUlvpL9dHSq wwAeMql5sJe7jB4IUgCmsIuY7ugY0qD2gYLauxpKRVwq1SmRJRI+cw3RLq3QMmiRI81k opMFs+eSKjMQIIZ6jWu36LjcsDui+dlMImiJpkNRg7HuwjtMy4dVfu0wx9Sy7Gka6bvA X5Lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232645; x=1723837445; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=C/Kjdpw4GZGWgN0ZxVg8kngRskxTTrucp/2D4JfCR5g=; b=dRQJpC64By6h9BnmXKLqAyAfaSIfB0bVQUGYKRzCIjMeEjH/1H4BG9ZS7G3cnaB3Dk 71F2PlfoBbLawnwVo1phjyhFGhwokKmoM08fmXWeDNhduk9j9Upb9THRYkrW9Um8bEEJ 3ARXs2/hqHetAuL3thYKQIX9N9MpMBiF3Dof0k2MacpaA8D1/DCDBtyjEM5JydkJibT5 RuIM6z1zniWd8jiMYyEYXDPoPMVBCFYsOo9usa/NCvJZDekMi4aivtt+Xva7HoQh/n3e ZKkexVURlbUdDHDFTbWN5iCYEZdMQkalxxMaPyN2d1Q7aG4Thn2i6W+tiikRoW1YtXH6 U7bA== X-Forwarded-Encrypted: i=1; AJvYcCUJR/AToDmSMqlI1lGT4z9tyO9w6H0IJgxlhhvgVGRn19fW1asfR2mmLNJOwDw7lJZHB45LP8x6pRFr5WrTZsUZSMIEAdxzV99iMc7E X-Gm-Message-State: AOJu0YzC65LJ5s/vnQVEPk9MXGH9Qy9L6BwxyYtekyBlvvIQVj9+/Oms 3l1c8cBlGU54NQRf7HO0VlnBeZjVERNw0YGzdBTtBqJQBaTYVlnNJswfn7hQcGexuk1/E1/vsmU wRw== X-Google-Smtp-Source: AGHT+IHlnphgTRNzeVdx0O31fPI0LuneFaWMABXGnKSnF6B8xUcCs6tohBFzlE1Hi/BkKbGcyYXHhyb9HbY= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:d507:b0:1fc:2ee3:d46c with SMTP id d9443c01a7336-200ae2549b2mr2179795ad.0.1723232644737; Fri, 09 Aug 2024 12:44:04 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:25 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-14-seanjc@google.com> Subject: [PATCH 13/22] KVM: x86/mmu: Honor NEED_RESCHED when zapping rmaps and blocking is allowed From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Convert kvm_unmap_gfn_range(), which is the helper that zaps rmap SPTEs in response to an mmu_notifier invalidation, to use __kvm_rmap_zap_gfn_range() and feed in range->may_block. In other words, honor NEED_RESCHED by way of cond_resched() when zapping rmaps. This fixes a long-standing issue where KVM could process an absurd number of rmap entries without ever yielding, e.g. if an mmu_notifier fired on a PUD (or larger) range. Opportunistically rename __kvm_zap_rmap() to kvm_zap_rmap(), and drop the old kvm_zap_rmap(). Ideally, the shuffling would be done in a different patch, but that just makes the compiler unhappy, e.g. arch/x86/kvm/mmu/mmu.c:1462:13: error: =E2=80=98kvm_zap_rmap=E2=80=99 def= ined but not used Reported-by: Peter Xu Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 4016f63d03e8..0a33857d668a 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1453,16 +1453,10 @@ static bool kvm_vcpu_write_protect_gfn(struct kvm_v= cpu *vcpu, u64 gfn) return kvm_mmu_slot_gfn_write_protect(vcpu->kvm, slot, gfn, PG_LEVEL_4K); } =20 -static bool __kvm_zap_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_hea= d, - const struct kvm_memory_slot *slot) -{ - return kvm_zap_all_rmap_sptes(kvm, rmap_head); -} - static bool kvm_zap_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, - struct kvm_memory_slot *slot, gfn_t gfn, int level) + const struct kvm_memory_slot *slot) { - return __kvm_zap_rmap(kvm, rmap_head, slot); + return kvm_zap_all_rmap_sptes(kvm, rmap_head); } =20 struct slot_rmap_walk_iterator { @@ -1597,7 +1591,7 @@ static bool __kvm_rmap_zap_gfn_range(struct kvm *kvm, gfn_t start, gfn_t end, bool can_yield, bool flush) { - return __walk_slot_rmaps(kvm, slot, __kvm_zap_rmap, + return __walk_slot_rmaps(kvm, slot, kvm_zap_rmap, PG_LEVEL_4K, KVM_MAX_HUGEPAGE_LEVEL, start, end - 1, can_yield, true, flush); } @@ -1626,7 +1620,9 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_= gfn_range *range) bool flush =3D false; =20 if (kvm_memslots_have_rmaps(kvm)) - flush =3D kvm_handle_gfn_range(kvm, range, kvm_zap_rmap); + flush =3D __kvm_rmap_zap_gfn_range(kvm, range->slot, + range->start, range->end, + range->may_block, flush); =20 if (tdp_mmu_enabled) flush =3D kvm_tdp_mmu_unmap_gfn_range(kvm, range, flush); --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 641BD199250 for ; Fri, 9 Aug 2024 19:44:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232648; cv=none; b=IJo/MelVXHJHriixYbd5A/Uc2xuEtoyH7/uZn+MVS/kgoWj49QaYZbmHBPINQabhfRcAJ/Vf49BZ0UOwUtu2hMZ0/8RfIiPxDizGopJWhPt3MAGQiuBO5+jYqPeILZapoKRfRdwjUuI5F90Rdws1A7LAilt/YgIyakTVQI5Rypc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232648; c=relaxed/simple; bh=Ed1G/Nb2CnyIN7eoTZ0EijA3UdmeGBcXP9PKO/DqlAo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=XtZAqXQHI7YyC2QlBp3mxThFaaEL/GtgnRTqDm9nr7p2DkoO4ZoOsTBI9Up5GnINQn9yWppZxWEVZU8lHH/LRt/nUloDnYU1Rghy4b5cdb95t5ElWvRT6hdNc+MvHbGa2aLXmwh/FzxHd4OjOjfQtwdhXMzZxgg6O4eu8/uBzVQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WJov4la9; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WJov4la9" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-7a30753fe30so2302560a12.3 for ; Fri, 09 Aug 2024 12:44:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232647; x=1723837447; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=xgf0knUlsf9OwxiOIKoJyEqysjyCp256HfjOAsBxKsQ=; b=WJov4la9g6mT6plMsSYQXNdHAX9fpke2GXA9PibfRBbl/km/cDZ6eW/g5uljUZLvAL lNLemc0pucMcfpvsH3IdogpZWpONwLKd8tBK/GBCNq6lgjtI6uANf8AlLT3edj234NSL +uZt+QPjGBTWNdcLCCvZnPRFFBAJy/EKXjwHm5AW/Qw5iu1vUFG2icWu9+044H4FjRm1 hzvIb+qDAKcfpUZau8lLIi/7iknVionVOe75FxTM9NUT6MwZTxHdv8oDAPV7E0++nobf 4cW2h7jf7KzoDaAUl74AlvfOcgkrgy66YrFQEWCQPu4OPZwnCsKRofdLV/UF2+XpFQn1 ZMSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232647; x=1723837447; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=xgf0knUlsf9OwxiOIKoJyEqysjyCp256HfjOAsBxKsQ=; b=NyjLNcx0jTyD/YTqhC0J/stdshfF9yJ3AzCTTqxE5G52YjiF7DAxYzYovRxZQmvdkg aR85gyRaKknQF/7n3BoeR7NnjpwqEvry6/+AFPD9p6BK+WY+F1v7Rq74yaOUUtWXwkfq lPLNVFvAw2N1N1fMiQ8ZWRDdlxINeD74wluD1vmxujUt+4Fbcqy7zdOGj8e2HFj7vkYY dSuVt+r3DFC0bUNtBk7mQendLxxRBd8jwzQZKBO9Tj4rL+85kFV/IODn5Lg49yedjBvS avXwPReKoQFybGUltWT+jkWw56EFMfepjXNsymCZeCBqY9+2cQFmpnoRMtkyO2+qjsHc tdIg== X-Forwarded-Encrypted: i=1; AJvYcCWwH2xWbUXRbUk3nMeO5Qddp4GUD5yoDfTOYOW+Kr6HCjB0T8Bjv3M9EVCWg2OYE/3UcAjP3wcUQZ1zrbs417FKdlg8H/OLQGFmNflf X-Gm-Message-State: AOJu0Yw0cBsmqFcXOeQ8rtXhF3gXIXamyRZumZaDXIfCuojjNiWf4Yg5 L8Tu6PA+saLNmy+gQrYgSkJLyBBIGBqFMIicl+aV5DKTM/FW43PYW6ow9/ajQ5VaX48QPpjvFM8 5oQ== X-Google-Smtp-Source: AGHT+IH1270TZbw7dwgOHkJEckbD2SAane4OZgA6vncKd5lKiIhglKNNCAxVffQiCers7rx90OBnP67l4os= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a63:6d46:0:b0:718:665d:65c9 with SMTP id 41be03b00d2f7-7c3d2bd2a84mr4911a12.6.1723232646717; Fri, 09 Aug 2024 12:44:06 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:26 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-15-seanjc@google.com> Subject: [PATCH 14/22] KVM: x86/mmu: Morph kvm_handle_gfn_range() into an aging specific helper From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Rework kvm_handle_gfn_range() into an aging-specic helper, kvm_rmap_age_gfn_range(). In addition to purging a bunch of unnecessary boilerplate code, this sets the stage for aging rmap SPTEs outside of mmu_lock. Note, there's a small functional change, as kvm_test_age_gfn() will now return immediately if a young SPTE is found, whereas previously KVM would continue iterating over other levels. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 68 ++++++++++++++---------------------------- 1 file changed, 22 insertions(+), 46 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 0a33857d668a..88b656a1453d 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1596,25 +1596,6 @@ static bool __kvm_rmap_zap_gfn_range(struct kvm *kvm, start, end - 1, can_yield, true, flush); } =20 -typedef bool (*rmap_handler_t)(struct kvm *kvm, struct kvm_rmap_head *rmap= _head, - struct kvm_memory_slot *slot, gfn_t gfn, - int level); - -static __always_inline bool kvm_handle_gfn_range(struct kvm *kvm, - struct kvm_gfn_range *range, - rmap_handler_t handler) -{ - struct slot_rmap_walk_iterator iterator; - bool ret =3D false; - - for_each_slot_rmap_range(range->slot, PG_LEVEL_4K, KVM_MAX_HUGEPAGE_LEVEL, - range->start, range->end - 1, &iterator) - ret |=3D handler(kvm, iterator.rmap, range->slot, iterator.gfn, - iterator.level); - - return ret; -} - bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) { bool flush =3D false; @@ -1634,31 +1615,6 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm= _gfn_range *range) return flush; } =20 -static bool kvm_age_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, - struct kvm_memory_slot *slot, gfn_t gfn, int level) -{ - u64 *sptep; - struct rmap_iterator iter; - int young =3D 0; - - for_each_rmap_spte(rmap_head, &iter, sptep) - young |=3D mmu_spte_age(sptep); - - return young; -} - -static bool kvm_test_age_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_= head, - struct kvm_memory_slot *slot, gfn_t gfn, int level) -{ - u64 *sptep; - struct rmap_iterator iter; - - for_each_rmap_spte(rmap_head, &iter, sptep) - if (is_accessed_spte(*sptep)) - return true; - return false; -} - #define RMAP_RECYCLE_THRESHOLD 1000 =20 static void __rmap_add(struct kvm *kvm, @@ -1693,12 +1649,32 @@ static void rmap_add(struct kvm_vcpu *vcpu, const s= truct kvm_memory_slot *slot, __rmap_add(vcpu->kvm, cache, slot, spte, gfn, access); } =20 +static bool kvm_rmap_age_gfn_range(struct kvm *kvm, + struct kvm_gfn_range *range, bool test_only) +{ + struct slot_rmap_walk_iterator iterator; + struct rmap_iterator iter; + bool young =3D false; + u64 *sptep; + + for_each_slot_rmap_range(range->slot, PG_LEVEL_4K, KVM_MAX_HUGEPAGE_LEVEL, + range->start, range->end - 1, &iterator) { + for_each_rmap_spte(iterator.rmap, &iter, sptep) { + if (test_only && is_accessed_spte(*sptep)) + return true; + + young =3D mmu_spte_age(sptep); + } + } + return young; +} + bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { bool young =3D false; =20 if (kvm_memslots_have_rmaps(kvm)) - young =3D kvm_handle_gfn_range(kvm, range, kvm_age_rmap); + young =3D kvm_rmap_age_gfn_range(kvm, range, false); =20 if (tdp_mmu_enabled) young |=3D kvm_tdp_mmu_age_gfn_range(kvm, range); @@ -1711,7 +1687,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn= _range *range) bool young =3D false; =20 if (kvm_memslots_have_rmaps(kvm)) - young =3D kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap); + young =3D kvm_rmap_age_gfn_range(kvm, range, true); =20 if (tdp_mmu_enabled) young |=3D kvm_tdp_mmu_test_age_gfn(kvm, range); --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 580FD19939E for ; Fri, 9 Aug 2024 19:44:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232650; cv=none; b=GO4OHUGJETDvZtRpGUxkuMt+J/mccA5kG2EJ0LIQ6d0PnMZxdNP89GVxbp8P+3tUdZK09Seaz0TUqt9HBXfKvzHXsuWXbac35dImnoKUihDUtxExTRXo4fioD3FzVKeSO7C9WOnoMV/BxWh1pS/Wp3wDBDELkEGgMoBtejsnoIo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232650; c=relaxed/simple; bh=VG0ehnLRBNQjGjdZ62CiB3/PrgwQ8Ke9eyr0F1VXaGU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=lD+bOJdwCx5XeSs2dLoKWN9Zn+preR2HDgHQvVkYqnKSUgNPikRgmkY/2yngs3xraHE6cEXFbs39ZsKujLbyMuBwAk4AoLc0ciqOsFBgij44lVJs2m+olQ5HY/tHhff38YFWMdVAiYjhiy5d3g9i6vb5tGsTE7OsbcTHk+47Jqc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=3qlLhUcC; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="3qlLhUcC" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-71050384c9aso2463665b3a.1 for ; Fri, 09 Aug 2024 12:44:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232648; x=1723837448; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=Ai8EH7U6wTbiOPvc4tDQEcYQ+E/EtVyC6HGLhPebPF8=; b=3qlLhUcC97lB6vCpdUPWtkuyd4ObXkDIAeSa+M9klnKyX5L6H2S67M6HxaE6pIJQV6 VHdblS5NGUk7ROWxOvP08JH655dAuNTTe8bDHkbjBLuLlJWNHZOvNYsedBgMLnZ3Ih42 Ad+pyOTzngUWy5i90jrJOZ0+SSHdUBuePexfbsgSQEV1H8ND7msrLyv0YAvH8y7jBIWm oFzpkQT2F+GqtzX8yg8tr1Ydq6lDdvbJZAY+TqklSgsy00ybax5BxOMqpTSelcGvuWzp oWOsMgmdN5ioNYnLAAavhTjP8buGZ16QCyHZ6whtp79/Xf2+Tfxio2j6bc0BV9nowmCY BmAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232648; x=1723837448; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Ai8EH7U6wTbiOPvc4tDQEcYQ+E/EtVyC6HGLhPebPF8=; b=kO4wp274WAzHlqakQIXWo767dueYbhJNHkm/DIQHiPr0IqJma0+twmUwX+FmgHtPDq SUiSB2gb2oeogHZdanFL3Nx1Ty6VYrEuf0w1s6glRiTwYxMyR2MNGsrxfQWKcJF/c6L5 FBThAzFemca70VVjSkOiDkA3boi3yN9lEdImfjJZ0r1e5OLFTqwH+2eF9CXL6ROqqaNw X4lahBzhVGfTzcF0RhMj8S/a9aFg+OzGeAqYl9pmTy5PvdKmzk//A25m40NSPJUW0i02 Q/2GIs3VTakAY8Wabhwp/rbqT8RK1N9MoTXanvNC53SeuvXy4+lHFvVHDwcmq0K8CbCt ESuQ== X-Forwarded-Encrypted: i=1; AJvYcCUXbqoAwkAScxQvEDq68nomIoKkMT1DNLD+5egkASTbcfqd3fHsrhCUQc+RtQOpddQxCFbyuYaW9rfMlIYRNE1G7iTJZGPFBy/EgdHd X-Gm-Message-State: AOJu0Yxdvd7MVxmVK6EoNn+EID/NgYVnk1L4UOWlFJ+zrphtOYVD23+u VoRktP8J18kS4COdyir4uX7WhS5BAZhRRBcbnjKgq8Cqi2UurughJLY1gfny+ZCVkmx8lCA6Q5t p2g== X-Google-Smtp-Source: AGHT+IHNNFbniYTOQTi7bihux9iGSGXPTBj4ES/WY0YDfyXV3hzhWFmxyp3a86iRwwurPT1lphBR34jXitQ= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6a00:9167:b0:70d:1b0d:a15d with SMTP id d2e1a72fcca58-710ccb04a71mr122147b3a.3.1723232648511; Fri, 09 Aug 2024 12:44:08 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:27 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-16-seanjc@google.com> Subject: [PATCH 15/22] KVM: x86/mmu: Fold mmu_spte_age() into kvm_rmap_age_gfn_range() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Fold mmu_spte_age() into its sole caller now that aging and testing for young SPTEs is handled in a common location, i.e. doesn't require more helpers. Opportunistically remove the use of mmu_spte_get_lockless(), as mmu_lock is held (for write!), and marking SPTEs for access tracking outside of mmu_lock is unsafe (at least, as written). I.e. using the lockless accessor is quite misleading. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 50 +++++++++++++++++++----------------------- 1 file changed, 22 insertions(+), 28 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 88b656a1453d..c536a069d6b9 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -614,32 +614,6 @@ static u64 mmu_spte_get_lockless(u64 *sptep) return __get_spte_lockless(sptep); } =20 -/* Returns the Accessed status of the PTE and resets it at the same time. = */ -static bool mmu_spte_age(u64 *sptep) -{ - u64 spte =3D mmu_spte_get_lockless(sptep); - - if (!is_accessed_spte(spte)) - return false; - - if (spte_ad_enabled(spte)) { - clear_bit((ffs(shadow_accessed_mask) - 1), - (unsigned long *)sptep); - } else { - /* - * Capture the dirty status of the page, so that it doesn't get - * lost when the SPTE is marked for access tracking. - */ - if (is_writable_pte(spte)) - kvm_set_pfn_dirty(spte_to_pfn(spte)); - - spte =3D mark_spte_for_access_track(spte); - mmu_spte_update_no_track(sptep, spte); - } - - return true; -} - static inline bool is_tdp_mmu_active(struct kvm_vcpu *vcpu) { return tdp_mmu_enabled && vcpu->arch.mmu->root_role.direct; @@ -1660,10 +1634,30 @@ static bool kvm_rmap_age_gfn_range(struct kvm *kvm, for_each_slot_rmap_range(range->slot, PG_LEVEL_4K, KVM_MAX_HUGEPAGE_LEVEL, range->start, range->end - 1, &iterator) { for_each_rmap_spte(iterator.rmap, &iter, sptep) { - if (test_only && is_accessed_spte(*sptep)) + u64 spte =3D *sptep; + + if (!is_accessed_spte(spte)) + continue; + + if (test_only) return true; =20 - young =3D mmu_spte_age(sptep); + if (spte_ad_enabled(spte)) { + clear_bit((ffs(shadow_accessed_mask) - 1), + (unsigned long *)sptep); + } else { + /* + * Capture the dirty status of the page, so that + * it doesn't get lost when the SPTE is marked + * for access tracking. + */ + if (is_writable_pte(spte)) + kvm_set_pfn_dirty(spte_to_pfn(spte)); + + spte =3D mark_spte_for_access_track(spte); + mmu_spte_update_no_track(sptep, spte); + } + young =3D true; } } return young; --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D2B0199E9A for ; Fri, 9 Aug 2024 19:44:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232652; cv=none; b=c7XUzvERC9Z8KLCRfVUHubNOiSFWP4rI9DM192KIjtmVJOnolMHuDdXWj4QGj2fUwVSg9XCPV94CCrvQXp0U1BBsJbctXgCaEcWQmnp779rQzAHXVWX1+w3F1dAfsv64olfS603Z5c8cgOqxJLsAIzcFgPNwDBUYRDjEjIjHuI0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232652; c=relaxed/simple; bh=aE+XjLSYWb149BZo4fuwEt6PAsJA4S1vOO2cjFVxeZw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=MqqUlWJ0+7KPMiKf8/tCe2kNxSAmF6xV3MGVcH9Q0Z1uAup/2CrimJnwBJKL9nvvTQhg0dxKCCD4PhurlOKNY52G2c1lU4h8NgFQIYFVnThW9GCksAsYQoyzBh2TLo/9J6DLZnThoHDxyiSfOFFXow78E2wOponsHCunuGCJJ5E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=QQP9GjY+; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="QQP9GjY+" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6506bfeaf64so47165037b3.1 for ; Fri, 09 Aug 2024 12:44:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232650; x=1723837450; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=GD7gycGsmp7NjONVo+Jkt5X+cTf6SoGnpMVjud9Ddns=; b=QQP9GjY+WoiTj2kofWBAusfKONlPvEtPCxIeJuyN4ZihH7zFfe/XPLtryV2o8qn8mx R+rLn7JG10drMi/oFbfw/jQvdNNEaW+8M01e8GKin1KUWnDNUUCvogIGH4tkPYZu/yGO Ireh6K9jYRkb0mX+4rQVqNlhMz7vORYY300xEEPEU5KRkjeaJbgMskmxBOGMYHd9bxTS jlaHKa1+ves/F9zIqeFLSxWG8KzoX6NBNuQS0n3jCCPok6ulpT6zZMHEqzbxPdKwVUGA MTnv0imu4dlp45J7CsWYPuDA8Fn4di3vWDbWWV4y5h2lH2aCjFyMUK5EuA/4O34EnXph KZ0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232650; x=1723837450; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=GD7gycGsmp7NjONVo+Jkt5X+cTf6SoGnpMVjud9Ddns=; b=s9bpGy7F0q2bZJ52BJTVBLEh784Xzx9MjYN+R5ZIHFuwcMH9S0ugaEdNYYafi2K9Y7 pLI85LYCZLMDI1OQxaFTrrm6mLbGvojtNH1G/vJYSWWrNgxi96DoTmjmA5BSySMkpG/V s4xgYWv7H1+lwAYxPXFwA9Pl1SrYnVWmBi2B82my5TUxREAW+BdGIUGnI1DFu62PUNBb dSD/cqmEq3HURg4SS2w1SUFVaRYKa6CfrIxmHXDyngPhEFTQ0wEj0KTnkAuApgTnlzM1 kGLIg2q3QoNqgSoa1zTnqka4NCjZkFw3YSQrWsgiruj2c8C652fq8yYUkBj2Uhn6ICUK uawQ== X-Forwarded-Encrypted: i=1; AJvYcCWT6xFmMconTgqVVPgQnarfA4dRSCCAYyfq1rb1EcSOVsj5cGXs0xcetukwVtH+Hyks37flLOJrp7qFY4ccpYCMMWu3BpzaJGnZZByU X-Gm-Message-State: AOJu0YwnRI6ONcjNmk/ZRRsxvbXDrNsWnk8E50CTmnlurm4ch64orOFQ QgchjvCKpSaK22rLONQnlbUxhSYQN1OUdYtSbg9d252USCE+QDMD5Js0eqjraJo+wa/wb0/YJKI 6hA== X-Google-Smtp-Source: AGHT+IG5SwO0DcW0swDPhRWsKN2wuZhEs3mbWTMQ2R+czhdomrVn6Ywvmw+UK+vkinS9d2PnRXVcF1Gcbp4= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:690c:290f:b0:615:32e1:d82c with SMTP id 00721157ae682-69ec8d85594mr560077b3.6.1723232650446; Fri, 09 Aug 2024 12:44:10 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:28 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-17-seanjc@google.com> Subject: [PATCH 16/22] KVM: x86/mmu: Add KVM_RMAP_MANY to replace open coded '1' and '1ul' literals From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Replace all of the open coded '1' literals used to mark a PTE list as having many/multiple entries with a proper define. It's hard enough to read the code with one magic bit, and a future patch to support "locking" a single rmap will add another. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index c536a069d6b9..73569979130d 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -912,6 +912,7 @@ static struct kvm_memory_slot *gfn_to_memslot_dirty_bit= map(struct kvm_vcpu *vcpu * in this rmap chain. Otherwise, (rmap_head->val & ~1) points to a struct * pte_list_desc containing more mappings. */ +#define KVM_RMAP_MANY BIT(0) =20 /* * Returns the number of pointers in the rmap chain, not counting the new = one. @@ -924,16 +925,16 @@ static int pte_list_add(struct kvm_mmu_memory_cache *= cache, u64 *spte, =20 if (!rmap_head->val) { rmap_head->val =3D (unsigned long)spte; - } else if (!(rmap_head->val & 1)) { + } else if (!(rmap_head->val & KVM_RMAP_MANY)) { desc =3D kvm_mmu_memory_cache_alloc(cache); desc->sptes[0] =3D (u64 *)rmap_head->val; desc->sptes[1] =3D spte; desc->spte_count =3D 2; desc->tail_count =3D 0; - rmap_head->val =3D (unsigned long)desc | 1; + rmap_head->val =3D (unsigned long)desc | KVM_RMAP_MANY; ++count; } else { - desc =3D (struct pte_list_desc *)(rmap_head->val & ~1ul); + desc =3D (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); count =3D desc->tail_count + desc->spte_count; =20 /* @@ -942,10 +943,10 @@ static int pte_list_add(struct kvm_mmu_memory_cache *= cache, u64 *spte, */ if (desc->spte_count =3D=3D PTE_LIST_EXT) { desc =3D kvm_mmu_memory_cache_alloc(cache); - desc->more =3D (struct pte_list_desc *)(rmap_head->val & ~1ul); + desc->more =3D (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY= ); desc->spte_count =3D 0; desc->tail_count =3D count; - rmap_head->val =3D (unsigned long)desc | 1; + rmap_head->val =3D (unsigned long)desc | KVM_RMAP_MANY; } desc->sptes[desc->spte_count++] =3D spte; } @@ -956,7 +957,7 @@ static void pte_list_desc_remove_entry(struct kvm *kvm, struct kvm_rmap_head *rmap_head, struct pte_list_desc *desc, int i) { - struct pte_list_desc *head_desc =3D (struct pte_list_desc *)(rmap_head->v= al & ~1ul); + struct pte_list_desc *head_desc =3D (struct pte_list_desc *)(rmap_head->v= al & ~KVM_RMAP_MANY); int j =3D head_desc->spte_count - 1; =20 /* @@ -985,7 +986,7 @@ static void pte_list_desc_remove_entry(struct kvm *kvm, if (!head_desc->more) rmap_head->val =3D 0; else - rmap_head->val =3D (unsigned long)head_desc->more | 1; + rmap_head->val =3D (unsigned long)head_desc->more | KVM_RMAP_MANY; mmu_free_pte_list_desc(head_desc); } =20 @@ -998,13 +999,13 @@ static void pte_list_remove(struct kvm *kvm, u64 *spt= e, if (KVM_BUG_ON_DATA_CORRUPTION(!rmap_head->val, kvm)) return; =20 - if (!(rmap_head->val & 1)) { + if (!(rmap_head->val & KVM_RMAP_MANY)) { if (KVM_BUG_ON_DATA_CORRUPTION((u64 *)rmap_head->val !=3D spte, kvm)) return; =20 rmap_head->val =3D 0; } else { - desc =3D (struct pte_list_desc *)(rmap_head->val & ~1ul); + desc =3D (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); while (desc) { for (i =3D 0; i < desc->spte_count; ++i) { if (desc->sptes[i] =3D=3D spte) { @@ -1037,12 +1038,12 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm, if (!rmap_head->val) return false; =20 - if (!(rmap_head->val & 1)) { + if (!(rmap_head->val & KVM_RMAP_MANY)) { mmu_spte_clear_track_bits(kvm, (u64 *)rmap_head->val); goto out; } =20 - desc =3D (struct pte_list_desc *)(rmap_head->val & ~1ul); + desc =3D (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); =20 for (; desc; desc =3D next) { for (i =3D 0; i < desc->spte_count; i++) @@ -1062,10 +1063,10 @@ unsigned int pte_list_count(struct kvm_rmap_head *r= map_head) =20 if (!rmap_head->val) return 0; - else if (!(rmap_head->val & 1)) + else if (!(rmap_head->val & KVM_RMAP_MANY)) return 1; =20 - desc =3D (struct pte_list_desc *)(rmap_head->val & ~1ul); + desc =3D (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); return desc->tail_count + desc->spte_count; } =20 @@ -1127,13 +1128,13 @@ static u64 *rmap_get_first(struct kvm_rmap_head *rm= ap_head, if (!rmap_head->val) return NULL; =20 - if (!(rmap_head->val & 1)) { + if (!(rmap_head->val & KVM_RMAP_MANY)) { iter->desc =3D NULL; sptep =3D (u64 *)rmap_head->val; goto out; } =20 - iter->desc =3D (struct pte_list_desc *)(rmap_head->val & ~1ul); + iter->desc =3D (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); iter->pos =3D 0; sptep =3D iter->desc->sptes[iter->pos]; out: --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 36A9519A288 for ; Fri, 9 Aug 2024 19:44:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232654; cv=none; b=EjRr+Fwin2xj98MfkTK2HhjVwX6w87ofILtEmPMPJ70RKbbuDO9Qbco0CqrpfTL4m+DmYgbcSy3z6TUVtEzVeb18aykM4602Qcfcj8G/KGkgGuqdECuR6k+YQmcQycoFc/FLquaZhfpvDfmF2UE5R2DX8nMJ5rDlRBRmmxYTdho= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232654; c=relaxed/simple; bh=KiAcec9zQkYgeR0W9i8MR49CVqVo71Q4X9zgQyrpB/0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=tpnNzIyqzZzRg/vH20mJSz6t6DsyvFzGkukufVZ3sHMtkIROg4aJ1XMjv5FLABOcE68vaZwgLJFbK2fsUad+0VrYxjD4zYZNpLAL3Vqg+0ixlZ4NhQMFzbttCT/rfqJ6W7CUDEI1JfOBd7won2f3JDV+PZRE71jewG4hzmG2cMA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=JeOmsrn8; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="JeOmsrn8" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-7a3c6dd559aso2628573a12.1 for ; Fri, 09 Aug 2024 12:44:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232652; x=1723837452; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=LJq/GJ0moPmdVQ8FJN/LFNxKQSz1PiS8pWAScNqsM+k=; b=JeOmsrn85iNSMEOfGAbluITjOVjH7/ODm5wI5Xc5P6E/CDN5hcORuyFfFbjPhBAsne 3TASZGUndyPHczwmB/cfFV79gMgrpXLzwD5iXaOQrNIGXLQgboG+yVaI5a72DDjEpaXR rwMnwc0EYi612YYOmEc9OlpTKeOlyNuU/W7EaJCCSK7WFp6juLnrYwnuEMdANW36t34m 4+W8AC21Tg1UhUHUWxz/Ts8vCSIpJRrszFmJed8+LPAOIquxyGXMIg2w0sXVoHovIcYO wEePlIftCbuFQ9xewWaiKu+KmZR9MHT+wJH8A6XLUCRQRKn3jn26PQeOQHnl+Tq5zrAH JGKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232652; x=1723837452; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=LJq/GJ0moPmdVQ8FJN/LFNxKQSz1PiS8pWAScNqsM+k=; b=QDUlZUyQkuj+W9LGAojpWh9D1N13PnTY24nDf4b4gSuAeX+oaL10fz08cSeTfHohHL 0/5rgtZS5jx5fPGdBGnbEEB7/cdcMy1j4NWCEsiJONtR8SeFmqmeV8Yu4DWQP1Ga7GCT oADPIxWyss5biz3GhdhjW/FneNsW/uu14jyM+ob4QiUed9JrHzMriDEJKPaQPIMIg4nL 0Bu7gpHnfU25qsl6lVc8/CPV5QAnMexSKP3+n13xcZyjq6x2Ava7a+pcdbnzsh53xEZn wQguMnCvEMyZ6wIslL3AmeYo2e4rQO3ydrZXgf4kFsXR2NLdkfMxiBz4YRAtVDYbSgz8 ns/A== X-Forwarded-Encrypted: i=1; AJvYcCVEpe5Ec+6l59+JrsaELYgVWPXo+crmQViKZLLDCIZq4SYVrXFBe5+Eid5vAg/2Z69e8GPyt3f5zsVZxWHfhnhB5cJj4wz8CcDAU77p X-Gm-Message-State: AOJu0YwKn8fNlB43UIPdKJLP5Ew8SmkZFNUEbIFb6PKLBFBQP8++3kCz 1WRRmqldzel4uwNxaoH21csq6ZjbHAaLy/TObOy8UXSQjfOtHlHOTYd7qmJPe+cKWpfidLYUBQc 1EQ== X-Google-Smtp-Source: AGHT+IG9iS1KQa10Rf8CvbcbSW/02Vokoqft8BTRa6S8EoVMPZ81K0TgdGJVD2ND164posnnJw8HgWteDns= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a63:3714:0:b0:717:a912:c302 with SMTP id 41be03b00d2f7-7c3d2b78b02mr5155a12.1.1723232652349; Fri, 09 Aug 2024 12:44:12 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:29 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-18-seanjc@google.com> Subject: [PATCH 17/22] KVM: x86/mmu: Refactor low level rmap helpers to prep for walking w/o mmu_lock From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Refactor the pte_list and rmap code to always read and write rmap_head->val exactly once, e.g. by collecting changes in a local variable and then propagating those changes back to rmap_head->val as appropriate. This will allow implementing a per-rmap rwlock (of sorts) by adding a LOCKED bit into the rmap value alongside the MANY bit. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 83 +++++++++++++++++++++++++----------------- 1 file changed, 50 insertions(+), 33 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 73569979130d..49c1f6cc1526 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -920,21 +920,24 @@ static struct kvm_memory_slot *gfn_to_memslot_dirty_b= itmap(struct kvm_vcpu *vcpu static int pte_list_add(struct kvm_mmu_memory_cache *cache, u64 *spte, struct kvm_rmap_head *rmap_head) { + unsigned long old_val, new_val; struct pte_list_desc *desc; int count =3D 0; =20 - if (!rmap_head->val) { - rmap_head->val =3D (unsigned long)spte; - } else if (!(rmap_head->val & KVM_RMAP_MANY)) { + old_val =3D rmap_head->val; + + if (!old_val) { + new_val =3D (unsigned long)spte; + } else if (!(old_val & KVM_RMAP_MANY)) { desc =3D kvm_mmu_memory_cache_alloc(cache); - desc->sptes[0] =3D (u64 *)rmap_head->val; + desc->sptes[0] =3D (u64 *)old_val; desc->sptes[1] =3D spte; desc->spte_count =3D 2; desc->tail_count =3D 0; - rmap_head->val =3D (unsigned long)desc | KVM_RMAP_MANY; + new_val =3D (unsigned long)desc | KVM_RMAP_MANY; ++count; } else { - desc =3D (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); + desc =3D (struct pte_list_desc *)(old_val & ~KVM_RMAP_MANY); count =3D desc->tail_count + desc->spte_count; =20 /* @@ -943,21 +946,25 @@ static int pte_list_add(struct kvm_mmu_memory_cache *= cache, u64 *spte, */ if (desc->spte_count =3D=3D PTE_LIST_EXT) { desc =3D kvm_mmu_memory_cache_alloc(cache); - desc->more =3D (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY= ); + desc->more =3D (struct pte_list_desc *)(old_val & ~KVM_RMAP_MANY); desc->spte_count =3D 0; desc->tail_count =3D count; - rmap_head->val =3D (unsigned long)desc | KVM_RMAP_MANY; + new_val =3D (unsigned long)desc | KVM_RMAP_MANY; + } else { + new_val =3D old_val; } desc->sptes[desc->spte_count++] =3D spte; } + + rmap_head->val =3D new_val; + return count; } =20 -static void pte_list_desc_remove_entry(struct kvm *kvm, - struct kvm_rmap_head *rmap_head, +static void pte_list_desc_remove_entry(struct kvm *kvm, unsigned long *rma= p_val, struct pte_list_desc *desc, int i) { - struct pte_list_desc *head_desc =3D (struct pte_list_desc *)(rmap_head->v= al & ~KVM_RMAP_MANY); + struct pte_list_desc *head_desc =3D (struct pte_list_desc *)(*rmap_val & = ~KVM_RMAP_MANY); int j =3D head_desc->spte_count - 1; =20 /* @@ -984,9 +991,9 @@ static void pte_list_desc_remove_entry(struct kvm *kvm, * head at the next descriptor, i.e. the new head. */ if (!head_desc->more) - rmap_head->val =3D 0; + *rmap_val =3D 0; else - rmap_head->val =3D (unsigned long)head_desc->more | KVM_RMAP_MANY; + *rmap_val =3D (unsigned long)head_desc->more | KVM_RMAP_MANY; mmu_free_pte_list_desc(head_desc); } =20 @@ -994,24 +1001,26 @@ static void pte_list_remove(struct kvm *kvm, u64 *sp= te, struct kvm_rmap_head *rmap_head) { struct pte_list_desc *desc; + unsigned long rmap_val; int i; =20 - if (KVM_BUG_ON_DATA_CORRUPTION(!rmap_head->val, kvm)) - return; + rmap_val =3D rmap_head->val; + if (KVM_BUG_ON_DATA_CORRUPTION(!rmap_val, kvm)) + goto out; =20 - if (!(rmap_head->val & KVM_RMAP_MANY)) { - if (KVM_BUG_ON_DATA_CORRUPTION((u64 *)rmap_head->val !=3D spte, kvm)) - return; + if (!(rmap_val & KVM_RMAP_MANY)) { + if (KVM_BUG_ON_DATA_CORRUPTION((u64 *)rmap_val !=3D spte, kvm)) + goto out; =20 - rmap_head->val =3D 0; + rmap_val =3D 0; } else { - desc =3D (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); + desc =3D (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY); while (desc) { for (i =3D 0; i < desc->spte_count; ++i) { if (desc->sptes[i] =3D=3D spte) { - pte_list_desc_remove_entry(kvm, rmap_head, + pte_list_desc_remove_entry(kvm, &rmap_val, desc, i); - return; + goto out; } } desc =3D desc->more; @@ -1019,6 +1028,9 @@ static void pte_list_remove(struct kvm *kvm, u64 *spt= e, =20 KVM_BUG_ON_DATA_CORRUPTION(true, kvm); } + +out: + rmap_head->val =3D rmap_val; } =20 static void kvm_zap_one_rmap_spte(struct kvm *kvm, @@ -1033,17 +1045,19 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm, struct kvm_rmap_head *rmap_head) { struct pte_list_desc *desc, *next; + unsigned long rmap_val; int i; =20 - if (!rmap_head->val) + rmap_val =3D rmap_head->val; + if (!rmap_val) return false; =20 - if (!(rmap_head->val & KVM_RMAP_MANY)) { - mmu_spte_clear_track_bits(kvm, (u64 *)rmap_head->val); + if (!(rmap_val & KVM_RMAP_MANY)) { + mmu_spte_clear_track_bits(kvm, (u64 *)rmap_val); goto out; } =20 - desc =3D (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); + desc =3D (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY); =20 for (; desc; desc =3D next) { for (i =3D 0; i < desc->spte_count; i++) @@ -1059,14 +1073,15 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm, =20 unsigned int pte_list_count(struct kvm_rmap_head *rmap_head) { + unsigned long rmap_val =3D rmap_head->val; struct pte_list_desc *desc; =20 - if (!rmap_head->val) + if (!rmap_val) return 0; - else if (!(rmap_head->val & KVM_RMAP_MANY)) + else if (!(rmap_val & KVM_RMAP_MANY)) return 1; =20 - desc =3D (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); + desc =3D (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY); return desc->tail_count + desc->spte_count; } =20 @@ -1109,6 +1124,7 @@ static void rmap_remove(struct kvm *kvm, u64 *spte) */ struct rmap_iterator { /* private fields */ + struct rmap_head *head; struct pte_list_desc *desc; /* holds the sptep if not NULL */ int pos; /* index of the sptep */ }; @@ -1123,18 +1139,19 @@ struct rmap_iterator { static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head, struct rmap_iterator *iter) { + unsigned long rmap_val =3D rmap_head->val; u64 *sptep; =20 - if (!rmap_head->val) + if (!rmap_val) return NULL; =20 - if (!(rmap_head->val & KVM_RMAP_MANY)) { + if (!(rmap_val & KVM_RMAP_MANY)) { iter->desc =3D NULL; - sptep =3D (u64 *)rmap_head->val; + sptep =3D (u64 *)rmap_val; goto out; } =20 - iter->desc =3D (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); + iter->desc =3D (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY); iter->pos =3D 0; sptep =3D iter->desc->sptes[iter->pos]; out: --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5008F19AA68 for ; Fri, 9 Aug 2024 19:44:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232656; cv=none; b=CW2K5c7kiOtqph/4vQU0HkYzlAQO/LnlWKEvfnSOKq4/0Zow3ANqOeEb2cGLKEPRGuDfgYHmDI3w9q7TUtUV06DcOz0+q/V21Ll11B50SI6RxDujttCBs85lLMYESMniEtJyb1LOASL3h0AHDX9M0FPx49Qo8f3Od05zEGg+dRE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232656; c=relaxed/simple; bh=lGLMFgBOB+zT1nvZAyf/Zc7DkG5GaKVPXMH41Wm07lY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=qayJVZY8a9GtCSVeUY8R74Lyrzoh68iJA8JaVgdZozGe1OkNmnSwdsDk0499a83LTA2RYnQd/Y91c6X/8f8sErOq2zmR3kk3t3PUw3gR3/pA4macwrs93rGMPO9UNLgeiNj+FreNZg35e2b4icFqjW0pk/WSArx3TWCLQ1DhcMs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=tDvNCmaS; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="tDvNCmaS" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-1fd774c3b8eso29510745ad.0 for ; Fri, 09 Aug 2024 12:44:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232654; x=1723837454; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=gQyjT/J4Ba5hX1ofSr3HGlFYc34S1qyVyBiZGiETo24=; b=tDvNCmaSDyqOauFuB0GWJbupGiwGO8rmxX2bpgQzlrI5g57Kcf7X1Faa8FSlpolcLL GvFtAEyKjHolXIqtxftBtftG/Z+k8t1KRJE42VhaKYK4Jw7866V0YIpmVGdSRHR5Fi3I SsiE9MshFRX+GuDK4ZoVsRyj48km10uh7kVg/juZWCCgbyxhWzWg2+HBXOfOIHA0Up4n aXYMr6EBW5K7GwI2+fXxrE70TbcG83xrjQdqxh0I3sAMqPqDuxEsgEZf/gf0jNT+bwEl M7p/NEh1zNOc3mC3UwznIb5eBLgT3UEDUkzeogRIHbQB0EC4A+I/swdOwWHZe1cIDi3K V4aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232654; x=1723837454; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=gQyjT/J4Ba5hX1ofSr3HGlFYc34S1qyVyBiZGiETo24=; b=nZFQqyaBjA/BOLz+6Ucb4eVbvraKrBl3orK7aEX7/ZOAax0A0Z9Z2LLKCIeh0Ryubf SDhLtX32pULA0txVLKbHi3MSt2w9xqv/HuZDtfqw1i65UzIZ+p7fqed9HCatyYB96OWA V4j6L06CYJpQCL3ReJtIApKLqlwitqW1nkhQonbCZcaXzxBKMeXkyJs+Rxw5Ddp1O30Z c5XiYTjEh7u51hGSiMdwa/vJQ6FdCjr/RFZH0OOqHFWZO/n/xBsHwsTuJ4y+GwgPcETZ j/9l+4ZzlxoD0KHhQI/dLZ+45hKrHLIGUdCEqND/GOy07Vkm/576UPGheHr0RRagPpPJ 5YUQ== X-Forwarded-Encrypted: i=1; AJvYcCV8MujZvKeJRovOetLpJaQeyoyUOAkXLsTd0UnpejVDEgpc9fFWzMhaaOFSwGcgPwoBXTu9gUWycSZ1e6+6MwL3nRz0Ggf60CEVQ1B0 X-Gm-Message-State: AOJu0YyDALv9qd9ic21MJtDsxF7lDBwNj/Xuj9KdwHYeEso+u38jqiNT jTpnd7KfCE7qv0LslCSM0BrEP2yy29qM6M+knnfEK8wTJcSq9eRVpmqhhxqbj87VXtItRnQ46z3 XJQ== X-Google-Smtp-Source: AGHT+IE7aWrPpp/r00UqCmma5DjJiz0mB/4c0ahEJTDpFl71+olIXEHExH9+iNKK22JWV2AmgsyXWas3vJg= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:f689:b0:1fa:1a78:b5bc with SMTP id d9443c01a7336-200ae2a6d55mr1531575ad.0.1723232654489; Fri, 09 Aug 2024 12:44:14 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:30 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-19-seanjc@google.com> Subject: [PATCH 18/22] KVM: x86/mmu: Use KVM_PAGES_PER_HPAGE() instead of an open coded equivalent From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use KVM_PAGES_PER_HPAGE() instead of open coding equivalent logic that is anything but obvious. No functional change intended, and verified by compiling with the below assertions: BUILD_BUG_ON((1UL << KVM_HPAGE_GFN_SHIFT(PG_LEVEL_4K)) !=3D KVM_PAGES_PER_HPAGE(PG_LEVEL_4K)); BUILD_BUG_ON((1UL << KVM_HPAGE_GFN_SHIFT(PG_LEVEL_2M)) !=3D KVM_PAGES_PER_HPAGE(PG_LEVEL_2M)); BUILD_BUG_ON((1UL << KVM_HPAGE_GFN_SHIFT(PG_LEVEL_1G)) !=3D KVM_PAGES_PER_HPAGE(PG_LEVEL_1G)); Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 49c1f6cc1526..8ca7f51c2da3 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1499,7 +1499,7 @@ static bool slot_rmap_walk_okay(struct slot_rmap_walk= _iterator *iterator) static void slot_rmap_walk_next(struct slot_rmap_walk_iterator *iterator) { while (++iterator->rmap <=3D iterator->end_rmap) { - iterator->gfn +=3D (1UL << KVM_HPAGE_GFN_SHIFT(iterator->level)); + iterator->gfn +=3D KVM_PAGES_PER_HPAGE(iterator->level); =20 if (iterator->rmap->val) return; --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 768BF19ADB0 for ; Fri, 9 Aug 2024 19:44:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232659; cv=none; b=gYlo8Jij1aZ6m4VKQhCNjRH2R+fcyIhfsYfh5sKQ2Foow3IhoFsCdTBp2F1Sl7I2mTtxiWH5lXB+aYYuFmGgA1nEr1jQRPKpQd1JPOsqB+bG2uDfeY4jufavEXdr6ssH9yoyGqVE//qnonv93MhG+iL8ErNre/ysijnX0iaU4wY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232659; c=relaxed/simple; bh=5rReX2g7eJHcUC+j0NjSl5Msvx0o5Y1O/Iq1eeDsNj4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=IoF7pdMD+aHRrfFq8GqvHbE+SY/WG/EzWY+EVpeMZGXkQa34+J7a3Oio0XFblCDOUvXP/qTDOeyBpoaIctqrB0JJTWUOfE2ocpfo7fSspNzHVdTaqJAuMQKI7kUEuNx51RV+xGaLDtupD+I9Mj1N5wMWHnwFMHsI9hEUJ4oPF8g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=UGmUI3G0; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="UGmUI3G0" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-672bea19bedso56724557b3.3 for ; Fri, 09 Aug 2024 12:44:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232656; x=1723837456; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=kaQkfw19FaVNluamR3vKVuhloxrU0TCLWsyQPJoUOaw=; b=UGmUI3G0LPbglxhO0HahDxJaWVCfKe2K1LDPTW0hprlQiKr4n4ItnsaK7ttuAIOyGA WaLQW4G7HMbvnfb7XULss7ilkhWNYwz6IzwtmFz3r7HgIMG1a5tVIg6eYjOnOkjZZYpN CAdUTLn16zZSU3Vqr5pAdMsydj8qIIIMucciSlVE2WJw0xTB0DNwswId5yOFQen4yuFY LG36TKXU/sNFMv40d8rfgytSYowHforLdGojtETZxR3NNhTRDh+pxr0WZAQSeROrvi3+ HCSDN2jtmUcgq/NipXxX69oynrW/s7DeKhLpyPB/sGu1pZzYDo/EcMF8dNLDJ3RKEg3f Sl4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232656; x=1723837456; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=kaQkfw19FaVNluamR3vKVuhloxrU0TCLWsyQPJoUOaw=; b=X+BFiujxWfRfkIxJKjBBkJL0YMQIyORkznOik+W/TNGIqCnRXk9odT7oncaxcKYKFs ILskcvTgKa1l6Ff83um4KJUObVJc0iN27Q59rtFQEmeaDvY5RRYHNCdgZ8WE9xNDkxnp bGQyY6uN0zf+WN/arIuumM9cNlfpj6Upn2n2LI4VC2FIB9/rVzRJ66T4XlegGEg/wwzH aN85GcbyTaAMiNOHAikiGX0Qg/SlTlfv249yox78CyIYVBJkXQZl9wtdti4iMvpc4pEB h8DCjXsIiSnUFmjlM/Qsmaa614xpVA/GGoR86yMoQKUsKNxzE9pe1Q4XgWgjR0J1w1FY mS/w== X-Forwarded-Encrypted: i=1; AJvYcCXGw8AVykpiOgviPqDoexNp11FH+Rp0c26vf4R5tKG0zUEKYOXiLRyT/Ql/iPrZ9pNxeVX9SGWiIWPly7ixbRpbQJannRjair3xTcyo X-Gm-Message-State: AOJu0YzUNrlGatKevYQjGb0I9Kp0B9/GoIbOanKgdkHIJuXoKZFBTldy 0Sp1ORNGyxt6CAwKiBH1qHMhnM05M5TierQMwbu4DSmGS9V0hcE/+X3qaQ45d73DmCTV5m9IDFn e6Q== X-Google-Smtp-Source: AGHT+IEBMWPkgI1k2a5Dd4JPhwd4nlairCp4gJ2z3JSwf2Dhnd/SN6RWtBOMd6oz6Ibb0/SIzT9/HBsYOT8= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a5b:e8e:0:b0:e0b:cce3:45c7 with SMTP id 3f1490d57ef6-e0eb99ff70amr4436276.9.1723232656534; Fri, 09 Aug 2024 12:44:16 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:31 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-20-seanjc@google.com> Subject: [PATCH 19/22] KVM: x86/mmu: Add infrastructure to allow walking rmaps outside of mmu_lock From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Steal another bit from rmap entries (which are word aligned pointers, i.e. have 2 free bits on 32-bit KVM, and 3 free bits on 64-bit KVM), and use the bit to implement a *very* rudimentary per-rmap spinlock. The only anticipated usage of the lock outside of mmu_lock is for aging gfns, and collisions between aging and other MMU rmap operations are quite rare, e.g. unless userspace is being silly and aging a tiny range over and over in a tight loop, time between contention when aging an actively running VM is O(seconds). In short, a more sophisticated locking scheme shouldn't be necessary. Note, the lock only protects the rmap structure itself, SPTEs that are pointed at by a locked rmap can still be modified and zapped by another task (KVM drops/zaps SPTEs before deleting the rmap entries) Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 80 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 71 insertions(+), 9 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 8ca7f51c2da3..a683b5fc4026 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -909,11 +909,73 @@ static struct kvm_memory_slot *gfn_to_memslot_dirty_b= itmap(struct kvm_vcpu *vcpu * About rmap_head encoding: * * If the bit zero of rmap_head->val is clear, then it points to the only = spte - * in this rmap chain. Otherwise, (rmap_head->val & ~1) points to a struct + * in this rmap chain. Otherwise, (rmap_head->val & ~3) points to a struct * pte_list_desc containing more mappings. */ #define KVM_RMAP_MANY BIT(0) =20 +/* + * rmaps and PTE lists are mostly protected by mmu_lock (the shadow MMU al= ways + * operates with mmu_lock held for write), but rmaps can be walked without + * holding mmu_lock so long as the caller can tolerate SPTEs in the rmap c= hain + * being zapped/dropped _while the rmap is locked_. + * + * Other than the KVM_RMAP_LOCKED flag, modifications to rmap entries must= be + * done while holding mmu_lock for write. This allows a task walking rmaps + * without holding mmu_lock to concurrently walk the same entries as a task + * that is holding mmu_lock but _not_ the rmap lock. Neither task will mo= dify + * the rmaps, thus the walks are stable. + * + * As alluded to above, SPTEs in rmaps are _not_ protected by KVM_RMAP_LOC= KED, + * only the rmap chains themselves are protected. E.g. holding an rmap's = lock + * ensures all "struct pte_list_desc" fields are stable. + */ +#define KVM_RMAP_LOCKED BIT(1) + +static unsigned long kvm_rmap_lock(struct kvm_rmap_head *rmap_head) +{ + unsigned long old_val, new_val; + + old_val =3D READ_ONCE(rmap_head->val); + if (!old_val) + return 0; + + do { + /* + * If the rmap is locked, wait for it to be unlocked before + * trying acquire the lock, e.g. to bounce the cache line. + */ + while (old_val & KVM_RMAP_LOCKED) { + old_val =3D READ_ONCE(rmap_head->val); + cpu_relax(); + } + + /* + * Recheck for an empty rmap, it may have been purged by the + * task that held the lock. + */ + if (!old_val) + return 0; + + new_val =3D old_val | KVM_RMAP_LOCKED; + } while (!try_cmpxchg(&rmap_head->val, &old_val, new_val)); + + /* Return the old value, i.e. _without_ the LOCKED bit set. */ + return old_val; +} + +static void kvm_rmap_unlock(struct kvm_rmap_head *rmap_head, + unsigned long new_val) +{ + WARN_ON_ONCE(new_val & KVM_RMAP_LOCKED); + WRITE_ONCE(rmap_head->val, new_val); +} + +static unsigned long kvm_rmap_get(struct kvm_rmap_head *rmap_head) +{ + return READ_ONCE(rmap_head->val) & ~KVM_RMAP_LOCKED; +} + /* * Returns the number of pointers in the rmap chain, not counting the new = one. */ @@ -924,7 +986,7 @@ static int pte_list_add(struct kvm_mmu_memory_cache *ca= che, u64 *spte, struct pte_list_desc *desc; int count =3D 0; =20 - old_val =3D rmap_head->val; + old_val =3D kvm_rmap_lock(rmap_head); =20 if (!old_val) { new_val =3D (unsigned long)spte; @@ -956,7 +1018,7 @@ static int pte_list_add(struct kvm_mmu_memory_cache *c= ache, u64 *spte, desc->sptes[desc->spte_count++] =3D spte; } =20 - rmap_head->val =3D new_val; + kvm_rmap_unlock(rmap_head, new_val); =20 return count; } @@ -1004,7 +1066,7 @@ static void pte_list_remove(struct kvm *kvm, u64 *spt= e, unsigned long rmap_val; int i; =20 - rmap_val =3D rmap_head->val; + rmap_val =3D kvm_rmap_lock(rmap_head); if (KVM_BUG_ON_DATA_CORRUPTION(!rmap_val, kvm)) goto out; =20 @@ -1030,7 +1092,7 @@ static void pte_list_remove(struct kvm *kvm, u64 *spt= e, } =20 out: - rmap_head->val =3D rmap_val; + kvm_rmap_unlock(rmap_head, rmap_val); } =20 static void kvm_zap_one_rmap_spte(struct kvm *kvm, @@ -1048,7 +1110,7 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm, unsigned long rmap_val; int i; =20 - rmap_val =3D rmap_head->val; + rmap_val =3D kvm_rmap_lock(rmap_head); if (!rmap_val) return false; =20 @@ -1067,13 +1129,13 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm, } out: /* rmap_head is meaningless now, remember to reset it */ - rmap_head->val =3D 0; + kvm_rmap_unlock(rmap_head, 0); return true; } =20 unsigned int pte_list_count(struct kvm_rmap_head *rmap_head) { - unsigned long rmap_val =3D rmap_head->val; + unsigned long rmap_val =3D kvm_rmap_get(rmap_head); struct pte_list_desc *desc; =20 if (!rmap_val) @@ -1139,7 +1201,7 @@ struct rmap_iterator { static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head, struct rmap_iterator *iter) { - unsigned long rmap_val =3D rmap_head->val; + unsigned long rmap_val =3D kvm_rmap_get(rmap_head); u64 *sptep; =20 if (!rmap_val) --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1985319B3E2 for ; Fri, 9 Aug 2024 19:44:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232660; cv=none; b=bZP4ztqMJaCx7We3UuUOQRWKb7irleGhDrBJvuI6yzyycH1c13FbYR11IPv2ANoYnTE+A656P4A9tIF1L1Hj6+VFs125MBdPjQqG+O0M4VWC3Alp+m5qxObdBipLjO5Rmv7FsH61lqdk7a3gIWeOSKB7XxBSTDsJ59QdqmJOQvY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232660; c=relaxed/simple; bh=UJIPN/Wqw44nvsetoo32cIAvD96dImRh3JnAszJC0xg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=PBlGDHw1oYonVMDWQ4UZsRxby32UvEY1EJsg6QUwixhw9iTIkghwtcxoZ64x8F2YIFyTeUguUyu+Bg3GVSSL85kSsx0kaKzF29CdfKeJhJwAmxRCCLR1jMXwtCl2XuvcnsBh39xmzLL4FFVKxUaqk1pxClzw+lOFGgEf9Vja+Oc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=MCNWDJlp; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="MCNWDJlp" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-1fc658a161bso20006595ad.0 for ; Fri, 09 Aug 2024 12:44:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232658; x=1723837458; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=/UfIOuTzGEIV2Af0nH4NhXy855i+DLQnfbOw4mjIiPM=; b=MCNWDJlp/eBX60WFUNv999DoOKjZAIiTfD6dX5B43XmFxdoJXe86Ng0oE9uaGtAyXw YpZMW6YKqMduUVt8xbcv3RpfviAG2X7QYOyEFcTTXuyB/vcrJFU6d/LCaff5jcJukoo4 tpUb6cA8QoWSW4OJoUYHzQ9q604cURYqDIHex99DHuhsS2A9Y1G7yH2DWztDGkXSzvsp Ne14ArelML/Pc/yDL0h+Mw58YNmOJ53KIL5M7tsU9X8UK2HiFEhb42qx1uoLcBp0vp9E RiVWZLs+jOnGp4SwYcq3nnh5zaZBjIhCrRp+YSyfi00OZM4f3uoYfdoeScgXId6olF56 lR2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232658; x=1723837458; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/UfIOuTzGEIV2Af0nH4NhXy855i+DLQnfbOw4mjIiPM=; b=czKki5SeaQHDkFenjsSF2DIMN7rtrKTty7HurL+SsTQjIvDoJSiXmhkXvnv94Z2Z3K Ph/g3ByIMyzwn6Wt07j4GtLQ4ZOjV4q2yY2VIpTr2MHdfugHydSsFP6uiydhIpTyXqoR 0NGD9ncf4Z/7lH5QTrFYOHN9/Z3Gu9eLGpenpVurD875jA+S3d5diaQdY9DJ5TCpELaN aEGSQ+J/E+ahqQ/c7x3AkZEik8zVX8e/OxEjL96oEqMcafBaAC8VzJJjJICUU/7++Xsq RZKDexnJxFDzqmTlOoARiFbZIXWimJH1vZq5RbwbyImDqI28Qgv5XxkcRHRoO1+Zs2Aq KtyA== X-Forwarded-Encrypted: i=1; AJvYcCVH5Ke/LL2RsezowjqW+OLSWMHbBTelJQDxnQgkP4xCoTBGnAaYK1ap2hrZEOooIo3r9Z6KxJw3Hd68ddsiRu2XmC0U0arGLJ5vRhrD X-Gm-Message-State: AOJu0YyeAlNfxasTM9KI/fEi+s/JB4hXgfK1Ajp78ViJgcP6pYecGOn6 6kQSgB1eC2Ts8irjTeZAYlomDWtdPgFzR2DYUN1l+8FiZBNg8/QcxvQONv/SVA1NkyxBdVEnnYw STA== X-Google-Smtp-Source: AGHT+IGIFyto8PfbQoPuRgcHzc8y0pWxF/kkpM1SeNnq1V1bACAA4fUhirUzmNwmEGbz15VyaKrug3Yf37Q= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:e74e:b0:1ff:39d7:a1a4 with SMTP id d9443c01a7336-200ae5d8a1cmr1519355ad.12.1723232658496; Fri, 09 Aug 2024 12:44:18 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:32 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-21-seanjc@google.com> Subject: [PATCH 20/22] KVM: x86/mmu: Add support for lockless walks of rmap SPTEs From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a lockless version of for_each_rmap_spte(), which is pretty much the same as the normal version, except that it doesn't BUG() the host if a non-present SPTE is encountered. When mmu_lock is held, it should be impossible for a different task to zap a SPTE, _and_ zapped SPTEs must be removed from their rmap chain prior to dropping mmu_lock. Thus, the normal walker BUG()s if a non-present SPTE is encountered as something is wildly broken. When walking rmaps without holding mmu_lock, the SPTEs pointed at by the rmap chain can be zapped/dropped, and so a lockless walk can observe a non-present SPTE if it runs concurrently with a different operation that is zapping SPTEs. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 86 +++++++++++++++++++++++++++++++----------- 1 file changed, 63 insertions(+), 23 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a683b5fc4026..48e8608c2738 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -932,10 +932,16 @@ static struct kvm_memory_slot *gfn_to_memslot_dirty_b= itmap(struct kvm_vcpu *vcpu */ #define KVM_RMAP_LOCKED BIT(1) =20 -static unsigned long kvm_rmap_lock(struct kvm_rmap_head *rmap_head) +static unsigned long __kvm_rmap_lock(struct kvm_rmap_head *rmap_head) { unsigned long old_val, new_val; =20 + /* + * Elide the lock if the rmap is empty, as lockless walkers (read-only + * mode) don't need to (and can't) walk an empty rmap, nor can they add + * entries to the rmap. I.e. the only paths that process empty rmaps + * do so while holding mmu_lock for write, and are mutually exclusive. + */ old_val =3D READ_ONCE(rmap_head->val); if (!old_val) return 0; @@ -960,17 +966,53 @@ static unsigned long kvm_rmap_lock(struct kvm_rmap_he= ad *rmap_head) new_val =3D old_val | KVM_RMAP_LOCKED; } while (!try_cmpxchg(&rmap_head->val, &old_val, new_val)); =20 - /* Return the old value, i.e. _without_ the LOCKED bit set. */ + /* + * Return the old value, i.e. _without_ the LOCKED bit set. It's + * impossible for the return value to be 0 (see above), i.e. the read- + * only unlock flow can't get a false positive and fail to unlock. + */ return old_val; } =20 +static unsigned long kvm_rmap_lock(struct kvm_rmap_head *rmap_head) +{ + /* + * TODO: Plumb in @kvm and add a lockdep assertion that mmu_lock is + * held for write. + */ + return __kvm_rmap_lock(rmap_head); +} + static void kvm_rmap_unlock(struct kvm_rmap_head *rmap_head, unsigned long new_val) { - WARN_ON_ONCE(new_val & KVM_RMAP_LOCKED); + KVM_MMU_WARN_ON(new_val & KVM_RMAP_LOCKED); WRITE_ONCE(rmap_head->val, new_val); } =20 +/* + * If mmu_lock isn't held, rmaps can only locked in read-only mode. The a= ctual + * locking is the same, but the caller is disallowed from modifying the rm= ap, + * and so the unlock flow is a nop if the rmap is/was empty. + */ +__maybe_unused +static unsigned long kvm_rmap_lock_readonly(struct kvm_rmap_head *rmap_hea= d) +{ + return __kvm_rmap_lock(rmap_head); +} + +__maybe_unused +static void kvm_rmap_unlock_readonly(struct kvm_rmap_head *rmap_head, + unsigned long old_val) +{ + if (!old_val) + return; + + KVM_MMU_WARN_ON(old_val !=3D (rmap_head->val & ~KVM_RMAP_LOCKED)); + WRITE_ONCE(rmap_head->val, old_val); +} + + static unsigned long kvm_rmap_get(struct kvm_rmap_head *rmap_head) { return READ_ONCE(rmap_head->val) & ~KVM_RMAP_LOCKED; @@ -1202,23 +1244,18 @@ static u64 *rmap_get_first(struct kvm_rmap_head *rm= ap_head, struct rmap_iterator *iter) { unsigned long rmap_val =3D kvm_rmap_get(rmap_head); - u64 *sptep; =20 if (!rmap_val) return NULL; =20 if (!(rmap_val & KVM_RMAP_MANY)) { iter->desc =3D NULL; - sptep =3D (u64 *)rmap_val; - goto out; + return (u64 *)rmap_val; } =20 iter->desc =3D (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY); iter->pos =3D 0; - sptep =3D iter->desc->sptes[iter->pos]; -out: - BUG_ON(!is_shadow_present_pte(*sptep)); - return sptep; + return iter->desc->sptes[iter->pos]; } =20 /* @@ -1228,14 +1265,11 @@ static u64 *rmap_get_first(struct kvm_rmap_head *rm= ap_head, */ static u64 *rmap_get_next(struct rmap_iterator *iter) { - u64 *sptep; - if (iter->desc) { if (iter->pos < PTE_LIST_EXT - 1) { ++iter->pos; - sptep =3D iter->desc->sptes[iter->pos]; - if (sptep) - goto out; + if (iter->desc->sptes[iter->pos]) + return iter->desc->sptes[iter->pos]; } =20 iter->desc =3D iter->desc->more; @@ -1243,20 +1277,26 @@ static u64 *rmap_get_next(struct rmap_iterator *ite= r) if (iter->desc) { iter->pos =3D 0; /* desc->sptes[0] cannot be NULL */ - sptep =3D iter->desc->sptes[iter->pos]; - goto out; + return iter->desc->sptes[iter->pos]; } } =20 return NULL; -out: - BUG_ON(!is_shadow_present_pte(*sptep)); - return sptep; } =20 -#define for_each_rmap_spte(_rmap_head_, _iter_, _spte_) \ - for (_spte_ =3D rmap_get_first(_rmap_head_, _iter_); \ - _spte_; _spte_ =3D rmap_get_next(_iter_)) +#define __for_each_rmap_spte(_rmap_head_, _iter_, _sptep_) \ + for (_sptep_ =3D rmap_get_first(_rmap_head_, _iter_); \ + _sptep_; _sptep_ =3D rmap_get_next(_iter_)) + +#define for_each_rmap_spte(_rmap_head_, _iter_, _sptep_) \ + __for_each_rmap_spte(_rmap_head_, _iter_, _sptep_) \ + if (!is_shadow_present_pte(*(_sptep_))) \ + BUG(); \ + else + +#define for_each_rmap_spte_lockless(_rmap_head_, _iter_, _sptep_, _spte_) \ + __for_each_rmap_spte(_rmap_head_, _iter_, _sptep_) \ + if (is_shadow_present_pte(_spte_ =3D mmu_spte_get_lockless(sptep))) =20 static void drop_spte(struct kvm *kvm, u64 *sptep) { --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 65D891684B0 for ; Fri, 9 Aug 2024 19:44:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232662; cv=none; b=JG7n7EIiEUI1459qyV8g/BEcG7bvLS568mpdhXA++ISpSMnKTJDk61Eeh/v15a2tDXDXuf86LFz3r0PJyea2iw9LDS3NwIC7NRIu7KgroOIZFOxHbcBF6aKFUivLRVzG4x3goYSZr6FUr9mr4FZJqLZHWlDIl78nHWkWePFyyW0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232662; c=relaxed/simple; bh=Asf1nEQnWkrE4726hXYeTADyaj7scYaajKYH4kYYdl0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=sAEzAcAvH2GPhDhNM1fkRoDFfaE9Vhnq7ogtXcZiacVQq6L3srpGB2aAs1cgvoeYLgNOEPQD/NhlHvS21W5hOL1OkovfpGCY2g9WwNPpDwDC8O+51oWYRCY/+m39+01nz8EB/FB2rq8ceLKMJbUKNVTd+ZwFVfnbJ3Sz8uZrEvM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=lHwvgnkw; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="lHwvgnkw" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-70d1c28cd89so2270837b3a.1 for ; Fri, 09 Aug 2024 12:44:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232661; x=1723837461; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=mm5+yuh4XjI5vBcow3bFf5ciAXmemsZ5SboXxBoCy4Q=; b=lHwvgnkwEOcCOyfLX4TbZy3ZzhtRRG5E2K+HBQ2UV26ory3BHqVB+UC48r5lTei6IH CtssaU9bCLPu/GGQYC1hcUNdwHyC59GMiBcWoQXPJoCFdVMLgGt5IFj8HKFs6f4zaz4h 9irWYwagwJ7GVsPLjzbPn6R8h8cxkJoQMoE+3q/NIgnVLAvjio4uGxArss8qWTx1cQ1f IGsVCls9f93fLoBzQ5NReP5b51ZiFuMT44l5pFMW9dImqDhm3o+x4tehBuSU6rdOZn/u tlG8G3s5ueQrc9dZD2uqvgNnl02y7Nfomt0div4WempbIeDM+5z6fg0VgneiXW1g0lVW A9+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232661; x=1723837461; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=mm5+yuh4XjI5vBcow3bFf5ciAXmemsZ5SboXxBoCy4Q=; b=mWc9zP0U4yvE/aieUtaSYSxnYHv0kMtkUV5fO5rmMhTCidWiDFWP0Plb/6H9MkC7rw uP/Xkb7P/zL4gLr6PKsRXCaHAfpPbVrTW2Z1KQbY9BzBSNivNAA9Usar9VerlJNcNvEe eDTTzvMsHYFubGLmViQ2VBxNxsxPnsfdWko5EOrUc02vT5w0CzqK15ZgZTC4MlUxiTjX heDwicik2NPyBTUjb7q3O/b0NMgQgs457jy/8a/dTKPo601ZSmzt/rPpcvax2dAnWVHb l/vEyDfHn+6aBTKxQz2vE/Kq8ML+dHXXKsRdqKVZbSPSFCHueHm0y9nS8ln7BFTgHe81 d7fA== X-Forwarded-Encrypted: i=1; AJvYcCUINW4PeWZHQRcv/aotcQMNAUXzpbKSPtw9PUEfEyk1x7pxc4IBrF5U/8k2K70acAo692Vz8Z2pK/InTuEyatyuTaTAD21CHuBSNaCi X-Gm-Message-State: AOJu0YzEzuiMw/kvlFdJuUXrUCEgxk33Jh/d/eG0lvFvtGR0qaChbhDd Lz5zNz5Hw5eis0Yl/7cpbFrixcKtWE6fAszW4ic1cQ4Io6Bvrq4BFZdtRBYuiFE0Oki3ciBhDOG VQA== X-Google-Smtp-Source: AGHT+IF/O6pTeFfoAlWjuVv6pzCpox1wsfwaTSxqhna+jt0NCRVn34Qv+aJ7MSoWohdt3Pb9tXUIOw8xzCs= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6a00:91e4:b0:710:4d6c:a504 with SMTP id d2e1a72fcca58-710dcb27574mr83474b3a.4.1723232660388; Fri, 09 Aug 2024 12:44:20 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:33 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-22-seanjc@google.com> Subject: [PATCH 21/22] KVM: x86/mmu: Support rmap walks without holding mmu_lock when aging gfns From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" DO NOT MERGE, yet... Cc: James Houghton Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 63 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 59 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 48e8608c2738..9df6b465de06 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -995,13 +995,11 @@ static void kvm_rmap_unlock(struct kvm_rmap_head *rma= p_head, * locking is the same, but the caller is disallowed from modifying the rm= ap, * and so the unlock flow is a nop if the rmap is/was empty. */ -__maybe_unused static unsigned long kvm_rmap_lock_readonly(struct kvm_rmap_head *rmap_hea= d) { return __kvm_rmap_lock(rmap_head); } =20 -__maybe_unused static void kvm_rmap_unlock_readonly(struct kvm_rmap_head *rmap_head, unsigned long old_val) { @@ -1743,8 +1741,53 @@ static void rmap_add(struct kvm_vcpu *vcpu, const st= ruct kvm_memory_slot *slot, __rmap_add(vcpu->kvm, cache, slot, spte, gfn, access); } =20 -static bool kvm_rmap_age_gfn_range(struct kvm *kvm, - struct kvm_gfn_range *range, bool test_only) +static bool kvm_rmap_age_gfn_range_lockless(struct kvm *kvm, + struct kvm_gfn_range *range, + bool test_only) +{ + struct kvm_rmap_head *rmap_head; + struct rmap_iterator iter; + unsigned long rmap_val; + bool young =3D false; + u64 *sptep; + gfn_t gfn; + int level; + u64 spte; + + for (level =3D PG_LEVEL_4K; level <=3D KVM_MAX_HUGEPAGE_LEVEL; level++) { + for (gfn =3D range->start; gfn < range->end; + gfn +=3D KVM_PAGES_PER_HPAGE(level)) { + rmap_head =3D gfn_to_rmap(gfn, level, range->slot); + rmap_val =3D kvm_rmap_lock_readonly(rmap_head); + + for_each_rmap_spte_lockless(rmap_head, &iter, sptep, spte) { + if (!is_accessed_spte(spte)) + continue; + + if (test_only) { + kvm_rmap_unlock_readonly(rmap_head, rmap_val); + return true; + } + + /* + * Marking SPTEs for access tracking outside of + * mmu_lock is unsupported. Report the page as + * young, but otherwise leave it as-is. + */ + if (spte_ad_enabled(spte)) + clear_bit((ffs(shadow_accessed_mask) - 1), + (unsigned long *)sptep); + young =3D true; + } + + kvm_rmap_unlock_readonly(rmap_head, rmap_val); + } + } + return young; +} + +static bool __kvm_rmap_age_gfn_range(struct kvm *kvm, + struct kvm_gfn_range *range, bool test_only) { struct slot_rmap_walk_iterator iterator; struct rmap_iterator iter; @@ -1783,6 +1826,18 @@ static bool kvm_rmap_age_gfn_range(struct kvm *kvm, return young; } =20 + +static bool kvm_rmap_age_gfn_range(struct kvm *kvm, + struct kvm_gfn_range *range, bool test_only) +{ + /* FIXME: This also needs to be guarded with something like range->fast_o= nly. */ + if (kvm_ad_enabled()) + return kvm_rmap_age_gfn_range_lockless(kvm, range, test_only); + + lockdep_assert_held_write(&kvm->mmu_lock); + return __kvm_rmap_age_gfn_range(kvm, range, test_only); +} + bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { bool young =3D false; --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 10:16:38 2026 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0650C19CD0C for ; Fri, 9 Aug 2024 19:44:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232664; cv=none; b=JJFg25mZQZ9JUEe/oL80IQlMP29j0mAHdtRc/vwQgOieruEQw0W4R4zv0xXbpzqlFrAayIb5xBbC23Xv1J8i3A5EsATjXxQS6FKfqOeDieCEPYf/ng1x9QheOFIOIEOVWQ9sV71+1GIK0j2MdcB9CSPy3UeTQAvGJ5taXnxRJm4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232664; c=relaxed/simple; bh=DAzHpkMD+wsg+IgGyvY1zUDoM4DEg8WSG8aKxUnrcM0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=URsr5V0j0WXzC2OMCATC7b+nLzRwxIqfFH13ISmqpg1zwuVEJVLpHVwKFskKuRy6R4joBahicdYAHLZaxLzpCZeoMqwMDDW6Kc832VBrhs4zFfHfw0L9LQUA3d2kegIm0hvXJcI9j+qMpybbR2C3Z1WmhfUixwViWzSLG4QD+O0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=SNmfM4vo; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="SNmfM4vo" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-7b696999c65so2036182a12.3 for ; Fri, 09 Aug 2024 12:44:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232662; x=1723837462; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=8pI2Dk2CV1cTItnOXgPwVCgdPRwBcss9J2eOD4s9n2E=; b=SNmfM4voWXCyX9uc9PSsvFRT3o6CR1p7akCFejZ3L16cl0dRqL4ZZAHEpyQIHLz7OB oHxKLn61By1WM6WA4UzQWCZw8vPjAot5+XnYQZElx1cMYOKbBcJi3ZonX5oNDPgjk9MD rFUP1825C4QGb6A9X2oB3bjzorR/sOYtuyj7OtfCR4j9I9/4qY2WgnQGAKA83xopTliv Anm1wsZ0D9ErP0cZjnLFZwZdLaHxLS//YNzB3SeC2WAG3QuzLGywocwgPRQIDz0oGDEy t1dL+qCgLbPE7QdDkPyKxKOVm03BvAip+HArlxweMRIls9mJ5mN5lGfPEJgh1xNU/ubb jMLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232662; x=1723837462; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=8pI2Dk2CV1cTItnOXgPwVCgdPRwBcss9J2eOD4s9n2E=; b=rpzcDYK3gBEmm0nyAeTziZPQgxXSFH5Zb2XRkPfcgSoBjBYo+931qTUF27IUK8YmZn pjISN+3nZh9NWgr7YQlGkay2fjqS/asyFmv1aQBZSJfND84JU7l1BZNThWwRT/pzbg1d snpEALnp1Pv1SE1sNIC5HXMrCh7c9hOTABLYjoCtMaIVoadRYM9S0bDGcLftrZbL+24B Yt6i5whYMJ6tBt8h6N4GrKaVB8q3CzM5varJoVtW6ximGgnoRCxbSQUJRYtuQ95NrqCq EDJXCSOB5O0Vu+MLo5Q6jF01nKTPX1PYxkccxUrsrae7MN3kOeqdx0LHcr3ZH438RuTE 4zdw== X-Forwarded-Encrypted: i=1; AJvYcCWt7qvMwJishd1iNqbU6KW4gWYMBJ7Gz0KOrIuWqfO3zDQtOUXKehk2/ev5icbfxfOi0qdsC1KpHVoSmsoH+rB4MoZD/yaeODGlQpeB X-Gm-Message-State: AOJu0YwZrBw4faF4EZ4ssosT5+6CXgtagb3yf9QxB9xRe5bXTRTH5uu9 5eRaimXbgyXtMIkqlq+JWFlxgYSnHEodYFEcFKf94YuqZ42C3B05FmHlzlRTz+fs1OruaIzLk91 cCw== X-Google-Smtp-Source: AGHT+IFfyFmWIFbiqu94F+TDoLpelUA2Pyb+SCf4Vx96bmcRj1adsvoVMyySdsBmrnDzs3PRgMYuDO6IScE= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:90a:9c0c:b0:2c9:967d:94a4 with SMTP id 98e67ed59e1d1-2d1e80e6a81mr12876a91.5.1723232662118; Fri, 09 Aug 2024 12:44:22 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:34 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-23-seanjc@google.com> Subject: [PATCH 22/22] ***HACK*** KVM: x86: Don't take mmu_lock when aging gfns From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" DO NOT MERGE, this is a horrific hack, breaks TDP MMU, etc. Not-signed-off-by: Sean Christopherson --- arch/x86/kvm/svm/svm.c | 2 ++ arch/x86/kvm/vmx/vmx.c | 2 ++ virt/kvm/kvm_main.c | 7 ++++--- 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index c115d26844f7..e5c5d0f9a69d 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -5300,6 +5300,8 @@ static __init int svm_hardware_setup(void) if (!boot_cpu_has(X86_FEATURE_NPT)) npt_enabled =3D false; =20 + npt_enabled =3D false; + /* Force VM NPT level equal to the host's paging level */ kvm_configure_mmu(npt_enabled, get_npt_level(), get_npt_level(), PG_LEVEL_1G); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index f18c2d8c7476..28f3493d6391 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -8419,6 +8419,8 @@ __init int vmx_hardware_setup(void) !cpu_has_vmx_invept_global()) enable_ept =3D 0; =20 + enable_ept =3D 0; + /* NX support is required for shadow paging. */ if (!enable_ept && !boot_cpu_has(X86_FEATURE_NX)) { pr_err_ratelimited("NX (Execute Disable) not supported\n"); diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index d0788d0a72cc..1b9b5dea2ac8 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -642,10 +642,11 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_= range(struct kvm *kvm, gfn_range.slot =3D slot; =20 if (!r.found_memslot) { - r.found_memslot =3D true; - KVM_MMU_LOCK(kvm); - if (!IS_KVM_NULL_FN(range->on_lock)) + if (!IS_KVM_NULL_FN(range->on_lock)) { + r.found_memslot =3D true; + KVM_MMU_LOCK(kvm); range->on_lock(kvm); + } =20 if (IS_KVM_NULL_FN(range->handler)) goto mmu_unlock; --=20 2.46.0.76.ge559c4bf1a-goog