From nobody Fri Dec 19 14:31:20 2025 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B08AB18B484 for ; Sat, 15 Mar 2025 03:06:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742007996; cv=none; b=cCWoGs2LOXPNjbkEB7/R5PZlaAoIO7cp8uQKnYwxnsoAoFTJbfS2T4wvAJOOxf+zirJhpbANgwW09XESzvnN8tZ9WOeqt8W3dwgBPBIjmYKW0x6l++zOwTrhk1GFZ0eardDNzncWN0cPfKHZzvGHdwhhtikzh463cxhSeLAvYws= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742007996; c=relaxed/simple; bh=ORxMRxanum7dJ4QhbbOoW6MS6dV40SegF61qcn5cQK8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=YWyUurtlCfZT2BoWRVolhoodbNSibC5lEIn366zqFPDwYzMntxo4aM9ldV8pLOk1B3MEyRtMZ/7Zpf4/ixpJCR4LOZrpT28+7eAOonxGVkwSFoDt/AzvEHuFNZQbnqiVlWlPLdxinehNxyg85ewav1fhjVq1WXaTfxgi8xZXdIA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=UAMT/Ldj; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="UAMT/Ldj" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2ff854a2541so494224a91.0 for ; Fri, 14 Mar 2025 20:06:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1742007994; x=1742612794; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=A+D7/CZc6x+lcAi6xbF5IFcWIU9h8go1MiK4EHRJXP4=; b=UAMT/LdjeX0edyYremYgTvkrkszCPGCpjVyAYX62V4ktJUi/Z/M1s/Qewwj03Gvphb 2H4ZickLHWjxa5xLDisv8t4jq7XBY5DL0zFG28iyXQduvGGaiYRSUJz8PCPav5XOQZqW YEBYZMlqBkAzLI4SQPS4b7QYouAKWh+8laX7hFkwSCjyp+f2XgR8dTuHLeGztbx9r7zo OMfHDtzBE+fj4iYBAdd/PvUeILJy+kD/YFbt4YNzL5LuHQMJjA+TcnOzFIGuX5iL8Un5 6HbE1FnABlkL9nZ8+uURlYasmbds3ssIkqV+0UF+xpdNZFZev7Bi17V9BkDo+kK54cw9 e1iA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742007994; x=1742612794; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=A+D7/CZc6x+lcAi6xbF5IFcWIU9h8go1MiK4EHRJXP4=; b=iquvMETaao9lu7QBKy7EzbM9Z3tyb2gulNTO1iT/C7bXHKGEQEPlOngx65hNLKMrWf 0fx1KAm1Ak+8CURX4JYK72lJVhORdjXmgMoEIEWXoXLSsCmbMqZQ5BSrjwpciz64m8zs LS4tN4thCxImn574r768L2Vp2ucmh1ZulwbgBDVFq09K5wdGLf7fFX7HTi2XAxnqVwY7 C8kQaug+TVHGW0bzi/qwxAqeABdkIln46td0Cc4RncPzNNleGzu1ARwzZGuDabmtb+13 cQShjJkfidloefUmDlfVVLiMNZ7wqs5OaUsd8ZaMoPkb8PPb+tnvIP8zt7Ofk3hV3klU mQLw== X-Gm-Message-State: AOJu0YxC3t2/nXbvEIF1KRxtCby+J4AiYmKR7Qiv4bYEJyGFYVQ+V2AF nUcWAoG7CzLZcXHs1WGi54W5h5IAsbd7lXscQ2LVt+j34Z4eZ6vrIVJejq90UQUDpuVa++kutL4 NsA== X-Google-Smtp-Source: AGHT+IH3X+L+I/21qttxgDTkKxHuhAtYSP/nASSYaa/gcSlKKoQ0V6Lk918oh7tBqn4FZDeDooNPZD4EXAU= X-Received: from pjl6.prod.google.com ([2002:a17:90b:2f86:b0:2ee:3128:390f]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90a:fc4c:b0:2f4:4500:bb4d with SMTP id 98e67ed59e1d1-30151cc8c11mr5807708a91.20.1742007993991; Fri, 14 Mar 2025 20:06:33 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 14 Mar 2025 20:06:22 -0700 In-Reply-To: <20250315030630.2371712-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250315030630.2371712-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.rc1.451.g8f38331e32-goog Message-ID: <20250315030630.2371712-2-seanjc@google.com> Subject: [PATCH 1/8] x86/irq: Ensure initial PIR loads are performed exactly once From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jacob Pan , Jim Mattson Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Ensure the PIR is read exactly once at the start of handle_pending_pir(), to guarantee that checking for an outstanding posted interrupt in a given chuck doesn't reload the chunk from the "real" PIR. Functionally, a reload is benign, but it would defeat the purpose of pre-loading into a copy. Fixes: 1b03d82ba15e ("x86/irq: Install posted MSI notification handler") Cc: Jacob Pan Signed-off-by: Sean Christopherson Reviewed-by: Thomas Gleixner --- arch/x86/kernel/irq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index 385e3a5fc304..9e5263887ff6 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -412,7 +412,7 @@ static __always_inline bool handle_pending_pir(u64 *pir= , struct pt_regs *regs) bool handled =3D false; =20 for (i =3D 0; i < 4; i++) - pir_copy[i] =3D pir[i]; + pir_copy[i] =3D READ_ONCE(pir[i]); =20 for (i =3D 0; i < 4; i++) { if (!pir_copy[i]) --=20 2.49.0.rc1.451.g8f38331e32-goog From nobody Fri Dec 19 14:31:20 2025 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6B0F518E050 for ; Sat, 15 Mar 2025 03:06:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742007997; cv=none; b=F0ThQ7fibt4ldDBay5ddKTQy9Lpk0Xn2o+LXB+6DIS4cIYAnu9pYTUzYoRXzTfFBZ8sbYUX7OlLQOD1qXYy1n0cYBBDZI2tF53jpLzJ9vCt44LzKfnQ4GSrbcl2sQFJvkuUsl5NIgC/Mfb5qF2qJgSqJywWBuO6o91Vc4MRG9JE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742007997; c=relaxed/simple; bh=+WK1bltbWDTrbagBZEBAiy7h0kT2VEa0FdxHN4yhE9Y=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=NegGDO3hofPz7FvBsBQzsCA9als5aEzHB30EeLBTM1LROcBDmKnjPPtSaf/MG5lLI0kHm3adAI/e4SgeQ74lCjNcgnzHmFkCmctDZnzydpiTkNSILO/vV2u4ySA6HBptwvKSDGVUhA6vZcMjf4dNv7r4M9VGbN8h8/0x7MxQIz4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=rkaKDIb5; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="rkaKDIb5" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2ff78bd3026so448776a91.1 for ; Fri, 14 Mar 2025 20:06:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1742007996; x=1742612796; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=vHe6yayu6Pa5fekV1jp+JV4SQqE+N7492v9RWvirBCU=; b=rkaKDIb5IkP2/N6o1yzoZwGPAerPT2LrFdFzI6HhSTjTCFDQ7TLxX1y9HjPlTNhSUk 2UZU4RIiQiugLzYpehug2+xrSxYHZb4JIVLgCd/hmE2SbTZC/lIEKTPsdx7PfRr/iis6 YCtiSH7EKRXmstg1DMbiy9/i2ynO5xOjkxqqOLm560glBJXGSp301U1kOuv6EwX+J76n SVECsJc66XIj5MXLZgeGvFfUaRlA0WfMzmTKXiPFxgcJXm+2qwCeId/ZJZClkMCEja7Q GK5PYr9bxWsxMgWInBILKkXSgIiFcHnSMvc9kXIWfjQlb4XQkJcNMev4SS6mAe6qzlyT /2jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742007996; x=1742612796; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=vHe6yayu6Pa5fekV1jp+JV4SQqE+N7492v9RWvirBCU=; b=JextnhLgyfpX01+lOIYAitoyRntiUZrYVpxB4Hhb1Jer0wzCByM703OuK8f39xu4Jk eXNDOaH/wCE0Qp8PaBX3OwUHIBSleQDSw6EGgYFvDgF+CRgTR8RTq7ATNGwPOB8DRAzJ OdfS5/Tis2Pnl9uTBL+B3+ejquhft8e4PA2nsTeO2/ze7x2rC+9z/EJMR53kZx4zNO9U 2biFYYYinTumfKhayozBzh3QKh+fV44MdZ8fWVa6KPjMZr9uNa5zY02gWxXdvJyCAwBn aQLlcF11Hc207Kh1nQZKwiFj/krHlQEiT+v16plFVD3F4va9f+SPVe7/INS/IgvGU5Ur TNkg== X-Gm-Message-State: AOJu0YzxWS6WazKkxLUjSPDVh9mS1FaKAijDQpK2gvf8zAahn+l2lfww q2Kw0KamHBK0DTm5gtyZXqOgFbfEeNG6qSmnQlM+R8qZdvJSBddeGDsYKH+J70tPQtmqbY5iaoK nlQ== X-Google-Smtp-Source: AGHT+IHm3ikpa45S+MzJ9ChgrVNIn//2DPUl62WT86s6estUNWscdmElpdksr3RkSuK4oMZ2hPanzmR+FmA= X-Received: from pjbsc17.prod.google.com ([2002:a17:90b:5111:b0:2ea:aa56:49c]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:5387:b0:2ff:4a8d:74f8 with SMTP id 98e67ed59e1d1-30151c9a341mr5024590a91.6.1742007995777; Fri, 14 Mar 2025 20:06:35 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 14 Mar 2025 20:06:23 -0700 In-Reply-To: <20250315030630.2371712-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250315030630.2371712-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.rc1.451.g8f38331e32-goog Message-ID: <20250315030630.2371712-3-seanjc@google.com> Subject: [PATCH 2/8] x86/irq: Track if IRQ was found in PIR during initial loop (to load PIR vals) From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jacob Pan , Jim Mattson Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Track whether or not at least one IRQ was found in PIR during the initial loop to load PIR chunks from memory. Doing so generates slightly better code (arguably), especially for the case where there are no pending IRQs. Note, while PIR can be modified between the initial load and the XCHG, it can only _gain_ new IRQs, i.e. there is no danger of a false positive due to the final version of pir_copy[] being empty. Opportunistically rename the boolean in anticipation of moving the PIR accesses to a common helper that can be shared by posted MSIs and KVM. Old: <+74>: test %rdx,%rdx <+77>: je 0xffffffff812bbeb0 <+88>: mov $0x1,%dl> <+90>: test %rsi,%rsi <+93>: je 0xffffffff812bbe8c <+106>: mov $0x1,%dl <+108>: test %rcx,%rcx <+111>: je 0xffffffff812bbe9e <+124>: mov $0x1,%dl <+126>: test %rax,%rax <+129>: je 0xffffffff812bbeb9 <+142>: jmp 0xffffffff812bbec1 <+144>: xor %edx,%edx <+146>: test %rsi,%rsi <+149>: jne 0xffffffff812bbe7f <+151>: jmp 0xffffffff812bbe8c <+153>: test %dl,%dl <+155>: je 0xffffffff812bbf8e New: <+74>: mov %rax,%r8 <+77>: or %rcx,%r8 <+80>: or %rdx,%r8 <+83>: or %rsi,%r8 <+86>: setne %bl <+89>: je 0xffffffff812bbf88 <+95>: test %rsi,%rsi <+98>: je 0xffffffff812bbe8d <+109>: test %rdx,%rdx <+112>: je 0xffffffff812bbe9d <+125>: test %rcx,%rcx <+128>: je 0xffffffff812bbead <+141>: test %rax,%rax <+144>: je 0xffffffff812bbebd Signed-off-by: Sean Christopherson --- arch/x86/kernel/irq.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index 9e5263887ff6..3f95b00ccd7f 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -409,25 +409,28 @@ static __always_inline bool handle_pending_pir(u64 *p= ir, struct pt_regs *regs) { int i, vec =3D FIRST_EXTERNAL_VECTOR; unsigned long pir_copy[4]; - bool handled =3D false; + bool found_irq =3D false; =20 - for (i =3D 0; i < 4; i++) + for (i =3D 0; i < 4; i++) { pir_copy[i] =3D READ_ONCE(pir[i]); + if (pir_copy[i]) + found_irq =3D true; + } + + if (!found_irq) + return false; =20 for (i =3D 0; i < 4; i++) { if (!pir_copy[i]) continue; =20 pir_copy[i] =3D arch_xchg(&pir[i], 0); - handled =3D true; } =20 - if (handled) { - for_each_set_bit_from(vec, pir_copy, FIRST_SYSTEM_VECTOR) - call_irq_handler(vec, regs); - } + for_each_set_bit_from(vec, pir_copy, FIRST_SYSTEM_VECTOR) + call_irq_handler(vec, regs); =20 - return handled; + return true; } =20 /* --=20 2.49.0.rc1.451.g8f38331e32-goog From nobody Fri Dec 19 14:31:20 2025 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D49601917D4 for ; Sat, 15 Mar 2025 03:06:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742007999; cv=none; b=M4pjlyu3950BSvaOjUTTwZzvA5V+u3IYkeDXARAJoumcGVe4ZqxEexl1ac7SjynCdV6dNssKIkwWop9lZqEb3lcPWk0xRIKww4RnLbEm+kLwEoVc2wzXEatRHc8Yb/DnAGAKTOag54DbGEYc4oWTbc72XxdNlPUZ4mpoKRsAUJY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742007999; c=relaxed/simple; bh=3oNkXBgr8EIWda3cEEvkfRlryoj/MBVyRM1tKseboAk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=XPOS3KIJ/IMGFb9j/FQ45U0pf5I1+H075szp6ZcOAjIAP/EroXxHdo6E8a9cRQmTqt9gBOYA7oVr1VmS+4cs7wmpuLsZ48iULuHQhXi1vgGB6VMWEogarI5a1/SPp9Oy7fxxHUBweB+5HgQSQ3VYxlU269y7JqKp6nsGJK4/zIs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=DNe9Xnby; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="DNe9Xnby" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-3011cfa31f6so482463a91.3 for ; Fri, 14 Mar 2025 20:06:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1742007997; x=1742612797; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=+rGaV7EYztjwjHpumFCPp4cbhOrFaDcpQIEQlQEwrkg=; b=DNe9XnbyHkEO2O9917aoGeXFKUsA0CwhCFQts+2Z6d16hTD0cDo2z+tcdm3vi6pGRA mSGxOg1adjI3cwv98CcSuEaL5G0xzWwlGXooeAmp/Kie0SsGcV2w2Iys7hSBy4sErbd1 kI6ZiY2Auut6Ljo+4KkQO/n7p33prOYE5h+XYJ+PeN1bgpZn4fCCAzCF5ELhN+n9qICp lTT6vYBM/dXcCUfjbY8C223SFc3P9ddVos8YA+9jYVDjbaIKLpXDUo3NcrIvF/s9Bjx4 x6dU9nfkSvIzLt/dsUvWRJP2mFBMeuEKxPZMmW9k/OhNTYXaiU/bwxCdGAOL4dnOSqDq 5Yhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742007997; x=1742612797; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+rGaV7EYztjwjHpumFCPp4cbhOrFaDcpQIEQlQEwrkg=; b=D2X/jlDmy3bsskvoGkU0dqLpRat5WEiI8ZxEpKZtx4dRJokTJtDFQ0mBT/dcnPr316 6pWXAYny0QMFvSXEeVUpkdi/8cbMenTFCjbLMST/bMBOj8g5Deafx69Athy/ASlC6nQn OxraUzaeDrPodPx6qo7fdar2kxRJpxaA9cSYqOJ/E7KSPomk7AP+UmRXZzKjj7NIoBbz IiVY4b/p5SEbFX7g6xkTdUEQ+t3Zd4lnk0uoAWJkaLsTFFtnRg0APgki1WHWEqnaI9ab Z3Fz0qcOJACkfnHopmft+aMXUdVSCbT1ccdtob3eSVD959BuQINVy6Dl8gwZ7psjIBSN JKoA== X-Gm-Message-State: AOJu0YwYoqpLGo5YZT8vqJCBQOoApQ8New1ZS9dHBFjWx3nu2qpu/cd/ WSmyyEe1oyXKnuAA0fKNY9ddRtSTSVU7L6jqlOk8xVsne/WTFgRTpovTot9NYj3nsHDL3lMgjWZ /Gw== X-Google-Smtp-Source: AGHT+IHToyeqUaA2FjyTnWkl7XfPw0+f99lKhBPLNQjpZdYLGzY/BKFwEs/MhlWIFqJRpb5ryLcdl7TDFNM= X-Received: from pjbrr7.prod.google.com ([2002:a17:90b:2b47:b0:2ee:4b69:50e1]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:4ec3:b0:2ee:d7d3:3019 with SMTP id 98e67ed59e1d1-30151c7a384mr7020001a91.12.1742007997207; Fri, 14 Mar 2025 20:06:37 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 14 Mar 2025 20:06:24 -0700 In-Reply-To: <20250315030630.2371712-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250315030630.2371712-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.rc1.451.g8f38331e32-goog Message-ID: <20250315030630.2371712-4-seanjc@google.com> Subject: [PATCH 3/8] KVM: VMX: Ensure vIRR isn't reloaded at odd times when sync'ing PIR From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jacob Pan , Jim Mattson Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Read each vIRR exactly once when shuffling IRQs from the PIR to the vAPIC to ensure getting the highest priority IRQ from the chunk doesn't reload from the vIRR. In practice, a reload is functionally benign as vcpu->mutex is held and so IRQs can be consumed, i.e. new IRQs can appear, but existing IRQs can't disappear. Signed-off-by: Sean Christopherson --- arch/x86/kvm/lapic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index 9dbc0f5d9865..cb4aeab914eb 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -667,7 +667,7 @@ bool __kvm_apic_update_irr(u32 *pir, void *regs, int *m= ax_irr) for (i =3D vec =3D 0; i <=3D 7; i++, vec +=3D 32) { u32 *p_irr =3D (u32 *)(regs + APIC_IRR + i * 0x10); =20 - irr_val =3D *p_irr; + irr_val =3D READ_ONCE(*p_irr); pir_val =3D READ_ONCE(pir[i]); =20 if (pir_val) { --=20 2.49.0.rc1.451.g8f38331e32-goog From nobody Fri Dec 19 14:31:20 2025 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 88909194AC7 for ; Sat, 15 Mar 2025 03:06:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742008001; cv=none; b=GHX8vq+uqkASCCspy3tVybdA1Pa8r4i2ai0lEefJ8sb2+J/wkljMyw1sGD8dFAwGRBNVyR0y4FF0irayaZjZnu/eEDLMBD3fvuDd2XGkFU0r6dlc4aO7X1w9atvrrw4+pm3qpnbM6kq8/gGQDQkPlhXL008wuiwEc7Q/0BQ0Wj8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742008001; c=relaxed/simple; bh=Uta9a/3o/kiGmJmq8wNCcF+U80EDf0HSkBBuI1bpzK4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=QjHXSckPYN+wgpUV4if0dGZFKLA6GeL5GxtxLlNozfAL8t2vDWG+eq7GG7DMQTIjBksXgi2CPFipkQbpAmzi+629PWJWaa2BLqyNjUy4pXb4yTAJlF3TstjXz+z2UGkTSBUc3gN9xl7Kk7anuzg2gS7L/vUq+xIKr/i74joVbOc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=GyccclzN; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="GyccclzN" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-3011bee1751so522637a91.1 for ; Fri, 14 Mar 2025 20:06:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1742007999; x=1742612799; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=eXE4hRNdIiKSnjJQabn9+9Z7l4STEDkxZZthIWAaxQM=; b=GyccclzNZKflDwYfwsuNwTREjeT4y5XYdu85Irz+7wHGQgoBvF1CVb0oTBO79nelqj vY9ywEWq6u7ugedqR+nkgGzFK+nPh+UWKBa8lMiVzPg3ey7hVDscObqm8+Q81menAr+c Fxprb0qardBlT0onT1OHmHNBIxy5WpOEUCp02HLNeZ303Y25olnvxzJqdUkOKa0IX2O/ ARz0agITwnJNH9/Z07NuC8dU+Q8AcHB5KpQvOUNNt10qxexeVJo3s3w+HZHEGC8M5DZx arVTByS4z03GtR+nWUFn2aL2WOYxb3sGp9SHg2skwSiR+ClTQDCf3tusF+2PrSC5ZCg8 ns5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742007999; x=1742612799; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=eXE4hRNdIiKSnjJQabn9+9Z7l4STEDkxZZthIWAaxQM=; b=Pi/TuQ8/iyUdrtczZ+v6jDcLRC8jwH/bQWoUItz/Cw84yU6Rhijj1XgrvJTMyZX5pZ efOsTcakv9WJuP+EsgFZ3ZYstVm7hnzONceERQt1lr98TyNRGhMuamWXkjoF5TwvA1gy dnF4e5Sl5xo6mF8fxRsABsrLaV6aRmO3Fqi6r+o3YlYCD+KYNlXPOpxiRePrsUMBj/qi QOj+4RWXmgYzxILbuqxNaf+ClaLeric1JcVDxjjpnc1XKMwEiwq3CjXmMa+vcnZMnpYn eRnIIdfjbUNvefRKDT2CDiZxErDQ/Rrfmb3SUf5N6R6zC7F0zm3fpwa2BFZgVP7RJn3n Kf0Q== X-Gm-Message-State: AOJu0Yy4dwLAy6yoVRKXsg3oD4WXkIFVSThpjk9xYRRXuhgJxROdXOz6 /BzmDR7dGSGThHcwBVH1LsTlGmC6CDubsDi9XEt2rxtbQrukevdHnj0lRkJxh4X6ES4QTadcgPd Onw== X-Google-Smtp-Source: AGHT+IF1o7Gq8dxb5C/i4W3LaVHx9Xt9tEF6sWZDA8tMLg9rmMb0LEnExUDVXCga37ha7MMcuP+qjT1SPsA= X-Received: from pjbtc11.prod.google.com ([2002:a17:90b:540b:b0:2ff:6e58:8a03]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2690:b0:2ee:af31:a7bd with SMTP id 98e67ed59e1d1-30151cb59afmr5565985a91.5.1742007998857; Fri, 14 Mar 2025 20:06:38 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 14 Mar 2025 20:06:25 -0700 In-Reply-To: <20250315030630.2371712-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250315030630.2371712-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.rc1.451.g8f38331e32-goog Message-ID: <20250315030630.2371712-5-seanjc@google.com> Subject: [PATCH 4/8] x86/irq: KVM: Track PIR bitmap as an "unsigned long" array From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jacob Pan , Jim Mattson Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Track the PIR bitmap in posted interrupt descriptor structures as an array of unsigned longs instead of using unionized arrays for KVM (u32s) versus IRQ management (u64s). In practice, because the non-KVM usage is (sanely) restricted to 64-bit kernels, all existing usage of the u64 variant is already working with unsigned longs. Using "unsigned long" for the array will allow reworking KVM's processing of the bitmap to read/write in 64-bit chunks on 64-bit kernels, i.e. will allow optimizing KVM by reducing the number of atomic accesses to PIR. Opportunstically replace the open coded literals in the posted MSIs code with the appropriate macro. Deliberately don't use ARRAY_SIZE() in the for-loops, even though it would be cleaner from a certain perspective, in anticipation of decoupling the processing from the array declaration. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/include/asm/posted_intr.h | 14 +++++++------- arch/x86/kernel/irq.c | 12 ++++++------ arch/x86/kvm/lapic.c | 9 +++++---- arch/x86/kvm/lapic.h | 4 ++-- arch/x86/kvm/vmx/posted_intr.h | 2 +- 5 files changed, 21 insertions(+), 20 deletions(-) diff --git a/arch/x86/include/asm/posted_intr.h b/arch/x86/include/asm/post= ed_intr.h index de788b400fba..c3e6e4221a5b 100644 --- a/arch/x86/include/asm/posted_intr.h +++ b/arch/x86/include/asm/posted_intr.h @@ -8,12 +8,12 @@ =20 #define PID_TABLE_ENTRY_VALID 1 =20 +#define NR_PIR_VECTORS 256 +#define NR_PIR_WORDS (NR_PIR_VECTORS / BITS_PER_LONG) + /* Posted-Interrupt Descriptor */ struct pi_desc { - union { - u32 pir[8]; /* Posted interrupt requested */ - u64 pir64[4]; - }; + unsigned long pir[NR_PIR_WORDS]; /* Posted interrupt requested */ union { struct { u16 notifications; /* Suppress and outstanding bits */ @@ -43,12 +43,12 @@ static inline bool pi_test_and_clear_sn(struct pi_desc = *pi_desc) =20 static inline bool pi_test_and_set_pir(int vector, struct pi_desc *pi_desc) { - return test_and_set_bit(vector, (unsigned long *)pi_desc->pir); + return test_and_set_bit(vector, pi_desc->pir); } =20 static inline bool pi_is_pir_empty(struct pi_desc *pi_desc) { - return bitmap_empty((unsigned long *)pi_desc->pir, NR_VECTORS); + return bitmap_empty(pi_desc->pir, NR_VECTORS); } =20 static inline void pi_set_sn(struct pi_desc *pi_desc) @@ -105,7 +105,7 @@ static inline bool pi_pending_this_cpu(unsigned int vec= tor) if (WARN_ON_ONCE(vector > NR_VECTORS || vector < FIRST_EXTERNAL_VECTOR)) return false; =20 - return test_bit(vector, (unsigned long *)pid->pir); + return test_bit(vector, pid->pir); } =20 extern void intel_posted_msi_init(void); diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index 3f95b00ccd7f..704c104ff7a4 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -405,13 +405,13 @@ void intel_posted_msi_init(void) * instead of: * read, xchg, read, xchg, read, xchg, read, xchg */ -static __always_inline bool handle_pending_pir(u64 *pir, struct pt_regs *r= egs) +static __always_inline bool handle_pending_pir(unsigned long *pir, struct = pt_regs *regs) { int i, vec =3D FIRST_EXTERNAL_VECTOR; - unsigned long pir_copy[4]; + unsigned long pir_copy[NR_PIR_WORDS]; bool found_irq =3D false; =20 - for (i =3D 0; i < 4; i++) { + for (i =3D 0; i < NR_PIR_WORDS; i++) { pir_copy[i] =3D READ_ONCE(pir[i]); if (pir_copy[i]) found_irq =3D true; @@ -420,7 +420,7 @@ static __always_inline bool handle_pending_pir(u64 *pir= , struct pt_regs *regs) if (!found_irq) return false; =20 - for (i =3D 0; i < 4; i++) { + for (i =3D 0; i < NR_PIR_WORDS; i++) { if (!pir_copy[i]) continue; =20 @@ -460,7 +460,7 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_posted_msi_notification) * MAX_POSTED_MSI_COALESCING_LOOP - 1 loops are executed here. */ while (++i < MAX_POSTED_MSI_COALESCING_LOOP) { - if (!handle_pending_pir(pid->pir64, regs)) + if (!handle_pending_pir(pid->pir, regs)) break; } =20 @@ -475,7 +475,7 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_posted_msi_notification) * process PIR bits one last time such that handling the new interrupts * are not delayed until the next IRQ. */ - handle_pending_pir(pid->pir64, regs); + handle_pending_pir(pid->pir, regs); =20 apic_eoi(); irq_exit(); diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index cb4aeab914eb..893e7d06e0e6 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -655,8 +655,9 @@ static u8 count_vectors(void *bitmap) return count; } =20 -bool __kvm_apic_update_irr(u32 *pir, void *regs, int *max_irr) +bool __kvm_apic_update_irr(unsigned long *pir, void *regs, int *max_irr) { + u32 *__pir =3D (void *)pir; u32 i, vec; u32 pir_val, irr_val, prev_irr_val; int max_updated_irr; @@ -668,10 +669,10 @@ bool __kvm_apic_update_irr(u32 *pir, void *regs, int = *max_irr) u32 *p_irr =3D (u32 *)(regs + APIC_IRR + i * 0x10); =20 irr_val =3D READ_ONCE(*p_irr); - pir_val =3D READ_ONCE(pir[i]); + pir_val =3D READ_ONCE(__pir[i]); =20 if (pir_val) { - pir_val =3D xchg(&pir[i], 0); + pir_val =3D xchg(&__pir[i], 0); =20 prev_irr_val =3D irr_val; do { @@ -691,7 +692,7 @@ bool __kvm_apic_update_irr(u32 *pir, void *regs, int *m= ax_irr) } EXPORT_SYMBOL_GPL(__kvm_apic_update_irr); =20 -bool kvm_apic_update_irr(struct kvm_vcpu *vcpu, u32 *pir, int *max_irr) +bool kvm_apic_update_irr(struct kvm_vcpu *vcpu, unsigned long *pir, int *m= ax_irr) { struct kvm_lapic *apic =3D vcpu->arch.apic; bool irr_updated =3D __kvm_apic_update_irr(pir, apic->regs, max_irr); diff --git a/arch/x86/kvm/lapic.h b/arch/x86/kvm/lapic.h index 1a8553ebdb42..0d41780852e4 100644 --- a/arch/x86/kvm/lapic.h +++ b/arch/x86/kvm/lapic.h @@ -101,8 +101,8 @@ bool kvm_apic_match_dest(struct kvm_vcpu *vcpu, struct = kvm_lapic *source, int shorthand, unsigned int dest, int dest_mode); int kvm_apic_compare_prio(struct kvm_vcpu *vcpu1, struct kvm_vcpu *vcpu2); void kvm_apic_clear_irr(struct kvm_vcpu *vcpu, int vec); -bool __kvm_apic_update_irr(u32 *pir, void *regs, int *max_irr); -bool kvm_apic_update_irr(struct kvm_vcpu *vcpu, u32 *pir, int *max_irr); +bool __kvm_apic_update_irr(unsigned long *pir, void *regs, int *max_irr); +bool kvm_apic_update_irr(struct kvm_vcpu *vcpu, unsigned long *pir, int *m= ax_irr); void kvm_apic_update_ppr(struct kvm_vcpu *vcpu); int kvm_apic_set_irq(struct kvm_vcpu *vcpu, struct kvm_lapic_irq *irq, struct dest_map *dest_map); diff --git a/arch/x86/kvm/vmx/posted_intr.h b/arch/x86/kvm/vmx/posted_intr.h index ad9116a99bcc..4ff9d720dec0 100644 --- a/arch/x86/kvm/vmx/posted_intr.h +++ b/arch/x86/kvm/vmx/posted_intr.h @@ -18,7 +18,7 @@ static inline int pi_find_highest_vector(struct pi_desc *= pi_desc) { int vec; =20 - vec =3D find_last_bit((unsigned long *)pi_desc->pir, 256); + vec =3D find_last_bit(pi_desc->pir, 256); return vec < 256 ? vec : -1; } =20 --=20 2.49.0.rc1.451.g8f38331e32-goog From nobody Fri Dec 19 14:31:20 2025 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0DCE91990D9 for ; Sat, 15 Mar 2025 03:06:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742008002; cv=none; b=fQDCUq0Eh7fVMYkrJZ4dgWKfqOQEBxJQ7tTvO22ZE3HLAFRHBFRC/7MUbF9brqecz0titXGH1eRUcjCfvwXeVRD0yJ/45skJntzjZAlaD1ObXvifsqYv+nsABvmBjTmWDgQqrEpCMh1XOnOPvtw0KYnux1s9OwpA+datoBVQD4w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742008002; c=relaxed/simple; bh=YPH0PHJw1tpa8l8R9QQeeIkiVyk20jHhX+FQL8dEQKY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=dB88zZ37F+R1Cguk6c+JLkqvm2CQhfRnTi7uW6K/NRW5z5b061Qo7aau/NQUSVO6k0GH+Sk3Aid+p7+G7k3t/4l3ltIdnH7glADEYlN/oO0vCSZUK9KaFyVPaqeHpyCmiTaoOOpkCR34UPzaXO4dgCI5qerxVJQAO21Nkm1d1KA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Ojr76eZ7; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Ojr76eZ7" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-300fefb8e25so462429a91.3 for ; Fri, 14 Mar 2025 20:06:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1742008000; x=1742612800; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=yig8PS2Kt8/pb1c+Gkh8z0ntzMG9R4wz5XbuI5FRVMQ=; b=Ojr76eZ7/Cnv0xzFfKrdSeyZ72bzbBnuOeyFAMXAU9GWiWoKoOFRVz/QbIhLzMxrX6 ImRZBcHvQg6dEabtMGjRYoqvjou2z8+JD8X70Vw+cswDHnHNQx1PlQ50ikcNqSOQkTd9 KehFp2sdWkkuNSEZOxGQff+6Uclkmkxq62ngovRrGu8uXfBZUdlGxK5E39dDAqOW2J2A tRu/3gxr4sTpa/6oBCCA6KZz7YtakVlvkil7BIPd7wG8q6MeWVUA5vf7Xq/sp2e30uO9 hFPM9wl4AiNo1UnOM563eaqhXYCt9QBITMzrN/IpOSUOyQSf4q8/WgSIM09D8MApxvnl XyDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742008000; x=1742612800; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=yig8PS2Kt8/pb1c+Gkh8z0ntzMG9R4wz5XbuI5FRVMQ=; b=U+MGsfCfFfA1ris4MNP/T/ETM4HsmOjsooFbLgL/dH39ohprM1DD2q/vph98stb7/p IsyGnnVGdxQ75dcpsilSuicezdECIEZB+nIvhcysi+HNHo73kk0D6UnkFcxUayyQKrRm WfjiXxscbjotzAwWBbJNvIWjhFMuXzsdrD9twuyxRbjSaUIm4nQu71q4nvJqxeU0gF2U LoY+cFMCkjyGdnVJQ8cpAjxlAhbVMKVu1qwaNBYLN/L69boN7wSouPM7D0TTtdMDFsZu IyVfng5rDWSL5CztuJ3eqKk5Ft12PtKb+opNA18msbSOWtE5FS2+oTWjXyHqP6yqSjF4 MDqw== X-Gm-Message-State: AOJu0YzJTw9Maujk9dKB/B0/O16z//PygfFmOF8nXnyjoXgN54zBZOdM q+nNlJMgcbvd5cBVA9G47J24EP02nFU+NeQT0JPZM9buH7dg30GWbM3YS7z16jlyYr0xGBL2PMX iog== X-Google-Smtp-Source: AGHT+IE+tLpYYn+7SIxiMQbvt6BNA1OW+IsM1LNLKbpBuX9tC4yl9q4XDM8kwytvK/R01RnOYlCyet2fS44= X-Received: from pjl6.prod.google.com ([2002:a17:90b:2f86:b0:2ee:3128:390f]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90a:e18f:b0:2ee:b6c5:1def with SMTP id 98e67ed59e1d1-30151ce1224mr6192064a91.8.1742008000654; Fri, 14 Mar 2025 20:06:40 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 14 Mar 2025 20:06:26 -0700 In-Reply-To: <20250315030630.2371712-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250315030630.2371712-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.rc1.451.g8f38331e32-goog Message-ID: <20250315030630.2371712-6-seanjc@google.com> Subject: [PATCH 5/8] KVM: VMX: Process PIR using 64-bit accesses on 64-bit kernels From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jacob Pan , Jim Mattson Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Process the PIR at the natural kernel width, i.e. in 64-bit chunks on 64-bit kernels, so that the worst case of having a posted IRQ in each chunk of the vIRR only requires 4 loads and xchgs from/to the PIR, not 8. Deliberately use a "continue" to skip empty entries so that the code is a carbon copy of handle_pending_pir(), in anticipation of deduplicating KVM and posted MSI logic. Suggested-by: Jim Mattson Signed-off-by: Sean Christopherson --- arch/x86/kvm/lapic.c | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index 893e7d06e0e6..e4f182ee9340 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -657,26 +657,32 @@ static u8 count_vectors(void *bitmap) =20 bool __kvm_apic_update_irr(unsigned long *pir, void *regs, int *max_irr) { - u32 *__pir =3D (void *)pir; + unsigned long pir_vals[NR_PIR_WORDS]; + u32 *__pir =3D (void *)pir_vals; u32 i, vec; - u32 pir_val, irr_val, prev_irr_val; + u32 irr_val, prev_irr_val; int max_updated_irr; =20 max_updated_irr =3D -1; *max_irr =3D -1; =20 + for (i =3D 0; i < NR_PIR_WORDS; i++) { + pir_vals[i] =3D READ_ONCE(pir[i]); + if (!pir_vals[i]) + continue; + + pir_vals[i] =3D xchg(&pir[i], 0); + } + for (i =3D vec =3D 0; i <=3D 7; i++, vec +=3D 32) { u32 *p_irr =3D (u32 *)(regs + APIC_IRR + i * 0x10); =20 irr_val =3D READ_ONCE(*p_irr); - pir_val =3D READ_ONCE(__pir[i]); - - if (pir_val) { - pir_val =3D xchg(&__pir[i], 0); =20 + if (__pir[i]) { prev_irr_val =3D irr_val; do { - irr_val =3D prev_irr_val | pir_val; + irr_val =3D prev_irr_val | __pir[i]; } while (prev_irr_val !=3D irr_val && !try_cmpxchg(p_irr, &prev_irr_val, irr_val)); =20 --=20 2.49.0.rc1.451.g8f38331e32-goog From nobody Fri Dec 19 14:31:20 2025 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0794C199E8D for ; Sat, 15 Mar 2025 03:06:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742008004; cv=none; b=RSaBQkhcJE+20+2VFsC+PSwImGOLGtRbMwCXJ1jEf4zf9CgC/7VAZYIRLFk8lVZNC353kna9fF3lJpm7q55ivsqqVP5AlQg9QASJUWnaNQrp0vIBGsXTm7XNQmcuxOvxfSCZ+kkhWVn1uur1212Xhe2CFvzwM7mxqAcJN0i353k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742008004; c=relaxed/simple; bh=DIDA37JL8eeKhWGYjcN5PxZYRrpmJV0kRjJacIWUfhQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ri8tFKuWTKx7vP9js73kOTfmsX4XC93OrmqLMu4cs7+9ut5kPPbXxBjHScOXk9wHG7ynQdNt/3JCiSbi1M+pAalHkva+h7VhHwU9QHlsls5pzLcc7dyV2UtdtY5o6GprPI3v+Amfc2Kv3T3scmsI6OLxzz3FM1GQsNIQLdXjZkM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=1dv00/Uc; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="1dv00/Uc" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2ff68033070so473344a91.2 for ; Fri, 14 Mar 2025 20:06:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1742008002; x=1742612802; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=hv39/O/PnMVFihpfgZT9SpR99lFAgaV130aLFM9qjm4=; b=1dv00/UcK4oER/YLiOMaxFVrKEWirmRovwQWwX6yItUbKsCfoh7wT1wHH0k4wITgBC Q6Ad4AnqU8nJ1xIQevzuGzQPCrCNeX2URXUpPBUAhxbcMWgpvmhMGNWL+g0rVIrXDA/Y Qc7/pDe+k2f4vAd485redXeir59iSCOD32B/ZIgtrIhn12koiU2VyMqZFD1YD+SUjLy0 dB6Ua0QKW1juHyHTktBsqcg/ChpXS6LYe0CmX5YG5z1oK66ZE1CIVetlExdo47Y2gU5q U7sc03bYLwFU6qjIDpp5Ga4Tu1xJBgoemKAdxVfWx3kG8zjK8892umX+6JJZ/M0yMZUi ACqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742008002; x=1742612802; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=hv39/O/PnMVFihpfgZT9SpR99lFAgaV130aLFM9qjm4=; b=q/5iMvwJwFrzU2RtNlLwuaLOkljb0Z7WIW+jermJ6TPwVCwct1J15+FzxPL3ReIvBp nFjD4I6+ohzc3dSuQnpB2xIvlynV7a4DHZq+idvk3GbAHXdR42jYG+XxuRAkak4cyNOA +Uj9i4JoWgS7lJO+Fq7sCmQLfCizmb2bmXK2aEa/1f00/jrBlI/1LCqhEbHgByq5YMzz DNBhUOGfwYCk+KYJilwwpsUQ45DFXFTIbtFSVS3JZeJZleZXqgjcXXvQdr0CHqpkSuqi DQWlkFtaWyu2VuTxD57CqalPoMvEqhnj9YHB3QCjqwyKOZBeIdHr5DKZY8bJ57Qh3L6w saCQ== X-Gm-Message-State: AOJu0YxdOc47B1VTNYO/LtgTvcd4Jrjw8D6Jo4SygmJ8ZETEEn7FcfLj dhC1aJ5fiRgTMeyIw0iwwlIxDYmxTdLgs5/va7IGviQdqZcHOAQMANEgTJWnB8BL0ZP7yXV3zcj 3MQ== X-Google-Smtp-Source: AGHT+IHWjt/BJeVYOLv32GnaoqzmTC0s6P3U8PoNtZJT15p0guIAJtE+fq85gGZBlDG7yQuICOSFHSPkotI= X-Received: from pjbph6.prod.google.com ([2002:a17:90b:3bc6:b0:2fc:11a0:c549]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:514e:b0:2fa:157e:c78e with SMTP id 98e67ed59e1d1-30151cab344mr6435643a91.7.1742008002348; Fri, 14 Mar 2025 20:06:42 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 14 Mar 2025 20:06:27 -0700 In-Reply-To: <20250315030630.2371712-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250315030630.2371712-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.rc1.451.g8f38331e32-goog Message-ID: <20250315030630.2371712-7-seanjc@google.com> Subject: [PATCH 6/8] KVM: VMX: Isolate pure loads from atomic XCHG when processing PIR From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jacob Pan , Jim Mattson Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Rework KVM's processing of the PIR to use the same algorithm as posted MSIs, i.e. to do READ(x4) =3D> XCHG(x4) instead of (READ+XCHG)(x4). Given KVM's long-standing, sub-optimal use of 32-bit accesses to the PIR, it's safe to say far more thought and investigation was put into handling the PIR for posted MSIs, i.e. there's no reason to assume KVM's existing logic is meaningful, let alone superior. Matching the processing done by posted MSIs will also allow deduplicating the code between KVM and posted MSIs. See the comment for handle_pending_pir() added by commit 1b03d82ba15e ("x86/irq: Install posted MSI notification handler") for details on why isolating loads from XCHG is desirable. Suggested-by: Jim Mattson Signed-off-by: Sean Christopherson --- arch/x86/kvm/lapic.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index e4f182ee9340..d7e36faffc72 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -659,6 +659,7 @@ bool __kvm_apic_update_irr(unsigned long *pir, void *re= gs, int *max_irr) { unsigned long pir_vals[NR_PIR_WORDS]; u32 *__pir =3D (void *)pir_vals; + bool found_irq =3D false; u32 i, vec; u32 irr_val, prev_irr_val; int max_updated_irr; @@ -668,6 +669,14 @@ bool __kvm_apic_update_irr(unsigned long *pir, void *r= egs, int *max_irr) =20 for (i =3D 0; i < NR_PIR_WORDS; i++) { pir_vals[i] =3D READ_ONCE(pir[i]); + if (pir_vals[i]) + found_irq =3D true; + } + + if (!found_irq) + return false; + + for (i =3D 0; i < NR_PIR_WORDS; i++) { if (!pir_vals[i]) continue; =20 --=20 2.49.0.rc1.451.g8f38331e32-goog From nobody Fri Dec 19 14:31:20 2025 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7C58119C546 for ; Sat, 15 Mar 2025 03:06:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742008005; cv=none; b=gIMwf+zHOyeecVsXdoV62ns0vRgqHjMeVCL3dv3KYAeokR7TJa8WQSAgDss+0M/4lIn62Vb9ABPY8fL7paIvEww40APLdhS+mYgPvlXzQFyUTqvrk1w70pH5s6yA7sXUPO3m1TfQi4sWmLyzDclp2Hh5nELuaKU6S0afOi06bJA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742008005; c=relaxed/simple; bh=kyZPgKDP+G4NiFhQkapY7Z40kUAzSIF8X2Em11vw2i8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=uHtHq8j3UkwRI2r41hB3gZCgCVkDfIKdw7Oo6Rt+PxE7puxhtTiVGDVCHnrONvKqN8NLB6durDrHlFs4E1juGdVT49meqm+XMArBh6lVKTN4ddClAyPnFx6+OmQJ9uE7SomoreCf4D+mE5mS/LNyK9q6VRKr2FRUsEyPfBeJiMk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=gdiikDAQ; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="gdiikDAQ" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-225505d1ca5so45191445ad.2 for ; Fri, 14 Mar 2025 20:06:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1742008004; x=1742612804; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=G6aJw5/ES1UOfEaKxfAACWN89ZnVy5JuxHZXx05VAEo=; b=gdiikDAQmhkZVpccgyQyzXAoXEyPBvXk7uxwtMnsIEoCMG/6J/BXs7ntlpXlvhaXA1 qQ2LV5Hub8fmwTAxiZrRsS7Mz8b6GrBvz68AtJVzAugBDu0ah73wDHoDEgJ+CapP0m/q EGXDQM7H2T47tzcJmExK1A9Dy1gMDJXJm7AhnGDP8O47sqB5Lvj993ozXe4kca2xFmOI s6yvHWTJZZd1PmuroBrrlZCg6S08dtxaU7fWrgmxz+ZUGaSFKZN90ThcSj33hQzpLw6X lrO6ftmct1pjCMTkIvTNYuTStn+DYlbCrnlyatbV33ODq3YtPYzHRtmIIR49Tkp8ttWZ CSeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742008004; x=1742612804; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=G6aJw5/ES1UOfEaKxfAACWN89ZnVy5JuxHZXx05VAEo=; b=WsLX0/u+cczfe1rRgpeEEZ1FyNv6pG21QG/uQhJI5q41tqx/HXKAeJCXdh6vJnbTYB uZpZoak1N5/APIGOISUrTrnOhW5Rq5r/Sc6nO76Zzo7HO1f5veEUCfCRBJhk66UF9IOd zHJwOVDorYy8K3qNP3KusZ/ZnCleTiZ4S5vofErgYf1eZoC9gcggSPcnzPjfqYz0046e /jXiS4jj+1wcr9t2HRck2VjSTSVS1lkLdRAO64D551QeGcv7P5dibNsV8DCvQ+tTdsxX tduhkiCtcqKxoKL0OjIFUw6E9pmJy98xrNa67brvgITs34uArgTnHn8wEPIhz+DhU5W4 hJ0A== X-Gm-Message-State: AOJu0YyeIRZhQIO50wbRLoQ6beDrym6nDVRzbel6/kWR1VORJM5RRgBh Y9nsfwzWxgfx8EtwTJ5piNJNyGqOEXxaUFDblt38pNMCPkpbwqDYl6oXcg7Tmbbd8pPSCKQnBaw fCQ== X-Google-Smtp-Source: AGHT+IESdoU5JvhAzuOhjydI2CjAPYyqye5HZ3O+0f0UA0gPpp7LXZq02t7B33tZ8tgvi8g0NZmeMCZy9I0= X-Received: from pjbee15.prod.google.com ([2002:a17:90a:fc4f:b0:2fe:d556:ec6e]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:e548:b0:223:653e:eb09 with SMTP id d9443c01a7336-225e0a369c4mr48322735ad.7.1742008004023; Fri, 14 Mar 2025 20:06:44 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 14 Mar 2025 20:06:28 -0700 In-Reply-To: <20250315030630.2371712-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250315030630.2371712-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.rc1.451.g8f38331e32-goog Message-ID: <20250315030630.2371712-8-seanjc@google.com> Subject: [PATCH 7/8] KVM: VMX: Use arch_xchg() when processing PIR to avoid instrumentation From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jacob Pan , Jim Mattson Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use arch_xchg() when moving IRQs from the PIR to the vIRR, purely to avoid instrumentation so that KVM is compatible with the needs of posted MSI. This will allow extracting the core PIR logic to common code and sharing it between KVM and posted MSI handling. Signed-off-by: Sean Christopherson --- arch/x86/kvm/lapic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index d7e36faffc72..b65e0f7223fe 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -680,7 +680,7 @@ bool __kvm_apic_update_irr(unsigned long *pir, void *re= gs, int *max_irr) if (!pir_vals[i]) continue; =20 - pir_vals[i] =3D xchg(&pir[i], 0); + pir_vals[i] =3D arch_xchg(&pir[i], 0); } =20 for (i =3D vec =3D 0; i <=3D 7; i++, vec +=3D 32) { --=20 2.49.0.rc1.451.g8f38331e32-goog From nobody Fri Dec 19 14:31:20 2025 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 60E9519E997 for ; Sat, 15 Mar 2025 03:06:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742008009; cv=none; b=jiXHR9C339XPQqVDhU9jOwixqEovp+K6XpSPOdEs+zQpRWaow8wEaVsJk6tLP+AWqUTg0/8/lTBuiGsY2A0CzpbIoNLGMt4y4artD/C7HJz/oUMuuuildxW/KnELS3QtM/ZqCA0HqwNlegSaayXfB6fpiD23qs6Qa8Hh7OLeKKs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742008009; c=relaxed/simple; bh=6Ct4ueqTACT+40mgDDmV2LJlgiyUgvMZWTdwVdt+2AI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=PO8WDkfei+aUtfnYeT0DwB6DKHRNPwUyX9I62FHD+Ww+vyesmo1Ne2DZkWNJYJpR8dN5KLXyKBLAuLSFHXWu1WPmdT7LtG663Pyt38MqIk19inN4SOftm6w6C0bEcenVL7zguiUnWvGs1jOJ0fAt+xwn67LDH9e83AWc8vQU40o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ZnE+4jTn; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ZnE+4jTn" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2feb47c6757so346902a91.3 for ; Fri, 14 Mar 2025 20:06:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1742008005; x=1742612805; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=Rqh78WRgDr7mgiGuvPqk9U1mgup3ydPWLYkuR7DIMcg=; b=ZnE+4jTnlbpy00U4j0HY28sde+eEOjoiEsddr8wce1yFa3en54ot9o407cEAIIoB5F qBnLJBUllfU6zfE6AAE0HuOBVVctF9EcfV+mz1a1b7sjkre0Se+Hrq0pJlM8YYKzldGI nC4p7MjI/YIT29mE1IKONotGxwL/hJS1DSNvW5N/LMoO6Pg6IUg5pYiuqWLUHUbmiTEm alGFNDOP+bY0Uzeab7rSiT9jFAthbxnx94DQrNb6sBSaOrQXenCegqLs+xYdfJDHE0gu NW/SE0XQhu6GdMroMvM8VgOOjfmIDLsc64gxmG50+gFZSk9BbWAb455oq5pFn7UExP9E 4hRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742008005; x=1742612805; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Rqh78WRgDr7mgiGuvPqk9U1mgup3ydPWLYkuR7DIMcg=; b=Nixy/RpvMDpcRFXrsw2h7HS0vkhdzannBdnsMNkiysZnF9F53pBQdIf4C1voTAZckm lPNPf4zxkSyeNlHqgbUlrn4TmT/oqtR/ItZUZLg56m9zjUFJcPbp9cmTLXF+ZYqAe5T+ 24MTH7Xa/90uMvsDKsQasgyTJce3PCXPo/T/CY9Kc8mBxhsxtQ+TY+C+MZ80fDGCAcd9 c7kKUrB5Bpa8SECK4suwLIzjS50NQCfxeVMdQhM0s7QCyEwEGou7z6+893KzVr0Eo2LF kFrwJ0fkxboPvq+I6Q6rTFs1DM0i16WlUFdm0XPO7y4P0iCae4y6RqcvnyFLZu1Vum6Z SpEg== X-Gm-Message-State: AOJu0YxsbdgCyQp+YOPnMDPf11NYxKKCpqz3qELd9j3DTyyxYghF09CR s26ii70kOyrNUAEOmMHBLZMSwQe9GHS3IltYkN/9jePcL5u1TEZvUKqtEBdvnKH6rLPpSSIzw1m U8w== X-Google-Smtp-Source: AGHT+IFf+R/IdkB9dMtQz0tp3pLrCC/oKPbyU6izvyqg+RZIyQsLWC+utkXYHnwPh9OG4z6WE2OE7PdpDaY= X-Received: from pjbdy6.prod.google.com ([2002:a17:90b:6c6:b0:2f8:49ad:406c]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:4d10:b0:2fe:b470:dde4 with SMTP id 98e67ed59e1d1-30151cd796dmr7085921a91.12.1742008005673; Fri, 14 Mar 2025 20:06:45 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 14 Mar 2025 20:06:29 -0700 In-Reply-To: <20250315030630.2371712-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250315030630.2371712-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.rc1.451.g8f38331e32-goog Message-ID: <20250315030630.2371712-9-seanjc@google.com> Subject: [PATCH 8/8] x86/irq: KVM: Add helper for harvesting PIR to deduplicate KVM and posted MSIs From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jacob Pan , Jim Mattson Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that posted MSI and KVM harvesting of PIR is identical, extract the code (and posted MSI's wonderful comment) to a common helper. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/include/asm/posted_intr.h | 65 ++++++++++++++++++++++++++++++ arch/x86/kernel/irq.c | 50 +---------------------- arch/x86/kvm/lapic.c | 16 +------- 3 files changed, 68 insertions(+), 63 deletions(-) diff --git a/arch/x86/include/asm/posted_intr.h b/arch/x86/include/asm/post= ed_intr.h index c3e6e4221a5b..6b1ddebbf06a 100644 --- a/arch/x86/include/asm/posted_intr.h +++ b/arch/x86/include/asm/posted_intr.h @@ -1,8 +1,13 @@ /* SPDX-License-Identifier: GPL-2.0 */ #ifndef _X86_POSTED_INTR_H #define _X86_POSTED_INTR_H + +#include +#include #include =20 +#include + #define POSTED_INTR_ON 0 #define POSTED_INTR_SN 1 =20 @@ -26,6 +31,66 @@ struct pi_desc { u32 rsvd[6]; } __aligned(64); =20 +/* + * De-multiplexing posted interrupts is on the performance path, the code + * below is written to optimize the cache performance based on the followi= ng + * considerations: + * 1.Posted interrupt descriptor (PID) fits in a cache line that is freque= ntly + * accessed by both CPU and IOMMU. + * 2.During software processing of posted interrupts, the CPU needs to do + * natural width read and xchg for checking and clearing posted interrupt + * request (PIR), a 256 bit field within the PID. + * 3.On the other side, the IOMMU does atomic swaps of the entire PID cache + * line when posting interrupts and setting control bits. + * 4.The CPU can access the cache line a magnitude faster than the IOMMU. + * 5.Each time the IOMMU does interrupt posting to the PIR will evict the = PID + * cache line. The cache line states after each operation are as follows, + * assuming a 64-bit kernel: + * CPU IOMMU PID Cache line state + * --------------------------------------------------------------- + *...read64 exclusive + *...lock xchg64 modified + *... post/atomic swap invalid + *...------------------------------------------------------------- + * + * To reduce L1 data cache miss, it is important to avoid contention with + * IOMMU's interrupt posting/atomic swap. Therefore, a copy of PIR is used + * when processing posted interrupts in software, e.g. to dispatch interru= pt + * handlers for posted MSIs, or to move interrupts from the PIR to the vIRR + * in KVM. + * + * In addition, the code is trying to keep the cache line state consistent + * as much as possible. e.g. when making a copy and clearing the PIR + * (assuming non-zero PIR bits are present in the entire PIR), it does: + * read, read, read, read, xchg, xchg, xchg, xchg + * instead of: + * read, xchg, read, xchg, read, xchg, read, xchg + */ +static __always_inline bool pi_harvest_pir(unsigned long *pir, + unsigned long *pir_vals) +{ + bool found_irq =3D false; + int i; + + for (i =3D 0; i < NR_PIR_WORDS; i++) { + pir_vals[i] =3D READ_ONCE(pir[i]); + if (pir_vals[i]) + found_irq =3D true; + } + + if (!found_irq) + return false; + + for (i =3D 0; i < NR_PIR_WORDS; i++) { + if (!pir_vals[i]) + continue; + + pir_vals[i] =3D arch_xchg(&pir[i], 0); + } + + return true; +} + static inline bool pi_test_and_set_on(struct pi_desc *pi_desc) { return test_and_set_bit(POSTED_INTR_ON, (unsigned long *)&pi_desc->contro= l); diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index 704c104ff7a4..b98a5abdeaec 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -373,60 +373,14 @@ void intel_posted_msi_init(void) this_cpu_write(posted_msi_pi_desc.ndst, destination); } =20 -/* - * De-multiplexing posted interrupts is on the performance path, the code - * below is written to optimize the cache performance based on the followi= ng - * considerations: - * 1.Posted interrupt descriptor (PID) fits in a cache line that is freque= ntly - * accessed by both CPU and IOMMU. - * 2.During posted MSI processing, the CPU needs to do 64-bit read and xchg - * for checking and clearing posted interrupt request (PIR), a 256 bit f= ield - * within the PID. - * 3.On the other side, the IOMMU does atomic swaps of the entire PID cache - * line when posting interrupts and setting control bits. - * 4.The CPU can access the cache line a magnitude faster than the IOMMU. - * 5.Each time the IOMMU does interrupt posting to the PIR will evict the = PID - * cache line. The cache line states after each operation are as follows: - * CPU IOMMU PID Cache line state - * --------------------------------------------------------------- - *...read64 exclusive - *...lock xchg64 modified - *... post/atomic swap invalid - *...------------------------------------------------------------- - * - * To reduce L1 data cache miss, it is important to avoid contention with - * IOMMU's interrupt posting/atomic swap. Therefore, a copy of PIR is used - * to dispatch interrupt handlers. - * - * In addition, the code is trying to keep the cache line state consistent - * as much as possible. e.g. when making a copy and clearing the PIR - * (assuming non-zero PIR bits are present in the entire PIR), it does: - * read, read, read, read, xchg, xchg, xchg, xchg - * instead of: - * read, xchg, read, xchg, read, xchg, read, xchg - */ static __always_inline bool handle_pending_pir(unsigned long *pir, struct = pt_regs *regs) { - int i, vec =3D FIRST_EXTERNAL_VECTOR; + int vec =3D FIRST_EXTERNAL_VECTOR; unsigned long pir_copy[NR_PIR_WORDS]; - bool found_irq =3D false; =20 - for (i =3D 0; i < NR_PIR_WORDS; i++) { - pir_copy[i] =3D READ_ONCE(pir[i]); - if (pir_copy[i]) - found_irq =3D true; - } - - if (!found_irq) + if (!pi_harvest_pir(pir, pir_copy)) return false; =20 - for (i =3D 0; i < NR_PIR_WORDS; i++) { - if (!pir_copy[i]) - continue; - - pir_copy[i] =3D arch_xchg(&pir[i], 0); - } - for_each_set_bit_from(vec, pir_copy, FIRST_SYSTEM_VECTOR) call_irq_handler(vec, regs); =20 diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index b65e0f7223fe..1c611b84b8ab 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -659,7 +659,6 @@ bool __kvm_apic_update_irr(unsigned long *pir, void *re= gs, int *max_irr) { unsigned long pir_vals[NR_PIR_WORDS]; u32 *__pir =3D (void *)pir_vals; - bool found_irq =3D false; u32 i, vec; u32 irr_val, prev_irr_val; int max_updated_irr; @@ -667,22 +666,9 @@ bool __kvm_apic_update_irr(unsigned long *pir, void *r= egs, int *max_irr) max_updated_irr =3D -1; *max_irr =3D -1; =20 - for (i =3D 0; i < NR_PIR_WORDS; i++) { - pir_vals[i] =3D READ_ONCE(pir[i]); - if (pir_vals[i]) - found_irq =3D true; - } - - if (!found_irq) + if (!pi_harvest_pir(pir, pir_vals)) return false; =20 - for (i =3D 0; i < NR_PIR_WORDS; i++) { - if (!pir_vals[i]) - continue; - - pir_vals[i] =3D arch_xchg(&pir[i], 0); - } - for (i =3D vec =3D 0; i <=3D 7; i++, vec +=3D 32) { u32 *p_irr =3D (u32 *)(regs + APIC_IRR + i * 0x10); =20 --=20 2.49.0.rc1.451.g8f38331e32-goog