From nobody Mon Apr 6 16:28:11 2026 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 86000331A44 for ; Thu, 19 Mar 2026 06:15:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773900948; cv=none; b=fMsQBxSkI7l6iVEkGvClankJ9IHgXZuo2TC3Cgquo3r8yq6/RtY0MYaJJxi9RPxe62IKBPqekT64AuHQG4xuDfobBbvYad7gQ6fnHauVuR9dA3WZUHrjVQmcF/skhKHIJFonJD5kIMCjECcd26J9ABbQBIj8s2c5/LiY7v1BVh4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773900948; c=relaxed/simple; bh=0S8tvoyNB0MBm9GLPrKZBj/V5Cfw8tU439iV60Dpy+w=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=BjLolnOLkA3b10lxnj/3EcJq03nnSJlVx4Yold79Dz/fw9k4WyzJzN/GtkPtPJqYHQLGWjLtSA/n4QVkIwBocmKk/NW9to5VmuM9pPeEIGj9Nl3oSxMirkQk+ETKaq8VV7n7Sv+QM+JXmiQYABNe/URH+cgRP7uZvOOe37YF3ms= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=m6VWcpaT; arc=none smtp.client-ip=209.85.210.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="m6VWcpaT" Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-829ac4670c4so276743b3a.0 for ; Wed, 18 Mar 2026 23:15:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1773900946; x=1774505746; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8OXeZt9Y4f4f82jU5A2ObTC5XK08MnscssukO7f4OZg=; b=m6VWcpaTuzQ3NnWnDK3TATsy7f2+ooIW4LBUGZSfzhSE98+yTjmqfgyzBfCHKfHrjB 7X2z1Nbskpb/Xly4s8bta3WFY9CVDOBNse1+7aoiTHcLOUYo1HitLpg2aRYuxdUZhjAv SurPcsBu6iZsXlEGSjT3nvkpLcTcFT1cSPtCJ6PWLPT8UDQodsL3nR9i/UWsZliKwm0o V3mxvbh2GtYPBhEZyksmWzoAIEHpzEYROWAqtfTVPo/TppKWkQxSRgKrpTVcLkx6/Rds tzNQC2qowhPKAojddAQW4mKH6fjshoyWKwQ8o6L/4PIy/tMxip5iOLFqVKIOPb+Ip5PJ e+HQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1773900946; x=1774505746; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=8OXeZt9Y4f4f82jU5A2ObTC5XK08MnscssukO7f4OZg=; b=cWBmgIZMcHpDJ9hZ4FRf1romedHD7XZegPPQwgzlhT2ylL3cjQQ0LAUaN9zc9Z3LLi r0dIA2Kn6/2zbcbuYddiv0z0fcDe8WZbl34zZeLo119SAuKlOh8Cl37g+f5jqNUjiJwj 86aQhUXa1ZJEyOpMNJ6ROd/w/DXUdWMQG784GYEYALnwnlPSuAvimdJzeBHx9xhAaZ2K AHLQL5rTilubQffnHBN1hTWncY1/J6NMIYpfKXCAk/mkY9yet8YJZxl/Q0ef2T755b/Q xnZQTK9+uU3L5tp1RtKUTvW7cSI2aNoSIb4yY5JCRYPrrjkArKJX2oEepYq4okFrwhHx 6Pgw== X-Forwarded-Encrypted: i=1; AJvYcCVcXMtz+P3q/eGs+O/CQ9NvIlSV1KBSYWfZ+RJ2i+r7KlzTNRf+5Qd5CJOZneokvWh4zgw4rDiAXRtLYZA=@vger.kernel.org X-Gm-Message-State: AOJu0YybGJ21jVq7nBkuCun9GugAHcmzX7sgRyUrySLnNrxkqdS9v7Lt H1xv/APzUrknHIG17JxDDTreUIiKXOtDKT7N2uDXbCp/bdRYx7lqDN+/ X-Gm-Gg: ATEYQzzyjBj74tTTMc5bb0PFSSwXENKDttbUn8WUtDBTd6hyFtM3HjcaBc9asB27gE2 9hjaApK6XD6ELO5Th6kGBs0KeKMjUKAXz+8k1gyknynex4Sei9YPfT2ps/TjXhF/MlHAy1iZufV F8Et9CBKQkH3fRk7xn19H1VEz4LqnUX+TclFcdUFg/LxOxUGVQPF9Ju7E+dWvdlJxa+phW/dnfr seLjZOE195qjBRI6AjLsP87MwQe28Ju9g3ytuWRzjTvD24tGm7MahkgqFNLRyf+WTluLPoaeMVA dDdbtMUN/ikWQyL2Wwx5yoCfCQlNmXwAKM7P/4ihn3iDlglyoT59yVtWLRutaal3r09ruLmz5hE Mf1GstD856LOr8wZaukOHY5TtpLD0/dVV6o5xlm3LxLQHgnvctm4ZEaBkrwzEnedE+h/s0bTX8K zpkRKu7j6v7ysuPJgvSxD0wBaq7MdkYAjbyAccMNsPntNQyEELlip1+KGDiDQbMmkYPBF3UHZzE LLPJn0FdHxVgg== X-Received: by 2002:a05:6a00:399c:b0:827:3307:170 with SMTP id d2e1a72fcca58-82a6ae53fafmr5958993b3a.37.1773900945809; Wed, 18 Mar 2026 23:15:45 -0700 (PDT) Received: from AHUANG12-3ZHH9X.lenovo.com (61-221-208-111.hinet-ip.hinet.net. [61.221.208.111]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-82a6bef5ed8sm5660824b3a.57.2026.03.18.23.15.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Mar 2026 23:15:45 -0700 (PDT) From: "Adrian Huang (Lenovo)" To: Joerg Roedel , Suravee Suthikulpanit Cc: Will Deacon , Robin Murphy , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, ahuang12@lenovo.com, "Adrian Huang (Lenovo)" Subject: [PATCH 1/2] iommu/amd: Keep x2apic enabled when appending amd_iommu=off Date: Thu, 19 Mar 2026 14:15:06 +0800 Message-Id: <20260319061507.541-2-adrianhuang0701@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20260319061507.541-1-adrianhuang0701@gmail.com> References: <20260319061507.541-1-adrianhuang0701@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When booting with amd_iommu=3Doff, the AMD IOMMU driver currently disables the entire IOMMU subsystem, which also results in x2APIC being turned off. Consequently, the APIC mode falls back from x2APIC to physical flat mode. The dmesg log shows the transition: x2apic: enabled by BIOS, switching to x2apic ops ... APIC: Switched APIC routing to: cluster x2apic ... APIC: Switch to symmetric I/O mode setup x2apic: IRQ remapping doesn't support X2APIC mode x2apic disabled APIC: Switched APIC routing to: physical flat Since physical flat mode supports only up to 255 APIC IDs, systems with large CPU counts cannot fully initialize. For example, on a 448-core system, the kernel reports repeated errors such as: smpboot: CPU 112 has invalid APIC ID 100. Aborting bringup smpboot: CPU 113 has invalid APIC ID 102. Aborting bringup ... smp: Brought up 2 nodes, 224 CPUs Eventually, only 224 CPUs are brought up because of valid APIC IDs and the APIC ID limitation of the physical flat mode. In contrast, on an Intel platform with 960 cores, booting with intel_iommu=3Doff does not disable x2APIC: x2apic: enabled by BIOS, switching to x2apic ops ... APIC: Switched APIC routing to: cluster x2apic ... APIC: Switch to symmetric I/O mode setup DMAR: Host address width 46 ... smpboot: CPU0: Intel(R) Xeon(R) Platinum 8490H (family: 0x6, model: 0x8f, stepping: 0x6) ... smp: Brought up 8 nodes, 960 CPUs This confirms that x2APIC remains enabled when intel_iommu=3Doff is specified. Adjust the semantics of "amd_iommu=3Doff" so that: * DMA translation is disabled * x2apic remains enabled This preserves x2APIC functionality and allows large CPU count systems to operate correctly, while still disabling DMA remapping. With this patch, the system correctly brings up all 448 cores when booting with amd_iommu=3Doff, as verified by the logs below: x2apic: enabled by BIOS, switching to x2apic ops ... APIC: Switched APIC routing to: cluster x2apic ... APIC: Switch to symmetric I/O mode setup ... smp: Brought up 2 nodes, 448 CPUs ... AMD-Vi: Interrupt remapping enabled AMD-Vi: X2APIC enabled ... Signed-off-by: Adrian Huang (Lenovo) --- drivers/iommu/amd/amd_iommu.h | 1 + drivers/iommu/amd/init.c | 56 ++++++++++++++++++++++++++++------- drivers/iommu/amd/iommu.c | 14 +++++++++ 3 files changed, 61 insertions(+), 10 deletions(-) diff --git a/drivers/iommu/amd/amd_iommu.h b/drivers/iommu/amd/amd_iommu.h index 1342e764a548..82c10f55f0ea 100644 --- a/drivers/iommu/amd/amd_iommu.h +++ b/drivers/iommu/amd/amd_iommu.h @@ -22,6 +22,7 @@ void amd_iommu_restart_event_logging(struct amd_iommu *io= mmu); void amd_iommu_restart_ga_log(struct amd_iommu *iommu); void amd_iommu_restart_ppr_log(struct amd_iommu *iommu); void amd_iommu_set_rlookup_table(struct amd_iommu *iommu, u16 devid); +void amd_iommu_dev_set_pci_msi_domain(struct device *dev); void iommu_feature_enable(struct amd_iommu *iommu, u8 bit); void *__init iommu_alloc_4k_pages(struct amd_iommu *iommu, gfp_t gfp, size_t size); diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c index f3fd7f39efb4..0f577534702d 100644 --- a/drivers/iommu/amd/init.c +++ b/drivers/iommu/amd/init.c @@ -3396,6 +3396,41 @@ static __init void iommu_snp_enable(void) #endif } =20 +static int __init amd_iommu_devices_set_pci_msi_domain(void) +{ + struct pci_dev *dev =3D NULL; + struct amd_iommu *iommu; + int ret =3D 0; + + /* Register IRQ handler for each iommu device. */ + for_each_iommu(iommu) { + iommu->dev =3D pci_get_domain_bus_and_slot(iommu->pci_seg->id, + PCI_BUS_NUM(iommu->devid), + iommu->devid & 0xff); + if (!iommu->dev) + return -ENODEV; + + ret =3D iommu_init_irq(iommu); + if (ret) + return ret; + + iommu->dev->irq_managed =3D 1; + } + + /* + * In configurations where the IOMMU is disabled but x2APIC is + * required for high CPU counts (> 256), the kernel must explicitly + * map PCI Message Signaled Interrupt (MSI) domains to the IOMMU + * hardware's interrupt domain to ensure valid interrupt routing. + */ + for_each_pci_dev(dev) + amd_iommu_dev_set_pci_msi_domain(&dev->dev); + + print_iommu_info(); + + return ret; +} + /*************************************************************************= *** * * AMD IOMMU Initialization State Machine @@ -3416,13 +3451,8 @@ static int __init state_next(void) } break; case IOMMU_IVRS_DETECTED: - if (amd_iommu_disabled) { - init_state =3D IOMMU_CMDLINE_DISABLED; - ret =3D -EINVAL; - } else { - ret =3D early_amd_iommu_init(); - init_state =3D ret ? IOMMU_INIT_ERROR : IOMMU_ACPI_FINISHED; - } + ret =3D early_amd_iommu_init(); + init_state =3D ret ? IOMMU_INIT_ERROR : IOMMU_ACPI_FINISHED; break; case IOMMU_ACPI_FINISHED: early_enable_iommus(); @@ -3430,9 +3460,15 @@ static int __init state_next(void) init_state =3D IOMMU_ENABLED; break; case IOMMU_ENABLED: - register_syscore(&amd_iommu_syscore); - iommu_snp_enable(); - ret =3D amd_iommu_init_pci(); + if (amd_iommu_disabled) { + amd_iommu_devices_set_pci_msi_domain(); + init_state =3D IOMMU_CMDLINE_DISABLED; + ret =3D -EINVAL; + } else { + register_syscore(&amd_iommu_syscore); + iommu_snp_enable(); + ret =3D amd_iommu_init_pci(); + } init_state =3D ret ? IOMMU_INIT_ERROR : IOMMU_PCI_INIT; break; case IOMMU_PCI_INIT: diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c index 81c4d7733872..75b10eca9ecf 100644 --- a/drivers/iommu/amd/iommu.c +++ b/drivers/iommu/amd/iommu.c @@ -2439,6 +2439,20 @@ static void detach_device(struct device *dev) mutex_unlock(&dev_data->mutex); } =20 +void amd_iommu_dev_set_pci_msi_domain(struct device *dev) +{ + struct amd_iommu *iommu; + + if (!check_device(dev)) + return; + + iommu =3D rlookup_amd_iommu(dev); + if (!iommu) + return; + + amd_iommu_set_pci_msi_domain(dev, iommu); +} + static struct iommu_device *amd_iommu_probe_device(struct device *dev) { struct iommu_device *iommu_dev; --=20 2.47.3 From nobody Mon Apr 6 16:28:11 2026 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 012A734572F for ; Thu, 19 Mar 2026 06:15:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773900951; cv=none; b=MuW7C2BiLVTDJk82TalBbb/uULC6t56MKXtwzzQesD3LF84Lgpl3yShjC+5PUpijebGZ12i9Ydbv5fCpRLp+YLrB9dIvNUwpZHo7C667JfWxrLgYWDp83NToQBmBsUky3ZYUpsuO2RJIaXUCmH1HDvY5XviW9XssF4Slq+8EFbU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773900951; c=relaxed/simple; bh=2DHIOR/xu0FQ/B1GFdC6D07OnelCvpt03qpXdP8Z//s=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=GUu7f5zfc7lsUhwNNMMCfEPOQNID8jbDaV9pzEZCxFwXfx+THO3ODnh/lM/naEGljunKP6235KtM52+s7d9uet2rT6VGUeZeXaf13C1SBV0aApJ8FtSZmvBm/dB03bpP97d2/s5hfPtUYF5ZSLED6vfPsR84MXt1xzVnih6anpk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=RzJYyNmm; arc=none smtp.client-ip=209.85.210.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RzJYyNmm" Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-8297e0b27e5so396314b3a.1 for ; Wed, 18 Mar 2026 23:15:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1773900949; x=1774505749; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NsT73j4cLvpq8BjBjqE6tomVf3siHf2xqAbxNaDSyB0=; b=RzJYyNmmNwZbTIMLVTW+Tc1dmr9HA1SYcrq7xWbrRDAc87z7T3s56PsWh75J0+J59b Ob25zBJi5jjIrutSYjKiBiGTdceODDNrsbYMa9GGzWFt3dg7LUGES6pKXR45sswQEEcN NbiqMu3byVfn+GeqmLT5oytwLT3MzOMQzavEmuzMuP1YgOUqeEWMmZEjs1ios0HTIDPO ey3LjadjbX+SdA0553P7AyZnCxc9Uj98cSNb+jDeEpifg6V1hJnboD/VSUs4xcE9VgEE EQhdZZxtL83mIX0iNq2GbfEd+HCbOfjHgBtXxDaDjt6FuEf43O0ENuON6K4algncVN5K fG2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1773900949; x=1774505749; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=NsT73j4cLvpq8BjBjqE6tomVf3siHf2xqAbxNaDSyB0=; b=Aw4ySzG0C2I5ZhXXtc1qz4u53bcNfwc/cB4Ip5EMv9NDPN77VNMP7wSqLrMNXUjYJi ruHDOfm2W+RWVval0uCWLYiz0oG/V0atG59pbDvb6OxRc25eySqNlTn+lVq6PE2gh2Hk cEqOJpda2UEo2mBey85phFHapBUCh5FYpNwVxZjSSAGZxSyjw4E/PoSBeHIpdJ7a1I5Y votNYkqc9cI9lYplkfRVsnbwjGa5aNise1wAGuh4kXxPsU5NGvpFjlslfR7Oef3Zs+C7 NWfUm5AfSAJUe0bNI7f4e+QNWoSZB+IVz7iaaVEQ1xVqviL/9WZH+A5U0nYGvbeHr+KB aZ3g== X-Forwarded-Encrypted: i=1; AJvYcCUilVdKbuhk+m91N3BqO3df4J6J9XoQ6rBmrz/5jOentY9/oYMgfjj+UEnswJk2B+c7KiPvVXxW7tVRC94=@vger.kernel.org X-Gm-Message-State: AOJu0Yxf6ZzIE8GLwyLzuRUDhvinmpGGmT7gvzpUyhxU76R8r5YwuP4r pWzvf1VtEo+UjEgSnhwJFoE5RiodGpk9NNemuddDNH+KIjc2WF/EM4Ne X-Gm-Gg: ATEYQzz2XPN+KQnj1B2ml/b4JL2Ds3zJKKe9h5bHLw8f0vZ0VQ8/HrRFkolw5uoudth nB4OU72OLB3vsU6Ciqvcznk3mwnlATeWoGqZXQgYBeVjt0vgiauNg6Ng/OczjUvy88qygxZbrDY eFt4BCZsz4Qr49yQ0B+67FGHf5nyNw1AAsMUr5jAmRloadNqdMHJ/mKUgKHg+vFyBoiC/Z0G/1P A98LiDIPJqElZZ3wtWDwWz5hIoQgaCv7yij10I/ScjQN8/g5F1YrOIuQRwlWLnNulS5J9iXjyt5 vS7bWQvRJvLq65IRnD3LwVUKPbCAyGx46wPa/igiBRxotm8pLay48OPuYBoKenXBfw/wqu9Y3ad XH0vcs1BH2KnsnM0bWAoa8QiT2owHDmLQnu2XQcckph4tZb0BBiewMA8HrT2iQmaT/xGBKq2CCJ YK6ZBH1QdDTrW7fesKIWEs7tjb68I2t3j3wDdXZlxmMIyQ+Gyf2g/PcJnPtE0JRDEZMF6E3+qMf 1RQRpUy1T/lnw== X-Received: by 2002:a05:6a00:190b:b0:829:7d31:dd99 with SMTP id d2e1a72fcca58-82a6ae7084fmr5481255b3a.38.1773900949283; Wed, 18 Mar 2026 23:15:49 -0700 (PDT) Received: from AHUANG12-3ZHH9X.lenovo.com (61-221-208-111.hinet-ip.hinet.net. [61.221.208.111]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-82a6bef5ed8sm5660824b3a.57.2026.03.18.23.15.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Mar 2026 23:15:48 -0700 (PDT) From: "Adrian Huang (Lenovo)" To: Joerg Roedel , Suravee Suthikulpanit Cc: Will Deacon , Robin Murphy , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, ahuang12@lenovo.com, "Adrian Huang (Lenovo)" Subject: [PATCH 2/2] iommu/amd: Register PCI bus notifier when appending amd_iommu=off Date: Thu, 19 Mar 2026 14:15:07 +0800 Message-Id: <20260319061507.541-3-adrianhuang0701@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20260319061507.541-1-adrianhuang0701@gmail.com> References: <20260319061507.541-1-adrianhuang0701@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When the amd_iommu=3Doff kernel parameter is used, dynamically adding PCI devices (e.g., creating SR-IOV VFs) triggers a call trace and IOMMU fault events. For example, enabling VFs via sysfs: # echo 4 > /sys/class/net/eth1/device/sriov_numvfs It results in an INVALID_DEVICE_REQUEST and a warning in __irq_msi_compose_msg+0xa3/0xb0: pci 0000:41:03.0: [14e4:16c1] type 00 class 0x020000 PCIe Endpoint bnxt_en 0000:41:03.0: enabling device (0000 -> 0002) bnxt_en 0000:41:03.0 eth4: Broadcom NetXtreme-E Ethernet Virtual Function= found at mem 381c7ff40000, node addr 92:42:39:64:ad:24 ... WARNING: arch/x86/kernel/apic/apic.c:2313 at __irq_msi_compose_msg+0xa3/0= xb0, CPU#2: NetworkManager/4268 bnxt_en 0000:41:03.1: enabling device (0000 -> 0002) CPU: 2 UID: 0 PID: 4268 Comm: NetworkManager Kdump: loaded Not tainted 7.= 0.0-rc4+ #23 PREEMPT(lazy) ... RIP: 0010:__irq_msi_compose_msg+0xa3/0xb0 ... Call Trace: irq_chip_compose_msi_msg+0x2e/0x50 msi_domain_activate+0x41/0x90 __irq_domain_activate_irq+0x53/0x90 ? mtree_load+0x26e/0x2b0 irq_domain_activate_irq+0x29/0x40 __setup_irq+0x322/0x790 ? __pfx_bnxt_msix+0x10/0x10 [bnxt_en] request_threaded_irq+0x109/0x1c0 bnxt_request_irq+0xee/0x270 [bnxt_en] ... AMD-Vi: Event logged [INVALID_DEVICE_REQUEST device=3D0000:41:03.0 pasid= =3D0x00000 address=3D0xfffffffdf8220100 flags=3D0x0a00] The root cause is that when IOMMU is disabled via the command line, the PCI bus notifier is not registered. Consequently, the MSI domain for dynamically added devices is not properly associated with the IOMMU interrupt remapping domain. Even when the IOMMU is "off," certain interrupt remapping configurations must still be handled for newly enumerated devices to prevent activation failures. Fix this by ensuring the PCI bus notifier is registered even when amd_iommu=3Doff is set, allowing the IOMMU interrupt remapping domain to be correctly assigned to new PCI devices. Signed-off-by: Adrian Huang (Lenovo) --- drivers/iommu/amd/init.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c index 0f577534702d..5509ade0cf0d 100644 --- a/drivers/iommu/amd/init.c +++ b/drivers/iommu/amd/init.c @@ -3431,6 +3431,30 @@ static int __init amd_iommu_devices_set_pci_msi_doma= in(void) return ret; } =20 +static int amd_iommu_pci_bus_notifier(struct notifier_block *nb, + unsigned long action, void *data) +{ + struct pci_dev *pdev =3D to_pci_dev(data); + + /* We only care about the add event. */ + if (action !=3D BUS_NOTIFY_ADD_DEVICE) + return NOTIFY_DONE; + + amd_iommu_dev_set_pci_msi_domain(&pdev->dev); + + return NOTIFY_OK; +} + +static struct notifier_block amd_iommu_pci_bus_nb =3D { + .notifier_call =3D amd_iommu_pci_bus_notifier, + .priority =3D 1, +}; + +static void __init amd_iommu_register_bus_notifier(void) +{ + bus_register_notifier(&pci_bus_type, &amd_iommu_pci_bus_nb); +} + /*************************************************************************= *** * * AMD IOMMU Initialization State Machine @@ -3462,6 +3486,7 @@ static int __init state_next(void) case IOMMU_ENABLED: if (amd_iommu_disabled) { amd_iommu_devices_set_pci_msi_domain(); + amd_iommu_register_bus_notifier(); init_state =3D IOMMU_CMDLINE_DISABLED; ret =3D -EINVAL; } else { --=20 2.47.3