From nobody Wed Dec 17 08:52:32 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B0D128DF0E for ; Tue, 13 May 2025 16:31:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747153912; cv=none; b=PBM/jdbN98gRitE9F/4u7wygrmYqmAJqb3RBkA2ETScO5PuDnL4EFrtdrP/iH7jjhpUpBAdBwv6VajBjegsrUSg7iJJzvrKU6lbBRRMQWqKJ+g1AlX2SheCk6bZ0GL2+lv25LMLud7IOLFRuXA8+JiXxLFS3RkguDpq3nJDzYFI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747153912; c=relaxed/simple; bh=RcZzjfbX3puoN0l0SuXE8X3V8BVXm+FxxK0G7lqM4z8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jlFWDp9HhhqJCn9GE4cGXaodMUJhAbPSwUBO+SB0BAST0msqgCKuPhJSmN/hscbHsn1tIzqVdeeXOX8wQasbgR+Q5xFQCW9JUprGxLSJb545UoFVEbgChlKgQT9q9WHfe/0jMmIxb5wUVEfdYRCWPdtltd0DBcL6mNXlZm+hO6g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DVeIbSW8; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DVeIbSW8" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0384CC4CEED; Tue, 13 May 2025 16:31:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1747153912; bh=RcZzjfbX3puoN0l0SuXE8X3V8BVXm+FxxK0G7lqM4z8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DVeIbSW8TnkKNmuUizY8RG+l6Zonk3cN72jhwf9HqSXsapUKLlJ8AwnqBF8PdLSK2 RNVRdIhNl3pFvXk27FMO+0e9Q3tBJJ4bxDEO9MSVfUSADpgt7y00e79e8kATICEZye /+FTm7r3OuG7Ylo1gwiI5LYo8nAUGhGsnCYPDivZm8EJa3S2JiT3AjR4mvdhkFdsyP CvXB/RWkWiGC12sPNmN6vkSUw6Qy788qBtezjaBjmI5sZhP59ypUmaqe9RfNDuTMyh 7cIJ302UJ9x9j9JEkmZYDmBhA/C8dQFgNAfL0EOlMAlHbIa7sILgVtQBpGkbb1Hgqw gSNgGsHdXCnCw== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1uEsXm-00EaLc-0g; Tue, 13 May 2025 17:31:50 +0100 From: Marc Zyngier To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Thomas Gleixner , Lorenzo Pieralisi , Sascha Bischoff , Timothy Hayes Subject: [PATCH v2 1/5] genirq/msi: Add .msi_teardown() callback as the reverse of .msi_prepare() Date: Tue, 13 May 2025 17:31:40 +0100 Message-Id: <20250513163144.2215824-2-maz@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250513163144.2215824-1-maz@kernel.org> References: <20250513163144.2215824-1-maz@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, tglx@linutronix.de, lpieralisi@kernel.org, sascha.bischoff@arm.com, timothy.hayes@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Content-Type: text/plain; charset="utf-8" While the MSI ops do have a .msi_prepare() callback that is responsible for setting up the relevant (usually per-device) allocation, there is no callback reversing this setup. For this purpose, add .msi_teardown() callback. In order to avoid breaking the ITS driver that suffers from related issues, do not call the call back just yet. Signed-off-by: Marc Zyngier --- include/linux/msi.h | 10 ++++++++-- kernel/irq/msi.c | 8 ++++++++ 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/include/linux/msi.h b/include/linux/msi.h index 8c0ec9fc05a39..63c23003ec9b7 100644 --- a/include/linux/msi.h +++ b/include/linux/msi.h @@ -423,6 +423,7 @@ struct msi_domain_info; * @msi_init: Domain specific init function for MSI interrupts * @msi_free: Domain specific function to free a MSI interrupts * @msi_prepare: Prepare the allocation of the interrupts in the domain + * @msi_teardown: Reverse the effects of @msi_prepare * @prepare_desc: Optional function to prepare the allocated MSI descriptor * in the domain * @set_desc: Set the msi descriptor for an interrupt @@ -438,8 +439,9 @@ struct msi_domain_info; * @get_hwirq, @msi_init and @msi_free are callbacks used by the underlying * irqdomain. * - * @msi_check, @msi_prepare, @prepare_desc and @set_desc are callbacks use= d by the - * msi_domain_alloc/free_irqs*() variants. + * @msi_check, @msi_prepare, @msi_teardown, @prepare_desc and + * @set_desc are callbacks used by the msi_domain_alloc/free_irqs*() + * variants. * * @domain_alloc_irqs, @domain_free_irqs can be used to override the * default allocation/free functions (__msi_domain_alloc/free_irqs). This @@ -461,6 +463,8 @@ struct msi_domain_ops { int (*msi_prepare)(struct irq_domain *domain, struct device *dev, int nvec, msi_alloc_info_t *arg); + void (*msi_teardown)(struct irq_domain *domain, + msi_alloc_info_t *arg); void (*prepare_desc)(struct irq_domain *domain, msi_alloc_info_t *arg, struct msi_desc *desc); void (*set_desc)(msi_alloc_info_t *arg, @@ -489,6 +493,7 @@ struct msi_domain_ops { * @handler: Optional: associated interrupt flow handler * @handler_data: Optional: associated interrupt flow handler data * @handler_name: Optional: associated interrupt flow handler name + * @alloc_data: Optional: associated interrupt allocation data * @data: Optional: domain specific data */ struct msi_domain_info { @@ -501,6 +506,7 @@ struct msi_domain_info { irq_flow_handler_t handler; void *handler_data; const char *handler_name; + msi_alloc_info_t *alloc_data; void *data; }; =20 diff --git a/kernel/irq/msi.c b/kernel/irq/msi.c index 2b3ef2333dbc8..a3b34c3c599be 100644 --- a/kernel/irq/msi.c +++ b/kernel/irq/msi.c @@ -795,6 +795,11 @@ static int msi_domain_ops_prepare(struct irq_domain *d= omain, struct device *dev, return 0; } =20 +static void msi_domain_ops_teardown(struct irq_domain *domain, + msi_alloc_info_t *arg) +{ +} + static void msi_domain_ops_set_desc(msi_alloc_info_t *arg, struct msi_desc *desc) { @@ -820,6 +825,7 @@ static struct msi_domain_ops msi_domain_ops_default =3D= { .get_hwirq =3D msi_domain_ops_get_hwirq, .msi_init =3D msi_domain_ops_init, .msi_prepare =3D msi_domain_ops_prepare, + .msi_teardown =3D msi_domain_ops_teardown, .set_desc =3D msi_domain_ops_set_desc, }; =20 @@ -841,6 +847,8 @@ static void msi_domain_update_dom_ops(struct msi_domain= _info *info) ops->msi_init =3D msi_domain_ops_default.msi_init; if (ops->msi_prepare =3D=3D NULL) ops->msi_prepare =3D msi_domain_ops_default.msi_prepare; + if (ops->msi_teardown =3D=3D NULL) + ops->msi_teardown =3D msi_domain_ops_default.msi_teardown; if (ops->set_desc =3D=3D NULL) ops->set_desc =3D msi_domain_ops_default.set_desc; } --=20 2.39.2 From nobody Wed Dec 17 08:52:32 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B1C02BE0E6 for ; Tue, 13 May 2025 16:31:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747153912; cv=none; b=jxmo3QTKE3FLUTDPjAy+hdwIMrbL4EUr4nv76iSK4Mgp7NNBJ3SigNJVdcktCu26FLjg64qs3e25OJ8JyIszrzlqevRFYjymMR/GAY3GDqLRodiLQiiOuV1S00m9xJq0crFe3hsp/obOmp2nomhl/715SfV9SVpQ8DCEPvsq8IE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747153912; c=relaxed/simple; bh=EnmoQWrnaV1Hvaf1udlclzJvTLDhJHEEja5GIK61C/I=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=g7iWDpnWyx0mDHTyzDgdANEOO0RMwdm+NkUfiVyy/K7whbAFhlUBw8n6ZQ93FpCZ8qPOnrn/tEIGUj6hI9vIJcKpJwvxlH65dXDDDa6HJO/KC0er3pW3Z0DN7MOKB4KRusZ0nTtRS6gSRXtRf+veKuEOtJa7hw6On1qaQW8zI2Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=fFJsX6Td; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="fFJsX6Td" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0A6E2C4CEF1; Tue, 13 May 2025 16:31:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1747153912; bh=EnmoQWrnaV1Hvaf1udlclzJvTLDhJHEEja5GIK61C/I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fFJsX6TdE3Ukgp937aSEcaY4ZWy0xuVyFyqU4fCsA7l/5mg80Bcgltlp6srb+BIvc oLY4Nfhbph0bTJLDcF5Z0YHuARDg37/CAdJP4OVcQwX98d4dy2El7jfD2Tfaxlw7w/ 7MjFLMikWTE2e2W4uJ/CWkWoWfmXRVVfeL5Johj00F9an59BGiCYqYZVOL14m725JF NQqb7r4m6hPnICEGTgrCdTkECOM7NBiMKt/BSxyFRtsNyCQR7Q/tJCxjg65/2uHM/W MptUIh7cpv2DuWbDCSje9Jrm1b4L+/xrjHHTodi+upO20fzVpUY+GFLfycsZn8lkV4 q3AzgMyWNoRVQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1uEsXm-00EaLc-65; Tue, 13 May 2025 17:31:50 +0100 From: Marc Zyngier To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Thomas Gleixner , Lorenzo Pieralisi , Sascha Bischoff , Timothy Hayes Subject: [PATCH v2 2/5] irqchip/gic-v3-its: Implement .msi_teardown() callback Date: Tue, 13 May 2025 17:31:41 +0100 Message-Id: <20250513163144.2215824-3-maz@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250513163144.2215824-1-maz@kernel.org> References: <20250513163144.2215824-1-maz@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, tglx@linutronix.de, lpieralisi@kernel.org, sascha.bischoff@arm.com, timothy.hayes@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Content-Type: text/plain; charset="utf-8" The ITS driver currently nukes the structure representing an endpoint device translating via an ITS on freeing the last LPI allocated for it. That's an unfortunate state of affair, as it is pretty common for a driver to allocate a single MSI, do something clever, teardown this MSI, and reallocate a whole bunch of them. The nvme driver does exactly that, amongst others. What happens in that case is that the core code is buggy enough to issue another .msi_prepare() call, even if it shouldn't. This luckily cancels the above behaviour and hides the problem. In order to fix the core code, start by implementing the new .msi_teardown() callback. Nothing calls it yet, so a side effect is that the its_dev structure will not be freed and that the DID will stay mapped. Not a big deal, and this will be solved in following patches. Signed-off-by: Marc Zyngier --- drivers/irqchip/irq-gic-v3-its-msi-parent.c | 10 +++++ drivers/irqchip/irq-gic-v3-its.c | 46 +++++++++++---------- kernel/irq/msi.c | 3 +- 3 files changed, 36 insertions(+), 23 deletions(-) diff --git a/drivers/irqchip/irq-gic-v3-its-msi-parent.c b/drivers/irqchip/= irq-gic-v3-its-msi-parent.c index 68f9ba4085ce5..958736622fa57 100644 --- a/drivers/irqchip/irq-gic-v3-its-msi-parent.c +++ b/drivers/irqchip/irq-gic-v3-its-msi-parent.c @@ -167,6 +167,14 @@ static int its_pmsi_prepare(struct irq_domain *domain,= struct device *dev, dev, nvec, info); } =20 +static void its_msi_teardown(struct irq_domain *domain, msi_alloc_info_t *= info) +{ + struct msi_domain_info *msi_info; + + msi_info =3D msi_get_domain_info(domain->parent); + msi_info->ops->msi_teardown(domain->parent, info); +} + static bool its_init_dev_msi_info(struct device *dev, struct irq_domain *d= omain, struct irq_domain *real_parent, struct msi_domain_info *info) { @@ -190,6 +198,7 @@ static bool its_init_dev_msi_info(struct device *dev, s= truct irq_domain *domain, * %MSI_MAX_INDEX. */ info->ops->msi_prepare =3D its_pci_msi_prepare; + info->ops->msi_teardown =3D its_msi_teardown; break; case DOMAIN_BUS_DEVICE_MSI: case DOMAIN_BUS_WIRED_TO_MSI: @@ -198,6 +207,7 @@ static bool its_init_dev_msi_info(struct device *dev, s= truct irq_domain *domain, * size is also known at domain creation time. */ info->ops->msi_prepare =3D its_pmsi_prepare; + info->ops->msi_teardown =3D its_msi_teardown; break; default: /* Confused. How did the lib return true? */ diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-= its.c index fd6e7c170d37e..a77f11e23ad6c 100644 --- a/drivers/irqchip/irq-gic-v3-its.c +++ b/drivers/irqchip/irq-gic-v3-its.c @@ -3620,8 +3620,33 @@ static int its_msi_prepare(struct irq_domain *domain= , struct device *dev, return err; } =20 +static void its_msi_teardown(struct irq_domain *domain, msi_alloc_info_t *= info) +{ + struct its_device *its_dev =3D info->scratchpad[0].ptr; + + guard(mutex)(&its_dev->its->dev_alloc_lock); + + /* If the device is shared, keep everything around */ + if (its_dev->shared) + return; + + /* LPIs should have been already unmapped at this stage */ + if (WARN_ON_ONCE(!bitmap_empty(its_dev->event_map.lpi_map, + its_dev->event_map.nr_lpis))) + return; + + its_lpi_free(its_dev->event_map.lpi_map, + its_dev->event_map.lpi_base, + its_dev->event_map.nr_lpis); + + /* Unmap device/itt, and get rid of the tracking */ + its_send_mapd(its_dev, 0); + its_free_device(its_dev); +} + static struct msi_domain_ops its_msi_domain_ops =3D { .msi_prepare =3D its_msi_prepare, + .msi_teardown =3D its_msi_teardown, }; =20 static int its_irq_gic_domain_alloc(struct irq_domain *domain, @@ -3722,7 +3747,6 @@ static void its_irq_domain_free(struct irq_domain *do= main, unsigned int virq, { struct irq_data *d =3D irq_domain_get_irq_data(domain, virq); struct its_device *its_dev =3D irq_data_get_irq_chip_data(d); - struct its_node *its =3D its_dev->its; int i; =20 bitmap_release_region(its_dev->event_map.lpi_map, @@ -3736,26 +3760,6 @@ static void its_irq_domain_free(struct irq_domain *d= omain, unsigned int virq, irq_domain_reset_irq_data(data); } =20 - mutex_lock(&its->dev_alloc_lock); - - /* - * If all interrupts have been freed, start mopping the - * floor. This is conditioned on the device not being shared. - */ - if (!its_dev->shared && - bitmap_empty(its_dev->event_map.lpi_map, - its_dev->event_map.nr_lpis)) { - its_lpi_free(its_dev->event_map.lpi_map, - its_dev->event_map.lpi_base, - its_dev->event_map.nr_lpis); - - /* Unmap device/itt */ - its_send_mapd(its_dev, 0); - its_free_device(its_dev); - } - - mutex_unlock(&its->dev_alloc_lock); - irq_domain_free_irqs_parent(domain, virq, nr_irqs); } =20 diff --git a/kernel/irq/msi.c b/kernel/irq/msi.c index a3b34c3c599be..31378a2535fb9 100644 --- a/kernel/irq/msi.c +++ b/kernel/irq/msi.c @@ -795,8 +795,7 @@ static int msi_domain_ops_prepare(struct irq_domain *do= main, struct device *dev, return 0; } =20 -static void msi_domain_ops_teardown(struct irq_domain *domain, - msi_alloc_info_t *arg) +static void msi_domain_ops_teardown(struct irq_domain *domain, msi_alloc_i= nfo_t *arg) { } =20 --=20 2.39.2 From nobody Wed Dec 17 08:52:32 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A37922BE7BC for ; Tue, 13 May 2025 16:31:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747153912; cv=none; b=AqUvmuNVTBP92raR883UPlnBsndyK9Lvuslhg2YcyM9NQUoyxLhQdip5lVyHx/1xdgXc5mX0tKAoZR4M70P9LK+KSK6kzdETkWvRL3gotSQnCsGnuoNfNLDJhgQ6R+tF/0MvofQf3Fvxf+S0FsqD6zAlzGpGASYXOQSb/vh+lB0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747153912; c=relaxed/simple; bh=WVNzJ1QslGOZvrM8+INHuUbrNxIwVxMRLbrLof8UGE0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=X2iYytWlZbRtbRk/9cMikgOBeLmo0IMzGD2mCN5977wPBTT/q3uoYo56nduBMynJ+n5floRij24LVqQmEs+ud9rbOba8VpyPpbqq76Z/iMwJiISJxO2GNN4hKlhSaw6n9FdoVo5+0UV7uNy+SxnrhxcNygiAc7iGjAcuOI2D/9g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=h9kMzXXD; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="h9kMzXXD" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 628CAC4CEF4; Tue, 13 May 2025 16:31:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1747153912; bh=WVNzJ1QslGOZvrM8+INHuUbrNxIwVxMRLbrLof8UGE0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=h9kMzXXDBuRa3vfoSYnrhhdB5pvIx2WCKZ71M0Pmc2Lept3bgCBcto6sgWuvVOqro bBcP5w5R2OtBt//zq3TGAw24/hnSQcAUBq0MDr0LiyVbOtAR02KfPej14Aey3jH/4t hC2A/uBZasZ8xAw7a09lCVCzznRqJ9nEgWmNctrkza/7yBewxiIDQhxWfqP9neKFVA NLY8E9bYBe0hrXcqDdiTDMJ3DHOxPRUWSffMDrUoQcfklHtrg3+xkGh7jhR7m01zSj ZvKOIaFI4A8l8H5fzHJi9OiKWFr5ra38eRKnk/xoZkkT1Bk8DSc3ar50F4KdFRE0Kc mX5Jj+HKhMtOw== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1uEsXm-00EaLc-Ba; Tue, 13 May 2025 17:31:50 +0100 From: Marc Zyngier To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Thomas Gleixner , Lorenzo Pieralisi , Sascha Bischoff , Timothy Hayes Subject: [PATCH v2 3/5] genirq/msi: Move prepare() call to per-device allocation Date: Tue, 13 May 2025 17:31:42 +0100 Message-Id: <20250513163144.2215824-4-maz@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250513163144.2215824-1-maz@kernel.org> References: <20250513163144.2215824-1-maz@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, tglx@linutronix.de, lpieralisi@kernel.org, sascha.bischoff@arm.com, timothy.hayes@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Content-Type: text/plain; charset="utf-8" The current device MSI infrastructure is subtly broken, as it will issue an .msi_prepare() callback into the MSI controller driver every time it needs to allocate an MSI. That's pretty wrong, as the contract (or unwarranted assumption, depending who you ask) between the MSI controller and the core code is that .msi_prepare() is called exactly once per device. This leads to some subtle breakage in said MSI controller drivers, as it gives the impression that there are multiple endpoints sharing a bus identifier (RID in PCI parlance, DID for GICv3+). It implies that whatever allocation the ITS driver (for example) has done on behalf of these devices cannot be undone, as there is no way to track the shared state. This is particularly bad for wire-MSI devices, for which .msi_prepare() is called for. each. input. line. To address this issue, move the call to .msi_prepare() to take place at the point of irq domain allocation, which is the only place that makes sense. The msi_alloc_info_t structure is made part of the msi_domain_template, so that its life-cycle is that of the domain as well. Finally, the msi_info::alloc_data field is made to point at this allocation tracking structure, ensuring that it is carried around the block. This is all pretty straightforward, except for the non-device-MSI leftovers, which still have to call .msi_prepare() at the old spot. One day... Signed-off-by: Marc Zyngier Tested-by: Zenghui Yu --- include/linux/msi.h | 2 ++ kernel/irq/msi.c | 35 +++++++++++++++++++++++++++++++---- 2 files changed, 33 insertions(+), 4 deletions(-) diff --git a/include/linux/msi.h b/include/linux/msi.h index 63c23003ec9b7..ba1c77a829a1c 100644 --- a/include/linux/msi.h +++ b/include/linux/msi.h @@ -516,12 +516,14 @@ struct msi_domain_info { * @chip: Interrupt chip for this domain * @ops: MSI domain ops * @info: MSI domain info data + * @alloc_info: MSI domain allocation data (arch specific) */ struct msi_domain_template { char name[48]; struct irq_chip chip; struct msi_domain_ops ops; struct msi_domain_info info; + msi_alloc_info_t alloc_info; }; =20 /* diff --git a/kernel/irq/msi.c b/kernel/irq/msi.c index 31378a2535fb9..07eb857efd15e 100644 --- a/kernel/irq/msi.c +++ b/kernel/irq/msi.c @@ -59,7 +59,8 @@ struct msi_ctrl { static void msi_domain_free_locked(struct device *dev, struct msi_ctrl *ct= rl); static unsigned int msi_domain_get_hwsize(struct device *dev, unsigned int= domid); static inline int msi_sysfs_create_group(struct device *dev); - +static int msi_domain_prepare_irqs(struct irq_domain *domain, struct devic= e *dev, + int nvec, msi_alloc_info_t *arg); =20 /** * msi_alloc_desc - Allocate an initialized msi_desc @@ -1023,6 +1024,7 @@ bool msi_create_device_irq_domain(struct device *dev,= unsigned int domid, bundle->info.ops =3D &bundle->ops; bundle->info.data =3D domain_data; bundle->info.chip_data =3D chip_data; + bundle->info.alloc_data =3D &bundle->alloc_info; =20 pops =3D parent->msi_parent_ops; snprintf(bundle->name, sizeof(bundle->name), "%s%s-%s", @@ -1061,11 +1063,18 @@ bool msi_create_device_irq_domain(struct device *de= v, unsigned int domid, if (!domain) return false; =20 + domain->dev =3D dev; + dev->msi.data->__domains[domid].domain =3D domain; + + if (msi_domain_prepare_irqs(domain, dev, hwsize, &bundle->alloc_info)) { + dev->msi.data->__domains[domid].domain =3D NULL; + irq_domain_remove(domain); + return false; + } + /* @bundle and @fwnode_alloced are now in use. Prevent cleanup */ retain_and_null_ptr(bundle); retain_and_null_ptr(fwnode_alloced); - domain->dev =3D dev; - dev->msi.data->__domains[domid].domain =3D domain; return true; } =20 @@ -1232,6 +1241,24 @@ static int msi_init_virq(struct irq_domain *domain, = int virq, unsigned int vflag return 0; } =20 +static int populate_alloc_info(struct irq_domain *domain, struct device *d= ev, + unsigned int nirqs, msi_alloc_info_t *arg) +{ + struct msi_domain_info *info =3D domain->host_data; + + /* + * If the caller has provided a template alloc info, use that. Once + * all users of msi_create_irq_domain() have been eliminated, this + * should be the only source of allocation information, and the + * prepare call below should be finally removed. + */ + if (!info->alloc_data) + return msi_domain_prepare_irqs(domain, dev, nirqs, arg); + + *arg =3D *info->alloc_data; + return 0; +} + static int __msi_domain_alloc_irqs(struct device *dev, struct irq_domain *= domain, struct msi_ctrl *ctrl) { @@ -1244,7 +1271,7 @@ static int __msi_domain_alloc_irqs(struct device *dev= , struct irq_domain *domain unsigned long idx; int i, ret, virq; =20 - ret =3D msi_domain_prepare_irqs(domain, dev, ctrl->nirqs, &arg); + ret =3D populate_alloc_info(domain, dev, ctrl->nirqs, &arg); if (ret) return ret; =20 --=20 2.39.2 From nobody Wed Dec 17 08:52:32 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B85F11A38E1 for ; Tue, 13 May 2025 16:31:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747153912; cv=none; b=aCP15tlvEMn/uCjUkEAbZdjbP+MO/OaM56k5dN62K5efaLvqBzQUQSdpi+D9W4KwHj4s00WakOkxfuEotVQsHLeOEwHcq3CSM87kljg9I9gTf/Jp/R7yKSQEImQeAoEqK3mEOA0PA2W248+3QHtdi/aO6GQ2ruJixHItd/22d34= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747153912; c=relaxed/simple; bh=5YnLuW4gz7cYBwmJ4dAR13BTuttnlScp3AJ2nVWpIOU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OTBJglVWsBBnOMyhvtTMzVSEGDIaoXVd5b9FzJmTilMmorPv8b5CW8UH8DIKQMEn/FnGB0G1Uz358gJQmCX4CbHj7Dt3vWoiNKREBq3HtOnxM94d3DSCkL14lWacQo4Nl1YrwFPb5RYBYfmt35NqiPAlodnB9TpJLXtP+fpoT1c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=md5u7IbZ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="md5u7IbZ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6B35BC4CEF5; Tue, 13 May 2025 16:31:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1747153912; bh=5YnLuW4gz7cYBwmJ4dAR13BTuttnlScp3AJ2nVWpIOU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=md5u7IbZqXvOU/Kx+Gy17zrOrU35ta7gml8kWoB5xRMwCAMx8uj3mYTrLLQMwNTMg rlADE+MFc3dfObEs2QiRLKgr/r7Jr6GxlS24GMZgAOU1yyC+/z5dznGRGJ9LOFwZp9 3wTisr7Tr4EZQPyK+z3gGvqc5gYz6fX9et4LbVAMQe9y377hvtUKswz6eAKFVeXbFe ir1oPkZhAcQ9EWuxL37my5rFeXWpfJLL4Dl6zZhj9vleNW0XowCRLv1L6pyLf16rCD 7HH27Wyy6xh3+U51T3wqgavV1GAYqw0+csEcxjV/vkhhcIF51Y+0raJvGNCHp9M7IZ /D47zCDtyk14g== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1uEsXm-00EaLc-HK; Tue, 13 May 2025 17:31:50 +0100 From: Marc Zyngier To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Thomas Gleixner , Lorenzo Pieralisi , Sascha Bischoff , Timothy Hayes Subject: [PATCH v2 4/5] genirq/msi: Engage the .msi_teardown() callback on domain removal Date: Tue, 13 May 2025 17:31:43 +0100 Message-Id: <20250513163144.2215824-5-maz@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250513163144.2215824-1-maz@kernel.org> References: <20250513163144.2215824-1-maz@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, tglx@linutronix.de, lpieralisi@kernel.org, sascha.bischoff@arm.com, timothy.hayes@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Content-Type: text/plain; charset="utf-8" Kindly inform the MSI driver that we are tearing down the domain, providing the allocation context previously populated on domain creation. Signed-off-by: Marc Zyngier --- kernel/irq/msi.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/irq/msi.c b/kernel/irq/msi.c index 07eb857efd15e..8f3d9f5ff1eb4 100644 --- a/kernel/irq/msi.c +++ b/kernel/irq/msi.c @@ -1096,6 +1096,9 @@ void msi_remove_device_irq_domain(struct device *dev,= unsigned int domid) =20 dev->msi.data->__domains[domid].domain =3D NULL; info =3D domain->host_data; + + info->ops->msi_teardown(domain, info->alloc_data); + if (irq_domain_is_msi_device(domain)) fwnode =3D domain->fwnode; irq_domain_remove(domain); --=20 2.39.2 From nobody Wed Dec 17 08:52:32 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F04282BEC5A for ; Tue, 13 May 2025 16:31:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747153913; cv=none; b=RUl42RqdrH9QPWrnAAlE+Mmg/OJGbPBm6xxGdfgi/Yg6DGpku7Td5BHJ9uHaC/GTJeB3aEdayrXHDQL8ErBgMnMyY0GnmVjFX3Wj7DZffpaeIkEUohTHvTzQIQp0KzyaELR2Yyzel/LoRpr0NJkV53Finv05trBIaX9fZm9zkqM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747153913; c=relaxed/simple; bh=17/pUMQ3E5yHwJFwSxMS9cCKFYZ8IU9JtUyr4n/x9f4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Tc37mpFD689NGxYtaWyOuGSawZxnCUwjpVR/UiYgy0l3j8aMOG3mt/DjumX5AyF/VdWyCHl2XPrjIf70AO7q/UxqSq34Ff8MYkJ9ydaOLXg6qPYEwX2uIDZZ8NX92beGi/jA5XSNZO74iWb5gKq+kr2DiB1wDXC2Ox1FTwPnS/c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=gKIAKPyZ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="gKIAKPyZ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 718F3C4CEEF; Tue, 13 May 2025 16:31:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1747153912; bh=17/pUMQ3E5yHwJFwSxMS9cCKFYZ8IU9JtUyr4n/x9f4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gKIAKPyZNE8558Z0epfjFt5ZcJ86ATAzkd0LRVgH/k9K3aQh9Co/nhlK4dMRY6vUn AvZhb5tu72Wio6HMJqBQtu/VVP3dVo9ZFVPXtExMv8Bjem4k/og61Pf/DNGsMuYJ+1 8yPxs5+mZGuqu8p3tbL/G9VIcoXr/+xpCPsoXV6D2kLg2LzROHVoSVdWjnYlkFNhEa 0c3Czuh+7tsWlcmW0UI0x+MU880Usim2vVLQzpMQnzDB4Lz2Qi6Oot0BOVyQo+uoe6 OSWbFcu3WHMCW35JdgExQLCD8jfgNTsAzMd/goD7ku2VilZr3a6FErGGrJYvtWMJHa zBFeoiRdtKvlA== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1uEsXm-00EaLc-Mi; Tue, 13 May 2025 17:31:50 +0100 From: Marc Zyngier To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Thomas Gleixner , Lorenzo Pieralisi , Sascha Bischoff , Timothy Hayes Subject: [PATCH v2 5/5] irqchip/gic-v3-its: Use allocation size from the prepare call Date: Tue, 13 May 2025 17:31:44 +0100 Message-Id: <20250513163144.2215824-6-maz@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250513163144.2215824-1-maz@kernel.org> References: <20250513163144.2215824-1-maz@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, tglx@linutronix.de, lpieralisi@kernel.org, sascha.bischoff@arm.com, timothy.hayes@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Content-Type: text/plain; charset="utf-8" Now that .msi_prepare() gets called at the right time and not with semi-random parameters, remove the ugly hack that tried to fix up the number of allocated vectors. It is now correct by construction. Signed-off-by: Marc Zyngier --- drivers/irqchip/irq-gic-v3-its-msi-parent.c | 19 ------------------- 1 file changed, 19 deletions(-) diff --git a/drivers/irqchip/irq-gic-v3-its-msi-parent.c b/drivers/irqchip/= irq-gic-v3-its-msi-parent.c index 958736622fa57..6a5f64f120d4a 100644 --- a/drivers/irqchip/irq-gic-v3-its-msi-parent.c +++ b/drivers/irqchip/irq-gic-v3-its-msi-parent.c @@ -67,17 +67,6 @@ static int its_pci_msi_prepare(struct irq_domain *domain= , struct device *dev, /* ITS specific DeviceID, as the core ITS ignores dev. */ info->scratchpad[0].ul =3D pci_msi_domain_get_msi_rid(domain->parent, pde= v); =20 - /* - * @domain->msi_domain_info->hwsize contains the size of the - * MSI[-X] domain, but vector allocation happens one by one. This - * needs some thought when MSI comes into play as the size of MSI - * might be unknown at domain creation time and therefore set to - * MSI_MAX_INDEX. - */ - msi_info =3D msi_get_domain_info(domain); - if (msi_info->hwsize > nvec) - nvec =3D msi_info->hwsize; - /* * Always allocate a power of 2, and special case device 0 for * broken systems where the DevID is not wired (and all devices @@ -151,14 +140,6 @@ static int its_pmsi_prepare(struct irq_domain *domain,= struct device *dev, /* ITS specific DeviceID, as the core ITS ignores dev. */ info->scratchpad[0].ul =3D dev_id; =20 - /* - * @domain->msi_domain_info->hwsize contains the size of the device - * domain, but vector allocation happens one by one. - */ - msi_info =3D msi_get_domain_info(domain); - if (msi_info->hwsize > nvec) - nvec =3D msi_info->hwsize; - /* Allocate at least 32 MSIs, and always as a power of 2 */ nvec =3D max_t(int, 32, roundup_pow_of_two(nvec)); =20 --=20 2.39.2