From nobody Mon Jun 22 15:42:51 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D967C433EF for ; Mon, 21 Mar 2022 16:52:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351440AbiCUQx0 (ORCPT ); Mon, 21 Mar 2022 12:53:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51844 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234615AbiCUQxX (ORCPT ); Mon, 21 Mar 2022 12:53:23 -0400 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9ECCB174B8B for ; Mon, 21 Mar 2022 09:51:57 -0700 (PDT) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id 04C2D32004CE; Mon, 21 Mar 2022 12:51:55 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Mon, 21 Mar 2022 12:51:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=svenpeter.dev; h=cc:cc:content-transfer-encoding:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm1; bh=9y3qY8wrWheTjT yFytvtK0+0Iv19bnWaSl3QfjKyhtE=; b=ISBoBUMMt3OzLno+nSYoFli65K1ZSz AbdaLKHSwbymZunFKAyQYGiQBTTbK0RTc7lxD7MEheSKcLRR51CX5p38FmvlJ+cj hEj3/K/psPOfJ9IyFrIb8Bye44kElVoFb7I4nAP8k8/ESli4W6O4JKtSkjNan/lw TVndVZXWyhZtnYMlOopvgqDBo316+XeE1ttHovZmT0TmBViK9XHx0aK8TV+Rb86g Dbf5LYKwwQrpNsjcOYUclQrrJV+KC2RFe6RQlvdPPpK9EFQ66DAhpniEQ9Zz+d83 KeFx1FTUDzISoTOrthEYZzlSBIDc731uLWLrOS3g3QfJdEzjN1yKCohw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=9y3qY8 wrWheTjTyFytvtK0+0Iv19bnWaSl3QfjKyhtE=; b=X2haQ2aO9+NkS8qcP9sWqk FdHkclNHA73VxP5HDKLh0XdAICTW0TqDAC4+sfL2OhTmx8fEnXEZrvHaxgTy368P LYO8a3+FPqg+5hbWZ7tITE7tMlXxFsVy6eHgObOJOr0ZlIVMbYRaZU2sDKDBfNGu 2agmDFEeKIPLju6USst975OCpYi5dIWr1RreUkpasQMiqgPb1JciQhty2hEVGs0z 759oUJ+kBEnJWKqZ57QP98uaNJX3slSS6PCYnZDfbJ4pEvCFqpJe0F4CivIjQDQ6 uJre0OpG+8bvUFDesCZcRxFAGnNw+Q+9Ii85Jw1hugPww34YJH3+BeABL95HAO7A == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrudegfedgleehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefuvhgvnhcu rfgvthgvrhcuoehsvhgvnhesshhvvghnphgvthgvrhdruggvvheqnecuggftrfgrthhtvg hrnhephfefffejvdekleeitdffiefhhfffveffieejteefhedutdfgffeftdejhedtteeh necuffhomhgrihhnpeguvghvihgtvghtrhgvvgdrohhrghenucevlhhushhtvghrufhiii gvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsvhgvnhesshhvvghnphgvthgvrhdr uggvvh X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 21 Mar 2022 12:51:53 -0400 (EDT) From: Sven Peter To: Rob Herring Cc: Sven Peter , Hector Martin , Alyssa Rosenzweig , Arnd Bergmann , Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Marc Zyngier , devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org Subject: [PATCH 1/9] dt-bindings: soc: apple: Add Apple SART Date: Mon, 21 Mar 2022 17:50:41 +0100 Message-Id: <20220321165049.35985-2-sven@svenpeter.dev> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20220321165049.35985-1-sven@svenpeter.dev> References: <20220321165049.35985-1-sven@svenpeter.dev> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Apple SoCs such as the M1 come with a simple DMA address filter called SART. Unlike a real IOMMU no pagetables can be configured but instead DMA transactions can be allowed for up to 16 paddr regions. Signed-off-by: Sven Peter --- .../bindings/soc/apple/apple,sart.yaml | 52 +++++++++++++++++++ MAINTAINERS | 1 + 2 files changed, 53 insertions(+) create mode 100644 Documentation/devicetree/bindings/soc/apple/apple,sart.= yaml diff --git a/Documentation/devicetree/bindings/soc/apple/apple,sart.yaml b/= Documentation/devicetree/bindings/soc/apple/apple,sart.yaml new file mode 100644 index 000000000000..d8177b3a3fba --- /dev/null +++ b/Documentation/devicetree/bindings/soc/apple/apple,sart.yaml @@ -0,0 +1,52 @@ +# SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/soc/apple/apple,sart.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Apple SART DMA address filter + +maintainers: + - Sven Peter + +description: + Apple SART is a simple address filter for DMA transactions. Regions of + physical memory must be added to the SART's allow list before any + DMA can target these. Unlike a proper IOMMU no remapping can be done and + special support in the consumer driver is required since not all DMA + transactions of a single device are subject to SART filtering. + + SART1 has first been used since at least the A11 (iPhone 8 and iPhone X) + and allows 36 bit of physical address space and filter entries with sizes + up to 24 bit. + + SART2, first seen in A14 and M1, allows 36 bit of physical address space + and filter entry size up to 36 bit. + + SART3, first seen in M1 Pro/Max, extends both the address space and filt= er + entry size to 42 bit. + +properties: + compatible: + enum: + - apple,t6000-sart + - apple,t8103-sart + + reg: + maxItems: 1 + + power-domains: + maxItems: 1 + +required: + - compatible + - reg + +additionalProperties: false + +examples: + - | + sart@7bc50000 { + compatible =3D "apple,t8103-sart"; + reg =3D <0x7bc50000 0x4000>; + }; diff --git a/MAINTAINERS b/MAINTAINERS index cd0f68d4a34a..027c3b4ad61c 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1774,6 +1774,7 @@ F: Documentation/devicetree/bindings/mailbox/apple,ma= ilbox.yaml F: Documentation/devicetree/bindings/pci/apple,pcie.yaml F: Documentation/devicetree/bindings/pinctrl/apple,pinctrl.yaml F: Documentation/devicetree/bindings/power/apple* +F: Documentation/devicetree/bindings/soc/apple/* F: Documentation/devicetree/bindings/watchdog/apple,wdt.yaml F: arch/arm64/boot/dts/apple/ F: drivers/i2c/busses/i2c-pasemi-core.c --=20 2.25.1 From nobody Mon Jun 22 15:42:51 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50E3AC433F5 for ; Mon, 21 Mar 2022 16:52:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351455AbiCUQxb (ORCPT ); Mon, 21 Mar 2022 12:53:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52050 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1351441AbiCUQx1 (ORCPT ); Mon, 21 Mar 2022 12:53:27 -0400 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA63B174BA0 for ; Mon, 21 Mar 2022 09:52:01 -0700 (PDT) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id 22F833200EAD; Mon, 21 Mar 2022 12:52:00 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Mon, 21 Mar 2022 12:52:01 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=svenpeter.dev; h=cc:cc:content-transfer-encoding:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm1; bh=/pY3yz47Nuid96 s8eNEwkYTyhd2x8Impr+YNOi7OK8g=; b=ESpAxG2myY5Dbc+afq6j+05WbyAt+0 MrZ6+THFre7I2tBMBV3Nvgr6HhvrTaztmtsF8le7Gdv8Bp2AIqhXQroc/o0dKfp4 6UCv20df6zaE9Obt9QQKgh/zubYJnmx4dA9UWgiK0n6VX/vjAy+ni2+ke7/wNhdT wlfIxkfkFh7+RstRdhJyluZTX/BPGuVGvwZA7vxAUGvye2B6Nu7SAHnSWSRZkb8c Jdtn7mJBJkdsYEWmRpgm4i6/y7gzSFTXE3YsmlvmkEkZ6vuFEf/VdpFkPEL+HcrA OrJV3krJmZkWpJI1Gtm6D0vswGMqWY37pH+reKWz3CaITZ9t8UaGI4lw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=/pY3yz 47Nuid96s8eNEwkYTyhd2x8Impr+YNOi7OK8g=; b=D4JG78bsRw36sMeSUQHgoz VsjBjrGgwLeBSHk9onX1t6jc9poa8mnjFEW0PaOS/9V9nF91dPanB32WJwt/NuAJ 2/QJ78Ggiyd4Ng/jWIeoMPWCyLNA+UaeUzqyLi09/ahrjXww+4L9WYgm5H0REVAP hKhjkSoWc4RZwkuae+g1WL0RTsPcDuubO7GH6LrLWPsqEp8LhGSNW5lkjn4VBi9V ckebce0ViNaTMboZb+mhrTPLUF+VDHW/oXPC4SkMOD+SDHu/rrpdKQhQI7y4Wv8K rsv/fqDIMoh6RPVYE5M8W5Xg1dcgMvzAdSdGtwdZOeFM+xVL/eLY9/6hzjrivHYA == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrudegfedgleehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefuvhgvnhcu rfgvthgvrhcuoehsvhgvnhesshhvvghnphgvthgvrhdruggvvheqnecuggftrfgrthhtvg hrnhephfefffejvdekleeitdffiefhhfffveffieejteefhedutdfgffeftdejhedtteeh necuffhomhgrihhnpeguvghvihgtvghtrhgvvgdrohhrghenucevlhhushhtvghrufhiii gvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsvhgvnhesshhvvghnphgvthgvrhdr uggvvh X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 21 Mar 2022 12:51:57 -0400 (EDT) From: Sven Peter To: Rob Herring Cc: Sven Peter , Hector Martin , Alyssa Rosenzweig , Arnd Bergmann , Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Marc Zyngier , devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org Subject: [PATCH 2/9] dt-bindings: soc: apple: Add ANS NVMe Date: Mon, 21 Mar 2022 17:50:42 +0100 Message-Id: <20220321165049.35985-3-sven@svenpeter.dev> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20220321165049.35985-1-sven@svenpeter.dev> References: <20220321165049.35985-1-sven@svenpeter.dev> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Apple SoCs such as the M1 come with an embedded NVMe coprocessor called ANS2. Signed-off-by: Sven Peter --- .../bindings/soc/apple/apple,nvme-ans.yaml | 75 +++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 Documentation/devicetree/bindings/soc/apple/apple,nvme-= ans.yaml diff --git a/Documentation/devicetree/bindings/soc/apple/apple,nvme-ans.yam= l b/Documentation/devicetree/bindings/soc/apple/apple,nvme-ans.yaml new file mode 100644 index 000000000000..e1f4c1c572aa --- /dev/null +++ b/Documentation/devicetree/bindings/soc/apple/apple,nvme-ans.yaml @@ -0,0 +1,75 @@ +# SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/soc/apple/apple,nvme-ans.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Apple ANS NVM Express host controller + +maintainers: + - Sven Peter + +properties: + compatible: + items: + - enum: + - apple,t8103-nvme-ans2 + - apple,t6000-nvme-ans2 + - const: apple,nvme-ans2 + + reg: + items: + - description: NVMe and NVMMU registers + - description: ANS2 co-processor control registers + + reg-names: + items: + - const: nvme + - const: ans + + resets: + maxItems: 1 + + power-domains: true + + mboxes: + maxItems: 1 + description: Mailbox of the ANS2 co-processor + + interrupts: + maxItems: 1 + + apple,sart: + maxItems: 1 + $ref: /schemas/types.yaml#/definitions/phandle + description: | + Reference to the SART address filter. + + The SART address filter is documented in apple,sart.yaml. + +required: + - compatible + - reg + - reg-names + - resets + - mboxes + - interrupts + - apple,sart + +additionalProperties: false + +examples: + - | + #include + #include + + nvme@7bcc0000 { + compatible =3D "apple,t8103-nvme-ans2", "apple,nvme-ans2"; + reg =3D <0x7bcc0000 0x40000>, <0x77400000 0x4000>; + reg-names =3D "nvme", "ans"; + interrupts =3D ; + mboxes =3D <&ans>; + apple,sart =3D <&sart>; + power-domains =3D <&ps_ans2>; + resets =3D <&ps_ans2>; + }; --=20 2.25.1 From nobody Mon Jun 22 15:42:51 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76901C433EF for ; Mon, 21 Mar 2022 16:52:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351474AbiCUQxg (ORCPT ); Mon, 21 Mar 2022 12:53:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52184 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1351448AbiCUQxa (ORCPT ); Mon, 21 Mar 2022 12:53:30 -0400 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF352174BA8 for ; Mon, 21 Mar 2022 09:52:05 -0700 (PDT) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id 2781B3201DDB; Mon, 21 Mar 2022 12:52:04 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Mon, 21 Mar 2022 12:52:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=svenpeter.dev; h=cc:cc:content-transfer-encoding:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm1; bh=Bi1/K+ioRkAobi uxJd52yJR2Pj8oZxB/CV+UGWW2RrE=; b=Ow/vF7bGd5mVcJpF0oAf+PfH97F+z1 QwRM+o61GqMDteHOun8EzV80CcuuHXOuIj6ogQx4Aunri+CA52KRQW2SutwJ6Yos 5y8q4xBkfuCvREhw1ydrTQ4OxhmW0KYUms8gX8U27V0ALlR39zfLJtMsuXd8jWF5 IMmFwwOJeZ9a8YbLrOJ5l1CWkVidp5TEh6Fwy2tEH1CCWuD90ai9qyetl8qJSo3j ZRQKTWQaDaUVwiUTZg6gnWESaYuijdqW02RHACEVvQRNfhW69Jj4bk7Hf7WJrD9X IRJS6v558WbLL2eE2Y6d3esR8z/Uz648QB0ll6oW5VNFupcQifJ0Zmqw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=Bi1/K+ ioRkAobiuxJd52yJR2Pj8oZxB/CV+UGWW2RrE=; b=ZnfrVXH5KtTHqfz39jXMpW lMWw6tWSctYFWbvrJo2h2FaN35zoQ0ndoNqmJkdojyPQScwX48WLxNl1ZLBGgafs qpXVvWxrfd5rSUBmtxgQ/4Fij2E38I0lVbGZniwMv1WrxbeaukKuqRPzMhWUkwP2 Apk67LVkZXAJazuXavMaZ+w27ntL+5bRcEb0ENS9z1nVr0Q48NUDlTnIQENqIi75 hrESvgT1euHD18oUII9iauolhBG0XyP42ZwcmDh3bCUjhQGJT2kHl90NPhdZDUoi xxYI6cnzURDZio4+dD1FHaK+CfexEAk7gzcFU+67hOc+hnPF0YEbV2qPhcusv2AA == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrudegfedgleehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne hmihhsshhinhhgucfvqfcufhhivghlugculdeftddmnegoteeftdduqddtudculdduhedm necujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefuvhgvnh curfgvthgvrhcuoehsvhgvnhesshhvvghnphgvthgvrhdruggvvheqnecuggftrfgrthht vghrnheptedvkeetleeuffffhfekteetffeggffgveehieelueefvddtueffveevlefhfe ejnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepshhv vghnsehsvhgvnhhpvghtvghrrdguvghv X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 21 Mar 2022 12:52:01 -0400 (EDT) From: Sven Peter Cc: Sven Peter , Hector Martin , Alyssa Rosenzweig , Rob Herring , Arnd Bergmann , Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Marc Zyngier , devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org Subject: [PATCH 3/9] soc: apple: Always include Makefile Date: Mon, 21 Mar 2022 17:50:43 +0100 Message-Id: <20220321165049.35985-4-sven@svenpeter.dev> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20220321165049.35985-1-sven@svenpeter.dev> References: <20220321165049.35985-1-sven@svenpeter.dev> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable To: unlisted-recipients:; (no To-header on input) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" We want to allow the code inside drivers/soc/apple to be compiled with COMPILE_TEST but this will currently result in linking errors because ARCH_APPLE is not set and make will never recurse into drivers/soc/apple. Let's just unconditionally recurse into apple/ since all drivers in there are guarded by config options anyways. Signed-off-by: Sven Peter --- drivers/soc/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/soc/Makefile b/drivers/soc/Makefile index adb30c2d4fea..608f8ce8b600 100644 --- a/drivers/soc/Makefile +++ b/drivers/soc/Makefile @@ -4,7 +4,7 @@ # =20 obj-$(CONFIG_ARCH_ACTIONS) +=3D actions/ -obj-$(CONFIG_ARCH_APPLE) +=3D apple/ +obj-y +=3D apple/ obj-y +=3D aspeed/ obj-$(CONFIG_ARCH_AT91) +=3D atmel/ obj-y +=3D bcm/ --=20 2.25.1 From nobody Mon Jun 22 15:42:51 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8BE7C433EF for ; Mon, 21 Mar 2022 16:52:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351463AbiCUQxk (ORCPT ); Mon, 21 Mar 2022 12:53:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52372 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1351472AbiCUQxg (ORCPT ); Mon, 21 Mar 2022 12:53:36 -0400 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B0E4C17708B for ; Mon, 21 Mar 2022 09:52:09 -0700 (PDT) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id 2696832004ED; Mon, 21 Mar 2022 12:52:08 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Mon, 21 Mar 2022 12:52:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=svenpeter.dev; h=cc:cc:content-transfer-encoding:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm1; bh=rr03hFF2bNF2sz Mfr4QL+Usn5E1acu59izxNNk0mufA=; b=fGhrsuc/rO74iNobTJtrnOJTvJW6eO bsrdOUyBfe/pc7HzfL9tWgdbZz5aP00ujEktoe69QwdTSoI5steIUpe9vKf4x+BB cNxPfDRXnRnCXTKUBbC8N13bLyBkyAynvKjEQMR5ugEp0+d69lrkC+Xx4IsTxk1s srcbkoTKl4txzSUq4FUF/TmCcbWEkozNLddKzPD4cU8iEZ5y+BmM2Yd5xflRWuoQ yzdV3pzNW1zp7sacrdnpBso7SKsU+AJsPTIaCthJZ8QlKjc6LQe/0oV7sItUpkPp CsYQ0Mm1a7JNs0MNN9Yu4HcrGaT5t5Gs461/sawWUJagaBgevUVuP0ow== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=rr03hF F2bNF2szMfr4QL+Usn5E1acu59izxNNk0mufA=; b=ZGqjaCjzgP5RwV0eEqpuWo rUM042tt5dRkxbFqXLo6gzxARruBnMcjVUpyd5zDuEGiNTyqSsdPSj3iIwqHXV3z q5NWfhNMgk4ShLqmwFCAyEPgV7xapVPIWR1PgZoErrY9DQ7bthOABsDTQzlwQIk3 MMd8BAWIRI2tqhmIQqIxzNIjISh6EEcw6SoC5h2kOrsQQdFHwSRdlunGO26Pbwoo g6QB5SsN8U9jttHC6VZq4WimlZCCYfnIa0m7ZTiwsPBSaeLGSpLKJZpj6zoWVcds f1Kmk/0iBRlsO2zcxhSwt/S1ED8WaG+l5sWncT/g42MJ8HYlz4o8eZYZPTE2TXRQ == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrudegfedgleehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne hmihhsshhinhhgucfvqfcufhhivghlugculdeftddmnegoteeftdduqddtudculdduhedm necujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefuvhgvnh curfgvthgvrhcuoehsvhgvnhesshhvvghnphgvthgvrhdruggvvheqnecuggftrfgrthht vghrnheptedvkeetleeuffffhfekteetffeggffgveehieelueefvddtueffveevlefhfe ejnecuvehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepshhv vghnsehsvhgvnhhpvghtvghrrdguvghv X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 21 Mar 2022 12:52:05 -0400 (EDT) From: Sven Peter Cc: Sven Peter , Hector Martin , Alyssa Rosenzweig , Rob Herring , Arnd Bergmann , Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Marc Zyngier , devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org Subject: [PATCH 4/9] soc: apple: Add SART driver Date: Mon, 21 Mar 2022 17:50:44 +0100 Message-Id: <20220321165049.35985-5-sven@svenpeter.dev> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20220321165049.35985-1-sven@svenpeter.dev> References: <20220321165049.35985-1-sven@svenpeter.dev> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable To: unlisted-recipients:; (no To-header on input) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The NVMe co-processor on the Apple M1 uses a DMA address filter called SART for some DMA transactions. This adds a simple driver used to configure the memory regions from which DMA transactions are allowed. Co-developed-by: Hector Martin Signed-off-by: Hector Martin Signed-off-by: Sven Peter --- MAINTAINERS | 1 + drivers/soc/apple/Kconfig | 11 ++ drivers/soc/apple/Makefile | 3 + drivers/soc/apple/sart.c | 318 +++++++++++++++++++++++++++++++++ include/linux/soc/apple/sart.h | 77 ++++++++ 5 files changed, 410 insertions(+) create mode 100644 drivers/soc/apple/sart.c create mode 100644 include/linux/soc/apple/sart.h diff --git a/MAINTAINERS b/MAINTAINERS index 027c3b4ad61c..3d37fe7a0408 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1787,6 +1787,7 @@ F: drivers/watchdog/apple_wdt.c F: include/dt-bindings/interrupt-controller/apple-aic.h F: include/dt-bindings/pinctrl/apple.h F: include/linux/apple-mailbox.h +F: include/linux/soc/apple/* =20 ARM/ARTPEC MACHINE SUPPORT M: Jesper Nilsson diff --git a/drivers/soc/apple/Kconfig b/drivers/soc/apple/Kconfig index 9b8de31d6a8f..8c37ffd53fbd 100644 --- a/drivers/soc/apple/Kconfig +++ b/drivers/soc/apple/Kconfig @@ -17,6 +17,17 @@ config APPLE_PMGR_PWRSTATE controls for SoC devices. This driver manages them through the generic power domain framework, and also provides reset support. =20 +config APPLE_SART + tristate "Apple SART DMA address filter" + depends on ARCH_APPLE || COMPILE_TEST + default ARCH_APPLE + help + Apple SART is a simple DMA address filter used on Apple SoCs such + as the M1. It is usually required for the NVMe coprocessor which does + not use a proper IOMMU. + + Say 'y' here if you have an Apple SoC. + endmenu =20 endif diff --git a/drivers/soc/apple/Makefile b/drivers/soc/apple/Makefile index c114e84667e4..c83c66317098 100644 --- a/drivers/soc/apple/Makefile +++ b/drivers/soc/apple/Makefile @@ -1,2 +1,5 @@ # SPDX-License-Identifier: GPL-2.0-only obj-$(CONFIG_APPLE_PMGR_PWRSTATE) +=3D apple-pmgr-pwrstate.o + +obj-$(CONFIG_APPLE_SART) +=3D apple-sart.o +apple-sart-y =3D sart.o diff --git a/drivers/soc/apple/sart.c b/drivers/soc/apple/sart.c new file mode 100644 index 000000000000..836ea68db2fc --- /dev/null +++ b/drivers/soc/apple/sart.c @@ -0,0 +1,318 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +/* + * Apple SART device driver + * Copyright (C) The Asahi Linux Contributors + * + * Apple SART is a simple address filter for some DMA transactions. + * Regions of physical memory must be added to the SART's allow + * list before any DMA can target these. Unlike a proper + * IOMMU no remapping can be done and special support in the + * consumer driver is required since not all DMA transactions of + * a single device are subject to SART filtering. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define APPLE_SART_MAX_ENTRIES 16 + +/* This is probably a bitfield but the exact meaning of each bit is unknow= n. */ +#define APPLE_SART_FLAGS_ALLOW 0xff + +/* SARTv2 registers */ +#define APPLE_SART2_CONFIG(idx) (0x00 + 4 * (idx)) +#define APPLE_SART2_CONFIG_FLAGS GENMASK(31, 24) +#define APPLE_SART2_CONFIG_SIZE GENMASK(23, 0) +#define APPLE_SART2_CONFIG_SIZE_SHIFT 12 +#define APPLE_SART2_CONFIG_SIZE_MAX GENMASK(23, 0) + +#define APPLE_SART2_PADDR(idx) (0x40 + 4 * (idx)) +#define APPLE_SART2_PADDR_SHIFT 12 + +/* SARTv3 registers */ +#define APPLE_SART3_CONFIG(idx) (0x00 + 4 * (idx)) + +#define APPLE_SART3_PADDR(idx) (0x40 + 4 * (idx)) +#define APPLE_SART3_PADDR_SHIFT 12 + +#define APPLE_SART3_SIZE(idx) (0x80 + 4 * (idx)) +#define APPLE_SART3_SIZE_SHIFT 12 +#define APPLE_SART3_SIZE_MAX GENMASK(29, 0) + +struct apple_sart_ops { + void (*get_entry)(struct apple_sart *sart, int index, u8 *flags, + phys_addr_t *paddr, size_t *size); + int (*set_entry)(struct apple_sart *sart, int index, u8 flags, + phys_addr_t paddr, size_t size); +}; + +struct apple_sart { + struct device *dev; + void __iomem *regs; + + const struct apple_sart_ops *ops; + + unsigned long protected_entries; + unsigned long used_entries; +}; + +static void sart2_get_entry(struct apple_sart *sart, int index, u8 *flags, + phys_addr_t *paddr, size_t *size) +{ + u32 cfg =3D readl_relaxed(sart->regs + APPLE_SART2_CONFIG(index)); + u32 paddr_ =3D readl_relaxed(sart->regs + APPLE_SART2_PADDR(index)); + u32 size_ =3D FIELD_GET(APPLE_SART2_CONFIG_SIZE, cfg); + + *flags =3D FIELD_GET(APPLE_SART2_CONFIG_FLAGS, cfg); + *size =3D (size_t)size_ << APPLE_SART2_CONFIG_SIZE_SHIFT; + *paddr =3D (phys_addr_t)paddr_ << APPLE_SART2_PADDR_SHIFT; +} + +static int sart2_set_entry(struct apple_sart *sart, int index, u8 flags, + phys_addr_t paddr, size_t size) +{ + u32 cfg; + + if (size & ((1 << APPLE_SART2_CONFIG_SIZE_SHIFT) - 1)) + return -EINVAL; + if (paddr & ((1 << APPLE_SART2_PADDR_SHIFT) - 1)) + return -EINVAL; + + size >>=3D APPLE_SART2_CONFIG_SIZE_SHIFT; + paddr >>=3D APPLE_SART2_PADDR_SHIFT; + + if (size > APPLE_SART2_CONFIG_SIZE_MAX) + return -EINVAL; + + cfg =3D FIELD_PREP(APPLE_SART2_CONFIG_FLAGS, flags); + cfg |=3D FIELD_PREP(APPLE_SART2_CONFIG_SIZE, size); + + writel_relaxed(paddr, sart->regs + APPLE_SART2_PADDR(index)); + writel_relaxed(cfg, sart->regs + APPLE_SART2_CONFIG(index)); + + return 0; +} + +static struct apple_sart_ops sart_ops_v2 =3D { + .get_entry =3D sart2_get_entry, + .set_entry =3D sart2_set_entry, +}; + +static void sart3_get_entry(struct apple_sart *sart, int index, u8 *flags, + phys_addr_t *paddr, size_t *size) +{ + u32 paddr_ =3D readl_relaxed(sart->regs + APPLE_SART3_PADDR(index)); + u32 size_ =3D readl_relaxed(sart->regs + APPLE_SART3_SIZE(index)); + + *flags =3D readl_relaxed(sart->regs + APPLE_SART3_CONFIG(index)); + *size =3D (size_t)size_ << APPLE_SART3_SIZE_SHIFT; + *paddr =3D (phys_addr_t)paddr_ << APPLE_SART3_PADDR_SHIFT; +} + +static int sart3_set_entry(struct apple_sart *sart, int index, u8 flags, + phys_addr_t paddr, size_t size) +{ + if (size & ((1 << APPLE_SART3_SIZE_SHIFT) - 1)) + return -EINVAL; + if (paddr & ((1 << APPLE_SART3_PADDR_SHIFT) - 1)) + return -EINVAL; + + paddr >>=3D APPLE_SART3_PADDR_SHIFT; + size >>=3D APPLE_SART3_SIZE_SHIFT; + + if (size > APPLE_SART3_SIZE_MAX) + return -EINVAL; + + writel_relaxed(paddr, sart->regs + APPLE_SART3_PADDR(index)); + writel_relaxed(size, sart->regs + APPLE_SART3_SIZE(index)); + writel_relaxed(flags, sart->regs + APPLE_SART3_CONFIG(index)); + + return 0; +} + +static struct apple_sart_ops sart_ops_v3 =3D { + .get_entry =3D sart3_get_entry, + .set_entry =3D sart3_set_entry, +}; + +static int apple_sart_probe(struct platform_device *pdev) +{ + int i; + struct apple_sart *sart; + struct device *dev =3D &pdev->dev; + + sart =3D devm_kzalloc(dev, sizeof(*sart), GFP_KERNEL); + if (!sart) + return -ENOMEM; + + sart->dev =3D dev; + sart->ops =3D of_device_get_match_data(dev); + + sart->regs =3D devm_platform_ioremap_resource(pdev, 0); + if (IS_ERR(sart->regs)) + return PTR_ERR(sart->regs); + + for (i =3D 0; i < APPLE_SART_MAX_ENTRIES; ++i) { + u8 flags; + size_t size; + phys_addr_t paddr; + + sart->ops->get_entry(sart, i, &flags, &paddr, &size); + + if (!flags) + continue; + + dev_dbg(sart->dev, + "SART bootloader entry: index %02d; flags: 0x%02x; paddr: %pa; size: 0x= %zx\n", + i, flags, &paddr, size); + set_bit(i, &sart->protected_entries); + } + + platform_set_drvdata(pdev, sart); + return 0; +} + +struct apple_sart *apple_sart_get(struct device *dev) +{ + struct device_node *sart_node; + struct platform_device *sart_pdev; + struct apple_sart *sart; + + sart_node =3D of_parse_phandle(dev->of_node, "apple,sart", 0); + if (!sart_node) + return ERR_PTR(ENODEV); + + sart_pdev =3D of_find_device_by_node(sart_node); + of_node_put(sart_node); + + if (!sart_pdev) + return ERR_PTR(ENODEV); + + sart =3D dev_get_drvdata(&sart_pdev->dev); + if (!sart) + return ERR_PTR(EPROBE_DEFER); + + device_link_add(dev, &sart_pdev->dev, + DL_FLAG_PM_RUNTIME | DL_FLAG_AUTOREMOVE_SUPPLIER); + + return sart; +} +EXPORT_SYMBOL(apple_sart_get); + +int apple_sart_add_allowed_region(struct apple_sart *sart, phys_addr_t pad= dr, + size_t size) +{ + int i, ret; + + for (i =3D 0; i < APPLE_SART_MAX_ENTRIES; ++i) { + if (test_bit(i, &sart->protected_entries)) + continue; + if (test_and_set_bit(i, &sart->used_entries)) + continue; + + ret =3D sart->ops->set_entry(sart, i, APPLE_SART_FLAGS_ALLOW, + paddr, size); + if (ret) { + dev_dbg(sart->dev, + "unable to set entry %d to [%pa, 0x%zx]\n", + i, &paddr, size); + clear_bit(i, &sart->used_entries); + return ret; + } + + dev_dbg(sart->dev, "wrote [%pa, 0x%zx] to %d\n", &paddr, size, + i); + return 0; + } + + dev_warn(sart->dev, + "no free entries left to add [paddr: 0x%llx, size: 0x%zx]\n", + paddr, size); + + return -EBUSY; +} +EXPORT_SYMBOL(apple_sart_add_allowed_region); + +int apple_sart_remove_allowed_region(struct apple_sart *sart, phys_addr_t = paddr, + size_t size) +{ + int i; + + dev_dbg(sart->dev, + "will remove [paddr: %pa, size: 0x%zx] from allowed regions\n", + &paddr, size); + + for (i =3D 0; i < APPLE_SART_MAX_ENTRIES; ++i) { + u8 eflags; + size_t esize; + phys_addr_t epaddr; + + if (test_bit(i, &sart->protected_entries)) + continue; + + sart->ops->get_entry(sart, i, &eflags, &epaddr, &esize); + + if (epaddr !=3D paddr || esize !=3D size) + continue; + + sart->ops->set_entry(sart, i, 0, 0, 0); + + clear_bit(i, &sart->used_entries); + dev_dbg(sart->dev, "cleared entry %d\n", i); + return 0; + } + + dev_warn(sart->dev, "entry [paddr: 0x%llx, size: 0x%zx] not found\n", + paddr, size); + + return -EINVAL; +} +EXPORT_SYMBOL(apple_sart_remove_allowed_region); + +static void apple_sart_shutdown(struct platform_device *pdev) +{ + struct apple_sart *sart =3D dev_get_drvdata(&pdev->dev); + int i; + + for (i =3D 0; i < APPLE_SART_MAX_ENTRIES; ++i) { + if (test_bit(i, &sart->protected_entries)) + continue; + + sart->ops->set_entry(sart, i, 0, 0, 0); + } +} + +static const struct of_device_id apple_sart_of_match[] =3D { + { + .compatible =3D "apple,t6000-sart", + .data =3D &sart_ops_v3, + }, + { + .compatible =3D "apple,t8103-sart", + .data =3D &sart_ops_v2, + }, + {} +}; +MODULE_DEVICE_TABLE(of, apple_sart_of_match); + +static struct platform_driver apple_sart_driver =3D { + .driver =3D { + .name =3D "apple-sart", + .of_match_table =3D apple_sart_of_match, + }, + .probe =3D apple_sart_probe, + .shutdown =3D apple_sart_shutdown, +}; +module_platform_driver(apple_sart_driver); + +MODULE_LICENSE("Dual MIT/GPL"); +MODULE_AUTHOR("Sven Peter "); +MODULE_DESCRIPTION("Apple SART driver"); diff --git a/include/linux/soc/apple/sart.h b/include/linux/soc/apple/sart.h new file mode 100644 index 000000000000..4e3d4960be5a --- /dev/null +++ b/include/linux/soc/apple/sart.h @@ -0,0 +1,77 @@ +/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ +/* + * Apple SART device driver + * Copyright (C) The Asahi Linux Contributors + * + * Apple SART is a simple address filter for DMA transactions. + * Regions of physical memory must be added to the SART's allow + * list before any DMA can target these. Unlike a proper + * IOMMU no remapping can be done. + */ + +#ifndef _LINUX_SOC_APPLE_SART_H_ +#define _LINUX_SOC_APPLE_SART_H_ + +#include +#include +#include + +struct apple_sart; + +#if IS_ENABLED(CONFIG_APPLE_SART) + +/* + * Get a reference to the SART attached to dev. + * + * Looks for the phandle reference in apple,sart and returns a pointer + * to the corresponding apple_sart struct to be used with + * apple_sart_add_allowed_region and apple_sart_remove_allowed_region. + */ +struct apple_sart *apple_sart_get(struct device *dev); + +/* + * Adds the region [paddr, paddr+size] to the DMA allow list. + * + * @sart: SART reference + * @paddr: Start address of the region to be used for DMA + * @size: Size of the region to be used for DMA. + */ +int apple_sart_add_allowed_region(struct apple_sart *sart, phys_addr_t pad= dr, + size_t size); + +/* + * Removes the region [paddr, paddr+size] from the DMA allow list. + * + * Note that exact same paddr and size used for apple_sart_add_allowed_reg= ion + * have to be passed. + * + * @sart: SART reference + * @paddr: Start address of the region no longer used for DMA + * @size: Size of the region no longer used for DMA. + */ +int apple_sart_remove_allowed_region(struct apple_sart *sart, phys_addr_t = paddr, + size_t size); + +#else + +static inline struct apple_sart *apple_sart_get(struct device *dev) +{ + return ERR_PTR(-ENODEV); +} + +static inline int apple_sart_add_allowed_region(struct apple_sart *sart, + phys_addr_t paddr, size_t size) +{ + return -ENODEV; +} + +static inline int apple_sart_remove_allowed_region(struct apple_sart *sart, + phys_addr_t paddr, + size_t size) +{ + return -ENODEV; +} + +#endif /* IS_ENABLED(CONFIG_APPLE_SART) */ + +#endif /* _LINUX_SOC_APPLE_SART_H_ */ --=20 2.25.1 From nobody Mon Jun 22 15:42:51 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29C0EC433F5 for ; Mon, 21 Mar 2022 16:52:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351481AbiCUQxx (ORCPT ); Mon, 21 Mar 2022 12:53:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52720 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1351444AbiCUQxt (ORCPT ); Mon, 21 Mar 2022 12:53:49 -0400 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 03166181150 for ; Mon, 21 Mar 2022 09:52:13 -0700 (PDT) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id 32FD33201E3E; Mon, 21 Mar 2022 12:52:12 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Mon, 21 Mar 2022 12:52:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=svenpeter.dev; h=cc:cc:content-transfer-encoding:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm1; bh=p5Ys4eQa7puvj9 /ziOqLgg1MYgYcW6qkq+qG6HyFH5A=; b=AcAewwuTPkjuL+5CPltcc21ol3cyyt gIcuIiOPD+N6XgIEB6gzN2m+gb2CWt+L9SNS8p/Es8zFs2i6eQ9gKdpJJVYtbo61 /uZ2wy0/ScHfm9NcfFNgLpAnozV83upJLE5LIIb+4zAx2lsKNmqzw6FGFSMh0zMl nyfCxeEKMsfLsKwSvHDUJ13C7psnRi9obANt1fJy1LsXb5xefCOOUBpGqf7YOvuS Wfqom1O/FLT2fZATDsmyneIgEEnWFzr886inQIr66NpaZwJWjl7XFW3CSv/n+0FG 81F2JQ584/YX9TNofI0xgGkm+ZXClyEI6XtUgDGx0iIHDpRDtzxtCZFw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=p5Ys4e Qa7puvj9/ziOqLgg1MYgYcW6qkq+qG6HyFH5A=; b=Wcxn2xBqFZ/GbPV40m9Shv BW2rQs+rrqV/3f5ChxCYAeWi5EO1PFsFCNGCqE38UjVtHOx4DUObDRc4jG9peDov rfb0yEfcnfolcMYLf2CMP4tdX8GKVh75t/ybntTsp+IISFR+L3Y6C5ATDYo8SecD RVeF9u3igtlHTWGaeE8+us+jzLKt//nfiuPe+v3fa4kl5m5wgaxoxWbc4GDbYSJb aDhU8TRB9EnM9uWEjdPOZ6wqYFCGDbtBV6qhZVF9M8R6mdjj3zs1mS6+zr98+Jm9 HVPgds+Bdj+4AHKv3WHBm3hzMWkg9ukZezQJIEN7x+6E2Nxp29glaJADC2D6ueAg == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrudegfedgleehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne hmihhsshhinhhgucfvqfcufhhivghlugculdeftddmnegoteeftdduqddtudculdduhedm necujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefuvhgvnh curfgvthgvrhcuoehsvhgvnhesshhvvghnphgvthgvrhdruggvvheqnecuggftrfgrthht vghrnheptedvkeetleeuffffhfekteetffeggffgveehieelueefvddtueffveevlefhfe ejnecuvehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepshhv vghnsehsvhgvnhhpvghtvghrrdguvghv X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 21 Mar 2022 12:52:09 -0400 (EDT) From: Sven Peter Cc: Sven Peter , Hector Martin , Alyssa Rosenzweig , Rob Herring , Arnd Bergmann , Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Marc Zyngier , devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org Subject: [PATCH 5/9] soc: apple: Add RTKit IPC library Date: Mon, 21 Mar 2022 17:50:45 +0100 Message-Id: <20220321165049.35985-6-sven@svenpeter.dev> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20220321165049.35985-1-sven@svenpeter.dev> References: <20220321165049.35985-1-sven@svenpeter.dev> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable To: unlisted-recipients:; (no To-header on input) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Apple SoCs such as the M1 come with multiple embedded co-processors running proprietary firmware. Communication with those is established over a simple mailbox using the RTKit IPC protocol. Signed-off-by: Sven Peter --- drivers/soc/apple/Kconfig | 13 + drivers/soc/apple/Makefile | 3 + drivers/soc/apple/rtkit-crashlog.c | 147 +++++ drivers/soc/apple/rtkit-internal.h | 76 +++ drivers/soc/apple/rtkit.c | 842 +++++++++++++++++++++++++++++ include/linux/soc/apple/rtkit.h | 203 +++++++ 6 files changed, 1284 insertions(+) create mode 100644 drivers/soc/apple/rtkit-crashlog.c create mode 100644 drivers/soc/apple/rtkit-internal.h create mode 100644 drivers/soc/apple/rtkit.c create mode 100644 include/linux/soc/apple/rtkit.h diff --git a/drivers/soc/apple/Kconfig b/drivers/soc/apple/Kconfig index 8c37ffd53fbd..feb56419ac3c 100644 --- a/drivers/soc/apple/Kconfig +++ b/drivers/soc/apple/Kconfig @@ -17,6 +17,19 @@ config APPLE_PMGR_PWRSTATE controls for SoC devices. This driver manages them through the generic power domain framework, and also provides reset support. =20 +config APPLE_RTKIT + tristate "Apple RTKit co-processor IPC protocol" + depends on MAILBOX + depends on ARCH_APPLE || (COMPILE_TEST && 64BIT) + default ARCH_APPLE + help + Apple SoCs such as the M1 come with various co-processors running + their proprietary RTKit operating system. This option enables support + for the protocol library used to communicate with those. It is used + by various client drivers. + + Say 'y' here if you have an Apple SoC. + config APPLE_SART tristate "Apple SART DMA address filter" depends on ARCH_APPLE || COMPILE_TEST diff --git a/drivers/soc/apple/Makefile b/drivers/soc/apple/Makefile index c83c66317098..e293770cf66d 100644 --- a/drivers/soc/apple/Makefile +++ b/drivers/soc/apple/Makefile @@ -1,5 +1,8 @@ # SPDX-License-Identifier: GPL-2.0-only obj-$(CONFIG_APPLE_PMGR_PWRSTATE) +=3D apple-pmgr-pwrstate.o =20 +obj-$(CONFIG_APPLE_RTKIT) +=3D apple-rtkit.o +apple-rtkit-y =3D rtkit.o rtkit-crashlog.o + obj-$(CONFIG_APPLE_SART) +=3D apple-sart.o apple-sart-y =3D sart.o diff --git a/drivers/soc/apple/rtkit-crashlog.c b/drivers/soc/apple/rtkit-c= rashlog.c new file mode 100644 index 000000000000..4612c8997632 --- /dev/null +++ b/drivers/soc/apple/rtkit-crashlog.c @@ -0,0 +1,147 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +/* + * Apple RTKit IPC library + * Copyright (C) The Asahi Linux Contributors + */ +#include "rtkit-internal.h" + +#define FOURCC(a, b, c, d) = \ + (((u32)(a) << 24) | ((u32)(b) << 16) | ((u32)(c) << 8) | ((u32)(d))) + +#define APPLE_RTKIT_CRASHLOG_HEADER FOURCC('C', 'L', 'H', 'E') +#define APPLE_RTKIT_CRASHLOG_STR FOURCC('C', 's', 't', 'r') +#define APPLE_RTKIT_CRASHLOG_VERSION FOURCC('C', 'v', 'e', 'r') +#define APPLE_RTKIT_CRASHLOG_MBOX FOURCC('C', 'm', 'b', 'x') +#define APPLE_RTKIT_CRASHLOG_TIME FOURCC('C', 't', 'i', 'm') + +struct apple_rtkit_crashlog_header { + u32 fourcc; + u32 version; + u32 size; + u32 flags; + u8 _unk[16]; +}; +static_assert(sizeof(struct apple_rtkit_crashlog_header) =3D=3D 0x20); + +struct apple_rtkit_crashlog_mbox_entry { + u64 msg0; + u64 msg1; + u32 timestamp; + u8 _unk[4]; +}; +static_assert(sizeof(struct apple_rtkit_crashlog_mbox_entry) =3D=3D 0x18); + +static void apple_rtkit_crashlog_dump_str(struct apple_rtkit *rtk, u8 *bfr, + size_t size) +{ + u32 idx; + u8 *ptr, *end; + + memcpy(&idx, bfr, 4); + + ptr =3D bfr + 4; + end =3D bfr + size; + while (ptr < end) { + u8 *newline =3D memchr(ptr, '\n', end - ptr); + + if (newline) { + u8 tmp =3D *newline; + *newline =3D '\0'; + rtk_warn("Message (id=3D%x): %s\n", idx, ptr); + *newline =3D tmp; + ptr =3D newline + 1; + } else { + rtk_warn("Message (id=3D%x): %s", idx, ptr); + break; + } + } +} + +static void apple_rtkit_crashlog_dump_version(struct apple_rtkit *rtk, u8 = *bfr, + size_t size) +{ + rtk_warn("Version: %s", bfr + 16); +} + +static void apple_rtkit_crashlog_dump_time(struct apple_rtkit *rtk, u8 *bf= r, + size_t size) +{ + u64 crash_time; + + memcpy(&crash_time, bfr, 8); + rtk_warn("Crash time: %lld", crash_time); +} + +static void apple_rtkit_crashlog_dump_mailbox(struct apple_rtkit *rtk, u8 = *bfr, + size_t size) +{ + u32 type, index, i; + size_t n_messages; + struct apple_rtkit_crashlog_mbox_entry entry; + + memcpy(&type, bfr + 16, 4); + memcpy(&index, bfr + 24, 4); + n_messages =3D (size - 28) / sizeof(entry); + + rtk_warn("Mailbox history (type =3D %d, index =3D %d)", type, index); + for (i =3D 0; i < n_messages; ++i) { + memcpy(&entry, bfr + 28 + i * sizeof(entry), sizeof(entry)); + rtk_warn(" #%03d@%08x: %016llx %016llx", i, entry.timestamp, + entry.msg0, entry.msg1); + } +} + +void apple_rtkit_crashlog_dump(struct apple_rtkit *rtk, u8 *bfr, size_t si= ze) +{ + size_t offset; + u32 section_fourcc, section_size; + struct apple_rtkit_crashlog_header header; + + memcpy(&header, bfr, sizeof(header)); + if (header.fourcc !=3D APPLE_RTKIT_CRASHLOG_HEADER) { + rtk_warn("Expected crashlog header but got %x", header.fourcc); + return; + } + + if (header.size > size) { + rtk_warn("Crashlog size (%x) is too large", header.size); + return; + } + + size =3D header.size; + offset =3D sizeof(header); + + while (offset < size) { + memcpy(§ion_fourcc, bfr + offset, 4); + memcpy(§ion_size, bfr + offset + 12, 4); + + switch (section_fourcc) { + case APPLE_RTKIT_CRASHLOG_HEADER: + rtk_dbg("End of crashlog reached"); + return; + case APPLE_RTKIT_CRASHLOG_STR: + apple_rtkit_crashlog_dump_str(rtk, bfr + offset + 16, + section_size); + break; + case APPLE_RTKIT_CRASHLOG_VERSION: + apple_rtkit_crashlog_dump_version( + rtk, bfr + offset + 16, section_size); + break; + case APPLE_RTKIT_CRASHLOG_MBOX: + apple_rtkit_crashlog_dump_mailbox( + rtk, bfr + offset + 16, section_size); + break; + case APPLE_RTKIT_CRASHLOG_TIME: + apple_rtkit_crashlog_dump_time(rtk, bfr + offset + 16, + section_size); + break; + default: + rtk_warn("Unknown crashlog section: %x", + section_fourcc); + } + + offset +=3D section_size; + } + + rtk_warn("End of crashlog reached but no footer present"); +} diff --git a/drivers/soc/apple/rtkit-internal.h b/drivers/soc/apple/rtkit-i= nternal.h new file mode 100644 index 000000000000..6ff8b2cd2532 --- /dev/null +++ b/drivers/soc/apple/rtkit-internal.h @@ -0,0 +1,76 @@ +/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ +/* + * Apple RTKit IPC library + * Copyright (C) The Asahi Linux Contributors + */ + +#ifndef _APPLE_RTKIT_INTERAL_H +#define _APPLE_RTKIT_INTERAL_H + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define rtk_err(format, arg...) dev_err(rtk->dev, "RTKit: " format, ##arg) +#define rtk_warn(format, arg...) dev_warn(rtk->dev, "RTKit: " format, ##ar= g) +#define rtk_info(format, arg...) dev_info(rtk->dev, "RTKit: " format, ##ar= g) +#define rtk_dbg(format, arg...) dev_dbg(rtk->dev, "RTKit: " format, ##arg) + + +#define APPLE_RTKIT_APP_ENDPOINT_START 0x20 +#define APPLE_RTKIT_MAX_ENDPOINTS 0x100 + +struct apple_rtkit_work { + unsigned int type; + struct apple_mbox_msg msg; +}; + +struct apple_rtkit { + void *cookie; + const struct apple_rtkit_ops *ops; + struct device *dev; + struct mbox_client mbox_cl; + struct mbox_chan *mbox_chan; + + struct completion epmap_completion; + struct completion reinit_completion; + struct completion iop_pwr_ack_completion; + struct completion ap_pwr_ack_completion; + + int boot_result; + int version; + + unsigned int iop_power_state; + unsigned int ap_power_state; + bool crashed; + + struct task_struct *task; + + struct wait_queue_head wq; + DECLARE_KFIFO(work_fifo, struct apple_rtkit_work, 64); + spinlock_t work_lock; + + DECLARE_BITMAP(endpoints, APPLE_RTKIT_MAX_ENDPOINTS); + + struct apple_rtkit_shmem ioreport_buffer; + struct apple_rtkit_shmem crashlog_buffer; + + struct apple_rtkit_shmem syslog_buffer; + char *syslog_msg_buffer; + size_t syslog_n_entries; + size_t syslog_msg_size; +}; + +void apple_rtkit_crashlog_dump(struct apple_rtkit *rtk, u8 *bfr, size_t si= ze); + +#endif diff --git a/drivers/soc/apple/rtkit.c b/drivers/soc/apple/rtkit.c new file mode 100644 index 000000000000..7a93c6a99ae9 --- /dev/null +++ b/drivers/soc/apple/rtkit.c @@ -0,0 +1,842 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +/* + * Apple RTKit IPC library + * Copyright (C) The Asahi Linux Contributors + */ + +//#define DEBUG + +#include "rtkit-internal.h" + +enum { APPLE_RTKIT_WORK_MSG, + APPLE_RTKIT_WORK_REINIT, +}; + +enum { APPLE_RTKIT_PWR_STATE_OFF =3D 0x00, + APPLE_RTKIT_PWR_STATE_SLEEP =3D 0x01, + APPLE_RTKIT_PWR_STATE_GATED =3D 0x02, + APPLE_RTKIT_PWR_STATE_QUIESCED =3D 0x10, + APPLE_RTKIT_PWR_STATE_ON =3D 0x20, +}; + +enum { APPLE_RTKIT_EP_MGMT =3D 0, + APPLE_RTKIT_EP_CRASHLOG =3D 1, + APPLE_RTKIT_EP_SYSLOG =3D 2, + APPLE_RTKIT_EP_DEBUG =3D 3, + APPLE_RTKIT_EP_IOREPORT =3D 4, + APPLE_RTKIT_EP_OSLOG =3D 8, +}; + +#define APPLE_RTKIT_MGMT_TYPE GENMASK(59, 52) + +enum { APPLE_RTKIT_MGMT_HELLO =3D 1, + APPLE_RTKIT_MGMT_HELLO_REPLY =3D 2, + APPLE_RTKIT_MGMT_STARTEP =3D 5, + APPLE_RTKIT_MGMT_SET_IOP_PWR_STATE =3D 6, + APPLE_RTKIT_MGMT_SET_IOP_PWR_STATE_ACK =3D 7, + APPLE_RTKIT_MGMT_EPMAP =3D 8, + APPLE_RTKIT_MGMT_EPMAP_REPLY =3D 8, + APPLE_RTKIT_MGMT_SET_AP_PWR_STATE =3D 0xb, + APPLE_RTKIT_MGMT_SET_AP_PWR_STATE_ACK =3D 0xb, +}; + +#define APPLE_RTKIT_MGMT_HELLO_MINVER GENMASK(15, 0) +#define APPLE_RTKIT_MGMT_HELLO_MAXVER GENMASK(31, 16) + +#define APPLE_RTKIT_MGMT_EPMAP_LAST BIT(51) +#define APPLE_RTKIT_MGMT_EPMAP_BASE GENMASK(34, 32) +#define APPLE_RTKIT_MGMT_EPMAP_BITMAP GENMASK(31, 0) + +#define APPLE_RTKIT_MGMT_EPMAP_REPLY_MORE BIT(0) + +#define APPLE_RTKIT_MGMT_STARTEP_EP GENMASK(39, 32) +#define APPLE_RTKIT_MGMT_STARTEP_FLAG BIT(1) + +#define APPLE_RTKIT_MGMT_PWR_STATE GENMASK(15, 0) + +#define APPLE_RTKIT_CRASHLOG_CRASH 1 + +#define APPLE_RTKIT_BUFFER_REQUEST 1 +#define APPLE_RTKIT_BUFFER_REQUEST_SIZE GENMASK(51, 44) +#define APPLE_RTKIT_BUFFER_REQUEST_IOVA GENMASK(41, 0) + +#define APPLE_RTKIT_SYSLOG_TYPE GENMASK(59, 52) + +#define APPLE_RTKIT_SYSLOG_LOG 5 + +#define APPLE_RTKIT_SYSLOG_INIT 8 +#define APPLE_RTKIT_SYSLOG_N_ENTRIES GENMASK(7, 0) +#define APPLE_RTKIT_SYSLOG_MSG_SIZE GENMASK(31, 24) + +#define APPLE_RTKIT_OSLOG_TYPE GENMASK(63, 56) +#define APPLE_RTKIT_OSLOG_INIT 1 +#define APPLE_RTKIT_OSLOG_ACK 3 + +#define APPLE_RTKIT_MIN_SUPPORTED_VERSION 11 +#define APPLE_RTKIT_MAX_SUPPORTED_VERSION 12 + +bool apple_rtkit_is_running(struct apple_rtkit *rtk) +{ + if (rtk->crashed) + return false; + if ((rtk->iop_power_state & 0xff) !=3D APPLE_RTKIT_PWR_STATE_ON) + return false; + if ((rtk->ap_power_state & 0xff) !=3D APPLE_RTKIT_PWR_STATE_ON) + return false; + return true; +} +EXPORT_SYMBOL_GPL(apple_rtkit_is_running); + +bool apple_rtkit_is_crashed(struct apple_rtkit *rtk) +{ + return rtk->crashed; +} +EXPORT_SYMBOL_GPL(apple_rtkit_is_crashed); + +static void apple_rtkit_management_send(struct apple_rtkit *rtk, u8 type, + u64 msg) +{ + msg &=3D ~APPLE_RTKIT_MGMT_TYPE; + msg |=3D FIELD_PREP(APPLE_RTKIT_MGMT_TYPE, type); + apple_rtkit_send_message(rtk, APPLE_RTKIT_EP_MGMT, msg); +} + +static void apple_rtkit_management_rx_hello(struct apple_rtkit *rtk, u64 m= sg) +{ + u64 reply; + + int min_ver =3D FIELD_GET(APPLE_RTKIT_MGMT_HELLO_MINVER, msg); + int max_ver =3D FIELD_GET(APPLE_RTKIT_MGMT_HELLO_MAXVER, msg); + int want_ver =3D min(APPLE_RTKIT_MAX_SUPPORTED_VERSION, max_ver); + + rtk_dbg("Min ver %d, max ver %d\n", min_ver, max_ver); + + if (min_ver > APPLE_RTKIT_MAX_SUPPORTED_VERSION) { + rtk_err("Firmware min version %d is too new\n", min_ver); + goto abort_boot; + } + + if (max_ver < APPLE_RTKIT_MIN_SUPPORTED_VERSION) { + rtk_err("Firmware max version %d is too old\n", max_ver); + goto abort_boot; + } + + rtk_info("Initializing (protocol version %d)\n", want_ver); + rtk->version =3D want_ver; + + reply =3D FIELD_PREP(APPLE_RTKIT_MGMT_HELLO_MINVER, want_ver); + reply |=3D FIELD_PREP(APPLE_RTKIT_MGMT_HELLO_MAXVER, want_ver); + apple_rtkit_management_send(rtk, APPLE_RTKIT_MGMT_HELLO_REPLY, reply); + + return; + +abort_boot: + rtk->boot_result =3D -EINVAL; + complete_all(&rtk->epmap_completion); +} + +static void apple_rtkit_management_rx_epmap(struct apple_rtkit *rtk, u64 m= sg) +{ + int i, ep; + u64 reply; + unsigned long bitmap =3D FIELD_GET(APPLE_RTKIT_MGMT_EPMAP_BITMAP, msg); + u32 base =3D FIELD_GET(APPLE_RTKIT_MGMT_EPMAP_BASE, msg); + + rtk_dbg("received endpoint bitmap 0x%lx with base 0x%x\n", bitmap, + base); + + for_each_set_bit(i, &bitmap, 32) { + ep =3D 32 * base + i; + rtk_dbg("Discovered endpoint 0x%02x\n", ep); + set_bit(ep, rtk->endpoints); + } + + reply =3D FIELD_PREP(APPLE_RTKIT_MGMT_EPMAP_BASE, base); + if (msg & APPLE_RTKIT_MGMT_EPMAP_LAST) + reply |=3D APPLE_RTKIT_MGMT_EPMAP_LAST; + else + reply |=3D APPLE_RTKIT_MGMT_EPMAP_REPLY_MORE; + + apple_rtkit_management_send(rtk, APPLE_RTKIT_MGMT_EPMAP_REPLY, reply); + + if (!(msg & APPLE_RTKIT_MGMT_EPMAP_LAST)) + return; + + for_each_set_bit(ep, rtk->endpoints, APPLE_RTKIT_APP_ENDPOINT_START) { + switch (ep) { + /* the management endpoint is started by default */ + case APPLE_RTKIT_EP_MGMT: + break; + + /* without starting these RTKit refuses to boot */ + case APPLE_RTKIT_EP_SYSLOG: + case APPLE_RTKIT_EP_CRASHLOG: + case APPLE_RTKIT_EP_DEBUG: + case APPLE_RTKIT_EP_IOREPORT: + case APPLE_RTKIT_EP_OSLOG: + rtk_dbg("Starting system endpoint 0x%02x\n", ep); + apple_rtkit_start_ep(rtk, ep); + break; + + default: + rtk_warn("Unknown system endpoint: 0x%02x\n", ep); + } + } + + complete_all(&rtk->epmap_completion); +} + +static void apple_rtkit_management_rx_iop_pwr_ack(struct apple_rtkit *rtk, + u64 msg) +{ + unsigned int new_state =3D FIELD_GET(APPLE_RTKIT_MGMT_PWR_STATE, msg); + + rtk_dbg("IOP power state transition: 0x%x -> 0x%x\n", + rtk->iop_power_state, new_state); + rtk->iop_power_state =3D new_state; + + complete_all(&rtk->iop_pwr_ack_completion); +} + +static void apple_rtkit_management_rx_ap_pwr_ack(struct apple_rtkit *rtk, + u64 msg) +{ + unsigned int new_state =3D FIELD_GET(APPLE_RTKIT_MGMT_PWR_STATE, msg); + + rtk_dbg("AP power state transition: 0x%x -> 0x%x\n", + rtk->ap_power_state, new_state); + rtk->ap_power_state =3D new_state; + + complete_all(&rtk->ap_pwr_ack_completion); +} + +static void apple_rtkit_management_rx(struct apple_rtkit *rtk, u64 msg) +{ + u8 type =3D FIELD_GET(APPLE_RTKIT_MGMT_TYPE, msg); + + switch (type) { + case APPLE_RTKIT_MGMT_HELLO: + apple_rtkit_management_rx_hello(rtk, msg); + break; + case APPLE_RTKIT_MGMT_EPMAP: + apple_rtkit_management_rx_epmap(rtk, msg); + break; + case APPLE_RTKIT_MGMT_SET_IOP_PWR_STATE_ACK: + apple_rtkit_management_rx_iop_pwr_ack(rtk, msg); + break; + case APPLE_RTKIT_MGMT_SET_AP_PWR_STATE_ACK: + apple_rtkit_management_rx_ap_pwr_ack(rtk, msg); + break; + default: + rtk_warn("unknown management message: 0x%llx (type: 0x%02x)\n", + msg, type); + } +} + +static int apple_rtkit_common_rx_get_buffer(struct apple_rtkit *rtk, + struct apple_rtkit_shmem *buffer, + u8 ep, u64 msg) +{ + size_t n_4kpages =3D FIELD_GET(APPLE_RTKIT_BUFFER_REQUEST_SIZE, msg); + size_t size =3D n_4kpages << 12; + dma_addr_t iova =3D FIELD_GET(APPLE_RTKIT_BUFFER_REQUEST_IOVA, msg); + u64 reply; + int err; + + rtk_dbg("buffer request for 0x%zx bytes at %pad\n", size, &iova); + + if (iova && (!rtk->ops->shmem_setup || !rtk->ops->shmem_destroy)) + return -EINVAL; + + if (rtk->ops->shmem_setup) { + err =3D rtk->ops->shmem_setup(rtk->cookie, buffer, iova, size); + if (err < 0) + return err; + } else { + buffer->buffer =3D + dma_alloc_coherent(rtk->dev, size, &iova, GFP_KERNEL); + if (!buffer->buffer) + return -ENOMEM; + + buffer->size =3D size; + buffer->iova =3D iova; + } + + if (!buffer->is_mapped) { + reply =3D FIELD_PREP(APPLE_RTKIT_SYSLOG_TYPE, + APPLE_RTKIT_BUFFER_REQUEST); + reply |=3D FIELD_PREP(APPLE_RTKIT_BUFFER_REQUEST_SIZE, n_4kpages); + reply |=3D FIELD_PREP(APPLE_RTKIT_BUFFER_REQUEST_IOVA, + buffer->iova); + apple_rtkit_send_message(rtk, ep, reply); + } + + return 0; +} + +static void apple_rtkit_free_buffer(struct apple_rtkit *rtk, + struct apple_rtkit_shmem *bfr) +{ + if (bfr->size =3D=3D 0) + return; + + if (rtk->ops->shmem_destroy) + rtk->ops->shmem_destroy(rtk->cookie, bfr); + else if (bfr->buffer) + dma_free_coherent(rtk->dev, bfr->size, bfr->buffer, bfr->iova); + + bfr->buffer =3D NULL; + bfr->iomem =3D NULL; + bfr->iova =3D 0; + bfr->size =3D 0; + bfr->is_mapped =3D false; +} + +static void apple_rtkit_memcpy(struct apple_rtkit *rtk, void *dst, + struct apple_rtkit_shmem *bfr, size_t offset, + size_t len) +{ + if (bfr->iomem) + memcpy_fromio(dst, bfr->iomem + offset, len); + else + memcpy(dst, bfr->buffer + offset, len); +} + +static void apple_rtkit_crashlog_rx(struct apple_rtkit *rtk, u64 msg) +{ + u8 type =3D FIELD_GET(APPLE_RTKIT_SYSLOG_TYPE, msg); + u8 *bfr; + + if (type !=3D APPLE_RTKIT_CRASHLOG_CRASH) { + rtk_warn("Unknown crashlog message: %llx\n", msg); + return; + } + + if (!rtk->crashlog_buffer.size) { + apple_rtkit_common_rx_get_buffer(rtk, &rtk->crashlog_buffer, + APPLE_RTKIT_EP_CRASHLOG, msg); + return; + } + + rtk_err("co-processor has crashed.\n"); + + /* + * create a shadow copy here to make sure the co-processor isn't able + * to change the log while we're dumping it. this also ensures + * the buffer is in normal memory and not iomem for e.g. the SMC + */ + bfr =3D kzalloc(rtk->crashlog_buffer.size, GFP_KERNEL); + if (bfr) { + apple_rtkit_memcpy(rtk, bfr, &rtk->crashlog_buffer, 0, + rtk->crashlog_buffer.size); + apple_rtkit_crashlog_dump(rtk, bfr, rtk->crashlog_buffer.size); + kfree(bfr); + } else { + rtk_err("Couldn't allocate crashlog shadow buffer."); + } + + rtk->crashed =3D true; + if (rtk->ops->crashed) + rtk->ops->crashed(rtk->cookie); +} + +static void apple_rtkit_ioreport_rx(struct apple_rtkit *rtk, u64 msg) +{ + u8 type =3D FIELD_GET(APPLE_RTKIT_SYSLOG_TYPE, msg); + + switch (type) { + case APPLE_RTKIT_BUFFER_REQUEST: + apple_rtkit_common_rx_get_buffer(rtk, &rtk->ioreport_buffer, + APPLE_RTKIT_EP_IOREPORT, msg); + break; + /* unknown, must be ACKed or the co-processor will hang */ + case 0x8: + case 0xc: + apple_rtkit_send_message(rtk, APPLE_RTKIT_EP_IOREPORT, msg); + break; + default: + rtk_warn("Unknown ioreport message: %llx\n", msg); + } +} + +static void apple_rtkit_syslog_rx_init(struct apple_rtkit *rtk, u64 msg) +{ + rtk->syslog_n_entries =3D FIELD_GET(APPLE_RTKIT_SYSLOG_N_ENTRIES, msg); + rtk->syslog_msg_size =3D FIELD_GET(APPLE_RTKIT_SYSLOG_MSG_SIZE, msg); + + rtk->syslog_msg_buffer =3D kzalloc(rtk->syslog_msg_size, GFP_KERNEL); + + rtk_dbg("syslog initialized: entries: %zd, msg_size: %zd\n", + rtk->syslog_n_entries, rtk->syslog_msg_size); +} + +static void apple_rtkit_syslog_rx_log(struct apple_rtkit *rtk, u64 msg) +{ + u8 idx =3D msg & 0xff; + char log_context[24]; + size_t entry_size =3D 0x20 + rtk->syslog_msg_size; + + if (!rtk->syslog_buffer.size) { + rtk_warn( + "received syslog message but syslog_buffer.size is zero"); + goto done; + } + if (!rtk->syslog_buffer.buffer && !rtk->syslog_buffer.iomem) { + rtk_warn("received syslog message but no syslog_buffer.buffer or syslog_= buffer.iomem"); + goto done; + } + if (idx > rtk->syslog_n_entries) { + rtk_warn("syslog index %d out of range", idx); + goto done; + } + + apple_rtkit_memcpy(rtk, log_context, &rtk->syslog_buffer, + idx * entry_size + 8, sizeof(log_context)); + apple_rtkit_memcpy(rtk, rtk->syslog_msg_buffer, &rtk->syslog_buffer, + idx * entry_size + 8 + sizeof(log_context), + rtk->syslog_msg_size); + + log_context[sizeof(log_context) - 1] =3D 0; + rtk->syslog_msg_buffer[rtk->syslog_msg_size - 1] =3D 0; + rtk_info("syslog message: %s: %s", log_context, rtk->syslog_msg_buffer); + +done: + apple_rtkit_send_message(rtk, APPLE_RTKIT_EP_SYSLOG, msg); +} + +static void apple_rtkit_syslog_rx(struct apple_rtkit *rtk, u64 msg) +{ + u8 type =3D FIELD_GET(APPLE_RTKIT_SYSLOG_TYPE, msg); + + switch (type) { + case APPLE_RTKIT_BUFFER_REQUEST: + apple_rtkit_common_rx_get_buffer(rtk, &rtk->syslog_buffer, + APPLE_RTKIT_EP_SYSLOG, msg); + break; + case APPLE_RTKIT_SYSLOG_INIT: + apple_rtkit_syslog_rx_init(rtk, msg); + break; + case APPLE_RTKIT_SYSLOG_LOG: + apple_rtkit_syslog_rx_log(rtk, msg); + break; + default: + rtk_warn("Unknown syslog message: %llx\n", msg); + } +} + +static void apple_rtkit_oslog_rx_init(struct apple_rtkit *rtk, u64 msg) +{ + u64 ack; + + rtk_dbg("oslog init: msg: 0x%llx\n", msg); + ack =3D FIELD_PREP(APPLE_RTKIT_OSLOG_TYPE, APPLE_RTKIT_OSLOG_ACK); + apple_rtkit_send_message(rtk, APPLE_RTKIT_EP_OSLOG, ack); +} + +static void apple_rtkit_oslog_rx(struct apple_rtkit *rtk, u64 msg) +{ + u8 type =3D FIELD_GET(APPLE_RTKIT_OSLOG_TYPE, msg); + + switch (type) { + case APPLE_RTKIT_OSLOG_INIT: + apple_rtkit_oslog_rx_init(rtk, msg); + break; + default: + rtk_warn("Unknown oslog message: %llx\n", msg); + } +} + +static void apple_rtkit_rx(struct apple_rtkit *rtk, struct apple_mbox_msg = *msg) +{ + u8 ep =3D msg->msg1; + + if (!test_bit(ep, rtk->endpoints)) + rtk_warn("Message to undiscovered endpoint 0x%02x", ep); + + switch (ep) { + case APPLE_RTKIT_EP_MGMT: + apple_rtkit_management_rx(rtk, msg->msg0); + break; + case APPLE_RTKIT_EP_CRASHLOG: + apple_rtkit_crashlog_rx(rtk, msg->msg0); + break; + case APPLE_RTKIT_EP_SYSLOG: + apple_rtkit_syslog_rx(rtk, msg->msg0); + break; + case APPLE_RTKIT_EP_IOREPORT: + apple_rtkit_ioreport_rx(rtk, msg->msg0); + break; + case APPLE_RTKIT_EP_OSLOG: + apple_rtkit_oslog_rx(rtk, msg->msg0); + break; + case APPLE_RTKIT_APP_ENDPOINT_START ... 0xff: + rtk->ops->recv_message(rtk->cookie, ep, msg->msg0); + break; + default: + rtk_warn("message to unknown endpoint %02x: %llx\n", ep, + msg->msg0); + } +} + +static void apple_rtkit_do_reinit(struct apple_rtkit *rtk) +{ + apple_rtkit_free_buffer(rtk, &rtk->ioreport_buffer); + apple_rtkit_free_buffer(rtk, &rtk->crashlog_buffer); + apple_rtkit_free_buffer(rtk, &rtk->syslog_buffer); + + kfree(rtk->syslog_msg_buffer); + + rtk->syslog_msg_buffer =3D NULL; + rtk->syslog_n_entries =3D 0; + rtk->syslog_msg_size =3D 0; + + bitmap_zero(rtk->endpoints, APPLE_RTKIT_MAX_ENDPOINTS); + set_bit(APPLE_RTKIT_EP_MGMT, rtk->endpoints); + + reinit_completion(&rtk->epmap_completion); + reinit_completion(&rtk->iop_pwr_ack_completion); + reinit_completion(&rtk->ap_pwr_ack_completion); + + rtk->crashed =3D false; + rtk->iop_power_state =3D APPLE_RTKIT_PWR_STATE_OFF; + rtk->ap_power_state =3D APPLE_RTKIT_PWR_STATE_OFF; + + complete_all(&rtk->reinit_completion); +} + +static int apple_rtkit_worker(void *data) +{ + struct apple_rtkit *rtk =3D data; + struct apple_rtkit_work work; + + while (!kthread_should_stop()) { + wait_event_interruptible(rtk->wq, + kfifo_len(&rtk->work_fifo) > 0 || + kthread_should_stop()); + + if (kthread_should_stop()) + break; + + while (kfifo_out_spinlocked(&rtk->work_fifo, &work, 1, + &rtk->work_lock) =3D=3D 1) { + switch (work.type) { + case APPLE_RTKIT_WORK_MSG: + apple_rtkit_rx(rtk, &work.msg); + break; + case APPLE_RTKIT_WORK_REINIT: + apple_rtkit_do_reinit(rtk); + break; + } + } + } + + return 0; +} + +static void apple_rtkit_rx_callback(struct mbox_client *cl, void *mssg) +{ + struct apple_rtkit *rtk =3D container_of(cl, struct apple_rtkit, mbox_cl); + struct apple_mbox_msg *msg =3D mssg; + struct apple_rtkit_work work; + + dma_rmb(); + + memcpy(&work.msg, msg, sizeof(*msg)); + work.type =3D APPLE_RTKIT_WORK_MSG; + + kfifo_in_spinlocked(&rtk->work_fifo, &work, 1, &rtk->work_lock); + wake_up(&rtk->wq); +} + +int apple_rtkit_send_message(struct apple_rtkit *rtk, u8 ep, u64 message) +{ + struct apple_mbox_msg msg; + + if (rtk->crashed) + return -EINVAL; + if (ep >=3D APPLE_RTKIT_APP_ENDPOINT_START && + !apple_rtkit_is_running(rtk)) + return -EINVAL; + + msg.msg0 =3D (u64)message; + msg.msg1 =3D ep; + dma_wmb(); + + return mbox_send_message(rtk->mbox_chan, &msg); +} +EXPORT_SYMBOL_GPL(apple_rtkit_send_message); + +int apple_rtkit_start_ep(struct apple_rtkit *rtk, u8 endpoint) +{ + u64 msg; + + if (!test_bit(endpoint, rtk->endpoints)) + return -EINVAL; + if (endpoint >=3D APPLE_RTKIT_APP_ENDPOINT_START && + !apple_rtkit_is_running(rtk)) + return -EINVAL; + + msg =3D FIELD_PREP(APPLE_RTKIT_MGMT_STARTEP_EP, endpoint); + msg |=3D APPLE_RTKIT_MGMT_STARTEP_FLAG; + apple_rtkit_management_send(rtk, APPLE_RTKIT_MGMT_STARTEP, msg); + + return 0; +} +EXPORT_SYMBOL_GPL(apple_rtkit_start_ep); + +static int apple_rtkit_start_worker(struct apple_rtkit *rtk) +{ + rtk->task =3D kthread_run(apple_rtkit_worker, rtk, "%s-rtkit-worker", + dev_name(rtk->dev)); + if (IS_ERR(rtk->task)) + return PTR_ERR(rtk->task); + return 0; +} + +struct apple_rtkit *apple_rtkit_init(struct device *dev, void *cookie, + const char *mbox_name, int mbox_idx, + const struct apple_rtkit_ops *ops) +{ + struct apple_rtkit *rtk; + int ret; + + if (!ops) + return ERR_PTR(-EINVAL); + + rtk =3D kzalloc(sizeof(*rtk), GFP_KERNEL); + if (!rtk) + return ERR_PTR(-ENOMEM); + + rtk->dev =3D dev; + rtk->cookie =3D cookie; + rtk->ops =3D ops; + + INIT_KFIFO(rtk->work_fifo); + spin_lock_init(&rtk->work_lock); + init_waitqueue_head(&rtk->wq); + init_completion(&rtk->epmap_completion); + init_completion(&rtk->reinit_completion); + init_completion(&rtk->iop_pwr_ack_completion); + init_completion(&rtk->ap_pwr_ack_completion); + + bitmap_zero(rtk->endpoints, APPLE_RTKIT_MAX_ENDPOINTS); + set_bit(APPLE_RTKIT_EP_MGMT, rtk->endpoints); + + ret =3D apple_rtkit_start_worker(rtk); + if (ret) + return ERR_PTR(ret); + + rtk->mbox_cl.dev =3D dev; + rtk->mbox_cl.tx_block =3D true; + rtk->mbox_cl.knows_txdone =3D false; + rtk->mbox_cl.rx_callback =3D &apple_rtkit_rx_callback; + + if (mbox_name) + rtk->mbox_chan =3D + mbox_request_channel_byname(&rtk->mbox_cl, mbox_name); + else + rtk->mbox_chan =3D mbox_request_channel(&rtk->mbox_cl, mbox_idx); + + if (IS_ERR(rtk->mbox_chan)) + return (struct apple_rtkit *)rtk->mbox_chan; + + return rtk; +} +EXPORT_SYMBOL_GPL(apple_rtkit_init); + +static int apple_rtkit_wait_for_completion(struct completion *c) +{ + long t; + + t =3D wait_for_completion_interruptible_timeout(c, + msecs_to_jiffies(1000)); + if (t =3D=3D -ERESTARTSYS) + return t; + else if (t =3D=3D 0) + return -ETIME; + else + return 0; +} + +int apple_rtkit_reinit(struct apple_rtkit *rtk) +{ + struct apple_rtkit_work work; + + reinit_completion(&rtk->reinit_completion); + + work.type =3D APPLE_RTKIT_WORK_REINIT; + kfifo_in_spinlocked(&rtk->work_fifo, &work, 1, &rtk->work_lock); + wake_up(&rtk->wq); + + return apple_rtkit_wait_for_completion(&rtk->reinit_completion); +} +EXPORT_SYMBOL_GPL(apple_rtkit_reinit); + +static int apple_rtkit_set_ap_power_state(struct apple_rtkit *rtk, + unsigned int state) +{ + u64 msg; + int ret; + + reinit_completion(&rtk->ap_pwr_ack_completion); + + msg =3D FIELD_PREP(APPLE_RTKIT_MGMT_PWR_STATE, state); + apple_rtkit_management_send(rtk, APPLE_RTKIT_MGMT_SET_AP_PWR_STATE, + msg); + + ret =3D apple_rtkit_wait_for_completion(&rtk->ap_pwr_ack_completion); + if (ret) + return ret; + + if (rtk->ap_power_state !=3D state) + return -EINVAL; + return 0; +} + +static int apple_rtkit_set_iop_power_state(struct apple_rtkit *rtk, + unsigned int state) +{ + u64 msg; + int ret; + + reinit_completion(&rtk->iop_pwr_ack_completion); + + msg =3D FIELD_PREP(APPLE_RTKIT_MGMT_PWR_STATE, state); + apple_rtkit_management_send(rtk, APPLE_RTKIT_MGMT_SET_IOP_PWR_STATE, + msg); + + ret =3D apple_rtkit_wait_for_completion(&rtk->iop_pwr_ack_completion); + if (ret) + return ret; + + if (rtk->iop_power_state !=3D state) + return -EINVAL; + return 0; +} + +int apple_rtkit_boot(struct apple_rtkit *rtk) +{ + int ret; + + if (apple_rtkit_is_running(rtk)) + return 0; + if (rtk->crashed) + return -EINVAL; + + rtk_dbg("waiting for boot to finish\n"); + ret =3D apple_rtkit_wait_for_completion(&rtk->epmap_completion); + if (ret) + return ret; + if (rtk->boot_result) + return rtk->boot_result; + + rtk_dbg("waiting for IOP power state ACK\n"); + ret =3D apple_rtkit_wait_for_completion(&rtk->iop_pwr_ack_completion); + if (ret) + return ret; + + return apple_rtkit_set_ap_power_state(rtk, APPLE_RTKIT_PWR_STATE_ON); +} +EXPORT_SYMBOL_GPL(apple_rtkit_boot); + +int apple_rtkit_shutdown(struct apple_rtkit *rtk) +{ + int ret; + + /* if OFF is used here the co-processor will not wake up again */ + ret =3D apple_rtkit_set_ap_power_state(rtk, + APPLE_RTKIT_PWR_STATE_QUIESCED); + if (ret) + return ret; + + ret =3D apple_rtkit_set_iop_power_state(rtk, APPLE_RTKIT_PWR_STATE_SLEEP); + if (ret) + return ret; + + return apple_rtkit_reinit(rtk); +} +EXPORT_SYMBOL_GPL(apple_rtkit_shutdown); + +int apple_rtkit_hibernate(struct apple_rtkit *rtk) +{ + int ret; + + ret =3D apple_rtkit_set_ap_power_state(rtk, + APPLE_RTKIT_PWR_STATE_QUIESCED); + if (ret) + return ret; + + ret =3D apple_rtkit_set_iop_power_state(rtk, + APPLE_RTKIT_PWR_STATE_QUIESCED); + if (ret) + return ret; + + ret =3D apple_rtkit_reinit(rtk); + if (ret) + return ret; + + // TODO: apple_rtkit_reinit resets these so we have to restore them here = :/ + rtk->iop_power_state =3D APPLE_RTKIT_PWR_STATE_QUIESCED; + rtk->ap_power_state =3D APPLE_RTKIT_PWR_STATE_QUIESCED; + return 0; +} +EXPORT_SYMBOL_GPL(apple_rtkit_hibernate); + +int apple_rtkit_wake(struct apple_rtkit *rtk) +{ + u64 msg; + + if (apple_rtkit_is_running(rtk)) + return -EINVAL; + + reinit_completion(&rtk->iop_pwr_ack_completion); + + /* + * Use open-coded apple_rtkit_set_iop_power_state since apple_rtkit_boot + * will wait for the completion anyway. + */ + msg =3D FIELD_PREP(APPLE_RTKIT_MGMT_PWR_STATE, APPLE_RTKIT_PWR_STATE_ON); + apple_rtkit_management_send(rtk, APPLE_RTKIT_MGMT_SET_IOP_PWR_STATE, + msg); + + return apple_rtkit_boot(rtk); +} +EXPORT_SYMBOL_GPL(apple_rtkit_wake); + +void apple_rtkit_free(struct apple_rtkit *rtk) +{ + kthread_stop(rtk->task); + mbox_free_channel(rtk->mbox_chan); + + apple_rtkit_free_buffer(rtk, &rtk->ioreport_buffer); + apple_rtkit_free_buffer(rtk, &rtk->crashlog_buffer); + apple_rtkit_free_buffer(rtk, &rtk->syslog_buffer); + + kfree(rtk->syslog_msg_buffer); + kfree(rtk); +} +EXPORT_SYMBOL_GPL(apple_rtkit_free); + +struct apple_rtkit *devm_apple_rtkit_init(struct device *dev, void *cookie, + const char *mbox_name, int mbox_idx, + const struct apple_rtkit_ops *ops) +{ + struct apple_rtkit *rtk; + int ret; + + rtk =3D apple_rtkit_init(dev, cookie, mbox_name, mbox_idx, ops); + if (IS_ERR(rtk)) + return rtk; + + ret =3D devm_add_action_or_reset(dev, (void (*)(void *))apple_rtkit_free, + rtk); + if (ret) + return ERR_PTR(ret); + + return rtk; +} +EXPORT_SYMBOL_GPL(devm_apple_rtkit_init); + +MODULE_LICENSE("Dual MIT/GPL"); +MODULE_AUTHOR("Sven Peter "); +MODULE_DESCRIPTION("Apple RTKit driver"); diff --git a/include/linux/soc/apple/rtkit.h b/include/linux/soc/apple/rtki= t.h new file mode 100644 index 000000000000..a1beb514fff6 --- /dev/null +++ b/include/linux/soc/apple/rtkit.h @@ -0,0 +1,203 @@ +/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ +/* + * Apple RTKit IPC Library + * Copyright (C) The Asahi Linux Contributors + * + * Apple's SoCs come with various co-processors running their RTKit operat= ing + * system. This protocol library is used by client drivers to use the + * features provided by them. + */ +#ifndef _LINUX_APPLE_RTKIT_H_ +#define _LINUX_APPLE_RTKIT_H_ + +#include +#include +#include +#include + +/* + * Struct to represent implementation-specific RTKit operations. + * + * @buffer: Shared memory buffer allocated inside normal RAM. + * @iomem: Shared memory buffer controlled by the co-processors. + * @size: Size of the shared memory buffer. + * @iova: Device VA of shared memory buffer. + * @is_mapped: Shared memory buffer is managed by the co-processor. + */ + +struct apple_rtkit_shmem { + void *buffer; + void __iomem *iomem; + size_t size; + dma_addr_t iova; + bool is_mapped; +}; + +/* + * Struct to represent implementation-specific RTKit operations. + * + * @crashed: Called when the co-processor has crashed. + * @recv_message: Function called when a message from RTKit is received + * on a non-system endpoint. Called from a worker thread. + * @shmem_setup: Setup shared memory buffer. If bfr.is_iomem is true the + * buffer is managed by the co-processor and needs to be m= apped. + * Otherwise the buffer is managed by Linux and needs to be + * allocated. If not specified dma_alloc_coherent is used. + * @shmem_destroy: Undo the shared memory buffer setup in shmem_setup. If = not + * specified dma_free_coherent is used. + */ +struct apple_rtkit_ops { + void (*crashed)(void *cookie); + void (*recv_message)(void *cookie, u8 endpoint, u64 message); + int (*shmem_setup)(void *cookie, struct apple_rtkit_shmem *bfr, + dma_addr_t addr, size_t len); + void (*shmem_destroy)(void *cookie, struct apple_rtkit_shmem *bfr); +}; + +struct apple_rtkit; + +#if IS_ENABLED(CONFIG_APPLE_RTKIT) + +/* + * Initializes the internal state required to handle RTKit. This + * should usually be called within _probe. + * + * @dev: Pointer to the device node this coprocessor is assocated with + * @cookie: opaque cookie passed to all functions defined in rtkit_ops + * @mbox_name: mailbox name used to communicate with the co-processor + * @mbox_idx: mailbox index to be used if mbox_name is NULL + * @ops: pointer to rtkit_ops to be used for this co-processor + */ +struct apple_rtkit *apple_rtkit_init(struct device *dev, void *cookie, + const char *mbox_name, int mbox_idx, + const struct apple_rtkit_ops *ops); + +/* + * Dev-res managed version of apple_rtkit_init. + */ +struct apple_rtkit *devm_apple_rtkit_init(struct device *dev, void *cookie, + const char *mbox_name, int mbox_idx, + const struct apple_rtkit_ops *ops); + +/* + * Free internal structures. + */ +void apple_rtkit_free(struct apple_rtkit *rtk); + +/* + * Reinitialize internal structures. Must only be called with the co-proce= ssor + * is held in reset. + */ +int apple_rtkit_reinit(struct apple_rtkit *rtk); + +/* + * Handle RTKit's boot process. Should be called after the CPU of the + * co-processor has been started. + */ +int apple_rtkit_boot(struct apple_rtkit *rtk); + +/* + * Hibernate the co-processor. + */ +int apple_rtkit_hibernate(struct apple_rtkit *rtk); + +/* + * Wake the co-processor up from hibernation mode. + */ +int apple_rtkit_wake(struct apple_rtkit *rtk); + +/* + * Shutdown the co-processor + */ +int apple_rtkit_shutdown(struct apple_rtkit *rtk); + +/* + * Checks if RTKit is running and ready to handle messages. + */ +bool apple_rtkit_is_running(struct apple_rtkit *rtk); + +/* + * Checks if RTKit has crashed. + */ +bool apple_rtkit_is_crashed(struct apple_rtkit *rtk); + +/* + * Starts an endpoint. Must be called after boot but before any messages c= an be + * sent or received from that endpoint. + */ +int apple_rtkit_start_ep(struct apple_rtkit *rtk, u8 endpoint); + +/* + * Send a message to the given endpoint. + */ +int apple_rtkit_send_message(struct apple_rtkit *rtk, u8 ep, u64 message); + +#else + +static inline struct apple_rtkit * +apple_rtkit_init(struct device *dev, void *cookie, const char *mbox_name, + int mbox_idx, const struct apple_rtkit_ops *ops) +{ + return ERR_PTR(-ENODEV); +} + +static inline struct apple_rtkit * +devm_apple_rtkit_init(struct device *dev, void *cookie, const char *mbox_n= ame, + int mbox_idx, const struct apple_rtkit_ops *ops) +{ + return ERR_PTR(-ENODEV); +} + +static inline void apple_rtkit_free(struct apple_rtkit *rtk) +{ +} + +static inline int apple_rtkit_reinit(struct apple_rtkit *rtk) +{ + return -ENODEV; +} + +static inline int apple_rtkit_boot(struct apple_rtkit *rtk) +{ + return -ENODEV; +} + +static inline int apple_rtkit_hibernate(struct apple_rtkit *rtk) +{ + return -ENODEV; +} + +static inline int apple_rtkit_wake(struct apple_rtkit *rtk) +{ + return -ENODEV; +} + +static inline int apple_rtkit_shutdown(struct apple_rtkit *rtk) +{ + return -ENODEV; +} + +static inline bool apple_rtkit_is_running(struct apple_rtkit *rtk) +{ + return false; +} + +static inline bool apple_rtkit_is_crashed(struct apple_rtkit *rtk) +{ + return false; +} + +static inline int apple_rtkit_start_ep(struct apple_rtkit *rtk, u8 endpoin= t) +{ + return -ENODEV; +} + +static inline int apple_rtkit_send_message(struct apple_rtkit *rtk, u8 ep, + u64 message) +{ + return -ENODEV; +} + +#endif /* IS_ENABLED(CONFIG_APPLE_RTKIT) */ + +#endif /* _LINUX_APPLE_RTKIT_H_ */ --=20 2.25.1 From nobody Mon Jun 22 15:42:51 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5905C433FE for ; Mon, 21 Mar 2022 16:52:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351487AbiCUQx7 (ORCPT ); Mon, 21 Mar 2022 12:53:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52586 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1351467AbiCUQxu (ORCPT ); Mon, 21 Mar 2022 12:53:50 -0400 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 04819186893 for ; Mon, 21 Mar 2022 09:52:18 -0700 (PDT) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id 15AA93201DFB; Mon, 21 Mar 2022 12:52:17 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Mon, 21 Mar 2022 12:52:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=svenpeter.dev; h=cc:cc:content-transfer-encoding:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm1; bh=+GxCEVkHec55pC AMUveltxl9mi/nio9ez9UhFD9sPw0=; b=opFneBZPvcfTeXh68hgCfstuMYdR7i kReKAVVxbLnKad7Fnf72DGR/vMjE+yxjxVe3UVoOwW9syKPdLetHrIuwa9USBbhY zok51rkTZGADAWtJmi72J84hM7vgABUEg9eo7vJBMmgzCi/UF4ncDdM6UkToRCNI P+OjVF6FAAgqNKUCOlcv4bT/jhKVB0QwUzjZjfl+BbTjx9sure9LhiQv2wo3NRKx uXPDgt78GFUIHumiAjkHhI5SvRaAq5Ysoeq7KacLexyY/9+mR86c9RHogzOSCMVc Hh3Q31XS12Y1wd8sm+MzpGHz2LPbqsU+xh60jGp5lBRyLTHBhcnvWIZw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=+GxCEV kHec55pCAMUveltxl9mi/nio9ez9UhFD9sPw0=; b=ZjNFY+MHWjI4WEdi1H6mWr ODL7PV5FleVwlL+KQLyEMwel8pmpyKvPUoi88Qs1QxNWGaI5/N8vIccDD2HIN2J8 ZxKEjt0r5G6zo4MMxVznygVKh7R9vInIMODQZ5e3r4vnEhgHaF8uqfJ4Q9EcmRuh 7Vzo8CwcgQpiAJH5I+us3wjFh2MOJ7bZoT/QBSKq/GBJyPWn6cmdFJx6F4J6ZbEG WlkfjGu7/da6qgSpZZzQlkuJW0vClBdAkZ/wi8ixsgl7tZpVYqREsObvm/+MOjfc k4jpKLw2JjvVEOdKsZJkZ9aYic6+3s/Gn+6ja4/X8u3ivaYD6zN7rqpw7YFFD0WA == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrudegfedgleehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefuvhgvnhcu rfgvthgvrhcuoehsvhgvnhesshhvvghnphgvthgvrhdruggvvheqnecuggftrfgrthhtvg hrnhepjefgjeefheeuiefgheehieekhfffgfetleeihefhtddutdegffdtjeekueffgeek necuffhomhgrihhnpehgihhthhhusgdrtghomhenucevlhhushhtvghrufhiiigvpedtne curfgrrhgrmhepmhgrihhlfhhrohhmpehsvhgvnhesshhvvghnphgvthgvrhdruggvvh X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 21 Mar 2022 12:52:13 -0400 (EDT) From: Sven Peter To: Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg Cc: Sven Peter , Hector Martin , Alyssa Rosenzweig , Rob Herring , Arnd Bergmann , Marc Zyngier , devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org Subject: [PATCH 6/9] nvme-apple: Add initial Apple SoC NVMe driver Date: Mon, 21 Mar 2022 17:50:46 +0100 Message-Id: <20220321165049.35985-7-sven@svenpeter.dev> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20220321165049.35985-1-sven@svenpeter.dev> References: <20220321165049.35985-1-sven@svenpeter.dev> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Apple SoCs such as the M1 come with an embedded NVMe controller that is not attached to any PCIe bus. Additionally, it doesn't confirm to the NVMe specification and requires a bunch of changes to command submission and IOMMU configuration to work. Signed-off-by: Sven Peter --- MAINTAINERS | 1 + drivers/nvme/host/Kconfig | 12 + drivers/nvme/host/Makefile | 3 + drivers/nvme/host/apple.c | 1456 ++++++++++++++++++++++++++++++++++++ 4 files changed, 1472 insertions(+) create mode 100644 drivers/nvme/host/apple.c diff --git a/MAINTAINERS b/MAINTAINERS index 3d37fe7a0408..0a493298b6ff 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1781,6 +1781,7 @@ F: drivers/i2c/busses/i2c-pasemi-core.c F: drivers/i2c/busses/i2c-pasemi-platform.c F: drivers/irqchip/irq-apple-aic.c F: drivers/mailbox/apple-mailbox.c +F: drivers/nvme/host/apple.c F: drivers/pinctrl/pinctrl-apple-gpio.c F: drivers/soc/apple/* F: drivers/watchdog/apple_wdt.c diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig index dc0450ca23a3..439848a4b685 100644 --- a/drivers/nvme/host/Kconfig +++ b/drivers/nvme/host/Kconfig @@ -83,3 +83,15 @@ config NVME_TCP from https://github.com/linux-nvme/nvme-cli. =20 If unsure, say N. + +config NVME_APPLE + tristate "Apple ANS2 NVM Express host driver" + depends on OF && BLOCK + depends on (APPLE_RTKIT && APPLE_SART && ARCH_APPLE) || COMPILE_TEST + select NVME_CORE + help + This provides support for the NVMe controller embedded in Apple SoCs + such as the M1. + + To compile this driver as a module, choose M here: the + module will be called nvme-apple. diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile index dfaacd472e5d..2927820c70a3 100644 --- a/drivers/nvme/host/Makefile +++ b/drivers/nvme/host/Makefile @@ -8,6 +8,7 @@ obj-$(CONFIG_NVME_FABRICS) +=3D nvme-fabrics.o obj-$(CONFIG_NVME_RDMA) +=3D nvme-rdma.o obj-$(CONFIG_NVME_FC) +=3D nvme-fc.o obj-$(CONFIG_NVME_TCP) +=3D nvme-tcp.o +obj-$(CONFIG_NVME_APPLE) +=3D nvme-apple.o =20 nvme-core-y :=3D core.o ioctl.o nvme-core-$(CONFIG_TRACING) +=3D trace.o @@ -25,3 +26,5 @@ nvme-rdma-y +=3D rdma.o nvme-fc-y +=3D fc.o =20 nvme-tcp-y +=3D tcp.o + +nvme-apple-y +=3D apple.o diff --git a/drivers/nvme/host/apple.c b/drivers/nvme/host/apple.c new file mode 100644 index 000000000000..587d6c7014a0 --- /dev/null +++ b/drivers/nvme/host/apple.c @@ -0,0 +1,1456 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Apple ANS NVM Express device driver + * Copyright The Asahi Linux Contributors + * + * Based on the pci.c NVM Express device driver + * Copyright (c) 2011-2014, Intel Corporation. + * and on the rdma.c NVMe over Fabrics RDMA host code. + * Copyright (c) 2015-2016 HGST, a Western Digital Company. + */ + +//#define DEBUG + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "nvme.h" + +#define APPLE_ANS_BOOT_TIMEOUT USEC_PER_SEC +#define APPLE_ANS_MAX_QUEUE_DEPTH 64 + +#define APPLE_ANS_COPROC_CPU_CONTROL 0x44 +#define APPLE_ANS_COPROC_CPU_CONTROL_RUN BIT(4) + +#define APPLE_ANS_ACQ_DB 0x1004 +#define APPLE_ANS_IOCQ_DB 0x100c + +#define APPLE_ANS_MAX_PEND_CMDS_CTRL 0x1210 + +#define APPLE_ANS_BOOT_STATUS 0x1300 +#define APPLE_ANS_BOOT_STATUS_OK 0xde71ce55 + +#define APPLE_ANS_UNKNOWN_CTRL 0x24008 +#define APPLE_ANS_PRP_NULL_CHECK BIT(11) + +#define APPLE_ANS_LINEAR_SQ_CTRL 0x24908 +#define APPLE_ANS_LINEAR_SQ_EN BIT(0) + +#define APPLE_ANS_LINEAR_ASQ_DB 0x2490c +#define APPLE_ANS_LINEAR_IOSQ_DB 0x24910 + +#define APPLE_NVMMU_NUM_TCBS 0x28100 +#define APPLE_NVMMU_ASQ_TCB_BASE 0x28108 +#define APPLE_NVMMU_IOSQ_TCB_BASE 0x28110 +#define APPLE_NVMMU_TCB_INVAL 0x28118 +#define APPLE_NVMMU_TCB_STAT 0x28120 + +/* NVM Express NVM Command Set Specification, Revision 1.0a, Figure 18 */ +#define NVME_OPCODE_DATA_XFER_HOST_TO_CTRL BIT(0) +#define NVME_OPCODE_DATA_XFER_CTRL_TO_HOST BIT(1) + +/* + * This controller is a bit weird in the way command tags works: Both the + * admin and the IO queue share the same tag space. Additionally, tags + * cannot be higher than 0x40 which effectively limits the combined + * queue depth to 0x40. Instead of wasting half of that on the admin queue + * which gets much less traffic we instead reduce its size here. + * The controller also doesn't support async event such that no space must + * be reserved for NVME_NR_AEN_COMMANDS. + */ +#define APPLE_NVME_AQ_DEPTH 2 +#define APPLE_NVME_AQ_MQ_TAG_DEPTH (APPLE_NVME_AQ_DEPTH - 1) + +/* + * These can be higher, but we need to ensure that any command doesn't + * require an sg allocation that needs more than a page of data. + */ +#define NVME_MAX_KB_SZ 4096 +#define NVME_MAX_SEGS 127 + +/* + * This controller comes with an embedded IOMMU known as NVMMU. + * The NVMMU is pointed to an array of TCBs indexed by the command tag. + * Each command must be configured inside this structure before it's allow= ed + * to execute, including commands that don't require DMA transfers. + * + * An exception to this are Apple's vendor-specific commands (opcode 0xD8 = on the + * admin queue): Those commands must still be added to the NVMMU but the D= MA + * buffers cannot be represented as PRPs and must instead be allowed using= SART. + * + * Programming the PRPs to the same values as those in the submission queue + * looks rather silly at first. This hardware is however designed for a ke= rnel + * that runs the NVMMU code in a higher exception level than the NVMe driv= er. + * In that setting the NVMe driver first programs the submission queue ent= ry + * and then executes a hypercall to the code that is allowed to program the + * NVMMU. The NVMMU driver then creates a shadow copy of the PRPs while + * verifying that they don't point to kernel text, data, pagetables, or si= milar + * protected areas before programming the TCB to point to this shadow copy. + * Since Linux doesn't do any of that we may as well just point both the q= ueue + * and the TCB PRP pointer to the same memory. + */ +struct apple_nvmmu_tcb { + u8 opcode; + +#define APPLE_ANS_TCB_DMA_FROM_DEVICE BIT(0) +#define APPLE_ANS_TCB_DMA_TO_DEVICE BIT(1) + u8 dma_flags; + + u8 command_id; + u8 _unk0; + u32 length; + u8 _unk1[16]; + u64 prp1; + u64 prp2; + u8 _unk2[16]; + u8 aes_iv[8]; + u8 _aes_unk[64]; +}; + +/* + * The Apple NVMe controller only supports a single admin and a single IO = queue + * which are both limited to 64 entries and share a single interrupt. + * + * The completion queue works as usual. The submission "queue" instead is + * an array indexed by the command tag on this hardware. Commands must als= o be + * present in the NVMMU's tcb array. They are triggered by writing their t= ag to + * a MMIO register. + */ +struct apple_nvme_queue { + struct nvme_command *sqes; + struct nvme_completion *cqes; + struct apple_nvmmu_tcb *tcbs; + + dma_addr_t sq_dma_addr; + dma_addr_t cq_dma_addr; + dma_addr_t tcb_dma_addr; + + u32 __iomem *sq_db; + u32 __iomem *cq_db; + + u16 cq_head; + u8 cq_phase; + + bool is_adminq; + bool enabled; +}; + +/* + * The apple_nvme_iod describes the data in an I/O. + * + * The sg pointer contains the list of PRP chunk allocations in addition + * to the actual struct scatterlist. + */ +struct apple_nvme_iod { + struct nvme_request req; + struct nvme_command cmd; + struct apple_nvme_queue *q; + int npages; /* In the PRP list. 0 means small pool in use */ + int nents; /* Used in scatterlist */ + dma_addr_t first_dma; + unsigned int dma_len; /* length of single DMA segment mapping */ + struct scatterlist *sg; +}; + +struct apple_nvme { + struct device *dev; + + void __iomem *mmio_coproc; + void __iomem *mmio_nvme; + + struct apple_sart *sart; + struct apple_rtkit *rtk; + struct reset_control *reset; + + struct dma_pool *prp_page_pool; + struct dma_pool *prp_small_pool; + mempool_t *iod_mempool; + + struct nvme_ctrl ctrl; + struct work_struct remove_work; + + struct apple_nvme_queue adminq; + struct apple_nvme_queue ioq; + + struct blk_mq_tag_set admin_tagset; + struct blk_mq_tag_set tagset; + + int irq; + spinlock_t lock; +}; + +static_assert(sizeof(struct nvme_command) =3D=3D 64); +static_assert(sizeof(struct apple_nvmmu_tcb) =3D=3D 128); + +static inline struct apple_nvme *ctrl_to_apple_nvme(struct nvme_ctrl *ctrl) +{ + return container_of(ctrl, struct apple_nvme, ctrl); +} + +static inline struct apple_nvme *queue_to_apple_nvme(struct apple_nvme_que= ue *q) +{ + if (q->is_adminq) + return container_of(q, struct apple_nvme, adminq); + else + return container_of(q, struct apple_nvme, ioq); +} + +static unsigned int apple_nvme_queue_depth(struct apple_nvme_queue *q) +{ + if (q->is_adminq) + return APPLE_NVME_AQ_DEPTH; + else + return APPLE_ANS_MAX_QUEUE_DEPTH; +} + +static void apple_nvme_rtkit_crashed(void *cookie) +{ + struct apple_nvme *anv =3D cookie; + + dev_warn(anv->dev, "RTKit crashed; unable to recover without a reboot"); + nvme_reset_ctrl(&anv->ctrl); +} + +static void apple_nvme_rtkit_recv(void *cookie, u8 endpoint, u64 message) +{ + struct apple_nvme *anv =3D cookie; + + dev_warn(anv->dev, "Received unexpected message to EP%02d: %llx", + endpoint, message); +} + +static int apple_nvme_sart_dma_setup(void *cookie, struct apple_rtkit_shme= m *bfr, + dma_addr_t iova, size_t size) +{ + struct apple_nvme *anv =3D cookie; + int ret; + + if (iova) + return -EINVAL; + + bfr->buffer =3D dma_alloc_coherent(anv->dev, size, &iova, GFP_KERNEL); + if (!bfr->buffer) + return -ENOMEM; + + ret =3D apple_sart_add_allowed_region(anv->sart, iova, size); + if (ret) { + dma_free_coherent(anv->dev, size, bfr->buffer, iova); + bfr->buffer =3D NULL; + return -ENOMEM; + } + + bfr->size =3D size; + bfr->iova =3D iova; + + return 0; +} + +static void apple_nvme_sart_dma_destroy(void *cookie, struct apple_rtkit_s= hmem *bfr) +{ + struct apple_nvme *anv =3D cookie; + + apple_sart_remove_allowed_region(anv->sart, bfr->iova, bfr->size); + dma_free_coherent(anv->dev, bfr->size, bfr->buffer, bfr->iova); +} + +static const struct apple_rtkit_ops apple_nvme_rtkit_ops =3D { + .crashed =3D apple_nvme_rtkit_crashed, + .recv_message =3D apple_nvme_rtkit_recv, + .shmem_setup =3D apple_nvme_sart_dma_setup, + .shmem_destroy =3D apple_nvme_sart_dma_destroy, +}; + +static void apple_nvmmu_inval(struct apple_nvme_queue *q, unsigned int tag) +{ + struct apple_nvme *anv =3D queue_to_apple_nvme(q); + + writel(tag, anv->mmio_nvme + APPLE_NVMMU_TCB_INVAL); + if (readl_relaxed(anv->mmio_nvme + APPLE_NVMMU_TCB_STAT)) + dev_warn(anv->dev, "NVMMU TCB invalidation failed\n"); +} + +static void apple_nvme_submit_cmd(struct apple_nvme_queue *q, + struct nvme_command *cmd) +{ + u32 tag =3D nvme_tag_from_cid(cmd->common.command_id); + struct apple_nvmmu_tcb *tcb =3D &q->tcbs[tag]; + + tcb->opcode =3D cmd->common.opcode; + tcb->prp1 =3D cmd->common.dptr.prp1; + tcb->prp2 =3D cmd->common.dptr.prp2; + tcb->length =3D cmd->rw.length; + tcb->command_id =3D tag; + tcb->dma_flags =3D 0; + + if (cmd->common.opcode & NVME_OPCODE_DATA_XFER_HOST_TO_CTRL) + tcb->dma_flags |=3D APPLE_ANS_TCB_DMA_TO_DEVICE; + if (cmd->common.opcode & NVME_OPCODE_DATA_XFER_CTRL_TO_HOST) + tcb->dma_flags |=3D APPLE_ANS_TCB_DMA_FROM_DEVICE; + + memcpy(&q->sqes[tag], cmd, sizeof(*cmd)); + writel(tag, q->sq_db); +} + +/* + * From pci.c: + * Will slightly overestimate the number of pages needed. This is OK + * as it only leads to a small amount of wasted memory for the lifetime of + * the I/O. + */ +static inline size_t apple_nvme_iod_alloc_size(void) +{ + const unsigned int nprps =3D DIV_ROUND_UP( + NVME_MAX_KB_SZ + NVME_CTRL_PAGE_SIZE, NVME_CTRL_PAGE_SIZE); + const int npages =3D DIV_ROUND_UP(8 * nprps, PAGE_SIZE - 8); + const size_t alloc_size =3D sizeof(__le64 *) * npages + + sizeof(struct scatterlist) * NVME_MAX_SEGS; + + return alloc_size; +} + +static void **apple_nvme_iod_list(struct request *req) +{ + struct apple_nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + + return (void **)(iod->sg + blk_rq_nr_phys_segments(req)); +} + +static void apple_nvme_free_prps(struct apple_nvme *anv, struct request *r= eq) +{ + const int last_prp =3D NVME_CTRL_PAGE_SIZE / sizeof(__le64) - 1; + struct apple_nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + dma_addr_t dma_addr =3D iod->first_dma; + int i; + + for (i =3D 0; i < iod->npages; i++) { + __le64 *prp_list =3D apple_nvme_iod_list(req)[i]; + dma_addr_t next_dma_addr =3D prp_list[last_prp]; + + dma_pool_free(anv->prp_page_pool, prp_list, dma_addr); + dma_addr =3D next_dma_addr; + } +} + +static void apple_nvme_unmap_data(struct apple_nvme *anv, struct request *= req) +{ + struct apple_nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + + if (iod->dma_len) { + dma_unmap_page(anv->dev, iod->first_dma, iod->dma_len, + rq_dma_dir(req)); + return; + } + + WARN_ON_ONCE(!iod->nents); + + dma_unmap_sg(anv->dev, iod->sg, iod->nents, rq_dma_dir(req)); + if (iod->npages =3D=3D 0) + dma_pool_free(anv->prp_small_pool, apple_nvme_iod_list(req)[0], + iod->first_dma); + else + apple_nvme_free_prps(anv, req); + mempool_free(iod->sg, anv->iod_mempool); +} + +static void apple_nvme_print_sgl(struct scatterlist *sgl, int nents) +{ + int i; + struct scatterlist *sg; + + for_each_sg(sgl, sg, nents, i) { + dma_addr_t phys =3D sg_phys(sg); + + pr_warn("sg[%d] phys_addr:%pad offset:%d length:%d dma_address:%pad dma_= length:%d\n", + i, &phys, sg->offset, sg->length, &sg_dma_address(sg), + sg_dma_len(sg)); + } +} + +static blk_status_t apple_nvme_setup_prps(struct apple_nvme *anv, + struct request *req, + struct nvme_rw_command *cmnd) +{ + struct apple_nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + struct dma_pool *pool; + int length =3D blk_rq_payload_bytes(req); + struct scatterlist *sg =3D iod->sg; + int dma_len =3D sg_dma_len(sg); + u64 dma_addr =3D sg_dma_address(sg); + int offset =3D dma_addr & (NVME_CTRL_PAGE_SIZE - 1); + __le64 *prp_list; + void **list =3D apple_nvme_iod_list(req); + dma_addr_t prp_dma; + int nprps, i; + + length -=3D (NVME_CTRL_PAGE_SIZE - offset); + if (length <=3D 0) { + iod->first_dma =3D 0; + goto done; + } + + dma_len -=3D (NVME_CTRL_PAGE_SIZE - offset); + if (dma_len) { + dma_addr +=3D (NVME_CTRL_PAGE_SIZE - offset); + } else { + sg =3D sg_next(sg); + dma_addr =3D sg_dma_address(sg); + dma_len =3D sg_dma_len(sg); + } + + if (length <=3D NVME_CTRL_PAGE_SIZE) { + iod->first_dma =3D dma_addr; + goto done; + } + + nprps =3D DIV_ROUND_UP(length, NVME_CTRL_PAGE_SIZE); + if (nprps <=3D (256 / 8)) { + pool =3D anv->prp_small_pool; + iod->npages =3D 0; + } else { + pool =3D anv->prp_page_pool; + iod->npages =3D 1; + } + + prp_list =3D dma_pool_alloc(pool, GFP_ATOMIC, &prp_dma); + if (!prp_list) { + iod->first_dma =3D dma_addr; + iod->npages =3D -1; + return BLK_STS_RESOURCE; + } + list[0] =3D prp_list; + iod->first_dma =3D prp_dma; + i =3D 0; + for (;;) { + if (i =3D=3D NVME_CTRL_PAGE_SIZE >> 3) { + __le64 *old_prp_list =3D prp_list; + + prp_list =3D dma_pool_alloc(pool, GFP_ATOMIC, &prp_dma); + if (!prp_list) + goto free_prps; + list[iod->npages++] =3D prp_list; + prp_list[0] =3D old_prp_list[i - 1]; + old_prp_list[i - 1] =3D prp_dma; + i =3D 1; + } + prp_list[i++] =3D dma_addr; + dma_len -=3D NVME_CTRL_PAGE_SIZE; + dma_addr +=3D NVME_CTRL_PAGE_SIZE; + length -=3D NVME_CTRL_PAGE_SIZE; + if (length <=3D 0) + break; + if (dma_len > 0) + continue; + if (unlikely(dma_len < 0)) + goto bad_sgl; + sg =3D sg_next(sg); + dma_addr =3D sg_dma_address(sg); + dma_len =3D sg_dma_len(sg); + } +done: + cmnd->dptr.prp1 =3D sg_dma_address(iod->sg); + cmnd->dptr.prp2 =3D iod->first_dma; + return BLK_STS_OK; +free_prps: + apple_nvme_free_prps(anv, req); + return BLK_STS_RESOURCE; +bad_sgl: + WARN(DO_ONCE(apple_nvme_print_sgl, iod->sg, iod->nents), + "Invalid SGL for payload:%d nents:%d\n", blk_rq_payload_bytes(req), + iod->nents); + return BLK_STS_IOERR; +} + +static blk_status_t apple_nvme_setup_prp_simple(struct apple_nvme *anv, + struct request *req, + struct nvme_rw_command *cmnd, + struct bio_vec *bv) +{ + struct apple_nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + unsigned int offset =3D bv->bv_offset & (NVME_CTRL_PAGE_SIZE - 1); + unsigned int first_prp_len =3D NVME_CTRL_PAGE_SIZE - offset; + + iod->first_dma =3D dma_map_bvec(anv->dev, bv, rq_dma_dir(req), 0); + if (dma_mapping_error(anv->dev, iod->first_dma)) + return BLK_STS_RESOURCE; + iod->dma_len =3D bv->bv_len; + + cmnd->dptr.prp1 =3D iod->first_dma; + if (bv->bv_len > first_prp_len) + cmnd->dptr.prp2 =3D iod->first_dma + first_prp_len; + return BLK_STS_OK; +} + +static blk_status_t apple_nvme_map_data(struct apple_nvme *anv, + struct request *req, + struct nvme_command *cmnd) +{ + struct apple_nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + blk_status_t ret =3D BLK_STS_RESOURCE; + int nr_mapped; + + if (blk_rq_nr_phys_segments(req) =3D=3D 1) { + struct bio_vec bv =3D req_bvec(req); + + if (bv.bv_offset + bv.bv_len <=3D NVME_CTRL_PAGE_SIZE * 2) + return apple_nvme_setup_prp_simple(anv, req, &cmnd->rw, + &bv); + } + + iod->dma_len =3D 0; + iod->sg =3D mempool_alloc(anv->iod_mempool, GFP_ATOMIC); + if (!iod->sg) + return BLK_STS_RESOURCE; + sg_init_table(iod->sg, blk_rq_nr_phys_segments(req)); + iod->nents =3D blk_rq_map_sg(req->q, req, iod->sg); + if (!iod->nents) + goto out_free_sg; + + nr_mapped =3D dma_map_sg_attrs(anv->dev, iod->sg, iod->nents, + rq_dma_dir(req), DMA_ATTR_NO_WARN); + if (!nr_mapped) + goto out_free_sg; + + ret =3D apple_nvme_setup_prps(anv, req, &cmnd->rw); + if (ret !=3D BLK_STS_OK) + goto out_unmap_sg; + return BLK_STS_OK; + +out_unmap_sg: + dma_unmap_sg(anv->dev, iod->sg, iod->nents, rq_dma_dir(req)); +out_free_sg: + mempool_free(iod->sg, anv->iod_mempool); + return ret; +} + +static __always_inline void apple_nvme_unmap_rq(struct request *req) +{ + struct apple_nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + struct apple_nvme *anv =3D queue_to_apple_nvme(iod->q); + + if (blk_rq_nr_phys_segments(req)) + apple_nvme_unmap_data(anv, req); +} + +static void apple_nvme_complete_rq(struct request *req) +{ + apple_nvme_unmap_rq(req); + nvme_complete_rq(req); +} + +static void apple_nvme_complete_batch(struct io_comp_batch *iob) +{ + nvme_complete_batch(iob, apple_nvme_unmap_rq); +} + +static inline bool apple_nvme_cqe_pending(struct apple_nvme_queue *q) +{ + struct nvme_completion *hcqe =3D &q->cqes[q->cq_head]; + + return (READ_ONCE(hcqe->status) & 1) =3D=3D q->cq_phase; +} + +static inline struct blk_mq_tags * +apple_nvme_queue_tagset(struct apple_nvme *anv, struct apple_nvme_queue *q) +{ + if (q->is_adminq) + return anv->admin_tagset.tags[0]; + else + return anv->tagset.tags[0]; +} + +static inline void apple_nvme_handle_cqe(struct apple_nvme_queue *q, + struct io_comp_batch *iob, u16 idx) +{ + struct apple_nvme *anv =3D queue_to_apple_nvme(q); + struct nvme_completion *cqe =3D &q->cqes[idx]; + __u16 command_id =3D READ_ONCE(cqe->command_id); + struct request *req; + + apple_nvmmu_inval(q, command_id); + + req =3D nvme_find_rq(apple_nvme_queue_tagset(anv, q), command_id); + if (unlikely(!req)) { + dev_warn(anv->dev, "invalid id %d completed", command_id); + return; + } + + if (!nvme_try_complete_req(req, cqe->status, cqe->result) && + !blk_mq_add_to_batch(req, iob, nvme_req(req)->status, + apple_nvme_complete_batch)) + apple_nvme_complete_rq(req); +} + +static inline void apple_nvme_update_cq_head(struct apple_nvme_queue *q) +{ + u32 tmp =3D q->cq_head + 1; + + if (tmp =3D=3D apple_nvme_queue_depth(q)) { + q->cq_head =3D 0; + q->cq_phase ^=3D 1; + } else { + q->cq_head =3D tmp; + } +} + +static bool apple_nvme_poll_cq(struct apple_nvme_queue *q, + struct io_comp_batch *iob) +{ + bool found =3D false; + + while (apple_nvme_cqe_pending(q)) { + found =3D true; + + /* + * load-load control dependency between phase and the rest of + * the cqe requires a full read memory barrier + */ + dma_rmb(); + apple_nvme_handle_cqe(q, iob, q->cq_head); + apple_nvme_update_cq_head(q); + } + + if (found) + writel_relaxed(q->cq_head, q->cq_db); + + return found; +} + +static bool apple_nvme_handle_cq(struct apple_nvme_queue *q, bool force) +{ + bool found; + DEFINE_IO_COMP_BATCH(iob); + + if (!READ_ONCE(q->enabled) && !force) + return false; + + found =3D apple_nvme_poll_cq(q, &iob); + + if (!rq_list_empty(iob.req_list)) + apple_nvme_complete_batch(&iob); + + return found; +} + +static irqreturn_t apple_nvme_irq(int irq, void *data) +{ + struct apple_nvme *anv =3D data; + bool handled =3D false; + unsigned long flags; + + spin_lock_irqsave(&anv->lock, flags); + if (apple_nvme_handle_cq(&anv->ioq, false)) + handled =3D true; + if (apple_nvme_handle_cq(&anv->adminq, false)) + handled =3D true; + spin_unlock_irqrestore(&anv->lock, flags); + + if (handled) + return IRQ_HANDLED; + return IRQ_NONE; +} + +static int apple_nvme_create_cq(struct apple_nvme *anv) +{ + struct nvme_command c =3D {}; + + /* + * Note: we (ab)use the fact that the prp fields survive if no data + * is attached to the request. + */ + c.create_cq.opcode =3D nvme_admin_create_cq; + c.create_cq.prp1 =3D anv->ioq.cq_dma_addr; + c.create_cq.cqid =3D 1; + c.create_cq.qsize =3D APPLE_ANS_MAX_QUEUE_DEPTH - 1; + c.create_cq.cq_flags =3D NVME_QUEUE_PHYS_CONTIG | NVME_CQ_IRQ_ENABLED; + c.create_cq.irq_vector =3D 0; + + return nvme_submit_sync_cmd(anv->ctrl.admin_q, &c, NULL, 0); +} + +static int apple_nvme_remove_cq(struct apple_nvme *anv) +{ + struct nvme_command c =3D {}; + + c.delete_queue.opcode =3D nvme_admin_delete_cq; + c.delete_queue.qid =3D 1; + + return nvme_submit_sync_cmd(anv->ctrl.admin_q, &c, NULL, 0); +} + +static int apple_nvme_create_sq(struct apple_nvme *anv) +{ + struct nvme_command c =3D {}; + + /* + * Note: we (ab)use the fact that the prp fields survive if no data + * is attached to the request. + */ + c.create_sq.opcode =3D nvme_admin_create_sq; + c.create_sq.prp1 =3D anv->ioq.sq_dma_addr; + c.create_sq.sqid =3D 1; + c.create_sq.qsize =3D APPLE_ANS_MAX_QUEUE_DEPTH - 1; + c.create_sq.sq_flags =3D NVME_QUEUE_PHYS_CONTIG; + c.create_sq.cqid =3D 1; + + return nvme_submit_sync_cmd(anv->ctrl.admin_q, &c, NULL, 0); +} + +static int apple_nvme_remove_sq(struct apple_nvme *anv) +{ + struct nvme_command c =3D {}; + + c.delete_queue.opcode =3D nvme_admin_delete_sq; + c.delete_queue.qid =3D 1; + + return nvme_submit_sync_cmd(anv->ctrl.admin_q, &c, NULL, 0); +} + +static blk_status_t apple_nvme_queue_rq(struct blk_mq_hw_ctx *hctx, + const struct blk_mq_queue_data *bd) +{ + struct nvme_ns *ns =3D hctx->queue->queuedata; + struct apple_nvme_queue *q =3D hctx->driver_data; + struct apple_nvme *anv =3D queue_to_apple_nvme(q); + struct request *req =3D bd->rq; + struct apple_nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + struct nvme_command *cmnd =3D &iod->cmd; + blk_status_t ret; + + iod->npages =3D -1; + iod->nents =3D 0; + + /* + * We should not need to do this, but we're still using this to + * ensure we can drain requests on a dying queue. + */ + if (unlikely(!READ_ONCE(q->enabled))) + return BLK_STS_IOERR; + + if (!nvme_check_ready(&anv->ctrl, req, true)) + return nvme_fail_nonready_command(&anv->ctrl, req); + + ret =3D nvme_setup_cmd(ns, req); + if (ret) + return ret; + + if (blk_rq_nr_phys_segments(req)) { + ret =3D apple_nvme_map_data(anv, req, cmnd); + if (ret) + goto out_free_cmd; + } + + blk_mq_start_request(req); + apple_nvme_submit_cmd(q, cmnd); + return BLK_STS_OK; + +out_free_cmd: + nvme_cleanup_cmd(req); + return ret; +} + +static int apple_nvme_init_hctx(struct blk_mq_hw_ctx *hctx, void *data, + unsigned int hctx_idx) +{ + hctx->driver_data =3D data; + return 0; +} + +static int apple_nvme_init_request(struct blk_mq_tag_set *set, + struct request *req, unsigned int hctx_idx, + unsigned int numa_node) +{ + struct apple_nvme_queue *q =3D set->driver_data; + struct apple_nvme *anv =3D queue_to_apple_nvme(q); + struct apple_nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + struct nvme_request *nreq =3D nvme_req(req); + + iod->q =3D q; + nreq->ctrl =3D &anv->ctrl; + nreq->cmd =3D &iod->cmd; + + return 0; +} + +static void apple_nvme_disable(struct apple_nvme *anv, bool shutdown) +{ + u32 csts =3D readl(anv->mmio_nvme + NVME_REG_CSTS); + bool dead =3D false, freeze =3D false; + unsigned long flags; + + if (apple_rtkit_is_crashed(anv->rtk)) + dead =3D true; + if (!(csts & NVME_CSTS_RDY)) + dead =3D true; + if (csts & NVME_CSTS_CFS) + dead =3D true; + + if (anv->ctrl.state =3D=3D NVME_CTRL_LIVE || + anv->ctrl.state =3D=3D NVME_CTRL_RESETTING) { + freeze =3D true; + nvme_start_freeze(&anv->ctrl); + } + + /* + * Give the controller a chance to complete all entered requests if + * doing a safe shutdown. + */ + if (!dead && shutdown && freeze) + nvme_wait_freeze_timeout(&anv->ctrl, NVME_IO_TIMEOUT); + + nvme_stop_queues(&anv->ctrl); + + if (!dead) { + if (READ_ONCE(anv->ioq.enabled)) { + apple_nvme_remove_sq(anv); + apple_nvme_remove_cq(anv); + } + + if (shutdown) + nvme_shutdown_ctrl(&anv->ctrl); + nvme_disable_ctrl(&anv->ctrl); + } + + WRITE_ONCE(anv->ioq.enabled, false); + WRITE_ONCE(anv->adminq.enabled, false); + mb(); /* ensure that nvme_queue_rq() sees that enabled is cleared */ + nvme_stop_admin_queue(&anv->ctrl); + + /* last chance to complete any requests before nvme_cancel_request */ + spin_lock_irqsave(&anv->lock, flags); + apple_nvme_handle_cq(&anv->ioq, true); + apple_nvme_handle_cq(&anv->adminq, true); + spin_unlock_irqrestore(&anv->lock, flags); + + blk_mq_tagset_busy_iter(&anv->tagset, nvme_cancel_request, &anv->ctrl); + blk_mq_tagset_busy_iter(&anv->admin_tagset, nvme_cancel_request, + &anv->ctrl); + blk_mq_tagset_wait_completed_request(&anv->tagset); + blk_mq_tagset_wait_completed_request(&anv->admin_tagset); + + /* + * The driver will not be starting up queues again if shutting down so + * must flush all entered requests to their failed completion to avoid + * deadlocking blk-mq hot-cpu notifier. + */ + if (shutdown) { + nvme_start_queues(&anv->ctrl); + nvme_start_admin_queue(&anv->ctrl); + } +} + +static enum blk_eh_timer_return apple_nvme_timeout(struct request *req, + bool reserved) +{ + struct apple_nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + struct apple_nvme_queue *q =3D iod->q; + struct apple_nvme *anv =3D queue_to_apple_nvme(q); + unsigned long flags; + u32 csts =3D readl_relaxed(anv->mmio_nvme + NVME_REG_CSTS); + + if (anv->ctrl.state !=3D NVME_CTRL_LIVE) { + /* + * From rdma.c: + * If we are resetting, connecting or deleting we should + * complete immediately because we may block controller + * teardown or setup sequence + * - ctrl disable/shutdown fabrics requests + * - connect requests + * - initialization admin requests + * - I/O requests that entered after unquiescing and + * the controller stopped responding + * + * All other requests should be cancelled by the error + * recovery work, so it's fine that we fail it here. + */ + dev_warn(anv->dev, + "I/O %d(aq:%d) timeout while not in live state\n", + req->tag, q->is_adminq); + if (blk_mq_request_started(req) && + !blk_mq_request_completed(req)) { + nvme_req(req)->status =3D NVME_SC_HOST_ABORTED_CMD; + blk_mq_complete_request(req); + } + return BLK_EH_DONE; + } + + /* check if we just missed an interrupt if we're still alive */ + if (!apple_rtkit_is_crashed(anv->rtk) && !(csts & NVME_CSTS_CFS)) { + spin_lock_irqsave(&anv->lock, flags); + apple_nvme_handle_cq(q, false); + spin_unlock_irqrestore(&anv->lock, flags); + if (blk_mq_request_completed(req)) { + dev_warn(anv->dev, + "I/O %d(aq:%d) timeout: completion polled\n", + req->tag, q->is_adminq); + return BLK_EH_DONE; + } + } + + /* + * aborting commands isn't supported which leaves a full reset as our + * only option here + */ + dev_warn(anv->dev, "I/O %d(aq:%d) timeout: resetting controller\n", + req->tag, q->is_adminq); + nvme_req(req)->flags |=3D NVME_REQ_CANCELLED; + apple_nvme_disable(anv, false); + nvme_reset_ctrl(&anv->ctrl); + return BLK_EH_DONE; +} + +static int apple_nvme_poll(struct blk_mq_hw_ctx *hctx, + struct io_comp_batch *iob) +{ + struct apple_nvme_queue *q =3D hctx->driver_data; + struct apple_nvme *anv =3D queue_to_apple_nvme(q); + bool found; + unsigned long flags; + + spin_lock_irqsave(&anv->lock, flags); + found =3D apple_nvme_poll_cq(q, iob); + spin_unlock_irqrestore(&anv->lock, flags); + + return found; +} + +static const struct blk_mq_ops apple_nvme_mq_admin_ops =3D { + .queue_rq =3D apple_nvme_queue_rq, + .complete =3D apple_nvme_complete_rq, + .init_hctx =3D apple_nvme_init_hctx, + .init_request =3D apple_nvme_init_request, + .timeout =3D apple_nvme_timeout, +}; + +static const struct blk_mq_ops apple_nvme_mq_ops =3D { + .queue_rq =3D apple_nvme_queue_rq, + .complete =3D apple_nvme_complete_rq, + .init_hctx =3D apple_nvme_init_hctx, + .init_request =3D apple_nvme_init_request, + .timeout =3D apple_nvme_timeout, + .poll =3D apple_nvme_poll, +}; + +static void apple_nvme_init_queue(struct apple_nvme_queue *q) +{ + unsigned int depth =3D apple_nvme_queue_depth(q); + + q->cq_head =3D 0; + q->cq_phase =3D 1; + memset(q->tcbs, 0, + APPLE_ANS_MAX_QUEUE_DEPTH * sizeof(struct apple_nvmmu_tcb)); + memset(q->cqes, 0, depth * sizeof(struct nvme_completion)); + WRITE_ONCE(q->enabled, true); + wmb(); /* ensure the first interrupt sees the initialization */ +} + +static void apple_nvme_reset_work(struct work_struct *work) +{ + unsigned int nr_io_queues =3D 1; + int ret; + u32 boot_status, aqa; + struct apple_nvme *anv =3D + container_of(work, struct apple_nvme, ctrl.reset_work); + + if (anv->ctrl.state !=3D NVME_CTRL_RESETTING) { + dev_warn(anv->dev, "ctrl state %d is not RESETTING\n", + anv->ctrl.state); + ret =3D -ENODEV; + goto out; + } + + /* there's unfortunately no known way to recover if RTKit crashed :( */ + if (apple_rtkit_is_crashed(anv->rtk)) { + dev_err(anv->dev, + "RTKit has crashed without any way to recover."); + ret =3D -EIO; + goto out; + } + + if (anv->ctrl.ctrl_config & NVME_CC_ENABLE) + apple_nvme_disable(anv, false); + + /* RTKit must be shut down cleanly for the (soft)-reset to work */ + if (apple_rtkit_is_running(anv->rtk)) { + dev_dbg(anv->dev, "Trying to shut down RTKit before reset."); + ret =3D apple_rtkit_shutdown(anv->rtk); + if (ret) + goto out; + } + + writel_relaxed(0, anv->mmio_coproc + APPLE_ANS_COPROC_CPU_CONTROL); + (void)readl_relaxed(anv->mmio_coproc + APPLE_ANS_COPROC_CPU_CONTROL); + + ret =3D reset_control_assert(anv->reset); + if (ret) + goto out; + + ret =3D apple_rtkit_reinit(anv->rtk); + if (ret) + goto out; + + ret =3D reset_control_deassert(anv->reset); + if (ret) + goto out; + + writel_relaxed(APPLE_ANS_COPROC_CPU_CONTROL_RUN, + anv->mmio_coproc + APPLE_ANS_COPROC_CPU_CONTROL); + (void)readl_relaxed(anv->mmio_coproc + APPLE_ANS_COPROC_CPU_CONTROL); + ret =3D apple_rtkit_boot(anv->rtk); + if (ret) { + dev_err(anv->dev, "ANS did not boot"); + goto out; + } + + ret =3D readl_relaxed_poll_timeout( + anv->mmio_nvme + APPLE_ANS_BOOT_STATUS, boot_status, + boot_status =3D=3D APPLE_ANS_BOOT_STATUS_OK, USEC_PER_MSEC, + APPLE_ANS_BOOT_TIMEOUT); + if (ret) { + dev_err(anv->dev, "ANS did not initialize"); + goto out; + } + + dev_dbg(anv->dev, "ANS booted successfully."); + + /* + * Limit the max command size to prevent iod->sg allocations going + * over a single page. + */ + anv->ctrl.max_hw_sectors =3D min_t(u32, NVME_MAX_KB_SZ << 1, + dma_max_mapping_size(anv->dev) >> 9); + anv->ctrl.max_segments =3D NVME_MAX_SEGS; + + /* + * Enable NVMMU and linear submission queues. + * While we could keep those disabled and pretend this is slightly + * more common NVMe controller we'd still need some quirks (e.g. + * sq entries will be 128 bytes) and Apple might drop support for + * that mode in the future. + */ + writel_relaxed(APPLE_ANS_LINEAR_SQ_EN, + anv->mmio_nvme + APPLE_ANS_LINEAR_SQ_CTRL); + + /* Allow as many pending command as possible for both queues */ + writel_relaxed(APPLE_ANS_MAX_QUEUE_DEPTH | + (APPLE_ANS_MAX_QUEUE_DEPTH << 16), + anv->mmio_nvme + APPLE_ANS_MAX_PEND_CMDS_CTRL); + + /* Setup the NVMMU for the maximum admin and IO queue depth */ + writel_relaxed(APPLE_ANS_MAX_QUEUE_DEPTH - 1, + anv->mmio_nvme + APPLE_NVMMU_NUM_TCBS); + + /* + * This is probably a chicken bit: without it all commands where any PRP + * is set to zero (including those that don't use that field) fail and + * the co-processor complains about "completed with err BAD_CMD-" or + * a "NULL_PRP_PTR_ERR" in the syslog + */ + writel_relaxed(readl_relaxed(anv->mmio_nvme + APPLE_ANS_UNKNOWN_CTRL) & + ~APPLE_ANS_PRP_NULL_CHECK, + anv->mmio_nvme + APPLE_ANS_UNKNOWN_CTRL); + + /* Setup the admin queue */ + aqa =3D APPLE_NVME_AQ_DEPTH - 1; + aqa |=3D aqa << 16; + writel_relaxed(aqa, anv->mmio_nvme + NVME_REG_AQA); + lo_hi_writeq_relaxed(anv->adminq.sq_dma_addr, + anv->mmio_nvme + NVME_REG_ASQ); + lo_hi_writeq_relaxed(anv->adminq.cq_dma_addr, + anv->mmio_nvme + NVME_REG_ACQ); + + /* Setup NVMMU for both queues */ + lo_hi_writeq_relaxed(anv->adminq.tcb_dma_addr, + anv->mmio_nvme + APPLE_NVMMU_ASQ_TCB_BASE); + lo_hi_writeq_relaxed(anv->ioq.tcb_dma_addr, + anv->mmio_nvme + APPLE_NVMMU_IOSQ_TCB_BASE); + + anv->ctrl.sqsize =3D + APPLE_ANS_MAX_QUEUE_DEPTH - 1; /* 0's based queue depth */ + anv->ctrl.cap =3D lo_hi_readq_relaxed(anv->mmio_nvme + NVME_REG_CAP); + + dev_dbg(anv->dev, "Enabling controller now"); + ret =3D nvme_enable_ctrl(&anv->ctrl); + if (ret) + goto out; + + dev_dbg(anv->dev, "Starting admin queue"); + apple_nvme_init_queue(&anv->adminq); + nvme_start_admin_queue(&anv->ctrl); + + if (!nvme_change_ctrl_state(&anv->ctrl, NVME_CTRL_CONNECTING)) { + dev_warn(anv->ctrl.device, + "failed to mark controller CONNECTING\n"); + ret =3D -ENODEV; + goto out; + } + + ret =3D nvme_init_ctrl_finish(&anv->ctrl); + if (ret) + goto out; + + dev_dbg(anv->dev, "Creating IOCQ"); + ret =3D apple_nvme_create_cq(anv); + if (ret) + goto out; + dev_dbg(anv->dev, "Creating IOSQ"); + ret =3D apple_nvme_create_sq(anv); + if (ret) + goto out_remove_cq; + + apple_nvme_init_queue(&anv->ioq); + nr_io_queues =3D 1; + ret =3D nvme_set_queue_count(&anv->ctrl, &nr_io_queues); + if (ret) + goto out_remove_sq; + if (nr_io_queues !=3D 1) { + ret =3D -ENXIO; + goto out_remove_sq; + } + + anv->ctrl.queue_count =3D nr_io_queues + 1; + + nvme_start_queues(&anv->ctrl); + nvme_wait_freeze(&anv->ctrl); + blk_mq_update_nr_hw_queues(&anv->tagset, 1); + nvme_unfreeze(&anv->ctrl); + + if (!nvme_change_ctrl_state(&anv->ctrl, NVME_CTRL_LIVE)) { + dev_warn(anv->ctrl.device, + "failed to mark controller live state\n"); + ret =3D -ENODEV; + goto out_remove_sq; + } + + nvme_start_ctrl(&anv->ctrl); + + dev_dbg(anv->dev, "ANS boot and NVMe init completed."); + return; + +out_remove_sq: + apple_nvme_remove_sq(anv); +out_remove_cq: + apple_nvme_remove_cq(anv); +out: + dev_warn(anv->ctrl.device, "Reset failure status: %d\n", ret); + nvme_change_ctrl_state(&anv->ctrl, NVME_CTRL_DELETING); + nvme_get_ctrl(&anv->ctrl); + apple_nvme_disable(anv, false); + nvme_kill_queues(&anv->ctrl); + if (!queue_work(nvme_wq, &anv->remove_work)) + nvme_put_ctrl(&anv->ctrl); +} + +static void apple_nvme_remove_dead_ctrl_work(struct work_struct *work) +{ + struct apple_nvme *anv =3D + container_of(work, struct apple_nvme, remove_work); + + nvme_put_ctrl(&anv->ctrl); + device_release_driver(anv->dev); +} + +static int apple_nvme_reg_read32(struct nvme_ctrl *ctrl, u32 off, u32 *val) +{ + *val =3D readl_relaxed(ctrl_to_apple_nvme(ctrl)->mmio_nvme + off); + return 0; +} + +static int apple_nvme_reg_write32(struct nvme_ctrl *ctrl, u32 off, u32 val) +{ + writel_relaxed(val, ctrl_to_apple_nvme(ctrl)->mmio_nvme + off); + return 0; +} + +static int apple_nvme_reg_read64(struct nvme_ctrl *ctrl, u32 off, u64 *val) +{ + *val =3D lo_hi_readq_relaxed(ctrl_to_apple_nvme(ctrl)->mmio_nvme + off); + return 0; +} + +static int apple_nvme_get_address(struct nvme_ctrl *ctrl, char *buf, int s= ize) +{ + struct device *dev =3D ctrl_to_apple_nvme(ctrl)->dev; + + return snprintf(buf, size, "%s\n", dev_name(dev)); +} + +static void apple_nvme_free_ctrl(struct nvme_ctrl *ctrl) +{ +} + +static const struct nvme_ctrl_ops nvme_ctrl_ops =3D { + .name =3D "apple-nvme", + .module =3D THIS_MODULE, + .flags =3D 0, + .reg_read32 =3D apple_nvme_reg_read32, + .reg_write32 =3D apple_nvme_reg_write32, + .reg_read64 =3D apple_nvme_reg_read64, + .free_ctrl =3D apple_nvme_free_ctrl, + .get_address =3D apple_nvme_get_address, +}; + +static void apple_nvme_async_probe(void *data, async_cookie_t cookie) +{ + struct apple_nvme *anv =3D data; + + flush_work(&anv->ctrl.reset_work); + flush_work(&anv->ctrl.scan_work); + nvme_put_ctrl(&anv->ctrl); +} + +static int apple_nvme_alloc_tagsets(struct apple_nvme *anv) +{ + int ret; + + anv->admin_tagset.ops =3D &apple_nvme_mq_admin_ops; + anv->admin_tagset.nr_hw_queues =3D 1; + anv->admin_tagset.queue_depth =3D APPLE_NVME_AQ_MQ_TAG_DEPTH; + anv->admin_tagset.timeout =3D NVME_ADMIN_TIMEOUT; + anv->admin_tagset.numa_node =3D NUMA_NO_NODE; + anv->admin_tagset.cmd_size =3D sizeof(struct apple_nvme_iod); + anv->admin_tagset.flags =3D BLK_MQ_F_NO_SCHED; + anv->admin_tagset.driver_data =3D &anv->adminq; + + ret =3D blk_mq_alloc_tag_set(&anv->admin_tagset); + if (ret) + return ret; + ret =3D devm_add_action_or_reset(anv->dev, + (void (*)(void *))blk_mq_free_tag_set, + &anv->admin_tagset); + if (ret) + return ret; + + anv->tagset.ops =3D &apple_nvme_mq_ops; + anv->tagset.nr_hw_queues =3D 1; + anv->tagset.nr_maps =3D 1; + /* + * Tags are used as an index to the NVMMU and must be unique across + * both queues. The admin queue gets the first APPLE_NVME_AQ_DEPTH which + * must be marked as reserved in the IO queue. + */ + anv->tagset.reserved_tags =3D APPLE_NVME_AQ_DEPTH; + anv->tagset.queue_depth =3D APPLE_ANS_MAX_QUEUE_DEPTH - 1; + anv->tagset.timeout =3D NVME_IO_TIMEOUT; + anv->tagset.numa_node =3D NUMA_NO_NODE; + anv->tagset.cmd_size =3D sizeof(struct apple_nvme_iod); + anv->tagset.flags =3D BLK_MQ_F_SHOULD_MERGE; + anv->tagset.driver_data =3D &anv->ioq; + + ret =3D blk_mq_alloc_tag_set(&anv->tagset); + if (ret) + return ret; + ret =3D devm_add_action_or_reset( + anv->dev, (void (*)(void *))blk_mq_free_tag_set, &anv->tagset); + if (ret) + return ret; + + anv->ctrl.admin_tagset =3D &anv->admin_tagset; + anv->ctrl.tagset =3D &anv->tagset; + + return 0; +} + +static int apple_nvme_queue_alloc(struct apple_nvme *anv, + struct apple_nvme_queue *q) +{ + unsigned int depth =3D apple_nvme_queue_depth(q); + + q->cqes =3D dmam_alloc_coherent(anv->dev, + depth * sizeof(struct nvme_completion), + &q->cq_dma_addr, GFP_KERNEL); + if (!q->cqes) + return -ENOMEM; + + q->sqes =3D dmam_alloc_coherent(anv->dev, + depth * sizeof(struct nvme_command), + &q->sq_dma_addr, GFP_KERNEL); + if (!q->sqes) + return -ENOMEM; + + /* + * We need the maximum queue depth here because the NVMMU only has a + * single depth configuration shared between both queues. + */ + q->tcbs =3D dmam_alloc_coherent(anv->dev, + APPLE_ANS_MAX_QUEUE_DEPTH * + sizeof(struct apple_nvmmu_tcb), + &q->tcb_dma_addr, GFP_KERNEL); + if (!q->tcbs) + return -ENOMEM; + + return 0; +} + +static int apple_nvme_probe(struct platform_device *pdev) +{ + struct device *dev =3D &pdev->dev; + struct apple_nvme *anv; + int ret; + + anv =3D devm_kzalloc(dev, sizeof(*anv), GFP_KERNEL); + if (!anv) + return -ENOMEM; + + anv->dev =3D dev; + anv->adminq.is_adminq =3D true; + platform_set_drvdata(pdev, anv); + + if (dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64))) + return -ENXIO; + + anv->irq =3D platform_get_irq(pdev, 0); + if (anv->irq < 0) + return anv->irq; + if (!anv->irq) + return -ENXIO; + + anv->mmio_coproc =3D devm_platform_ioremap_resource_byname(pdev, "ans"); + if (IS_ERR(anv->mmio_coproc)) + return PTR_ERR(anv->mmio_coproc); + anv->mmio_nvme =3D devm_platform_ioremap_resource_byname(pdev, "nvme"); + if (IS_ERR(anv->mmio_nvme)) + return PTR_ERR(anv->mmio_nvme); + + anv->adminq.sq_db =3D anv->mmio_nvme + APPLE_ANS_LINEAR_ASQ_DB; + anv->adminq.cq_db =3D anv->mmio_nvme + APPLE_ANS_ACQ_DB; + anv->ioq.sq_db =3D anv->mmio_nvme + APPLE_ANS_LINEAR_IOSQ_DB; + anv->ioq.cq_db =3D anv->mmio_nvme + APPLE_ANS_IOCQ_DB; + + anv->sart =3D apple_sart_get(dev); + if (IS_ERR(anv->sart)) + return dev_err_probe(dev, PTR_ERR(anv->sart), + "Failed to initialize SART"); + + anv->reset =3D devm_reset_control_array_get_exclusive(anv->dev); + if (IS_ERR(anv->reset)) + return dev_err_probe(dev, PTR_ERR(anv->reset), + "Failed to get reset control"); + + INIT_WORK(&anv->ctrl.reset_work, apple_nvme_reset_work); + INIT_WORK(&anv->remove_work, apple_nvme_remove_dead_ctrl_work); + spin_lock_init(&anv->lock); + + ret =3D apple_nvme_queue_alloc(anv, &anv->adminq); + if (ret) + return ret; + ret =3D apple_nvme_queue_alloc(anv, &anv->ioq); + if (ret) + return ret; + + anv->prp_page_pool =3D dmam_pool_create("prp list page", anv->dev, + NVME_CTRL_PAGE_SIZE, + NVME_CTRL_PAGE_SIZE, 0); + if (!anv->prp_page_pool) + return -ENOMEM; + + anv->prp_small_pool =3D + dmam_pool_create("prp list 256", anv->dev, 256, 256, 0); + if (!anv->prp_small_pool) + return -ENOMEM; + + WARN_ON_ONCE(apple_nvme_iod_alloc_size() > PAGE_SIZE); + anv->iod_mempool =3D + mempool_create_kmalloc_pool(1, apple_nvme_iod_alloc_size()); + if (!anv->iod_mempool) + return -ENOMEM; + ret =3D devm_add_action_or_reset( + anv->dev, (void (*)(void *))mempool_destroy, anv->iod_mempool); + if (ret) + return ret; + + ret =3D apple_nvme_alloc_tagsets(anv); + if (ret) + return ret; + + ret =3D devm_request_irq(anv->dev, anv->irq, apple_nvme_irq, 0, + "nvme-apple", anv); + if (ret) + return dev_err_probe(dev, ret, "Failed to request IRQ"); + + anv->rtk =3D + devm_apple_rtkit_init(dev, anv, NULL, 0, &apple_nvme_rtkit_ops); + if (IS_ERR(anv->rtk)) + return dev_err_probe(dev, PTR_ERR(anv->rtk), + "Failed to initialize RTKit"); + + ret =3D nvme_init_ctrl(&anv->ctrl, anv->dev, &nvme_ctrl_ops, + NVME_QUIRK_SKIP_CID_GEN); + if (ret) + return dev_err_probe(dev, ret, + "Failed to initialize nvme_ctrl"); + + anv->ctrl.admin_q =3D blk_mq_init_queue(&anv->admin_tagset); + if (IS_ERR(anv->ctrl.admin_q)) + return -ENOMEM; + + nvme_reset_ctrl(&anv->ctrl); + async_schedule(apple_nvme_async_probe, anv); + + return 0; +} + +static int apple_nvme_remove(struct platform_device *pdev) +{ + struct apple_nvme *anv =3D platform_get_drvdata(pdev); + + nvme_change_ctrl_state(&anv->ctrl, NVME_CTRL_DELETING); + flush_work(&anv->ctrl.reset_work); + nvme_stop_ctrl(&anv->ctrl); + nvme_remove_namespaces(&anv->ctrl); + apple_nvme_disable(anv, true); + nvme_uninit_ctrl(&anv->ctrl); + + if (apple_rtkit_is_running(anv->rtk)) + apple_rtkit_shutdown(anv->rtk); + + return 0; +} + +static void apple_nvme_shutdown(struct platform_device *pdev) +{ + struct apple_nvme *anv =3D platform_get_drvdata(pdev); + + apple_nvme_disable(anv, true); + if (apple_rtkit_is_running(anv->rtk)) + apple_rtkit_shutdown(anv->rtk); +} + +static const struct of_device_id apple_nvme_of_match[] =3D { + { .compatible =3D "apple,nvme-ans2" }, + {}, +}; +MODULE_DEVICE_TABLE(of, apple_nvme_of_match); + +static struct platform_driver apple_nvme_driver =3D { + .driver =3D { + .name =3D "nvme-apple", + .of_match_table =3D apple_nvme_of_match, + }, + .probe =3D apple_nvme_probe, + .remove =3D apple_nvme_remove, + .shutdown =3D apple_nvme_shutdown, +}; +module_platform_driver(apple_nvme_driver); + +MODULE_AUTHOR("Sven Peter "); +MODULE_LICENSE("GPL"); --=20 2.25.1 From nobody Mon Jun 22 15:42:51 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1F64C433F5 for ; Mon, 21 Mar 2022 16:52:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351494AbiCUQyC (ORCPT ); Mon, 21 Mar 2022 12:54:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52722 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1351475AbiCUQxv (ORCPT ); Mon, 21 Mar 2022 12:53:51 -0400 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 755FA174BA1 for ; Mon, 21 Mar 2022 09:52:23 -0700 (PDT) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id 8343E3201DA2; Mon, 21 Mar 2022 12:52:21 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Mon, 21 Mar 2022 12:52:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=svenpeter.dev; h=cc:cc:content-transfer-encoding:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm1; bh=5nQpEZL6uoaGwo qyCOgh7yCZkXCs5xzAO7/dEUKTT+0=; b=CAtQijKbAa1HWvg5Tpk7zi2d5FkuFF aLTNGI76WDQ22B00KJKLI34LbvcaU1EIdqWbga6GcEcquNCAtkIuNQnX1gkJ0CfS qeynsyLAicgu0w6mh4UFOfN4I+UhwfFJ6fQ/dGF4xV1lCG2CeB8L3oXexxLX3MTu cFsXC7a+o3WyWjek76mwkgfXU4lLpPkcRT3U4zEk+V+aYop+0oDBoyXy6oXm5Jx8 CN75O1hNTC0zvOKD+4Gn/4VmVnzKoa+5X+2lNikK/I/AQutTfItUqs3hTJF6LRo9 zvrtvBug2La3VEmVJbGH2BYeE0yKN2OvU+augQNt/FhqElCdFoxaAL6w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=5nQpEZ L6uoaGwoqyCOgh7yCZkXCs5xzAO7/dEUKTT+0=; b=E9MPCUh24viES5QQonaStG Emdx3GaIzqVI3u1pDCc4LgmYtxl6uSuFQVewDbo4g96i/1oEtGVwFHToqzhEXq5d pB/oaaimaAOP6G80vau2wYTzqllX1aS2GOzf+KvyJJ+MH6EBMAy/WcSLEbgjcQyG KQwCWEI9v0vLJAHDRN5Xmt8yjxyttI0VX1ZRMwdKQc5loImHSCA3h3jFsQU8jyt5 GZXH4ofkItVZUff2bF34rt21WcQheTlId9TDDPKi5GR/Z22LmeiqSEdKErISxFSk xmunL5eQz6XaZNcMv+1hhlXEsBa61MwIxsOKhRUfnx5OIL9ekpR6CrEMuFYIfHdQ == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrudegfedgleehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefuvhgvnhcu rfgvthgvrhcuoehsvhgvnhesshhvvghnphgvthgvrhdruggvvheqnecuggftrfgrthhtvg hrnheptedvkeetleeuffffhfekteetffeggffgveehieelueefvddtueffveevlefhfeej necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepshhvvg hnsehsvhgvnhhpvghtvghrrdguvghv X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 21 Mar 2022 12:52:18 -0400 (EDT) From: Sven Peter To: Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg Cc: Sven Peter , Hector Martin , Alyssa Rosenzweig , Rob Herring , Arnd Bergmann , Marc Zyngier , devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, Jens Axboe Subject: [PATCH 7/9] nvme-apple: Serialize command issue Date: Mon, 21 Mar 2022 17:50:47 +0100 Message-Id: <20220321165049.35985-8-sven@svenpeter.dev> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20220321165049.35985-1-sven@svenpeter.dev> References: <20220321165049.35985-1-sven@svenpeter.dev> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Jens Axboe This controller shouldn't need serialization of command issue since the SQ is replaced by a simple array and commands are issued by writing the array index to a MMIO register. Without serialization however sometimes commands are still executed correctly and appear in the CQ but never trigger an interrupt. Signed-off-by: Jens Axboe [sven: added our best guess why this needs to be done] Signed-off-by: Sven Peter --- drivers/nvme/host/apple.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/drivers/nvme/host/apple.c b/drivers/nvme/host/apple.c index 587d6c7014a0..a4193429564e 100644 --- a/drivers/nvme/host/apple.c +++ b/drivers/nvme/host/apple.c @@ -292,6 +292,7 @@ static void apple_nvmmu_inval(struct apple_nvme_queue *= q, unsigned int tag) static void apple_nvme_submit_cmd(struct apple_nvme_queue *q, struct nvme_command *cmd) { + struct apple_nvme *anv =3D queue_to_apple_nvme(q); u32 tag =3D nvme_tag_from_cid(cmd->common.command_id); struct apple_nvmmu_tcb *tcb =3D &q->tcbs[tag]; =20 @@ -308,7 +309,18 @@ static void apple_nvme_submit_cmd(struct apple_nvme_qu= eue *q, tcb->dma_flags |=3D APPLE_ANS_TCB_DMA_FROM_DEVICE; =20 memcpy(&q->sqes[tag], cmd, sizeof(*cmd)); + + /* + * This lock here doesn't make much sense at a first glace but + * removing it will result in occasional missed completetion + * interrupts even though the commands still appear on the CQ. + * It's unclear why this happens but our best guess is that + * there is a bug in the firmware triggered when a write to the + * CQ and the SQ happen simultaneously. + */ + spin_lock_irq(&anv->lock); writel(tag, q->sq_db); + spin_unlock_irq(&anv->lock); } =20 /* --=20 2.25.1 From nobody Mon Jun 22 15:42:51 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 954C2C433EF for ; Mon, 21 Mar 2022 16:52:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351500AbiCUQyH (ORCPT ); Mon, 21 Mar 2022 12:54:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52728 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1351465AbiCUQxx (ORCPT ); Mon, 21 Mar 2022 12:53:53 -0400 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8C01B176D1C for ; Mon, 21 Mar 2022 09:52:27 -0700 (PDT) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id D8A853201F01; Mon, 21 Mar 2022 12:52:25 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Mon, 21 Mar 2022 12:52:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=svenpeter.dev; h=cc:cc:content-transfer-encoding:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm1; bh=ESszksCEES65bf hqx+u2sQyVlxPEvY4CKlsiAv1DpcI=; b=MnjD+VCawenSZyPsGTG6DQkUoHqoBF 4K/AhJXs9FTJUM6uCDUaLrC6u/ZGu1fyb+riCwkazbTOYVXolXl1S3qWdb810D+F olCUQZjQwDS8CZYPznbMJON+u5/UNOAuG/R2pnJ1PsygpPLPfeGo2EjiXHTxuW9l 79XHI1mTFoUcGALwcjYTHHS6UaW3m+ZYQiIGrW4Q7dO54K2WIn8mLYVI05L4BQpl 4RSWWQbSFc43t3HB5tZ0h1J+hYSMcMQ9ZWeH1y5eNXFHLUWytW9rs7JH2Jwe8BBr ymrYwwZylrHNZoxqAbLx0r1EhnmWiT6K9D9xIP7KPziZfUgbLiU78iCg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=ESszks CEES65bfhqx+u2sQyVlxPEvY4CKlsiAv1DpcI=; b=i4uZGESsv9zgKKUfYbkePZ mbKl67WKeZYpLZ6J64ZibwH4h6TKuD2q8U0BKQdUe+RkzUNtlCjEI7XORqI8ExtS a9ctIlDquf4z53UNt1k67Urtj3M5RZjkuqmFQdHvTOfY6uE/rS3qcYhOO4aHZPow J2R+/oUhvUEweiniWHMNG3N3rZ9ZFKekNS4HR0UrYOTE4W+GsBgf0y82N17lWTRn AfhSYHEBucKSLBqsFHykAUp7iC4RQBSXNJ0+CmR8lzm5/a7G1RPbb2Dd11uC+87f +5T3e0w8ZuhCcbn9haMospKkkQrD1YTw2YL/B1yxWkTTI+HFeSpS/nKPR6rTCqGw == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrudegfedgleehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefuvhgvnhcu rfgvthgvrhcuoehsvhgvnhesshhvvghnphgvthgvrhdruggvvheqnecuggftrfgrthhtvg hrnheptedvkeetleeuffffhfekteetffeggffgveehieelueefvddtueffveevlefhfeej necuvehluhhsthgvrhfuihiivgepfeenucfrrghrrghmpehmrghilhhfrhhomhepshhvvg hnsehsvhgvnhhpvghtvghrrdguvghv X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 21 Mar 2022 12:52:23 -0400 (EDT) From: Sven Peter To: Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg Cc: Sven Peter , Hector Martin , Alyssa Rosenzweig , Rob Herring , Arnd Bergmann , Marc Zyngier , devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org Subject: [PATCH 8/9] nvme-apple: Add support for multiple power domains Date: Mon, 21 Mar 2022 17:50:48 +0100 Message-Id: <20220321165049.35985-9-sven@svenpeter.dev> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20220321165049.35985-1-sven@svenpeter.dev> References: <20220321165049.35985-1-sven@svenpeter.dev> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Hector Martin Turns out we really need this, as the APCIE_ST*_SYS domains really do hard-depend on ANS2, at least on t6000. Signed-off-by: Hector Martin Signed-off-by: Sven Peter --- drivers/nvme/host/apple.c | 67 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) diff --git a/drivers/nvme/host/apple.c b/drivers/nvme/host/apple.c index a4193429564e..d89b4ab80169 100644 --- a/drivers/nvme/host/apple.c +++ b/drivers/nvme/host/apple.c @@ -29,6 +29,7 @@ #include #include #include +#include #include #include #include @@ -178,6 +179,10 @@ struct apple_nvme { void __iomem *mmio_coproc; void __iomem *mmio_nvme; =20 + struct device **pd_dev; + struct device_link **pd_link; + int pd_count; + struct apple_sart *sart; struct apple_rtkit *rtk; struct reset_control *reset; @@ -1313,6 +1318,62 @@ static int apple_nvme_queue_alloc(struct apple_nvme = *anv, return 0; } =20 +static void apple_nvme_detach_genpd(struct apple_nvme *anv) +{ + int i; + + if (anv->pd_count <=3D 1) + return; + + for (i =3D anv->pd_count - 1; i >=3D 0; i--) { + if (anv->pd_link[i]) + device_link_del(anv->pd_link[i]); + if (!IS_ERR_OR_NULL(anv->pd_dev[i])) + dev_pm_domain_detach(anv->pd_dev[i], true); + } +} + +static int apple_nvme_attach_genpd(struct apple_nvme *anv) +{ + struct device *dev =3D anv->dev; + int i; + + anv->pd_count =3D of_count_phandle_with_args(dev->of_node, + "power-domains", + "#power-domain-cells"); + if (anv->pd_count <=3D 1) + return 0; + + anv->pd_dev =3D devm_kcalloc(dev, anv->pd_count, sizeof(*anv->pd_dev), + GFP_KERNEL); + if (!anv->pd_dev) + return -ENOMEM; + + anv->pd_link =3D devm_kcalloc(dev, anv->pd_count, sizeof(*anv->pd_link), + GFP_KERNEL); + if (!anv->pd_link) + return -ENOMEM; + + for (i =3D 0; i < anv->pd_count; i++) { + anv->pd_dev[i] =3D dev_pm_domain_attach_by_id(dev, i); + if (IS_ERR(anv->pd_dev[i])) { + apple_nvme_detach_genpd(anv); + return PTR_ERR(anv->pd_dev[i]); + } + + anv->pd_link[i] =3D device_link_add(dev, anv->pd_dev[i], + DL_FLAG_STATELESS | + DL_FLAG_PM_RUNTIME | + DL_FLAG_RPM_ACTIVE); + if (!anv->pd_link[i]) { + apple_nvme_detach_genpd(anv); + return -EINVAL; + } + } + + return 0; +} + static int apple_nvme_probe(struct platform_device *pdev) { struct device *dev =3D &pdev->dev; @@ -1327,6 +1388,10 @@ static int apple_nvme_probe(struct platform_device *= pdev) anv->adminq.is_adminq =3D true; platform_set_drvdata(pdev, anv); =20 + ret =3D apple_nvme_attach_genpd(anv); + if (ret < 0) + return dev_err_probe(dev, ret, "Failed to attach power domains"); + if (dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64))) return -ENXIO; =20 @@ -1435,6 +1500,8 @@ static int apple_nvme_remove(struct platform_device *= pdev) if (apple_rtkit_is_running(anv->rtk)) apple_rtkit_shutdown(anv->rtk); =20 + apple_nvme_detach_genpd(anv); + return 0; } =20 --=20 2.25.1 From nobody Mon Jun 22 15:42:51 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36C75C433EF for ; Mon, 21 Mar 2022 16:52:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351513AbiCUQyV (ORCPT ); Mon, 21 Mar 2022 12:54:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53498 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1351506AbiCUQyR (ORCPT ); Mon, 21 Mar 2022 12:54:17 -0400 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8BCDF186885 for ; Mon, 21 Mar 2022 09:52:31 -0700 (PDT) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id DB4733201F73; Mon, 21 Mar 2022 12:52:29 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Mon, 21 Mar 2022 12:52:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=svenpeter.dev; h=cc:cc:content-transfer-encoding:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm1; bh=AYdSGwRMGldmW9 2q2sOAENktvfn95kOLjUtd9ziO5OE=; b=QWAsrxB+Mxp/m+32deSN0+TmPt92QW B1KIGBAagmFxx6rfzS/9rnuiDW+GsQZJQWBEcC++fgWpCFfxH2jPxKPXwVxkP3Dj qs9GD5evComStOb97lopehFMx4v9B6j0/F5cHWQEzxw1EC69ffDJ0IIyC3QxIO4V HPZtsHFOgKwWw6MF/h+A2LI5gOYeIiIH8K8aLRIu0xzRpg5YYT7AKic9OGhlkeQu xVj+A3CwFw0dxcffeMlcMdYLvxr/aJoWlu9/9UCvFl1mZLYNa4BJr1DuiBqxTSN7 dIaf5Veg76UadS3HLIgAipznQSp5VcPD2N7SmOMJoy0Eqkt+S7klKpkA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=AYdSGw RMGldmW92q2sOAENktvfn95kOLjUtd9ziO5OE=; b=g61PwCZVDEDaX/4YvLPIYr tD+Q6W5NuXbmwXxf16U82zGn9dBkLPNRuCGZgH0VsY6hEfRfCowDjEVNKzdwCLeR aLl9Swu/QnY9yPEMTBuN8t5iN4Kr2rHSU9HDk6WkknMOysUco5/M+NbPyitlPqPM LwRHlEwTPOTqtjWAs3Ru7u/RmuIAnOdztQXDUKzUDrQCITJBaHNQFCClvfdkCYlO pfDelp2EZY2ICZ6qkx7dZm9DGRrXmAN/NFGLlB1dmBWECf3cjNyzBkGAF3CcUwqI FKR4o9MGglGoVu4Ef78Kqr94yHTzG7J5oU8zSc/0dFv8NiwBNhESHfPSW+gK2Qpw == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrudegfedgleehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefuvhgvnhcu rfgvthgvrhcuoehsvhgvnhesshhvvghnphgvthgvrhdruggvvheqnecuggftrfgrthhtvg hrnheptedvkeetleeuffffhfekteetffeggffgveehieelueefvddtueffveevlefhfeej necuvehluhhsthgvrhfuihiivgepfeenucfrrghrrghmpehmrghilhhfrhhomhepshhvvg hnsehsvhgvnhhpvghtvghrrdguvghv X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 21 Mar 2022 12:52:27 -0400 (EDT) From: Sven Peter To: Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg Cc: Sven Peter , Hector Martin , Alyssa Rosenzweig , Rob Herring , Arnd Bergmann , Marc Zyngier , devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org Subject: [PATCH 9/9] nvme-apple: Add support for suspend/resume Date: Mon, 21 Mar 2022 17:50:49 +0100 Message-Id: <20220321165049.35985-10-sven@svenpeter.dev> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20220321165049.35985-1-sven@svenpeter.dev> References: <20220321165049.35985-1-sven@svenpeter.dev> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Hector Martin Signed-off-by: Hector Martin Signed-off-by: Sven Peter --- drivers/nvme/host/apple.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/drivers/nvme/host/apple.c b/drivers/nvme/host/apple.c index d89b4ab80169..c949a18ec690 100644 --- a/drivers/nvme/host/apple.c +++ b/drivers/nvme/host/apple.c @@ -1514,6 +1514,36 @@ static void apple_nvme_shutdown(struct platform_devi= ce *pdev) apple_rtkit_shutdown(anv->rtk); } =20 +#ifdef CONFIG_PM_SLEEP +static int apple_nvme_resume(struct device *dev) +{ + struct apple_nvme *anv =3D dev_get_drvdata(dev); + + return nvme_reset_ctrl(&anv->ctrl); +} + +static int apple_nvme_suspend(struct device *dev) +{ + struct apple_nvme *anv =3D dev_get_drvdata(dev); + int ret =3D 0; + + apple_nvme_disable(anv, true); + + if (apple_rtkit_is_running(anv->rtk)) + ret =3D apple_rtkit_shutdown(anv->rtk); + + writel_relaxed(0, anv->mmio_coproc + APPLE_ANS_COPROC_CPU_CONTROL); + (void)readl_relaxed(anv->mmio_coproc + APPLE_ANS_COPROC_CPU_CONTROL); + + return ret; +} + +static const struct dev_pm_ops apple_nvme_pm_ops =3D { + .suspend =3D apple_nvme_suspend, + .resume =3D apple_nvme_resume, +}; +#endif + static const struct of_device_id apple_nvme_of_match[] =3D { { .compatible =3D "apple,nvme-ans2" }, {}, @@ -1524,6 +1554,9 @@ static struct platform_driver apple_nvme_driver =3D { .driver =3D { .name =3D "nvme-apple", .of_match_table =3D apple_nvme_of_match, +#ifdef CONFIG_PM_SLEEP + .pm =3D &apple_nvme_pm_ops, +#endif }, .probe =3D apple_nvme_probe, .remove =3D apple_nvme_remove, --=20 2.25.1