From nobody Wed May 15 05:10:18 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1694427469; cv=none; d=zohomail.com; s=zohoarc; b=LlgQXyRr8PncvOJb96K9DcANx/JQOngvdsvLO4JyP/hGjQIfiDGwo1sxrJROSH1Lm3ehVKmAH+b5J5l7cszcCShM0rpQGJm7mrEdFDmT5LdApKAa/j3kgtMJh1ebqpPJ2YaEzGXi19/EpEsrHG20+jguKnPj2ci27NmS8kpKOoI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1694427469; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To; bh=h5jyXIL4MylxL7KTKKCeGpCc0CK6eglna7KZ9e7/cIc=; b=bo+R6hU2+eX6bifGeeIngQ4dTlf2yFDBM1a7RoyIj9O2MfN2Ew6Oc/p/8UaOqNGkpG/8K5/K6JdhWAotRR58F/OevWJ8xf8v3LfTKJ1kVJP2V4u4OR+cMFOuUVjkdUS0FO/gn4aWSOGqAbF5RBjW1DCZS3ZzYkiVM6D8Up4SffA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1694427469760913.0174472206108; Mon, 11 Sep 2023 03:17:49 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qfdys-0001sY-3c; Mon, 11 Sep 2023 06:17:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qfdyq-0001sO-Hs for qemu-devel@nongnu.org; Mon, 11 Sep 2023 06:17:20 -0400 Received: from mail-ed1-x52a.google.com ([2a00:1450:4864:20::52a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qfdyn-0001vO-S3 for qemu-devel@nongnu.org; Mon, 11 Sep 2023 06:17:20 -0400 Received: by mail-ed1-x52a.google.com with SMTP id 4fb4d7f45d1cf-52c88a03f99so5145946a12.2 for ; Mon, 11 Sep 2023 03:17:17 -0700 (PDT) Received: from [192.168.200.206] (83.21.94.247.ipv4.supernova.orange.pl. [83.21.94.247]) by smtp.gmail.com with ESMTPSA id m14-20020aa7d34e000000b0052a1a623267sm4396109edr.62.2023.09.11.03.17.14 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 11 Sep 2023 03:17:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1694427436; x=1695032236; darn=nongnu.org; h=content-transfer-encoding:subject:organization:from:cc:to :content-language:user-agent:mime-version:date:message-id:from:to:cc :subject:date:message-id:reply-to; bh=h5jyXIL4MylxL7KTKKCeGpCc0CK6eglna7KZ9e7/cIc=; b=ll5J3St02RYICNkQjedie/tRB397Tf3YXCkzdtSaKvtOzF6ejTfSL5LJfQajqr9VSf SJbJqoXHmvR4aoBH52qYv1oVSq3vGRPdIrf+TPH0zpuyiWScbe9avdenfMnXx7MllH39 PJ7uEqBaF0GbCOYHgrl4/B9+MQy/R1fd+5AEjUwJcagrAwSoij1UdoStbDsa21GkzC2t LmtcO7Css8YRMv8HpjlQomJIt7/AZ5QFSKlDL0qg7t5VRod7uxvpNaPGE1Ab5tLGC0su 4wfS3OYNLzxli+WNkq8Mas+KplhqMv5lhWdRjN/uj4/03LUYHe3KRFqoSLUQAiJnKFKu YTRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694427436; x=1695032236; h=content-transfer-encoding:subject:organization:from:cc:to :content-language:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=h5jyXIL4MylxL7KTKKCeGpCc0CK6eglna7KZ9e7/cIc=; b=PHfUFwaUPT9B0ZPzqt+/wuiD32h2qiCz76xPNwvqF5ovmJ0ggG49GOVxy/GUg/xj+6 jGT4SpkRqQVPFpMs/fvpzJM6kb9WCtotTGcVukHrFoKfQZAWsZjpWghQbMzHl57K8VsS PxF5jO8pbB6RxbjQnsqCCWlBlwBPubGUL4BF8yRB6HoeQrW2BELftTI5fCmsLIq6RoT5 YkN8Dci793LzdT3arI0EFH19Z0iRsERb8gqdpC67nTZiPkOlYMFsz8z8SoVwHemfezmo FOXNhVGaetENl25YD/srf77LGMIwqHZVHKJxUOkpHwmmUK2LIR9RGWEWfohyK/f9O61z sSsA== X-Gm-Message-State: AOJu0YwhkqastKiVxj+VtAzaXX7vkSSUKawOgIDJpeGNu7olSptgFSyg FBFuVgfzucCWPMpnqsKGlqixuJ4L/J7mT7arC8DNcg== X-Google-Smtp-Source: AGHT+IH5oOMHEloCGvgyg7MGjJE5xABIh1O8v7NeY5pYQGATUYZkMsAoc8QBNMKPiXbfMctqskc6dw== X-Received: by 2002:a50:ee0b:0:b0:523:18db:e3ab with SMTP id g11-20020a50ee0b000000b0052318dbe3abmr6897828eds.39.1694427435472; Mon, 11 Sep 2023 03:17:15 -0700 (PDT) Message-ID: <6a0b9345-a447-4cf9-ba9a-88c420e7ab79@linaro.org> Date: Mon, 11 Sep 2023 12:17:13 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: pl-PL, en-GB, en-HK To: qemu-devel@nongnu.org Cc: "Michael S. Tsirkin" , Marcel Apfelbaum From: Marcin Juszkiewicz Organization: Linaro Subject: Should pcie-pci-bridge use 32-bit non-prefetchable memory? Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::52a; envelope-from=marcin.juszkiewicz@linaro.org; helo=mail-ed1-x52a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @linaro.org) X-ZM-MESSAGEID: 1694427471543100003 Content-Type: text/plain; charset="utf-8"; format="flowed" I am working on aarch64/sbsa-ref machine so people can have virtual machine to test their OS against something reminding standards compliant system. One of tools I use is BSA ACS (Base System Architecture - Architecture Compliance Suite) [1] written by Arm. It runs set of tests to check does system conforms to BSA specification. 1. https://github.com/ARM-software/bsa-acs To run tests I use my "boot-sbsa-ref.sh" script [2] which allows me to run exactly same setup each time. 2. https://github.com/hrw/sbsa-ref-status/blob/main/boot-sbsa-ref.sh Since we have ITS support in whole stack (qemu, tf-a, edk2) I use overcomplicated PCIe setup: -device igb -device pcie-root-port,id=3Droot_port_for_switch1,chassis=3D0,slot=3D0 -device x3130-upstream,id=3Dupstream_port1,bus=3Droot_port_for_switch1 -device xio3130-downstream,id=3Ddownstream_port1,bus=3Dupstream_port1,ch= assis=3D1,slot=3D0 -device ac97,bus=3Ddownstream_port1 -device pcie-root-port,id=3Droot_port_for_nvme1,chassis=3D2,slot=3D0 -device nvme,serial=3Ddeadbeef,bus=3Droot_port_for_nvme1 -device pcie-root-port,id=3Droot_port_for_igb,chassis=3D3,slot=3D0 -device igb,bus=3Droot_port_for_igb -device pcie-root-port,id=3Droot_port_for_xhci,chassis=3D4,slot=3D0 -device qemu-xhci,bus=3Droot_port_for_xhci -device pcie-root-port,id=3Droot_port_for_rng,chassis=3D5,slot=3D0 -device virtio-rng-pci,bus=3Droot_port_for_rng -device pcie-root-port,id=3Droot_port_for_pci,chassis=3D6,slot=3D0 -device pcie-pci-bridge,id=3Dpci,bus=3Droot_port_for_pci -device es1370,bus=3Dpci,addr=3D9 -device e1000,bus=3Dpci,addr=3D10 -device pxb-pcie,id=3Dpxb1,bus_nr=3D1 -device pcie-root-port,id=3Droot_port_for_ahci,bus=3Dpxb1,chassis=3D10,sl= ot=3D0 -device ahci,bus=3Droot_port_for_ahci BSA ACS test 841 checks do Type-1 PCIe devices have 32-bit non-prefetchable memory. And fails on pcie-pci-bridge: Operating System View: 841 : NP type-1 PCIe supp 32-bit only START BDF - 0x400 BDF - 0x500 BDF - 0x600 BDF - 0x700 BDF - 0x800 BDF - 0x900 BDF - 0x10000 BDF - 0x30000 Skipping Non Type-1 headers BDF - 0x40000 Skipping Non Type-1 headers BDF - 0x50000 Skipping Non Type-1 headers BDF - 0x60000 Skipping Non Type-1 headers BDF - 0x70000 NP type-1 pcie is not 32-bit mem type Failed on PE - 0 PCI_MM_04 Checkpoint -- 1 : Result: FAIL 0x70000 is pcie-pci-bridge card. I opened issue for BSA ACS [3] and asked where the problem is. 3. https://github.com/ARM-software/bsa-acs/issues/197 Got quote from BSA (Arm Base System Architecture) [4] chapter E.2 PCI Express Memory Space: > When PCI Express memory space is mapped as normal memory, the system > must support unaligned accesses to that region. > PCI Type 1 headers, used in PCI-to-PCI bridges, and therefore in root > ports and switches, have to be programmed with the address space > resources claimed by the given bridge. > For non-prefetchable (NP) memory, Type 1 headers only support 32-bit > addresses. This implies that endpoints on the other end of a > PCI-to-PCI bridge only support 32-bit NP BARs 4. https://developer.arm.com/documentation/den0094/latest/ I looked at code and tried to switch pcie-pci-bridge to 32-bit: ------------------------------------------------------------------------ diff --git a/hw/pci-bridge/pcie_pci_bridge.c b/hw/pci-bridge/pcie_pci_bridg= e.c index 2301b2ca0b..45199d2fa0 100644 --- a/hw/pci-bridge/pcie_pci_bridge.c +++ b/hw/pci-bridge/pcie_pci_bridge.c @@ -82,7 +82,7 @@ static void pcie_pci_bridge_realize(PCIDevice *d, Error *= *errp) } } pci_register_bar(d, 0, PCI_BASE_ADDRESS_SPACE_MEMORY | - PCI_BASE_ADDRESS_MEM_TYPE_64, &pcie_br->shpc_bar); + PCI_BASE_ADDRESS_MEM_TYPE_32, &pcie_br->shpc_bar); return; msi_error: ------------------------------------------------------------------------ With it, test 841 passes. The difference in "lspci -vvvv" output suggests that this region address was 32-bit in both cases: - Region 0: Memory at 81c00000 (64-bit, non-prefetchable) [size=3D256] + Region 0: Memory at 81c00000 (32-bit, non-prefetchable) [size=3D256] Any ideas how to continue from here?