From nobody Fri May 3 14:42:45 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 14985543472601004.5717737200833; Tue, 27 Jun 2017 02:05:47 -0700 (PDT) Received: from localhost ([::1]:51351 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPmRL-0005zJ-NJ for importer@patchew.org; Tue, 27 Jun 2017 05:05:43 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51286) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPmPf-0004qO-Pq for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:00 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dPmPe-00030m-WE for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:03:59 -0400 Received: from mx1.redhat.com ([209.132.183.28]:37914) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dPmPe-00030X-Qa for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:03:58 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D2925C04574C for ; Tue, 27 Jun 2017 09:03:57 +0000 (UTC) Received: from pxdev.xzpeter.org.com (ovpn-12-81.pek2.redhat.com [10.72.12.81]) by smtp.corp.redhat.com (Postfix) with ESMTP id 622F07F81B; Tue, 27 Jun 2017 09:03:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com D2925C04574C Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=peterx@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com D2925C04574C From: Peter Xu To: qemu-devel@nongnu.org Date: Tue, 27 Jun 2017 17:03:32 +0800 Message-Id: <1498554219-4942-2-git-send-email-peterx@redhat.com> In-Reply-To: <1498554219-4942-1-git-send-email-peterx@redhat.com> References: <1498554219-4942-1-git-send-email-peterx@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Tue, 27 Jun 2017 09:03:58 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 1/8] intel_iommu: fix VTD_PAGE_MASK X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Wang , peterx@redhat.com, "Michael S . Tsirkin" Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" IOMMUTLBEntry.iova is returned incorrectly in one PT path (though mostly we cannot really trigger this path, even if we do, we are mostly disgarding this value, so it didn't break anything). Fix it by converting the VTD_PAGE_MASK into normal definition (normally it should be pfn mask, not offset mask), then switch the other user of it. Fixes: b93130 ("intel_iommu: cleanup vtd_{do_}iommu_translate()") Signed-off-by: Peter Xu --- hw/i386/intel_iommu.c | 2 +- hw/i386/intel_iommu_internal.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c index a9b59bd..a5c83dd 100644 --- a/hw/i386/intel_iommu.c +++ b/hw/i386/intel_iommu.c @@ -1141,7 +1141,7 @@ static bool vtd_do_iommu_translate(VTDAddressSpace *v= td_as, PCIBus *bus, if (vtd_ce_get_type(&ce) =3D=3D VTD_CONTEXT_TT_PASS_THROUGH) { entry->iova =3D addr & VTD_PAGE_MASK; entry->translated_addr =3D entry->iova; - entry->addr_mask =3D VTD_PAGE_MASK; + entry->addr_mask =3D ~VTD_PAGE_MASK; entry->perm =3D IOMMU_RW; trace_vtd_translate_pt(source_id, entry->iova); =20 diff --git a/hw/i386/intel_iommu_internal.h b/hw/i386/intel_iommu_internal.h index f50ecd8..d1d6290 100644 --- a/hw/i386/intel_iommu_internal.h +++ b/hw/i386/intel_iommu_internal.h @@ -384,7 +384,7 @@ typedef struct VTDIOTLBPageInvInfo VTDIOTLBPageInvInfo; /* Pagesize of VTD paging structures, including root and context tables */ #define VTD_PAGE_SHIFT 12 #define VTD_PAGE_SIZE (1ULL << VTD_PAGE_SHIFT) -#define VTD_PAGE_MASK (VTD_PAGE_SIZE - 1) +#define VTD_PAGE_MASK ~(VTD_PAGE_SIZE - 1) =20 #define VTD_PAGE_SHIFT_4K 12 #define VTD_PAGE_MASK_4K (~((1ULL << VTD_PAGE_SHIFT_4K) - 1)) --=20 2.7.4 From nobody Fri May 3 14:42:45 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 149855435397512.971699079257746; Tue, 27 Jun 2017 02:05:53 -0700 (PDT) Received: from localhost ([::1]:51352 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPmRS-00063C-CW for importer@patchew.org; Tue, 27 Jun 2017 05:05:50 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51314) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPmPi-0004s3-FQ for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dPmPh-00033x-Cd for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:02 -0400 Received: from mx1.redhat.com ([209.132.183.28]:38010) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dPmPh-00032y-4R for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:01 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3211CC04574C for ; Tue, 27 Jun 2017 09:04:00 +0000 (UTC) Received: from pxdev.xzpeter.org.com (ovpn-12-81.pek2.redhat.com [10.72.12.81]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4BEC47FB4C; Tue, 27 Jun 2017 09:03:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 3211CC04574C Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=peterx@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 3211CC04574C From: Peter Xu To: qemu-devel@nongnu.org Date: Tue, 27 Jun 2017 17:03:33 +0800 Message-Id: <1498554219-4942-3-git-send-email-peterx@redhat.com> In-Reply-To: <1498554219-4942-1-git-send-email-peterx@redhat.com> References: <1498554219-4942-1-git-send-email-peterx@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Tue, 27 Jun 2017 09:04:00 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 2/8] hmp: add info iommu X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Wang , peterx@redhat.com, "Michael S . Tsirkin" Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Introducing a new HMP interface "info iommu" to dump IOMMU information. This command will be only used for developers' debugging purpose, and no possible use for users. So QMP interface will not be implemented. This patch only implements the stub one. We can provide arch-dependent status dump in the future. Signed-off-by: Peter Xu --- hmp-commands-info.hx | 14 ++++++++++++++ hmp.c | 6 ++++++ hmp.h | 1 + include/hw/iommu.h | 9 +++++++++ stubs/Makefile.objs | 1 + stubs/iommu.c | 9 +++++++++ 6 files changed, 40 insertions(+) create mode 100644 include/hw/iommu.h create mode 100644 stubs/iommu.c diff --git a/hmp-commands-info.hx b/hmp-commands-info.hx index ae16901..a39243d 100644 --- a/hmp-commands-info.hx +++ b/hmp-commands-info.hx @@ -802,6 +802,20 @@ Dump all the ramblocks of the system. ETEXI =20 { + .name =3D "iommu", + .args_type =3D "", + .params =3D "", + .help =3D "Display system IOMMU information", + .cmd =3D hmp_info_iommu, + }, + +STEXI +@item info ramblock +@findex ramblock +Dump all the ramblocks of the system. +ETEXI + + { .name =3D "hotpluggable-cpus", .args_type =3D "", .params =3D "", diff --git a/hmp.c b/hmp.c index 8c72c58..68994af 100644 --- a/hmp.c +++ b/hmp.c @@ -42,6 +42,7 @@ #include "qemu/error-report.h" #include "exec/ramlist.h" #include "hw/intc/intc.h" +#include "hw/iommu.h" #include "migration/snapshot.h" =20 #ifdef CONFIG_SPICE @@ -2817,3 +2818,8 @@ void hmp_info_vm_generation_id(Monitor *mon, const QD= ict *qdict) hmp_handle_error(mon, &err); qapi_free_GuidInfo(info); } + +void hmp_info_iommu(Monitor *mon, const QDict *qdict) +{ + arch_iommu_info(mon, qdict); +} diff --git a/hmp.h b/hmp.h index d8b94ce..ed01c49 100644 --- a/hmp.h +++ b/hmp.h @@ -143,5 +143,6 @@ void hmp_info_dump(Monitor *mon, const QDict *qdict); void hmp_info_ramblock(Monitor *mon, const QDict *qdict); void hmp_hotpluggable_cpus(Monitor *mon, const QDict *qdict); void hmp_info_vm_generation_id(Monitor *mon, const QDict *qdict); +void hmp_info_iommu(Monitor *mon, const QDict *qdict); =20 #endif diff --git a/include/hw/iommu.h b/include/hw/iommu.h new file mode 100644 index 0000000..5201a8d --- /dev/null +++ b/include/hw/iommu.h @@ -0,0 +1,9 @@ +#ifndef __HW_IOMMU_H__ +#define __HW_IOMMU_H__ + +#include "qemu/typedefs.h" +#include "qapi/qmp/qdict.h" + +void arch_iommu_info(Monitor *mon, const QDict *qdict); + +#endif diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs index f5b47bf..dfd5569 100644 --- a/stubs/Makefile.objs +++ b/stubs/Makefile.objs @@ -39,3 +39,4 @@ stub-obj-y +=3D pc_madt_cpu_entry.o stub-obj-y +=3D vmgenid.o stub-obj-y +=3D xen-common.o stub-obj-y +=3D xen-hvm.o +stub-obj-y +=3D iommu.o diff --git a/stubs/iommu.c b/stubs/iommu.c new file mode 100644 index 0000000..75b4f4c --- /dev/null +++ b/stubs/iommu.c @@ -0,0 +1,9 @@ +#include "qemu/osdep.h" +#include "monitor/monitor.h" +#include "hw/iommu.h" + +void arch_iommu_info(Monitor *mon, const QDict *qdict) +{ + monitor_printf(mon, "This command is not supported " + "on this platform.\n"); +} --=20 2.7.4 From nobody Fri May 3 14:42:45 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1498554490638285.2541389389563; Tue, 27 Jun 2017 02:08:10 -0700 (PDT) Received: from localhost ([::1]:51361 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPmTh-0008Rl-Ao for importer@patchew.org; Tue, 27 Jun 2017 05:08:09 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51326) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPmPk-0004tK-9J for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dPmPj-00034T-AO for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:04 -0400 Received: from mx1.redhat.com ([209.132.183.28]:33778) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dPmPj-00034E-2C for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:03 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 20935C04B93B for ; Tue, 27 Jun 2017 09:04:02 +0000 (UTC) Received: from pxdev.xzpeter.org.com (ovpn-12-81.pek2.redhat.com [10.72.12.81]) by smtp.corp.redhat.com (Postfix) with ESMTP id 989867FB53; Tue, 27 Jun 2017 09:04:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 20935C04B93B Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=peterx@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 20935C04B93B From: Peter Xu To: qemu-devel@nongnu.org Date: Tue, 27 Jun 2017 17:03:34 +0800 Message-Id: <1498554219-4942-4-git-send-email-peterx@redhat.com> In-Reply-To: <1498554219-4942-1-git-send-email-peterx@redhat.com> References: <1498554219-4942-1-git-send-email-peterx@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Tue, 27 Jun 2017 09:04:02 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 3/8] intel_iommu: support "info iommu" X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Wang , peterx@redhat.com, "Michael S . Tsirkin" Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Dump critical information for VT-d. Sample output: (qemu) info iommu Version: 1 Cap: 0x12008c22260286 Extended Cap: 0xf00f5a DMAR: enabled, root=3D0x7435f000 (extended=3D0) IR: enabled, root=3D0x17a400000, size=3D0x10000 (eim=3D1) QI: enabled, root=3D0x17aadf000, head=3D156, tail=3D156, size=3D256 Caching-mode: enabled Misc: next_frr=3D0, context_gen=3D2, buggy_eim=3D0 Signed-off-by: Peter Xu --- hw/i386/intel_iommu.c | 39 +++++++++++++++++++++++++++++++++++++++ hw/i386/x86-iommu.c | 17 +++++++++++++++++ include/hw/i386/x86-iommu.h | 5 +++++ 3 files changed, 61 insertions(+) diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c index a5c83dd..39f772a 100644 --- a/hw/i386/intel_iommu.c +++ b/hw/i386/intel_iommu.c @@ -2991,6 +2991,44 @@ static bool vtd_decide_config(IntelIOMMUState *s, Er= ror **errp) return true; } =20 +#define DUMP(...) monitor_printf(mon, ## __VA_ARGS__) +static void vtd_info_dump(X86IOMMUState *x86_iommu, Monitor *mon, + const QDict *qdict) +{ + IntelIOMMUState *s =3D INTEL_IOMMU_DEVICE(x86_iommu); + + DUMP("Version: %d\n", 1); + DUMP("Cap: 0x%"PRIx64"\n", s->cap); + DUMP("Extended Cap: 0x%"PRIx64"\n", s->ecap); + + DUMP("DMAR: %s", s->dmar_enabled ? "enabled" : "disabled"); + if (s->dmar_enabled) { + DUMP(", root=3D0x%"PRIx64" (extended=3D%d)", + s->root, s->root_extended); + } + DUMP("\n"); + + DUMP("IR: %s", s->intr_enabled ? "enabled" : "disabled"); + if (s->intr_enabled) { + DUMP(", root=3D0x%"PRIx64", size=3D0x%"PRIx32" (eim=3D%d)", + s->intr_root, s->intr_size, s->intr_eime); + } + DUMP("\n"); + + DUMP("QI: %s", s->qi_enabled ? "enabled" : "disabled"); + if (s->qi_enabled) { + DUMP(", root=3D0x%"PRIx64", head=3D%u, tail=3D%u, size=3D%u", + s->iq, s->iq_head, s->iq_tail, s->iq_size); + } + DUMP("\n"); + + DUMP("Caching-mode: %s\n", s->caching_mode ? "enabled" : "disabled"); + DUMP("Misc: next_frr=3D%d, context_gen=3D%d, buggy_eim=3D%d\n", + s->next_frcd_reg, s->context_cache_gen, s->buggy_eim); + DUMP(" iotlb_size=3D%d\n", g_hash_table_size(s->iotlb)); +} +#undef DUMP + static void vtd_realize(DeviceState *dev, Error **errp) { MachineState *ms =3D MACHINE(qdev_get_machine()); @@ -3042,6 +3080,7 @@ static void vtd_class_init(ObjectClass *klass, void *= data) dc->hotpluggable =3D false; x86_class->realize =3D vtd_realize; x86_class->int_remap =3D vtd_int_remap; + x86_class->info_dump =3D vtd_info_dump; /* Supported by the pc-q35-* machine types */ dc->user_creatable =3D true; } diff --git a/hw/i386/x86-iommu.c b/hw/i386/x86-iommu.c index 293caf8..fed35b4 100644 --- a/hw/i386/x86-iommu.c +++ b/hw/i386/x86-iommu.c @@ -76,6 +76,23 @@ IommuType x86_iommu_get_type(void) return x86_iommu_default->type; } =20 +void arch_iommu_info(Monitor *mon, const QDict *qdict) +{ + X86IOMMUState *iommu =3D x86_iommu_get_default(); + X86IOMMUClass *class; + + if (!iommu) { + monitor_printf(mon, "No IOMMU is detected.\n"); + return; + } + + class =3D X86_IOMMU_GET_CLASS(iommu); + + if (class->info_dump) { + class->info_dump(iommu, mon, qdict); + } +} + static void x86_iommu_realize(DeviceState *dev, Error **errp) { X86IOMMUState *x86_iommu =3D X86_IOMMU_DEVICE(dev); diff --git a/include/hw/i386/x86-iommu.h b/include/hw/i386/x86-iommu.h index ef89c0c..c414b65 100644 --- a/include/hw/i386/x86-iommu.h +++ b/include/hw/i386/x86-iommu.h @@ -22,6 +22,8 @@ =20 #include "hw/sysbus.h" #include "hw/pci/pci.h" +#include "hw/iommu.h" +#include "monitor/monitor.h" =20 #define TYPE_X86_IOMMU_DEVICE ("x86-iommu") #define X86_IOMMU_DEVICE(obj) \ @@ -50,6 +52,9 @@ struct X86IOMMUClass { /* MSI-based interrupt remapping */ int (*int_remap)(X86IOMMUState *iommu, MSIMessage *src, MSIMessage *dst, uint16_t sid); + /* Dump IOMMU information */ + void (*info_dump)(X86IOMMUState *iommu, Monitor *mon, + const QDict *qdict); }; =20 /** --=20 2.7.4 From nobody Fri May 3 14:42:45 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1498554590025478.26372636058966; Tue, 27 Jun 2017 02:09:50 -0700 (PDT) Received: from localhost ([::1]:51366 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPmVG-0001Qg-P6 for importer@patchew.org; Tue, 27 Jun 2017 05:09:46 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51355) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPmPr-0004xi-2I for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:17 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dPmPp-00036O-Ur for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:11 -0400 Received: from mx1.redhat.com ([209.132.183.28]:38252) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dPmPp-000362-Ma for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:09 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B126CC05972E for ; Tue, 27 Jun 2017 09:04:08 +0000 (UTC) Received: from pxdev.xzpeter.org.com (ovpn-12-81.pek2.redhat.com [10.72.12.81]) by smtp.corp.redhat.com (Postfix) with ESMTP id 871BE18B4B; Tue, 27 Jun 2017 09:04:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com B126CC05972E Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=peterx@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com B126CC05972E From: Peter Xu To: qemu-devel@nongnu.org Date: Tue, 27 Jun 2017 17:03:35 +0800 Message-Id: <1498554219-4942-5-git-send-email-peterx@redhat.com> In-Reply-To: <1498554219-4942-1-git-send-email-peterx@redhat.com> References: <1498554219-4942-1-git-send-email-peterx@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Tue, 27 Jun 2017 09:04:08 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 4/8] intel_iommu: add iotlb/context cache statistics X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Wang , peterx@redhat.com, "Michael S . Tsirkin" Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Add statistics for the VT-d IOMMU DMA remapping. Now "info iommu" shows us this for extra: Statistics: iotlb=3D26.35% (6689/25388), context=3D99.99% (18697/18699) Signed-off-by: Peter Xu --- hw/i386/intel_iommu.c | 21 +++++++++++++++++++++ include/hw/i386/intel_iommu.h | 10 ++++++++++ 2 files changed, 31 insertions(+) diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c index 39f772a..45d0919 100644 --- a/hw/i386/intel_iommu.c +++ b/hw/i386/intel_iommu.c @@ -37,6 +37,11 @@ #include "kvm_i386.h" #include "trace.h" =20 +static void vtd_reset_stats(IntelIOMMUState *s) +{ + memset(&s->cache_stat, 0, sizeof(s->cache_stat)); +} + static void vtd_define_quad(IntelIOMMUState *s, hwaddr addr, uint64_t val, uint64_t wmask, uint64_t w1cmask) { @@ -1095,9 +1100,12 @@ static bool vtd_do_iommu_translate(VTDAddressSpace *= vtd_as, PCIBus *bus, */ assert(!vtd_is_interrupt_addr(addr)); =20 + s->cache_stat.iotlb_total++; + /* Try to fetch slpte form IOTLB */ iotlb_entry =3D vtd_lookup_iotlb(s, source_id, addr); if (iotlb_entry) { + s->cache_stat.iotlb_hit++; trace_vtd_iotlb_page_hit(source_id, addr, iotlb_entry->slpte, iotlb_entry->domain_id); slpte =3D iotlb_entry->slpte; @@ -1107,8 +1115,11 @@ static bool vtd_do_iommu_translate(VTDAddressSpace *= vtd_as, PCIBus *bus, goto out; } =20 + s->cache_stat.context_total++; + /* Try to fetch context-entry from cache first */ if (cc_entry->context_cache_gen =3D=3D s->context_cache_gen) { + s->cache_stat.context_hit++; trace_vtd_iotlb_cc_hit(bus_num, devfn, cc_entry->context_entry.hi, cc_entry->context_entry.lo, cc_entry->context_cache_gen); @@ -2875,6 +2886,7 @@ static void vtd_init(IntelIOMMUState *s) =20 vtd_reset_context_cache(s); vtd_reset_iotlb(s); + vtd_reset_stats(s); =20 /* Define registers with default values and bit semantics */ vtd_define_long(s, DMAR_VER_REG, 0x10UL, 0, 0); @@ -3022,6 +3034,15 @@ static void vtd_info_dump(X86IOMMUState *x86_iommu, = Monitor *mon, } DUMP("\n"); =20 + DUMP("Statistics: iotlb=3D%.2lf%% (%"PRIu64"/%"PRIu64"), " + "context=3D%.2lf%% (%"PRIu64"/%"PRIu64")\n", + (double)s->cache_stat.iotlb_hit / + s->cache_stat.iotlb_total * 100, + s->cache_stat.iotlb_hit, s->cache_stat.iotlb_total, + (double)s->cache_stat.context_hit / + s->cache_stat.context_total * 100, + s->cache_stat.context_hit, s->cache_stat.context_total); + DUMP("Caching-mode: %s\n", s->caching_mode ? "enabled" : "disabled"); DUMP("Misc: next_frr=3D%d, context_gen=3D%d, buggy_eim=3D%d\n", s->next_frcd_reg, s->context_cache_gen, s->buggy_eim); diff --git a/include/hw/i386/intel_iommu.h b/include/hw/i386/intel_iommu.h index 3e51876..fc69ff3 100644 --- a/include/hw/i386/intel_iommu.h +++ b/include/hw/i386/intel_iommu.h @@ -255,6 +255,13 @@ struct IntelIOMMUNotifierNode { QLIST_ENTRY(IntelIOMMUNotifierNode) next; }; =20 +typedef struct IOMMUCacheStats { + uint64_t iotlb_hit; + uint64_t iotlb_total; + uint64_t context_hit; + uint64_t context_total; +} IOMMUCacheStats; + /* The iommu (DMAR) device state struct */ struct IntelIOMMUState { X86IOMMUState x86_iommu; @@ -302,6 +309,9 @@ struct IntelIOMMUState { bool intr_eime; /* Extended interrupt mode enabled */ OnOffAuto intr_eim; /* Toggle for EIM cabability */ bool buggy_eim; /* Force buggy EIM unless eim=3Doff */ + + /* For statistics */ + IOMMUCacheStats cache_stat; }; =20 /* Find the VTD Address space associated with the given bus pointer, --=20 2.7.4 From nobody Fri May 3 14:42:45 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1498554489458586.228485674335; Tue, 27 Jun 2017 02:08:09 -0700 (PDT) Received: from localhost ([::1]:51360 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPmTc-0008OV-4t for importer@patchew.org; Tue, 27 Jun 2017 05:08:04 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51367) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPmPt-000500-OU for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:14 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dPmPs-00037B-RO for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:13 -0400 Received: from mx1.redhat.com ([209.132.183.28]:40756) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dPmPs-00036z-LX for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:12 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A8278A8321 for ; Tue, 27 Jun 2017 09:04:11 +0000 (UTC) Received: from pxdev.xzpeter.org.com (ovpn-12-81.pek2.redhat.com [10.72.12.81]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2305618B4B; Tue, 27 Jun 2017 09:04:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com A8278A8321 Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=peterx@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com A8278A8321 From: Peter Xu To: qemu-devel@nongnu.org Date: Tue, 27 Jun 2017 17:03:36 +0800 Message-Id: <1498554219-4942-6-git-send-email-peterx@redhat.com> In-Reply-To: <1498554219-4942-1-git-send-email-peterx@redhat.com> References: <1498554219-4942-1-git-send-email-peterx@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Tue, 27 Jun 2017 09:04:11 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 5/8] intel_iommu: hmp: allow "-c" for "info iommu" X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Wang , peterx@redhat.com, "Michael S . Tsirkin" Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" New parameter "-c" for it to clear statistics. Other platforms can selectively support this (though none yet). Signed-off-by: Peter Xu --- hmp-commands-info.hx | 4 ++-- hw/i386/intel_iommu.c | 5 +++++ 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/hmp-commands-info.hx b/hmp-commands-info.hx index a39243d..2add941 100644 --- a/hmp-commands-info.hx +++ b/hmp-commands-info.hx @@ -803,8 +803,8 @@ ETEXI =20 { .name =3D "iommu", - .args_type =3D "", - .params =3D "", + .args_type =3D "clear_stats:-c", + .params =3D "[-c]", .help =3D "Display system IOMMU information", .cmd =3D hmp_info_iommu, }, diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c index 45d0919..72b39f0 100644 --- a/hw/i386/intel_iommu.c +++ b/hw/i386/intel_iommu.c @@ -3008,6 +3008,7 @@ static void vtd_info_dump(X86IOMMUState *x86_iommu, M= onitor *mon, const QDict *qdict) { IntelIOMMUState *s =3D INTEL_IOMMU_DEVICE(x86_iommu); + bool clear_stats =3D qdict_get_try_bool(qdict, "clear_stats", false); =20 DUMP("Version: %d\n", 1); DUMP("Cap: 0x%"PRIx64"\n", s->cap); @@ -3047,6 +3048,10 @@ static void vtd_info_dump(X86IOMMUState *x86_iommu, = Monitor *mon, DUMP("Misc: next_frr=3D%d, context_gen=3D%d, buggy_eim=3D%d\n", s->next_frcd_reg, s->context_cache_gen, s->buggy_eim); DUMP(" iotlb_size=3D%d\n", g_hash_table_size(s->iotlb)); + + if (clear_stats) { + vtd_reset_stats(s); + } } #undef DUMP =20 --=20 2.7.4 From nobody Fri May 3 14:42:45 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1498554443764249.76767887780488; Tue, 27 Jun 2017 02:07:23 -0700 (PDT) Received: from localhost ([::1]:51359 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPmSs-0007Ic-M6 for importer@patchew.org; Tue, 27 Jun 2017 05:07:19 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51400) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPmPz-00055X-RW for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dPmPu-00037v-TY for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:19 -0400 Received: from mx1.redhat.com ([209.132.183.28]:34024) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dPmPu-00037a-Kv for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:14 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A9F91C04B93B for ; Tue, 27 Jun 2017 09:04:13 +0000 (UTC) Received: from pxdev.xzpeter.org.com (ovpn-12-81.pek2.redhat.com [10.72.12.81]) by smtp.corp.redhat.com (Postfix) with ESMTP id 192BB7FB69; Tue, 27 Jun 2017 09:04:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com A9F91C04B93B Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=peterx@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com A9F91C04B93B From: Peter Xu To: qemu-devel@nongnu.org Date: Tue, 27 Jun 2017 17:03:37 +0800 Message-Id: <1498554219-4942-7-git-send-email-peterx@redhat.com> In-Reply-To: <1498554219-4942-1-git-send-email-peterx@redhat.com> References: <1498554219-4942-1-git-send-email-peterx@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Tue, 27 Jun 2017 09:04:13 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 6/8] intel_iommu: let iotlb size tunable X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Wang , peterx@redhat.com, "Michael S . Tsirkin" Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We were having static IOTLB size as 1024. Let it be a tunable. We can also turns IOTLB off if we want, by specify the size as zero. The tunable is named as "x-iotlb-size" since that should not really be something used by user yet, but mostly for debugging purpose now. Signed-off-by: Peter Xu --- hw/i386/intel_iommu.c | 14 ++++++++++++-- hw/i386/intel_iommu_internal.h | 1 - include/hw/i386/intel_iommu.h | 1 + 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c index 72b39f0..fc05764 100644 --- a/hw/i386/intel_iommu.c +++ b/hw/i386/intel_iommu.c @@ -227,6 +227,10 @@ static VTDIOTLBEntry *vtd_lookup_iotlb(IntelIOMMUState= *s, uint16_t source_id, uint64_t key; int level; =20 + if (s->iotlb_size =3D=3D 0) { + return NULL; + } + for (level =3D VTD_SL_PT_LEVEL; level < VTD_SL_PML4_LEVEL; level++) { key =3D vtd_get_iotlb_key(vtd_get_iotlb_gfn(addr, level), source_id, level); @@ -249,8 +253,12 @@ static void vtd_update_iotlb(IntelIOMMUState *s, uint1= 6_t source_id, uint64_t *key =3D g_malloc(sizeof(*key)); uint64_t gfn =3D vtd_get_iotlb_gfn(addr, level); =20 + if (s->iotlb_size =3D=3D 0) { + return; + } + trace_vtd_iotlb_page_update(source_id, addr, slpte, domain_id); - if (g_hash_table_size(s->iotlb) >=3D VTD_IOTLB_MAX_SIZE) { + if (g_hash_table_size(s->iotlb) >=3D s->iotlb_size) { trace_vtd_iotlb_reset("iotlb exceeds size limit"); vtd_reset_iotlb(s); } @@ -2388,6 +2396,7 @@ static Property vtd_properties[] =3D { ON_OFF_AUTO_AUTO), DEFINE_PROP_BOOL("x-buggy-eim", IntelIOMMUState, buggy_eim, false), DEFINE_PROP_BOOL("caching-mode", IntelIOMMUState, caching_mode, FALSE), + DEFINE_PROP_UINT16("x-iotlb-size", IntelIOMMUState, iotlb_size, 1024), DEFINE_PROP_END_OF_LIST(), }; =20 @@ -3047,7 +3056,8 @@ static void vtd_info_dump(X86IOMMUState *x86_iommu, M= onitor *mon, DUMP("Caching-mode: %s\n", s->caching_mode ? "enabled" : "disabled"); DUMP("Misc: next_frr=3D%d, context_gen=3D%d, buggy_eim=3D%d\n", s->next_frcd_reg, s->context_cache_gen, s->buggy_eim); - DUMP(" iotlb_size=3D%d\n", g_hash_table_size(s->iotlb)); + DUMP(" iotlb_size=3D%d/%d\n", g_hash_table_size(s->iotlb), + s->iotlb_size); =20 if (clear_stats) { vtd_reset_stats(s); diff --git a/hw/i386/intel_iommu_internal.h b/hw/i386/intel_iommu_internal.h index d1d6290..dc0257c 100644 --- a/hw/i386/intel_iommu_internal.h +++ b/hw/i386/intel_iommu_internal.h @@ -116,7 +116,6 @@ /* The shift of source_id in the key of IOTLB hash table */ #define VTD_IOTLB_SID_SHIFT 36 #define VTD_IOTLB_LVL_SHIFT 52 -#define VTD_IOTLB_MAX_SIZE 1024 /* Max size of the hash table = */ =20 /* IOTLB_REG */ #define VTD_TLB_GLOBAL_FLUSH (1ULL << 60) /* Global invalidation */ diff --git a/include/hw/i386/intel_iommu.h b/include/hw/i386/intel_iommu.h index fc69ff3..947c153 100644 --- a/include/hw/i386/intel_iommu.h +++ b/include/hw/i386/intel_iommu.h @@ -295,6 +295,7 @@ struct IntelIOMMUState { =20 uint32_t context_cache_gen; /* Should be in [1,MAX] */ GHashTable *iotlb; /* IOTLB */ + uint16_t iotlb_size; /* IOTLB max cache entries */ =20 MemoryRegionIOMMUOps iommu_ops; GHashTable *vtd_as_by_busptr; /* VTDBus objects indexed by PCIBus* r= eference */ --=20 2.7.4 From nobody Fri May 3 14:42:45 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1498554651776453.71462444212807; Tue, 27 Jun 2017 02:10:51 -0700 (PDT) Received: from localhost ([::1]:51375 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPmWI-00027P-9V for importer@patchew.org; Tue, 27 Jun 2017 05:10:50 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51401) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPmPz-00055Z-Ry for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:20 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dPmPw-00038a-N2 for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:19 -0400 Received: from mx1.redhat.com ([209.132.183.28]:44654) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dPmPw-00038J-EL for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:16 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7BF017A165 for ; Tue, 27 Jun 2017 09:04:15 +0000 (UTC) Received: from pxdev.xzpeter.org.com (ovpn-12-81.pek2.redhat.com [10.72.12.81]) by smtp.corp.redhat.com (Postfix) with ESMTP id 19E6918B4B; Tue, 27 Jun 2017 09:04:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 7BF017A165 Authentication-Results: ext-mx04.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx04.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=peterx@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 7BF017A165 From: Peter Xu To: qemu-devel@nongnu.org Date: Tue, 27 Jun 2017 17:03:38 +0800 Message-Id: <1498554219-4942-8-git-send-email-peterx@redhat.com> In-Reply-To: <1498554219-4942-1-git-send-email-peterx@redhat.com> References: <1498554219-4942-1-git-send-email-peterx@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Tue, 27 Jun 2017 09:04:15 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 7/8] intel_iommu: use access_flags for iotlb X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Wang , peterx@redhat.com, "Michael S . Tsirkin" Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" It was cached by read/write separately. Let's merge them. Signed-off-by: Peter Xu --- hw/i386/intel_iommu.c | 15 +++++++-------- include/hw/i386/intel_iommu.h | 3 +-- 2 files changed, 8 insertions(+), 10 deletions(-) diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c index fc05764..c2b2683 100644 --- a/hw/i386/intel_iommu.c +++ b/hw/i386/intel_iommu.c @@ -246,8 +246,7 @@ out: =20 static void vtd_update_iotlb(IntelIOMMUState *s, uint16_t source_id, uint16_t domain_id, hwaddr addr, uint64_t slp= te, - bool read_flags, bool write_flags, - uint32_t level) + uint8_t access_flags, uint32_t level) { VTDIOTLBEntry *entry =3D g_malloc(sizeof(*entry)); uint64_t *key =3D g_malloc(sizeof(*key)); @@ -266,8 +265,7 @@ static void vtd_update_iotlb(IntelIOMMUState *s, uint16= _t source_id, entry->gfn =3D gfn; entry->domain_id =3D domain_id; entry->slpte =3D slpte; - entry->read_flags =3D read_flags; - entry->write_flags =3D write_flags; + entry->access_flags =3D access_flags; entry->mask =3D vtd_slpt_level_page_mask(level); *key =3D vtd_get_iotlb_key(gfn, source_id, level); g_hash_table_replace(s->iotlb, key, entry); @@ -1100,6 +1098,7 @@ static bool vtd_do_iommu_translate(VTDAddressSpace *v= td_as, PCIBus *bus, bool is_fpd_set =3D false; bool reads =3D true; bool writes =3D true; + uint8_t access_flags; VTDIOTLBEntry *iotlb_entry; =20 /* @@ -1117,8 +1116,7 @@ static bool vtd_do_iommu_translate(VTDAddressSpace *v= td_as, PCIBus *bus, trace_vtd_iotlb_page_hit(source_id, addr, iotlb_entry->slpte, iotlb_entry->domain_id); slpte =3D iotlb_entry->slpte; - reads =3D iotlb_entry->read_flags; - writes =3D iotlb_entry->write_flags; + access_flags =3D iotlb_entry->access_flags; page_mask =3D iotlb_entry->mask; goto out; } @@ -1191,13 +1189,14 @@ static bool vtd_do_iommu_translate(VTDAddressSpace = *vtd_as, PCIBus *bus, } =20 page_mask =3D vtd_slpt_level_page_mask(level); + access_flags =3D IOMMU_ACCESS_FLAG(reads, writes); vtd_update_iotlb(s, source_id, VTD_CONTEXT_ENTRY_DID(ce.hi), addr, slp= te, - reads, writes, level); + access_flags, level); out: entry->iova =3D addr & page_mask; entry->translated_addr =3D vtd_get_slpte_addr(slpte) & page_mask; entry->addr_mask =3D ~page_mask; - entry->perm =3D IOMMU_ACCESS_FLAG(reads, writes); + entry->perm =3D access_flags; return true; =20 error: diff --git a/include/hw/i386/intel_iommu.h b/include/hw/i386/intel_iommu.h index 947c153..4960f8d 100644 --- a/include/hw/i386/intel_iommu.h +++ b/include/hw/i386/intel_iommu.h @@ -101,8 +101,7 @@ struct VTDIOTLBEntry { uint16_t domain_id; uint64_t slpte; uint64_t mask; - bool read_flags; - bool write_flags; + uint8_t access_flags; }; =20 /* VT-d Source-ID Qualifier types */ --=20 2.7.4 From nobody Fri May 3 14:42:45 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1498554557059439.4776172512662; Tue, 27 Jun 2017 02:09:17 -0700 (PDT) Received: from localhost ([::1]:51365 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPmUl-00015v-OE for importer@patchew.org; Tue, 27 Jun 2017 05:09:15 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51415) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPmQ0-000562-CU for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dPmPy-00039F-Ol for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:20 -0400 Received: from mx1.redhat.com ([209.132.183.28]:52360) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dPmPy-00038v-G0 for qemu-devel@nongnu.org; Tue, 27 Jun 2017 05:04:18 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 730152E97CE for ; Tue, 27 Jun 2017 09:04:17 +0000 (UTC) Received: from pxdev.xzpeter.org.com (ovpn-12-81.pek2.redhat.com [10.72.12.81]) by smtp.corp.redhat.com (Postfix) with ESMTP id F33FD7FB55; Tue, 27 Jun 2017 09:04:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 730152E97CE Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=peterx@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 730152E97CE From: Peter Xu To: qemu-devel@nongnu.org Date: Tue, 27 Jun 2017 17:03:39 +0800 Message-Id: <1498554219-4942-9-git-send-email-peterx@redhat.com> In-Reply-To: <1498554219-4942-1-git-send-email-peterx@redhat.com> References: <1498554219-4942-1-git-send-email-peterx@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Tue, 27 Jun 2017 09:04:17 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 8/8] intel_iommu: implement mru list for iotlb X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Wang , peterx@redhat.com, "Michael S . Tsirkin" Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" It is not wise to disgard all the IOTLB cache when cache size reaches max, but that's what we do now. A slightly better (but still simple) way to do this is, we just throw away the least recent used cache entry. This patch implemented MRU list algorithm for VT-d IOTLB. The main logic is to maintain a Most Recently Used list for the IOTLB entries. The hash table is still used for lookup, though a new list field is added to each IOTLB entry for a iotlb MRU list. For each active IOTLB entry, it's both in the hash table in s->iotlb, and also linked into the MRU list of s->iotlb_head. The hash helps in fast lookup, and the MRU list helps in knowing whether the cache is still hot. After we have such a MRU list, replacing all the iterators of IOTLB entries by using list iterations rather than hash table iterations. Signed-off-by: Peter Xu --- hw/i386/intel_iommu.c | 75 +++++++++++++++++++++++++-------------= ---- hw/i386/intel_iommu_internal.h | 8 ----- hw/i386/trace-events | 1 - include/hw/i386/intel_iommu.h | 6 +++- 4 files changed, 50 insertions(+), 40 deletions(-) diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c index c2b2683..8375fc3 100644 --- a/hw/i386/intel_iommu.c +++ b/hw/i386/intel_iommu.c @@ -37,6 +37,9 @@ #include "kvm_i386.h" #include "trace.h" =20 +#define FOREACH_IOTLB_SAFE(entry, s, entry_n) \ + QTAILQ_FOREACH_SAFE(entry, &(s)->iotlb_head, link, entry_n) + static void vtd_reset_stats(IntelIOMMUState *s) { memset(&s->cache_stat, 0, sizeof(s->cache_stat)); @@ -144,14 +147,6 @@ static guint vtd_uint64_hash(gconstpointer v) return (guint)*(const uint64_t *)v; } =20 -static gboolean vtd_hash_remove_by_domain(gpointer key, gpointer value, - gpointer user_data) -{ - VTDIOTLBEntry *entry =3D (VTDIOTLBEntry *)value; - uint16_t domain_id =3D *(uint16_t *)user_data; - return entry->domain_id =3D=3D domain_id; -} - /* The shift of an addr for a certain level of paging structure */ static inline uint32_t vtd_slpt_level_shift(uint32_t level) { @@ -164,18 +159,6 @@ static inline uint64_t vtd_slpt_level_page_mask(uint32= _t level) return ~((1ULL << vtd_slpt_level_shift(level)) - 1); } =20 -static gboolean vtd_hash_remove_by_page(gpointer key, gpointer value, - gpointer user_data) -{ - VTDIOTLBEntry *entry =3D (VTDIOTLBEntry *)value; - VTDIOTLBPageInvInfo *info =3D (VTDIOTLBPageInvInfo *)user_data; - uint64_t gfn =3D (info->addr >> VTD_PAGE_SHIFT_4K) & info->mask; - uint64_t gfn_tlb =3D (info->addr & entry->mask) >> VTD_PAGE_SHIFT_4K; - return (entry->domain_id =3D=3D info->domain_id) && - (((entry->gfn & info->mask) =3D=3D gfn) || - (entry->gfn =3D=3D gfn_tlb)); -} - /* Reset all the gen of VTDAddressSpace to zero and set the gen of * IntelIOMMUState to 1. */ @@ -206,6 +189,7 @@ static void vtd_reset_iotlb(IntelIOMMUState *s) { assert(s->iotlb); g_hash_table_remove_all(s->iotlb); + QTAILQ_INIT(&s->iotlb_head); } =20 static uint64_t vtd_get_iotlb_key(uint64_t gfn, uint16_t source_id, @@ -236,6 +220,9 @@ static VTDIOTLBEntry *vtd_lookup_iotlb(IntelIOMMUState = *s, uint16_t source_id, source_id, level); entry =3D g_hash_table_lookup(s->iotlb, &key); if (entry) { + /* Move the entry to the head of MRU list */ + QTAILQ_REMOVE(&s->iotlb_head, entry, link); + QTAILQ_INSERT_HEAD(&s->iotlb_head, entry, link); goto out; } } @@ -244,11 +231,23 @@ out: return entry; } =20 +static void vtd_iotlb_remove_entry(IntelIOMMUState *s, VTDIOTLBEntry *entr= y) +{ + uint64_t key =3D entry->key; + + /* + * To remove an entry, we need to both remove it from the MRU + * list, and also from the hash table. + */ + QTAILQ_REMOVE(&s->iotlb_head, entry, link); + g_hash_table_remove(s->iotlb, &key); +} + static void vtd_update_iotlb(IntelIOMMUState *s, uint16_t source_id, uint16_t domain_id, hwaddr addr, uint64_t slp= te, uint8_t access_flags, uint32_t level) { - VTDIOTLBEntry *entry =3D g_malloc(sizeof(*entry)); + VTDIOTLBEntry *entry =3D g_new0(VTDIOTLBEntry, 1), *last; uint64_t *key =3D g_malloc(sizeof(*key)); uint64_t gfn =3D vtd_get_iotlb_gfn(addr, level); =20 @@ -258,8 +257,9 @@ static void vtd_update_iotlb(IntelIOMMUState *s, uint16= _t source_id, =20 trace_vtd_iotlb_page_update(source_id, addr, slpte, domain_id); if (g_hash_table_size(s->iotlb) >=3D s->iotlb_size) { - trace_vtd_iotlb_reset("iotlb exceeds size limit"); - vtd_reset_iotlb(s); + /* Remove the Least Recently Used cache */ + last =3D QTAILQ_LAST(&s->iotlb_head, VTDIOTLBEntryHead); + vtd_iotlb_remove_entry(s, last); } =20 entry->gfn =3D gfn; @@ -268,7 +268,11 @@ static void vtd_update_iotlb(IntelIOMMUState *s, uint1= 6_t source_id, entry->access_flags =3D access_flags; entry->mask =3D vtd_slpt_level_page_mask(level); *key =3D vtd_get_iotlb_key(gfn, source_id, level); + entry->key =3D *key; g_hash_table_replace(s->iotlb, key, entry); + + /* Update MRU list */ + QTAILQ_INSERT_HEAD(&s->iotlb_head, entry, link); } =20 /* Given the reg addr of both the message data and address, generate an @@ -1365,11 +1369,15 @@ static void vtd_iotlb_domain_invalidate(IntelIOMMUS= tate *s, uint16_t domain_id) IntelIOMMUNotifierNode *node; VTDContextEntry ce; VTDAddressSpace *vtd_as; + VTDIOTLBEntry *entry, *entry_n; =20 trace_vtd_inv_desc_iotlb_domain(domain_id); =20 - g_hash_table_foreach_remove(s->iotlb, vtd_hash_remove_by_domain, - &domain_id); + FOREACH_IOTLB_SAFE(entry, s, entry_n) { + if (entry->domain_id =3D=3D domain_id) { + vtd_iotlb_remove_entry(s, entry); + } + } =20 QLIST_FOREACH(node, &s->notifiers_list, next) { vtd_as =3D node->vtd_as; @@ -1411,15 +1419,22 @@ static void vtd_iotlb_page_invalidate_notify(IntelI= OMMUState *s, static void vtd_iotlb_page_invalidate(IntelIOMMUState *s, uint16_t domain_= id, hwaddr addr, uint8_t am) { - VTDIOTLBPageInvInfo info; + VTDIOTLBEntry *entry, *entry_n; + uint64_t gfn, mask; =20 trace_vtd_inv_desc_iotlb_pages(domain_id, addr, am); =20 assert(am <=3D VTD_MAMV); - info.domain_id =3D domain_id; - info.addr =3D addr; - info.mask =3D ~((1 << am) - 1); - g_hash_table_foreach_remove(s->iotlb, vtd_hash_remove_by_page, &info); + + mask =3D ~((1 << am) - 1); + gfn =3D (addr >> VTD_PAGE_SHIFT) & mask; + + FOREACH_IOTLB_SAFE(entry, s, entry_n) { + if (entry->domain_id =3D=3D domain_id && (entry->gfn & mask) =3D= =3D gfn) { + vtd_iotlb_remove_entry(s, entry); + } + } + vtd_iotlb_page_invalidate_notify(s, domain_id, addr, am); } =20 diff --git a/hw/i386/intel_iommu_internal.h b/hw/i386/intel_iommu_internal.h index dc0257c..84e68e1 100644 --- a/hw/i386/intel_iommu_internal.h +++ b/hw/i386/intel_iommu_internal.h @@ -372,14 +372,6 @@ typedef union VTDInvDesc VTDInvDesc; #define VTD_INV_DESC_DEVICE_IOTLB_RSVD_HI 0xffeULL #define VTD_INV_DESC_DEVICE_IOTLB_RSVD_LO 0xffff0000ffe0fff8 =20 -/* Information about page-selective IOTLB invalidate */ -struct VTDIOTLBPageInvInfo { - uint16_t domain_id; - uint64_t addr; - uint8_t mask; -}; -typedef struct VTDIOTLBPageInvInfo VTDIOTLBPageInvInfo; - /* Pagesize of VTD paging structures, including root and context tables */ #define VTD_PAGE_SHIFT 12 #define VTD_PAGE_SIZE (1ULL << VTD_PAGE_SHIFT) diff --git a/hw/i386/trace-events b/hw/i386/trace-events index 5f111d6..edf0dca 100644 --- a/hw/i386/trace-events +++ b/hw/i386/trace-events @@ -34,7 +34,6 @@ vtd_iotlb_page_hit(uint16_t sid, uint64_t addr, uint64_t = slpte, uint16_t domain) vtd_iotlb_page_update(uint16_t sid, uint64_t addr, uint64_t slpte, uint16_= t domain) "IOTLB page update sid 0x%"PRIx16" iova 0x%"PRIx64" slpte 0x%"PRI= x64" domain 0x%"PRIx16 vtd_iotlb_cc_hit(uint8_t bus, uint8_t devfn, uint64_t high, uint64_t low, = uint32_t gen) "IOTLB context hit bus 0x%"PRIx8" devfn 0x%"PRIx8" high 0x%"P= RIx64" low 0x%"PRIx64" gen %"PRIu32 vtd_iotlb_cc_update(uint8_t bus, uint8_t devfn, uint64_t high, uint64_t lo= w, uint32_t gen1, uint32_t gen2) "IOTLB context update bus 0x%"PRIx8" devfn= 0x%"PRIx8" high 0x%"PRIx64" low 0x%"PRIx64" gen %"PRIu32" -> gen %"PRIu32 -vtd_iotlb_reset(const char *reason) "IOTLB reset (reason: %s)" vtd_fault_disabled(void) "Fault processing disabled for context entry" vtd_replay_ce_valid(uint8_t bus, uint8_t dev, uint8_t fn, uint16_t domain,= uint64_t hi, uint64_t lo) "replay valid context device %02"PRIx8":%02"PRIx= 8".%02"PRIx8" domain 0x%"PRIx16" hi 0x%"PRIx64" lo 0x%"PRIx64 vtd_replay_ce_invalid(uint8_t bus, uint8_t dev, uint8_t fn) "replay invali= d context device %02"PRIx8":%02"PRIx8".%02"PRIx8 diff --git a/include/hw/i386/intel_iommu.h b/include/hw/i386/intel_iommu.h index 4960f8d..bd17a24 100644 --- a/include/hw/i386/intel_iommu.h +++ b/include/hw/i386/intel_iommu.h @@ -98,9 +98,11 @@ struct VTDBus { =20 struct VTDIOTLBEntry { uint64_t gfn; - uint16_t domain_id; uint64_t slpte; uint64_t mask; + uint64_t key; + QTAILQ_ENTRY(VTDIOTLBEntry) link; + uint16_t domain_id; uint8_t access_flags; }; =20 @@ -295,6 +297,8 @@ struct IntelIOMMUState { uint32_t context_cache_gen; /* Should be in [1,MAX] */ GHashTable *iotlb; /* IOTLB */ uint16_t iotlb_size; /* IOTLB max cache entries */ + /* Head of IOTLB MRU list */ + QTAILQ_HEAD(VTDIOTLBEntryHead, VTDIOTLBEntry) iotlb_head; =20 MemoryRegionIOMMUOps iommu_ops; GHashTable *vtd_as_by_busptr; /* VTDBus objects indexed by PCIBus* r= eference */ --=20 2.7.4