Hopefully last target-arm queue before softfreeze; this one's largest part is the remainder of the SVE patches, but there are a selection of other minor things too. thanks -- PMM The following changes since commit 109b25045b3651f9c5d02c3766c0b3ff63e6d193: Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into staging (2018-06-29 12:30:29 +0100) are available in the Git repository at: git://git.linaro.org/people/pmaydell/qemu-arm.git tags/pull-target-arm-20180629 for you to fetch changes up to 802abf4024d23e48d45373ac3f2b580124b54b47: target/arm: Add ID_ISAR6 (2018-06-29 15:30:54 +0100) ---------------------------------------------------------------- target-arm queue: * last of the SVE patches; SVE is now enabled for aarch64 linux-user * sd: Don't trace SDRequest crc field (coverity bugfix) * target/arm: Mark PMINTENSET accesses as possibly doing IO * clean up v7VE feature bit handling * i.mx7d: minor cleanups * target/arm: support reading of CNT[VCT|FRQ]_EL0 from user-space * target/arm: Implement ARMv8.2-DotProd * virt: add addresses to dt node names (which stops dtc from complaining that they're not correctly named) * cleanups: replace error_setg(&error_fatal) by error_report() + exit() ---------------------------------------------------------------- Aaron Lindsay (3): target/arm: Add ARM_FEATURE_V7VE for v7 Virtualization Extensions target/arm: Remove redundant DIV detection for KVM target/arm: Mark PMINTENSET accesses as possibly doing IO Alex Bennée (1): target/arm: support reading of CNT[VCT|FRQ]_EL0 from user-space Eric Auger (3): device_tree: Add qemu_fdt_node_unit_path hw/arm/virt: Silence dtc /intc warnings hw/arm/virt: Silence dtc /memory warning Jean-Christophe Dubois (3): i.mx7d: Remove unused header files i.mx7d: Change SRC unimplemented device name from sdma to src i.mx7d: Change IRQ number type from hwaddr to int Peter Maydell (1): sd: Don't trace SDRequest crc field Philippe Mathieu-Daudé (4): hw/block/fdc: Replace error_setg(&error_abort) by assert() hw/arm/sysbus-fdt: Replace error_setg(&error_fatal) by error_report() + exit() device_tree: Replace error_setg(&error_fatal) by error_report() + exit() sdcard: Use the ldst API Richard Henderson (40): target/arm: Implement SVE Memory Contiguous Load Group target/arm: Implement SVE Contiguous Load, first-fault and no-fault target/arm: Implement SVE Memory Contiguous Store Group target/arm: Implement SVE load and broadcast quadword target/arm: Implement SVE integer convert to floating-point target/arm: Implement SVE floating-point arithmetic (predicated) target/arm: Implement SVE FP Multiply-Add Group target/arm: Implement SVE Floating Point Accumulating Reduction Group target/arm: Implement SVE load and broadcast element target/arm: Implement SVE store vector/predicate register target/arm: Implement SVE scatter stores target/arm: Implement SVE prefetches target/arm: Implement SVE gather loads target/arm: Implement SVE first-fault gather loads target/arm: Implement SVE scatter store vector immediate target/arm: Implement SVE floating-point compare vectors target/arm: Implement SVE floating-point arithmetic with immediate target/arm: Implement SVE Floating Point Multiply Indexed Group target/arm: Implement SVE FP Fast Reduction Group target/arm: Implement SVE Floating Point Unary Operations - Unpredicated Group target/arm: Implement SVE FP Compare with Zero Group target/arm: Implement SVE floating-point trig multiply-add coefficient target/arm: Implement SVE floating-point convert precision target/arm: Implement SVE floating-point convert to integer target/arm: Implement SVE floating-point round to integral value target/arm: Implement SVE floating-point unary operations target/arm: Implement SVE MOVPRFX target/arm: Implement SVE floating-point complex add target/arm: Implement SVE fp complex multiply add target/arm: Pass index to AdvSIMD FCMLA (indexed) target/arm: Implement SVE fp complex multiply add (indexed) target/arm: Implement SVE dot product (vectors) target/arm: Implement SVE dot product (indexed) target/arm: Enable SVE for aarch64-linux-user target/arm: Implement ARMv8.2-DotProd target/arm: Fix SVE signed division vs x86 overflow exception target/arm: Fix SVE system register access checks target/arm: Prune a57 features from max target/arm: Prune a15 features from max target/arm: Add ID_ISAR6 include/sysemu/device_tree.h | 16 + target/arm/cpu.h | 3 + target/arm/helper-sve.h | 682 +++++++++++++++ target/arm/helper.h | 44 +- device_tree.c | 78 +- hw/arm/boot.c | 41 +- hw/arm/fsl-imx7.c | 8 +- hw/arm/mcimx7d-sabre.c | 2 - hw/arm/sysbus-fdt.c | 53 +- hw/arm/virt.c | 70 +- hw/block/fdc.c | 9 +- hw/sd/bcm2835_sdhost.c | 13 +- hw/sd/core.c | 2 +- hw/sd/milkymist-memcard.c | 3 +- hw/sd/omap_mmc.c | 6 +- hw/sd/pl181.c | 11 +- hw/sd/sdhci.c | 15 +- hw/sd/ssi-sd.c | 6 +- linux-user/elfload.c | 2 + target/arm/cpu.c | 36 +- target/arm/cpu64.c | 13 +- target/arm/helper.c | 44 +- target/arm/kvm32.c | 27 +- target/arm/sve_helper.c | 1875 +++++++++++++++++++++++++++++++++++++++++- target/arm/translate-a64.c | 62 +- target/arm/translate-sve.c | 1688 ++++++++++++++++++++++++++++++++++++- target/arm/translate.c | 102 ++- target/arm/vec_helper.c | 311 ++++++- hw/sd/trace-events | 2 +- target/arm/sve.decode | 427 ++++++++++ 30 files changed, 5394 insertions(+), 257 deletions(-) From: Philippe Mathieu-Daudé <f4bug@amsat.org> Use assert() instead of error_setg(&error_abort), as suggested by the "qapi/error.h" documentation: Please don't error_setg(&error_fatal, ...), use error_report() and exit(), because that's more obvious. Likewise, don't error_setg(&error_abort, ...), use assert(). Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Acked-by: John Snow <jsnow@redhat.com> Message-id: 20180625165749.3910-2-f4bug@amsat.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- hw/block/fdc.c | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/hw/block/fdc.c b/hw/block/fdc.c index XXXXXXX..XXXXXXX 100644 --- a/hw/block/fdc.c +++ b/hw/block/fdc.c @@ -XXX,XX +XXX,XX @@ static int pick_geometry(FDrive *drv) nb_sectors, FloppyDriveType_str(parse->drive)); } + assert(type_match != -1 && "misconfigured fd_format"); match = type_match; } - - /* No match of any kind found -- fd_format is misconfigured, abort. */ - if (match == -1) { - error_setg(&error_abort, "No candidate geometries present in table " - " for floppy drive type '%s'", - FloppyDriveType_str(drv->drive)); - } - parse = &(fd_formats[match]); out: -- 2.17.1 From: Philippe Mathieu-Daudé <f4bug@amsat.org> Use error_report() + exit() instead of error_setg(&error_fatal), as suggested by the "qapi/error.h" documentation: Please don't error_setg(&error_fatal, ...), use error_report() and exit(), because that's more obvious. This fixes CID 1352173: "Passing null pointer dt_name to qemu_fdt_node_path, which dereferences it." And this also fixes: hw/arm/sysbus-fdt.c:322:9: warning: Array access (from variable 'node_path') results in a null pointer dereference if (node_path[1]) { ^~~~~~~~~~~~ Fixes: Coverity CID 1352173 (Dereference after null check) Suggested-by: Eric Blake <eblake@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Eric Auger <eric.auger@redhat.com> Message-id: 20180625165749.3910-3-f4bug@amsat.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- hw/arm/sysbus-fdt.c | 53 +++++++++++++++++++++++++-------------------- 1 file changed, 30 insertions(+), 23 deletions(-) diff --git a/hw/arm/sysbus-fdt.c b/hw/arm/sysbus-fdt.c index XXXXXXX..XXXXXXX 100644 --- a/hw/arm/sysbus-fdt.c +++ b/hw/arm/sysbus-fdt.c @@ -XXX,XX +XXX,XX @@ static void copy_properties_from_host(HostProperty *props, int nb_props, r = qemu_fdt_getprop(host_fdt, node_path, props[i].name, &prop_len, - props[i].optional ? &err : &error_fatal); + &err); if (r) { qemu_fdt_setprop(guest_fdt, nodename, props[i].name, r, prop_len); } else { - if (prop_len != -FDT_ERR_NOTFOUND) { - /* optional property not returned although property exists */ - error_report_err(err); - } else { + if (props[i].optional && prop_len == -FDT_ERR_NOTFOUND) { + /* optional property does not exist */ error_free(err); + } else { + error_report_err(err); + } + if (!props[i].optional) { + /* mandatory property not found: bail out */ + exit(1); } } } @@ -XXX,XX +XXX,XX @@ static void fdt_build_clock_node(void *host_fdt, void *guest_fdt, node_offset = fdt_node_offset_by_phandle(host_fdt, host_phandle); if (node_offset <= 0) { - error_setg(&error_fatal, - "not able to locate clock handle %d in host device tree", - host_phandle); + error_report("not able to locate clock handle %d in host device tree", + host_phandle); + exit(1); } node_path = g_malloc(path_len); while ((ret = fdt_get_path(host_fdt, node_offset, node_path, path_len)) @@ -XXX,XX +XXX,XX @@ static void fdt_build_clock_node(void *host_fdt, void *guest_fdt, node_path = g_realloc(node_path, path_len); } if (ret < 0) { - error_setg(&error_fatal, - "not able to retrieve node path for clock handle %d", - host_phandle); + error_report("not able to retrieve node path for clock handle %d", + host_phandle); + exit(1); } r = qemu_fdt_getprop(host_fdt, node_path, "compatible", &prop_len, &error_fatal); if (strcmp(r, "fixed-clock")) { - error_setg(&error_fatal, - "clock handle %d is not a fixed clock", host_phandle); + error_report("clock handle %d is not a fixed clock", host_phandle); + exit(1); } nodename = strrchr(node_path, '/'); @@ -XXX,XX +XXX,XX @@ static int add_amd_xgbe_fdt_node(SysBusDevice *sbdev, void *opaque) dt_name = sysfs_to_dt_name(vbasedev->name); if (!dt_name) { - error_setg(&error_fatal, "%s incorrect sysfs device name %s", - __func__, vbasedev->name); + error_report("%s incorrect sysfs device name %s", + __func__, vbasedev->name); + exit(1); } node_path = qemu_fdt_node_path(host_fdt, dt_name, vdev->compat, &error_fatal); if (!node_path || !node_path[0]) { - error_setg(&error_fatal, "%s unable to retrieve node path for %s/%s", - __func__, dt_name, vdev->compat); + error_report("%s unable to retrieve node path for %s/%s", + __func__, dt_name, vdev->compat); + exit(1); } if (node_path[1]) { - error_setg(&error_fatal, "%s more than one node matching %s/%s!", - __func__, dt_name, vdev->compat); + error_report("%s more than one node matching %s/%s!", + __func__, dt_name, vdev->compat); + exit(1); } g_free(dt_name); if (vbasedev->num_regions != 5) { - error_setg(&error_fatal, "%s Does the host dt node combine XGBE/PHY?", - __func__); + error_report("%s Does the host dt node combine XGBE/PHY?", __func__); + exit(1); } /* generate nodes for DMA_CLK and PTP_CLK */ r = qemu_fdt_getprop(host_fdt, node_path[0], "clocks", &prop_len, &error_fatal); if (prop_len != 8) { - error_setg(&error_fatal, "%s clocks property should contain 2 handles", - __func__); + error_report("%s clocks property should contain 2 handles", __func__); + exit(1); } host_clock_phandles = (uint32_t *)r; guest_clock_phandles[0] = qemu_fdt_alloc_phandle(guest_fdt); -- 2.17.1 From: Philippe Mathieu-Daudé <f4bug@amsat.org> Use error_report() + exit() instead of error_setg(&error_fatal), as suggested by the "qapi/error.h" documentation: Please don't error_setg(&error_fatal, ...), use error_report() and exit(), because that's more obvious. Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Eric Auger <eric.auger@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: David Gibson <david@gibson.dropbear.id.au> Message-id: 20180625165749.3910-4-f4bug@amsat.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- device_tree.c | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/device_tree.c b/device_tree.c index XXXXXXX..XXXXXXX 100644 --- a/device_tree.c +++ b/device_tree.c @@ -XXX,XX +XXX,XX @@ static void read_fstree(void *fdt, const char *dirname) const char *parent_node; if (strstr(dirname, root_dir) != dirname) { - error_setg(&error_fatal, "%s: %s must be searched within %s", - __func__, dirname, root_dir); + error_report("%s: %s must be searched within %s", + __func__, dirname, root_dir); + exit(1); } parent_node = &dirname[strlen(SYSFS_DT_BASEDIR)]; d = opendir(dirname); if (!d) { - error_setg(&error_fatal, "%s cannot open %s", __func__, dirname); - return; + error_report("%s cannot open %s", __func__, dirname); + exit(1); } while ((de = readdir(d)) != NULL) { @@ -XXX,XX +XXX,XX @@ static void read_fstree(void *fdt, const char *dirname) tmpnam = g_strdup_printf("%s/%s", dirname, de->d_name); if (lstat(tmpnam, &st) < 0) { - error_setg(&error_fatal, "%s cannot lstat %s", __func__, tmpnam); + error_report("%s cannot lstat %s", __func__, tmpnam); + exit(1); } if (S_ISREG(st.st_mode)) { @@ -XXX,XX +XXX,XX @@ static void read_fstree(void *fdt, const char *dirname) gsize len; if (!g_file_get_contents(tmpnam, &val, &len, NULL)) { - error_setg(&error_fatal, "%s not able to extract info from %s", - __func__, tmpnam); + error_report("%s not able to extract info from %s", + __func__, tmpnam); + exit(1); } if (strlen(parent_node) > 0) { @@ -XXX,XX +XXX,XX @@ void *load_device_tree_from_sysfs(void) host_fdt = create_device_tree(&host_fdt_size); read_fstree(host_fdt, SYSFS_DT_BASEDIR); if (fdt_check_header(host_fdt)) { - error_setg(&error_fatal, - "%s host device tree extracted into memory is invalid", - __func__); + error_report("%s host device tree extracted into memory is invalid", + __func__); + exit(1); } return host_fdt; } -- 2.17.1 From: Eric Auger <eric.auger@redhat.com> This helper allows to retrieve the paths of nodes whose name match node-name or node-name@unit-address patterns. Signed-off-by: Eric Auger <eric.auger@redhat.com> Message-id: 1530044492-24921-2-git-send-email-eric.auger@redhat.com Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- include/sysemu/device_tree.h | 16 +++++++++++ device_tree.c | 55 ++++++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+) diff --git a/include/sysemu/device_tree.h b/include/sysemu/device_tree.h index XXXXXXX..XXXXXXX 100644 --- a/include/sysemu/device_tree.h +++ b/include/sysemu/device_tree.h @@ -XXX,XX +XXX,XX @@ void *load_device_tree_from_sysfs(void); char **qemu_fdt_node_path(void *fdt, const char *name, char *compat, Error **errp); +/** + * qemu_fdt_node_unit_path: return the paths of nodes matching a given + * node-name, ie. node-name and node-name@unit-address + * @fdt: pointer to the dt blob + * @name: node name + * @errp: handle to an error object + * + * returns a newly allocated NULL-terminated array of node paths. + * Use g_strfreev() to free it. If one or more nodes were found, the + * array contains the path of each node and the last element equals to + * NULL. If there is no error but no matching node was found, the + * returned array contains a single element equal to NULL. If an error + * was encountered when parsing the blob, the function returns NULL + */ +char **qemu_fdt_node_unit_path(void *fdt, const char *name, Error **errp); + int qemu_fdt_setprop(void *fdt, const char *node_path, const char *property, const void *val, int size); int qemu_fdt_setprop_cell(void *fdt, const char *node_path, diff --git a/device_tree.c b/device_tree.c index XXXXXXX..XXXXXXX 100644 --- a/device_tree.c +++ b/device_tree.c @@ -XXX,XX +XXX,XX @@ static int findnode_nofail(void *fdt, const char *node_path) return offset; } +char **qemu_fdt_node_unit_path(void *fdt, const char *name, Error **errp) +{ + char *prefix = g_strdup_printf("%s@", name); + unsigned int path_len = 16, n = 0; + GSList *path_list = NULL, *iter; + const char *iter_name; + int offset, len, ret; + char **path_array; + + offset = fdt_next_node(fdt, -1, NULL); + + while (offset >= 0) { + iter_name = fdt_get_name(fdt, offset, &len); + if (!iter_name) { + offset = len; + break; + } + if (!strcmp(iter_name, name) || g_str_has_prefix(iter_name, prefix)) { + char *path; + + path = g_malloc(path_len); + while ((ret = fdt_get_path(fdt, offset, path, path_len)) + == -FDT_ERR_NOSPACE) { + path_len += 16; + path = g_realloc(path, path_len); + } + path_list = g_slist_prepend(path_list, path); + n++; + } + offset = fdt_next_node(fdt, offset, NULL); + } + g_free(prefix); + + if (offset < 0 && offset != -FDT_ERR_NOTFOUND) { + error_setg(errp, "%s: abort parsing dt for %s node units: %s", + __func__, name, fdt_strerror(offset)); + for (iter = path_list; iter; iter = iter->next) { + g_free(iter->data); + } + g_slist_free(path_list); + return NULL; + } + + path_array = g_new(char *, n + 1); + path_array[n--] = NULL; + + for (iter = path_list; iter; iter = iter->next) { + path_array[n--] = iter->data; + } + + g_slist_free(path_list); + + return path_array; +} + char **qemu_fdt_node_path(void *fdt, const char *name, char *compat, Error **errp) { -- 2.17.1 From: Eric Auger <eric.auger@redhat.com> When running dtc on the guest /proc/device-tree we get the following warnings: "Warning (unit_address_vs_reg): Node <name> has a reg or ranges property, but no unit name", with name: /intc, /intc/its, /intc/v2m. Nodes should have a name in the form <name>[@<unit-address>] where unit-address is the primary address used to access the device, listed in the node's reg property. This fix seems to make dtc happy. Signed-off-by: Eric Auger <eric.auger@redhat.com> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Message-id: 1530044492-24921-3-git-send-email-eric.auger@redhat.com Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- hw/arm/virt.c | 63 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 39 insertions(+), 24 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index XXXXXXX..XXXXXXX 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -XXX,XX +XXX,XX @@ static void fdt_add_cpu_nodes(const VirtMachineState *vms) static void fdt_add_its_gic_node(VirtMachineState *vms) { + char *nodename; + vms->msi_phandle = qemu_fdt_alloc_phandle(vms->fdt); - qemu_fdt_add_subnode(vms->fdt, "/intc/its"); - qemu_fdt_setprop_string(vms->fdt, "/intc/its", "compatible", + nodename = g_strdup_printf("/intc/its@%" PRIx64, + vms->memmap[VIRT_GIC_ITS].base); + qemu_fdt_add_subnode(vms->fdt, nodename); + qemu_fdt_setprop_string(vms->fdt, nodename, "compatible", "arm,gic-v3-its"); - qemu_fdt_setprop(vms->fdt, "/intc/its", "msi-controller", NULL, 0); - qemu_fdt_setprop_sized_cells(vms->fdt, "/intc/its", "reg", + qemu_fdt_setprop(vms->fdt, nodename, "msi-controller", NULL, 0); + qemu_fdt_setprop_sized_cells(vms->fdt, nodename, "reg", 2, vms->memmap[VIRT_GIC_ITS].base, 2, vms->memmap[VIRT_GIC_ITS].size); - qemu_fdt_setprop_cell(vms->fdt, "/intc/its", "phandle", vms->msi_phandle); + qemu_fdt_setprop_cell(vms->fdt, nodename, "phandle", vms->msi_phandle); + g_free(nodename); } static void fdt_add_v2m_gic_node(VirtMachineState *vms) { + char *nodename; + + nodename = g_strdup_printf("/intc/v2m@%" PRIx64, + vms->memmap[VIRT_GIC_V2M].base); vms->msi_phandle = qemu_fdt_alloc_phandle(vms->fdt); - qemu_fdt_add_subnode(vms->fdt, "/intc/v2m"); - qemu_fdt_setprop_string(vms->fdt, "/intc/v2m", "compatible", + qemu_fdt_add_subnode(vms->fdt, nodename); + qemu_fdt_setprop_string(vms->fdt, nodename, "compatible", "arm,gic-v2m-frame"); - qemu_fdt_setprop(vms->fdt, "/intc/v2m", "msi-controller", NULL, 0); - qemu_fdt_setprop_sized_cells(vms->fdt, "/intc/v2m", "reg", + qemu_fdt_setprop(vms->fdt, nodename, "msi-controller", NULL, 0); + qemu_fdt_setprop_sized_cells(vms->fdt, nodename, "reg", 2, vms->memmap[VIRT_GIC_V2M].base, 2, vms->memmap[VIRT_GIC_V2M].size); - qemu_fdt_setprop_cell(vms->fdt, "/intc/v2m", "phandle", vms->msi_phandle); + qemu_fdt_setprop_cell(vms->fdt, nodename, "phandle", vms->msi_phandle); + g_free(nodename); } static void fdt_add_gic_node(VirtMachineState *vms) { + char *nodename; + vms->gic_phandle = qemu_fdt_alloc_phandle(vms->fdt); qemu_fdt_setprop_cell(vms->fdt, "/", "interrupt-parent", vms->gic_phandle); - qemu_fdt_add_subnode(vms->fdt, "/intc"); - qemu_fdt_setprop_cell(vms->fdt, "/intc", "#interrupt-cells", 3); - qemu_fdt_setprop(vms->fdt, "/intc", "interrupt-controller", NULL, 0); - qemu_fdt_setprop_cell(vms->fdt, "/intc", "#address-cells", 0x2); - qemu_fdt_setprop_cell(vms->fdt, "/intc", "#size-cells", 0x2); - qemu_fdt_setprop(vms->fdt, "/intc", "ranges", NULL, 0); + nodename = g_strdup_printf("/intc@%" PRIx64, + vms->memmap[VIRT_GIC_DIST].base); + qemu_fdt_add_subnode(vms->fdt, nodename); + qemu_fdt_setprop_cell(vms->fdt, nodename, "#interrupt-cells", 3); + qemu_fdt_setprop(vms->fdt, nodename, "interrupt-controller", NULL, 0); + qemu_fdt_setprop_cell(vms->fdt, nodename, "#address-cells", 0x2); + qemu_fdt_setprop_cell(vms->fdt, nodename, "#size-cells", 0x2); + qemu_fdt_setprop(vms->fdt, nodename, "ranges", NULL, 0); if (vms->gic_version == 3) { int nb_redist_regions = virt_gicv3_redist_region_count(vms); - qemu_fdt_setprop_string(vms->fdt, "/intc", "compatible", + qemu_fdt_setprop_string(vms->fdt, nodename, "compatible", "arm,gic-v3"); - qemu_fdt_setprop_cell(vms->fdt, "/intc", + qemu_fdt_setprop_cell(vms->fdt, nodename, "#redistributor-regions", nb_redist_regions); if (nb_redist_regions == 1) { - qemu_fdt_setprop_sized_cells(vms->fdt, "/intc", "reg", + qemu_fdt_setprop_sized_cells(vms->fdt, nodename, "reg", 2, vms->memmap[VIRT_GIC_DIST].base, 2, vms->memmap[VIRT_GIC_DIST].size, 2, vms->memmap[VIRT_GIC_REDIST].base, 2, vms->memmap[VIRT_GIC_REDIST].size); } else { - qemu_fdt_setprop_sized_cells(vms->fdt, "/intc", "reg", + qemu_fdt_setprop_sized_cells(vms->fdt, nodename, "reg", 2, vms->memmap[VIRT_GIC_DIST].base, 2, vms->memmap[VIRT_GIC_DIST].size, 2, vms->memmap[VIRT_GIC_REDIST].base, @@ -XXX,XX +XXX,XX @@ static void fdt_add_gic_node(VirtMachineState *vms) } if (vms->virt) { - qemu_fdt_setprop_cells(vms->fdt, "/intc", "interrupts", + qemu_fdt_setprop_cells(vms->fdt, nodename, "interrupts", GIC_FDT_IRQ_TYPE_PPI, ARCH_GICV3_MAINT_IRQ, GIC_FDT_IRQ_FLAGS_LEVEL_HI); } } else { /* 'cortex-a15-gic' means 'GIC v2' */ - qemu_fdt_setprop_string(vms->fdt, "/intc", "compatible", + qemu_fdt_setprop_string(vms->fdt, nodename, "compatible", "arm,cortex-a15-gic"); - qemu_fdt_setprop_sized_cells(vms->fdt, "/intc", "reg", + qemu_fdt_setprop_sized_cells(vms->fdt, nodename, "reg", 2, vms->memmap[VIRT_GIC_DIST].base, 2, vms->memmap[VIRT_GIC_DIST].size, 2, vms->memmap[VIRT_GIC_CPU].base, 2, vms->memmap[VIRT_GIC_CPU].size); } - qemu_fdt_setprop_cell(vms->fdt, "/intc", "phandle", vms->gic_phandle); + qemu_fdt_setprop_cell(vms->fdt, nodename, "phandle", vms->gic_phandle); + g_free(nodename); } static void fdt_add_pmu_nodes(const VirtMachineState *vms) -- 2.17.1 From: Eric Auger <eric.auger@redhat.com> When running dtc on the guest /proc/device-tree we get the following warning: Warning (unit_address_vs_reg): Node /memory has a reg or ranges property, but no unit name". Let's fix that by adding the unit address to the node name. We also don't create the /memory node anymore in create_fdt(). We directly create it in load_dtb. /chosen still needs to be created in create_fdt as the uart needs it. In case the user provided his own dtb, we nop all memory nodes found in root and create new one(s). Signed-off-by: Eric Auger <eric.auger@redhat.com> Message-id: 1530044492-24921-4-git-send-email-eric.auger@redhat.com Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- hw/arm/boot.c | 41 +++++++++++++++++++++++------------------ hw/arm/virt.c | 7 +------ 2 files changed, 24 insertions(+), 24 deletions(-) diff --git a/hw/arm/boot.c b/hw/arm/boot.c index XXXXXXX..XXXXXXX 100644 --- a/hw/arm/boot.c +++ b/hw/arm/boot.c @@ -XXX,XX +XXX,XX @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, hwaddr addr_limit, AddressSpace *as) { void *fdt = NULL; - int size, rc; + int size, rc, n = 0; uint32_t acells, scells; char *nodename; unsigned int i; hwaddr mem_base, mem_len; + char **node_path; + Error *err = NULL; if (binfo->dtb_filename) { char *filename; @@ -XXX,XX +XXX,XX @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, goto fail; } + /* nop all root nodes matching /memory or /memory@unit-address */ + node_path = qemu_fdt_node_unit_path(fdt, "memory", &err); + if (err) { + error_report_err(err); + goto fail; + } + while (node_path[n]) { + if (g_str_has_prefix(node_path[n], "/memory")) { + qemu_fdt_nop_node(fdt, node_path[n]); + } + n++; + } + g_strfreev(node_path); + if (nb_numa_nodes > 0) { - /* - * Turn the /memory node created before into a NOP node, then create - * /memory@addr nodes for all numa nodes respectively. - */ - qemu_fdt_nop_node(fdt, "/memory"); mem_base = binfo->loader_start; for (i = 0; i < nb_numa_nodes; i++) { mem_len = numa_info[i].node_mem; @@ -XXX,XX +XXX,XX @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, g_free(nodename); } } else { - Error *err = NULL; + nodename = g_strdup_printf("/memory@%" PRIx64, binfo->loader_start); + qemu_fdt_add_subnode(fdt, nodename); + qemu_fdt_setprop_string(fdt, nodename, "device_type", "memory"); - rc = fdt_path_offset(fdt, "/memory"); - if (rc < 0) { - qemu_fdt_add_subnode(fdt, "/memory"); - } - - if (!qemu_fdt_getprop(fdt, "/memory", "device_type", NULL, &err)) { - qemu_fdt_setprop_string(fdt, "/memory", "device_type", "memory"); - } - - rc = qemu_fdt_setprop_sized_cells(fdt, "/memory", "reg", + rc = qemu_fdt_setprop_sized_cells(fdt, nodename, "reg", acells, binfo->loader_start, scells, binfo->ram_size); if (rc < 0) { - fprintf(stderr, "couldn't set /memory/reg\n"); + fprintf(stderr, "couldn't set %s reg\n", nodename); goto fail; } + g_free(nodename); } rc = fdt_path_offset(fdt, "/chosen"); diff --git a/hw/arm/virt.c b/hw/arm/virt.c index XXXXXXX..XXXXXXX 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -XXX,XX +XXX,XX @@ static void create_fdt(VirtMachineState *vms) qemu_fdt_setprop_cell(fdt, "/", "#address-cells", 0x2); qemu_fdt_setprop_cell(fdt, "/", "#size-cells", 0x2); - /* - * /chosen and /memory nodes must exist for load_dtb - * to fill in necessary properties later - */ + /* /chosen must exist for load_dtb to fill in necessary properties later */ qemu_fdt_add_subnode(fdt, "/chosen"); - qemu_fdt_add_subnode(fdt, "/memory"); - qemu_fdt_setprop_string(fdt, "/memory", "device_type", "memory"); /* Clock node, for the benefit of the UART. The kernel device tree * binding documentation claims the PL011 node clock properties are -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-2-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 35 +++++++++ target/arm/sve_helper.c | 153 +++++++++++++++++++++++++++++++++++++ target/arm/translate-sve.c | 121 +++++++++++++++++++++++++++++ target/arm/sve.decode | 34 +++++++++ 4 files changed, 343 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_5(gvec_rsqrts_s, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_5(gvec_rsqrts_d, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_4(sve_ld1bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld2bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld3bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld4bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ld1hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld2hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld3hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld4hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ld1ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld2ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld3ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld4ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ld1dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld2dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld3dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld4dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ld1bhu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1bsu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1bdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1bhs_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1bss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1bds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ld1hsu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1hdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1hss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1hds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ld1sdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1sds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ uint32_t HELPER(sve_while)(void *vd, uint32_t count, uint32_t pred_desc) return predtest_ones(d, oprsz, esz_mask); } + +/* + * Load contiguous data, protected by a governing predicate. + */ +#define DO_LD1(NAME, FN, TYPEE, TYPEM, H) \ +static void do_##NAME(CPUARMState *env, void *vd, void *vg, \ + target_ulong addr, intptr_t oprsz, \ + uintptr_t ra) \ +{ \ + intptr_t i = 0; \ + do { \ + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ + do { \ + TYPEM m = 0; \ + if (pg & 1) { \ + m = FN(env, addr, ra); \ + } \ + *(TYPEE *)(vd + H(i)) = m; \ + i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ + addr += sizeof(TYPEM); \ + } while (i & 15); \ + } while (i < oprsz); \ +} \ +void HELPER(NAME)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + do_##NAME(env, &env->vfp.zregs[simd_data(desc)], vg, \ + addr, simd_oprsz(desc), GETPC()); \ +} + +#define DO_LD2(NAME, FN, TYPEE, TYPEM, H) \ +void HELPER(NAME)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + intptr_t i, oprsz = simd_oprsz(desc); \ + intptr_t ra = GETPC(); \ + unsigned rd = simd_data(desc); \ + void *d1 = &env->vfp.zregs[rd]; \ + void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ + for (i = 0; i < oprsz; ) { \ + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ + do { \ + TYPEM m1 = 0, m2 = 0; \ + if (pg & 1) { \ + m1 = FN(env, addr, ra); \ + m2 = FN(env, addr + sizeof(TYPEM), ra); \ + } \ + *(TYPEE *)(d1 + H(i)) = m1; \ + *(TYPEE *)(d2 + H(i)) = m2; \ + i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ + addr += 2 * sizeof(TYPEM); \ + } while (i & 15); \ + } \ +} + +#define DO_LD3(NAME, FN, TYPEE, TYPEM, H) \ +void HELPER(NAME)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + intptr_t i, oprsz = simd_oprsz(desc); \ + intptr_t ra = GETPC(); \ + unsigned rd = simd_data(desc); \ + void *d1 = &env->vfp.zregs[rd]; \ + void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ + void *d3 = &env->vfp.zregs[(rd + 2) & 31]; \ + for (i = 0; i < oprsz; ) { \ + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ + do { \ + TYPEM m1 = 0, m2 = 0, m3 = 0; \ + if (pg & 1) { \ + m1 = FN(env, addr, ra); \ + m2 = FN(env, addr + sizeof(TYPEM), ra); \ + m3 = FN(env, addr + 2 * sizeof(TYPEM), ra); \ + } \ + *(TYPEE *)(d1 + H(i)) = m1; \ + *(TYPEE *)(d2 + H(i)) = m2; \ + *(TYPEE *)(d3 + H(i)) = m3; \ + i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ + addr += 3 * sizeof(TYPEM); \ + } while (i & 15); \ + } \ +} + +#define DO_LD4(NAME, FN, TYPEE, TYPEM, H) \ +void HELPER(NAME)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + intptr_t i, oprsz = simd_oprsz(desc); \ + intptr_t ra = GETPC(); \ + unsigned rd = simd_data(desc); \ + void *d1 = &env->vfp.zregs[rd]; \ + void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ + void *d3 = &env->vfp.zregs[(rd + 2) & 31]; \ + void *d4 = &env->vfp.zregs[(rd + 3) & 31]; \ + for (i = 0; i < oprsz; ) { \ + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ + do { \ + TYPEM m1 = 0, m2 = 0, m3 = 0, m4 = 0; \ + if (pg & 1) { \ + m1 = FN(env, addr, ra); \ + m2 = FN(env, addr + sizeof(TYPEM), ra); \ + m3 = FN(env, addr + 2 * sizeof(TYPEM), ra); \ + m4 = FN(env, addr + 3 * sizeof(TYPEM), ra); \ + } \ + *(TYPEE *)(d1 + H(i)) = m1; \ + *(TYPEE *)(d2 + H(i)) = m2; \ + *(TYPEE *)(d3 + H(i)) = m3; \ + *(TYPEE *)(d4 + H(i)) = m4; \ + i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ + addr += 4 * sizeof(TYPEM); \ + } while (i & 15); \ + } \ +} + +DO_LD1(sve_ld1bhu_r, cpu_ldub_data_ra, uint16_t, uint8_t, H1_2) +DO_LD1(sve_ld1bhs_r, cpu_ldsb_data_ra, uint16_t, int8_t, H1_2) +DO_LD1(sve_ld1bsu_r, cpu_ldub_data_ra, uint32_t, uint8_t, H1_4) +DO_LD1(sve_ld1bss_r, cpu_ldsb_data_ra, uint32_t, int8_t, H1_4) +DO_LD1(sve_ld1bdu_r, cpu_ldub_data_ra, uint64_t, uint8_t, ) +DO_LD1(sve_ld1bds_r, cpu_ldsb_data_ra, uint64_t, int8_t, ) + +DO_LD1(sve_ld1hsu_r, cpu_lduw_data_ra, uint32_t, uint16_t, H1_4) +DO_LD1(sve_ld1hss_r, cpu_ldsw_data_ra, uint32_t, int8_t, H1_4) +DO_LD1(sve_ld1hdu_r, cpu_lduw_data_ra, uint64_t, uint16_t, ) +DO_LD1(sve_ld1hds_r, cpu_ldsw_data_ra, uint64_t, int16_t, ) + +DO_LD1(sve_ld1sdu_r, cpu_ldl_data_ra, uint64_t, uint32_t, ) +DO_LD1(sve_ld1sds_r, cpu_ldl_data_ra, uint64_t, int32_t, ) + +DO_LD1(sve_ld1bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) +DO_LD2(sve_ld2bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) +DO_LD3(sve_ld3bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) +DO_LD4(sve_ld4bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) + +DO_LD1(sve_ld1hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) +DO_LD2(sve_ld2hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) +DO_LD3(sve_ld3hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) +DO_LD4(sve_ld4hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) + +DO_LD1(sve_ld1ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) +DO_LD2(sve_ld2ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) +DO_LD3(sve_ld3ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) +DO_LD4(sve_ld4ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) + +DO_LD1(sve_ld1dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) +DO_LD2(sve_ld2dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) +DO_LD3(sve_ld3dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) +DO_LD4(sve_ld4dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) + +#undef DO_LD1 +#undef DO_LD2 +#undef DO_LD3 +#undef DO_LD4 diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ typedef void gen_helper_gvec_flags_3(TCGv_i32, TCGv_ptr, TCGv_ptr, typedef void gen_helper_gvec_flags_4(TCGv_i32, TCGv_ptr, TCGv_ptr, TCGv_ptr, TCGv_ptr, TCGv_i32); +typedef void gen_helper_gvec_mem(TCGv_env, TCGv_ptr, TCGv_i64, TCGv_i32); + /* * Helpers for extracting complex instruction fields. */ @@ -XXX,XX +XXX,XX @@ static inline int expand_imm_sh8u(int x) return (uint8_t)x << (x & 0x100 ? 8 : 0); } +/* Convert a 2-bit memory size (msz) to a 4-bit data type (dtype) + * with unsigned data. C.f. SVE Memory Contiguous Load Group. + */ +static inline int msz_dtype(int msz) +{ + static const uint8_t dtype[4] = { 0, 5, 10, 15 }; + return dtype[msz]; +} + /* * Include the generated decoder. */ @@ -XXX,XX +XXX,XX @@ static bool trans_LDR_pri(DisasContext *s, arg_rri *a, uint32_t insn) } return true; } + +/* + *** SVE Memory - Contiguous Load Group + */ + +/* The memory mode of the dtype. */ +static const TCGMemOp dtype_mop[16] = { + MO_UB, MO_UB, MO_UB, MO_UB, + MO_SL, MO_UW, MO_UW, MO_UW, + MO_SW, MO_SW, MO_UL, MO_UL, + MO_SB, MO_SB, MO_SB, MO_Q +}; + +#define dtype_msz(x) (dtype_mop[x] & MO_SIZE) + +/* The vector element size of dtype. */ +static const uint8_t dtype_esz[16] = { + 0, 1, 2, 3, + 3, 1, 2, 3, + 3, 2, 2, 3, + 3, 2, 1, 3 +}; + +static void do_mem_zpa(DisasContext *s, int zt, int pg, TCGv_i64 addr, + gen_helper_gvec_mem *fn) +{ + unsigned vsz = vec_full_reg_size(s); + TCGv_ptr t_pg; + TCGv_i32 desc; + + /* For e.g. LD4, there are not enough arguments to pass all 4 + * registers as pointers, so encode the regno into the data field. + * For consistency, do this even for LD1. + */ + desc = tcg_const_i32(simd_desc(vsz, vsz, zt)); + t_pg = tcg_temp_new_ptr(); + + tcg_gen_addi_ptr(t_pg, cpu_env, pred_full_reg_offset(s, pg)); + fn(cpu_env, t_pg, addr, desc); + + tcg_temp_free_ptr(t_pg); + tcg_temp_free_i32(desc); +} + +static void do_ld_zpa(DisasContext *s, int zt, int pg, + TCGv_i64 addr, int dtype, int nreg) +{ + static gen_helper_gvec_mem * const fns[16][4] = { + { gen_helper_sve_ld1bb_r, gen_helper_sve_ld2bb_r, + gen_helper_sve_ld3bb_r, gen_helper_sve_ld4bb_r }, + { gen_helper_sve_ld1bhu_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1bsu_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1bdu_r, NULL, NULL, NULL }, + + { gen_helper_sve_ld1sds_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1hh_r, gen_helper_sve_ld2hh_r, + gen_helper_sve_ld3hh_r, gen_helper_sve_ld4hh_r }, + { gen_helper_sve_ld1hsu_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1hdu_r, NULL, NULL, NULL }, + + { gen_helper_sve_ld1hds_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1hss_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1ss_r, gen_helper_sve_ld2ss_r, + gen_helper_sve_ld3ss_r, gen_helper_sve_ld4ss_r }, + { gen_helper_sve_ld1sdu_r, NULL, NULL, NULL }, + + { gen_helper_sve_ld1bds_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1bss_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1bhs_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1dd_r, gen_helper_sve_ld2dd_r, + gen_helper_sve_ld3dd_r, gen_helper_sve_ld4dd_r }, + }; + gen_helper_gvec_mem *fn = fns[dtype][nreg]; + + /* While there are holes in the table, they are not + * accessible via the instruction encoding. + */ + assert(fn != NULL); + do_mem_zpa(s, zt, pg, addr, fn); +} + +static bool trans_LD_zprr(DisasContext *s, arg_rprr_load *a, uint32_t insn) +{ + if (a->rm == 31) { + return false; + } + if (sve_access_check(s)) { + TCGv_i64 addr = new_tmp_a64(s); + tcg_gen_muli_i64(addr, cpu_reg(s, a->rm), + (a->nreg + 1) << dtype_msz(a->dtype)); + tcg_gen_add_i64(addr, addr, cpu_reg_sp(s, a->rn)); + do_ld_zpa(s, a->rd, a->pg, addr, a->dtype, a->nreg); + } + return true; +} + +static bool trans_LD_zpri(DisasContext *s, arg_rpri_load *a, uint32_t insn) +{ + if (sve_access_check(s)) { + int vsz = vec_full_reg_size(s); + int elements = vsz >> dtype_esz[a->dtype]; + TCGv_i64 addr = new_tmp_a64(s); + + tcg_gen_addi_i64(addr, cpu_reg_sp(s, a->rn), + (a->imm * elements * (a->nreg + 1)) + << dtype_msz(a->dtype)); + do_ld_zpa(s, a->rd, a->pg, addr, a->dtype, a->nreg); + } + return true; +} diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ # Unsigned 8-bit immediate, optionally shifted left by 8. %sh8_i8u 5:9 !function=expand_imm_sh8u +# Unsigned load of msz into esz=2, represented as a dtype. +%msz_dtype 23:2 !function=msz_dtype + # Either a copy of rd (at bit 0), or a different source # as propagated via the MOVPRFX instruction. %reg_movprfx 0:5 @@ -XXX,XX +XXX,XX @@ &incdec2_cnt rd rn pat esz imm d u &incdec_pred rd pg esz d u &incdec2_pred rd rn pg esz d u +&rprr_load rd pg rn rm dtype nreg +&rpri_load rd pg rn imm dtype nreg ########################################################################### # Named instruction formats. These are generally used to @@ -XXX,XX +XXX,XX @@ @incdec2_pred ........ esz:2 .... .. ..... .. pg:4 rd:5 \ &incdec2_pred rn=%reg_movprfx +# Loads; user must fill in NREG. +@rprr_load_dt ....... dtype:4 rm:5 ... pg:3 rn:5 rd:5 &rprr_load +@rpri_load_dt ....... dtype:4 . imm:s4 ... pg:3 rn:5 rd:5 &rpri_load + +@rprr_load_msz ....... .... rm:5 ... pg:3 rn:5 rd:5 \ + &rprr_load dtype=%msz_dtype +@rpri_load_msz ....... .... . imm:s4 ... pg:3 rn:5 rd:5 \ + &rpri_load dtype=%msz_dtype + ########################################################################### # Instruction patterns. Grouped according to the SVE encodingindex.xhtml. @@ -XXX,XX +XXX,XX @@ LDR_pri 10000101 10 ...... 000 ... ..... 0 .... @pd_rn_i9 # SVE load vector register LDR_zri 10000101 10 ...... 010 ... ..... ..... @rd_rn_i9 + +### SVE Memory Contiguous Load Group + +# SVE contiguous load (scalar plus scalar) +LD_zprr 1010010 .... ..... 010 ... ..... ..... @rprr_load_dt nreg=0 + +# SVE contiguous load (scalar plus immediate) +LD_zpri 1010010 .... 0.... 101 ... ..... ..... @rpri_load_dt nreg=0 + +# SVE contiguous non-temporal load (scalar plus scalar) +# LDNT1B, LDNT1H, LDNT1W, LDNT1D +# SVE load multiple structures (scalar plus scalar) +# LD2B, LD2H, LD2W, LD2D; etc. +LD_zprr 1010010 .. nreg:2 ..... 110 ... ..... ..... @rprr_load_msz + +# SVE contiguous non-temporal load (scalar plus immediate) +# LDNT1B, LDNT1H, LDNT1W, LDNT1D +# SVE load multiple structures (scalar plus immediate) +# LD2B, LD2H, LD2W, LD2D; etc. +LD_zpri 1010010 .. nreg:2 0.... 111 ... ..... ..... @rpri_load_msz -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Tested-by: Alex Bennée <alex.bennee@linaro.org> Message-id: 20180627043328.11531-3-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 40 ++++++++++ target/arm/sve_helper.c | 157 +++++++++++++++++++++++++++++++++++++ target/arm/translate-sve.c | 69 ++++++++++++++++ target/arm/sve.decode | 6 ++ 4 files changed, 272 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_4(sve_ld1hds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ld1sdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ld1sds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ldff1bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1bhu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1bsu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1bdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1bhs_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1bss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1bds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ldff1hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1hsu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1hdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1hss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1hds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ldff1ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1sdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1sds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ldff1dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ldnf1bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1bhu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1bsu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1bdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1bhs_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1bss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1bds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ldnf1hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1hsu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1hdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1hss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1hds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ldnf1ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1sdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1sds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ldnf1dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ DO_LD4(sve_ld4dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) #undef DO_LD2 #undef DO_LD3 #undef DO_LD4 + +/* + * Load contiguous data, first-fault and no-fault. + */ + +#ifdef CONFIG_USER_ONLY + +/* Fault on byte I. All bits in FFR from I are cleared. The vector + * result from I is CONSTRAINED UNPREDICTABLE; we choose the MERGE + * option, which leaves subsequent data unchanged. + */ +static void record_fault(CPUARMState *env, uintptr_t i, uintptr_t oprsz) +{ + uint64_t *ffr = env->vfp.pregs[FFR_PRED_NUM].p; + + if (i & 63) { + ffr[i / 64] &= MAKE_64BIT_MASK(0, i & 63); + i = ROUND_UP(i, 64); + } + for (; i < oprsz; i += 64) { + ffr[i / 64] = 0; + } +} + +/* Hold the mmap lock during the operation so that there is no race + * between page_check_range and the load operation. We expect the + * usual case to have no faults at all, so we check the whole range + * first and if successful defer to the normal load operation. + * + * TODO: Change mmap_lock to a rwlock so that multiple readers + * can run simultaneously. This will probably help other uses + * within QEMU as well. + */ +#define DO_LDFF1(PART, FN, TYPEE, TYPEM, H) \ +static void do_sve_ldff1##PART(CPUARMState *env, void *vd, void *vg, \ + target_ulong addr, intptr_t oprsz, \ + bool first, uintptr_t ra) \ +{ \ + intptr_t i = 0; \ + do { \ + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ + do { \ + TYPEM m = 0; \ + if (pg & 1) { \ + if (!first && \ + unlikely(page_check_range(addr, sizeof(TYPEM), \ + PAGE_READ))) { \ + record_fault(env, i, oprsz); \ + return; \ + } \ + m = FN(env, addr, ra); \ + first = false; \ + } \ + *(TYPEE *)(vd + H(i)) = m; \ + i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ + addr += sizeof(TYPEM); \ + } while (i & 15); \ + } while (i < oprsz); \ +} \ +void HELPER(sve_ldff1##PART)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + intptr_t oprsz = simd_oprsz(desc); \ + unsigned rd = simd_data(desc); \ + void *vd = &env->vfp.zregs[rd]; \ + mmap_lock(); \ + if (likely(page_check_range(addr, oprsz, PAGE_READ) == 0)) { \ + do_sve_ld1##PART(env, vd, vg, addr, oprsz, GETPC()); \ + } else { \ + do_sve_ldff1##PART(env, vd, vg, addr, oprsz, true, GETPC()); \ + } \ + mmap_unlock(); \ +} + +/* No-fault loads are like first-fault loads without the + * first faulting special case. + */ +#define DO_LDNF1(PART) \ +void HELPER(sve_ldnf1##PART)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + intptr_t oprsz = simd_oprsz(desc); \ + unsigned rd = simd_data(desc); \ + void *vd = &env->vfp.zregs[rd]; \ + mmap_lock(); \ + if (likely(page_check_range(addr, oprsz, PAGE_READ) == 0)) { \ + do_sve_ld1##PART(env, vd, vg, addr, oprsz, GETPC()); \ + } else { \ + do_sve_ldff1##PART(env, vd, vg, addr, oprsz, false, GETPC()); \ + } \ + mmap_unlock(); \ +} + +#else + +/* TODO: System mode is not yet supported. + * This would probably use tlb_vaddr_to_host. + */ +#define DO_LDFF1(PART, FN, TYPEE, TYPEM, H) \ +void HELPER(sve_ldff1##PART)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + g_assert_not_reached(); \ +} + +#define DO_LDNF1(PART) \ +void HELPER(sve_ldnf1##PART)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + g_assert_not_reached(); \ +} + +#endif + +DO_LDFF1(bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) +DO_LDFF1(bhu_r, cpu_ldub_data_ra, uint16_t, uint8_t, H1_2) +DO_LDFF1(bhs_r, cpu_ldsb_data_ra, uint16_t, int8_t, H1_2) +DO_LDFF1(bsu_r, cpu_ldub_data_ra, uint32_t, uint8_t, H1_4) +DO_LDFF1(bss_r, cpu_ldsb_data_ra, uint32_t, int8_t, H1_4) +DO_LDFF1(bdu_r, cpu_ldub_data_ra, uint64_t, uint8_t, ) +DO_LDFF1(bds_r, cpu_ldsb_data_ra, uint64_t, int8_t, ) + +DO_LDFF1(hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) +DO_LDFF1(hsu_r, cpu_lduw_data_ra, uint32_t, uint16_t, H1_4) +DO_LDFF1(hss_r, cpu_ldsw_data_ra, uint32_t, int8_t, H1_4) +DO_LDFF1(hdu_r, cpu_lduw_data_ra, uint64_t, uint16_t, ) +DO_LDFF1(hds_r, cpu_ldsw_data_ra, uint64_t, int16_t, ) + +DO_LDFF1(ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) +DO_LDFF1(sdu_r, cpu_ldl_data_ra, uint64_t, uint32_t, ) +DO_LDFF1(sds_r, cpu_ldl_data_ra, uint64_t, int32_t, ) + +DO_LDFF1(dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) + +#undef DO_LDFF1 + +DO_LDNF1(bb_r) +DO_LDNF1(bhu_r) +DO_LDNF1(bhs_r) +DO_LDNF1(bsu_r) +DO_LDNF1(bss_r) +DO_LDNF1(bdu_r) +DO_LDNF1(bds_r) + +DO_LDNF1(hh_r) +DO_LDNF1(hsu_r) +DO_LDNF1(hss_r) +DO_LDNF1(hdu_r) +DO_LDNF1(hds_r) + +DO_LDNF1(ss_r) +DO_LDNF1(sdu_r) +DO_LDNF1(sds_r) + +DO_LDNF1(dd_r) + +#undef DO_LDNF1 diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ static bool trans_LD_zpri(DisasContext *s, arg_rpri_load *a, uint32_t insn) } return true; } + +static bool trans_LDFF1_zprr(DisasContext *s, arg_rprr_load *a, uint32_t insn) +{ + static gen_helper_gvec_mem * const fns[16] = { + gen_helper_sve_ldff1bb_r, + gen_helper_sve_ldff1bhu_r, + gen_helper_sve_ldff1bsu_r, + gen_helper_sve_ldff1bdu_r, + + gen_helper_sve_ldff1sds_r, + gen_helper_sve_ldff1hh_r, + gen_helper_sve_ldff1hsu_r, + gen_helper_sve_ldff1hdu_r, + + gen_helper_sve_ldff1hds_r, + gen_helper_sve_ldff1hss_r, + gen_helper_sve_ldff1ss_r, + gen_helper_sve_ldff1sdu_r, + + gen_helper_sve_ldff1bds_r, + gen_helper_sve_ldff1bss_r, + gen_helper_sve_ldff1bhs_r, + gen_helper_sve_ldff1dd_r, + }; + + if (sve_access_check(s)) { + TCGv_i64 addr = new_tmp_a64(s); + tcg_gen_shli_i64(addr, cpu_reg(s, a->rm), dtype_msz(a->dtype)); + tcg_gen_add_i64(addr, addr, cpu_reg_sp(s, a->rn)); + do_mem_zpa(s, a->rd, a->pg, addr, fns[a->dtype]); + } + return true; +} + +static bool trans_LDNF1_zpri(DisasContext *s, arg_rpri_load *a, uint32_t insn) +{ + static gen_helper_gvec_mem * const fns[16] = { + gen_helper_sve_ldnf1bb_r, + gen_helper_sve_ldnf1bhu_r, + gen_helper_sve_ldnf1bsu_r, + gen_helper_sve_ldnf1bdu_r, + + gen_helper_sve_ldnf1sds_r, + gen_helper_sve_ldnf1hh_r, + gen_helper_sve_ldnf1hsu_r, + gen_helper_sve_ldnf1hdu_r, + + gen_helper_sve_ldnf1hds_r, + gen_helper_sve_ldnf1hss_r, + gen_helper_sve_ldnf1ss_r, + gen_helper_sve_ldnf1sdu_r, + + gen_helper_sve_ldnf1bds_r, + gen_helper_sve_ldnf1bss_r, + gen_helper_sve_ldnf1bhs_r, + gen_helper_sve_ldnf1dd_r, + }; + + if (sve_access_check(s)) { + int vsz = vec_full_reg_size(s); + int elements = vsz >> dtype_esz[a->dtype]; + int off = (a->imm * elements) << dtype_msz(a->dtype); + TCGv_i64 addr = new_tmp_a64(s); + + tcg_gen_addi_i64(addr, cpu_reg_sp(s, a->rn), off); + do_mem_zpa(s, a->rd, a->pg, addr, fns[a->dtype]); + } + return true; +} diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ LDR_zri 10000101 10 ...... 010 ... ..... ..... @rd_rn_i9 # SVE contiguous load (scalar plus scalar) LD_zprr 1010010 .... ..... 010 ... ..... ..... @rprr_load_dt nreg=0 +# SVE contiguous first-fault load (scalar plus scalar) +LDFF1_zprr 1010010 .... ..... 011 ... ..... ..... @rprr_load_dt nreg=0 + # SVE contiguous load (scalar plus immediate) LD_zpri 1010010 .... 0.... 101 ... ..... ..... @rpri_load_dt nreg=0 +# SVE contiguous non-fault load (scalar plus immediate) +LDNF1_zpri 1010010 .... 1.... 101 ... ..... ..... @rpri_load_dt nreg=0 + # SVE contiguous non-temporal load (scalar plus scalar) # LDNT1B, LDNT1H, LDNT1W, LDNT1D # SVE load multiple structures (scalar plus scalar) -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-4-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 29 +++++ target/arm/sve_helper.c | 211 +++++++++++++++++++++++++++++++++++++ target/arm/translate-sve.c | 65 ++++++++++++ target/arm/sve.decode | 38 +++++++ 4 files changed, 343 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_4(sve_ldnf1sdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ldnf1sds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ldnf1dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_st1bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st2bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st3bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st4bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_st1hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st2hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st3hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st4hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_st1ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st2ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st3ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st4ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_st1dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st2dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st3dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st4dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_st1bh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st1bs_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st1bd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_st1hs_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st1hd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_st1sd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ DO_LDNF1(sds_r) DO_LDNF1(dd_r) #undef DO_LDNF1 + +/* + * Store contiguous data, protected by a governing predicate. + */ +#define DO_ST1(NAME, FN, TYPEE, TYPEM, H) \ +void HELPER(NAME)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + intptr_t i, oprsz = simd_oprsz(desc); \ + intptr_t ra = GETPC(); \ + unsigned rd = simd_data(desc); \ + void *vd = &env->vfp.zregs[rd]; \ + for (i = 0; i < oprsz; ) { \ + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ + do { \ + if (pg & 1) { \ + TYPEM m = *(TYPEE *)(vd + H(i)); \ + FN(env, addr, m, ra); \ + } \ + i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ + addr += sizeof(TYPEM); \ + } while (i & 15); \ + } \ +} + +#define DO_ST1_D(NAME, FN, TYPEM) \ +void HELPER(NAME)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + intptr_t i, oprsz = simd_oprsz(desc) / 8; \ + intptr_t ra = GETPC(); \ + unsigned rd = simd_data(desc); \ + uint64_t *d = &env->vfp.zregs[rd].d[0]; \ + uint8_t *pg = vg; \ + for (i = 0; i < oprsz; i += 1) { \ + if (pg[H1(i)] & 1) { \ + FN(env, addr, d[i], ra); \ + } \ + addr += sizeof(TYPEM); \ + } \ +} + +#define DO_ST2(NAME, FN, TYPEE, TYPEM, H) \ +void HELPER(NAME)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + intptr_t i, oprsz = simd_oprsz(desc); \ + intptr_t ra = GETPC(); \ + unsigned rd = simd_data(desc); \ + void *d1 = &env->vfp.zregs[rd]; \ + void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ + for (i = 0; i < oprsz; ) { \ + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ + do { \ + if (pg & 1) { \ + TYPEM m1 = *(TYPEE *)(d1 + H(i)); \ + TYPEM m2 = *(TYPEE *)(d2 + H(i)); \ + FN(env, addr, m1, ra); \ + FN(env, addr + sizeof(TYPEM), m2, ra); \ + } \ + i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ + addr += 2 * sizeof(TYPEM); \ + } while (i & 15); \ + } \ +} + +#define DO_ST3(NAME, FN, TYPEE, TYPEM, H) \ +void HELPER(NAME)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + intptr_t i, oprsz = simd_oprsz(desc); \ + intptr_t ra = GETPC(); \ + unsigned rd = simd_data(desc); \ + void *d1 = &env->vfp.zregs[rd]; \ + void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ + void *d3 = &env->vfp.zregs[(rd + 2) & 31]; \ + for (i = 0; i < oprsz; ) { \ + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ + do { \ + if (pg & 1) { \ + TYPEM m1 = *(TYPEE *)(d1 + H(i)); \ + TYPEM m2 = *(TYPEE *)(d2 + H(i)); \ + TYPEM m3 = *(TYPEE *)(d3 + H(i)); \ + FN(env, addr, m1, ra); \ + FN(env, addr + sizeof(TYPEM), m2, ra); \ + FN(env, addr + 2 * sizeof(TYPEM), m3, ra); \ + } \ + i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ + addr += 3 * sizeof(TYPEM); \ + } while (i & 15); \ + } \ +} + +#define DO_ST4(NAME, FN, TYPEE, TYPEM, H) \ +void HELPER(NAME)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + intptr_t i, oprsz = simd_oprsz(desc); \ + intptr_t ra = GETPC(); \ + unsigned rd = simd_data(desc); \ + void *d1 = &env->vfp.zregs[rd]; \ + void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ + void *d3 = &env->vfp.zregs[(rd + 2) & 31]; \ + void *d4 = &env->vfp.zregs[(rd + 3) & 31]; \ + for (i = 0; i < oprsz; ) { \ + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ + do { \ + if (pg & 1) { \ + TYPEM m1 = *(TYPEE *)(d1 + H(i)); \ + TYPEM m2 = *(TYPEE *)(d2 + H(i)); \ + TYPEM m3 = *(TYPEE *)(d3 + H(i)); \ + TYPEM m4 = *(TYPEE *)(d4 + H(i)); \ + FN(env, addr, m1, ra); \ + FN(env, addr + sizeof(TYPEM), m2, ra); \ + FN(env, addr + 2 * sizeof(TYPEM), m3, ra); \ + FN(env, addr + 3 * sizeof(TYPEM), m4, ra); \ + } \ + i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ + addr += 4 * sizeof(TYPEM); \ + } while (i & 15); \ + } \ +} + +DO_ST1(sve_st1bh_r, cpu_stb_data_ra, uint16_t, uint8_t, H1_2) +DO_ST1(sve_st1bs_r, cpu_stb_data_ra, uint32_t, uint8_t, H1_4) +DO_ST1_D(sve_st1bd_r, cpu_stb_data_ra, uint8_t) + +DO_ST1(sve_st1hs_r, cpu_stw_data_ra, uint32_t, uint16_t, H1_4) +DO_ST1_D(sve_st1hd_r, cpu_stw_data_ra, uint16_t) + +DO_ST1_D(sve_st1sd_r, cpu_stl_data_ra, uint32_t) + +DO_ST1(sve_st1bb_r, cpu_stb_data_ra, uint8_t, uint8_t, H1) +DO_ST2(sve_st2bb_r, cpu_stb_data_ra, uint8_t, uint8_t, H1) +DO_ST3(sve_st3bb_r, cpu_stb_data_ra, uint8_t, uint8_t, H1) +DO_ST4(sve_st4bb_r, cpu_stb_data_ra, uint8_t, uint8_t, H1) + +DO_ST1(sve_st1hh_r, cpu_stw_data_ra, uint16_t, uint16_t, H1_2) +DO_ST2(sve_st2hh_r, cpu_stw_data_ra, uint16_t, uint16_t, H1_2) +DO_ST3(sve_st3hh_r, cpu_stw_data_ra, uint16_t, uint16_t, H1_2) +DO_ST4(sve_st4hh_r, cpu_stw_data_ra, uint16_t, uint16_t, H1_2) + +DO_ST1(sve_st1ss_r, cpu_stl_data_ra, uint32_t, uint32_t, H1_4) +DO_ST2(sve_st2ss_r, cpu_stl_data_ra, uint32_t, uint32_t, H1_4) +DO_ST3(sve_st3ss_r, cpu_stl_data_ra, uint32_t, uint32_t, H1_4) +DO_ST4(sve_st4ss_r, cpu_stl_data_ra, uint32_t, uint32_t, H1_4) + +DO_ST1_D(sve_st1dd_r, cpu_stq_data_ra, uint64_t) + +void HELPER(sve_st2dd_r)(CPUARMState *env, void *vg, + target_ulong addr, uint32_t desc) +{ + intptr_t i, oprsz = simd_oprsz(desc) / 8; + intptr_t ra = GETPC(); + unsigned rd = simd_data(desc); + uint64_t *d1 = &env->vfp.zregs[rd].d[0]; + uint64_t *d2 = &env->vfp.zregs[(rd + 1) & 31].d[0]; + uint8_t *pg = vg; + + for (i = 0; i < oprsz; i += 1) { + if (pg[H1(i)] & 1) { + cpu_stq_data_ra(env, addr, d1[i], ra); + cpu_stq_data_ra(env, addr + 8, d2[i], ra); + } + addr += 2 * 8; + } +} + +void HELPER(sve_st3dd_r)(CPUARMState *env, void *vg, + target_ulong addr, uint32_t desc) +{ + intptr_t i, oprsz = simd_oprsz(desc) / 8; + intptr_t ra = GETPC(); + unsigned rd = simd_data(desc); + uint64_t *d1 = &env->vfp.zregs[rd].d[0]; + uint64_t *d2 = &env->vfp.zregs[(rd + 1) & 31].d[0]; + uint64_t *d3 = &env->vfp.zregs[(rd + 2) & 31].d[0]; + uint8_t *pg = vg; + + for (i = 0; i < oprsz; i += 1) { + if (pg[H1(i)] & 1) { + cpu_stq_data_ra(env, addr, d1[i], ra); + cpu_stq_data_ra(env, addr + 8, d2[i], ra); + cpu_stq_data_ra(env, addr + 16, d3[i], ra); + } + addr += 3 * 8; + } +} + +void HELPER(sve_st4dd_r)(CPUARMState *env, void *vg, + target_ulong addr, uint32_t desc) +{ + intptr_t i, oprsz = simd_oprsz(desc) / 8; + intptr_t ra = GETPC(); + unsigned rd = simd_data(desc); + uint64_t *d1 = &env->vfp.zregs[rd].d[0]; + uint64_t *d2 = &env->vfp.zregs[(rd + 1) & 31].d[0]; + uint64_t *d3 = &env->vfp.zregs[(rd + 2) & 31].d[0]; + uint64_t *d4 = &env->vfp.zregs[(rd + 3) & 31].d[0]; + uint8_t *pg = vg; + + for (i = 0; i < oprsz; i += 1) { + if (pg[H1(i)] & 1) { + cpu_stq_data_ra(env, addr, d1[i], ra); + cpu_stq_data_ra(env, addr + 8, d2[i], ra); + cpu_stq_data_ra(env, addr + 16, d3[i], ra); + cpu_stq_data_ra(env, addr + 24, d4[i], ra); + } + addr += 4 * 8; + } +} diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ static bool trans_LDNF1_zpri(DisasContext *s, arg_rpri_load *a, uint32_t insn) } return true; } + +static void do_st_zpa(DisasContext *s, int zt, int pg, TCGv_i64 addr, + int msz, int esz, int nreg) +{ + static gen_helper_gvec_mem * const fn_single[4][4] = { + { gen_helper_sve_st1bb_r, gen_helper_sve_st1bh_r, + gen_helper_sve_st1bs_r, gen_helper_sve_st1bd_r }, + { NULL, gen_helper_sve_st1hh_r, + gen_helper_sve_st1hs_r, gen_helper_sve_st1hd_r }, + { NULL, NULL, + gen_helper_sve_st1ss_r, gen_helper_sve_st1sd_r }, + { NULL, NULL, NULL, gen_helper_sve_st1dd_r }, + }; + static gen_helper_gvec_mem * const fn_multiple[3][4] = { + { gen_helper_sve_st2bb_r, gen_helper_sve_st2hh_r, + gen_helper_sve_st2ss_r, gen_helper_sve_st2dd_r }, + { gen_helper_sve_st3bb_r, gen_helper_sve_st3hh_r, + gen_helper_sve_st3ss_r, gen_helper_sve_st3dd_r }, + { gen_helper_sve_st4bb_r, gen_helper_sve_st4hh_r, + gen_helper_sve_st4ss_r, gen_helper_sve_st4dd_r }, + }; + gen_helper_gvec_mem *fn; + + if (nreg == 0) { + /* ST1 */ + fn = fn_single[msz][esz]; + } else { + /* ST2, ST3, ST4 -- msz == esz, enforced by encoding */ + assert(msz == esz); + fn = fn_multiple[nreg - 1][msz]; + } + assert(fn != NULL); + do_mem_zpa(s, zt, pg, addr, fn); +} + +static bool trans_ST_zprr(DisasContext *s, arg_rprr_store *a, uint32_t insn) +{ + if (a->rm == 31 || a->msz > a->esz) { + return false; + } + if (sve_access_check(s)) { + TCGv_i64 addr = new_tmp_a64(s); + tcg_gen_muli_i64(addr, cpu_reg(s, a->rm), (a->nreg + 1) << a->msz); + tcg_gen_add_i64(addr, addr, cpu_reg_sp(s, a->rn)); + do_st_zpa(s, a->rd, a->pg, addr, a->msz, a->esz, a->nreg); + } + return true; +} + +static bool trans_ST_zpri(DisasContext *s, arg_rpri_store *a, uint32_t insn) +{ + if (a->msz > a->esz) { + return false; + } + if (sve_access_check(s)) { + int vsz = vec_full_reg_size(s); + int elements = vsz >> a->esz; + TCGv_i64 addr = new_tmp_a64(s); + + tcg_gen_addi_i64(addr, cpu_reg_sp(s, a->rn), + (a->imm * elements * (a->nreg + 1)) << a->msz); + do_st_zpa(s, a->rd, a->pg, addr, a->msz, a->esz, a->nreg); + } + return true; +} diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ %imm7_22_16 22:2 16:5 %imm8_16_10 16:5 10:3 %imm9_16_10 16:s6 10:3 +%size_23 23:2 # A combination of tsz:imm3 -- extract esize. %tszimm_esz 22:2 5:5 !function=tszimm_esz @@ -XXX,XX +XXX,XX @@ &incdec2_pred rd rn pg esz d u &rprr_load rd pg rn rm dtype nreg &rpri_load rd pg rn imm dtype nreg +&rprr_store rd pg rn rm msz esz nreg +&rpri_store rd pg rn imm msz esz nreg ########################################################################### # Named instruction formats. These are generally used to @@ -XXX,XX +XXX,XX @@ @rpri_load_msz ....... .... . imm:s4 ... pg:3 rn:5 rd:5 \ &rpri_load dtype=%msz_dtype +# Stores; user must fill in ESZ, MSZ, NREG as needed. +@rprr_store ....... .. .. rm:5 ... pg:3 rn:5 rd:5 &rprr_store +@rpri_store_msz ....... msz:2 .. . imm:s4 ... pg:3 rn:5 rd:5 &rpri_store +@rprr_store_esz_n0 ....... .. esz:2 rm:5 ... pg:3 rn:5 rd:5 \ + &rprr_store nreg=0 + ########################################################################### # Instruction patterns. Grouped according to the SVE encodingindex.xhtml. @@ -XXX,XX +XXX,XX @@ LD_zprr 1010010 .. nreg:2 ..... 110 ... ..... ..... @rprr_load_msz # SVE load multiple structures (scalar plus immediate) # LD2B, LD2H, LD2W, LD2D; etc. LD_zpri 1010010 .. nreg:2 0.... 111 ... ..... ..... @rpri_load_msz + +### SVE Memory Store Group + +# SVE contiguous store (scalar plus immediate) +# ST1B, ST1H, ST1W, ST1D; require msz <= esz +ST_zpri 1110010 .. esz:2 0.... 111 ... ..... ..... \ + @rpri_store_msz nreg=0 + +# SVE contiguous store (scalar plus scalar) +# ST1B, ST1H, ST1W, ST1D; require msz <= esz +# Enumerate msz lest we conflict with STR_zri. +ST_zprr 1110010 00 .. ..... 010 ... ..... ..... \ + @rprr_store_esz_n0 msz=0 +ST_zprr 1110010 01 .. ..... 010 ... ..... ..... \ + @rprr_store_esz_n0 msz=1 +ST_zprr 1110010 10 .. ..... 010 ... ..... ..... \ + @rprr_store_esz_n0 msz=2 +ST_zprr 1110010 11 11 ..... 010 ... ..... ..... \ + @rprr_store msz=3 esz=3 nreg=0 + +# SVE contiguous non-temporal store (scalar plus immediate) (nreg == 0) +# SVE store multiple structures (scalar plus immediate) (nreg != 0) +ST_zpri 1110010 .. nreg:2 1.... 111 ... ..... ..... \ + @rpri_store_msz esz=%size_23 + +# SVE contiguous non-temporal store (scalar plus scalar) (nreg == 0) +# SVE store multiple structures (scalar plus scalar) (nreg != 0) +ST_zprr 1110010 msz:2 nreg:2 ..... 011 ... ..... ..... \ + @rprr_store esz=%size_23 -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-5-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/translate-sve.c | 52 ++++++++++++++++++++++++++++++++++++++ target/arm/sve.decode | 9 +++++++ 2 files changed, 61 insertions(+) diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ static bool trans_LDNF1_zpri(DisasContext *s, arg_rpri_load *a, uint32_t insn) return true; } +static void do_ldrq(DisasContext *s, int zt, int pg, TCGv_i64 addr, int msz) +{ + static gen_helper_gvec_mem * const fns[4] = { + gen_helper_sve_ld1bb_r, gen_helper_sve_ld1hh_r, + gen_helper_sve_ld1ss_r, gen_helper_sve_ld1dd_r, + }; + unsigned vsz = vec_full_reg_size(s); + TCGv_ptr t_pg; + TCGv_i32 desc; + + /* Load the first quadword using the normal predicated load helpers. */ + desc = tcg_const_i32(simd_desc(16, 16, zt)); + t_pg = tcg_temp_new_ptr(); + + tcg_gen_addi_ptr(t_pg, cpu_env, pred_full_reg_offset(s, pg)); + fns[msz](cpu_env, t_pg, addr, desc); + + tcg_temp_free_ptr(t_pg); + tcg_temp_free_i32(desc); + + /* Replicate that first quadword. */ + if (vsz > 16) { + unsigned dofs = vec_full_reg_offset(s, zt); + tcg_gen_gvec_dup_mem(4, dofs + 16, dofs, vsz - 16, vsz - 16); + } +} + +static bool trans_LD1RQ_zprr(DisasContext *s, arg_rprr_load *a, uint32_t insn) +{ + if (a->rm == 31) { + return false; + } + if (sve_access_check(s)) { + int msz = dtype_msz(a->dtype); + TCGv_i64 addr = new_tmp_a64(s); + tcg_gen_shli_i64(addr, cpu_reg(s, a->rm), msz); + tcg_gen_add_i64(addr, addr, cpu_reg_sp(s, a->rn)); + do_ldrq(s, a->rd, a->pg, addr, msz); + } + return true; +} + +static bool trans_LD1RQ_zpri(DisasContext *s, arg_rpri_load *a, uint32_t insn) +{ + if (sve_access_check(s)) { + TCGv_i64 addr = new_tmp_a64(s); + tcg_gen_addi_i64(addr, cpu_reg_sp(s, a->rn), a->imm * 16); + do_ldrq(s, a->rd, a->pg, addr, dtype_msz(a->dtype)); + } + return true; +} + static void do_st_zpa(DisasContext *s, int zt, int pg, TCGv_i64 addr, int msz, int esz, int nreg) { diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ LD_zprr 1010010 .. nreg:2 ..... 110 ... ..... ..... @rprr_load_msz # LD2B, LD2H, LD2W, LD2D; etc. LD_zpri 1010010 .. nreg:2 0.... 111 ... ..... ..... @rpri_load_msz +# SVE load and broadcast quadword (scalar plus scalar) +LD1RQ_zprr 1010010 .. 00 ..... 000 ... ..... ..... \ + @rprr_load_msz nreg=0 + +# SVE load and broadcast quadword (scalar plus immediate) +# LD1RQB, LD1RQH, LD1RQS, LD1RQD +LD1RQ_zpri 1010010 .. 00 0.... 001 ... ..... ..... \ + @rpri_load_msz nreg=0 + ### SVE Memory Store Group # SVE contiguous store (scalar plus immediate) -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> [PMM: fixed typo] Message-id: 20180627043328.11531-6-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 30 +++++++++++++ target/arm/sve_helper.c | 38 ++++++++++++++++ target/arm/translate-sve.c | 90 ++++++++++++++++++++++++++++++++++++++ target/arm/sve.decode | 22 ++++++++++ 4 files changed, 180 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_5(gvec_rsqrts_s, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_5(gvec_rsqrts_d, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_scvt_hh, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_scvt_sh, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_scvt_dh, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_scvt_ss, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_scvt_sd, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_scvt_ds, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_scvt_dd, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_5(sve_ucvt_hh, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_ucvt_sh, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_ucvt_dh, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_ucvt_ss, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_ucvt_sd, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_ucvt_ds, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_ucvt_dd, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) + DEF_HELPER_FLAGS_4(sve_ld1bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ld2bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ld3bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ uint32_t HELPER(sve_while)(void *vd, uint32_t count, uint32_t pred_desc) return predtest_ones(d, oprsz, esz_mask); } +/* Fully general two-operand expander, controlled by a predicate, + * With the extra float_status parameter. + */ +#define DO_ZPZ_FP(NAME, TYPE, H, OP) \ +void HELPER(NAME)(void *vd, void *vn, void *vg, void *status, uint32_t desc) \ +{ \ + intptr_t i = simd_oprsz(desc); \ + uint64_t *g = vg; \ + do { \ + uint64_t pg = g[(i - 1) >> 6]; \ + do { \ + i -= sizeof(TYPE); \ + if (likely((pg >> (i & 63)) & 1)) { \ + TYPE nn = *(TYPE *)(vn + H(i)); \ + *(TYPE *)(vd + H(i)) = OP(nn, status); \ + } \ + } while (i & 63); \ + } while (i != 0); \ +} + +DO_ZPZ_FP(sve_scvt_hh, uint16_t, H1_2, int16_to_float16) +DO_ZPZ_FP(sve_scvt_sh, uint32_t, H1_4, int32_to_float16) +DO_ZPZ_FP(sve_scvt_ss, uint32_t, H1_4, int32_to_float32) +DO_ZPZ_FP(sve_scvt_sd, uint64_t, , int32_to_float64) +DO_ZPZ_FP(sve_scvt_dh, uint64_t, , int64_to_float16) +DO_ZPZ_FP(sve_scvt_ds, uint64_t, , int64_to_float32) +DO_ZPZ_FP(sve_scvt_dd, uint64_t, , int64_to_float64) + +DO_ZPZ_FP(sve_ucvt_hh, uint16_t, H1_2, uint16_to_float16) +DO_ZPZ_FP(sve_ucvt_sh, uint32_t, H1_4, uint32_to_float16) +DO_ZPZ_FP(sve_ucvt_ss, uint32_t, H1_4, uint32_to_float32) +DO_ZPZ_FP(sve_ucvt_sd, uint64_t, , uint32_to_float64) +DO_ZPZ_FP(sve_ucvt_dh, uint64_t, , uint64_to_float16) +DO_ZPZ_FP(sve_ucvt_ds, uint64_t, , uint64_to_float32) +DO_ZPZ_FP(sve_ucvt_dd, uint64_t, , uint64_to_float64) + +#undef DO_ZPZ_FP + /* * Load contiguous data, protected by a governing predicate. */ diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ DO_FP3(FRSQRTS, rsqrts) #undef DO_FP3 + +/* + *** SVE Floating Point Unary Operations Predicated Group + */ + +static bool do_zpz_ptr(DisasContext *s, int rd, int rn, int pg, + bool is_fp16, gen_helper_gvec_3_ptr *fn) +{ + if (sve_access_check(s)) { + unsigned vsz = vec_full_reg_size(s); + TCGv_ptr status = get_fpstatus_ptr(is_fp16); + tcg_gen_gvec_3_ptr(vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), + pred_full_reg_offset(s, pg), + status, vsz, vsz, 0, fn); + tcg_temp_free_ptr(status); + } + return true; +} + +static bool trans_SCVTF_hh(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, true, gen_helper_sve_scvt_hh); +} + +static bool trans_SCVTF_sh(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, true, gen_helper_sve_scvt_sh); +} + +static bool trans_SCVTF_dh(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, true, gen_helper_sve_scvt_dh); +} + +static bool trans_SCVTF_ss(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_scvt_ss); +} + +static bool trans_SCVTF_ds(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_scvt_ds); +} + +static bool trans_SCVTF_sd(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_scvt_sd); +} + +static bool trans_SCVTF_dd(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_scvt_dd); +} + +static bool trans_UCVTF_hh(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, true, gen_helper_sve_ucvt_hh); +} + +static bool trans_UCVTF_sh(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, true, gen_helper_sve_ucvt_sh); +} + +static bool trans_UCVTF_dh(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, true, gen_helper_sve_ucvt_dh); +} + +static bool trans_UCVTF_ss(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_ucvt_ss); +} + +static bool trans_UCVTF_ds(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_ucvt_ds); +} + +static bool trans_UCVTF_sd(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_ucvt_sd); +} + +static bool trans_UCVTF_dd(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_ucvt_dd); +} + /* *** SVE Memory - 32-bit Gather and Unsized Contiguous Group */ diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ @rd_pg_rn ........ esz:2 ... ... ... pg:3 rn:5 rd:5 &rpr_esz @rd_pg4_pn ........ esz:2 ... ... .. pg:4 . rn:4 rd:5 &rpr_esz +# One register operand, with governing predicate, no vector element size +@rd_pg_rn_e0 ........ .. ... ... ... pg:3 rn:5 rd:5 &rpr_esz esz=0 + # Two register operands with a 6-bit signed immediate. @rd_rn_i6 ........ ... rn:5 ..... imm:s6 rd:5 &rri @@ -XXX,XX +XXX,XX @@ FTSMUL 01100101 .. 0 ..... 000 011 ..... ..... @rd_rn_rm FRECPS 01100101 .. 0 ..... 000 110 ..... ..... @rd_rn_rm FRSQRTS 01100101 .. 0 ..... 000 111 ..... ..... @rd_rn_rm +### SVE FP Unary Operations Predicated Group + +# SVE integer convert to floating-point +SCVTF_hh 01100101 01 010 01 0 101 ... ..... ..... @rd_pg_rn_e0 +SCVTF_sh 01100101 01 010 10 0 101 ... ..... ..... @rd_pg_rn_e0 +SCVTF_dh 01100101 01 010 11 0 101 ... ..... ..... @rd_pg_rn_e0 +SCVTF_ss 01100101 10 010 10 0 101 ... ..... ..... @rd_pg_rn_e0 +SCVTF_sd 01100101 11 010 00 0 101 ... ..... ..... @rd_pg_rn_e0 +SCVTF_ds 01100101 11 010 10 0 101 ... ..... ..... @rd_pg_rn_e0 +SCVTF_dd 01100101 11 010 11 0 101 ... ..... ..... @rd_pg_rn_e0 + +UCVTF_hh 01100101 01 010 01 1 101 ... ..... ..... @rd_pg_rn_e0 +UCVTF_sh 01100101 01 010 10 1 101 ... ..... ..... @rd_pg_rn_e0 +UCVTF_dh 01100101 01 010 11 1 101 ... ..... ..... @rd_pg_rn_e0 +UCVTF_ss 01100101 10 010 10 1 101 ... ..... ..... @rd_pg_rn_e0 +UCVTF_sd 01100101 11 010 00 1 101 ... ..... ..... @rd_pg_rn_e0 +UCVTF_ds 01100101 11 010 10 1 101 ... ..... ..... @rd_pg_rn_e0 +UCVTF_dd 01100101 11 010 11 1 101 ... ..... ..... @rd_pg_rn_e0 + ### SVE Memory - 32-bit Gather and Unsized Contiguous Group # SVE load predicate register -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-7-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 77 +++++++++++++++++++++++++++++++++ target/arm/sve_helper.c | 89 ++++++++++++++++++++++++++++++++++++++ target/arm/translate-sve.c | 46 ++++++++++++++++++++ target/arm/sve.decode | 17 ++++++++ 4 files changed, 229 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_5(gvec_rsqrts_s, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_5(gvec_rsqrts_d, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fadd_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fadd_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fadd_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fsub_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fsub_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fsub_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fmul_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fmul_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fmul_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fdiv_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fdiv_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fdiv_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fmin_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fmin_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fmin_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fmax_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fmax_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fmax_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fminnum_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fminnum_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fminnum_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fmaxnum_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fmaxnum_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fmaxnum_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fabd_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fabd_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fabd_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fscalbn_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fscalbn_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fscalbn_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fmulx_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fmulx_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fmulx_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + DEF_HELPER_FLAGS_5(sve_scvt_hh, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_5(sve_scvt_sh, TCG_CALL_NO_RWG, diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ uint32_t HELPER(sve_while)(void *vd, uint32_t count, uint32_t pred_desc) return predtest_ones(d, oprsz, esz_mask); } +/* Fully general three-operand expander, controlled by a predicate, + * With the extra float_status parameter. + */ +#define DO_ZPZZ_FP(NAME, TYPE, H, OP) \ +void HELPER(NAME)(void *vd, void *vn, void *vm, void *vg, \ + void *status, uint32_t desc) \ +{ \ + intptr_t i = simd_oprsz(desc); \ + uint64_t *g = vg; \ + do { \ + uint64_t pg = g[(i - 1) >> 6]; \ + do { \ + i -= sizeof(TYPE); \ + if (likely((pg >> (i & 63)) & 1)) { \ + TYPE nn = *(TYPE *)(vn + H(i)); \ + TYPE mm = *(TYPE *)(vm + H(i)); \ + *(TYPE *)(vd + H(i)) = OP(nn, mm, status); \ + } \ + } while (i & 63); \ + } while (i != 0); \ +} + +DO_ZPZZ_FP(sve_fadd_h, uint16_t, H1_2, float16_add) +DO_ZPZZ_FP(sve_fadd_s, uint32_t, H1_4, float32_add) +DO_ZPZZ_FP(sve_fadd_d, uint64_t, , float64_add) + +DO_ZPZZ_FP(sve_fsub_h, uint16_t, H1_2, float16_sub) +DO_ZPZZ_FP(sve_fsub_s, uint32_t, H1_4, float32_sub) +DO_ZPZZ_FP(sve_fsub_d, uint64_t, , float64_sub) + +DO_ZPZZ_FP(sve_fmul_h, uint16_t, H1_2, float16_mul) +DO_ZPZZ_FP(sve_fmul_s, uint32_t, H1_4, float32_mul) +DO_ZPZZ_FP(sve_fmul_d, uint64_t, , float64_mul) + +DO_ZPZZ_FP(sve_fdiv_h, uint16_t, H1_2, float16_div) +DO_ZPZZ_FP(sve_fdiv_s, uint32_t, H1_4, float32_div) +DO_ZPZZ_FP(sve_fdiv_d, uint64_t, , float64_div) + +DO_ZPZZ_FP(sve_fmin_h, uint16_t, H1_2, float16_min) +DO_ZPZZ_FP(sve_fmin_s, uint32_t, H1_4, float32_min) +DO_ZPZZ_FP(sve_fmin_d, uint64_t, , float64_min) + +DO_ZPZZ_FP(sve_fmax_h, uint16_t, H1_2, float16_max) +DO_ZPZZ_FP(sve_fmax_s, uint32_t, H1_4, float32_max) +DO_ZPZZ_FP(sve_fmax_d, uint64_t, , float64_max) + +DO_ZPZZ_FP(sve_fminnum_h, uint16_t, H1_2, float16_minnum) +DO_ZPZZ_FP(sve_fminnum_s, uint32_t, H1_4, float32_minnum) +DO_ZPZZ_FP(sve_fminnum_d, uint64_t, , float64_minnum) + +DO_ZPZZ_FP(sve_fmaxnum_h, uint16_t, H1_2, float16_maxnum) +DO_ZPZZ_FP(sve_fmaxnum_s, uint32_t, H1_4, float32_maxnum) +DO_ZPZZ_FP(sve_fmaxnum_d, uint64_t, , float64_maxnum) + +static inline float16 abd_h(float16 a, float16 b, float_status *s) +{ + return float16_abs(float16_sub(a, b, s)); +} + +static inline float32 abd_s(float32 a, float32 b, float_status *s) +{ + return float32_abs(float32_sub(a, b, s)); +} + +static inline float64 abd_d(float64 a, float64 b, float_status *s) +{ + return float64_abs(float64_sub(a, b, s)); +} + +DO_ZPZZ_FP(sve_fabd_h, uint16_t, H1_2, abd_h) +DO_ZPZZ_FP(sve_fabd_s, uint32_t, H1_4, abd_s) +DO_ZPZZ_FP(sve_fabd_d, uint64_t, , abd_d) + +static inline float64 scalbn_d(float64 a, int64_t b, float_status *s) +{ + int b_int = MIN(MAX(b, INT_MIN), INT_MAX); + return float64_scalbn(a, b_int, s); +} + +DO_ZPZZ_FP(sve_fscalbn_h, int16_t, H1_2, float16_scalbn) +DO_ZPZZ_FP(sve_fscalbn_s, int32_t, H1_4, float32_scalbn) +DO_ZPZZ_FP(sve_fscalbn_d, int64_t, , scalbn_d) + +DO_ZPZZ_FP(sve_fmulx_h, uint16_t, H1_2, helper_advsimd_mulxh) +DO_ZPZZ_FP(sve_fmulx_s, uint32_t, H1_4, helper_vfp_mulxs) +DO_ZPZZ_FP(sve_fmulx_d, uint64_t, , helper_vfp_mulxd) + +#undef DO_ZPZZ_FP + /* Fully general two-operand expander, controlled by a predicate, * With the extra float_status parameter. */ diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ DO_FP3(FRSQRTS, rsqrts) #undef DO_FP3 +/* + *** SVE Floating Point Arithmetic - Predicated Group + */ + +static bool do_zpzz_fp(DisasContext *s, arg_rprr_esz *a, + gen_helper_gvec_4_ptr *fn) +{ + if (fn == NULL) { + return false; + } + if (sve_access_check(s)) { + unsigned vsz = vec_full_reg_size(s); + TCGv_ptr status = get_fpstatus_ptr(a->esz == MO_16); + tcg_gen_gvec_4_ptr(vec_full_reg_offset(s, a->rd), + vec_full_reg_offset(s, a->rn), + vec_full_reg_offset(s, a->rm), + pred_full_reg_offset(s, a->pg), + status, vsz, vsz, 0, fn); + tcg_temp_free_ptr(status); + } + return true; +} + +#define DO_FP3(NAME, name) \ +static bool trans_##NAME(DisasContext *s, arg_rprr_esz *a, uint32_t insn) \ +{ \ + static gen_helper_gvec_4_ptr * const fns[4] = { \ + NULL, gen_helper_sve_##name##_h, \ + gen_helper_sve_##name##_s, gen_helper_sve_##name##_d \ + }; \ + return do_zpzz_fp(s, a, fns[a->esz]); \ +} + +DO_FP3(FADD_zpzz, fadd) +DO_FP3(FSUB_zpzz, fsub) +DO_FP3(FMUL_zpzz, fmul) +DO_FP3(FMIN_zpzz, fmin) +DO_FP3(FMAX_zpzz, fmax) +DO_FP3(FMINNM_zpzz, fminnum) +DO_FP3(FMAXNM_zpzz, fmaxnum) +DO_FP3(FABD, fabd) +DO_FP3(FSCALE, fscalbn) +DO_FP3(FDIV, fdiv) +DO_FP3(FMULX, fmulx) + +#undef DO_FP3 /* *** SVE Floating Point Unary Operations Predicated Group diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ FTSMUL 01100101 .. 0 ..... 000 011 ..... ..... @rd_rn_rm FRECPS 01100101 .. 0 ..... 000 110 ..... ..... @rd_rn_rm FRSQRTS 01100101 .. 0 ..... 000 111 ..... ..... @rd_rn_rm +### SVE FP Arithmetic Predicated Group + +# SVE floating-point arithmetic (predicated) +FADD_zpzz 01100101 .. 00 0000 100 ... ..... ..... @rdn_pg_rm +FSUB_zpzz 01100101 .. 00 0001 100 ... ..... ..... @rdn_pg_rm +FMUL_zpzz 01100101 .. 00 0010 100 ... ..... ..... @rdn_pg_rm +FSUB_zpzz 01100101 .. 00 0011 100 ... ..... ..... @rdm_pg_rn # FSUBR +FMAXNM_zpzz 01100101 .. 00 0100 100 ... ..... ..... @rdn_pg_rm +FMINNM_zpzz 01100101 .. 00 0101 100 ... ..... ..... @rdn_pg_rm +FMAX_zpzz 01100101 .. 00 0110 100 ... ..... ..... @rdn_pg_rm +FMIN_zpzz 01100101 .. 00 0111 100 ... ..... ..... @rdn_pg_rm +FABD 01100101 .. 00 1000 100 ... ..... ..... @rdn_pg_rm +FSCALE 01100101 .. 00 1001 100 ... ..... ..... @rdn_pg_rm +FMULX 01100101 .. 00 1010 100 ... ..... ..... @rdn_pg_rm +FDIV 01100101 .. 00 1100 100 ... ..... ..... @rdm_pg_rn # FDIVR +FDIV 01100101 .. 00 1101 100 ... ..... ..... @rdn_pg_rm + ### SVE FP Unary Operations Predicated Group # SVE integer convert to floating-point -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Message-id: 20180627043328.11531-8-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 16 ++++ target/arm/sve_helper.c | 158 +++++++++++++++++++++++++++++++++++++ target/arm/translate-sve.c | 49 ++++++++++++ target/arm/sve.decode | 18 +++++ 4 files changed, 241 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_5(sve_ucvt_ds, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_5(sve_ucvt_dd, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_3(sve_fmla_zpzzz_h, TCG_CALL_NO_RWG, void, env, ptr, i32) +DEF_HELPER_FLAGS_3(sve_fmla_zpzzz_s, TCG_CALL_NO_RWG, void, env, ptr, i32) +DEF_HELPER_FLAGS_3(sve_fmla_zpzzz_d, TCG_CALL_NO_RWG, void, env, ptr, i32) + +DEF_HELPER_FLAGS_3(sve_fmls_zpzzz_h, TCG_CALL_NO_RWG, void, env, ptr, i32) +DEF_HELPER_FLAGS_3(sve_fmls_zpzzz_s, TCG_CALL_NO_RWG, void, env, ptr, i32) +DEF_HELPER_FLAGS_3(sve_fmls_zpzzz_d, TCG_CALL_NO_RWG, void, env, ptr, i32) + +DEF_HELPER_FLAGS_3(sve_fnmla_zpzzz_h, TCG_CALL_NO_RWG, void, env, ptr, i32) +DEF_HELPER_FLAGS_3(sve_fnmla_zpzzz_s, TCG_CALL_NO_RWG, void, env, ptr, i32) +DEF_HELPER_FLAGS_3(sve_fnmla_zpzzz_d, TCG_CALL_NO_RWG, void, env, ptr, i32) + +DEF_HELPER_FLAGS_3(sve_fnmls_zpzzz_h, TCG_CALL_NO_RWG, void, env, ptr, i32) +DEF_HELPER_FLAGS_3(sve_fnmls_zpzzz_s, TCG_CALL_NO_RWG, void, env, ptr, i32) +DEF_HELPER_FLAGS_3(sve_fnmls_zpzzz_d, TCG_CALL_NO_RWG, void, env, ptr, i32) + DEF_HELPER_FLAGS_4(sve_ld1bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ld2bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ld3bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ DO_ZPZ_FP(sve_ucvt_dd, uint64_t, , uint64_to_float64) #undef DO_ZPZ_FP +/* 4-operand predicated multiply-add. This requires 7 operands to pass + * "properly", so we need to encode some of the registers into DESC. + */ +QEMU_BUILD_BUG_ON(SIMD_DATA_SHIFT + 20 > 32); + +static void do_fmla_zpzzz_h(CPUARMState *env, void *vg, uint32_t desc, + uint16_t neg1, uint16_t neg3) +{ + intptr_t i = simd_oprsz(desc); + unsigned rd = extract32(desc, SIMD_DATA_SHIFT, 5); + unsigned rn = extract32(desc, SIMD_DATA_SHIFT + 5, 5); + unsigned rm = extract32(desc, SIMD_DATA_SHIFT + 10, 5); + unsigned ra = extract32(desc, SIMD_DATA_SHIFT + 15, 5); + void *vd = &env->vfp.zregs[rd]; + void *vn = &env->vfp.zregs[rn]; + void *vm = &env->vfp.zregs[rm]; + void *va = &env->vfp.zregs[ra]; + uint64_t *g = vg; + + do { + uint64_t pg = g[(i - 1) >> 6]; + do { + i -= 2; + if (likely((pg >> (i & 63)) & 1)) { + float16 e1, e2, e3, r; + + e1 = *(uint16_t *)(vn + H1_2(i)) ^ neg1; + e2 = *(uint16_t *)(vm + H1_2(i)); + e3 = *(uint16_t *)(va + H1_2(i)) ^ neg3; + r = float16_muladd(e1, e2, e3, 0, &env->vfp.fp_status); + *(uint16_t *)(vd + H1_2(i)) = r; + } + } while (i & 63); + } while (i != 0); +} + +void HELPER(sve_fmla_zpzzz_h)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_h(env, vg, desc, 0, 0); +} + +void HELPER(sve_fmls_zpzzz_h)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_h(env, vg, desc, 0x8000, 0); +} + +void HELPER(sve_fnmla_zpzzz_h)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_h(env, vg, desc, 0x8000, 0x8000); +} + +void HELPER(sve_fnmls_zpzzz_h)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_h(env, vg, desc, 0, 0x8000); +} + +static void do_fmla_zpzzz_s(CPUARMState *env, void *vg, uint32_t desc, + uint32_t neg1, uint32_t neg3) +{ + intptr_t i = simd_oprsz(desc); + unsigned rd = extract32(desc, SIMD_DATA_SHIFT, 5); + unsigned rn = extract32(desc, SIMD_DATA_SHIFT + 5, 5); + unsigned rm = extract32(desc, SIMD_DATA_SHIFT + 10, 5); + unsigned ra = extract32(desc, SIMD_DATA_SHIFT + 15, 5); + void *vd = &env->vfp.zregs[rd]; + void *vn = &env->vfp.zregs[rn]; + void *vm = &env->vfp.zregs[rm]; + void *va = &env->vfp.zregs[ra]; + uint64_t *g = vg; + + do { + uint64_t pg = g[(i - 1) >> 6]; + do { + i -= 4; + if (likely((pg >> (i & 63)) & 1)) { + float32 e1, e2, e3, r; + + e1 = *(uint32_t *)(vn + H1_4(i)) ^ neg1; + e2 = *(uint32_t *)(vm + H1_4(i)); + e3 = *(uint32_t *)(va + H1_4(i)) ^ neg3; + r = float32_muladd(e1, e2, e3, 0, &env->vfp.fp_status); + *(uint32_t *)(vd + H1_4(i)) = r; + } + } while (i & 63); + } while (i != 0); +} + +void HELPER(sve_fmla_zpzzz_s)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_s(env, vg, desc, 0, 0); +} + +void HELPER(sve_fmls_zpzzz_s)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_s(env, vg, desc, 0x80000000, 0); +} + +void HELPER(sve_fnmla_zpzzz_s)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_s(env, vg, desc, 0x80000000, 0x80000000); +} + +void HELPER(sve_fnmls_zpzzz_s)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_s(env, vg, desc, 0, 0x80000000); +} + +static void do_fmla_zpzzz_d(CPUARMState *env, void *vg, uint32_t desc, + uint64_t neg1, uint64_t neg3) +{ + intptr_t i = simd_oprsz(desc); + unsigned rd = extract32(desc, SIMD_DATA_SHIFT, 5); + unsigned rn = extract32(desc, SIMD_DATA_SHIFT + 5, 5); + unsigned rm = extract32(desc, SIMD_DATA_SHIFT + 10, 5); + unsigned ra = extract32(desc, SIMD_DATA_SHIFT + 15, 5); + void *vd = &env->vfp.zregs[rd]; + void *vn = &env->vfp.zregs[rn]; + void *vm = &env->vfp.zregs[rm]; + void *va = &env->vfp.zregs[ra]; + uint64_t *g = vg; + + do { + uint64_t pg = g[(i - 1) >> 6]; + do { + i -= 8; + if (likely((pg >> (i & 63)) & 1)) { + float64 e1, e2, e3, r; + + e1 = *(uint64_t *)(vn + i) ^ neg1; + e2 = *(uint64_t *)(vm + i); + e3 = *(uint64_t *)(va + i) ^ neg3; + r = float64_muladd(e1, e2, e3, 0, &env->vfp.fp_status); + *(uint64_t *)(vd + i) = r; + } + } while (i & 63); + } while (i != 0); +} + +void HELPER(sve_fmla_zpzzz_d)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_d(env, vg, desc, 0, 0); +} + +void HELPER(sve_fmls_zpzzz_d)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_d(env, vg, desc, INT64_MIN, 0); +} + +void HELPER(sve_fnmla_zpzzz_d)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_d(env, vg, desc, INT64_MIN, INT64_MIN); +} + +void HELPER(sve_fnmls_zpzzz_d)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_d(env, vg, desc, 0, INT64_MIN); +} + /* * Load contiguous data, protected by a governing predicate. */ diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ DO_FP3(FMULX, fmulx) #undef DO_FP3 +typedef void gen_helper_sve_fmla(TCGv_env, TCGv_ptr, TCGv_i32); + +static bool do_fmla(DisasContext *s, arg_rprrr_esz *a, gen_helper_sve_fmla *fn) +{ + if (fn == NULL) { + return false; + } + if (!sve_access_check(s)) { + return true; + } + + unsigned vsz = vec_full_reg_size(s); + unsigned desc; + TCGv_i32 t_desc; + TCGv_ptr pg = tcg_temp_new_ptr(); + + /* We would need 7 operands to pass these arguments "properly". + * So we encode all the register numbers into the descriptor. + */ + desc = deposit32(a->rd, 5, 5, a->rn); + desc = deposit32(desc, 10, 5, a->rm); + desc = deposit32(desc, 15, 5, a->ra); + desc = simd_desc(vsz, vsz, desc); + + t_desc = tcg_const_i32(desc); + tcg_gen_addi_ptr(pg, cpu_env, pred_full_reg_offset(s, a->pg)); + fn(cpu_env, pg, t_desc); + tcg_temp_free_i32(t_desc); + tcg_temp_free_ptr(pg); + return true; +} + +#define DO_FMLA(NAME, name) \ +static bool trans_##NAME(DisasContext *s, arg_rprrr_esz *a, uint32_t insn) \ +{ \ + static gen_helper_sve_fmla * const fns[4] = { \ + NULL, gen_helper_sve_##name##_h, \ + gen_helper_sve_##name##_s, gen_helper_sve_##name##_d \ + }; \ + return do_fmla(s, a, fns[a->esz]); \ +} + +DO_FMLA(FMLA_zpzzz, fmla_zpzzz) +DO_FMLA(FMLS_zpzzz, fmls_zpzzz) +DO_FMLA(FNMLA_zpzzz, fnmla_zpzzz) +DO_FMLA(FNMLS_zpzzz, fnmls_zpzzz) + +#undef DO_FMLA + /* *** SVE Floating Point Unary Operations Predicated Group */ diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ &rprrr_esz ra=%reg_movprfx @rdn_pg_ra_rm ........ esz:2 . rm:5 ... pg:3 ra:5 rd:5 \ &rprrr_esz rn=%reg_movprfx +@rdn_pg_rm_ra ........ esz:2 . ra:5 ... pg:3 rm:5 rd:5 \ + &rprrr_esz rn=%reg_movprfx # One register operand, with governing predicate, vector element size @rd_pg_rn ........ esz:2 ... ... ... pg:3 rn:5 rd:5 &rpr_esz @@ -XXX,XX +XXX,XX @@ FMULX 01100101 .. 00 1010 100 ... ..... ..... @rdn_pg_rm FDIV 01100101 .. 00 1100 100 ... ..... ..... @rdm_pg_rn # FDIVR FDIV 01100101 .. 00 1101 100 ... ..... ..... @rdn_pg_rm +### SVE FP Multiply-Add Group + +# SVE floating-point multiply-accumulate writing addend +FMLA_zpzzz 01100101 .. 1 ..... 000 ... ..... ..... @rda_pg_rn_rm +FMLS_zpzzz 01100101 .. 1 ..... 001 ... ..... ..... @rda_pg_rn_rm +FNMLA_zpzzz 01100101 .. 1 ..... 010 ... ..... ..... @rda_pg_rn_rm +FNMLS_zpzzz 01100101 .. 1 ..... 011 ... ..... ..... @rda_pg_rn_rm + +# SVE floating-point multiply-accumulate writing multiplicand +# Alter the operand extraction order and reuse the helpers from above. +# FMAD, FMSB, FNMAD, FNMS +FMLA_zpzzz 01100101 .. 1 ..... 100 ... ..... ..... @rdn_pg_rm_ra +FMLS_zpzzz 01100101 .. 1 ..... 101 ... ..... ..... @rdn_pg_rm_ra +FNMLA_zpzzz 01100101 .. 1 ..... 110 ... ..... ..... @rdn_pg_rm_ra +FNMLS_zpzzz 01100101 .. 1 ..... 111 ... ..... ..... @rdn_pg_rm_ra + ### SVE FP Unary Operations Predicated Group # SVE integer convert to floating-point -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-9-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 7 +++++ target/arm/sve_helper.c | 56 ++++++++++++++++++++++++++++++++++++++ target/arm/translate-sve.c | 45 ++++++++++++++++++++++++++++++ target/arm/sve.decode | 5 ++++ 4 files changed, 113 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_5(gvec_rsqrts_s, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_5(gvec_rsqrts_d, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fadda_h, TCG_CALL_NO_RWG, + i64, i64, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fadda_s, TCG_CALL_NO_RWG, + i64, i64, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fadda_d, TCG_CALL_NO_RWG, + i64, i64, ptr, ptr, ptr, i32) + DEF_HELPER_FLAGS_6(sve_fadd_h, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_6(sve_fadd_s, TCG_CALL_NO_RWG, diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ uint32_t HELPER(sve_while)(void *vd, uint32_t count, uint32_t pred_desc) return predtest_ones(d, oprsz, esz_mask); } +uint64_t HELPER(sve_fadda_h)(uint64_t nn, void *vm, void *vg, + void *status, uint32_t desc) +{ + intptr_t i = 0, opr_sz = simd_oprsz(desc); + float16 result = nn; + + do { + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); + do { + if (pg & 1) { + float16 mm = *(float16 *)(vm + H1_2(i)); + result = float16_add(result, mm, status); + } + i += sizeof(float16), pg >>= sizeof(float16); + } while (i & 15); + } while (i < opr_sz); + + return result; +} + +uint64_t HELPER(sve_fadda_s)(uint64_t nn, void *vm, void *vg, + void *status, uint32_t desc) +{ + intptr_t i = 0, opr_sz = simd_oprsz(desc); + float32 result = nn; + + do { + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); + do { + if (pg & 1) { + float32 mm = *(float32 *)(vm + H1_2(i)); + result = float32_add(result, mm, status); + } + i += sizeof(float32), pg >>= sizeof(float32); + } while (i & 15); + } while (i < opr_sz); + + return result; +} + +uint64_t HELPER(sve_fadda_d)(uint64_t nn, void *vm, void *vg, + void *status, uint32_t desc) +{ + intptr_t i = 0, opr_sz = simd_oprsz(desc) / 8; + uint64_t *m = vm; + uint8_t *pg = vg; + + for (i = 0; i < opr_sz; i++) { + if (pg[H1(i)] & 1) { + nn = float64_add(nn, m[i], status); + } + } + + return nn; +} + /* Fully general three-operand expander, controlled by a predicate, * With the extra float_status parameter. */ diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ DO_ZZI(UMIN, umin) #undef DO_ZZI +/* + *** SVE Floating Point Accumulating Reduction Group + */ + +static bool trans_FADDA(DisasContext *s, arg_rprr_esz *a, uint32_t insn) +{ + typedef void fadda_fn(TCGv_i64, TCGv_i64, TCGv_ptr, + TCGv_ptr, TCGv_ptr, TCGv_i32); + static fadda_fn * const fns[3] = { + gen_helper_sve_fadda_h, + gen_helper_sve_fadda_s, + gen_helper_sve_fadda_d, + }; + unsigned vsz = vec_full_reg_size(s); + TCGv_ptr t_rm, t_pg, t_fpst; + TCGv_i64 t_val; + TCGv_i32 t_desc; + + if (a->esz == 0) { + return false; + } + if (!sve_access_check(s)) { + return true; + } + + t_val = load_esz(cpu_env, vec_reg_offset(s, a->rn, 0, a->esz), a->esz); + t_rm = tcg_temp_new_ptr(); + t_pg = tcg_temp_new_ptr(); + tcg_gen_addi_ptr(t_rm, cpu_env, vec_full_reg_offset(s, a->rm)); + tcg_gen_addi_ptr(t_pg, cpu_env, pred_full_reg_offset(s, a->pg)); + t_fpst = get_fpstatus_ptr(a->esz == MO_16); + t_desc = tcg_const_i32(simd_desc(vsz, vsz, 0)); + + fns[a->esz - 1](t_val, t_val, t_rm, t_pg, t_fpst, t_desc); + + tcg_temp_free_i32(t_desc); + tcg_temp_free_ptr(t_fpst); + tcg_temp_free_ptr(t_pg); + tcg_temp_free_ptr(t_rm); + + write_fp_dreg(s, a->rd, t_val); + tcg_temp_free_i64(t_val); + return true; +} + /* *** SVE Floating Point Arithmetic - Unpredicated Group */ diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ UMIN_zzi 00100101 .. 101 011 110 ........ ..... @rdn_i8u # SVE integer multiply immediate (unpredicated) MUL_zzi 00100101 .. 110 000 110 ........ ..... @rdn_i8s +### SVE FP Accumulating Reduction Group + +# SVE floating-point serial reduction (predicated) +FADDA 01100101 .. 011 000 001 ... ..... ..... @rdn_pg_rm + ### SVE Floating Point Arithmetic - Unpredicated Group # SVE floating-point arithmetic (unpredicated) -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-10-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 5 +++ target/arm/sve_helper.c | 41 +++++++++++++++++++++++++ target/arm/translate-sve.c | 62 ++++++++++++++++++++++++++++++++++++++ target/arm/sve.decode | 5 +++ 4 files changed, 113 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_3(sve_clr_h, TCG_CALL_NO_RWG, void, ptr, ptr, i32) DEF_HELPER_FLAGS_3(sve_clr_s, TCG_CALL_NO_RWG, void, ptr, ptr, i32) DEF_HELPER_FLAGS_3(sve_clr_d, TCG_CALL_NO_RWG, void, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(sve_movz_b, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(sve_movz_h, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(sve_movz_s, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(sve_movz_d, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) + DEF_HELPER_FLAGS_4(sve_asr_zpzi_b, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(sve_asr_zpzi_h, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(sve_asr_zpzi_s, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ void HELPER(sve_clr_d)(void *vd, void *vg, uint32_t desc) } } +/* Copy Zn into Zd, and store zero into inactive elements. */ +void HELPER(sve_movz_b)(void *vd, void *vn, void *vg, uint32_t desc) +{ + intptr_t i, opr_sz = simd_oprsz(desc) / 8; + uint64_t *d = vd, *n = vn; + uint8_t *pg = vg; + for (i = 0; i < opr_sz; i += 1) { + d[i] = n[i] & expand_pred_b(pg[H1(i)]); + } +} + +void HELPER(sve_movz_h)(void *vd, void *vn, void *vg, uint32_t desc) +{ + intptr_t i, opr_sz = simd_oprsz(desc) / 8; + uint64_t *d = vd, *n = vn; + uint8_t *pg = vg; + for (i = 0; i < opr_sz; i += 1) { + d[i] = n[i] & expand_pred_h(pg[H1(i)]); + } +} + +void HELPER(sve_movz_s)(void *vd, void *vn, void *vg, uint32_t desc) +{ + intptr_t i, opr_sz = simd_oprsz(desc) / 8; + uint64_t *d = vd, *n = vn; + uint8_t *pg = vg; + for (i = 0; i < opr_sz; i += 1) { + d[i] = n[i] & expand_pred_s(pg[H1(i)]); + } +} + +void HELPER(sve_movz_d)(void *vd, void *vn, void *vg, uint32_t desc) +{ + intptr_t i, opr_sz = simd_oprsz(desc) / 8; + uint64_t *d = vd, *n = vn; + uint8_t *pg = vg; + for (i = 0; i < opr_sz; i += 1) { + d[i] = n[1] & -(uint64_t)(pg[H1(i)] & 1); + } +} + /* Three-operand expander, immediate operand, controlled by a predicate. */ #define DO_ZPZI(NAME, TYPE, H, OP) \ diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ static bool do_clr_zp(DisasContext *s, int rd, int pg, int esz) return true; } +/* Copy Zn into Zd, storing zeros into inactive elements. */ +static void do_movz_zpz(DisasContext *s, int rd, int rn, int pg, int esz) +{ + static gen_helper_gvec_3 * const fns[4] = { + gen_helper_sve_movz_b, gen_helper_sve_movz_h, + gen_helper_sve_movz_s, gen_helper_sve_movz_d, + }; + unsigned vsz = vec_full_reg_size(s); + tcg_gen_gvec_3_ool(vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), + pred_full_reg_offset(s, pg), + vsz, vsz, 0, fns[esz]); +} + static bool do_zpzi_ool(DisasContext *s, arg_rpri_esz *a, gen_helper_gvec_3 *fn) { @@ -XXX,XX +XXX,XX @@ static bool trans_LD1RQ_zpri(DisasContext *s, arg_rpri_load *a, uint32_t insn) return true; } +/* Load and broadcast element. */ +static bool trans_LD1R_zpri(DisasContext *s, arg_rpri_load *a, uint32_t insn) +{ + if (!sve_access_check(s)) { + return true; + } + + unsigned vsz = vec_full_reg_size(s); + unsigned psz = pred_full_reg_size(s); + unsigned esz = dtype_esz[a->dtype]; + TCGLabel *over = gen_new_label(); + TCGv_i64 temp; + + /* If the guarding predicate has no bits set, no load occurs. */ + if (psz <= 8) { + /* Reduce the pred_esz_masks value simply to reduce the + * size of the code generated here. + */ + uint64_t psz_mask = MAKE_64BIT_MASK(0, psz * 8); + temp = tcg_temp_new_i64(); + tcg_gen_ld_i64(temp, cpu_env, pred_full_reg_offset(s, a->pg)); + tcg_gen_andi_i64(temp, temp, pred_esz_masks[esz] & psz_mask); + tcg_gen_brcondi_i64(TCG_COND_EQ, temp, 0, over); + tcg_temp_free_i64(temp); + } else { + TCGv_i32 t32 = tcg_temp_new_i32(); + find_last_active(s, t32, esz, a->pg); + tcg_gen_brcondi_i32(TCG_COND_LT, t32, 0, over); + tcg_temp_free_i32(t32); + } + + /* Load the data. */ + temp = tcg_temp_new_i64(); + tcg_gen_addi_i64(temp, cpu_reg_sp(s, a->rn), a->imm << esz); + tcg_gen_qemu_ld_i64(temp, temp, get_mem_index(s), + s->be_data | dtype_mop[a->dtype]); + + /* Broadcast to *all* elements. */ + tcg_gen_gvec_dup_i64(esz, vec_full_reg_offset(s, a->rd), + vsz, vsz, temp); + tcg_temp_free_i64(temp); + + /* Zero the inactive elements. */ + gen_set_label(over); + do_movz_zpz(s, a->rd, a->rd, a->pg, esz); + return true; +} + static void do_st_zpa(DisasContext *s, int zt, int pg, TCGv_i64 addr, int msz, int esz, int nreg) { diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ %imm8_16_10 16:5 10:3 %imm9_16_10 16:s6 10:3 %size_23 23:2 +%dtype_23_13 23:2 13:2 # A combination of tsz:imm3 -- extract esize. %tszimm_esz 22:2 5:5 !function=tszimm_esz @@ -XXX,XX +XXX,XX @@ LDR_pri 10000101 10 ...... 000 ... ..... 0 .... @pd_rn_i9 # SVE load vector register LDR_zri 10000101 10 ...... 010 ... ..... ..... @rd_rn_i9 +# SVE load and broadcast element +LD1R_zpri 1000010 .. 1 imm:6 1.. pg:3 rn:5 rd:5 \ + &rpri_load dtype=%dtype_23_13 nreg=0 + ### SVE Memory Contiguous Load Group # SVE contiguous load (scalar plus scalar) -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-11-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/translate-sve.c | 103 +++++++++++++++++++++++++++++++++++++ target/arm/sve.decode | 6 +++ 2 files changed, 109 insertions(+) diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ static void do_ldr(DisasContext *s, uint32_t vofs, uint32_t len, tcg_temp_free_i64(t0); } +/* Similarly for stores. */ +static void do_str(DisasContext *s, uint32_t vofs, uint32_t len, + int rn, int imm) +{ + uint32_t len_align = QEMU_ALIGN_DOWN(len, 8); + uint32_t len_remain = len % 8; + uint32_t nparts = len / 8 + ctpop8(len_remain); + int midx = get_mem_index(s); + TCGv_i64 addr, t0; + + addr = tcg_temp_new_i64(); + t0 = tcg_temp_new_i64(); + + /* Note that unpredicated load/store of vector/predicate registers + * are defined as a stream of bytes, which equates to little-endian + * operations on larger quantities. There is no nice way to force + * a little-endian store for aarch64_be-linux-user out of line. + * + * Attempt to keep code expansion to a minimum by limiting the + * amount of unrolling done. + */ + if (nparts <= 4) { + int i; + + for (i = 0; i < len_align; i += 8) { + tcg_gen_ld_i64(t0, cpu_env, vofs + i); + tcg_gen_addi_i64(addr, cpu_reg_sp(s, rn), imm + i); + tcg_gen_qemu_st_i64(t0, addr, midx, MO_LEQ); + } + } else { + TCGLabel *loop = gen_new_label(); + TCGv_ptr t2, i = tcg_const_local_ptr(0); + + gen_set_label(loop); + + t2 = tcg_temp_new_ptr(); + tcg_gen_add_ptr(t2, cpu_env, i); + tcg_gen_ld_i64(t0, t2, vofs); + + /* Minimize the number of local temps that must be re-read from + * the stack each iteration. Instead, re-compute values other + * than the loop counter. + */ + tcg_gen_addi_ptr(t2, i, imm); + tcg_gen_extu_ptr_i64(addr, t2); + tcg_gen_add_i64(addr, addr, cpu_reg_sp(s, rn)); + tcg_temp_free_ptr(t2); + + tcg_gen_qemu_st_i64(t0, addr, midx, MO_LEQ); + + tcg_gen_addi_ptr(i, i, 8); + + tcg_gen_brcondi_ptr(TCG_COND_LTU, i, len_align, loop); + tcg_temp_free_ptr(i); + } + + /* Predicate register stores can be any multiple of 2. */ + if (len_remain) { + tcg_gen_ld_i64(t0, cpu_env, vofs + len_align); + tcg_gen_addi_i64(addr, cpu_reg_sp(s, rn), imm + len_align); + + switch (len_remain) { + case 2: + case 4: + case 8: + tcg_gen_qemu_st_i64(t0, addr, midx, MO_LE | ctz32(len_remain)); + break; + + case 6: + tcg_gen_qemu_st_i64(t0, addr, midx, MO_LEUL); + tcg_gen_addi_i64(addr, addr, 4); + tcg_gen_shri_i64(t0, t0, 32); + tcg_gen_qemu_st_i64(t0, addr, midx, MO_LEUW); + break; + + default: + g_assert_not_reached(); + } + } + tcg_temp_free_i64(addr); + tcg_temp_free_i64(t0); +} + static bool trans_LDR_zri(DisasContext *s, arg_rri *a, uint32_t insn) { if (sve_access_check(s)) { @@ -XXX,XX +XXX,XX @@ static bool trans_LDR_pri(DisasContext *s, arg_rri *a, uint32_t insn) return true; } +static bool trans_STR_zri(DisasContext *s, arg_rri *a, uint32_t insn) +{ + if (sve_access_check(s)) { + int size = vec_full_reg_size(s); + int off = vec_full_reg_offset(s, a->rd); + do_str(s, off, size, a->rn, a->imm * size); + } + return true; +} + +static bool trans_STR_pri(DisasContext *s, arg_rri *a, uint32_t insn) +{ + if (sve_access_check(s)) { + int size = pred_full_reg_size(s); + int off = pred_full_reg_offset(s, a->rd); + do_str(s, off, size, a->rn, a->imm * size); + } + return true; +} + /* *** SVE Memory - Contiguous Load Group */ diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ LD1RQ_zpri 1010010 .. 00 0.... 001 ... ..... ..... \ ### SVE Memory Store Group +# SVE store predicate register +STR_pri 1110010 11 0. ..... 000 ... ..... 0 .... @pd_rn_i9 + +# SVE store vector register +STR_zri 1110010 11 0. ..... 010 ... ..... ..... @rd_rn_i9 + # SVE contiguous store (scalar plus immediate) # ST1B, ST1H, ST1W, ST1D; require msz <= esz ST_zpri 1110010 .. esz:2 0.... 111 ... ..... ..... \ -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-12-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 41 +++++++++++++++++++++ target/arm/sve_helper.c | 61 +++++++++++++++++++++++++++++++ target/arm/translate-sve.c | 75 ++++++++++++++++++++++++++++++++++++++ target/arm/sve.decode | 39 ++++++++++++++++++++ 4 files changed, 216 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_4(sve_st1hs_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_st1hd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_st1sd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_6(sve_stbs_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_sths_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stss_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) + +DEF_HELPER_FLAGS_6(sve_stbs_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_sths_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stss_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) + +DEF_HELPER_FLAGS_6(sve_stbd_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_sthd_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stsd_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stdd_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) + +DEF_HELPER_FLAGS_6(sve_stbd_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_sthd_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stsd_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stdd_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) + +DEF_HELPER_FLAGS_6(sve_stbd_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_sthd_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stsd_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stdd_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ void HELPER(sve_st4dd_r)(CPUARMState *env, void *vg, addr += 4 * 8; } } + +/* Stores with a vector index. */ + +#define DO_ST1_ZPZ_S(NAME, TYPEI, FN) \ +void HELPER(NAME)(CPUARMState *env, void *vd, void *vg, void *vm, \ + target_ulong base, uint32_t desc) \ +{ \ + intptr_t i, oprsz = simd_oprsz(desc); \ + unsigned scale = simd_data(desc); \ + uintptr_t ra = GETPC(); \ + for (i = 0; i < oprsz; ) { \ + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ + do { \ + if (likely(pg & 1)) { \ + target_ulong off = *(TYPEI *)(vm + H1_4(i)); \ + uint32_t d = *(uint32_t *)(vd + H1_4(i)); \ + FN(env, base + (off << scale), d, ra); \ + } \ + i += sizeof(uint32_t), pg >>= sizeof(uint32_t); \ + } while (i & 15); \ + } \ +} + +#define DO_ST1_ZPZ_D(NAME, TYPEI, FN) \ +void HELPER(NAME)(CPUARMState *env, void *vd, void *vg, void *vm, \ + target_ulong base, uint32_t desc) \ +{ \ + intptr_t i, oprsz = simd_oprsz(desc) / 8; \ + unsigned scale = simd_data(desc); \ + uintptr_t ra = GETPC(); \ + uint64_t *d = vd, *m = vm; uint8_t *pg = vg; \ + for (i = 0; i < oprsz; i++) { \ + if (likely(pg[H1(i)] & 1)) { \ + target_ulong off = (target_ulong)(TYPEI)m[i] << scale; \ + FN(env, base + off, d[i], ra); \ + } \ + } \ +} + +DO_ST1_ZPZ_S(sve_stbs_zsu, uint32_t, cpu_stb_data_ra) +DO_ST1_ZPZ_S(sve_sths_zsu, uint32_t, cpu_stw_data_ra) +DO_ST1_ZPZ_S(sve_stss_zsu, uint32_t, cpu_stl_data_ra) + +DO_ST1_ZPZ_S(sve_stbs_zss, int32_t, cpu_stb_data_ra) +DO_ST1_ZPZ_S(sve_sths_zss, int32_t, cpu_stw_data_ra) +DO_ST1_ZPZ_S(sve_stss_zss, int32_t, cpu_stl_data_ra) + +DO_ST1_ZPZ_D(sve_stbd_zsu, uint32_t, cpu_stb_data_ra) +DO_ST1_ZPZ_D(sve_sthd_zsu, uint32_t, cpu_stw_data_ra) +DO_ST1_ZPZ_D(sve_stsd_zsu, uint32_t, cpu_stl_data_ra) +DO_ST1_ZPZ_D(sve_stdd_zsu, uint32_t, cpu_stq_data_ra) + +DO_ST1_ZPZ_D(sve_stbd_zss, int32_t, cpu_stb_data_ra) +DO_ST1_ZPZ_D(sve_sthd_zss, int32_t, cpu_stw_data_ra) +DO_ST1_ZPZ_D(sve_stsd_zss, int32_t, cpu_stl_data_ra) +DO_ST1_ZPZ_D(sve_stdd_zss, int32_t, cpu_stq_data_ra) + +DO_ST1_ZPZ_D(sve_stbd_zd, uint64_t, cpu_stb_data_ra) +DO_ST1_ZPZ_D(sve_sthd_zd, uint64_t, cpu_stw_data_ra) +DO_ST1_ZPZ_D(sve_stsd_zd, uint64_t, cpu_stl_data_ra) +DO_ST1_ZPZ_D(sve_stdd_zd, uint64_t, cpu_stq_data_ra) diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ typedef void gen_helper_gvec_flags_4(TCGv_i32, TCGv_ptr, TCGv_ptr, TCGv_ptr, TCGv_ptr, TCGv_i32); typedef void gen_helper_gvec_mem(TCGv_env, TCGv_ptr, TCGv_i64, TCGv_i32); +typedef void gen_helper_gvec_mem_scatter(TCGv_env, TCGv_ptr, TCGv_ptr, + TCGv_ptr, TCGv_i64, TCGv_i32); /* * Helpers for extracting complex instruction fields. @@ -XXX,XX +XXX,XX @@ static bool trans_ST_zpri(DisasContext *s, arg_rpri_store *a, uint32_t insn) } return true; } + +/* + *** SVE gather loads / scatter stores + */ + +static void do_mem_zpz(DisasContext *s, int zt, int pg, int zm, int scale, + TCGv_i64 scalar, gen_helper_gvec_mem_scatter *fn) +{ + unsigned vsz = vec_full_reg_size(s); + TCGv_i32 desc = tcg_const_i32(simd_desc(vsz, vsz, scale)); + TCGv_ptr t_zm = tcg_temp_new_ptr(); + TCGv_ptr t_pg = tcg_temp_new_ptr(); + TCGv_ptr t_zt = tcg_temp_new_ptr(); + + tcg_gen_addi_ptr(t_pg, cpu_env, pred_full_reg_offset(s, pg)); + tcg_gen_addi_ptr(t_zm, cpu_env, vec_full_reg_offset(s, zm)); + tcg_gen_addi_ptr(t_zt, cpu_env, vec_full_reg_offset(s, zt)); + fn(cpu_env, t_zt, t_pg, t_zm, scalar, desc); + + tcg_temp_free_ptr(t_zt); + tcg_temp_free_ptr(t_zm); + tcg_temp_free_ptr(t_pg); + tcg_temp_free_i32(desc); +} + +static bool trans_ST1_zprz(DisasContext *s, arg_ST1_zprz *a, uint32_t insn) +{ + /* Indexed by [xs][msz]. */ + static gen_helper_gvec_mem_scatter * const fn32[2][3] = { + { gen_helper_sve_stbs_zsu, + gen_helper_sve_sths_zsu, + gen_helper_sve_stss_zsu, }, + { gen_helper_sve_stbs_zss, + gen_helper_sve_sths_zss, + gen_helper_sve_stss_zss, }, + }; + /* Note that we overload xs=2 to indicate 64-bit offset. */ + static gen_helper_gvec_mem_scatter * const fn64[3][4] = { + { gen_helper_sve_stbd_zsu, + gen_helper_sve_sthd_zsu, + gen_helper_sve_stsd_zsu, + gen_helper_sve_stdd_zsu, }, + { gen_helper_sve_stbd_zss, + gen_helper_sve_sthd_zss, + gen_helper_sve_stsd_zss, + gen_helper_sve_stdd_zss, }, + { gen_helper_sve_stbd_zd, + gen_helper_sve_sthd_zd, + gen_helper_sve_stsd_zd, + gen_helper_sve_stdd_zd, }, + }; + gen_helper_gvec_mem_scatter *fn; + + if (a->esz < a->msz || (a->msz == 0 && a->scale)) { + return false; + } + if (!sve_access_check(s)) { + return true; + } + switch (a->esz) { + case MO_32: + fn = fn32[a->xs][a->msz]; + break; + case MO_64: + fn = fn64[a->xs][a->msz]; + break; + default: + g_assert_not_reached(); + } + do_mem_zpz(s, a->rd, a->pg, a->rm, a->scale * a->msz, + cpu_reg_sp(s, a->rn), fn); + return true; +} diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ &rpri_load rd pg rn imm dtype nreg &rprr_store rd pg rn rm msz esz nreg &rpri_store rd pg rn imm msz esz nreg +&rprr_scatter_store rd pg rn rm esz msz xs scale ########################################################################### # Named instruction formats. These are generally used to @@ -XXX,XX +XXX,XX @@ @rpri_store_msz ....... msz:2 .. . imm:s4 ... pg:3 rn:5 rd:5 &rpri_store @rprr_store_esz_n0 ....... .. esz:2 rm:5 ... pg:3 rn:5 rd:5 \ &rprr_store nreg=0 +@rprr_scatter_store ....... msz:2 .. rm:5 ... pg:3 rn:5 rd:5 \ + &rprr_scatter_store ########################################################################### # Instruction patterns. Grouped according to the SVE encodingindex.xhtml. @@ -XXX,XX +XXX,XX @@ ST_zpri 1110010 .. nreg:2 1.... 111 ... ..... ..... \ # SVE store multiple structures (scalar plus scalar) (nreg != 0) ST_zprr 1110010 msz:2 nreg:2 ..... 011 ... ..... ..... \ @rprr_store esz=%size_23 + +# SVE 32-bit scatter store (scalar plus 32-bit scaled offsets) +# Require msz > 0 && msz <= esz. +ST1_zprz 1110010 .. 11 ..... 100 ... ..... ..... \ + @rprr_scatter_store xs=0 esz=2 scale=1 +ST1_zprz 1110010 .. 11 ..... 110 ... ..... ..... \ + @rprr_scatter_store xs=1 esz=2 scale=1 + +# SVE 32-bit scatter store (scalar plus 32-bit unscaled offsets) +# Require msz <= esz. +ST1_zprz 1110010 .. 10 ..... 100 ... ..... ..... \ + @rprr_scatter_store xs=0 esz=2 scale=0 +ST1_zprz 1110010 .. 10 ..... 110 ... ..... ..... \ + @rprr_scatter_store xs=1 esz=2 scale=0 + +# SVE 64-bit scatter store (scalar plus 64-bit scaled offset) +# Require msz > 0 +ST1_zprz 1110010 .. 01 ..... 101 ... ..... ..... \ + @rprr_scatter_store xs=2 esz=3 scale=1 + +# SVE 64-bit scatter store (scalar plus 64-bit unscaled offset) +ST1_zprz 1110010 .. 00 ..... 101 ... ..... ..... \ + @rprr_scatter_store xs=2 esz=3 scale=0 + +# SVE 64-bit scatter store (scalar plus unpacked 32-bit scaled offset) +# Require msz > 0 +ST1_zprz 1110010 .. 01 ..... 100 ... ..... ..... \ + @rprr_scatter_store xs=0 esz=3 scale=1 +ST1_zprz 1110010 .. 01 ..... 110 ... ..... ..... \ + @rprr_scatter_store xs=1 esz=3 scale=1 + +# SVE 64-bit scatter store (scalar plus unpacked 32-bit unscaled offset) +ST1_zprz 1110010 .. 00 ..... 100 ... ..... ..... \ + @rprr_scatter_store xs=0 esz=3 scale=0 +ST1_zprz 1110010 .. 00 ..... 110 ... ..... ..... \ + @rprr_scatter_store xs=1 esz=3 scale=0 -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-13-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/translate-sve.c | 21 +++++++++++++++++++++ target/arm/sve.decode | 23 +++++++++++++++++++++++ 2 files changed, 44 insertions(+) diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ static bool trans_ST1_zprz(DisasContext *s, arg_ST1_zprz *a, uint32_t insn) cpu_reg_sp(s, a->rn), fn); return true; } + +/* + * Prefetches + */ + +static bool trans_PRF(DisasContext *s, arg_PRF *a, uint32_t insn) +{ + /* Prefetch is a nop within QEMU. */ + sve_access_check(s); + return true; +} + +static bool trans_PRF_rr(DisasContext *s, arg_PRF_rr *a, uint32_t insn) +{ + if (a->rm == 31) { + return false; + } + /* Prefetch is a nop within QEMU. */ + sve_access_check(s); + return true; +} diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ LD1RQ_zprr 1010010 .. 00 ..... 000 ... ..... ..... \ LD1RQ_zpri 1010010 .. 00 0.... 001 ... ..... ..... \ @rpri_load_msz nreg=0 +# SVE 32-bit gather prefetch (scalar plus 32-bit scaled offsets) +PRF 1000010 00 -1 ----- 0-- --- ----- 0 ---- + +# SVE 32-bit gather prefetch (vector plus immediate) +PRF 1000010 -- 00 ----- 111 --- ----- 0 ---- + +# SVE contiguous prefetch (scalar plus immediate) +PRF 1000010 11 1- ----- 0-- --- ----- 0 ---- + +# SVE contiguous prefetch (scalar plus scalar) +PRF_rr 1000010 -- 00 rm:5 110 --- ----- 0 ---- + +### SVE Memory 64-bit Gather Group + +# SVE 64-bit gather prefetch (scalar plus 64-bit scaled offsets) +PRF 1100010 00 11 ----- 1-- --- ----- 0 ---- + +# SVE 64-bit gather prefetch (scalar plus unpacked 32-bit scaled offsets) +PRF 1100010 00 -1 ----- 0-- --- ----- 0 ---- + +# SVE 64-bit gather prefetch (vector plus immediate) +PRF 1100010 -- 00 ----- 111 --- ----- 0 ---- + ### SVE Memory Store Group # SVE store predicate register -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Message-id: 20180627043328.11531-14-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 67 +++++++++++++++++++++++++ target/arm/sve_helper.c | 77 ++++++++++++++++++++++++++++ target/arm/translate-sve.c | 100 +++++++++++++++++++++++++++++++++++++ target/arm/sve.decode | 57 +++++++++++++++++++++ 4 files changed, 301 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_4(sve_st1hd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_st1sd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldbsu_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldhsu_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldssu_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldbss_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldhss_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) + +DEF_HELPER_FLAGS_6(sve_ldbsu_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldhsu_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldssu_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldbss_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldhss_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) + +DEF_HELPER_FLAGS_6(sve_ldbdu_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldhdu_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldsdu_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldddu_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldbds_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldhds_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldsds_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) + +DEF_HELPER_FLAGS_6(sve_ldbdu_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldhdu_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldsdu_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldddu_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldbds_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldhds_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldsds_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) + +DEF_HELPER_FLAGS_6(sve_ldbdu_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldhdu_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldsdu_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldddu_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldbds_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldhds_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldsds_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) + DEF_HELPER_FLAGS_6(sve_stbs_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_sths_zsu, TCG_CALL_NO_WG, diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ void HELPER(sve_st4dd_r)(CPUARMState *env, void *vg, } } +/* Loads with a vector index. */ + +#define DO_LD1_ZPZ_S(NAME, TYPEI, TYPEM, FN) \ +void HELPER(NAME)(CPUARMState *env, void *vd, void *vg, void *vm, \ + target_ulong base, uint32_t desc) \ +{ \ + intptr_t i, oprsz = simd_oprsz(desc); \ + unsigned scale = simd_data(desc); \ + uintptr_t ra = GETPC(); \ + for (i = 0; i < oprsz; i++) { \ + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ + do { \ + TYPEM m = 0; \ + if (pg & 1) { \ + target_ulong off = *(TYPEI *)(vm + H1_4(i)); \ + m = FN(env, base + (off << scale), ra); \ + } \ + *(uint32_t *)(vd + H1_4(i)) = m; \ + i += 4, pg >>= 4; \ + } while (i & 15); \ + } \ +} + +#define DO_LD1_ZPZ_D(NAME, TYPEI, TYPEM, FN) \ +void HELPER(NAME)(CPUARMState *env, void *vd, void *vg, void *vm, \ + target_ulong base, uint32_t desc) \ +{ \ + intptr_t i, oprsz = simd_oprsz(desc) / 8; \ + unsigned scale = simd_data(desc); \ + uintptr_t ra = GETPC(); \ + uint64_t *d = vd, *m = vm; uint8_t *pg = vg; \ + for (i = 0; i < oprsz; i++) { \ + TYPEM mm = 0; \ + if (pg[H1(i)] & 1) { \ + target_ulong off = (TYPEI)m[i]; \ + mm = FN(env, base + (off << scale), ra); \ + } \ + d[i] = mm; \ + } \ +} + +DO_LD1_ZPZ_S(sve_ldbsu_zsu, uint32_t, uint8_t, cpu_ldub_data_ra) +DO_LD1_ZPZ_S(sve_ldhsu_zsu, uint32_t, uint16_t, cpu_lduw_data_ra) +DO_LD1_ZPZ_S(sve_ldssu_zsu, uint32_t, uint32_t, cpu_ldl_data_ra) +DO_LD1_ZPZ_S(sve_ldbss_zsu, uint32_t, int8_t, cpu_ldub_data_ra) +DO_LD1_ZPZ_S(sve_ldhss_zsu, uint32_t, int16_t, cpu_lduw_data_ra) + +DO_LD1_ZPZ_S(sve_ldbsu_zss, int32_t, uint8_t, cpu_ldub_data_ra) +DO_LD1_ZPZ_S(sve_ldhsu_zss, int32_t, uint16_t, cpu_lduw_data_ra) +DO_LD1_ZPZ_S(sve_ldssu_zss, int32_t, uint32_t, cpu_ldl_data_ra) +DO_LD1_ZPZ_S(sve_ldbss_zss, int32_t, int8_t, cpu_ldub_data_ra) +DO_LD1_ZPZ_S(sve_ldhss_zss, int32_t, int16_t, cpu_lduw_data_ra) + +DO_LD1_ZPZ_D(sve_ldbdu_zsu, uint32_t, uint8_t, cpu_ldub_data_ra) +DO_LD1_ZPZ_D(sve_ldhdu_zsu, uint32_t, uint16_t, cpu_lduw_data_ra) +DO_LD1_ZPZ_D(sve_ldsdu_zsu, uint32_t, uint32_t, cpu_ldl_data_ra) +DO_LD1_ZPZ_D(sve_ldddu_zsu, uint32_t, uint64_t, cpu_ldq_data_ra) +DO_LD1_ZPZ_D(sve_ldbds_zsu, uint32_t, int8_t, cpu_ldub_data_ra) +DO_LD1_ZPZ_D(sve_ldhds_zsu, uint32_t, int16_t, cpu_lduw_data_ra) +DO_LD1_ZPZ_D(sve_ldsds_zsu, uint32_t, int32_t, cpu_ldl_data_ra) + +DO_LD1_ZPZ_D(sve_ldbdu_zss, int32_t, uint8_t, cpu_ldub_data_ra) +DO_LD1_ZPZ_D(sve_ldhdu_zss, int32_t, uint16_t, cpu_lduw_data_ra) +DO_LD1_ZPZ_D(sve_ldsdu_zss, int32_t, uint32_t, cpu_ldl_data_ra) +DO_LD1_ZPZ_D(sve_ldddu_zss, int32_t, uint64_t, cpu_ldq_data_ra) +DO_LD1_ZPZ_D(sve_ldbds_zss, int32_t, int8_t, cpu_ldub_data_ra) +DO_LD1_ZPZ_D(sve_ldhds_zss, int32_t, int16_t, cpu_lduw_data_ra) +DO_LD1_ZPZ_D(sve_ldsds_zss, int32_t, int32_t, cpu_ldl_data_ra) + +DO_LD1_ZPZ_D(sve_ldbdu_zd, uint64_t, uint8_t, cpu_ldub_data_ra) +DO_LD1_ZPZ_D(sve_ldhdu_zd, uint64_t, uint16_t, cpu_lduw_data_ra) +DO_LD1_ZPZ_D(sve_ldsdu_zd, uint64_t, uint32_t, cpu_ldl_data_ra) +DO_LD1_ZPZ_D(sve_ldddu_zd, uint64_t, uint64_t, cpu_ldq_data_ra) +DO_LD1_ZPZ_D(sve_ldbds_zd, uint64_t, int8_t, cpu_ldub_data_ra) +DO_LD1_ZPZ_D(sve_ldhds_zd, uint64_t, int16_t, cpu_lduw_data_ra) +DO_LD1_ZPZ_D(sve_ldsds_zd, uint64_t, int32_t, cpu_ldl_data_ra) + /* Stores with a vector index. */ #define DO_ST1_ZPZ_S(NAME, TYPEI, FN) \ diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ static void do_mem_zpz(DisasContext *s, int zt, int pg, int zm, int scale, tcg_temp_free_i32(desc); } +/* Indexed by [ff][xs][u][msz]. */ +static gen_helper_gvec_mem_scatter * const gather_load_fn32[2][2][2][3] = { + { { { gen_helper_sve_ldbss_zsu, + gen_helper_sve_ldhss_zsu, + NULL, }, + { gen_helper_sve_ldbsu_zsu, + gen_helper_sve_ldhsu_zsu, + gen_helper_sve_ldssu_zsu, } }, + { { gen_helper_sve_ldbss_zss, + gen_helper_sve_ldhss_zss, + NULL, }, + { gen_helper_sve_ldbsu_zss, + gen_helper_sve_ldhsu_zss, + gen_helper_sve_ldssu_zss, } } }, + /* TODO fill in first-fault handlers */ +}; + +/* Note that we overload xs=2 to indicate 64-bit offset. */ +static gen_helper_gvec_mem_scatter * const gather_load_fn64[2][3][2][4] = { + { { { gen_helper_sve_ldbds_zsu, + gen_helper_sve_ldhds_zsu, + gen_helper_sve_ldsds_zsu, + NULL, }, + { gen_helper_sve_ldbdu_zsu, + gen_helper_sve_ldhdu_zsu, + gen_helper_sve_ldsdu_zsu, + gen_helper_sve_ldddu_zsu, } }, + { { gen_helper_sve_ldbds_zss, + gen_helper_sve_ldhds_zss, + gen_helper_sve_ldsds_zss, + NULL, }, + { gen_helper_sve_ldbdu_zss, + gen_helper_sve_ldhdu_zss, + gen_helper_sve_ldsdu_zss, + gen_helper_sve_ldddu_zss, } }, + { { gen_helper_sve_ldbds_zd, + gen_helper_sve_ldhds_zd, + gen_helper_sve_ldsds_zd, + NULL, }, + { gen_helper_sve_ldbdu_zd, + gen_helper_sve_ldhdu_zd, + gen_helper_sve_ldsdu_zd, + gen_helper_sve_ldddu_zd, } } }, + /* TODO fill in first-fault handlers */ +}; + +static bool trans_LD1_zprz(DisasContext *s, arg_LD1_zprz *a, uint32_t insn) +{ + gen_helper_gvec_mem_scatter *fn = NULL; + + if (!sve_access_check(s)) { + return true; + } + + switch (a->esz) { + case MO_32: + fn = gather_load_fn32[a->ff][a->xs][a->u][a->msz]; + break; + case MO_64: + fn = gather_load_fn64[a->ff][a->xs][a->u][a->msz]; + break; + } + assert(fn != NULL); + + do_mem_zpz(s, a->rd, a->pg, a->rm, a->scale * a->msz, + cpu_reg_sp(s, a->rn), fn); + return true; +} + +static bool trans_LD1_zpiz(DisasContext *s, arg_LD1_zpiz *a, uint32_t insn) +{ + gen_helper_gvec_mem_scatter *fn = NULL; + TCGv_i64 imm; + + if (a->esz < a->msz || (a->esz == a->msz && !a->u)) { + return false; + } + if (!sve_access_check(s)) { + return true; + } + + switch (a->esz) { + case MO_32: + fn = gather_load_fn32[a->ff][0][a->u][a->msz]; + break; + case MO_64: + fn = gather_load_fn64[a->ff][2][a->u][a->msz]; + break; + } + assert(fn != NULL); + + /* Treat LD1_zpiz (zn[x] + imm) the same way as LD1_zprz (rn + zm[x]) + * by loading the immediate into the scalar parameter. + */ + imm = tcg_const_i64(a->imm << a->msz); + do_mem_zpz(s, a->rd, a->pg, a->rn, 0, imm, fn); + tcg_temp_free_i64(imm); + return true; +} + static bool trans_ST1_zprz(DisasContext *s, arg_ST1_zprz *a, uint32_t insn) { /* Indexed by [xs][msz]. */ diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ &rpri_load rd pg rn imm dtype nreg &rprr_store rd pg rn rm msz esz nreg &rpri_store rd pg rn imm msz esz nreg +&rprr_gather_load rd pg rn rm esz msz u ff xs scale +&rpri_gather_load rd pg rn imm esz msz u ff &rprr_scatter_store rd pg rn rm esz msz xs scale ########################################################################### @@ -XXX,XX +XXX,XX @@ @rpri_load_msz ....... .... . imm:s4 ... pg:3 rn:5 rd:5 \ &rpri_load dtype=%msz_dtype +# Gather Loads. +@rprr_g_load_u ....... .. . . rm:5 . u:1 ff:1 pg:3 rn:5 rd:5 \ + &rprr_gather_load xs=2 +@rprr_g_load_xs_u ....... .. xs:1 . rm:5 . u:1 ff:1 pg:3 rn:5 rd:5 \ + &rprr_gather_load +@rprr_g_load_xs_u_sc ....... .. xs:1 scale:1 rm:5 . u:1 ff:1 pg:3 rn:5 rd:5 \ + &rprr_gather_load +@rprr_g_load_xs_sc ....... .. xs:1 scale:1 rm:5 . . ff:1 pg:3 rn:5 rd:5 \ + &rprr_gather_load +@rprr_g_load_u_sc ....... .. . scale:1 rm:5 . u:1 ff:1 pg:3 rn:5 rd:5 \ + &rprr_gather_load xs=2 +@rprr_g_load_sc ....... .. . scale:1 rm:5 . . ff:1 pg:3 rn:5 rd:5 \ + &rprr_gather_load xs=2 +@rpri_g_load ....... msz:2 .. imm:5 . u:1 ff:1 pg:3 rn:5 rd:5 \ + &rpri_gather_load + # Stores; user must fill in ESZ, MSZ, NREG as needed. @rprr_store ....... .. .. rm:5 ... pg:3 rn:5 rd:5 &rprr_store @rpri_store_msz ....... msz:2 .. . imm:s4 ... pg:3 rn:5 rd:5 &rpri_store @@ -XXX,XX +XXX,XX @@ LDR_zri 10000101 10 ...... 010 ... ..... ..... @rd_rn_i9 LD1R_zpri 1000010 .. 1 imm:6 1.. pg:3 rn:5 rd:5 \ &rpri_load dtype=%dtype_23_13 nreg=0 +# SVE 32-bit gather load (scalar plus 32-bit unscaled offsets) +# SVE 32-bit gather load (scalar plus 32-bit scaled offsets) +LD1_zprz 1000010 00 .0 ..... 0.. ... ..... ..... \ + @rprr_g_load_xs_u esz=2 msz=0 scale=0 +LD1_zprz 1000010 01 .. ..... 0.. ... ..... ..... \ + @rprr_g_load_xs_u_sc esz=2 msz=1 +LD1_zprz 1000010 10 .. ..... 01. ... ..... ..... \ + @rprr_g_load_xs_sc esz=2 msz=2 u=1 + +# SVE 32-bit gather load (vector plus immediate) +LD1_zpiz 1000010 .. 01 ..... 1.. ... ..... ..... \ + @rpri_g_load esz=2 + ### SVE Memory Contiguous Load Group # SVE contiguous load (scalar plus scalar) @@ -XXX,XX +XXX,XX @@ PRF_rr 1000010 -- 00 rm:5 110 --- ----- 0 ---- ### SVE Memory 64-bit Gather Group +# SVE 64-bit gather load (scalar plus 32-bit unpacked unscaled offsets) +# SVE 64-bit gather load (scalar plus 32-bit unpacked scaled offsets) +LD1_zprz 1100010 00 .0 ..... 0.. ... ..... ..... \ + @rprr_g_load_xs_u esz=3 msz=0 scale=0 +LD1_zprz 1100010 01 .. ..... 0.. ... ..... ..... \ + @rprr_g_load_xs_u_sc esz=3 msz=1 +LD1_zprz 1100010 10 .. ..... 0.. ... ..... ..... \ + @rprr_g_load_xs_u_sc esz=3 msz=2 +LD1_zprz 1100010 11 .. ..... 01. ... ..... ..... \ + @rprr_g_load_xs_sc esz=3 msz=3 u=1 + +# SVE 64-bit gather load (scalar plus 64-bit unscaled offsets) +# SVE 64-bit gather load (scalar plus 64-bit scaled offsets) +LD1_zprz 1100010 00 10 ..... 1.. ... ..... ..... \ + @rprr_g_load_u esz=3 msz=0 scale=0 +LD1_zprz 1100010 01 1. ..... 1.. ... ..... ..... \ + @rprr_g_load_u_sc esz=3 msz=1 +LD1_zprz 1100010 10 1. ..... 1.. ... ..... ..... \ + @rprr_g_load_u_sc esz=3 msz=2 +LD1_zprz 1100010 11 1. ..... 11. ... ..... ..... \ + @rprr_g_load_sc esz=3 msz=3 u=1 + +# SVE 64-bit gather load (vector plus immediate) +LD1_zpiz 1100010 .. 01 ..... 1.. ... ..... ..... \ + @rpri_g_load esz=3 + # SVE 64-bit gather prefetch (scalar plus 64-bit scaled offsets) PRF 1100010 00 11 ----- 1-- --- ----- 0 ---- -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-15-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 67 +++++++++++++++++++++++++++++ target/arm/sve_helper.c | 88 ++++++++++++++++++++++++++++++++++++++ target/arm/translate-sve.c | 40 ++++++++++++++++- 3 files changed, 193 insertions(+), 2 deletions(-) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_6(sve_ldhds_zd, TCG_CALL_NO_WG, DEF_HELPER_FLAGS_6(sve_ldsds_zd, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffbsu_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffhsu_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffssu_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffbss_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffhss_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) + +DEF_HELPER_FLAGS_6(sve_ldffbsu_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffhsu_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffssu_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffbss_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffhss_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) + +DEF_HELPER_FLAGS_6(sve_ldffbdu_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffhdu_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffsdu_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffddu_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffbds_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffhds_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffsds_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) + +DEF_HELPER_FLAGS_6(sve_ldffbdu_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffhdu_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffsdu_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffddu_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffbds_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffhds_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffsds_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) + +DEF_HELPER_FLAGS_6(sve_ldffbdu_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffhdu_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffsdu_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffddu_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffbds_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffhds_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffsds_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) + DEF_HELPER_FLAGS_6(sve_stbs_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_sths_zsu, TCG_CALL_NO_WG, diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ DO_LD1_ZPZ_D(sve_ldbds_zd, uint64_t, int8_t, cpu_ldub_data_ra) DO_LD1_ZPZ_D(sve_ldhds_zd, uint64_t, int16_t, cpu_lduw_data_ra) DO_LD1_ZPZ_D(sve_ldsds_zd, uint64_t, int32_t, cpu_ldl_data_ra) +/* First fault loads with a vector index. */ + +#ifdef CONFIG_USER_ONLY + +#define DO_LDFF1_ZPZ(NAME, TYPEE, TYPEI, TYPEM, FN, H) \ +void HELPER(NAME)(CPUARMState *env, void *vd, void *vg, void *vm, \ + target_ulong base, uint32_t desc) \ +{ \ + intptr_t i, oprsz = simd_oprsz(desc); \ + unsigned scale = simd_data(desc); \ + uintptr_t ra = GETPC(); \ + bool first = true; \ + mmap_lock(); \ + for (i = 0; i < oprsz; i++) { \ + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ + do { \ + TYPEM m = 0; \ + if (pg & 1) { \ + target_ulong off = *(TYPEI *)(vm + H(i)); \ + target_ulong addr = base + (off << scale); \ + if (!first && \ + page_check_range(addr, sizeof(TYPEM), PAGE_READ)) { \ + record_fault(env, i, oprsz); \ + goto exit; \ + } \ + m = FN(env, addr, ra); \ + first = false; \ + } \ + *(TYPEE *)(vd + H(i)) = m; \ + i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ + } while (i & 15); \ + } \ + exit: \ + mmap_unlock(); \ +} + +#else + +#define DO_LDFF1_ZPZ(NAME, TYPEE, TYPEI, TYPEM, FN, H) \ +void HELPER(NAME)(CPUARMState *env, void *vd, void *vg, void *vm, \ + target_ulong base, uint32_t desc) \ +{ \ + g_assert_not_reached(); \ +} + +#endif + +#define DO_LDFF1_ZPZ_S(NAME, TYPEI, TYPEM, FN) \ + DO_LDFF1_ZPZ(NAME, uint32_t, TYPEI, TYPEM, FN, H1_4) +#define DO_LDFF1_ZPZ_D(NAME, TYPEI, TYPEM, FN) \ + DO_LDFF1_ZPZ(NAME, uint64_t, TYPEI, TYPEM, FN, ) + +DO_LDFF1_ZPZ_S(sve_ldffbsu_zsu, uint32_t, uint8_t, cpu_ldub_data_ra) +DO_LDFF1_ZPZ_S(sve_ldffhsu_zsu, uint32_t, uint16_t, cpu_lduw_data_ra) +DO_LDFF1_ZPZ_S(sve_ldffssu_zsu, uint32_t, uint32_t, cpu_ldl_data_ra) +DO_LDFF1_ZPZ_S(sve_ldffbss_zsu, uint32_t, int8_t, cpu_ldub_data_ra) +DO_LDFF1_ZPZ_S(sve_ldffhss_zsu, uint32_t, int16_t, cpu_lduw_data_ra) + +DO_LDFF1_ZPZ_S(sve_ldffbsu_zss, int32_t, uint8_t, cpu_ldub_data_ra) +DO_LDFF1_ZPZ_S(sve_ldffhsu_zss, int32_t, uint16_t, cpu_lduw_data_ra) +DO_LDFF1_ZPZ_S(sve_ldffssu_zss, int32_t, uint32_t, cpu_ldl_data_ra) +DO_LDFF1_ZPZ_S(sve_ldffbss_zss, int32_t, int8_t, cpu_ldub_data_ra) +DO_LDFF1_ZPZ_S(sve_ldffhss_zss, int32_t, int16_t, cpu_lduw_data_ra) + +DO_LDFF1_ZPZ_D(sve_ldffbdu_zsu, uint32_t, uint8_t, cpu_ldub_data_ra) +DO_LDFF1_ZPZ_D(sve_ldffhdu_zsu, uint32_t, uint16_t, cpu_lduw_data_ra) +DO_LDFF1_ZPZ_D(sve_ldffsdu_zsu, uint32_t, uint32_t, cpu_ldl_data_ra) +DO_LDFF1_ZPZ_D(sve_ldffddu_zsu, uint32_t, uint64_t, cpu_ldq_data_ra) +DO_LDFF1_ZPZ_D(sve_ldffbds_zsu, uint32_t, int8_t, cpu_ldub_data_ra) +DO_LDFF1_ZPZ_D(sve_ldffhds_zsu, uint32_t, int16_t, cpu_lduw_data_ra) +DO_LDFF1_ZPZ_D(sve_ldffsds_zsu, uint32_t, int32_t, cpu_ldl_data_ra) + +DO_LDFF1_ZPZ_D(sve_ldffbdu_zss, int32_t, uint8_t, cpu_ldub_data_ra) +DO_LDFF1_ZPZ_D(sve_ldffhdu_zss, int32_t, uint16_t, cpu_lduw_data_ra) +DO_LDFF1_ZPZ_D(sve_ldffsdu_zss, int32_t, uint32_t, cpu_ldl_data_ra) +DO_LDFF1_ZPZ_D(sve_ldffddu_zss, int32_t, uint64_t, cpu_ldq_data_ra) +DO_LDFF1_ZPZ_D(sve_ldffbds_zss, int32_t, int8_t, cpu_ldub_data_ra) +DO_LDFF1_ZPZ_D(sve_ldffhds_zss, int32_t, int16_t, cpu_lduw_data_ra) +DO_LDFF1_ZPZ_D(sve_ldffsds_zss, int32_t, int32_t, cpu_ldl_data_ra) + +DO_LDFF1_ZPZ_D(sve_ldffbdu_zd, uint64_t, uint8_t, cpu_ldub_data_ra) +DO_LDFF1_ZPZ_D(sve_ldffhdu_zd, uint64_t, uint16_t, cpu_lduw_data_ra) +DO_LDFF1_ZPZ_D(sve_ldffsdu_zd, uint64_t, uint32_t, cpu_ldl_data_ra) +DO_LDFF1_ZPZ_D(sve_ldffddu_zd, uint64_t, uint64_t, cpu_ldq_data_ra) +DO_LDFF1_ZPZ_D(sve_ldffbds_zd, uint64_t, int8_t, cpu_ldub_data_ra) +DO_LDFF1_ZPZ_D(sve_ldffhds_zd, uint64_t, int16_t, cpu_lduw_data_ra) +DO_LDFF1_ZPZ_D(sve_ldffsds_zd, uint64_t, int32_t, cpu_ldl_data_ra) + /* Stores with a vector index. */ #define DO_ST1_ZPZ_S(NAME, TYPEI, FN) \ diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ static gen_helper_gvec_mem_scatter * const gather_load_fn32[2][2][2][3] = { { gen_helper_sve_ldbsu_zss, gen_helper_sve_ldhsu_zss, gen_helper_sve_ldssu_zss, } } }, - /* TODO fill in first-fault handlers */ + + { { { gen_helper_sve_ldffbss_zsu, + gen_helper_sve_ldffhss_zsu, + NULL, }, + { gen_helper_sve_ldffbsu_zsu, + gen_helper_sve_ldffhsu_zsu, + gen_helper_sve_ldffssu_zsu, } }, + { { gen_helper_sve_ldffbss_zss, + gen_helper_sve_ldffhss_zss, + NULL, }, + { gen_helper_sve_ldffbsu_zss, + gen_helper_sve_ldffhsu_zss, + gen_helper_sve_ldffssu_zss, } } } }; /* Note that we overload xs=2 to indicate 64-bit offset. */ @@ -XXX,XX +XXX,XX @@ static gen_helper_gvec_mem_scatter * const gather_load_fn64[2][3][2][4] = { gen_helper_sve_ldhdu_zd, gen_helper_sve_ldsdu_zd, gen_helper_sve_ldddu_zd, } } }, - /* TODO fill in first-fault handlers */ + + { { { gen_helper_sve_ldffbds_zsu, + gen_helper_sve_ldffhds_zsu, + gen_helper_sve_ldffsds_zsu, + NULL, }, + { gen_helper_sve_ldffbdu_zsu, + gen_helper_sve_ldffhdu_zsu, + gen_helper_sve_ldffsdu_zsu, + gen_helper_sve_ldffddu_zsu, } }, + { { gen_helper_sve_ldffbds_zss, + gen_helper_sve_ldffhds_zss, + gen_helper_sve_ldffsds_zss, + NULL, }, + { gen_helper_sve_ldffbdu_zss, + gen_helper_sve_ldffhdu_zss, + gen_helper_sve_ldffsdu_zss, + gen_helper_sve_ldffddu_zss, } }, + { { gen_helper_sve_ldffbds_zd, + gen_helper_sve_ldffhds_zd, + gen_helper_sve_ldffsds_zd, + NULL, }, + { gen_helper_sve_ldffbdu_zd, + gen_helper_sve_ldffhdu_zd, + gen_helper_sve_ldffsdu_zd, + gen_helper_sve_ldffddu_zd, } } } }; static bool trans_LD1_zprz(DisasContext *s, arg_LD1_zprz *a, uint32_t insn) -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-16-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/translate-sve.c | 85 ++++++++++++++++++++++++++------------ target/arm/sve.decode | 11 +++++ 2 files changed, 70 insertions(+), 26 deletions(-) diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ static bool trans_LD1_zpiz(DisasContext *s, arg_LD1_zpiz *a, uint32_t insn) return true; } +/* Indexed by [xs][msz]. */ +static gen_helper_gvec_mem_scatter * const scatter_store_fn32[2][3] = { + { gen_helper_sve_stbs_zsu, + gen_helper_sve_sths_zsu, + gen_helper_sve_stss_zsu, }, + { gen_helper_sve_stbs_zss, + gen_helper_sve_sths_zss, + gen_helper_sve_stss_zss, }, +}; + +/* Note that we overload xs=2 to indicate 64-bit offset. */ +static gen_helper_gvec_mem_scatter * const scatter_store_fn64[3][4] = { + { gen_helper_sve_stbd_zsu, + gen_helper_sve_sthd_zsu, + gen_helper_sve_stsd_zsu, + gen_helper_sve_stdd_zsu, }, + { gen_helper_sve_stbd_zss, + gen_helper_sve_sthd_zss, + gen_helper_sve_stsd_zss, + gen_helper_sve_stdd_zss, }, + { gen_helper_sve_stbd_zd, + gen_helper_sve_sthd_zd, + gen_helper_sve_stsd_zd, + gen_helper_sve_stdd_zd, }, +}; + static bool trans_ST1_zprz(DisasContext *s, arg_ST1_zprz *a, uint32_t insn) { - /* Indexed by [xs][msz]. */ - static gen_helper_gvec_mem_scatter * const fn32[2][3] = { - { gen_helper_sve_stbs_zsu, - gen_helper_sve_sths_zsu, - gen_helper_sve_stss_zsu, }, - { gen_helper_sve_stbs_zss, - gen_helper_sve_sths_zss, - gen_helper_sve_stss_zss, }, - }; - /* Note that we overload xs=2 to indicate 64-bit offset. */ - static gen_helper_gvec_mem_scatter * const fn64[3][4] = { - { gen_helper_sve_stbd_zsu, - gen_helper_sve_sthd_zsu, - gen_helper_sve_stsd_zsu, - gen_helper_sve_stdd_zsu, }, - { gen_helper_sve_stbd_zss, - gen_helper_sve_sthd_zss, - gen_helper_sve_stsd_zss, - gen_helper_sve_stdd_zss, }, - { gen_helper_sve_stbd_zd, - gen_helper_sve_sthd_zd, - gen_helper_sve_stsd_zd, - gen_helper_sve_stdd_zd, }, - }; gen_helper_gvec_mem_scatter *fn; if (a->esz < a->msz || (a->msz == 0 && a->scale)) { @@ -XXX,XX +XXX,XX @@ static bool trans_ST1_zprz(DisasContext *s, arg_ST1_zprz *a, uint32_t insn) } switch (a->esz) { case MO_32: - fn = fn32[a->xs][a->msz]; + fn = scatter_store_fn32[a->xs][a->msz]; break; case MO_64: - fn = fn64[a->xs][a->msz]; + fn = scatter_store_fn64[a->xs][a->msz]; break; default: g_assert_not_reached(); @@ -XXX,XX +XXX,XX @@ static bool trans_ST1_zprz(DisasContext *s, arg_ST1_zprz *a, uint32_t insn) return true; } +static bool trans_ST1_zpiz(DisasContext *s, arg_ST1_zpiz *a, uint32_t insn) +{ + gen_helper_gvec_mem_scatter *fn = NULL; + TCGv_i64 imm; + + if (a->esz < a->msz) { + return false; + } + if (!sve_access_check(s)) { + return true; + } + + switch (a->esz) { + case MO_32: + fn = scatter_store_fn32[0][a->msz]; + break; + case MO_64: + fn = scatter_store_fn64[2][a->msz]; + break; + } + assert(fn != NULL); + + /* Treat ST1_zpiz (zn[x] + imm) the same way as ST1_zprz (rn + zm[x]) + * by loading the immediate into the scalar parameter. + */ + imm = tcg_const_i64(a->imm << a->msz); + do_mem_zpz(s, a->rd, a->pg, a->rn, 0, imm, fn); + tcg_temp_free_i64(imm); + return true; +} + /* * Prefetches */ diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ &rprr_gather_load rd pg rn rm esz msz u ff xs scale &rpri_gather_load rd pg rn imm esz msz u ff &rprr_scatter_store rd pg rn rm esz msz xs scale +&rpri_scatter_store rd pg rn imm esz msz ########################################################################### # Named instruction formats. These are generally used to @@ -XXX,XX +XXX,XX @@ &rprr_store nreg=0 @rprr_scatter_store ....... msz:2 .. rm:5 ... pg:3 rn:5 rd:5 \ &rprr_scatter_store +@rpri_scatter_store ....... msz:2 .. imm:5 ... pg:3 rn:5 rd:5 \ + &rpri_scatter_store ########################################################################### # Instruction patterns. Grouped according to the SVE encodingindex.xhtml. @@ -XXX,XX +XXX,XX @@ ST1_zprz 1110010 .. 01 ..... 101 ... ..... ..... \ ST1_zprz 1110010 .. 00 ..... 101 ... ..... ..... \ @rprr_scatter_store xs=2 esz=3 scale=0 +# SVE 64-bit scatter store (vector plus immediate) +ST1_zpiz 1110010 .. 10 ..... 101 ... ..... ..... \ + @rpri_scatter_store esz=3 + +# SVE 32-bit scatter store (vector plus immediate) +ST1_zpiz 1110010 .. 11 ..... 101 ... ..... ..... \ + @rpri_scatter_store esz=2 + # SVE 64-bit scatter store (scalar plus unpacked 32-bit scaled offset) # Require msz > 0 ST1_zprz 1110010 .. 01 ..... 100 ... ..... ..... \ -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-17-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 49 ++++++++++++++++++++++++++++++ target/arm/sve_helper.c | 62 ++++++++++++++++++++++++++++++++++++++ target/arm/translate-sve.c | 40 ++++++++++++++++++++++++ target/arm/sve.decode | 11 +++++++ 4 files changed, 162 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_5(sve_ucvt_ds, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_5(sve_ucvt_dd, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fcmge_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fcmge_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fcmge_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fcmgt_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fcmgt_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fcmgt_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fcmeq_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fcmeq_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fcmeq_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fcmne_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fcmne_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fcmne_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fcmuo_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fcmuo_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fcmuo_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_facge_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_facge_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_facge_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_facgt_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_facgt_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_facgt_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + DEF_HELPER_FLAGS_3(sve_fmla_zpzzz_h, TCG_CALL_NO_RWG, void, env, ptr, i32) DEF_HELPER_FLAGS_3(sve_fmla_zpzzz_s, TCG_CALL_NO_RWG, void, env, ptr, i32) DEF_HELPER_FLAGS_3(sve_fmla_zpzzz_d, TCG_CALL_NO_RWG, void, env, ptr, i32) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ void HELPER(sve_fnmls_zpzzz_d)(CPUARMState *env, void *vg, uint32_t desc) do_fmla_zpzzz_d(env, vg, desc, 0, INT64_MIN); } +/* Two operand floating-point comparison controlled by a predicate. + * Unlike the integer version, we are not allowed to optimistically + * compare operands, since the comparison may have side effects wrt + * the FPSR. + */ +#define DO_FPCMP_PPZZ(NAME, TYPE, H, OP) \ +void HELPER(NAME)(void *vd, void *vn, void *vm, void *vg, \ + void *status, uint32_t desc) \ +{ \ + intptr_t i = simd_oprsz(desc), j = (i - 1) >> 6; \ + uint64_t *d = vd, *g = vg; \ + do { \ + uint64_t out = 0, pg = g[j]; \ + do { \ + i -= sizeof(TYPE), out <<= sizeof(TYPE); \ + if (likely((pg >> (i & 63)) & 1)) { \ + TYPE nn = *(TYPE *)(vn + H(i)); \ + TYPE mm = *(TYPE *)(vm + H(i)); \ + out |= OP(TYPE, nn, mm, status); \ + } \ + } while (i & 63); \ + d[j--] = out; \ + } while (i > 0); \ +} + +#define DO_FPCMP_PPZZ_H(NAME, OP) \ + DO_FPCMP_PPZZ(NAME##_h, float16, H1_2, OP) +#define DO_FPCMP_PPZZ_S(NAME, OP) \ + DO_FPCMP_PPZZ(NAME##_s, float32, H1_4, OP) +#define DO_FPCMP_PPZZ_D(NAME, OP) \ + DO_FPCMP_PPZZ(NAME##_d, float64, , OP) + +#define DO_FPCMP_PPZZ_ALL(NAME, OP) \ + DO_FPCMP_PPZZ_H(NAME, OP) \ + DO_FPCMP_PPZZ_S(NAME, OP) \ + DO_FPCMP_PPZZ_D(NAME, OP) + +#define DO_FCMGE(TYPE, X, Y, ST) TYPE##_compare(Y, X, ST) <= 0 +#define DO_FCMGT(TYPE, X, Y, ST) TYPE##_compare(Y, X, ST) < 0 +#define DO_FCMEQ(TYPE, X, Y, ST) TYPE##_compare_quiet(X, Y, ST) == 0 +#define DO_FCMNE(TYPE, X, Y, ST) TYPE##_compare_quiet(X, Y, ST) != 0 +#define DO_FCMUO(TYPE, X, Y, ST) \ + TYPE##_compare_quiet(X, Y, ST) == float_relation_unordered +#define DO_FACGE(TYPE, X, Y, ST) \ + TYPE##_compare(TYPE##_abs(Y), TYPE##_abs(X), ST) <= 0 +#define DO_FACGT(TYPE, X, Y, ST) \ + TYPE##_compare(TYPE##_abs(Y), TYPE##_abs(X), ST) < 0 + +DO_FPCMP_PPZZ_ALL(sve_fcmge, DO_FCMGE) +DO_FPCMP_PPZZ_ALL(sve_fcmgt, DO_FCMGT) +DO_FPCMP_PPZZ_ALL(sve_fcmeq, DO_FCMEQ) +DO_FPCMP_PPZZ_ALL(sve_fcmne, DO_FCMNE) +DO_FPCMP_PPZZ_ALL(sve_fcmuo, DO_FCMUO) +DO_FPCMP_PPZZ_ALL(sve_facge, DO_FACGE) +DO_FPCMP_PPZZ_ALL(sve_facgt, DO_FACGT) + +#undef DO_FPCMP_PPZZ_ALL +#undef DO_FPCMP_PPZZ_D +#undef DO_FPCMP_PPZZ_S +#undef DO_FPCMP_PPZZ_H +#undef DO_FPCMP_PPZZ + /* * Load contiguous data, protected by a governing predicate. */ diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ DO_FP3(FMULX, fmulx) #undef DO_FP3 +static bool do_fp_cmp(DisasContext *s, arg_rprr_esz *a, + gen_helper_gvec_4_ptr *fn) +{ + if (fn == NULL) { + return false; + } + if (sve_access_check(s)) { + unsigned vsz = vec_full_reg_size(s); + TCGv_ptr status = get_fpstatus_ptr(a->esz == MO_16); + tcg_gen_gvec_4_ptr(pred_full_reg_offset(s, a->rd), + vec_full_reg_offset(s, a->rn), + vec_full_reg_offset(s, a->rm), + pred_full_reg_offset(s, a->pg), + status, vsz, vsz, 0, fn); + tcg_temp_free_ptr(status); + } + return true; +} + +#define DO_FPCMP(NAME, name) \ +static bool trans_##NAME##_ppzz(DisasContext *s, arg_rprr_esz *a, \ + uint32_t insn) \ +{ \ + static gen_helper_gvec_4_ptr * const fns[4] = { \ + NULL, gen_helper_sve_##name##_h, \ + gen_helper_sve_##name##_s, gen_helper_sve_##name##_d \ + }; \ + return do_fp_cmp(s, a, fns[a->esz]); \ +} + +DO_FPCMP(FCMGE, fcmge) +DO_FPCMP(FCMGT, fcmgt) +DO_FPCMP(FCMEQ, fcmeq) +DO_FPCMP(FCMNE, fcmne) +DO_FPCMP(FCMUO, fcmuo) +DO_FPCMP(FACGE, facge) +DO_FPCMP(FACGT, facgt) + +#undef DO_FPCMP + typedef void gen_helper_sve_fmla(TCGv_env, TCGv_ptr, TCGv_i32); static bool do_fmla(DisasContext *s, arg_rprrr_esz *a, gen_helper_sve_fmla *fn) diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ UXTH 00000100 .. 010 011 101 ... ..... ..... @rd_pg_rn SXTW 00000100 .. 010 100 101 ... ..... ..... @rd_pg_rn UXTW 00000100 .. 010 101 101 ... ..... ..... @rd_pg_rn +### SVE Floating Point Compare - Vectors Group + +# SVE floating-point compare vectors +FCMGE_ppzz 01100101 .. 0 ..... 010 ... ..... 0 .... @pd_pg_rn_rm +FCMGT_ppzz 01100101 .. 0 ..... 010 ... ..... 1 .... @pd_pg_rn_rm +FCMEQ_ppzz 01100101 .. 0 ..... 011 ... ..... 0 .... @pd_pg_rn_rm +FCMNE_ppzz 01100101 .. 0 ..... 011 ... ..... 1 .... @pd_pg_rn_rm +FCMUO_ppzz 01100101 .. 0 ..... 110 ... ..... 0 .... @pd_pg_rn_rm +FACGE_ppzz 01100101 .. 0 ..... 110 ... ..... 1 .... @pd_pg_rn_rm +FACGT_ppzz 01100101 .. 0 ..... 111 ... ..... 1 .... @pd_pg_rn_rm + ### SVE Integer Multiply-Add Group # SVE integer multiply-add writing addend (predicated) -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-18-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 56 ++++++++++++++++++++++++++++ target/arm/sve_helper.c | 69 +++++++++++++++++++++++++++++++++++ target/arm/translate-sve.c | 75 ++++++++++++++++++++++++++++++++++++++ target/arm/sve.decode | 14 +++++++ 4 files changed, 214 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_6(sve_fmulx_s, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_6(sve_fmulx_d, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fadds_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fadds_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fadds_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fsubs_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fsubs_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fsubs_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fmuls_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fmuls_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fmuls_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fsubrs_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fsubrs_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fsubrs_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fmaxnms_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fmaxnms_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fmaxnms_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fminnms_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fminnms_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fminnms_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fmaxs_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fmaxs_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fmaxs_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) + +DEF_HELPER_FLAGS_6(sve_fmins_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fmins_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fmins_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, i64, ptr, i32) + DEF_HELPER_FLAGS_5(sve_scvt_hh, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_5(sve_scvt_sh, TCG_CALL_NO_RWG, diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ DO_ZPZZ_FP(sve_fmulx_d, uint64_t, , helper_vfp_mulxd) #undef DO_ZPZZ_FP +/* Three-operand expander, with one scalar operand, controlled by + * a predicate, with the extra float_status parameter. + */ +#define DO_ZPZS_FP(NAME, TYPE, H, OP) \ +void HELPER(NAME)(void *vd, void *vn, void *vg, uint64_t scalar, \ + void *status, uint32_t desc) \ +{ \ + intptr_t i = simd_oprsz(desc); \ + uint64_t *g = vg; \ + TYPE mm = scalar; \ + do { \ + uint64_t pg = g[(i - 1) >> 6]; \ + do { \ + i -= sizeof(TYPE); \ + if (likely((pg >> (i & 63)) & 1)) { \ + TYPE nn = *(TYPE *)(vn + H(i)); \ + *(TYPE *)(vd + H(i)) = OP(nn, mm, status); \ + } \ + } while (i & 63); \ + } while (i != 0); \ +} + +DO_ZPZS_FP(sve_fadds_h, float16, H1_2, float16_add) +DO_ZPZS_FP(sve_fadds_s, float32, H1_4, float32_add) +DO_ZPZS_FP(sve_fadds_d, float64, , float64_add) + +DO_ZPZS_FP(sve_fsubs_h, float16, H1_2, float16_sub) +DO_ZPZS_FP(sve_fsubs_s, float32, H1_4, float32_sub) +DO_ZPZS_FP(sve_fsubs_d, float64, , float64_sub) + +DO_ZPZS_FP(sve_fmuls_h, float16, H1_2, float16_mul) +DO_ZPZS_FP(sve_fmuls_s, float32, H1_4, float32_mul) +DO_ZPZS_FP(sve_fmuls_d, float64, , float64_mul) + +static inline float16 subr_h(float16 a, float16 b, float_status *s) +{ + return float16_sub(b, a, s); +} + +static inline float32 subr_s(float32 a, float32 b, float_status *s) +{ + return float32_sub(b, a, s); +} + +static inline float64 subr_d(float64 a, float64 b, float_status *s) +{ + return float64_sub(b, a, s); +} + +DO_ZPZS_FP(sve_fsubrs_h, float16, H1_2, subr_h) +DO_ZPZS_FP(sve_fsubrs_s, float32, H1_4, subr_s) +DO_ZPZS_FP(sve_fsubrs_d, float64, , subr_d) + +DO_ZPZS_FP(sve_fmaxnms_h, float16, H1_2, float16_maxnum) +DO_ZPZS_FP(sve_fmaxnms_s, float32, H1_4, float32_maxnum) +DO_ZPZS_FP(sve_fmaxnms_d, float64, , float64_maxnum) + +DO_ZPZS_FP(sve_fminnms_h, float16, H1_2, float16_minnum) +DO_ZPZS_FP(sve_fminnms_s, float32, H1_4, float32_minnum) +DO_ZPZS_FP(sve_fminnms_d, float64, , float64_minnum) + +DO_ZPZS_FP(sve_fmaxs_h, float16, H1_2, float16_max) +DO_ZPZS_FP(sve_fmaxs_s, float32, H1_4, float32_max) +DO_ZPZS_FP(sve_fmaxs_d, float64, , float64_max) + +DO_ZPZS_FP(sve_fmins_h, float16, H1_2, float16_min) +DO_ZPZS_FP(sve_fmins_s, float32, H1_4, float32_min) +DO_ZPZS_FP(sve_fmins_d, float64, , float64_min) + /* Fully general two-operand expander, controlled by a predicate, * With the extra float_status parameter. */ diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ #include "exec/log.h" #include "trace-tcg.h" #include "translate-a64.h" +#include "fpu/softfloat.h" typedef void GVecGen2sFn(unsigned, uint32_t, uint32_t, @@ -XXX,XX +XXX,XX @@ DO_FP3(FMULX, fmulx) #undef DO_FP3 +typedef void gen_helper_sve_fp2scalar(TCGv_ptr, TCGv_ptr, TCGv_ptr, + TCGv_i64, TCGv_ptr, TCGv_i32); + +static void do_fp_scalar(DisasContext *s, int zd, int zn, int pg, bool is_fp16, + TCGv_i64 scalar, gen_helper_sve_fp2scalar *fn) +{ + unsigned vsz = vec_full_reg_size(s); + TCGv_ptr t_zd, t_zn, t_pg, status; + TCGv_i32 desc; + + t_zd = tcg_temp_new_ptr(); + t_zn = tcg_temp_new_ptr(); + t_pg = tcg_temp_new_ptr(); + tcg_gen_addi_ptr(t_zd, cpu_env, vec_full_reg_offset(s, zd)); + tcg_gen_addi_ptr(t_zn, cpu_env, vec_full_reg_offset(s, zn)); + tcg_gen_addi_ptr(t_pg, cpu_env, pred_full_reg_offset(s, pg)); + + status = get_fpstatus_ptr(is_fp16); + desc = tcg_const_i32(simd_desc(vsz, vsz, 0)); + fn(t_zd, t_zn, t_pg, scalar, status, desc); + + tcg_temp_free_i32(desc); + tcg_temp_free_ptr(status); + tcg_temp_free_ptr(t_pg); + tcg_temp_free_ptr(t_zn); + tcg_temp_free_ptr(t_zd); +} + +static void do_fp_imm(DisasContext *s, arg_rpri_esz *a, uint64_t imm, + gen_helper_sve_fp2scalar *fn) +{ + TCGv_i64 temp = tcg_const_i64(imm); + do_fp_scalar(s, a->rd, a->rn, a->pg, a->esz == MO_16, temp, fn); + tcg_temp_free_i64(temp); +} + +#define DO_FP_IMM(NAME, name, const0, const1) \ +static bool trans_##NAME##_zpzi(DisasContext *s, arg_rpri_esz *a, \ + uint32_t insn) \ +{ \ + static gen_helper_sve_fp2scalar * const fns[3] = { \ + gen_helper_sve_##name##_h, \ + gen_helper_sve_##name##_s, \ + gen_helper_sve_##name##_d \ + }; \ + static uint64_t const val[3][2] = { \ + { float16_##const0, float16_##const1 }, \ + { float32_##const0, float32_##const1 }, \ + { float64_##const0, float64_##const1 }, \ + }; \ + if (a->esz == 0) { \ + return false; \ + } \ + if (sve_access_check(s)) { \ + do_fp_imm(s, a, val[a->esz - 1][a->imm], fns[a->esz - 1]); \ + } \ + return true; \ +} + +#define float16_two make_float16(0x4000) +#define float32_two make_float32(0x40000000) +#define float64_two make_float64(0x4000000000000000ULL) + +DO_FP_IMM(FADD, fadds, half, one) +DO_FP_IMM(FSUB, fsubs, half, one) +DO_FP_IMM(FMUL, fmuls, half, two) +DO_FP_IMM(FSUBR, fsubrs, half, one) +DO_FP_IMM(FMAXNM, fmaxnms, zero, one) +DO_FP_IMM(FMINNM, fminnms, zero, one) +DO_FP_IMM(FMAX, fmaxs, zero, one) +DO_FP_IMM(FMIN, fmins, zero, one) + +#undef DO_FP_IMM + static bool do_fp_cmp(DisasContext *s, arg_rprr_esz *a, gen_helper_gvec_4_ptr *fn) { diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ @rdn_pg4 ........ esz:2 .. pg:4 ... ........ rd:5 \ &rpri_esz rn=%reg_movprfx +# Two register operand, one one-bit floating-point operand. +@rdn_i1 ........ esz:2 ......... pg:3 .... imm:1 rd:5 \ + &rpri_esz rn=%reg_movprfx + # Two register operand, one encoded bitmask. @rdn_dbm ........ .. .... dbm:13 rd:5 \ &rr_dbm rn=%reg_movprfx @@ -XXX,XX +XXX,XX @@ FMULX 01100101 .. 00 1010 100 ... ..... ..... @rdn_pg_rm FDIV 01100101 .. 00 1100 100 ... ..... ..... @rdm_pg_rn # FDIVR FDIV 01100101 .. 00 1101 100 ... ..... ..... @rdn_pg_rm +# SVE floating-point arithmetic with immediate (predicated) +FADD_zpzi 01100101 .. 011 000 100 ... 0000 . ..... @rdn_i1 +FSUB_zpzi 01100101 .. 011 001 100 ... 0000 . ..... @rdn_i1 +FMUL_zpzi 01100101 .. 011 010 100 ... 0000 . ..... @rdn_i1 +FSUBR_zpzi 01100101 .. 011 011 100 ... 0000 . ..... @rdn_i1 +FMAXNM_zpzi 01100101 .. 011 100 100 ... 0000 . ..... @rdn_i1 +FMINNM_zpzi 01100101 .. 011 101 100 ... 0000 . ..... @rdn_i1 +FMAX_zpzi 01100101 .. 011 110 100 ... 0000 . ..... @rdn_i1 +FMIN_zpzi 01100101 .. 011 111 100 ... 0000 . ..... @rdn_i1 + ### SVE FP Multiply-Add Group # SVE floating-point multiply-accumulate writing addend -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-19-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper.h | 14 +++++++++++ target/arm/translate-sve.c | 50 ++++++++++++++++++++++++++++++++++++++ target/arm/vec_helper.c | 48 ++++++++++++++++++++++++++++++++++++ target/arm/sve.decode | 19 +++++++++++++++ 4 files changed, 131 insertions(+) diff --git a/target/arm/helper.h b/target/arm/helper.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper.h +++ b/target/arm/helper.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_5(gvec_ftsmul_s, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_5(gvec_ftsmul_d, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(gvec_fmul_idx_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(gvec_fmul_idx_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(gvec_fmul_idx_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_6(gvec_fmla_idx_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(gvec_fmla_idx_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(gvec_fmla_idx_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + #ifdef TARGET_AARCH64 #include "helper-a64.h" #include "helper-sve.h" diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ DO_ZZI(UMIN, umin) #undef DO_ZZI +/* + *** SVE Floating Point Multiply-Add Indexed Group + */ + +static bool trans_FMLA_zzxz(DisasContext *s, arg_FMLA_zzxz *a, uint32_t insn) +{ + static gen_helper_gvec_4_ptr * const fns[3] = { + gen_helper_gvec_fmla_idx_h, + gen_helper_gvec_fmla_idx_s, + gen_helper_gvec_fmla_idx_d, + }; + + if (sve_access_check(s)) { + unsigned vsz = vec_full_reg_size(s); + TCGv_ptr status = get_fpstatus_ptr(a->esz == MO_16); + tcg_gen_gvec_4_ptr(vec_full_reg_offset(s, a->rd), + vec_full_reg_offset(s, a->rn), + vec_full_reg_offset(s, a->rm), + vec_full_reg_offset(s, a->ra), + status, vsz, vsz, (a->index << 1) | a->sub, + fns[a->esz - 1]); + tcg_temp_free_ptr(status); + } + return true; +} + +/* + *** SVE Floating Point Multiply Indexed Group + */ + +static bool trans_FMUL_zzx(DisasContext *s, arg_FMUL_zzx *a, uint32_t insn) +{ + static gen_helper_gvec_3_ptr * const fns[3] = { + gen_helper_gvec_fmul_idx_h, + gen_helper_gvec_fmul_idx_s, + gen_helper_gvec_fmul_idx_d, + }; + + if (sve_access_check(s)) { + unsigned vsz = vec_full_reg_size(s); + TCGv_ptr status = get_fpstatus_ptr(a->esz == MO_16); + tcg_gen_gvec_3_ptr(vec_full_reg_offset(s, a->rd), + vec_full_reg_offset(s, a->rn), + vec_full_reg_offset(s, a->rm), + status, vsz, vsz, a->index, fns[a->esz - 1]); + tcg_temp_free_ptr(status); + } + return true; +} + /* *** SVE Floating Point Accumulating Reduction Group */ diff --git a/target/arm/vec_helper.c b/target/arm/vec_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/vec_helper.c +++ b/target/arm/vec_helper.c @@ -XXX,XX +XXX,XX @@ DO_3OP(gvec_rsqrts_d, helper_rsqrtsf_f64, float64) #endif #undef DO_3OP + +/* For the indexed ops, SVE applies the index per 128-bit vector segment. + * For AdvSIMD, there is of course only one such vector segment. + */ + +#define DO_MUL_IDX(NAME, TYPE, H) \ +void HELPER(NAME)(void *vd, void *vn, void *vm, void *stat, uint32_t desc) \ +{ \ + intptr_t i, j, oprsz = simd_oprsz(desc), segment = 16 / sizeof(TYPE); \ + intptr_t idx = simd_data(desc); \ + TYPE *d = vd, *n = vn, *m = vm; \ + for (i = 0; i < oprsz / sizeof(TYPE); i += segment) { \ + TYPE mm = m[H(i + idx)]; \ + for (j = 0; j < segment; j++) { \ + d[i + j] = TYPE##_mul(n[i + j], mm, stat); \ + } \ + } \ +} + +DO_MUL_IDX(gvec_fmul_idx_h, float16, H2) +DO_MUL_IDX(gvec_fmul_idx_s, float32, H4) +DO_MUL_IDX(gvec_fmul_idx_d, float64, ) + +#undef DO_MUL_IDX + +#define DO_FMLA_IDX(NAME, TYPE, H) \ +void HELPER(NAME)(void *vd, void *vn, void *vm, void *va, \ + void *stat, uint32_t desc) \ +{ \ + intptr_t i, j, oprsz = simd_oprsz(desc), segment = 16 / sizeof(TYPE); \ + TYPE op1_neg = extract32(desc, SIMD_DATA_SHIFT, 1); \ + intptr_t idx = desc >> (SIMD_DATA_SHIFT + 1); \ + TYPE *d = vd, *n = vn, *m = vm, *a = va; \ + op1_neg <<= (8 * sizeof(TYPE) - 1); \ + for (i = 0; i < oprsz / sizeof(TYPE); i += segment) { \ + TYPE mm = m[H(i + idx)]; \ + for (j = 0; j < segment; j++) { \ + d[i + j] = TYPE##_muladd(n[i + j] ^ op1_neg, \ + mm, a[i + j], 0, stat); \ + } \ + } \ +} + +DO_FMLA_IDX(gvec_fmla_idx_h, float16, H2) +DO_FMLA_IDX(gvec_fmla_idx_s, float32, H4) +DO_FMLA_IDX(gvec_fmla_idx_d, float64, ) + +#undef DO_FMLA_IDX diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ %imm9_16_10 16:s6 10:3 %size_23 23:2 %dtype_23_13 23:2 13:2 +%index3_22_19 22:1 19:2 # A combination of tsz:imm3 -- extract esize. %tszimm_esz 22:2 5:5 !function=tszimm_esz @@ -XXX,XX +XXX,XX @@ UMIN_zzi 00100101 .. 101 011 110 ........ ..... @rdn_i8u # SVE integer multiply immediate (unpredicated) MUL_zzi 00100101 .. 110 000 110 ........ ..... @rdn_i8s +### SVE FP Multiply-Add Indexed Group + +# SVE floating-point multiply-add (indexed) +FMLA_zzxz 01100100 0.1 .. rm:3 00000 sub:1 rn:5 rd:5 \ + ra=%reg_movprfx index=%index3_22_19 esz=1 +FMLA_zzxz 01100100 101 index:2 rm:3 00000 sub:1 rn:5 rd:5 \ + ra=%reg_movprfx esz=2 +FMLA_zzxz 01100100 111 index:1 rm:4 00000 sub:1 rn:5 rd:5 \ + ra=%reg_movprfx esz=3 + +### SVE FP Multiply Indexed Group + +# SVE floating-point multiply (indexed) +FMUL_zzx 01100100 0.1 .. rm:3 001000 rn:5 rd:5 \ + index=%index3_22_19 esz=1 +FMUL_zzx 01100100 101 index:2 rm:3 001000 rn:5 rd:5 esz=2 +FMUL_zzx 01100100 111 index:1 rm:4 001000 rn:5 rd:5 esz=3 + ### SVE FP Accumulating Reduction Group # SVE floating-point serial reduction (predicated) -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-20-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 35 ++++++++++++++++++++++ target/arm/sve_helper.c | 61 ++++++++++++++++++++++++++++++++++++++ target/arm/translate-sve.c | 57 +++++++++++++++++++++++++++++++++++ target/arm/sve.decode | 8 +++++ 4 files changed, 161 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_5(gvec_rsqrts_s, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_5(gvec_rsqrts_d, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(sve_faddv_h, TCG_CALL_NO_RWG, + i64, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(sve_faddv_s, TCG_CALL_NO_RWG, + i64, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(sve_faddv_d, TCG_CALL_NO_RWG, + i64, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_4(sve_fmaxnmv_h, TCG_CALL_NO_RWG, + i64, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(sve_fmaxnmv_s, TCG_CALL_NO_RWG, + i64, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(sve_fmaxnmv_d, TCG_CALL_NO_RWG, + i64, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_4(sve_fminnmv_h, TCG_CALL_NO_RWG, + i64, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(sve_fminnmv_s, TCG_CALL_NO_RWG, + i64, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(sve_fminnmv_d, TCG_CALL_NO_RWG, + i64, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_4(sve_fmaxv_h, TCG_CALL_NO_RWG, + i64, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(sve_fmaxv_s, TCG_CALL_NO_RWG, + i64, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(sve_fmaxv_d, TCG_CALL_NO_RWG, + i64, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_4(sve_fminv_h, TCG_CALL_NO_RWG, + i64, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(sve_fminv_s, TCG_CALL_NO_RWG, + i64, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(sve_fminv_d, TCG_CALL_NO_RWG, + i64, ptr, ptr, ptr, i32) + DEF_HELPER_FLAGS_5(sve_fadda_h, TCG_CALL_NO_RWG, i64, i64, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_5(sve_fadda_s, TCG_CALL_NO_RWG, diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ uint32_t HELPER(sve_while)(void *vd, uint32_t count, uint32_t pred_desc) return predtest_ones(d, oprsz, esz_mask); } +/* Recursive reduction on a function; + * C.f. the ARM ARM function ReducePredicated. + * + * While it would be possible to write this without the DATA temporary, + * it is much simpler to process the predicate register this way. + * The recursion is bounded to depth 7 (128 fp16 elements), so there's + * little to gain with a more complex non-recursive form. + */ +#define DO_REDUCE(NAME, TYPE, H, FUNC, IDENT) \ +static TYPE NAME##_reduce(TYPE *data, float_status *status, uintptr_t n) \ +{ \ + if (n == 1) { \ + return *data; \ + } else { \ + uintptr_t half = n / 2; \ + TYPE lo = NAME##_reduce(data, status, half); \ + TYPE hi = NAME##_reduce(data + half, status, half); \ + return TYPE##_##FUNC(lo, hi, status); \ + } \ +} \ +uint64_t HELPER(NAME)(void *vn, void *vg, void *vs, uint32_t desc) \ +{ \ + uintptr_t i, oprsz = simd_oprsz(desc), maxsz = simd_maxsz(desc); \ + TYPE data[sizeof(ARMVectorReg) / sizeof(TYPE)]; \ + for (i = 0; i < oprsz; ) { \ + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ + do { \ + TYPE nn = *(TYPE *)(vn + H(i)); \ + *(TYPE *)((void *)data + i) = (pg & 1 ? nn : IDENT); \ + i += sizeof(TYPE), pg >>= sizeof(TYPE); \ + } while (i & 15); \ + } \ + for (; i < maxsz; i += sizeof(TYPE)) { \ + *(TYPE *)((void *)data + i) = IDENT; \ + } \ + return NAME##_reduce(data, vs, maxsz / sizeof(TYPE)); \ +} + +DO_REDUCE(sve_faddv_h, float16, H1_2, add, float16_zero) +DO_REDUCE(sve_faddv_s, float32, H1_4, add, float32_zero) +DO_REDUCE(sve_faddv_d, float64, , add, float64_zero) + +/* Identity is floatN_default_nan, without the function call. */ +DO_REDUCE(sve_fminnmv_h, float16, H1_2, minnum, 0x7E00) +DO_REDUCE(sve_fminnmv_s, float32, H1_4, minnum, 0x7FC00000) +DO_REDUCE(sve_fminnmv_d, float64, , minnum, 0x7FF8000000000000ULL) + +DO_REDUCE(sve_fmaxnmv_h, float16, H1_2, maxnum, 0x7E00) +DO_REDUCE(sve_fmaxnmv_s, float32, H1_4, maxnum, 0x7FC00000) +DO_REDUCE(sve_fmaxnmv_d, float64, , maxnum, 0x7FF8000000000000ULL) + +DO_REDUCE(sve_fminv_h, float16, H1_2, min, float16_infinity) +DO_REDUCE(sve_fminv_s, float32, H1_4, min, float32_infinity) +DO_REDUCE(sve_fminv_d, float64, , min, float64_infinity) + +DO_REDUCE(sve_fmaxv_h, float16, H1_2, max, float16_chs(float16_infinity)) +DO_REDUCE(sve_fmaxv_s, float32, H1_4, max, float32_chs(float32_infinity)) +DO_REDUCE(sve_fmaxv_d, float64, , max, float64_chs(float64_infinity)) + +#undef DO_REDUCE + uint64_t HELPER(sve_fadda_h)(uint64_t nn, void *vm, void *vg, void *status, uint32_t desc) { diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ static bool trans_FMUL_zzx(DisasContext *s, arg_FMUL_zzx *a, uint32_t insn) return true; } +/* + *** SVE Floating Point Fast Reduction Group + */ + +typedef void gen_helper_fp_reduce(TCGv_i64, TCGv_ptr, TCGv_ptr, + TCGv_ptr, TCGv_i32); + +static void do_reduce(DisasContext *s, arg_rpr_esz *a, + gen_helper_fp_reduce *fn) +{ + unsigned vsz = vec_full_reg_size(s); + unsigned p2vsz = pow2ceil(vsz); + TCGv_i32 t_desc = tcg_const_i32(simd_desc(vsz, p2vsz, 0)); + TCGv_ptr t_zn, t_pg, status; + TCGv_i64 temp; + + temp = tcg_temp_new_i64(); + t_zn = tcg_temp_new_ptr(); + t_pg = tcg_temp_new_ptr(); + + tcg_gen_addi_ptr(t_zn, cpu_env, vec_full_reg_offset(s, a->rn)); + tcg_gen_addi_ptr(t_pg, cpu_env, pred_full_reg_offset(s, a->pg)); + status = get_fpstatus_ptr(a->esz == MO_16); + + fn(temp, t_zn, t_pg, status, t_desc); + tcg_temp_free_ptr(t_zn); + tcg_temp_free_ptr(t_pg); + tcg_temp_free_ptr(status); + tcg_temp_free_i32(t_desc); + + write_fp_dreg(s, a->rd, temp); + tcg_temp_free_i64(temp); +} + +#define DO_VPZ(NAME, name) \ +static bool trans_##NAME(DisasContext *s, arg_rpr_esz *a, uint32_t insn) \ +{ \ + static gen_helper_fp_reduce * const fns[3] = { \ + gen_helper_sve_##name##_h, \ + gen_helper_sve_##name##_s, \ + gen_helper_sve_##name##_d, \ + }; \ + if (a->esz == 0) { \ + return false; \ + } \ + if (sve_access_check(s)) { \ + do_reduce(s, a, fns[a->esz - 1]); \ + } \ + return true; \ +} + +DO_VPZ(FADDV, faddv) +DO_VPZ(FMINNMV, fminnmv) +DO_VPZ(FMAXNMV, fmaxnmv) +DO_VPZ(FMINV, fminv) +DO_VPZ(FMAXV, fmaxv) + /* *** SVE Floating Point Accumulating Reduction Group */ diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ FMUL_zzx 01100100 0.1 .. rm:3 001000 rn:5 rd:5 \ FMUL_zzx 01100100 101 index:2 rm:3 001000 rn:5 rd:5 esz=2 FMUL_zzx 01100100 111 index:1 rm:4 001000 rn:5 rd:5 esz=3 +### SVE FP Fast Reduction Group + +FADDV 01100101 .. 000 000 001 ... ..... ..... @rd_pg_rn +FMAXNMV 01100101 .. 000 100 001 ... ..... ..... @rd_pg_rn +FMINNMV 01100101 .. 000 101 001 ... ..... ..... @rd_pg_rn +FMAXV 01100101 .. 000 110 001 ... ..... ..... @rd_pg_rn +FMINV 01100101 .. 000 111 001 ... ..... ..... @rd_pg_rn + ### SVE FP Accumulating Reduction Group # SVE floating-point serial reduction (predicated) -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-21-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper.h | 8 +++++++ target/arm/translate-sve.c | 47 ++++++++++++++++++++++++++++++++++++++ target/arm/vec_helper.c | 20 ++++++++++++++++ target/arm/sve.decode | 5 ++++ 4 files changed, 80 insertions(+) diff --git a/target/arm/helper.h b/target/arm/helper.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper.h +++ b/target/arm/helper.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_5(gvec_fcmlas_idx, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_5(gvec_fcmlad, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_frecpe_h, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_frecpe_s, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_frecpe_d, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_4(gvec_frsqrte_h, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_frsqrte_s, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_frsqrte_d, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) + DEF_HELPER_FLAGS_5(gvec_fadd_h, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_5(gvec_fadd_s, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_5(gvec_fadd_d, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ DO_VPZ(FMAXNMV, fmaxnmv) DO_VPZ(FMINV, fminv) DO_VPZ(FMAXV, fmaxv) +/* + *** SVE Floating Point Unary Operations - Unpredicated Group + */ + +static void do_zz_fp(DisasContext *s, arg_rr_esz *a, gen_helper_gvec_2_ptr *fn) +{ + unsigned vsz = vec_full_reg_size(s); + TCGv_ptr status = get_fpstatus_ptr(a->esz == MO_16); + + tcg_gen_gvec_2_ptr(vec_full_reg_offset(s, a->rd), + vec_full_reg_offset(s, a->rn), + status, vsz, vsz, 0, fn); + tcg_temp_free_ptr(status); +} + +static bool trans_FRECPE(DisasContext *s, arg_rr_esz *a, uint32_t insn) +{ + static gen_helper_gvec_2_ptr * const fns[3] = { + gen_helper_gvec_frecpe_h, + gen_helper_gvec_frecpe_s, + gen_helper_gvec_frecpe_d, + }; + if (a->esz == 0) { + return false; + } + if (sve_access_check(s)) { + do_zz_fp(s, a, fns[a->esz - 1]); + } + return true; +} + +static bool trans_FRSQRTE(DisasContext *s, arg_rr_esz *a, uint32_t insn) +{ + static gen_helper_gvec_2_ptr * const fns[3] = { + gen_helper_gvec_frsqrte_h, + gen_helper_gvec_frsqrte_s, + gen_helper_gvec_frsqrte_d, + }; + if (a->esz == 0) { + return false; + } + if (sve_access_check(s)) { + do_zz_fp(s, a, fns[a->esz - 1]); + } + return true; +} + /* *** SVE Floating Point Accumulating Reduction Group */ diff --git a/target/arm/vec_helper.c b/target/arm/vec_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/vec_helper.c +++ b/target/arm/vec_helper.c @@ -XXX,XX +XXX,XX @@ void HELPER(gvec_fcmlad)(void *vd, void *vn, void *vm, clear_tail(d, opr_sz, simd_maxsz(desc)); } +#define DO_2OP(NAME, FUNC, TYPE) \ +void HELPER(NAME)(void *vd, void *vn, void *stat, uint32_t desc) \ +{ \ + intptr_t i, oprsz = simd_oprsz(desc); \ + TYPE *d = vd, *n = vn; \ + for (i = 0; i < oprsz / sizeof(TYPE); i++) { \ + d[i] = FUNC(n[i], stat); \ + } \ +} + +DO_2OP(gvec_frecpe_h, helper_recpe_f16, float16) +DO_2OP(gvec_frecpe_s, helper_recpe_f32, float32) +DO_2OP(gvec_frecpe_d, helper_recpe_f64, float64) + +DO_2OP(gvec_frsqrte_h, helper_rsqrte_f16, float16) +DO_2OP(gvec_frsqrte_s, helper_rsqrte_f32, float32) +DO_2OP(gvec_frsqrte_d, helper_rsqrte_f64, float64) + +#undef DO_2OP + /* Floating-point trigonometric starting value. * See the ARM ARM pseudocode function FPTrigSMul. */ diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ FMINNMV 01100101 .. 000 101 001 ... ..... ..... @rd_pg_rn FMAXV 01100101 .. 000 110 001 ... ..... ..... @rd_pg_rn FMINV 01100101 .. 000 111 001 ... ..... ..... @rd_pg_rn +## SVE Floating Point Unary Operations - Unpredicated Group + +FRECPE 01100101 .. 001 110 001100 ..... ..... @rd_rn +FRSQRTE 01100101 .. 001 111 001100 ..... ..... @rd_rn + ### SVE FP Accumulating Reduction Group # SVE floating-point serial reduction (predicated) -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-22-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 42 +++++++++++++++++++++++++++++++++++++ target/arm/sve_helper.c | 43 ++++++++++++++++++++++++++++++++++++++ target/arm/translate-sve.c | 43 ++++++++++++++++++++++++++++++++++++++ target/arm/sve.decode | 10 +++++++++ 4 files changed, 138 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_5(sve_fadda_s, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_5(sve_fadda_d, TCG_CALL_NO_RWG, i64, i64, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcmge0_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcmge0_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcmge0_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_5(sve_fcmgt0_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcmgt0_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcmgt0_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_5(sve_fcmlt0_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcmlt0_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcmlt0_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_5(sve_fcmle0_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcmle0_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcmle0_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_5(sve_fcmeq0_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcmeq0_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcmeq0_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_5(sve_fcmne0_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcmne0_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcmne0_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) + DEF_HELPER_FLAGS_6(sve_fadd_h, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_6(sve_fadd_s, TCG_CALL_NO_RWG, diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ void HELPER(NAME)(void *vd, void *vn, void *vm, void *vg, \ #define DO_FCMGE(TYPE, X, Y, ST) TYPE##_compare(Y, X, ST) <= 0 #define DO_FCMGT(TYPE, X, Y, ST) TYPE##_compare(Y, X, ST) < 0 +#define DO_FCMLE(TYPE, X, Y, ST) TYPE##_compare(X, Y, ST) <= 0 +#define DO_FCMLT(TYPE, X, Y, ST) TYPE##_compare(X, Y, ST) < 0 #define DO_FCMEQ(TYPE, X, Y, ST) TYPE##_compare_quiet(X, Y, ST) == 0 #define DO_FCMNE(TYPE, X, Y, ST) TYPE##_compare_quiet(X, Y, ST) != 0 #define DO_FCMUO(TYPE, X, Y, ST) \ @@ -XXX,XX +XXX,XX @@ DO_FPCMP_PPZZ_ALL(sve_facgt, DO_FACGT) #undef DO_FPCMP_PPZZ_H #undef DO_FPCMP_PPZZ +/* One operand floating-point comparison against zero, controlled + * by a predicate. + */ +#define DO_FPCMP_PPZ0(NAME, TYPE, H, OP) \ +void HELPER(NAME)(void *vd, void *vn, void *vg, \ + void *status, uint32_t desc) \ +{ \ + intptr_t i = simd_oprsz(desc), j = (i - 1) >> 6; \ + uint64_t *d = vd, *g = vg; \ + do { \ + uint64_t out = 0, pg = g[j]; \ + do { \ + i -= sizeof(TYPE), out <<= sizeof(TYPE); \ + if ((pg >> (i & 63)) & 1) { \ + TYPE nn = *(TYPE *)(vn + H(i)); \ + out |= OP(TYPE, nn, 0, status); \ + } \ + } while (i & 63); \ + d[j--] = out; \ + } while (i > 0); \ +} + +#define DO_FPCMP_PPZ0_H(NAME, OP) \ + DO_FPCMP_PPZ0(NAME##_h, float16, H1_2, OP) +#define DO_FPCMP_PPZ0_S(NAME, OP) \ + DO_FPCMP_PPZ0(NAME##_s, float32, H1_4, OP) +#define DO_FPCMP_PPZ0_D(NAME, OP) \ + DO_FPCMP_PPZ0(NAME##_d, float64, , OP) + +#define DO_FPCMP_PPZ0_ALL(NAME, OP) \ + DO_FPCMP_PPZ0_H(NAME, OP) \ + DO_FPCMP_PPZ0_S(NAME, OP) \ + DO_FPCMP_PPZ0_D(NAME, OP) + +DO_FPCMP_PPZ0_ALL(sve_fcmge0, DO_FCMGE) +DO_FPCMP_PPZ0_ALL(sve_fcmgt0, DO_FCMGT) +DO_FPCMP_PPZ0_ALL(sve_fcmle0, DO_FCMLE) +DO_FPCMP_PPZ0_ALL(sve_fcmlt0, DO_FCMLT) +DO_FPCMP_PPZ0_ALL(sve_fcmeq0, DO_FCMEQ) +DO_FPCMP_PPZ0_ALL(sve_fcmne0, DO_FCMNE) + /* * Load contiguous data, protected by a governing predicate. */ diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ static bool trans_FRSQRTE(DisasContext *s, arg_rr_esz *a, uint32_t insn) return true; } +/* + *** SVE Floating Point Compare with Zero Group + */ + +static void do_ppz_fp(DisasContext *s, arg_rpr_esz *a, + gen_helper_gvec_3_ptr *fn) +{ + unsigned vsz = vec_full_reg_size(s); + TCGv_ptr status = get_fpstatus_ptr(a->esz == MO_16); + + tcg_gen_gvec_3_ptr(pred_full_reg_offset(s, a->rd), + vec_full_reg_offset(s, a->rn), + pred_full_reg_offset(s, a->pg), + status, vsz, vsz, 0, fn); + tcg_temp_free_ptr(status); +} + +#define DO_PPZ(NAME, name) \ +static bool trans_##NAME(DisasContext *s, arg_rpr_esz *a, uint32_t insn) \ +{ \ + static gen_helper_gvec_3_ptr * const fns[3] = { \ + gen_helper_sve_##name##_h, \ + gen_helper_sve_##name##_s, \ + gen_helper_sve_##name##_d, \ + }; \ + if (a->esz == 0) { \ + return false; \ + } \ + if (sve_access_check(s)) { \ + do_ppz_fp(s, a, fns[a->esz - 1]); \ + } \ + return true; \ +} + +DO_PPZ(FCMGE_ppz0, fcmge0) +DO_PPZ(FCMGT_ppz0, fcmgt0) +DO_PPZ(FCMLE_ppz0, fcmle0) +DO_PPZ(FCMLT_ppz0, fcmlt0) +DO_PPZ(FCMEQ_ppz0, fcmeq0) +DO_PPZ(FCMNE_ppz0, fcmne0) + +#undef DO_PPZ + /* *** SVE Floating Point Accumulating Reduction Group */ diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ # One register operand, with governing predicate, vector element size @rd_pg_rn ........ esz:2 ... ... ... pg:3 rn:5 rd:5 &rpr_esz @rd_pg4_pn ........ esz:2 ... ... .. pg:4 . rn:4 rd:5 &rpr_esz +@pd_pg_rn ........ esz:2 ... ... ... pg:3 rn:5 . rd:4 &rpr_esz # One register operand, with governing predicate, no vector element size @rd_pg_rn_e0 ........ .. ... ... ... pg:3 rn:5 rd:5 &rpr_esz esz=0 @@ -XXX,XX +XXX,XX @@ FMINV 01100101 .. 000 111 001 ... ..... ..... @rd_pg_rn FRECPE 01100101 .. 001 110 001100 ..... ..... @rd_rn FRSQRTE 01100101 .. 001 111 001100 ..... ..... @rd_rn +### SVE FP Compare with Zero Group + +FCMGE_ppz0 01100101 .. 0100 00 001 ... ..... 0 .... @pd_pg_rn +FCMGT_ppz0 01100101 .. 0100 00 001 ... ..... 1 .... @pd_pg_rn +FCMLT_ppz0 01100101 .. 0100 01 001 ... ..... 0 .... @pd_pg_rn +FCMLE_ppz0 01100101 .. 0100 01 001 ... ..... 1 .... @pd_pg_rn +FCMEQ_ppz0 01100101 .. 0100 10 001 ... ..... 0 .... @pd_pg_rn +FCMNE_ppz0 01100101 .. 0100 11 001 ... ..... 0 .... @pd_pg_rn + ### SVE FP Accumulating Reduction Group # SVE floating-point serial reduction (predicated) -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-23-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 4 +++ target/arm/sve_helper.c | 70 ++++++++++++++++++++++++++++++++++++++ target/arm/translate-sve.c | 27 +++++++++++++++ target/arm/sve.decode | 3 ++ 4 files changed, 104 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_3(sve_fnmls_zpzzz_h, TCG_CALL_NO_RWG, void, env, ptr, i32) DEF_HELPER_FLAGS_3(sve_fnmls_zpzzz_s, TCG_CALL_NO_RWG, void, env, ptr, i32) DEF_HELPER_FLAGS_3(sve_fnmls_zpzzz_d, TCG_CALL_NO_RWG, void, env, ptr, i32) +DEF_HELPER_FLAGS_5(sve_ftmad_h, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_ftmad_s, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_ftmad_d, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) + DEF_HELPER_FLAGS_4(sve_ld1bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ld2bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ld3bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ DO_FPCMP_PPZ0_ALL(sve_fcmlt0, DO_FCMLT) DO_FPCMP_PPZ0_ALL(sve_fcmeq0, DO_FCMEQ) DO_FPCMP_PPZ0_ALL(sve_fcmne0, DO_FCMNE) +/* FP Trig Multiply-Add. */ + +void HELPER(sve_ftmad_h)(void *vd, void *vn, void *vm, void *vs, uint32_t desc) +{ + static const float16 coeff[16] = { + 0x3c00, 0xb155, 0x2030, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, + 0x3c00, 0xb800, 0x293a, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, + }; + intptr_t i, opr_sz = simd_oprsz(desc) / sizeof(float16); + intptr_t x = simd_data(desc); + float16 *d = vd, *n = vn, *m = vm; + for (i = 0; i < opr_sz; i++) { + float16 mm = m[i]; + intptr_t xx = x; + if (float16_is_neg(mm)) { + mm = float16_abs(mm); + xx += 8; + } + d[i] = float16_muladd(n[i], mm, coeff[xx], 0, vs); + } +} + +void HELPER(sve_ftmad_s)(void *vd, void *vn, void *vm, void *vs, uint32_t desc) +{ + static const float32 coeff[16] = { + 0x3f800000, 0xbe2aaaab, 0x3c088886, 0xb95008b9, + 0x36369d6d, 0x00000000, 0x00000000, 0x00000000, + 0x3f800000, 0xbf000000, 0x3d2aaaa6, 0xbab60705, + 0x37cd37cc, 0x00000000, 0x00000000, 0x00000000, + }; + intptr_t i, opr_sz = simd_oprsz(desc) / sizeof(float32); + intptr_t x = simd_data(desc); + float32 *d = vd, *n = vn, *m = vm; + for (i = 0; i < opr_sz; i++) { + float32 mm = m[i]; + intptr_t xx = x; + if (float32_is_neg(mm)) { + mm = float32_abs(mm); + xx += 8; + } + d[i] = float32_muladd(n[i], mm, coeff[xx], 0, vs); + } +} + +void HELPER(sve_ftmad_d)(void *vd, void *vn, void *vm, void *vs, uint32_t desc) +{ + static const float64 coeff[16] = { + 0x3ff0000000000000ull, 0xbfc5555555555543ull, + 0x3f8111111110f30cull, 0xbf2a01a019b92fc6ull, + 0x3ec71de351f3d22bull, 0xbe5ae5e2b60f7b91ull, + 0x3de5d8408868552full, 0x0000000000000000ull, + 0x3ff0000000000000ull, 0xbfe0000000000000ull, + 0x3fa5555555555536ull, 0xbf56c16c16c13a0bull, + 0x3efa01a019b1e8d8ull, 0xbe927e4f7282f468ull, + 0x3e21ee96d2641b13ull, 0xbda8f76380fbb401ull, + }; + intptr_t i, opr_sz = simd_oprsz(desc) / sizeof(float64); + intptr_t x = simd_data(desc); + float64 *d = vd, *n = vn, *m = vm; + for (i = 0; i < opr_sz; i++) { + float64 mm = m[i]; + intptr_t xx = x; + if (float64_is_neg(mm)) { + mm = float64_abs(mm); + xx += 8; + } + d[i] = float64_muladd(n[i], mm, coeff[xx], 0, vs); + } +} + /* * Load contiguous data, protected by a governing predicate. */ diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ DO_PPZ(FCMNE_ppz0, fcmne0) #undef DO_PPZ +/* + *** SVE floating-point trig multiply-add coefficient + */ + +static bool trans_FTMAD(DisasContext *s, arg_FTMAD *a, uint32_t insn) +{ + static gen_helper_gvec_3_ptr * const fns[3] = { + gen_helper_sve_ftmad_h, + gen_helper_sve_ftmad_s, + gen_helper_sve_ftmad_d, + }; + + if (a->esz == 0) { + return false; + } + if (sve_access_check(s)) { + unsigned vsz = vec_full_reg_size(s); + TCGv_ptr status = get_fpstatus_ptr(a->esz == MO_16); + tcg_gen_gvec_3_ptr(vec_full_reg_offset(s, a->rd), + vec_full_reg_offset(s, a->rn), + vec_full_reg_offset(s, a->rm), + status, vsz, vsz, a->imm, fns[a->esz - 1]); + tcg_temp_free_ptr(status); + } + return true; +} + /* *** SVE Floating Point Accumulating Reduction Group */ diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ FMINNM_zpzi 01100101 .. 011 101 100 ... 0000 . ..... @rdn_i1 FMAX_zpzi 01100101 .. 011 110 100 ... 0000 . ..... @rdn_i1 FMIN_zpzi 01100101 .. 011 111 100 ... 0000 . ..... @rdn_i1 +# SVE floating-point trig multiply-add coefficient +FTMAD 01100101 esz:2 010 imm:3 100000 rm:5 rd:5 rn=%reg_movprfx + ### SVE FP Multiply-Add Group # SVE floating-point multiply-accumulate writing addend -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-24-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 13 +++++++++ target/arm/sve_helper.c | 55 ++++++++++++++++++++++++++++++++++++++ target/arm/translate-sve.c | 30 +++++++++++++++++++++ target/arm/sve.decode | 8 ++++++ 4 files changed, 106 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_6(sve_fmins_s, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_6(sve_fmins_d, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i64, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcvt_sh, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcvt_dh, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcvt_hs, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcvt_ds, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcvt_hd, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcvt_sd, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) + DEF_HELPER_FLAGS_5(sve_scvt_hh, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_5(sve_scvt_sh, TCG_CALL_NO_RWG, diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ void HELPER(NAME)(void *vd, void *vn, void *vg, void *status, uint32_t desc) \ } while (i != 0); \ } +/* SVE fp16 conversions always use IEEE mode. Like AdvSIMD, they ignore + * FZ16. When converting from fp16, this affects flushing input denormals; + * when converting to fp16, this affects flushing output denormals. + */ +static inline float32 sve_f16_to_f32(float16 f, float_status *fpst) +{ + flag save = get_flush_inputs_to_zero(fpst); + float32 ret; + + set_flush_inputs_to_zero(false, fpst); + ret = float16_to_float32(f, true, fpst); + set_flush_inputs_to_zero(save, fpst); + return ret; +} + +static inline float64 sve_f16_to_f64(float16 f, float_status *fpst) +{ + flag save = get_flush_inputs_to_zero(fpst); + float64 ret; + + set_flush_inputs_to_zero(false, fpst); + ret = float16_to_float64(f, true, fpst); + set_flush_inputs_to_zero(save, fpst); + return ret; +} + +static inline float16 sve_f32_to_f16(float32 f, float_status *fpst) +{ + flag save = get_flush_to_zero(fpst); + float16 ret; + + set_flush_to_zero(false, fpst); + ret = float32_to_float16(f, true, fpst); + set_flush_to_zero(save, fpst); + return ret; +} + +static inline float16 sve_f64_to_f16(float64 f, float_status *fpst) +{ + flag save = get_flush_to_zero(fpst); + float16 ret; + + set_flush_to_zero(false, fpst); + ret = float64_to_float16(f, true, fpst); + set_flush_to_zero(save, fpst); + return ret; +} + +DO_ZPZ_FP(sve_fcvt_sh, uint32_t, H1_4, sve_f32_to_f16) +DO_ZPZ_FP(sve_fcvt_hs, uint32_t, H1_4, sve_f16_to_f32) +DO_ZPZ_FP(sve_fcvt_dh, uint64_t, , sve_f64_to_f16) +DO_ZPZ_FP(sve_fcvt_hd, uint64_t, , sve_f16_to_f64) +DO_ZPZ_FP(sve_fcvt_ds, uint64_t, , float64_to_float32) +DO_ZPZ_FP(sve_fcvt_sd, uint64_t, , float32_to_float64) + DO_ZPZ_FP(sve_scvt_hh, uint16_t, H1_2, int16_to_float16) DO_ZPZ_FP(sve_scvt_sh, uint32_t, H1_4, int32_to_float16) DO_ZPZ_FP(sve_scvt_ss, uint32_t, H1_4, int32_to_float32) diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ static bool do_zpz_ptr(DisasContext *s, int rd, int rn, int pg, return true; } +static bool trans_FCVT_sh(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, true, gen_helper_sve_fcvt_sh); +} + +static bool trans_FCVT_hs(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_fcvt_hs); +} + +static bool trans_FCVT_dh(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, true, gen_helper_sve_fcvt_dh); +} + +static bool trans_FCVT_hd(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_fcvt_hd); +} + +static bool trans_FCVT_ds(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_fcvt_ds); +} + +static bool trans_FCVT_sd(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_fcvt_sd); +} + static bool trans_SCVTF_hh(DisasContext *s, arg_rpr_esz *a, uint32_t insn) { return do_zpz_ptr(s, a->rd, a->rn, a->pg, true, gen_helper_sve_scvt_hh); diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ FNMLS_zpzzz 01100101 .. 1 ..... 111 ... ..... ..... @rdn_pg_rm_ra ### SVE FP Unary Operations Predicated Group +# SVE floating-point convert precision +FCVT_sh 01100101 10 0010 00 101 ... ..... ..... @rd_pg_rn_e0 +FCVT_hs 01100101 10 0010 01 101 ... ..... ..... @rd_pg_rn_e0 +FCVT_dh 01100101 11 0010 00 101 ... ..... ..... @rd_pg_rn_e0 +FCVT_hd 01100101 11 0010 01 101 ... ..... ..... @rd_pg_rn_e0 +FCVT_ds 01100101 11 0010 10 101 ... ..... ..... @rd_pg_rn_e0 +FCVT_sd 01100101 11 0010 11 101 ... ..... ..... @rd_pg_rn_e0 + # SVE integer convert to floating-point SCVTF_hh 01100101 01 010 01 0 101 ... ..... ..... @rd_pg_rn_e0 SCVTF_sh 01100101 01 010 10 0 101 ... ..... ..... @rd_pg_rn_e0 -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-25-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 30 +++++++++++++ target/arm/helper.h | 12 +++--- target/arm/helper.c | 2 +- target/arm/sve_helper.c | 88 ++++++++++++++++++++++++++++++++++++++ target/arm/translate-sve.c | 70 ++++++++++++++++++++++++++++++ target/arm/sve.decode | 16 +++++++ 6 files changed, 211 insertions(+), 7 deletions(-) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_5(sve_fcvt_hd, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_5(sve_fcvt_sd, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcvtzs_hh, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcvtzs_hs, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcvtzs_ss, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcvtzs_ds, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcvtzs_hd, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcvtzs_sd, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcvtzs_dd, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_5(sve_fcvtzu_hh, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcvtzu_hs, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcvtzu_ss, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcvtzu_ds, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcvtzu_hd, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcvtzu_sd, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fcvtzu_dd, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) + DEF_HELPER_FLAGS_5(sve_scvt_hh, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_5(sve_scvt_sh, TCG_CALL_NO_RWG, diff --git a/target/arm/helper.h b/target/arm/helper.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper.h +++ b/target/arm/helper.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_2(vfp_touid, i32, f64, ptr) DEF_HELPER_2(vfp_touizh, i32, f16, ptr) DEF_HELPER_2(vfp_touizs, i32, f32, ptr) DEF_HELPER_2(vfp_touizd, i32, f64, ptr) -DEF_HELPER_2(vfp_tosih, i32, f16, ptr) -DEF_HELPER_2(vfp_tosis, i32, f32, ptr) -DEF_HELPER_2(vfp_tosid, i32, f64, ptr) -DEF_HELPER_2(vfp_tosizh, i32, f16, ptr) -DEF_HELPER_2(vfp_tosizs, i32, f32, ptr) -DEF_HELPER_2(vfp_tosizd, i32, f64, ptr) +DEF_HELPER_2(vfp_tosih, s32, f16, ptr) +DEF_HELPER_2(vfp_tosis, s32, f32, ptr) +DEF_HELPER_2(vfp_tosid, s32, f64, ptr) +DEF_HELPER_2(vfp_tosizh, s32, f16, ptr) +DEF_HELPER_2(vfp_tosizs, s32, f32, ptr) +DEF_HELPER_2(vfp_tosizd, s32, f64, ptr) DEF_HELPER_3(vfp_toshs_round_to_zero, i32, f32, i32, ptr) DEF_HELPER_3(vfp_tosls_round_to_zero, i32, f32, i32, ptr) diff --git a/target/arm/helper.c b/target/arm/helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper.c +++ b/target/arm/helper.c @@ -XXX,XX +XXX,XX @@ ftype HELPER(name)(uint32_t x, void *fpstp) \ } #define CONV_FTOI(name, ftype, fsz, sign, round) \ -uint32_t HELPER(name)(ftype x, void *fpstp) \ +sign##int32_t HELPER(name)(ftype x, void *fpstp) \ { \ float_status *fpst = fpstp; \ if (float##fsz##_is_any_nan(x)) { \ diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ static inline float16 sve_f64_to_f16(float64 f, float_status *fpst) return ret; } +static inline int16_t vfp_float16_to_int16_rtz(float16 f, float_status *s) +{ + if (float16_is_any_nan(f)) { + float_raise(float_flag_invalid, s); + return 0; + } + return float16_to_int16_round_to_zero(f, s); +} + +static inline int64_t vfp_float16_to_int64_rtz(float16 f, float_status *s) +{ + if (float16_is_any_nan(f)) { + float_raise(float_flag_invalid, s); + return 0; + } + return float16_to_int64_round_to_zero(f, s); +} + +static inline int64_t vfp_float32_to_int64_rtz(float32 f, float_status *s) +{ + if (float32_is_any_nan(f)) { + float_raise(float_flag_invalid, s); + return 0; + } + return float32_to_int64_round_to_zero(f, s); +} + +static inline int64_t vfp_float64_to_int64_rtz(float64 f, float_status *s) +{ + if (float64_is_any_nan(f)) { + float_raise(float_flag_invalid, s); + return 0; + } + return float64_to_int64_round_to_zero(f, s); +} + +static inline uint16_t vfp_float16_to_uint16_rtz(float16 f, float_status *s) +{ + if (float16_is_any_nan(f)) { + float_raise(float_flag_invalid, s); + return 0; + } + return float16_to_uint16_round_to_zero(f, s); +} + +static inline uint64_t vfp_float16_to_uint64_rtz(float16 f, float_status *s) +{ + if (float16_is_any_nan(f)) { + float_raise(float_flag_invalid, s); + return 0; + } + return float16_to_uint64_round_to_zero(f, s); +} + +static inline uint64_t vfp_float32_to_uint64_rtz(float32 f, float_status *s) +{ + if (float32_is_any_nan(f)) { + float_raise(float_flag_invalid, s); + return 0; + } + return float32_to_uint64_round_to_zero(f, s); +} + +static inline uint64_t vfp_float64_to_uint64_rtz(float64 f, float_status *s) +{ + if (float64_is_any_nan(f)) { + float_raise(float_flag_invalid, s); + return 0; + } + return float64_to_uint64_round_to_zero(f, s); +} + DO_ZPZ_FP(sve_fcvt_sh, uint32_t, H1_4, sve_f32_to_f16) DO_ZPZ_FP(sve_fcvt_hs, uint32_t, H1_4, sve_f16_to_f32) DO_ZPZ_FP(sve_fcvt_dh, uint64_t, , sve_f64_to_f16) @@ -XXX,XX +XXX,XX @@ DO_ZPZ_FP(sve_fcvt_hd, uint64_t, , sve_f16_to_f64) DO_ZPZ_FP(sve_fcvt_ds, uint64_t, , float64_to_float32) DO_ZPZ_FP(sve_fcvt_sd, uint64_t, , float32_to_float64) +DO_ZPZ_FP(sve_fcvtzs_hh, uint16_t, H1_2, vfp_float16_to_int16_rtz) +DO_ZPZ_FP(sve_fcvtzs_hs, uint32_t, H1_4, helper_vfp_tosizh) +DO_ZPZ_FP(sve_fcvtzs_ss, uint32_t, H1_4, helper_vfp_tosizs) +DO_ZPZ_FP(sve_fcvtzs_hd, uint64_t, , vfp_float16_to_int64_rtz) +DO_ZPZ_FP(sve_fcvtzs_sd, uint64_t, , vfp_float32_to_int64_rtz) +DO_ZPZ_FP(sve_fcvtzs_ds, uint64_t, , helper_vfp_tosizd) +DO_ZPZ_FP(sve_fcvtzs_dd, uint64_t, , vfp_float64_to_int64_rtz) + +DO_ZPZ_FP(sve_fcvtzu_hh, uint16_t, H1_2, vfp_float16_to_uint16_rtz) +DO_ZPZ_FP(sve_fcvtzu_hs, uint32_t, H1_4, helper_vfp_touizh) +DO_ZPZ_FP(sve_fcvtzu_ss, uint32_t, H1_4, helper_vfp_touizs) +DO_ZPZ_FP(sve_fcvtzu_hd, uint64_t, , vfp_float16_to_uint64_rtz) +DO_ZPZ_FP(sve_fcvtzu_sd, uint64_t, , vfp_float32_to_uint64_rtz) +DO_ZPZ_FP(sve_fcvtzu_ds, uint64_t, , helper_vfp_touizd) +DO_ZPZ_FP(sve_fcvtzu_dd, uint64_t, , vfp_float64_to_uint64_rtz) + DO_ZPZ_FP(sve_scvt_hh, uint16_t, H1_2, int16_to_float16) DO_ZPZ_FP(sve_scvt_sh, uint32_t, H1_4, int32_to_float16) DO_ZPZ_FP(sve_scvt_ss, uint32_t, H1_4, int32_to_float32) diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ static bool trans_FCVT_sd(DisasContext *s, arg_rpr_esz *a, uint32_t insn) return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_fcvt_sd); } +static bool trans_FCVTZS_hh(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, true, gen_helper_sve_fcvtzs_hh); +} + +static bool trans_FCVTZU_hh(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, true, gen_helper_sve_fcvtzu_hh); +} + +static bool trans_FCVTZS_hs(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, true, gen_helper_sve_fcvtzs_hs); +} + +static bool trans_FCVTZU_hs(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, true, gen_helper_sve_fcvtzu_hs); +} + +static bool trans_FCVTZS_hd(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, true, gen_helper_sve_fcvtzs_hd); +} + +static bool trans_FCVTZU_hd(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, true, gen_helper_sve_fcvtzu_hd); +} + +static bool trans_FCVTZS_ss(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_fcvtzs_ss); +} + +static bool trans_FCVTZU_ss(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_fcvtzu_ss); +} + +static bool trans_FCVTZS_sd(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_fcvtzs_sd); +} + +static bool trans_FCVTZU_sd(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_fcvtzu_sd); +} + +static bool trans_FCVTZS_ds(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_fcvtzs_ds); +} + +static bool trans_FCVTZU_ds(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_fcvtzu_ds); +} + +static bool trans_FCVTZS_dd(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_fcvtzs_dd); +} + +static bool trans_FCVTZU_dd(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_fcvtzu_dd); +} + static bool trans_SCVTF_hh(DisasContext *s, arg_rpr_esz *a, uint32_t insn) { return do_zpz_ptr(s, a->rd, a->rn, a->pg, true, gen_helper_sve_scvt_hh); diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ FCVT_hd 01100101 11 0010 01 101 ... ..... ..... @rd_pg_rn_e0 FCVT_ds 01100101 11 0010 10 101 ... ..... ..... @rd_pg_rn_e0 FCVT_sd 01100101 11 0010 11 101 ... ..... ..... @rd_pg_rn_e0 +# SVE floating-point convert to integer +FCVTZS_hh 01100101 01 011 01 0 101 ... ..... ..... @rd_pg_rn_e0 +FCVTZU_hh 01100101 01 011 01 1 101 ... ..... ..... @rd_pg_rn_e0 +FCVTZS_hs 01100101 01 011 10 0 101 ... ..... ..... @rd_pg_rn_e0 +FCVTZU_hs 01100101 01 011 10 1 101 ... ..... ..... @rd_pg_rn_e0 +FCVTZS_hd 01100101 01 011 11 0 101 ... ..... ..... @rd_pg_rn_e0 +FCVTZU_hd 01100101 01 011 11 1 101 ... ..... ..... @rd_pg_rn_e0 +FCVTZS_ss 01100101 10 011 10 0 101 ... ..... ..... @rd_pg_rn_e0 +FCVTZU_ss 01100101 10 011 10 1 101 ... ..... ..... @rd_pg_rn_e0 +FCVTZS_ds 01100101 11 011 00 0 101 ... ..... ..... @rd_pg_rn_e0 +FCVTZU_ds 01100101 11 011 00 1 101 ... ..... ..... @rd_pg_rn_e0 +FCVTZS_sd 01100101 11 011 10 0 101 ... ..... ..... @rd_pg_rn_e0 +FCVTZU_sd 01100101 11 011 10 1 101 ... ..... ..... @rd_pg_rn_e0 +FCVTZS_dd 01100101 11 011 11 0 101 ... ..... ..... @rd_pg_rn_e0 +FCVTZU_dd 01100101 11 011 11 1 101 ... ..... ..... @rd_pg_rn_e0 + # SVE integer convert to floating-point SCVTF_hh 01100101 01 010 01 0 101 ... ..... ..... @rd_pg_rn_e0 SCVTF_sh 01100101 01 010 10 0 101 ... ..... ..... @rd_pg_rn_e0 -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-26-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 14 +++++++ target/arm/sve_helper.c | 8 ++++ target/arm/translate-sve.c | 77 ++++++++++++++++++++++++++++++++++++++ target/arm/sve.decode | 9 +++++ 4 files changed, 108 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_5(sve_fcvtzu_sd, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_5(sve_fcvtzu_dd, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_frint_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_frint_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_frint_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_5(sve_frintx_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_frintx_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_frintx_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) + DEF_HELPER_FLAGS_5(sve_scvt_hh, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_5(sve_scvt_sh, TCG_CALL_NO_RWG, diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ DO_ZPZ_FP(sve_fcvtzu_sd, uint64_t, , vfp_float32_to_uint64_rtz) DO_ZPZ_FP(sve_fcvtzu_ds, uint64_t, , helper_vfp_touizd) DO_ZPZ_FP(sve_fcvtzu_dd, uint64_t, , vfp_float64_to_uint64_rtz) +DO_ZPZ_FP(sve_frint_h, uint16_t, H1_2, helper_advsimd_rinth) +DO_ZPZ_FP(sve_frint_s, uint32_t, H1_4, helper_rints) +DO_ZPZ_FP(sve_frint_d, uint64_t, , helper_rintd) + +DO_ZPZ_FP(sve_frintx_h, uint16_t, H1_2, float16_round_to_int) +DO_ZPZ_FP(sve_frintx_s, uint32_t, H1_4, float32_round_to_int) +DO_ZPZ_FP(sve_frintx_d, uint64_t, , float64_round_to_int) + DO_ZPZ_FP(sve_scvt_hh, uint16_t, H1_2, int16_to_float16) DO_ZPZ_FP(sve_scvt_sh, uint32_t, H1_4, int32_to_float16) DO_ZPZ_FP(sve_scvt_ss, uint32_t, H1_4, int32_to_float32) diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ static bool trans_FCVTZU_dd(DisasContext *s, arg_rpr_esz *a, uint32_t insn) return do_zpz_ptr(s, a->rd, a->rn, a->pg, false, gen_helper_sve_fcvtzu_dd); } +static gen_helper_gvec_3_ptr * const frint_fns[3] = { + gen_helper_sve_frint_h, + gen_helper_sve_frint_s, + gen_helper_sve_frint_d +}; + +static bool trans_FRINTI(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + if (a->esz == 0) { + return false; + } + return do_zpz_ptr(s, a->rd, a->rn, a->pg, a->esz == MO_16, + frint_fns[a->esz - 1]); +} + +static bool trans_FRINTX(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + static gen_helper_gvec_3_ptr * const fns[3] = { + gen_helper_sve_frintx_h, + gen_helper_sve_frintx_s, + gen_helper_sve_frintx_d + }; + if (a->esz == 0) { + return false; + } + return do_zpz_ptr(s, a->rd, a->rn, a->pg, a->esz == MO_16, fns[a->esz - 1]); +} + +static bool do_frint_mode(DisasContext *s, arg_rpr_esz *a, int mode) +{ + if (a->esz == 0) { + return false; + } + if (sve_access_check(s)) { + unsigned vsz = vec_full_reg_size(s); + TCGv_i32 tmode = tcg_const_i32(mode); + TCGv_ptr status = get_fpstatus_ptr(a->esz == MO_16); + + gen_helper_set_rmode(tmode, tmode, status); + + tcg_gen_gvec_3_ptr(vec_full_reg_offset(s, a->rd), + vec_full_reg_offset(s, a->rn), + pred_full_reg_offset(s, a->pg), + status, vsz, vsz, 0, frint_fns[a->esz - 1]); + + gen_helper_set_rmode(tmode, tmode, status); + tcg_temp_free_i32(tmode); + tcg_temp_free_ptr(status); + } + return true; +} + +static bool trans_FRINTN(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_frint_mode(s, a, float_round_nearest_even); +} + +static bool trans_FRINTP(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_frint_mode(s, a, float_round_up); +} + +static bool trans_FRINTM(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_frint_mode(s, a, float_round_down); +} + +static bool trans_FRINTZ(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_frint_mode(s, a, float_round_to_zero); +} + +static bool trans_FRINTA(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + return do_frint_mode(s, a, float_round_ties_away); +} + static bool trans_SCVTF_hh(DisasContext *s, arg_rpr_esz *a, uint32_t insn) { return do_zpz_ptr(s, a->rd, a->rn, a->pg, true, gen_helper_sve_scvt_hh); diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ FCVTZU_sd 01100101 11 011 10 1 101 ... ..... ..... @rd_pg_rn_e0 FCVTZS_dd 01100101 11 011 11 0 101 ... ..... ..... @rd_pg_rn_e0 FCVTZU_dd 01100101 11 011 11 1 101 ... ..... ..... @rd_pg_rn_e0 +# SVE floating-point round to integral value +FRINTN 01100101 .. 000 000 101 ... ..... ..... @rd_pg_rn +FRINTP 01100101 .. 000 001 101 ... ..... ..... @rd_pg_rn +FRINTM 01100101 .. 000 010 101 ... ..... ..... @rd_pg_rn +FRINTZ 01100101 .. 000 011 101 ... ..... ..... @rd_pg_rn +FRINTA 01100101 .. 000 100 101 ... ..... ..... @rd_pg_rn +FRINTX 01100101 .. 000 110 101 ... ..... ..... @rd_pg_rn +FRINTI 01100101 .. 000 111 101 ... ..... ..... @rd_pg_rn + # SVE integer convert to floating-point SCVTF_hh 01100101 01 010 01 0 101 ... ..... ..... @rd_pg_rn_e0 SCVTF_sh 01100101 01 010 10 0 101 ... ..... ..... @rd_pg_rn_e0 -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-27-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 14 ++++++++++++++ target/arm/sve_helper.c | 8 ++++++++ target/arm/translate-sve.c | 26 ++++++++++++++++++++++++++ target/arm/sve.decode | 4 ++++ 4 files changed, 52 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_5(sve_frintx_s, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_5(sve_frintx_d, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_frecpx_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_frecpx_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_frecpx_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_5(sve_fsqrt_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fsqrt_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(sve_fsqrt_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) + DEF_HELPER_FLAGS_5(sve_scvt_hh, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_5(sve_scvt_sh, TCG_CALL_NO_RWG, diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ DO_ZPZ_FP(sve_frintx_h, uint16_t, H1_2, float16_round_to_int) DO_ZPZ_FP(sve_frintx_s, uint32_t, H1_4, float32_round_to_int) DO_ZPZ_FP(sve_frintx_d, uint64_t, , float64_round_to_int) +DO_ZPZ_FP(sve_frecpx_h, uint16_t, H1_2, helper_frecpx_f16) +DO_ZPZ_FP(sve_frecpx_s, uint32_t, H1_4, helper_frecpx_f32) +DO_ZPZ_FP(sve_frecpx_d, uint64_t, , helper_frecpx_f64) + +DO_ZPZ_FP(sve_fsqrt_h, uint16_t, H1_2, float16_sqrt) +DO_ZPZ_FP(sve_fsqrt_s, uint32_t, H1_4, float32_sqrt) +DO_ZPZ_FP(sve_fsqrt_d, uint64_t, , float64_sqrt) + DO_ZPZ_FP(sve_scvt_hh, uint16_t, H1_2, int16_to_float16) DO_ZPZ_FP(sve_scvt_sh, uint32_t, H1_4, int32_to_float16) DO_ZPZ_FP(sve_scvt_ss, uint32_t, H1_4, int32_to_float32) diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ static bool trans_FRINTA(DisasContext *s, arg_rpr_esz *a, uint32_t insn) return do_frint_mode(s, a, float_round_ties_away); } +static bool trans_FRECPX(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + static gen_helper_gvec_3_ptr * const fns[3] = { + gen_helper_sve_frecpx_h, + gen_helper_sve_frecpx_s, + gen_helper_sve_frecpx_d + }; + if (a->esz == 0) { + return false; + } + return do_zpz_ptr(s, a->rd, a->rn, a->pg, a->esz == MO_16, fns[a->esz - 1]); +} + +static bool trans_FSQRT(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + static gen_helper_gvec_3_ptr * const fns[3] = { + gen_helper_sve_fsqrt_h, + gen_helper_sve_fsqrt_s, + gen_helper_sve_fsqrt_d + }; + if (a->esz == 0) { + return false; + } + return do_zpz_ptr(s, a->rd, a->rn, a->pg, a->esz == MO_16, fns[a->esz - 1]); +} + static bool trans_SCVTF_hh(DisasContext *s, arg_rpr_esz *a, uint32_t insn) { return do_zpz_ptr(s, a->rd, a->rn, a->pg, true, gen_helper_sve_scvt_hh); diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ FRINTA 01100101 .. 000 100 101 ... ..... ..... @rd_pg_rn FRINTX 01100101 .. 000 110 101 ... ..... ..... @rd_pg_rn FRINTI 01100101 .. 000 111 101 ... ..... ..... @rd_pg_rn +# SVE floating-point unary operations +FRECPX 01100101 .. 001 100 101 ... ..... ..... @rd_pg_rn +FSQRT 01100101 .. 001 101 101 ... ..... ..... @rd_pg_rn + # SVE integer convert to floating-point SCVTF_hh 01100101 01 010 01 0 101 ... ..... ..... @rd_pg_rn_e0 SCVTF_sh 01100101 01 010 10 0 101 ... ..... ..... @rd_pg_rn_e0 -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-28-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/translate-sve.c | 60 +++++++++++++++++++++++++++++++++++++- target/arm/sve.decode | 7 +++++ 2 files changed, 66 insertions(+), 1 deletion(-) diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ static bool do_zpzz_ool(DisasContext *s, arg_rprr_esz *a, gen_helper_gvec_4 *fn) return true; } +/* Select active elememnts from Zn and inactive elements from Zm, + * storing the result in Zd. + */ +static void do_sel_z(DisasContext *s, int rd, int rn, int rm, int pg, int esz) +{ + static gen_helper_gvec_4 * const fns[4] = { + gen_helper_sve_sel_zpzz_b, gen_helper_sve_sel_zpzz_h, + gen_helper_sve_sel_zpzz_s, gen_helper_sve_sel_zpzz_d + }; + unsigned vsz = vec_full_reg_size(s); + tcg_gen_gvec_4_ool(vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), + vec_full_reg_offset(s, rm), + pred_full_reg_offset(s, pg), + vsz, vsz, 0, fns[esz]); +} + #define DO_ZPZZ(NAME, name) \ static bool trans_##NAME##_zpzz(DisasContext *s, arg_rprr_esz *a, \ uint32_t insn) \ @@ -XXX,XX +XXX,XX @@ static bool trans_UDIV_zpzz(DisasContext *s, arg_rprr_esz *a, uint32_t insn) return do_zpzz_ool(s, a, fns[a->esz]); } -DO_ZPZZ(SEL, sel) +static bool trans_SEL_zpzz(DisasContext *s, arg_rprr_esz *a, uint32_t insn) +{ + if (sve_access_check(s)) { + do_sel_z(s, a->rd, a->rn, a->rm, a->pg, a->esz); + } + return true; +} #undef DO_ZPZZ @@ -XXX,XX +XXX,XX @@ static bool trans_PRF_rr(DisasContext *s, arg_PRF_rr *a, uint32_t insn) sve_access_check(s); return true; } + +/* + * Move Prefix + * + * TODO: The implementation so far could handle predicated merging movprfx. + * The helper functions as written take an extra source register to + * use in the operation, but the result is only written when predication + * succeeds. For unpredicated movprfx, we need to rearrange the helpers + * to allow the final write back to the destination to be unconditional. + * For predicated zeroing movprfx, we need to rearrange the helpers to + * allow the final write back to zero inactives. + * + * In the meantime, just emit the moves. + */ + +static bool trans_MOVPRFX(DisasContext *s, arg_MOVPRFX *a, uint32_t insn) +{ + return do_mov_z(s, a->rd, a->rn); +} + +static bool trans_MOVPRFX_m(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + if (sve_access_check(s)) { + do_sel_z(s, a->rd, a->rn, a->rd, a->pg, a->esz); + } + return true; +} + +static bool trans_MOVPRFX_z(DisasContext *s, arg_rpr_esz *a, uint32_t insn) +{ + if (sve_access_check(s)) { + do_movz_zpz(s, a->rd, a->rn, a->pg, a->esz); + } + return true; +} diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ ORV 00000100 .. 011 000 001 ... ..... ..... @rd_pg_rn EORV 00000100 .. 011 001 001 ... ..... ..... @rd_pg_rn ANDV 00000100 .. 011 010 001 ... ..... ..... @rd_pg_rn +# SVE constructive prefix (predicated) +MOVPRFX_z 00000100 .. 010 000 001 ... ..... ..... @rd_pg_rn +MOVPRFX_m 00000100 .. 010 001 001 ... ..... ..... @rd_pg_rn + # SVE integer add reduction (predicated) # Note that saddv requires size != 3. UADDV 00000100 .. 000 001 001 ... ..... ..... @rd_pg_rn @@ -XXX,XX +XXX,XX @@ ADR_p64 00000100 11 1 ..... 1010 .. ..... ..... @rd_rn_msz_rm ### SVE Integer Misc - Unpredicated Group +# SVE constructive prefix (unpredicated) +MOVPRFX 00000100 00 1 00000 101111 rn:5 rd:5 + # SVE floating-point exponential accelerator # Note esz != 0 FEXPA 00000100 .. 1 00000 101110 ..... ..... @rd_rn -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-29-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 7 +++ target/arm/sve_helper.c | 100 +++++++++++++++++++++++++++++++++++++ target/arm/translate-sve.c | 24 +++++++++ target/arm/sve.decode | 4 ++ 4 files changed, 135 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_6(sve_facgt_s, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_6(sve_facgt_d, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fcadd_h, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fcadd_s, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_6(sve_fcadd_d, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, ptr, i32) + DEF_HELPER_FLAGS_3(sve_fmla_zpzzz_h, TCG_CALL_NO_RWG, void, env, ptr, i32) DEF_HELPER_FLAGS_3(sve_fmla_zpzzz_s, TCG_CALL_NO_RWG, void, env, ptr, i32) DEF_HELPER_FLAGS_3(sve_fmla_zpzzz_d, TCG_CALL_NO_RWG, void, env, ptr, i32) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ void HELPER(sve_ftmad_d)(void *vd, void *vn, void *vm, void *vs, uint32_t desc) } } +/* + * FP Complex Add + */ + +void HELPER(sve_fcadd_h)(void *vd, void *vn, void *vm, void *vg, + void *vs, uint32_t desc) +{ + intptr_t j, i = simd_oprsz(desc); + uint64_t *g = vg; + float16 neg_imag = float16_set_sign(0, simd_data(desc)); + float16 neg_real = float16_chs(neg_imag); + + do { + uint64_t pg = g[(i - 1) >> 6]; + do { + float16 e0, e1, e2, e3; + + /* I holds the real index; J holds the imag index. */ + j = i - sizeof(float16); + i -= 2 * sizeof(float16); + + e0 = *(float16 *)(vn + H1_2(i)); + e1 = *(float16 *)(vm + H1_2(j)) ^ neg_real; + e2 = *(float16 *)(vn + H1_2(j)); + e3 = *(float16 *)(vm + H1_2(i)) ^ neg_imag; + + if (likely((pg >> (i & 63)) & 1)) { + *(float16 *)(vd + H1_2(i)) = float16_add(e0, e1, vs); + } + if (likely((pg >> (j & 63)) & 1)) { + *(float16 *)(vd + H1_2(j)) = float16_add(e2, e3, vs); + } + } while (i & 63); + } while (i != 0); +} + +void HELPER(sve_fcadd_s)(void *vd, void *vn, void *vm, void *vg, + void *vs, uint32_t desc) +{ + intptr_t j, i = simd_oprsz(desc); + uint64_t *g = vg; + float32 neg_imag = float32_set_sign(0, simd_data(desc)); + float32 neg_real = float32_chs(neg_imag); + + do { + uint64_t pg = g[(i - 1) >> 6]; + do { + float32 e0, e1, e2, e3; + + /* I holds the real index; J holds the imag index. */ + j = i - sizeof(float32); + i -= 2 * sizeof(float32); + + e0 = *(float32 *)(vn + H1_2(i)); + e1 = *(float32 *)(vm + H1_2(j)) ^ neg_real; + e2 = *(float32 *)(vn + H1_2(j)); + e3 = *(float32 *)(vm + H1_2(i)) ^ neg_imag; + + if (likely((pg >> (i & 63)) & 1)) { + *(float32 *)(vd + H1_2(i)) = float32_add(e0, e1, vs); + } + if (likely((pg >> (j & 63)) & 1)) { + *(float32 *)(vd + H1_2(j)) = float32_add(e2, e3, vs); + } + } while (i & 63); + } while (i != 0); +} + +void HELPER(sve_fcadd_d)(void *vd, void *vn, void *vm, void *vg, + void *vs, uint32_t desc) +{ + intptr_t j, i = simd_oprsz(desc); + uint64_t *g = vg; + float64 neg_imag = float64_set_sign(0, simd_data(desc)); + float64 neg_real = float64_chs(neg_imag); + + do { + uint64_t pg = g[(i - 1) >> 6]; + do { + float64 e0, e1, e2, e3; + + /* I holds the real index; J holds the imag index. */ + j = i - sizeof(float64); + i -= 2 * sizeof(float64); + + e0 = *(float64 *)(vn + H1_2(i)); + e1 = *(float64 *)(vm + H1_2(j)) ^ neg_real; + e2 = *(float64 *)(vn + H1_2(j)); + e3 = *(float64 *)(vm + H1_2(i)) ^ neg_imag; + + if (likely((pg >> (i & 63)) & 1)) { + *(float64 *)(vd + H1_2(i)) = float64_add(e0, e1, vs); + } + if (likely((pg >> (j & 63)) & 1)) { + *(float64 *)(vd + H1_2(j)) = float64_add(e2, e3, vs); + } + } while (i & 63); + } while (i != 0); +} + /* * Load contiguous data, protected by a governing predicate. */ diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ DO_FPCMP(FACGT, facgt) #undef DO_FPCMP +static bool trans_FCADD(DisasContext *s, arg_FCADD *a, uint32_t insn) +{ + static gen_helper_gvec_4_ptr * const fns[3] = { + gen_helper_sve_fcadd_h, + gen_helper_sve_fcadd_s, + gen_helper_sve_fcadd_d + }; + + if (a->esz == 0) { + return false; + } + if (sve_access_check(s)) { + unsigned vsz = vec_full_reg_size(s); + TCGv_ptr status = get_fpstatus_ptr(a->esz == MO_16); + tcg_gen_gvec_4_ptr(vec_full_reg_offset(s, a->rd), + vec_full_reg_offset(s, a->rn), + vec_full_reg_offset(s, a->rm), + pred_full_reg_offset(s, a->pg), + status, vsz, vsz, a->rot, fns[a->esz - 1]); + tcg_temp_free_ptr(status); + } + return true; +} + typedef void gen_helper_sve_fmla(TCGv_env, TCGv_ptr, TCGv_i32); static bool do_fmla(DisasContext *s, arg_rprrr_esz *a, gen_helper_sve_fmla *fn) diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ UMIN_zzi 00100101 .. 101 011 110 ........ ..... @rdn_i8u # SVE integer multiply immediate (unpredicated) MUL_zzi 00100101 .. 110 000 110 ........ ..... @rdn_i8s +# SVE floating-point complex add (predicated) +FCADD 01100100 esz:2 00000 rot:1 100 pg:3 rm:5 rd:5 \ + rn=%reg_movprfx + ### SVE FP Multiply-Add Indexed Group # SVE floating-point multiply-add (indexed) -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-30-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper-sve.h | 4 + target/arm/sve_helper.c | 162 +++++++++++++++++++++++++++++++++++++ target/arm/translate-sve.c | 37 +++++++++ target/arm/sve.decode | 4 + 4 files changed, 207 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_3(sve_fnmls_zpzzz_h, TCG_CALL_NO_RWG, void, env, ptr, i32) DEF_HELPER_FLAGS_3(sve_fnmls_zpzzz_s, TCG_CALL_NO_RWG, void, env, ptr, i32) DEF_HELPER_FLAGS_3(sve_fnmls_zpzzz_d, TCG_CALL_NO_RWG, void, env, ptr, i32) +DEF_HELPER_FLAGS_3(sve_fcmla_zpzzz_h, TCG_CALL_NO_RWG, void, env, ptr, i32) +DEF_HELPER_FLAGS_3(sve_fcmla_zpzzz_s, TCG_CALL_NO_RWG, void, env, ptr, i32) +DEF_HELPER_FLAGS_3(sve_fcmla_zpzzz_d, TCG_CALL_NO_RWG, void, env, ptr, i32) + DEF_HELPER_FLAGS_5(sve_ftmad_h, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_5(sve_ftmad_s, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_5(sve_ftmad_d, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ void HELPER(sve_fcadd_d)(void *vd, void *vn, void *vm, void *vg, } while (i != 0); } +/* + * FP Complex Multiply + */ + +QEMU_BUILD_BUG_ON(SIMD_DATA_SHIFT + 22 > 32); + +void HELPER(sve_fcmla_zpzzz_h)(CPUARMState *env, void *vg, uint32_t desc) +{ + intptr_t j, i = simd_oprsz(desc); + unsigned rd = extract32(desc, SIMD_DATA_SHIFT, 5); + unsigned rn = extract32(desc, SIMD_DATA_SHIFT + 5, 5); + unsigned rm = extract32(desc, SIMD_DATA_SHIFT + 10, 5); + unsigned ra = extract32(desc, SIMD_DATA_SHIFT + 15, 5); + unsigned rot = extract32(desc, SIMD_DATA_SHIFT + 20, 2); + bool flip = rot & 1; + float16 neg_imag, neg_real; + void *vd = &env->vfp.zregs[rd]; + void *vn = &env->vfp.zregs[rn]; + void *vm = &env->vfp.zregs[rm]; + void *va = &env->vfp.zregs[ra]; + uint64_t *g = vg; + + neg_imag = float16_set_sign(0, (rot & 2) != 0); + neg_real = float16_set_sign(0, rot == 1 || rot == 2); + + do { + uint64_t pg = g[(i - 1) >> 6]; + do { + float16 e1, e2, e3, e4, nr, ni, mr, mi, d; + + /* I holds the real index; J holds the imag index. */ + j = i - sizeof(float16); + i -= 2 * sizeof(float16); + + nr = *(float16 *)(vn + H1_2(i)); + ni = *(float16 *)(vn + H1_2(j)); + mr = *(float16 *)(vm + H1_2(i)); + mi = *(float16 *)(vm + H1_2(j)); + + e2 = (flip ? ni : nr); + e1 = (flip ? mi : mr) ^ neg_real; + e4 = e2; + e3 = (flip ? mr : mi) ^ neg_imag; + + if (likely((pg >> (i & 63)) & 1)) { + d = *(float16 *)(va + H1_2(i)); + d = float16_muladd(e2, e1, d, 0, &env->vfp.fp_status_f16); + *(float16 *)(vd + H1_2(i)) = d; + } + if (likely((pg >> (j & 63)) & 1)) { + d = *(float16 *)(va + H1_2(j)); + d = float16_muladd(e4, e3, d, 0, &env->vfp.fp_status_f16); + *(float16 *)(vd + H1_2(j)) = d; + } + } while (i & 63); + } while (i != 0); +} + +void HELPER(sve_fcmla_zpzzz_s)(CPUARMState *env, void *vg, uint32_t desc) +{ + intptr_t j, i = simd_oprsz(desc); + unsigned rd = extract32(desc, SIMD_DATA_SHIFT, 5); + unsigned rn = extract32(desc, SIMD_DATA_SHIFT + 5, 5); + unsigned rm = extract32(desc, SIMD_DATA_SHIFT + 10, 5); + unsigned ra = extract32(desc, SIMD_DATA_SHIFT + 15, 5); + unsigned rot = extract32(desc, SIMD_DATA_SHIFT + 20, 2); + bool flip = rot & 1; + float32 neg_imag, neg_real; + void *vd = &env->vfp.zregs[rd]; + void *vn = &env->vfp.zregs[rn]; + void *vm = &env->vfp.zregs[rm]; + void *va = &env->vfp.zregs[ra]; + uint64_t *g = vg; + + neg_imag = float32_set_sign(0, (rot & 2) != 0); + neg_real = float32_set_sign(0, rot == 1 || rot == 2); + + do { + uint64_t pg = g[(i - 1) >> 6]; + do { + float32 e1, e2, e3, e4, nr, ni, mr, mi, d; + + /* I holds the real index; J holds the imag index. */ + j = i - sizeof(float32); + i -= 2 * sizeof(float32); + + nr = *(float32 *)(vn + H1_2(i)); + ni = *(float32 *)(vn + H1_2(j)); + mr = *(float32 *)(vm + H1_2(i)); + mi = *(float32 *)(vm + H1_2(j)); + + e2 = (flip ? ni : nr); + e1 = (flip ? mi : mr) ^ neg_real; + e4 = e2; + e3 = (flip ? mr : mi) ^ neg_imag; + + if (likely((pg >> (i & 63)) & 1)) { + d = *(float32 *)(va + H1_2(i)); + d = float32_muladd(e2, e1, d, 0, &env->vfp.fp_status); + *(float32 *)(vd + H1_2(i)) = d; + } + if (likely((pg >> (j & 63)) & 1)) { + d = *(float32 *)(va + H1_2(j)); + d = float32_muladd(e4, e3, d, 0, &env->vfp.fp_status); + *(float32 *)(vd + H1_2(j)) = d; + } + } while (i & 63); + } while (i != 0); +} + +void HELPER(sve_fcmla_zpzzz_d)(CPUARMState *env, void *vg, uint32_t desc) +{ + intptr_t j, i = simd_oprsz(desc); + unsigned rd = extract32(desc, SIMD_DATA_SHIFT, 5); + unsigned rn = extract32(desc, SIMD_DATA_SHIFT + 5, 5); + unsigned rm = extract32(desc, SIMD_DATA_SHIFT + 10, 5); + unsigned ra = extract32(desc, SIMD_DATA_SHIFT + 15, 5); + unsigned rot = extract32(desc, SIMD_DATA_SHIFT + 20, 2); + bool flip = rot & 1; + float64 neg_imag, neg_real; + void *vd = &env->vfp.zregs[rd]; + void *vn = &env->vfp.zregs[rn]; + void *vm = &env->vfp.zregs[rm]; + void *va = &env->vfp.zregs[ra]; + uint64_t *g = vg; + + neg_imag = float64_set_sign(0, (rot & 2) != 0); + neg_real = float64_set_sign(0, rot == 1 || rot == 2); + + do { + uint64_t pg = g[(i - 1) >> 6]; + do { + float64 e1, e2, e3, e4, nr, ni, mr, mi, d; + + /* I holds the real index; J holds the imag index. */ + j = i - sizeof(float64); + i -= 2 * sizeof(float64); + + nr = *(float64 *)(vn + H1_2(i)); + ni = *(float64 *)(vn + H1_2(j)); + mr = *(float64 *)(vm + H1_2(i)); + mi = *(float64 *)(vm + H1_2(j)); + + e2 = (flip ? ni : nr); + e1 = (flip ? mi : mr) ^ neg_real; + e4 = e2; + e3 = (flip ? mr : mi) ^ neg_imag; + + if (likely((pg >> (i & 63)) & 1)) { + d = *(float64 *)(va + H1_2(i)); + d = float64_muladd(e2, e1, d, 0, &env->vfp.fp_status); + *(float64 *)(vd + H1_2(i)) = d; + } + if (likely((pg >> (j & 63)) & 1)) { + d = *(float64 *)(va + H1_2(j)); + d = float64_muladd(e4, e3, d, 0, &env->vfp.fp_status); + *(float64 *)(vd + H1_2(j)) = d; + } + } while (i & 63); + } while (i != 0); +} + /* * Load contiguous data, protected by a governing predicate. */ diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ DO_FMLA(FNMLS_zpzzz, fnmls_zpzzz) #undef DO_FMLA +static bool trans_FCMLA_zpzzz(DisasContext *s, + arg_FCMLA_zpzzz *a, uint32_t insn) +{ + static gen_helper_sve_fmla * const fns[3] = { + gen_helper_sve_fcmla_zpzzz_h, + gen_helper_sve_fcmla_zpzzz_s, + gen_helper_sve_fcmla_zpzzz_d, + }; + + if (a->esz == 0) { + return false; + } + if (sve_access_check(s)) { + unsigned vsz = vec_full_reg_size(s); + unsigned desc; + TCGv_i32 t_desc; + TCGv_ptr pg = tcg_temp_new_ptr(); + + /* We would need 7 operands to pass these arguments "properly". + * So we encode all the register numbers into the descriptor. + */ + desc = deposit32(a->rd, 5, 5, a->rn); + desc = deposit32(desc, 10, 5, a->rm); + desc = deposit32(desc, 15, 5, a->ra); + desc = deposit32(desc, 20, 2, a->rot); + desc = sextract32(desc, 0, 22); + desc = simd_desc(vsz, vsz, desc); + + t_desc = tcg_const_i32(desc); + tcg_gen_addi_ptr(pg, cpu_env, pred_full_reg_offset(s, a->pg)); + fns[a->esz - 1](cpu_env, pg, t_desc); + tcg_temp_free_i32(t_desc); + tcg_temp_free_ptr(pg); + } + return true; +} + /* *** SVE Floating Point Unary Operations Predicated Group */ diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ MUL_zzi 00100101 .. 110 000 110 ........ ..... @rdn_i8s FCADD 01100100 esz:2 00000 rot:1 100 pg:3 rm:5 rd:5 \ rn=%reg_movprfx +# SVE floating-point complex multiply-add (predicated) +FCMLA_zpzzz 01100100 esz:2 0 rm:5 0 rot:2 pg:3 rn:5 rd:5 \ + ra=%reg_movprfx + ### SVE FP Multiply-Add Indexed Group # SVE floating-point multiply-add (indexed) -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> For aa64 advsimd, we had been passing the pre-indexed vector. However, sve applies the index to each 128-bit segment, so we need to pass in the index separately. For aa32 advsimd, the fp32 operation always has index 0, but we failed to interpret the fp16 index correctly. Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Message-id: 20180627043328.11531-31-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/translate-a64.c | 21 ++++++++++++--------- target/arm/translate.c | 32 +++++++++++++++++++++++--------- target/arm/vec_helper.c | 10 ++++++---- 3 files changed, 41 insertions(+), 22 deletions(-) diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -XXX,XX +XXX,XX @@ static void disas_simd_indexed(DisasContext *s, uint32_t insn) case 0x13: /* FCMLA #90 */ case 0x15: /* FCMLA #180 */ case 0x17: /* FCMLA #270 */ - tcg_gen_gvec_3_ptr(vec_full_reg_offset(s, rd), - vec_full_reg_offset(s, rn), - vec_reg_offset(s, rm, index, size), fpst, - is_q ? 16 : 8, vec_full_reg_size(s), - extract32(insn, 13, 2), /* rot */ - size == MO_64 - ? gen_helper_gvec_fcmlas_idx - : gen_helper_gvec_fcmlah_idx); - tcg_temp_free_ptr(fpst); + { + int rot = extract32(insn, 13, 2); + int data = (index << 2) | rot; + tcg_gen_gvec_3_ptr(vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), + vec_full_reg_offset(s, rm), fpst, + is_q ? 16 : 8, vec_full_reg_size(s), data, + size == MO_64 + ? gen_helper_gvec_fcmlas_idx + : gen_helper_gvec_fcmlah_idx); + tcg_temp_free_ptr(fpst); + } return; } diff --git a/target/arm/translate.c b/target/arm/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate.c +++ b/target/arm/translate.c @@ -XXX,XX +XXX,XX @@ static int disas_neon_insn_3same_ext(DisasContext *s, uint32_t insn) static int disas_neon_insn_2reg_scalar_ext(DisasContext *s, uint32_t insn) { - int rd, rn, rm, rot, size, opr_sz; + gen_helper_gvec_3_ptr *fn_gvec_ptr; + int rd, rn, rm, opr_sz, data; TCGv_ptr fpst; bool q; q = extract32(insn, 6, 1); VFP_DREG_D(rd, insn); VFP_DREG_N(rn, insn); - VFP_DREG_M(rm, insn); if ((rd | rn) & q) { return 1; } if ((insn & 0xff000f10) == 0xfe000800) { /* VCMLA (indexed) -- 1111 1110 S.RR .... .... 1000 ...0 .... */ - rot = extract32(insn, 20, 2); - size = extract32(insn, 23, 1); - if (!arm_dc_feature(s, ARM_FEATURE_V8_FCMA) - || (!size && !arm_dc_feature(s, ARM_FEATURE_V8_FP16))) { + int rot = extract32(insn, 20, 2); + int size = extract32(insn, 23, 1); + int index; + + if (!arm_dc_feature(s, ARM_FEATURE_V8_FCMA)) { return 1; } + if (size == 0) { + if (!arm_dc_feature(s, ARM_FEATURE_V8_FP16)) { + return 1; + } + /* For fp16, rm is just Vm, and index is M. */ + rm = extract32(insn, 0, 4); + index = extract32(insn, 5, 1); + } else { + /* For fp32, rm is the usual M:Vm, and index is 0. */ + VFP_DREG_M(rm, insn); + index = 0; + } + data = (index << 2) | rot; + fn_gvec_ptr = (size ? gen_helper_gvec_fcmlas_idx + : gen_helper_gvec_fcmlah_idx); } else { return 1; } @@ -XXX,XX +XXX,XX @@ static int disas_neon_insn_2reg_scalar_ext(DisasContext *s, uint32_t insn) tcg_gen_gvec_3_ptr(vfp_reg_offset(1, rd), vfp_reg_offset(1, rn), vfp_reg_offset(1, rm), fpst, - opr_sz, opr_sz, rot, - size ? gen_helper_gvec_fcmlas_idx - : gen_helper_gvec_fcmlah_idx); + opr_sz, opr_sz, data, fn_gvec_ptr); tcg_temp_free_ptr(fpst); return 0; } diff --git a/target/arm/vec_helper.c b/target/arm/vec_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/vec_helper.c +++ b/target/arm/vec_helper.c @@ -XXX,XX +XXX,XX @@ void HELPER(gvec_fcmlah_idx)(void *vd, void *vn, void *vm, float_status *fpst = vfpst; intptr_t flip = extract32(desc, SIMD_DATA_SHIFT, 1); uint32_t neg_imag = extract32(desc, SIMD_DATA_SHIFT + 1, 1); + intptr_t index = extract32(desc, SIMD_DATA_SHIFT + 2, 2); uint32_t neg_real = flip ^ neg_imag; uintptr_t i; - float16 e1 = m[H2(flip)]; - float16 e3 = m[H2(1 - flip)]; + float16 e1 = m[H2(2 * index + flip)]; + float16 e3 = m[H2(2 * index + 1 - flip)]; /* Shift boolean to the sign bit so we can xor to negate. */ neg_real <<= 15; @@ -XXX,XX +XXX,XX @@ void HELPER(gvec_fcmlas_idx)(void *vd, void *vn, void *vm, float_status *fpst = vfpst; intptr_t flip = extract32(desc, SIMD_DATA_SHIFT, 1); uint32_t neg_imag = extract32(desc, SIMD_DATA_SHIFT + 1, 1); + intptr_t index = extract32(desc, SIMD_DATA_SHIFT + 2, 2); uint32_t neg_real = flip ^ neg_imag; uintptr_t i; - float32 e1 = m[H4(flip)]; - float32 e3 = m[H4(1 - flip)]; + float32 e1 = m[H4(2 * index + flip)]; + float32 e3 = m[H4(2 * index + 1 - flip)]; /* Shift boolean to the sign bit so we can xor to negate. */ neg_real <<= 31; -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Enhance the existing helpers to support SVE, which takes the index from each 128-bit segment. The change has no effect for AdvSIMD, since there is only one such segment. Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Message-id: 20180627043328.11531-32-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/translate-sve.c | 23 ++++++++++++++++++ target/arm/vec_helper.c | 50 +++++++++++++++++++++++--------------- target/arm/sve.decode | 6 +++++ 3 files changed, 59 insertions(+), 20 deletions(-) diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ static bool trans_FCMLA_zpzzz(DisasContext *s, return true; } +static bool trans_FCMLA_zzxz(DisasContext *s, arg_FCMLA_zzxz *a, uint32_t insn) +{ + static gen_helper_gvec_3_ptr * const fns[2] = { + gen_helper_gvec_fcmlah_idx, + gen_helper_gvec_fcmlas_idx, + }; + + tcg_debug_assert(a->esz == 1 || a->esz == 2); + tcg_debug_assert(a->rd == a->ra); + if (sve_access_check(s)) { + unsigned vsz = vec_full_reg_size(s); + TCGv_ptr status = get_fpstatus_ptr(a->esz == MO_16); + tcg_gen_gvec_3_ptr(vec_full_reg_offset(s, a->rd), + vec_full_reg_offset(s, a->rn), + vec_full_reg_offset(s, a->rm), + status, vsz, vsz, + a->index * 4 + a->rot, + fns[a->esz - 1]); + tcg_temp_free_ptr(status); + } + return true; +} + /* *** SVE Floating Point Unary Operations Predicated Group */ diff --git a/target/arm/vec_helper.c b/target/arm/vec_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/vec_helper.c +++ b/target/arm/vec_helper.c @@ -XXX,XX +XXX,XX @@ void HELPER(gvec_fcmlah_idx)(void *vd, void *vn, void *vm, uint32_t neg_imag = extract32(desc, SIMD_DATA_SHIFT + 1, 1); intptr_t index = extract32(desc, SIMD_DATA_SHIFT + 2, 2); uint32_t neg_real = flip ^ neg_imag; - uintptr_t i; - float16 e1 = m[H2(2 * index + flip)]; - float16 e3 = m[H2(2 * index + 1 - flip)]; + intptr_t elements = opr_sz / sizeof(float16); + intptr_t eltspersegment = 16 / sizeof(float16); + intptr_t i, j; /* Shift boolean to the sign bit so we can xor to negate. */ neg_real <<= 15; neg_imag <<= 15; - e1 ^= neg_real; - e3 ^= neg_imag; - for (i = 0; i < opr_sz / 2; i += 2) { - float16 e2 = n[H2(i + flip)]; - float16 e4 = e2; + for (i = 0; i < elements; i += eltspersegment) { + float16 mr = m[H2(i + 2 * index + 0)]; + float16 mi = m[H2(i + 2 * index + 1)]; + float16 e1 = neg_real ^ (flip ? mi : mr); + float16 e3 = neg_imag ^ (flip ? mr : mi); - d[H2(i)] = float16_muladd(e2, e1, d[H2(i)], 0, fpst); - d[H2(i + 1)] = float16_muladd(e4, e3, d[H2(i + 1)], 0, fpst); + for (j = i; j < i + eltspersegment; j += 2) { + float16 e2 = n[H2(j + flip)]; + float16 e4 = e2; + + d[H2(j)] = float16_muladd(e2, e1, d[H2(j)], 0, fpst); + d[H2(j + 1)] = float16_muladd(e4, e3, d[H2(j + 1)], 0, fpst); + } } clear_tail(d, opr_sz, simd_maxsz(desc)); } @@ -XXX,XX +XXX,XX @@ void HELPER(gvec_fcmlas_idx)(void *vd, void *vn, void *vm, uint32_t neg_imag = extract32(desc, SIMD_DATA_SHIFT + 1, 1); intptr_t index = extract32(desc, SIMD_DATA_SHIFT + 2, 2); uint32_t neg_real = flip ^ neg_imag; - uintptr_t i; - float32 e1 = m[H4(2 * index + flip)]; - float32 e3 = m[H4(2 * index + 1 - flip)]; + intptr_t elements = opr_sz / sizeof(float32); + intptr_t eltspersegment = 16 / sizeof(float32); + intptr_t i, j; /* Shift boolean to the sign bit so we can xor to negate. */ neg_real <<= 31; neg_imag <<= 31; - e1 ^= neg_real; - e3 ^= neg_imag; - for (i = 0; i < opr_sz / 4; i += 2) { - float32 e2 = n[H4(i + flip)]; - float32 e4 = e2; + for (i = 0; i < elements; i += eltspersegment) { + float32 mr = m[H4(i + 2 * index + 0)]; + float32 mi = m[H4(i + 2 * index + 1)]; + float32 e1 = neg_real ^ (flip ? mi : mr); + float32 e3 = neg_imag ^ (flip ? mr : mi); - d[H4(i)] = float32_muladd(e2, e1, d[H4(i)], 0, fpst); - d[H4(i + 1)] = float32_muladd(e4, e3, d[H4(i + 1)], 0, fpst); + for (j = i; j < i + eltspersegment; j += 2) { + float32 e2 = n[H4(j + flip)]; + float32 e4 = e2; + + d[H4(j)] = float32_muladd(e2, e1, d[H4(j)], 0, fpst); + d[H4(j + 1)] = float32_muladd(e4, e3, d[H4(j + 1)], 0, fpst); + } } clear_tail(d, opr_sz, simd_maxsz(desc)); } diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ FCADD 01100100 esz:2 00000 rot:1 100 pg:3 rm:5 rd:5 \ FCMLA_zpzzz 01100100 esz:2 0 rm:5 0 rot:2 pg:3 rn:5 rd:5 \ ra=%reg_movprfx +# SVE floating-point complex multiply-add (indexed) +FCMLA_zzxz 01100100 10 1 index:2 rm:3 0001 rot:2 rn:5 rd:5 \ + ra=%reg_movprfx esz=1 +FCMLA_zzxz 01100100 11 1 index:1 rm:4 0001 rot:2 rn:5 rd:5 \ + ra=%reg_movprfx esz=2 + ### SVE FP Multiply-Add Indexed Group # SVE floating-point multiply-add (indexed) -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-33-richard.henderson@linaro.org [PMM: moved 'ra=%reg_movprfx' here from following patch] Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper.h | 5 +++ target/arm/translate-sve.c | 17 ++++++++++ target/arm/vec_helper.c | 67 ++++++++++++++++++++++++++++++++++++++ target/arm/sve.decode | 3 ++ 4 files changed, 92 insertions(+) diff --git a/target/arm/helper.h b/target/arm/helper.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper.h +++ b/target/arm/helper.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_5(gvec_qrdmlah_s32, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_5(gvec_qrdmlsh_s32, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_sdot_b, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_udot_b, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_sdot_h, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_udot_h, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) + DEF_HELPER_FLAGS_5(gvec_fcaddh, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_5(gvec_fcadds, TCG_CALL_NO_RWG, diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ DO_ZZI(UMIN, umin) #undef DO_ZZI +static bool trans_DOT_zzz(DisasContext *s, arg_DOT_zzz *a, uint32_t insn) +{ + static gen_helper_gvec_3 * const fns[2][2] = { + { gen_helper_gvec_sdot_b, gen_helper_gvec_sdot_h }, + { gen_helper_gvec_udot_b, gen_helper_gvec_udot_h } + }; + + if (sve_access_check(s)) { + unsigned vsz = vec_full_reg_size(s); + tcg_gen_gvec_3_ool(vec_full_reg_offset(s, a->rd), + vec_full_reg_offset(s, a->rn), + vec_full_reg_offset(s, a->rm), + vsz, vsz, 0, fns[a->u][a->sz]); + } + return true; +} + /* *** SVE Floating Point Multiply-Add Indexed Group */ diff --git a/target/arm/vec_helper.c b/target/arm/vec_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/vec_helper.c +++ b/target/arm/vec_helper.c @@ -XXX,XX +XXX,XX @@ void HELPER(gvec_qrdmlsh_s32)(void *vd, void *vn, void *vm, clear_tail(d, opr_sz, simd_maxsz(desc)); } +/* Integer 8 and 16-bit dot-product. + * + * Note that for the loops herein, host endianness does not matter + * with respect to the ordering of data within the 64-bit lanes. + * All elements are treated equally, no matter where they are. + */ + +void HELPER(gvec_sdot_b)(void *vd, void *vn, void *vm, uint32_t desc) +{ + intptr_t i, opr_sz = simd_oprsz(desc); + uint32_t *d = vd; + int8_t *n = vn, *m = vm; + + for (i = 0; i < opr_sz / 4; ++i) { + d[i] += n[i * 4 + 0] * m[i * 4 + 0] + + n[i * 4 + 1] * m[i * 4 + 1] + + n[i * 4 + 2] * m[i * 4 + 2] + + n[i * 4 + 3] * m[i * 4 + 3]; + } + clear_tail(d, opr_sz, simd_maxsz(desc)); +} + +void HELPER(gvec_udot_b)(void *vd, void *vn, void *vm, uint32_t desc) +{ + intptr_t i, opr_sz = simd_oprsz(desc); + uint32_t *d = vd; + uint8_t *n = vn, *m = vm; + + for (i = 0; i < opr_sz / 4; ++i) { + d[i] += n[i * 4 + 0] * m[i * 4 + 0] + + n[i * 4 + 1] * m[i * 4 + 1] + + n[i * 4 + 2] * m[i * 4 + 2] + + n[i * 4 + 3] * m[i * 4 + 3]; + } + clear_tail(d, opr_sz, simd_maxsz(desc)); +} + +void HELPER(gvec_sdot_h)(void *vd, void *vn, void *vm, uint32_t desc) +{ + intptr_t i, opr_sz = simd_oprsz(desc); + uint64_t *d = vd; + int16_t *n = vn, *m = vm; + + for (i = 0; i < opr_sz / 8; ++i) { + d[i] += (int64_t)n[i * 4 + 0] * m[i * 4 + 0] + + (int64_t)n[i * 4 + 1] * m[i * 4 + 1] + + (int64_t)n[i * 4 + 2] * m[i * 4 + 2] + + (int64_t)n[i * 4 + 3] * m[i * 4 + 3]; + } + clear_tail(d, opr_sz, simd_maxsz(desc)); +} + +void HELPER(gvec_udot_h)(void *vd, void *vn, void *vm, uint32_t desc) +{ + intptr_t i, opr_sz = simd_oprsz(desc); + uint64_t *d = vd; + uint16_t *n = vn, *m = vm; + + for (i = 0; i < opr_sz / 8; ++i) { + d[i] += (uint64_t)n[i * 4 + 0] * m[i * 4 + 0] + + (uint64_t)n[i * 4 + 1] * m[i * 4 + 1] + + (uint64_t)n[i * 4 + 2] * m[i * 4 + 2] + + (uint64_t)n[i * 4 + 3] * m[i * 4 + 3]; + } + clear_tail(d, opr_sz, simd_maxsz(desc)); +} + void HELPER(gvec_fcaddh)(void *vd, void *vn, void *vm, void *vfpst, uint32_t desc) { diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ UMIN_zzi 00100101 .. 101 011 110 ........ ..... @rdn_i8u # SVE integer multiply immediate (unpredicated) MUL_zzi 00100101 .. 110 000 110 ........ ..... @rdn_i8s +# SVE integer dot product (unpredicated) +DOT_zzz 01000100 1 sz:1 0 rm:5 00000 u:1 rn:5 rd:5 ra=%reg_movprfx + # SVE floating-point complex add (predicated) FCADD 01100100 esz:2 00000 rot:1 100 pg:3 rm:5 rd:5 \ rn=%reg_movprfx -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Message-id: 20180627043328.11531-34-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper.h | 5 ++ target/arm/translate-sve.c | 18 ++++++ target/arm/vec_helper.c | 124 +++++++++++++++++++++++++++++++++++++ target/arm/sve.decode | 6 ++ 4 files changed, 153 insertions(+) diff --git a/target/arm/helper.h b/target/arm/helper.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper.h +++ b/target/arm/helper.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_4(gvec_udot_b, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(gvec_sdot_h, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(gvec_udot_h, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_sdot_idx_b, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_udot_idx_b, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_sdot_idx_h, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_udot_idx_h, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) + DEF_HELPER_FLAGS_5(gvec_fcaddh, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_5(gvec_fcadds, TCG_CALL_NO_RWG, diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -XXX,XX +XXX,XX @@ static bool trans_DOT_zzz(DisasContext *s, arg_DOT_zzz *a, uint32_t insn) return true; } +static bool trans_DOT_zzx(DisasContext *s, arg_DOT_zzx *a, uint32_t insn) +{ + static gen_helper_gvec_3 * const fns[2][2] = { + { gen_helper_gvec_sdot_idx_b, gen_helper_gvec_sdot_idx_h }, + { gen_helper_gvec_udot_idx_b, gen_helper_gvec_udot_idx_h } + }; + + if (sve_access_check(s)) { + unsigned vsz = vec_full_reg_size(s); + tcg_gen_gvec_3_ool(vec_full_reg_offset(s, a->rd), + vec_full_reg_offset(s, a->rn), + vec_full_reg_offset(s, a->rm), + vsz, vsz, a->index, fns[a->u][a->sz]); + } + return true; +} + + /* *** SVE Floating Point Multiply-Add Indexed Group */ diff --git a/target/arm/vec_helper.c b/target/arm/vec_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/vec_helper.c +++ b/target/arm/vec_helper.c @@ -XXX,XX +XXX,XX @@ void HELPER(gvec_udot_h)(void *vd, void *vn, void *vm, uint32_t desc) clear_tail(d, opr_sz, simd_maxsz(desc)); } +void HELPER(gvec_sdot_idx_b)(void *vd, void *vn, void *vm, uint32_t desc) +{ + intptr_t i, segend, opr_sz = simd_oprsz(desc), opr_sz_4 = opr_sz / 4; + intptr_t index = simd_data(desc); + uint32_t *d = vd; + int8_t *n = vn; + int8_t *m_indexed = (int8_t *)vm + index * 4; + + /* Notice the special case of opr_sz == 8, from aa64/aa32 advsimd. + * Otherwise opr_sz is a multiple of 16. + */ + segend = MIN(4, opr_sz_4); + i = 0; + do { + int8_t m0 = m_indexed[i * 4 + 0]; + int8_t m1 = m_indexed[i * 4 + 1]; + int8_t m2 = m_indexed[i * 4 + 2]; + int8_t m3 = m_indexed[i * 4 + 3]; + + do { + d[i] += n[i * 4 + 0] * m0 + + n[i * 4 + 1] * m1 + + n[i * 4 + 2] * m2 + + n[i * 4 + 3] * m3; + } while (++i < segend); + segend = i + 4; + } while (i < opr_sz_4); + + clear_tail(d, opr_sz, simd_maxsz(desc)); +} + +void HELPER(gvec_udot_idx_b)(void *vd, void *vn, void *vm, uint32_t desc) +{ + intptr_t i, segend, opr_sz = simd_oprsz(desc), opr_sz_4 = opr_sz / 4; + intptr_t index = simd_data(desc); + uint32_t *d = vd; + uint8_t *n = vn; + uint8_t *m_indexed = (uint8_t *)vm + index * 4; + + /* Notice the special case of opr_sz == 8, from aa64/aa32 advsimd. + * Otherwise opr_sz is a multiple of 16. + */ + segend = MIN(4, opr_sz_4); + i = 0; + do { + uint8_t m0 = m_indexed[i * 4 + 0]; + uint8_t m1 = m_indexed[i * 4 + 1]; + uint8_t m2 = m_indexed[i * 4 + 2]; + uint8_t m3 = m_indexed[i * 4 + 3]; + + do { + d[i] += n[i * 4 + 0] * m0 + + n[i * 4 + 1] * m1 + + n[i * 4 + 2] * m2 + + n[i * 4 + 3] * m3; + } while (++i < segend); + segend = i + 4; + } while (i < opr_sz_4); + + clear_tail(d, opr_sz, simd_maxsz(desc)); +} + +void HELPER(gvec_sdot_idx_h)(void *vd, void *vn, void *vm, uint32_t desc) +{ + intptr_t i, opr_sz = simd_oprsz(desc), opr_sz_8 = opr_sz / 8; + intptr_t index = simd_data(desc); + uint64_t *d = vd; + int16_t *n = vn; + int16_t *m_indexed = (int16_t *)vm + index * 4; + + /* This is supported by SVE only, so opr_sz is always a multiple of 16. + * Process the entire segment all at once, writing back the results + * only after we've consumed all of the inputs. + */ + for (i = 0; i < opr_sz_8 ; i += 2) { + uint64_t d0, d1; + + d0 = n[i * 4 + 0] * (int64_t)m_indexed[i * 4 + 0]; + d0 += n[i * 4 + 1] * (int64_t)m_indexed[i * 4 + 1]; + d0 += n[i * 4 + 2] * (int64_t)m_indexed[i * 4 + 2]; + d0 += n[i * 4 + 3] * (int64_t)m_indexed[i * 4 + 3]; + d1 = n[i * 4 + 4] * (int64_t)m_indexed[i * 4 + 0]; + d1 += n[i * 4 + 5] * (int64_t)m_indexed[i * 4 + 1]; + d1 += n[i * 4 + 6] * (int64_t)m_indexed[i * 4 + 2]; + d1 += n[i * 4 + 7] * (int64_t)m_indexed[i * 4 + 3]; + + d[i + 0] += d0; + d[i + 1] += d1; + } + + clear_tail(d, opr_sz, simd_maxsz(desc)); +} + +void HELPER(gvec_udot_idx_h)(void *vd, void *vn, void *vm, uint32_t desc) +{ + intptr_t i, opr_sz = simd_oprsz(desc), opr_sz_8 = opr_sz / 8; + intptr_t index = simd_data(desc); + uint64_t *d = vd; + uint16_t *n = vn; + uint16_t *m_indexed = (uint16_t *)vm + index * 4; + + /* This is supported by SVE only, so opr_sz is always a multiple of 16. + * Process the entire segment all at once, writing back the results + * only after we've consumed all of the inputs. + */ + for (i = 0; i < opr_sz_8 ; i += 2) { + uint64_t d0, d1; + + d0 = n[i * 4 + 0] * (uint64_t)m_indexed[i * 4 + 0]; + d0 += n[i * 4 + 1] * (uint64_t)m_indexed[i * 4 + 1]; + d0 += n[i * 4 + 2] * (uint64_t)m_indexed[i * 4 + 2]; + d0 += n[i * 4 + 3] * (uint64_t)m_indexed[i * 4 + 3]; + d1 = n[i * 4 + 4] * (uint64_t)m_indexed[i * 4 + 0]; + d1 += n[i * 4 + 5] * (uint64_t)m_indexed[i * 4 + 1]; + d1 += n[i * 4 + 6] * (uint64_t)m_indexed[i * 4 + 2]; + d1 += n[i * 4 + 7] * (uint64_t)m_indexed[i * 4 + 3]; + + d[i + 0] += d0; + d[i + 1] += d1; + } + + clear_tail(d, opr_sz, simd_maxsz(desc)); +} + void HELPER(gvec_fcaddh)(void *vd, void *vn, void *vm, void *vfpst, uint32_t desc) { diff --git a/target/arm/sve.decode b/target/arm/sve.decode index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -XXX,XX +XXX,XX @@ MUL_zzi 00100101 .. 110 000 110 ........ ..... @rdn_i8s # SVE integer dot product (unpredicated) DOT_zzz 01000100 1 sz:1 0 rm:5 00000 u:1 rn:5 rd:5 ra=%reg_movprfx +# SVE integer dot product (indexed) +DOT_zzx 01000100 101 index:2 rm:3 00000 u:1 rn:5 rd:5 \ + sz=0 ra=%reg_movprfx +DOT_zzx 01000100 111 index:1 rm:4 00000 u:1 rn:5 rd:5 \ + sz=1 ra=%reg_movprfx + # SVE floating-point complex add (predicated) FCADD 01100100 esz:2 00000 rot:1 100 pg:3 rm:5 rd:5 \ rn=%reg_movprfx -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Enable ARM_FEATURE_SVE for the generic "max" cpu. Tested-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-35-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- linux-user/elfload.c | 1 + target/arm/cpu.c | 7 +++++++ target/arm/cpu64.c | 1 + 3 files changed, 9 insertions(+) diff --git a/linux-user/elfload.c b/linux-user/elfload.c index XXXXXXX..XXXXXXX 100644 --- a/linux-user/elfload.c +++ b/linux-user/elfload.c @@ -XXX,XX +XXX,XX @@ static uint32_t get_elf_hwcap(void) GET_FEATURE(ARM_FEATURE_V8_ATOMICS, ARM_HWCAP_A64_ATOMICS); GET_FEATURE(ARM_FEATURE_V8_RDM, ARM_HWCAP_A64_ASIMDRDM); GET_FEATURE(ARM_FEATURE_V8_FCMA, ARM_HWCAP_A64_FCMA); + GET_FEATURE(ARM_FEATURE_SVE, ARM_HWCAP_A64_SVE); #undef GET_FEATURE return hwcaps; diff --git a/target/arm/cpu.c b/target/arm/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/cpu.c +++ b/target/arm/cpu.c @@ -XXX,XX +XXX,XX @@ static void arm_cpu_reset(CPUState *s) env->cp15.sctlr_el[1] |= SCTLR_UCT | SCTLR_UCI | SCTLR_DZE; /* and to the FP/Neon instructions */ env->cp15.cpacr_el1 = deposit64(env->cp15.cpacr_el1, 20, 2, 3); + /* and to the SVE instructions */ + env->cp15.cpacr_el1 = deposit64(env->cp15.cpacr_el1, 16, 2, 3); + env->cp15.cptr_el[3] |= CPTR_EZ; + /* with maximum vector length */ + env->vfp.zcr_el[1] = ARM_MAX_VQ - 1; + env->vfp.zcr_el[2] = ARM_MAX_VQ - 1; + env->vfp.zcr_el[3] = ARM_MAX_VQ - 1; #else /* Reset into the highest available EL */ if (arm_feature(env, ARM_FEATURE_EL3)) { diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/cpu64.c +++ b/target/arm/cpu64.c @@ -XXX,XX +XXX,XX @@ static void aarch64_max_initfn(Object *obj) set_feature(&cpu->env, ARM_FEATURE_V8_RDM); set_feature(&cpu->env, ARM_FEATURE_V8_FP16); set_feature(&cpu->env, ARM_FEATURE_V8_FCMA); + set_feature(&cpu->env, ARM_FEATURE_SVE); /* For usermode -cpu max we can use a larger and more efficient DCZ * blocksize since we don't have to follow what the hardware does. */ -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> We've already added the helpers with an SVE patch, all that remains is to wire up the aa64 and aa32 translators. Enable the feature within -cpu max for CONFIG_USER_ONLY. Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180627043328.11531-36-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/cpu.h | 1 + linux-user/elfload.c | 1 + target/arm/cpu.c | 1 + target/arm/cpu64.c | 1 + target/arm/translate-a64.c | 36 +++++++++++++++++++ target/arm/translate.c | 74 +++++++++++++++++++++++++++----------- 6 files changed, 93 insertions(+), 21 deletions(-) diff --git a/target/arm/cpu.h b/target/arm/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -XXX,XX +XXX,XX @@ enum arm_features { ARM_FEATURE_V8_SM4, /* implements SM4 part of v8 Crypto Extensions */ ARM_FEATURE_V8_ATOMICS, /* ARMv8.1-Atomics feature */ ARM_FEATURE_V8_RDM, /* implements v8.1 simd round multiply */ + ARM_FEATURE_V8_DOTPROD, /* implements v8.2 simd dot product */ ARM_FEATURE_V8_FP16, /* implements v8.2 half-precision float */ ARM_FEATURE_V8_FCMA, /* has complex number part of v8.3 extensions. */ ARM_FEATURE_M_MAIN, /* M profile Main Extension */ diff --git a/linux-user/elfload.c b/linux-user/elfload.c index XXXXXXX..XXXXXXX 100644 --- a/linux-user/elfload.c +++ b/linux-user/elfload.c @@ -XXX,XX +XXX,XX @@ static uint32_t get_elf_hwcap(void) ARM_HWCAP_A64_FPHP | ARM_HWCAP_A64_ASIMDHP); GET_FEATURE(ARM_FEATURE_V8_ATOMICS, ARM_HWCAP_A64_ATOMICS); GET_FEATURE(ARM_FEATURE_V8_RDM, ARM_HWCAP_A64_ASIMDRDM); + GET_FEATURE(ARM_FEATURE_V8_DOTPROD, ARM_HWCAP_A64_ASIMDDP); GET_FEATURE(ARM_FEATURE_V8_FCMA, ARM_HWCAP_A64_FCMA); GET_FEATURE(ARM_FEATURE_SVE, ARM_HWCAP_A64_SVE); #undef GET_FEATURE diff --git a/target/arm/cpu.c b/target/arm/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/cpu.c +++ b/target/arm/cpu.c @@ -XXX,XX +XXX,XX @@ static void arm_max_initfn(Object *obj) set_feature(&cpu->env, ARM_FEATURE_V8_PMULL); set_feature(&cpu->env, ARM_FEATURE_CRC); set_feature(&cpu->env, ARM_FEATURE_V8_RDM); + set_feature(&cpu->env, ARM_FEATURE_V8_DOTPROD); set_feature(&cpu->env, ARM_FEATURE_V8_FCMA); #endif } diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/cpu64.c +++ b/target/arm/cpu64.c @@ -XXX,XX +XXX,XX @@ static void aarch64_max_initfn(Object *obj) set_feature(&cpu->env, ARM_FEATURE_CRC); set_feature(&cpu->env, ARM_FEATURE_V8_ATOMICS); set_feature(&cpu->env, ARM_FEATURE_V8_RDM); + set_feature(&cpu->env, ARM_FEATURE_V8_DOTPROD); set_feature(&cpu->env, ARM_FEATURE_V8_FP16); set_feature(&cpu->env, ARM_FEATURE_V8_FCMA); set_feature(&cpu->env, ARM_FEATURE_SVE); diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -XXX,XX +XXX,XX @@ static void gen_gvec_op3(DisasContext *s, bool is_q, int rd, vec_full_reg_size(s), gvec_op); } +/* Expand a 3-operand operation using an out-of-line helper. */ +static void gen_gvec_op3_ool(DisasContext *s, bool is_q, int rd, + int rn, int rm, int data, gen_helper_gvec_3 *fn) +{ + tcg_gen_gvec_3_ool(vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), + vec_full_reg_offset(s, rm), + is_q ? 16 : 8, vec_full_reg_size(s), data, fn); +} + /* Expand a 3-operand + env pointer operation using * an out-of-line helper. */ @@ -XXX,XX +XXX,XX @@ static void disas_simd_three_reg_same_extra(DisasContext *s, uint32_t insn) } feature = ARM_FEATURE_V8_RDM; break; + case 0x02: /* SDOT (vector) */ + case 0x12: /* UDOT (vector) */ + if (size != MO_32) { + unallocated_encoding(s); + return; + } + feature = ARM_FEATURE_V8_DOTPROD; + break; case 0x8: /* FCMLA, #0 */ case 0x9: /* FCMLA, #90 */ case 0xa: /* FCMLA, #180 */ @@ -XXX,XX +XXX,XX @@ static void disas_simd_three_reg_same_extra(DisasContext *s, uint32_t insn) } return; + case 0x2: /* SDOT / UDOT */ + gen_gvec_op3_ool(s, is_q, rd, rn, rm, 0, + u ? gen_helper_gvec_udot_b : gen_helper_gvec_sdot_b); + return; + case 0x8: /* FCMLA, #0 */ case 0x9: /* FCMLA, #90 */ case 0xa: /* FCMLA, #180 */ @@ -XXX,XX +XXX,XX @@ static void disas_simd_indexed(DisasContext *s, uint32_t insn) return; } break; + case 0x0e: /* SDOT */ + case 0x1e: /* UDOT */ + if (size != MO_32 || !arm_dc_feature(s, ARM_FEATURE_V8_DOTPROD)) { + unallocated_encoding(s); + return; + } + break; case 0x11: /* FCMLA #0 */ case 0x13: /* FCMLA #90 */ case 0x15: /* FCMLA #180 */ @@ -XXX,XX +XXX,XX @@ static void disas_simd_indexed(DisasContext *s, uint32_t insn) } switch (16 * u + opcode) { + case 0x0e: /* SDOT */ + case 0x1e: /* UDOT */ + gen_gvec_op3_ool(s, is_q, rd, rn, rm, index, + u ? gen_helper_gvec_udot_idx_b + : gen_helper_gvec_sdot_idx_b); + return; case 0x11: /* FCMLA #0 */ case 0x13: /* FCMLA #90 */ case 0x15: /* FCMLA #180 */ diff --git a/target/arm/translate.c b/target/arm/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate.c +++ b/target/arm/translate.c @@ -XXX,XX +XXX,XX @@ static int disas_neon_data_insn(DisasContext *s, uint32_t insn) */ static int disas_neon_insn_3same_ext(DisasContext *s, uint32_t insn) { - gen_helper_gvec_3_ptr *fn_gvec_ptr; - int rd, rn, rm, rot, size, opr_sz; - TCGv_ptr fpst; + gen_helper_gvec_3 *fn_gvec = NULL; + gen_helper_gvec_3_ptr *fn_gvec_ptr = NULL; + int rd, rn, rm, opr_sz; + int data = 0; bool q; q = extract32(insn, 6, 1); @@ -XXX,XX +XXX,XX @@ static int disas_neon_insn_3same_ext(DisasContext *s, uint32_t insn) if ((insn & 0xfe200f10) == 0xfc200800) { /* VCMLA -- 1111 110R R.1S .... .... 1000 ...0 .... */ - size = extract32(insn, 20, 1); - rot = extract32(insn, 23, 2); + int size = extract32(insn, 20, 1); + data = extract32(insn, 23, 2); /* rot */ if (!arm_dc_feature(s, ARM_FEATURE_V8_FCMA) || (!size && !arm_dc_feature(s, ARM_FEATURE_V8_FP16))) { return 1; @@ -XXX,XX +XXX,XX @@ static int disas_neon_insn_3same_ext(DisasContext *s, uint32_t insn) fn_gvec_ptr = size ? gen_helper_gvec_fcmlas : gen_helper_gvec_fcmlah; } else if ((insn & 0xfea00f10) == 0xfc800800) { /* VCADD -- 1111 110R 1.0S .... .... 1000 ...0 .... */ - size = extract32(insn, 20, 1); - rot = extract32(insn, 24, 1); + int size = extract32(insn, 20, 1); + data = extract32(insn, 24, 1); /* rot */ if (!arm_dc_feature(s, ARM_FEATURE_V8_FCMA) || (!size && !arm_dc_feature(s, ARM_FEATURE_V8_FP16))) { return 1; } fn_gvec_ptr = size ? gen_helper_gvec_fcadds : gen_helper_gvec_fcaddh; + } else if ((insn & 0xfeb00f00) == 0xfc200d00) { + /* V[US]DOT -- 1111 1100 0.10 .... .... 1101 .Q.U .... */ + bool u = extract32(insn, 4, 1); + if (!arm_dc_feature(s, ARM_FEATURE_V8_DOTPROD)) { + return 1; + } + fn_gvec = u ? gen_helper_gvec_udot_b : gen_helper_gvec_sdot_b; } else { return 1; } @@ -XXX,XX +XXX,XX @@ static int disas_neon_insn_3same_ext(DisasContext *s, uint32_t insn) } opr_sz = (1 + q) * 8; - fpst = get_fpstatus_ptr(1); - tcg_gen_gvec_3_ptr(vfp_reg_offset(1, rd), - vfp_reg_offset(1, rn), - vfp_reg_offset(1, rm), fpst, - opr_sz, opr_sz, rot, fn_gvec_ptr); - tcg_temp_free_ptr(fpst); + if (fn_gvec_ptr) { + TCGv_ptr fpst = get_fpstatus_ptr(1); + tcg_gen_gvec_3_ptr(vfp_reg_offset(1, rd), + vfp_reg_offset(1, rn), + vfp_reg_offset(1, rm), fpst, + opr_sz, opr_sz, data, fn_gvec_ptr); + tcg_temp_free_ptr(fpst); + } else { + tcg_gen_gvec_3_ool(vfp_reg_offset(1, rd), + vfp_reg_offset(1, rn), + vfp_reg_offset(1, rm), + opr_sz, opr_sz, data, fn_gvec); + } return 0; } @@ -XXX,XX +XXX,XX @@ static int disas_neon_insn_3same_ext(DisasContext *s, uint32_t insn) static int disas_neon_insn_2reg_scalar_ext(DisasContext *s, uint32_t insn) { - gen_helper_gvec_3_ptr *fn_gvec_ptr; + gen_helper_gvec_3 *fn_gvec = NULL; + gen_helper_gvec_3_ptr *fn_gvec_ptr = NULL; int rd, rn, rm, opr_sz, data; - TCGv_ptr fpst; bool q; q = extract32(insn, 6, 1); @@ -XXX,XX +XXX,XX @@ static int disas_neon_insn_2reg_scalar_ext(DisasContext *s, uint32_t insn) data = (index << 2) | rot; fn_gvec_ptr = (size ? gen_helper_gvec_fcmlas_idx : gen_helper_gvec_fcmlah_idx); + } else if ((insn & 0xffb00f00) == 0xfe200d00) { + /* V[US]DOT -- 1111 1110 0.10 .... .... 1101 .Q.U .... */ + int u = extract32(insn, 4, 1); + if (!arm_dc_feature(s, ARM_FEATURE_V8_DOTPROD)) { + return 1; + } + fn_gvec = u ? gen_helper_gvec_udot_idx_b : gen_helper_gvec_sdot_idx_b; + /* rm is just Vm, and index is M. */ + data = extract32(insn, 5, 1); /* index */ + rm = extract32(insn, 0, 4); } else { return 1; } @@ -XXX,XX +XXX,XX @@ static int disas_neon_insn_2reg_scalar_ext(DisasContext *s, uint32_t insn) } opr_sz = (1 + q) * 8; - fpst = get_fpstatus_ptr(1); - tcg_gen_gvec_3_ptr(vfp_reg_offset(1, rd), - vfp_reg_offset(1, rn), - vfp_reg_offset(1, rm), fpst, - opr_sz, opr_sz, data, fn_gvec_ptr); - tcg_temp_free_ptr(fpst); + if (fn_gvec_ptr) { + TCGv_ptr fpst = get_fpstatus_ptr(1); + tcg_gen_gvec_3_ptr(vfp_reg_offset(1, rd), + vfp_reg_offset(1, rn), + vfp_reg_offset(1, rm), fpst, + opr_sz, opr_sz, data, fn_gvec_ptr); + tcg_temp_free_ptr(fpst); + } else { + tcg_gen_gvec_3_ool(vfp_reg_offset(1, rd), + vfp_reg_offset(1, rn), + vfp_reg_offset(1, rm), + opr_sz, opr_sz, data, fn_gvec); + } return 0; } -- 2.17.1 From: Alex Bennée <alex.bennee@linaro.org> Since kernel commit a86bd139f2 (arm64: arch_timer: Enable CNTVCT_EL0 trap..), released in kernel version v4.12, user-space has been able to read these system registers. As we can't use QEMUTimer's in linux-user mode we just directly call cpu_get_clock(). Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180625160009.17437-2-alex.bennee@linaro.org Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/target/arm/helper.c b/target/arm/helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper.c +++ b/target/arm/helper.c @@ -XXX,XX +XXX,XX @@ static const ARMCPRegInfo generic_timer_cp_reginfo[] = { }; #else -/* In user-mode none of the generic timer registers are accessible, - * and their implementation depends on QEMU_CLOCK_VIRTUAL and qdev gpio outputs, - * so instead just don't register any of them. + +/* In user-mode most of the generic timer registers are inaccessible + * however modern kernels (4.12+) allow access to cntvct_el0 */ + +static uint64_t gt_virt_cnt_read(CPUARMState *env, const ARMCPRegInfo *ri) +{ + /* Currently we have no support for QEMUTimer in linux-user so we + * can't call gt_get_countervalue(env), instead we directly + * call the lower level functions. + */ + return cpu_get_clock() / GTIMER_SCALE; +} + static const ARMCPRegInfo generic_timer_cp_reginfo[] = { + { .name = "CNTFRQ_EL0", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 3, .crn = 14, .crm = 0, .opc2 = 0, + .type = ARM_CP_CONST, .access = PL0_R /* no PL1_RW in linux-user */, + .fieldoffset = offsetof(CPUARMState, cp15.c14_cntfrq), + .resetvalue = NANOSECONDS_PER_SECOND / GTIMER_SCALE, + }, + { .name = "CNTVCT_EL0", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 3, .crn = 14, .crm = 0, .opc2 = 2, + .access = PL0_R, .type = ARM_CP_NO_RAW | ARM_CP_IO, + .readfn = gt_virt_cnt_read, + }, REGINFO_SENTINEL }; -- 2.17.1 From: Jean-Christophe Dubois <jcd@tribudubois.net> Signed-off-by: Jean-Christophe Dubois <jcd@tribudubois.net> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- hw/arm/mcimx7d-sabre.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/hw/arm/mcimx7d-sabre.c b/hw/arm/mcimx7d-sabre.c index XXXXXXX..XXXXXXX 100644 --- a/hw/arm/mcimx7d-sabre.c +++ b/hw/arm/mcimx7d-sabre.c @@ -XXX,XX +XXX,XX @@ #include "hw/arm/fsl-imx7.h" #include "hw/boards.h" #include "sysemu/sysemu.h" -#include "sysemu/device_tree.h" #include "qemu/error-report.h" #include "sysemu/qtest.h" -#include "net/net.h" typedef struct { FslIMX7State soc; -- 2.17.1 From: Jean-Christophe Dubois <jcd@tribudubois.net> Signed-off-by: Jean-Christophe Dubois <jcd@tribudubois.net> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- hw/arm/fsl-imx7.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/arm/fsl-imx7.c b/hw/arm/fsl-imx7.c index XXXXXXX..XXXXXXX 100644 --- a/hw/arm/fsl-imx7.c +++ b/hw/arm/fsl-imx7.c @@ -XXX,XX +XXX,XX @@ static void fsl_imx7_realize(DeviceState *dev, Error **errp) /* * SRC */ - create_unimplemented_device("sdma", FSL_IMX7_SRC_ADDR, FSL_IMX7_SRC_SIZE); + create_unimplemented_device("src", FSL_IMX7_SRC_ADDR, FSL_IMX7_SRC_SIZE); /* * Watchdog -- 2.17.1 From: Jean-Christophe Dubois <jcd@tribudubois.net> The qdev_get_gpio_in() function accept an int as second parameter. Signed-off-by: Jean-Christophe Dubois <jcd@tribudubois.net> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- hw/arm/fsl-imx7.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/hw/arm/fsl-imx7.c b/hw/arm/fsl-imx7.c index XXXXXXX..XXXXXXX 100644 --- a/hw/arm/fsl-imx7.c +++ b/hw/arm/fsl-imx7.c @@ -XXX,XX +XXX,XX @@ static void fsl_imx7_realize(DeviceState *dev, Error **errp) FSL_IMX7_ECSPI4_ADDR, }; - static const hwaddr FSL_IMX7_SPIn_IRQ[FSL_IMX7_NUM_ECSPIS] = { + static const int FSL_IMX7_SPIn_IRQ[FSL_IMX7_NUM_ECSPIS] = { FSL_IMX7_ECSPI1_IRQ, FSL_IMX7_ECSPI2_IRQ, FSL_IMX7_ECSPI3_IRQ, @@ -XXX,XX +XXX,XX @@ static void fsl_imx7_realize(DeviceState *dev, Error **errp) FSL_IMX7_I2C4_ADDR, }; - static const hwaddr FSL_IMX7_I2Cn_IRQ[FSL_IMX7_NUM_I2CS] = { + static const int FSL_IMX7_I2Cn_IRQ[FSL_IMX7_NUM_I2CS] = { FSL_IMX7_I2C1_IRQ, FSL_IMX7_I2C2_IRQ, FSL_IMX7_I2C3_IRQ, @@ -XXX,XX +XXX,XX @@ static void fsl_imx7_realize(DeviceState *dev, Error **errp) FSL_IMX7_USB3_ADDR, }; - static const hwaddr FSL_IMX7_USBn_IRQ[FSL_IMX7_NUM_USBS] = { + static const int FSL_IMX7_USBn_IRQ[FSL_IMX7_NUM_USBS] = { FSL_IMX7_USB1_IRQ, FSL_IMX7_USB2_IRQ, FSL_IMX7_USB3_IRQ, -- 2.17.1 From: Aaron Lindsay <alindsay@codeaurora.org> Signed-off-by: Aaron Lindsay <alindsay@codeaurora.org> Message-id: 1529699547-17044-5-git-send-email-alindsay@codeaurora.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/cpu.h | 1 + target/arm/cpu.c | 21 ++++++++++++++------- target/arm/kvm32.c | 8 ++++---- 3 files changed, 19 insertions(+), 11 deletions(-) diff --git a/target/arm/cpu.h b/target/arm/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -XXX,XX +XXX,XX @@ enum arm_features { ARM_FEATURE_OMAPCP, /* OMAP specific CP15 ops handling. */ ARM_FEATURE_THUMB2EE, ARM_FEATURE_V7MP, /* v7 Multiprocessing Extensions */ + ARM_FEATURE_V7VE, /* v7 Virtualization Extensions (non-EL2 parts) */ ARM_FEATURE_V4T, ARM_FEATURE_V5, ARM_FEATURE_STRONGARM, diff --git a/target/arm/cpu.c b/target/arm/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/cpu.c +++ b/target/arm/cpu.c @@ -XXX,XX +XXX,XX @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp) /* Some features automatically imply others: */ if (arm_feature(env, ARM_FEATURE_V8)) { - set_feature(env, ARM_FEATURE_V7); + set_feature(env, ARM_FEATURE_V7VE); + } + if (arm_feature(env, ARM_FEATURE_V7VE)) { + /* v7 Virtualization Extensions. In real hardware this implies + * EL2 and also the presence of the Security Extensions. + * For QEMU, for backwards-compatibility we implement some + * CPUs or CPU configs which have no actual EL2 or EL3 but do + * include the various other features that V7VE implies. + * Presence of EL2 itself is ARM_FEATURE_EL2, and of the + * Security Extensions is ARM_FEATURE_EL3. + */ set_feature(env, ARM_FEATURE_ARM_DIV); set_feature(env, ARM_FEATURE_LPAE); + set_feature(env, ARM_FEATURE_V7); } if (arm_feature(env, ARM_FEATURE_V7)) { set_feature(env, ARM_FEATURE_VAPA); @@ -XXX,XX +XXX,XX @@ static void cortex_a7_initfn(Object *obj) ARMCPU *cpu = ARM_CPU(obj); cpu->dtb_compatible = "arm,cortex-a7"; - set_feature(&cpu->env, ARM_FEATURE_V7); + set_feature(&cpu->env, ARM_FEATURE_V7VE); set_feature(&cpu->env, ARM_FEATURE_VFP4); set_feature(&cpu->env, ARM_FEATURE_NEON); set_feature(&cpu->env, ARM_FEATURE_THUMB2EE); - set_feature(&cpu->env, ARM_FEATURE_ARM_DIV); set_feature(&cpu->env, ARM_FEATURE_GENERIC_TIMER); set_feature(&cpu->env, ARM_FEATURE_DUMMY_C15_REGS); set_feature(&cpu->env, ARM_FEATURE_CBAR_RO); - set_feature(&cpu->env, ARM_FEATURE_LPAE); set_feature(&cpu->env, ARM_FEATURE_EL3); cpu->kvm_target = QEMU_KVM_ARM_TARGET_CORTEX_A7; cpu->midr = 0x410fc075; @@ -XXX,XX +XXX,XX @@ static void cortex_a15_initfn(Object *obj) ARMCPU *cpu = ARM_CPU(obj); cpu->dtb_compatible = "arm,cortex-a15"; - set_feature(&cpu->env, ARM_FEATURE_V7); + set_feature(&cpu->env, ARM_FEATURE_V7VE); set_feature(&cpu->env, ARM_FEATURE_VFP4); set_feature(&cpu->env, ARM_FEATURE_NEON); set_feature(&cpu->env, ARM_FEATURE_THUMB2EE); - set_feature(&cpu->env, ARM_FEATURE_ARM_DIV); set_feature(&cpu->env, ARM_FEATURE_GENERIC_TIMER); set_feature(&cpu->env, ARM_FEATURE_DUMMY_C15_REGS); set_feature(&cpu->env, ARM_FEATURE_CBAR_RO); - set_feature(&cpu->env, ARM_FEATURE_LPAE); set_feature(&cpu->env, ARM_FEATURE_EL3); cpu->kvm_target = QEMU_KVM_ARM_TARGET_CORTEX_A15; cpu->midr = 0x412fc0f1; diff --git a/target/arm/kvm32.c b/target/arm/kvm32.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/kvm32.c +++ b/target/arm/kvm32.c @@ -XXX,XX +XXX,XX @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf) /* Now we've retrieved all the register information we can * set the feature bits based on the ID register fields. * We can assume any KVM supporting CPU is at least a v7 - * with VFPv3, LPAE and the generic timers; this in turn implies - * most of the other feature bits, but a few must be tested. + * with VFPv3, virtualization extensions, and the generic + * timers; this in turn implies most of the other feature + * bits, but a few must be tested. */ - set_feature(&features, ARM_FEATURE_V7); + set_feature(&features, ARM_FEATURE_V7VE); set_feature(&features, ARM_FEATURE_VFP3); - set_feature(&features, ARM_FEATURE_LPAE); set_feature(&features, ARM_FEATURE_GENERIC_TIMER); switch (extract32(id_isar0, 24, 4)) { -- 2.17.1 From: Aaron Lindsay <alindsay@codeaurora.org> KVM implies V7VE, which implies ARM_DIV and THUMB_DIV. The conditional detection here is therefore unnecessary. Because V7VE is already unconditionally specified for all KVM hosts, ARM_DIV and THUMB_DIV are already indirectly specified and do not need to be included here at all. Signed-off-by: Aaron Lindsay <alindsay@codeaurora.org> Message-id: 1529699547-17044-6-git-send-email-alindsay@codeaurora.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/kvm32.c | 19 +------------------ 1 file changed, 1 insertion(+), 18 deletions(-) diff --git a/target/arm/kvm32.c b/target/arm/kvm32.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/kvm32.c +++ b/target/arm/kvm32.c @@ -XXX,XX +XXX,XX @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf) * and then query that CPU for the relevant ID registers. */ int i, ret, fdarray[3]; - uint32_t midr, id_pfr0, id_isar0, mvfr1; + uint32_t midr, id_pfr0, mvfr1; uint64_t features = 0; /* Old kernels may not know about the PREFERRED_TARGET ioctl: however * we know these will only support creating one kind of guest CPU, @@ -XXX,XX +XXX,XX @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf) | ENCODE_CP_REG(15, 0, 0, 0, 1, 0, 0), .addr = (uintptr_t)&id_pfr0, }, - { - .id = KVM_REG_ARM | KVM_REG_SIZE_U32 - | ENCODE_CP_REG(15, 0, 0, 0, 2, 0, 0), - .addr = (uintptr_t)&id_isar0, - }, { .id = KVM_REG_ARM | KVM_REG_SIZE_U32 | KVM_REG_ARM_VFP | KVM_REG_ARM_VFP_MVFR1, @@ -XXX,XX +XXX,XX @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf) set_feature(&features, ARM_FEATURE_VFP3); set_feature(&features, ARM_FEATURE_GENERIC_TIMER); - switch (extract32(id_isar0, 24, 4)) { - case 1: - set_feature(&features, ARM_FEATURE_THUMB_DIV); - break; - case 2: - set_feature(&features, ARM_FEATURE_ARM_DIV); - set_feature(&features, ARM_FEATURE_THUMB_DIV); - break; - default: - break; - } - if (extract32(id_pfr0, 12, 4) == 1) { set_feature(&features, ARM_FEATURE_THUMB2EE); } -- 2.17.1 From: Aaron Lindsay <alindsay@codeaurora.org> This makes it match its AArch64 equivalent, PMINTENSET_EL1 Signed-off-by: Aaron Lindsay <alindsay@codeaurora.org> Message-id: 1529699547-17044-13-git-send-email-alindsay@codeaurora.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/target/arm/helper.c b/target/arm/helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper.c +++ b/target/arm/helper.c @@ -XXX,XX +XXX,XX @@ static const ARMCPRegInfo v7_cp_reginfo[] = { .writefn = pmuserenr_write, .raw_writefn = raw_write }, { .name = "PMINTENSET", .cp = 15, .crn = 9, .crm = 14, .opc1 = 0, .opc2 = 1, .access = PL1_RW, .accessfn = access_tpm, - .type = ARM_CP_ALIAS, + .type = ARM_CP_ALIAS | ARM_CP_IO, .fieldoffset = offsetoflow32(CPUARMState, cp15.c9_pminten), .resetvalue = 0, .writefn = pmintenset_write, .raw_writefn = raw_write }, -- 2.17.1 We don't actually implement SD command CRC checking, because for almost all of our SD controllers the CRC generation is done in hardware, and so modelling CRC generation and checking would be a bit pointless. (The exception is that milkymist-memcard makes the guest software compute the CRC.) As a result almost all of our SD controller models don't bother to set the SDRequest crc field, and the SD card model doesn't check it. So the tracing of it in sdbus_do_command() provokes Coverity warnings about use of uninitialized data. Drop the CRC field from the trace; we can always add it back if and when we do anything useful with the CRC. Fixes Coverity issues 1386072, 1386074, 1386076, 1390571. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-id: 20180626180324.5537-1-peter.maydell@linaro.org --- hw/sd/core.c | 2 +- hw/sd/trace-events | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/hw/sd/core.c b/hw/sd/core.c index XXXXXXX..XXXXXXX 100644 --- a/hw/sd/core.c +++ b/hw/sd/core.c @@ -XXX,XX +XXX,XX @@ int sdbus_do_command(SDBus *sdbus, SDRequest *req, uint8_t *response) { SDState *card = get_card(sdbus); - trace_sdbus_command(sdbus_name(sdbus), req->cmd, req->arg, req->crc); + trace_sdbus_command(sdbus_name(sdbus), req->cmd, req->arg); if (card) { SDCardClass *sc = SD_CARD_GET_CLASS(card); diff --git a/hw/sd/trace-events b/hw/sd/trace-events index XXXXXXX..XXXXXXX 100644 --- a/hw/sd/trace-events +++ b/hw/sd/trace-events @@ -XXX,XX +XXX,XX @@ bcm2835_sdhost_edm_change(const char *why, uint32_t edm) "(%s) EDM now 0x%x" bcm2835_sdhost_update_irq(uint32_t irq) "IRQ bits 0x%x\n" # hw/sd/core.c -sdbus_command(const char *bus_name, uint8_t cmd, uint32_t arg, uint8_t crc) "@%s CMD%02d arg 0x%08x crc 0x%02x" +sdbus_command(const char *bus_name, uint8_t cmd, uint32_t arg) "@%s CMD%02d arg 0x%08x" sdbus_read(const char *bus_name, uint8_t value) "@%s value 0x%02x" sdbus_write(const char *bus_name, uint8_t value) "@%s value 0x%02x" sdbus_set_voltage(const char *bus_name, uint16_t millivolts) "@%s %u (mV)" -- 2.17.1 From: Philippe Mathieu-Daudé <f4bug@amsat.org> The load/store API will ease further code movement. Per the Physical Layer Simplified Spec. "3.6 Bus Protocol": "In the CMD line the Most Significant Bit (MSB) is transmitted first, the Least Significant Bit (LSB) is the last." Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- hw/sd/bcm2835_sdhost.c | 13 +++++-------- hw/sd/milkymist-memcard.c | 3 +-- hw/sd/omap_mmc.c | 6 ++---- hw/sd/pl181.c | 11 ++++------- hw/sd/sdhci.c | 15 +++++---------- hw/sd/ssi-sd.c | 6 ++---- 6 files changed, 19 insertions(+), 35 deletions(-) diff --git a/hw/sd/bcm2835_sdhost.c b/hw/sd/bcm2835_sdhost.c index XXXXXXX..XXXXXXX 100644 --- a/hw/sd/bcm2835_sdhost.c +++ b/hw/sd/bcm2835_sdhost.c @@ -XXX,XX +XXX,XX @@ static void bcm2835_sdhost_send_command(BCM2835SDHostState *s) goto error; } if (!(s->cmd & SDCMD_NO_RESPONSE)) { -#define RWORD(n) (((uint32_t)rsp[n] << 24) | (rsp[n + 1] << 16) \ - | (rsp[n + 2] << 8) | rsp[n + 3]) if (rlen == 0 || (rlen == 4 && (s->cmd & SDCMD_LONG_RESPONSE))) { goto error; } @@ -XXX,XX +XXX,XX @@ static void bcm2835_sdhost_send_command(BCM2835SDHostState *s) goto error; } if (rlen == 4) { - s->rsp[0] = RWORD(0); + s->rsp[0] = ldl_be_p(&rsp[0]); s->rsp[1] = s->rsp[2] = s->rsp[3] = 0; } else { - s->rsp[0] = RWORD(12); - s->rsp[1] = RWORD(8); - s->rsp[2] = RWORD(4); - s->rsp[3] = RWORD(0); + s->rsp[0] = ldl_be_p(&rsp[12]); + s->rsp[1] = ldl_be_p(&rsp[8]); + s->rsp[2] = ldl_be_p(&rsp[4]); + s->rsp[3] = ldl_be_p(&rsp[0]); } -#undef RWORD } /* We never really delay commands, so if this was a 'busywait' command * then we've completed it now and can raise the interrupt. diff --git a/hw/sd/milkymist-memcard.c b/hw/sd/milkymist-memcard.c index XXXXXXX..XXXXXXX 100644 --- a/hw/sd/milkymist-memcard.c +++ b/hw/sd/milkymist-memcard.c @@ -XXX,XX +XXX,XX @@ static void memcard_sd_command(MilkymistMemcardState *s) SDRequest req; req.cmd = s->command[0] & 0x3f; - req.arg = (s->command[1] << 24) | (s->command[2] << 16) - | (s->command[3] << 8) | s->command[4]; + req.arg = ldl_be_p(s->command + 1); req.crc = s->command[5]; s->response[0] = req.cmd; diff --git a/hw/sd/omap_mmc.c b/hw/sd/omap_mmc.c index XXXXXXX..XXXXXXX 100644 --- a/hw/sd/omap_mmc.c +++ b/hw/sd/omap_mmc.c @@ -XXX,XX +XXX,XX @@ static void omap_mmc_command(struct omap_mmc_s *host, int cmd, int dir, CID_CSD_OVERWRITE; if (host->sdio & (1 << 13)) mask |= AKE_SEQ_ERROR; - rspstatus = (response[0] << 24) | (response[1] << 16) | - (response[2] << 8) | (response[3] << 0); + rspstatus = ldl_be_p(response); break; case sd_r2: @@ -XXX,XX +XXX,XX @@ static void omap_mmc_command(struct omap_mmc_s *host, int cmd, int dir, } rsplen = 4; - rspstatus = (response[0] << 24) | (response[1] << 16) | - (response[2] << 8) | (response[3] << 0); + rspstatus = ldl_be_p(response); if (rspstatus & 0x80000000) host->status &= 0xe000; else diff --git a/hw/sd/pl181.c b/hw/sd/pl181.c index XXXXXXX..XXXXXXX 100644 --- a/hw/sd/pl181.c +++ b/hw/sd/pl181.c @@ -XXX,XX +XXX,XX @@ static void pl181_send_command(PL181State *s) if (rlen < 0) goto error; if (s->cmd & PL181_CMD_RESPONSE) { -#define RWORD(n) (((uint32_t)response[n] << 24) | (response[n + 1] << 16) \ - | (response[n + 2] << 8) | response[n + 3]) if (rlen == 0 || (rlen == 4 && (s->cmd & PL181_CMD_LONGRESP))) goto error; if (rlen != 4 && rlen != 16) goto error; - s->response[0] = RWORD(0); + s->response[0] = ldl_be_p(&response[0]); if (rlen == 4) { s->response[1] = s->response[2] = s->response[3] = 0; } else { - s->response[1] = RWORD(4); - s->response[2] = RWORD(8); - s->response[3] = RWORD(12) & ~1; + s->response[1] = ldl_be_p(&response[4]); + s->response[2] = ldl_be_p(&response[8]); + s->response[3] = ldl_be_p(&response[12]) & ~1; } DPRINTF("Response received\n"); s->status |= PL181_STATUS_CMDRESPEND; -#undef RWORD } else { DPRINTF("Command sent\n"); s->status |= PL181_STATUS_CMDSENT; diff --git a/hw/sd/sdhci.c b/hw/sd/sdhci.c index XXXXXXX..XXXXXXX 100644 --- a/hw/sd/sdhci.c +++ b/hw/sd/sdhci.c @@ -XXX,XX +XXX,XX @@ static void sdhci_send_command(SDHCIState *s) if (s->cmdreg & SDHC_CMD_RESPONSE) { if (rlen == 4) { - s->rspreg[0] = (response[0] << 24) | (response[1] << 16) | - (response[2] << 8) | response[3]; + s->rspreg[0] = ldl_be_p(response); s->rspreg[1] = s->rspreg[2] = s->rspreg[3] = 0; trace_sdhci_response4(s->rspreg[0]); } else if (rlen == 16) { - s->rspreg[0] = (response[11] << 24) | (response[12] << 16) | - (response[13] << 8) | response[14]; - s->rspreg[1] = (response[7] << 24) | (response[8] << 16) | - (response[9] << 8) | response[10]; - s->rspreg[2] = (response[3] << 24) | (response[4] << 16) | - (response[5] << 8) | response[6]; + s->rspreg[0] = ldl_be_p(&response[11]); + s->rspreg[1] = ldl_be_p(&response[7]); + s->rspreg[2] = ldl_be_p(&response[3]); s->rspreg[3] = (response[0] << 16) | (response[1] << 8) | response[2]; trace_sdhci_response16(s->rspreg[3], s->rspreg[2], @@ -XXX,XX +XXX,XX @@ static void sdhci_end_transfer(SDHCIState *s) trace_sdhci_end_transfer(request.cmd, request.arg); sdbus_do_command(&s->sdbus, &request, response); /* Auto CMD12 response goes to the upper Response register */ - s->rspreg[3] = (response[0] << 24) | (response[1] << 16) | - (response[2] << 8) | response[3]; + s->rspreg[3] = ldl_be_p(response); } s->prnsts &= ~(SDHC_DOING_READ | SDHC_DOING_WRITE | diff --git a/hw/sd/ssi-sd.c b/hw/sd/ssi-sd.c index XXXXXXX..XXXXXXX 100644 --- a/hw/sd/ssi-sd.c +++ b/hw/sd/ssi-sd.c @@ -XXX,XX +XXX,XX @@ static uint32_t ssi_sd_transfer(SSISlave *dev, uint32_t val) uint8_t longresp[16]; /* FIXME: Check CRC. */ request.cmd = s->cmd; - request.arg = (s->cmdarg[0] << 24) | (s->cmdarg[1] << 16) - | (s->cmdarg[2] << 8) | s->cmdarg[3]; + request.arg = ldl_be_p(s->cmdarg); DPRINTF("CMD%d arg 0x%08x\n", s->cmd, request.arg); s->arglen = sdbus_do_command(&s->sdbus, &request, longresp); if (s->arglen <= 0) { @@ -XXX,XX +XXX,XX @@ static uint32_t ssi_sd_transfer(SSISlave *dev, uint32_t val) /* CMD13 returns a 2-byte statuse work. Other commands only return the first byte. */ s->arglen = (s->cmd == 13) ? 2 : 1; - cardstatus = (longresp[0] << 24) | (longresp[1] << 16) - | (longresp[2] << 8) | longresp[3]; + cardstatus = ldl_be_p(longresp); status = 0; if (((cardstatus >> 9) & 0xf) < 4) status |= SSI_SDR_IDLE; -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> We already check for the same condition within the normal integer sdiv and sdiv64 helpers. Use a slightly different formation that does not require deducing the expression type. Fixes: f97cfd596ed Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Message-id: 20180629001538.11415-2-richard.henderson@linaro.org [PMM: reworded a comment] Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/sve_helper.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -XXX,XX +XXX,XX @@ void HELPER(NAME)(void *vd, void *vn, void *vm, void *vg, uint32_t desc) \ #define DO_MIN(N, M) ((N) >= (M) ? (M) : (N)) #define DO_ABD(N, M) ((N) >= (M) ? (N) - (M) : (M) - (N)) #define DO_MUL(N, M) (N * M) -#define DO_DIV(N, M) (M ? N / M : 0) + + +/* + * We must avoid the C undefined behaviour cases: division by + * zero and signed division of INT_MIN by -1. Both of these + * have architecturally defined required results for Arm. + * We special case all signed divisions by -1 to avoid having + * to deduce the minimum integer for the type involved. + */ +#define DO_SDIV(N, M) (unlikely(M == 0) ? 0 : unlikely(M == -1) ? -N : N / M) +#define DO_UDIV(N, M) (unlikely(M == 0) ? 0 : N / M) DO_ZPZZ(sve_and_zpzz_b, uint8_t, H1, DO_AND) DO_ZPZZ(sve_and_zpzz_h, uint16_t, H1_2, DO_AND) @@ -XXX,XX +XXX,XX @@ DO_ZPZZ(sve_umulh_zpzz_h, uint16_t, H1_2, do_mulh_h) DO_ZPZZ(sve_umulh_zpzz_s, uint32_t, H1_4, do_mulh_s) DO_ZPZZ_D(sve_umulh_zpzz_d, uint64_t, do_umulh_d) -DO_ZPZZ(sve_sdiv_zpzz_s, int32_t, H1_4, DO_DIV) -DO_ZPZZ_D(sve_sdiv_zpzz_d, int64_t, DO_DIV) +DO_ZPZZ(sve_sdiv_zpzz_s, int32_t, H1_4, DO_SDIV) +DO_ZPZZ_D(sve_sdiv_zpzz_d, int64_t, DO_SDIV) -DO_ZPZZ(sve_udiv_zpzz_s, uint32_t, H1_4, DO_DIV) -DO_ZPZZ_D(sve_udiv_zpzz_d, uint64_t, DO_DIV) +DO_ZPZZ(sve_udiv_zpzz_s, uint32_t, H1_4, DO_UDIV) +DO_ZPZZ_D(sve_udiv_zpzz_d, uint64_t, DO_UDIV) /* Note that all bits of the shift are significant and not modulo the element size. */ -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> Leave ARM_CP_SVE, removing ARM_CP_FPU; the sve_access_check produced by the flag already includes fp_access_check. If we also check ARM_CP_FPU the double fp_access_check asserts. Reported-by: Laurent Desnogues <laurent.desnogues@gmail.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Laurent Desnogues <laurent.desnogues@gmail.com> Message-id: 20180629001538.11415-3-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper.c | 8 ++++---- target/arm/translate-a64.c | 5 ++--- 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/target/arm/helper.c b/target/arm/helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper.c +++ b/target/arm/helper.c @@ -XXX,XX +XXX,XX @@ static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri, static const ARMCPRegInfo zcr_el1_reginfo = { .name = "ZCR_EL1", .state = ARM_CP_STATE_AA64, .opc0 = 3, .opc1 = 0, .crn = 1, .crm = 2, .opc2 = 0, - .access = PL1_RW, .type = ARM_CP_SVE | ARM_CP_FPU, + .access = PL1_RW, .type = ARM_CP_SVE, .fieldoffset = offsetof(CPUARMState, vfp.zcr_el[1]), .writefn = zcr_write, .raw_writefn = raw_write }; @@ -XXX,XX +XXX,XX @@ static const ARMCPRegInfo zcr_el1_reginfo = { static const ARMCPRegInfo zcr_el2_reginfo = { .name = "ZCR_EL2", .state = ARM_CP_STATE_AA64, .opc0 = 3, .opc1 = 4, .crn = 1, .crm = 2, .opc2 = 0, - .access = PL2_RW, .type = ARM_CP_SVE | ARM_CP_FPU, + .access = PL2_RW, .type = ARM_CP_SVE, .fieldoffset = offsetof(CPUARMState, vfp.zcr_el[2]), .writefn = zcr_write, .raw_writefn = raw_write }; @@ -XXX,XX +XXX,XX @@ static const ARMCPRegInfo zcr_el2_reginfo = { static const ARMCPRegInfo zcr_no_el2_reginfo = { .name = "ZCR_EL2", .state = ARM_CP_STATE_AA64, .opc0 = 3, .opc1 = 4, .crn = 1, .crm = 2, .opc2 = 0, - .access = PL2_RW, .type = ARM_CP_SVE | ARM_CP_FPU, + .access = PL2_RW, .type = ARM_CP_SVE, .readfn = arm_cp_read_zero, .writefn = arm_cp_write_ignore }; static const ARMCPRegInfo zcr_el3_reginfo = { .name = "ZCR_EL3", .state = ARM_CP_STATE_AA64, .opc0 = 3, .opc1 = 6, .crn = 1, .crm = 2, .opc2 = 0, - .access = PL3_RW, .type = ARM_CP_SVE | ARM_CP_FPU, + .access = PL3_RW, .type = ARM_CP_SVE, .fieldoffset = offsetof(CPUARMState, vfp.zcr_el[3]), .writefn = zcr_write, .raw_writefn = raw_write }; diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -XXX,XX +XXX,XX @@ static void handle_sys(DisasContext *s, uint32_t insn, bool isread, default: break; } - if ((ri->type & ARM_CP_SVE) && !sve_access_check(s)) { - return; - } if ((ri->type & ARM_CP_FPU) && !fp_access_check(s)) { return; + } else if ((ri->type & ARM_CP_SVE) && !sve_access_check(s)) { + return; } if ((tb_cflags(s->base.tb) & CF_USE_ICOUNT) && (ri->type & ARM_CP_IO)) { -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> There is no need to re-set these 9 features already implied by the call to aarch64_a57_initfn. Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Message-id: 20180629001538.11415-4-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/cpu64.c | 9 --------- 1 file changed, 9 deletions(-) diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/cpu64.c +++ b/target/arm/cpu64.c @@ -XXX,XX +XXX,XX @@ static void aarch64_max_initfn(Object *obj) * whereas the architecture requires them to be present in both if * present in either. */ - set_feature(&cpu->env, ARM_FEATURE_V8); - set_feature(&cpu->env, ARM_FEATURE_VFP4); - set_feature(&cpu->env, ARM_FEATURE_NEON); - set_feature(&cpu->env, ARM_FEATURE_AARCH64); - set_feature(&cpu->env, ARM_FEATURE_V8_AES); - set_feature(&cpu->env, ARM_FEATURE_V8_SHA1); - set_feature(&cpu->env, ARM_FEATURE_V8_SHA256); set_feature(&cpu->env, ARM_FEATURE_V8_SHA512); set_feature(&cpu->env, ARM_FEATURE_V8_SHA3); set_feature(&cpu->env, ARM_FEATURE_V8_SM3); set_feature(&cpu->env, ARM_FEATURE_V8_SM4); - set_feature(&cpu->env, ARM_FEATURE_V8_PMULL); - set_feature(&cpu->env, ARM_FEATURE_CRC); set_feature(&cpu->env, ARM_FEATURE_V8_ATOMICS); set_feature(&cpu->env, ARM_FEATURE_V8_RDM); set_feature(&cpu->env, ARM_FEATURE_V8_DOTPROD); -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> There is no need to re-set these 3 features already implied by the call to aarch64_a15_initfn. Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Message-id: 20180629001538.11415-5-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/cpu.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/target/arm/cpu.c b/target/arm/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/cpu.c +++ b/target/arm/cpu.c @@ -XXX,XX +XXX,XX @@ static void arm_max_initfn(Object *obj) * since we don't correctly set the ID registers to advertise them, */ set_feature(&cpu->env, ARM_FEATURE_V8); - set_feature(&cpu->env, ARM_FEATURE_VFP4); - set_feature(&cpu->env, ARM_FEATURE_NEON); - set_feature(&cpu->env, ARM_FEATURE_THUMB2EE); set_feature(&cpu->env, ARM_FEATURE_V8_AES); set_feature(&cpu->env, ARM_FEATURE_V8_SHA1); set_feature(&cpu->env, ARM_FEATURE_V8_SHA256); -- 2.17.1 From: Richard Henderson <richard.henderson@linaro.org> This register was added to aa32 state by ARMv8.2. Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Message-id: 20180629001538.11415-6-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/cpu.h | 1 + target/arm/cpu.c | 4 ++++ target/arm/cpu64.c | 2 ++ target/arm/helper.c | 5 ++--- 4 files changed, 9 insertions(+), 3 deletions(-) diff --git a/target/arm/cpu.h b/target/arm/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -XXX,XX +XXX,XX @@ struct ARMCPU { uint32_t id_isar3; uint32_t id_isar4; uint32_t id_isar5; + uint32_t id_isar6; uint64_t id_aa64pfr0; uint64_t id_aa64pfr1; uint64_t id_aa64dfr0; diff --git a/target/arm/cpu.c b/target/arm/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/cpu.c +++ b/target/arm/cpu.c @@ -XXX,XX +XXX,XX @@ static void cortex_m3_initfn(Object *obj) cpu->id_isar3 = 0x01111110; cpu->id_isar4 = 0x01310102; cpu->id_isar5 = 0x00000000; + cpu->id_isar6 = 0x00000000; } static void cortex_m4_initfn(Object *obj) @@ -XXX,XX +XXX,XX @@ static void cortex_m4_initfn(Object *obj) cpu->id_isar3 = 0x01111110; cpu->id_isar4 = 0x01310102; cpu->id_isar5 = 0x00000000; + cpu->id_isar6 = 0x00000000; } static void cortex_m33_initfn(Object *obj) @@ -XXX,XX +XXX,XX @@ static void cortex_m33_initfn(Object *obj) cpu->id_isar3 = 0x01111131; cpu->id_isar4 = 0x01310132; cpu->id_isar5 = 0x00000000; + cpu->id_isar6 = 0x00000000; cpu->clidr = 0x00000000; cpu->ctr = 0x8000c000; } @@ -XXX,XX +XXX,XX @@ static void cortex_r5_initfn(Object *obj) cpu->id_isar3 = 0x01112131; cpu->id_isar4 = 0x0010142; cpu->id_isar5 = 0x0; + cpu->id_isar6 = 0x0; cpu->mp_is_up = true; cpu->pmsav7_dregion = 16; define_arm_cp_regs(cpu, cortexr5_cp_reginfo); diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/cpu64.c +++ b/target/arm/cpu64.c @@ -XXX,XX +XXX,XX @@ static void aarch64_a57_initfn(Object *obj) cpu->id_isar3 = 0x01112131; cpu->id_isar4 = 0x00011142; cpu->id_isar5 = 0x00011121; + cpu->id_isar6 = 0; cpu->id_aa64pfr0 = 0x00002222; cpu->id_aa64dfr0 = 0x10305106; cpu->pmceid0 = 0x00000000; @@ -XXX,XX +XXX,XX @@ static void aarch64_a53_initfn(Object *obj) cpu->id_isar3 = 0x01112131; cpu->id_isar4 = 0x00011142; cpu->id_isar5 = 0x00011121; + cpu->id_isar6 = 0; cpu->id_aa64pfr0 = 0x00002222; cpu->id_aa64dfr0 = 0x10305106; cpu->id_aa64isar0 = 0x00011120; diff --git a/target/arm/helper.c b/target/arm/helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper.c +++ b/target/arm/helper.c @@ -XXX,XX +XXX,XX @@ void register_cp_regs_for_features(ARMCPU *cpu) .opc0 = 3, .opc1 = 0, .crn = 0, .crm = 2, .opc2 = 6, .access = PL1_R, .type = ARM_CP_CONST, .resetvalue = cpu->id_mmfr4 }, - /* 7 is as yet unallocated and must RAZ */ - { .name = "ID_ISAR7_RESERVED", .state = ARM_CP_STATE_BOTH, + { .name = "ID_ISAR6", .state = ARM_CP_STATE_BOTH, .opc0 = 3, .opc1 = 0, .crn = 0, .crm = 2, .opc2 = 7, .access = PL1_R, .type = ARM_CP_CONST, - .resetvalue = 0 }, + .resetvalue = cpu->id_isar6 }, REGINFO_SENTINEL }; define_arm_cp_regs(cpu, v6_idregs); -- 2.17.1 Hi; here's the latest arm pullreq... -- PMM The following changes since commit 03a3a62fbd0aa5227e978eef3c67d3978aec9e5f: Merge tag 'for-upstream' of https://gitlab.com/bonzini/qemu into staging (2023-09-07 10:29:06 -0400) are available in the Git repository at: https://git.linaro.org/people/pmaydell/qemu-arm.git tags/pull-target-arm-20230908 for you to fetch changes up to c8f2eb5d414b788420b938f2ffdde891aa6c3ae8: arm/kvm: Enable support for KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE (2023-09-08 16:41:36 +0100) ---------------------------------------------------------------- target-arm queue: * New CPU type: cortex-a710 * Implement new architectural features: - FEAT_PACQARMA3 - FEAT_EPAC - FEAT_Pauth2 - FEAT_FPAC - FEAT_FPACCOMBINE - FEAT_TIDCP1 * Xilinx Versal: Model the CFU/CFI * Implement RMR_ELx registers * Implement handling of HCR_EL2.TIDCP trap bit * arm/kvm: Enable support for KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE * hw/intc/arm_gicv3_its: Avoid maybe-uninitialized error in get_vte() * target/arm: Do not use gen_mte_checkN in trans_STGP * arm64: Restore trapless ptimer access ---------------------------------------------------------------- Aaron Lindsay (6): target/arm: Add ID_AA64ISAR2_EL1 target/arm: Add feature detection for FEAT_Pauth2 and extensions target/arm: Implement FEAT_EPAC target/arm: Implement FEAT_Pauth2 target/arm: Inform helpers whether a PAC instruction is 'combined' target/arm: Implement FEAT_FPAC and FEAT_FPACCOMBINE Colton Lewis (1): arm64: Restore trapless ptimer access Francisco Iglesias (8): hw/misc: Introduce the Xilinx CFI interface hw/misc: Introduce a model of Xilinx Versal's CFU_APB hw/misc/xlnx-versal-cfu: Introduce a model of Xilinx Versal CFU_FDRO hw/misc/xlnx-versal-cfu: Introduce a model of Xilinx Versal's CFU_SFR hw/misc: Introduce a model of Xilinx Versal's CFRAME_REG hw/misc: Introduce a model of Xilinx Versal's CFRAME_BCAST_REG hw/arm/xlnx-versal: Connect the CFU_APB, CFU_FDRO and CFU_SFR hw/arm/versal: Connect the CFRAME_REG and CFRAME_BCAST_REG Philippe Mathieu-Daudé (1): hw/intc/arm_gicv3_its: Avoid maybe-uninitialized error in get_vte() Richard Henderson (9): tests/tcg/aarch64: Adjust pauth tests for FEAT_FPAC target/arm: Don't change pauth features when changing algorithm target/arm: Implement FEAT_PACQARMA3 target/arm: Do not use gen_mte_checkN in trans_STGP target/arm: Implement RMR_ELx target/arm: Implement cortex-a710 target/arm: Implement HCR_EL2.TIDCP target/arm: Implement FEAT_TIDCP1 target/arm: Enable SCTLR_EL1.TIDCP for user-only Shameer Kolothum (1): arm/kvm: Enable support for KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE MAINTAINERS | 10 + docs/system/arm/cpu-features.rst | 21 +- docs/system/arm/emulation.rst | 8 + docs/system/arm/virt.rst | 1 + include/hw/arm/xlnx-versal.h | 85 +++ include/hw/misc/xlnx-cfi-if.h | 59 +++ include/hw/misc/xlnx-versal-cframe-reg.h | 303 +++++++++++ include/hw/misc/xlnx-versal-cfu.h | 258 ++++++++++ include/sysemu/kvm_int.h | 1 + target/arm/cpu.h | 54 +- target/arm/helper.h | 2 + target/arm/syndrome.h | 7 + target/arm/tcg/helper-a64.h | 4 + tests/tcg/aarch64/pauth.h | 23 + accel/kvm/kvm-all.c | 1 + hw/arm/virt.c | 1 + hw/arm/xlnx-versal.c | 155 +++++- hw/intc/arm_gicv3_its.c | 15 +- hw/misc/xlnx-cfi-if.c | 34 ++ hw/misc/xlnx-versal-cframe-reg.c | 858 +++++++++++++++++++++++++++++++ hw/misc/xlnx-versal-cfu.c | 563 ++++++++++++++++++++ target/arm/arm-qmp-cmds.c | 2 +- target/arm/cpu.c | 4 + target/arm/cpu64.c | 86 +++- target/arm/helper.c | 68 ++- target/arm/hvf/hvf.c | 1 + target/arm/kvm.c | 61 +++ target/arm/kvm64.c | 3 + target/arm/tcg/cpu64.c | 215 ++++++++ target/arm/tcg/op_helper.c | 33 ++ target/arm/tcg/pauth_helper.c | 180 +++++-- target/arm/tcg/translate-a64.c | 74 +-- target/arm/tcg/translate.c | 33 ++ tests/qtest/arm-cpu-features.c | 12 +- tests/tcg/aarch64/pauth-2.c | 54 +- tests/tcg/aarch64/pauth-4.c | 18 +- tests/tcg/aarch64/pauth-5.c | 10 + hw/misc/meson.build | 3 + qemu-options.hx | 15 + tests/tcg/aarch64/Makefile.target | 6 +- 40 files changed, 3184 insertions(+), 157 deletions(-) create mode 100644 include/hw/misc/xlnx-cfi-if.h create mode 100644 include/hw/misc/xlnx-versal-cframe-reg.h create mode 100644 include/hw/misc/xlnx-versal-cfu.h create mode 100644 tests/tcg/aarch64/pauth.h create mode 100644 hw/misc/xlnx-cfi-if.c create mode 100644 hw/misc/xlnx-versal-cframe-reg.c create mode 100644 hw/misc/xlnx-versal-cfu.c From: Richard Henderson <richard.henderson@linaro.org> With FEAT_FPAC, AUT* instructions that fail authentication do not produce an error value but instead fault. For pauth-2, install a signal handler and verify it gets called. For pauth-4 and pauth-5, we are explicitly testing the error value, so there's nothing to test with FEAT_FPAC, so exit early. Adjust the makefile to use -cpu neoverse-v1, which has FEAT_EPAC but not FEAT_FPAC. Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20230829232335.965414-2-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- tests/tcg/aarch64/pauth.h | 23 +++++++++++++ tests/tcg/aarch64/pauth-2.c | 54 ++++++++++++++++++++++++++----- tests/tcg/aarch64/pauth-4.c | 18 ++++++++--- tests/tcg/aarch64/pauth-5.c | 10 ++++++ tests/tcg/aarch64/Makefile.target | 6 +++- 5 files changed, 98 insertions(+), 13 deletions(-) create mode 100644 tests/tcg/aarch64/pauth.h diff --git a/tests/tcg/aarch64/pauth.h b/tests/tcg/aarch64/pauth.h new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/tests/tcg/aarch64/pauth.h @@ -XXX,XX +XXX,XX @@ +/* + * Helper for pauth test case + * + * Copyright (c) 2023 Linaro Ltd + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#include <assert.h> +#include <sys/auxv.h> + +static int get_pac_feature(void) +{ + unsigned long isar1, isar2; + + assert(getauxval(AT_HWCAP) & HWCAP_CPUID); + + asm("mrs %0, id_aa64isar1_el1" : "=r"(isar1)); + asm("mrs %0, S3_0_C0_C6_2" : "=r"(isar2)); /* id_aa64isar2_el1 */ + + return ((isar1 >> 4) & 0xf) /* APA */ + | ((isar1 >> 8) & 0xf) /* API */ + | ((isar2 >> 12) & 0xf); /* APA3 */ +} diff --git a/tests/tcg/aarch64/pauth-2.c b/tests/tcg/aarch64/pauth-2.c index XXXXXXX..XXXXXXX 100644 --- a/tests/tcg/aarch64/pauth-2.c +++ b/tests/tcg/aarch64/pauth-2.c @@ -XXX,XX +XXX,XX @@ #include <stdint.h> +#include <signal.h> +#include <stdlib.h> #include <assert.h> +#include "pauth.h" + + +static void sigill(int sig, siginfo_t *info, void *vuc) +{ + ucontext_t *uc = vuc; + uint64_t test; + + /* There is only one insn below that is allowed to fault. */ + asm volatile("adr %0, auth2_insn" : "=r"(test)); + assert(test == uc->uc_mcontext.pc); + exit(0); +} + +static int pac_feature; void do_test(uint64_t value) { @@ -XXX,XX +XXX,XX @@ void do_test(uint64_t value) * An invalid salt usually fails authorization, but again there * is a chance of choosing another salt that works. * Iterate until we find another salt which does fail. + * + * With FEAT_FPAC, this will SIGILL instead of producing a result. */ for (salt2 = salt1 + 1; ; salt2++) { - asm volatile("autda %0, %2" : "=r"(decode) : "0"(encode), "r"(salt2)); + asm volatile("auth2_insn: autda %0, %2" + : "=r"(decode) : "0"(encode), "r"(salt2)); if (decode != value) { break; } } + assert(pac_feature < 4); /* No FEAT_FPAC */ + /* The VA bits, bit 55, and the TBI bits, should be unchanged. */ assert(((decode ^ value) & 0xff80ffffffffffffull) == 0); /* - * Bits [54:53] are an error indicator based on the key used; - * the DA key above is keynumber 0, so error == 0b01. Otherwise - * bit 55 of the original is sign-extended into the rest of the auth. + * Without FEAT_Pauth2, bits [54:53] are an error indicator based on + * the key used; the DA key above is keynumber 0, so error == 0b01. + * Otherwise, bit 55 of the original is sign-extended into the rest + * of the auth. */ - if ((value >> 55) & 1) { - assert(((decode >> 48) & 0xff) == 0b10111111); - } else { - assert(((decode >> 48) & 0xff) == 0b00100000); + if (pac_feature < 3) { + if ((value >> 55) & 1) { + assert(((decode >> 48) & 0xff) == 0b10111111); + } else { + assert(((decode >> 48) & 0xff) == 0b00100000); + } } } int main() { + static const struct sigaction sa = { + .sa_sigaction = sigill, + .sa_flags = SA_SIGINFO + }; + + pac_feature = get_pac_feature(); + assert(pac_feature != 0); + + if (pac_feature >= 4) { + /* FEAT_FPAC */ + sigaction(SIGILL, &sa, NULL); + } + do_test(0); do_test(0xda004acedeadbeefull); return 0; diff --git a/tests/tcg/aarch64/pauth-4.c b/tests/tcg/aarch64/pauth-4.c index XXXXXXX..XXXXXXX 100644 --- a/tests/tcg/aarch64/pauth-4.c +++ b/tests/tcg/aarch64/pauth-4.c @@ -XXX,XX +XXX,XX @@ #include <assert.h> #include <stdio.h> #include <stdlib.h> +#include "pauth.h" #define TESTS 1000 int main() { + char base[TESTS]; int i, count = 0; float perc; - void *base = malloc(TESTS); + int pac_feature = get_pac_feature(); + + /* + * Exit if no PAuth or FEAT_FPAC, which will SIGILL on AUTIA failure + * rather than return an error for us to check below. + */ + if (pac_feature == 0 || pac_feature >= 4) { + return 0; + } for (i = 0; i < TESTS; i++) { uintptr_t in, x, y; @@ -XXX,XX +XXX,XX @@ int main() in = i + (uintptr_t) base; asm("mov %0, %[in]\n\t" - "pacia %0, sp\n\t" /* sigill if pauth not supported */ + "pacia %0, sp\n\t" "eor %0, %0, #4\n\t" /* corrupt single bit */ "mov %1, %0\n\t" "autia %1, sp\n\t" /* validate corrupted pointer */ @@ -XXX,XX +XXX,XX @@ int main() if (x != y) { count++; } - } + perc = (float) count / (float) TESTS; - printf("Checks Passed: %0.2f%%", perc * 100.0); + printf("Checks Passed: %0.2f%%\n", perc * 100.0); assert(perc > 0.95); return 0; } diff --git a/tests/tcg/aarch64/pauth-5.c b/tests/tcg/aarch64/pauth-5.c index XXXXXXX..XXXXXXX 100644 --- a/tests/tcg/aarch64/pauth-5.c +++ b/tests/tcg/aarch64/pauth-5.c @@ -XXX,XX +XXX,XX @@ #include <assert.h> +#include "pauth.h" static int x; @@ -XXX,XX +XXX,XX @@ int main() { int *p0 = &x, *p1, *p2, *p3; unsigned long salt = 0; + int pac_feature = get_pac_feature(); + + /* + * Exit if no PAuth or FEAT_FPAC, which will SIGILL on AUTDA failure + * rather than return an error for us to check below. + */ + if (pac_feature == 0 || pac_feature >= 4) { + return 0; + } /* * With TBI enabled and a 48-bit VA, there are 7 bits of auth, and so diff --git a/tests/tcg/aarch64/Makefile.target b/tests/tcg/aarch64/Makefile.target index XXXXXXX..XXXXXXX 100644 --- a/tests/tcg/aarch64/Makefile.target +++ b/tests/tcg/aarch64/Makefile.target @@ -XXX,XX +XXX,XX @@ endif ifneq ($(CROSS_CC_HAS_ARMV8_3),) AARCH64_TESTS += pauth-1 pauth-2 pauth-4 pauth-5 pauth-%: CFLAGS += -march=armv8.3-a -run-pauth-%: QEMU_OPTS += -cpu max +run-pauth-1: QEMU_OPTS += -cpu max +run-pauth-2: QEMU_OPTS += -cpu max +# Choose a cpu with FEAT_Pauth but without FEAT_FPAC for pauth-[45]. +run-pauth-4: QEMU_OPTS += -cpu neoverse-v1 +run-pauth-5: QEMU_OPTS += -cpu neoverse-v1 endif # BTI Tests -- 2.34.1 From: Aaron Lindsay <aaron@os.amperecomputing.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Aaron Lindsay <aaron@os.amperecomputing.com> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20230829232335.965414-3-richard.henderson@linaro.org [PMM: drop the HVF part of the patch and just comment that we need to do something when the register appears in that API] Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/cpu.h | 1 + target/arm/helper.c | 4 ++-- target/arm/hvf/hvf.c | 1 + target/arm/kvm64.c | 2 ++ 4 files changed, 6 insertions(+), 2 deletions(-) diff --git a/target/arm/cpu.h b/target/arm/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -XXX,XX +XXX,XX @@ struct ArchCPU { uint32_t dbgdevid1; uint64_t id_aa64isar0; uint64_t id_aa64isar1; + uint64_t id_aa64isar2; uint64_t id_aa64pfr0; uint64_t id_aa64pfr1; uint64_t id_aa64mmfr0; diff --git a/target/arm/helper.c b/target/arm/helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper.c +++ b/target/arm/helper.c @@ -XXX,XX +XXX,XX @@ void register_cp_regs_for_features(ARMCPU *cpu) .access = PL1_R, .type = ARM_CP_CONST, .accessfn = access_aa64_tid3, .resetvalue = cpu->isar.id_aa64isar1 }, - { .name = "ID_AA64ISAR2_EL1_RESERVED", .state = ARM_CP_STATE_AA64, + { .name = "ID_AA64ISAR2_EL1", .state = ARM_CP_STATE_AA64, .opc0 = 3, .opc1 = 0, .crn = 0, .crm = 6, .opc2 = 2, .access = PL1_R, .type = ARM_CP_CONST, .accessfn = access_aa64_tid3, - .resetvalue = 0 }, + .resetvalue = cpu->isar.id_aa64isar2 }, { .name = "ID_AA64ISAR3_EL1_RESERVED", .state = ARM_CP_STATE_AA64, .opc0 = 3, .opc1 = 0, .crn = 0, .crm = 6, .opc2 = 3, .access = PL1_R, .type = ARM_CP_CONST, diff --git a/target/arm/hvf/hvf.c b/target/arm/hvf/hvf.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/hvf/hvf.c +++ b/target/arm/hvf/hvf.c @@ -XXX,XX +XXX,XX @@ static bool hvf_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf) { HV_SYS_REG_ID_AA64DFR1_EL1, &host_isar.id_aa64dfr1 }, { HV_SYS_REG_ID_AA64ISAR0_EL1, &host_isar.id_aa64isar0 }, { HV_SYS_REG_ID_AA64ISAR1_EL1, &host_isar.id_aa64isar1 }, + /* Add ID_AA64ISAR2_EL1 here when HVF supports it */ { HV_SYS_REG_ID_AA64MMFR0_EL1, &host_isar.id_aa64mmfr0 }, { HV_SYS_REG_ID_AA64MMFR1_EL1, &host_isar.id_aa64mmfr1 }, { HV_SYS_REG_ID_AA64MMFR2_EL1, &host_isar.id_aa64mmfr2 }, diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/kvm64.c +++ b/target/arm/kvm64.c @@ -XXX,XX +XXX,XX @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf) ARM64_SYS_REG(3, 0, 0, 6, 0)); err |= read_sys_reg64(fdarray[2], &ahcf->isar.id_aa64isar1, ARM64_SYS_REG(3, 0, 0, 6, 1)); + err |= read_sys_reg64(fdarray[2], &ahcf->isar.id_aa64isar2, + ARM64_SYS_REG(3, 0, 0, 6, 2)); err |= read_sys_reg64(fdarray[2], &ahcf->isar.id_aa64mmfr0, ARM64_SYS_REG(3, 0, 0, 7, 0)); err |= read_sys_reg64(fdarray[2], &ahcf->isar.id_aa64mmfr1, -- 2.34.1 From: Aaron Lindsay <aaron@os.amperecomputing.com> Rename isar_feature_aa64_pauth_arch to isar_feature_aa64_pauth_qarma5 to distinguish the other architectural algorithm qarma3. Add ARMPauthFeature and isar_feature_pauth_feature to cover the other pauth conditions. Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Aaron Lindsay <aaron@os.amperecomputing.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20230829232335.965414-4-richard.henderson@linaro.org Message-Id: <20230609172324.982888-3-aaron@os.amperecomputing.com> [rth: Add ARMPauthFeature and eliminate most other predicates] Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/cpu.h | 47 +++++++++++++++++++++++++++++------ target/arm/tcg/pauth_helper.c | 2 +- 2 files changed, 40 insertions(+), 9 deletions(-) diff --git a/target/arm/cpu.h b/target/arm/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -XXX,XX +XXX,XX @@ static inline bool isar_feature_aa64_fcma(const ARMISARegisters *id) return FIELD_EX64(id->id_aa64isar1, ID_AA64ISAR1, FCMA) != 0; } +/* + * These are the values from APA/API/APA3. + * In general these must be compared '>=', per the normal Arm ARM + * treatment of fields in ID registers. + */ +typedef enum { + PauthFeat_None = 0, + PauthFeat_1 = 1, + PauthFeat_EPAC = 2, + PauthFeat_2 = 3, + PauthFeat_FPAC = 4, + PauthFeat_FPACCOMBINED = 5, +} ARMPauthFeature; + +static inline ARMPauthFeature +isar_feature_pauth_feature(const ARMISARegisters *id) +{ + /* + * Architecturally, only one of {APA,API,APA3} may be active (non-zero) + * and the other two must be zero. Thus we may avoid conditionals. + */ + return (FIELD_EX64(id->id_aa64isar1, ID_AA64ISAR1, APA) | + FIELD_EX64(id->id_aa64isar1, ID_AA64ISAR1, API) | + FIELD_EX64(id->id_aa64isar2, ID_AA64ISAR2, APA3)); +} + static inline bool isar_feature_aa64_pauth(const ARMISARegisters *id) { /* * Return true if any form of pauth is enabled, as this * predicate controls migration of the 128-bit keys. */ - return (id->id_aa64isar1 & - (FIELD_DP64(0, ID_AA64ISAR1, APA, 0xf) | - FIELD_DP64(0, ID_AA64ISAR1, API, 0xf) | - FIELD_DP64(0, ID_AA64ISAR1, GPA, 0xf) | - FIELD_DP64(0, ID_AA64ISAR1, GPI, 0xf))) != 0; + return isar_feature_pauth_feature(id) != PauthFeat_None; } -static inline bool isar_feature_aa64_pauth_arch(const ARMISARegisters *id) +static inline bool isar_feature_aa64_pauth_qarma5(const ARMISARegisters *id) { /* - * Return true if pauth is enabled with the architected QARMA algorithm. - * QEMU will always set APA+GPA to the same value. + * Return true if pauth is enabled with the architected QARMA5 algorithm. + * QEMU will always enable or disable both APA and GPA. */ return FIELD_EX64(id->id_aa64isar1, ID_AA64ISAR1, APA) != 0; } +static inline bool isar_feature_aa64_pauth_qarma3(const ARMISARegisters *id) +{ + /* + * Return true if pauth is enabled with the architected QARMA3 algorithm. + * QEMU will always enable or disable both APA3 and GPA3. + */ + return FIELD_EX64(id->id_aa64isar2, ID_AA64ISAR2, APA3) != 0; +} + static inline bool isar_feature_aa64_tlbirange(const ARMISARegisters *id) { return FIELD_EX64(id->id_aa64isar0, ID_AA64ISAR0, TLB) == 2; diff --git a/target/arm/tcg/pauth_helper.c b/target/arm/tcg/pauth_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/pauth_helper.c +++ b/target/arm/tcg/pauth_helper.c @@ -XXX,XX +XXX,XX @@ static uint64_t pauth_computepac_impdef(uint64_t data, uint64_t modifier, static uint64_t pauth_computepac(CPUARMState *env, uint64_t data, uint64_t modifier, ARMPACKey key) { - if (cpu_isar_feature(aa64_pauth_arch, env_archcpu(env))) { + if (cpu_isar_feature(aa64_pauth_qarma5, env_archcpu(env))) { return pauth_computepac_architected(data, modifier, key); } else { return pauth_computepac_impdef(data, modifier, key); -- 2.34.1 From: Richard Henderson <richard.henderson@linaro.org> We have cpu properties to adjust the pauth algorithm for the purpose of speed of emulation. Retain the set of pauth features supported by the cpu even as the algorithm changes. This already affects the neoverse-v1 cpu, which has FEAT_EPAC. Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20230829232335.965414-5-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/cpu64.c | 70 +++++++++++++++++++++++++++--------------- target/arm/tcg/cpu64.c | 2 ++ 2 files changed, 47 insertions(+), 25 deletions(-) diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/cpu64.c +++ b/target/arm/cpu64.c @@ -XXX,XX +XXX,XX @@ void aarch64_add_sme_properties(Object *obj) void arm_cpu_pauth_finalize(ARMCPU *cpu, Error **errp) { - int arch_val = 0, impdef_val = 0; - uint64_t t; + ARMPauthFeature features = cpu_isar_feature(pauth_feature, cpu); + uint64_t isar1; - /* Exit early if PAuth is enabled, and fall through to disable it */ - if ((kvm_enabled() || hvf_enabled()) && cpu->prop_pauth) { - if (!cpu_isar_feature(aa64_pauth, cpu)) { - error_setg(errp, "'pauth' feature not supported by %s on this host", - kvm_enabled() ? "KVM" : "hvf"); + /* + * These properties enable or disable Pauth as a whole, or change + * the pauth algorithm, but do not change the set of features that + * are present. We have saved a copy of those features above and + * will now place it into the field that chooses the algorithm. + * + * Begin by disabling all fields. + */ + isar1 = cpu->isar.id_aa64isar1; + isar1 = FIELD_DP64(isar1, ID_AA64ISAR1, APA, 0); + isar1 = FIELD_DP64(isar1, ID_AA64ISAR1, GPA, 0); + isar1 = FIELD_DP64(isar1, ID_AA64ISAR1, API, 0); + isar1 = FIELD_DP64(isar1, ID_AA64ISAR1, GPI, 0); + + if (kvm_enabled() || hvf_enabled()) { + /* + * Exit early if PAuth is enabled and fall through to disable it. + * The algorithm selection properties are not present. + */ + if (cpu->prop_pauth) { + if (features == 0) { + error_setg(errp, "'pauth' feature not supported by " + "%s on this host", current_accel_name()); + } + return; + } + } else { + /* Pauth properties are only present when the model supports it. */ + if (features == 0) { + assert(!cpu->prop_pauth); + return; } - return; - } - - /* TODO: Handle HaveEnhancedPAC, HaveEnhancedPAC2, HaveFPAC. */ - if (cpu->prop_pauth) { - if (cpu->prop_pauth_impdef) { - impdef_val = 1; - } else { - arch_val = 1; + if (cpu->prop_pauth) { + if (cpu->prop_pauth_impdef) { + isar1 = FIELD_DP64(isar1, ID_AA64ISAR1, API, features); + isar1 = FIELD_DP64(isar1, ID_AA64ISAR1, GPI, 1); + } else { + isar1 = FIELD_DP64(isar1, ID_AA64ISAR1, APA, features); + isar1 = FIELD_DP64(isar1, ID_AA64ISAR1, GPA, 1); + } + } else if (cpu->prop_pauth_impdef) { + error_setg(errp, "cannot enable pauth-impdef without pauth"); + error_append_hint(errp, "Add pauth=on to the CPU property list.\n"); } - } else if (cpu->prop_pauth_impdef) { - error_setg(errp, "cannot enable pauth-impdef without pauth"); - error_append_hint(errp, "Add pauth=on to the CPU property list.\n"); } - t = cpu->isar.id_aa64isar1; - t = FIELD_DP64(t, ID_AA64ISAR1, APA, arch_val); - t = FIELD_DP64(t, ID_AA64ISAR1, GPA, arch_val); - t = FIELD_DP64(t, ID_AA64ISAR1, API, impdef_val); - t = FIELD_DP64(t, ID_AA64ISAR1, GPI, impdef_val); - cpu->isar.id_aa64isar1 = t; + cpu->isar.id_aa64isar1 = isar1; } static Property arm_cpu_pauth_property = diff --git a/target/arm/tcg/cpu64.c b/target/arm/tcg/cpu64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/cpu64.c +++ b/target/arm/tcg/cpu64.c @@ -XXX,XX +XXX,XX @@ void aarch64_max_tcg_initfn(Object *obj) t = cpu->isar.id_aa64isar1; t = FIELD_DP64(t, ID_AA64ISAR1, DPB, 2); /* FEAT_DPB2 */ + t = FIELD_DP64(t, ID_AA64ISAR1, APA, PauthFeat_1); + t = FIELD_DP64(t, ID_AA64ISAR1, API, 1); t = FIELD_DP64(t, ID_AA64ISAR1, JSCVT, 1); /* FEAT_JSCVT */ t = FIELD_DP64(t, ID_AA64ISAR1, FCMA, 1); /* FEAT_FCMA */ t = FIELD_DP64(t, ID_AA64ISAR1, LRCPC, 2); /* FEAT_LRCPC2 */ -- 2.34.1 From: Richard Henderson <richard.henderson@linaro.org> Implement the QARMA3 cryptographic algorithm for PAC calculation. Implement a cpu feature to select the algorithm and document it. Signed-off-by: Aaron Lindsay <aaron@os.amperecomputing.com> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20230829232335.965414-6-richard.henderson@linaro.org Message-Id: <20230609172324.982888-4-aaron@os.amperecomputing.com> [rth: Merge cpu feature addition from another patch.] Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- docs/system/arm/cpu-features.rst | 21 ++++++++----- docs/system/arm/emulation.rst | 3 ++ target/arm/cpu.h | 1 + target/arm/arm-qmp-cmds.c | 2 +- target/arm/cpu64.c | 24 ++++++++++++-- target/arm/tcg/pauth_helper.c | 54 ++++++++++++++++++++++++++------ tests/qtest/arm-cpu-features.c | 12 ++++++- 7 files changed, 94 insertions(+), 23 deletions(-) diff --git a/docs/system/arm/cpu-features.rst b/docs/system/arm/cpu-features.rst index XXXXXXX..XXXXXXX 100644 --- a/docs/system/arm/cpu-features.rst +++ b/docs/system/arm/cpu-features.rst @@ -XXX,XX +XXX,XX @@ TCG VCPU Features TCG VCPU features are CPU features that are specific to TCG. Below is the list of TCG VCPU features and their descriptions. -``pauth-impdef`` - When ``FEAT_Pauth`` is enabled, either the *impdef* (Implementation - Defined) algorithm is enabled or the *architected* QARMA algorithm - is enabled. By default the impdef algorithm is disabled, and QARMA - is enabled. +``pauth`` + Enable or disable ``FEAT_Pauth`` entirely. - The architected QARMA algorithm has good cryptographic properties, - but can be quite slow to emulate. The impdef algorithm used by QEMU - is non-cryptographic but significantly faster. +``pauth-impdef`` + When ``pauth`` is enabled, select the QEMU implementation defined algorithm. + +``pauth-qarma3`` + When ``pauth`` is enabled, select the architected QARMA3 algorithm. + +Without either ``pauth-impdef`` or ``pauth-qarma3`` enabled, +the architected QARMA5 algorithm is used. The architected QARMA5 +and QARMA3 algorithms have good cryptographic properties, but can +be quite slow to emulate. The impdef algorithm used by QEMU is +non-cryptographic but significantly faster. SVE CPU Properties ================== diff --git a/docs/system/arm/emulation.rst b/docs/system/arm/emulation.rst index XXXXXXX..XXXXXXX 100644 --- a/docs/system/arm/emulation.rst +++ b/docs/system/arm/emulation.rst @@ -XXX,XX +XXX,XX @@ the following architecture extensions: - FEAT_MTE (Memory Tagging Extension) - FEAT_MTE2 (Memory Tagging Extension) - FEAT_MTE3 (MTE Asymmetric Fault Handling) +- FEAT_PACIMP (Pointer authentication - IMPLEMENTATION DEFINED algorithm) +- FEAT_PACQARMA3 (Pointer authentication - QARMA3 algorithm) +- FEAT_PACQARMA5 (Pointer authentication - QARMA5 algorithm) - FEAT_PAN (Privileged access never) - FEAT_PAN2 (AT S1E1R and AT S1E1W instruction variants affected by PSTATE.PAN) - FEAT_PAN3 (Support for SCTLR_ELx.EPAN) diff --git a/target/arm/cpu.h b/target/arm/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -XXX,XX +XXX,XX @@ struct ArchCPU { */ bool prop_pauth; bool prop_pauth_impdef; + bool prop_pauth_qarma3; bool prop_lpa2; /* DCZ blocksize, in log_2(words), ie low 4 bits of DCZID_EL0 */ diff --git a/target/arm/arm-qmp-cmds.c b/target/arm/arm-qmp-cmds.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/arm-qmp-cmds.c +++ b/target/arm/arm-qmp-cmds.c @@ -XXX,XX +XXX,XX @@ static const char *cpu_model_advertised_features[] = { "sve640", "sve768", "sve896", "sve1024", "sve1152", "sve1280", "sve1408", "sve1536", "sve1664", "sve1792", "sve1920", "sve2048", "kvm-no-adjvtime", "kvm-steal-time", - "pauth", "pauth-impdef", + "pauth", "pauth-impdef", "pauth-qarma3", NULL }; diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/cpu64.c +++ b/target/arm/cpu64.c @@ -XXX,XX +XXX,XX @@ void aarch64_add_sme_properties(Object *obj) void arm_cpu_pauth_finalize(ARMCPU *cpu, Error **errp) { ARMPauthFeature features = cpu_isar_feature(pauth_feature, cpu); - uint64_t isar1; + uint64_t isar1, isar2; /* * These properties enable or disable Pauth as a whole, or change @@ -XXX,XX +XXX,XX @@ void arm_cpu_pauth_finalize(ARMCPU *cpu, Error **errp) isar1 = FIELD_DP64(isar1, ID_AA64ISAR1, API, 0); isar1 = FIELD_DP64(isar1, ID_AA64ISAR1, GPI, 0); + isar2 = cpu->isar.id_aa64isar2; + isar2 = FIELD_DP64(isar2, ID_AA64ISAR2, APA3, 0); + isar2 = FIELD_DP64(isar2, ID_AA64ISAR2, GPA3, 0); + if (kvm_enabled() || hvf_enabled()) { /* * Exit early if PAuth is enabled and fall through to disable it. @@ -XXX,XX +XXX,XX @@ void arm_cpu_pauth_finalize(ARMCPU *cpu, Error **errp) } if (cpu->prop_pauth) { + if (cpu->prop_pauth_impdef && cpu->prop_pauth_qarma3) { + error_setg(errp, + "cannot enable both pauth-impdef and pauth-qarma3"); + return; + } + if (cpu->prop_pauth_impdef) { isar1 = FIELD_DP64(isar1, ID_AA64ISAR1, API, features); isar1 = FIELD_DP64(isar1, ID_AA64ISAR1, GPI, 1); + } else if (cpu->prop_pauth_qarma3) { + isar2 = FIELD_DP64(isar2, ID_AA64ISAR2, APA3, features); + isar2 = FIELD_DP64(isar2, ID_AA64ISAR2, GPA3, 1); } else { isar1 = FIELD_DP64(isar1, ID_AA64ISAR1, APA, features); isar1 = FIELD_DP64(isar1, ID_AA64ISAR1, GPA, 1); } - } else if (cpu->prop_pauth_impdef) { - error_setg(errp, "cannot enable pauth-impdef without pauth"); + } else if (cpu->prop_pauth_impdef || cpu->prop_pauth_qarma3) { + error_setg(errp, "cannot enable pauth-impdef or " + "pauth-qarma3 without pauth"); error_append_hint(errp, "Add pauth=on to the CPU property list.\n"); } } cpu->isar.id_aa64isar1 = isar1; + cpu->isar.id_aa64isar2 = isar2; } static Property arm_cpu_pauth_property = DEFINE_PROP_BOOL("pauth", ARMCPU, prop_pauth, true); static Property arm_cpu_pauth_impdef_property = DEFINE_PROP_BOOL("pauth-impdef", ARMCPU, prop_pauth_impdef, false); +static Property arm_cpu_pauth_qarma3_property = + DEFINE_PROP_BOOL("pauth-qarma3", ARMCPU, prop_pauth_qarma3, false); void aarch64_add_pauth_properties(Object *obj) { @@ -XXX,XX +XXX,XX @@ void aarch64_add_pauth_properties(Object *obj) cpu->prop_pauth = cpu_isar_feature(aa64_pauth, cpu); } else { qdev_property_add_static(DEVICE(obj), &arm_cpu_pauth_impdef_property); + qdev_property_add_static(DEVICE(obj), &arm_cpu_pauth_qarma3_property); } } diff --git a/target/arm/tcg/pauth_helper.c b/target/arm/tcg/pauth_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/pauth_helper.c +++ b/target/arm/tcg/pauth_helper.c @@ -XXX,XX +XXX,XX @@ static uint64_t pac_sub(uint64_t i) return o; } +static uint64_t pac_sub1(uint64_t i) +{ + static const uint8_t sub1[16] = { + 0xa, 0xd, 0xe, 0x6, 0xf, 0x7, 0x3, 0x5, + 0x9, 0x8, 0x0, 0xc, 0xb, 0x1, 0x2, 0x4, + }; + uint64_t o = 0; + int b; + + for (b = 0; b < 64; b += 4) { + o |= (uint64_t)sub1[(i >> b) & 0xf] << b; + } + return o; +} + static uint64_t pac_inv_sub(uint64_t i) { static const uint8_t inv_sub[16] = { @@ -XXX,XX +XXX,XX @@ static uint64_t tweak_inv_shuffle(uint64_t i) } static uint64_t pauth_computepac_architected(uint64_t data, uint64_t modifier, - ARMPACKey key) + ARMPACKey key, bool isqarma3) { static const uint64_t RC[5] = { 0x0000000000000000ull, @@ -XXX,XX +XXX,XX @@ static uint64_t pauth_computepac_architected(uint64_t data, uint64_t modifier, 0x452821E638D01377ull, }; const uint64_t alpha = 0xC0AC29B7C97C50DDull; + int iterations = isqarma3 ? 2 : 4; /* * Note that in the ARM pseudocode, key0 contains bits <127:64> * and key1 contains bits <63:0> of the 128-bit key. @@ -XXX,XX +XXX,XX @@ static uint64_t pauth_computepac_architected(uint64_t data, uint64_t modifier, runningmod = modifier; workingval = data ^ key0; - for (i = 0; i <= 4; ++i) { + for (i = 0; i <= iterations; ++i) { roundkey = key1 ^ runningmod; workingval ^= roundkey; workingval ^= RC[i]; @@ -XXX,XX +XXX,XX @@ static uint64_t pauth_computepac_architected(uint64_t data, uint64_t modifier, workingval = pac_cell_shuffle(workingval); workingval = pac_mult(workingval); } - workingval = pac_sub(workingval); + if (isqarma3) { + workingval = pac_sub1(workingval); + } else { + workingval = pac_sub(workingval); + } runningmod = tweak_shuffle(runningmod); } roundkey = modk0 ^ runningmod; workingval ^= roundkey; workingval = pac_cell_shuffle(workingval); workingval = pac_mult(workingval); - workingval = pac_sub(workingval); + if (isqarma3) { + workingval = pac_sub1(workingval); + } else { + workingval = pac_sub(workingval); + } workingval = pac_cell_shuffle(workingval); workingval = pac_mult(workingval); workingval ^= key1; workingval = pac_cell_inv_shuffle(workingval); - workingval = pac_inv_sub(workingval); + if (isqarma3) { + workingval = pac_sub1(workingval); + } else { + workingval = pac_inv_sub(workingval); + } workingval = pac_mult(workingval); workingval = pac_cell_inv_shuffle(workingval); workingval ^= key0; workingval ^= runningmod; - for (i = 0; i <= 4; ++i) { - workingval = pac_inv_sub(workingval); - if (i < 4) { + for (i = 0; i <= iterations; ++i) { + if (isqarma3) { + workingval = pac_sub1(workingval); + } else { + workingval = pac_inv_sub(workingval); + } + if (i < iterations) { workingval = pac_mult(workingval); workingval = pac_cell_inv_shuffle(workingval); } runningmod = tweak_inv_shuffle(runningmod); roundkey = key1 ^ runningmod; - workingval ^= RC[4 - i]; + workingval ^= RC[iterations - i]; workingval ^= roundkey; workingval ^= alpha; } @@ -XXX,XX +XXX,XX @@ static uint64_t pauth_computepac(CPUARMState *env, uint64_t data, uint64_t modifier, ARMPACKey key) { if (cpu_isar_feature(aa64_pauth_qarma5, env_archcpu(env))) { - return pauth_computepac_architected(data, modifier, key); + return pauth_computepac_architected(data, modifier, key, false); + } else if (cpu_isar_feature(aa64_pauth_qarma3, env_archcpu(env))) { + return pauth_computepac_architected(data, modifier, key, true); } else { return pauth_computepac_impdef(data, modifier, key); } diff --git a/tests/qtest/arm-cpu-features.c b/tests/qtest/arm-cpu-features.c index XXXXXXX..XXXXXXX 100644 --- a/tests/qtest/arm-cpu-features.c +++ b/tests/qtest/arm-cpu-features.c @@ -XXX,XX +XXX,XX @@ static void pauth_tests_default(QTestState *qts, const char *cpu_type) { assert_has_feature_enabled(qts, cpu_type, "pauth"); assert_has_feature_disabled(qts, cpu_type, "pauth-impdef"); + assert_has_feature_disabled(qts, cpu_type, "pauth-qarma3"); assert_set_feature(qts, cpu_type, "pauth", false); assert_set_feature(qts, cpu_type, "pauth", true); assert_set_feature(qts, cpu_type, "pauth-impdef", true); assert_set_feature(qts, cpu_type, "pauth-impdef", false); - assert_error(qts, cpu_type, "cannot enable pauth-impdef without pauth", + assert_set_feature(qts, cpu_type, "pauth-qarma3", true); + assert_set_feature(qts, cpu_type, "pauth-qarma3", false); + assert_error(qts, cpu_type, + "cannot enable pauth-impdef or pauth-qarma3 without pauth", "{ 'pauth': false, 'pauth-impdef': true }"); + assert_error(qts, cpu_type, + "cannot enable pauth-impdef or pauth-qarma3 without pauth", + "{ 'pauth': false, 'pauth-qarma3': true }"); + assert_error(qts, cpu_type, + "cannot enable both pauth-impdef and pauth-qarma3", + "{ 'pauth': true, 'pauth-impdef': true, 'pauth-qarma3': true }"); } static void test_query_cpu_model_expansion(const void *data) -- 2.34.1 From: Aaron Lindsay <aaron@os.amperecomputing.com> Signed-off-by: Aaron Lindsay <aaron@os.amperecomputing.com> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20230829232335.965414-7-richard.henderson@linaro.org Message-Id: <20230609172324.982888-5-aaron@os.amperecomputing.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- docs/system/arm/emulation.rst | 1 + target/arm/tcg/cpu64.c | 2 +- target/arm/tcg/pauth_helper.c | 16 +++++++++++----- 3 files changed, 13 insertions(+), 6 deletions(-) diff --git a/docs/system/arm/emulation.rst b/docs/system/arm/emulation.rst index XXXXXXX..XXXXXXX 100644 --- a/docs/system/arm/emulation.rst +++ b/docs/system/arm/emulation.rst @@ -XXX,XX +XXX,XX @@ the following architecture extensions: - FEAT_DotProd (Advanced SIMD dot product instructions) - FEAT_DoubleFault (Double Fault Extension) - FEAT_E0PD (Preventing EL0 access to halves of address maps) +- FEAT_EPAC (Enhanced pointer authentication) - FEAT_ETS (Enhanced Translation Synchronization) - FEAT_EVT (Enhanced Virtualization Traps) - FEAT_FCMA (Floating-point complex number instructions) diff --git a/target/arm/tcg/cpu64.c b/target/arm/tcg/cpu64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/cpu64.c +++ b/target/arm/tcg/cpu64.c @@ -XXX,XX +XXX,XX @@ void aarch64_max_tcg_initfn(Object *obj) t = cpu->isar.id_aa64isar1; t = FIELD_DP64(t, ID_AA64ISAR1, DPB, 2); /* FEAT_DPB2 */ - t = FIELD_DP64(t, ID_AA64ISAR1, APA, PauthFeat_1); + t = FIELD_DP64(t, ID_AA64ISAR1, APA, PauthFeat_EPAC); t = FIELD_DP64(t, ID_AA64ISAR1, API, 1); t = FIELD_DP64(t, ID_AA64ISAR1, JSCVT, 1); /* FEAT_JSCVT */ t = FIELD_DP64(t, ID_AA64ISAR1, FCMA, 1); /* FEAT_FCMA */ diff --git a/target/arm/tcg/pauth_helper.c b/target/arm/tcg/pauth_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/pauth_helper.c +++ b/target/arm/tcg/pauth_helper.c @@ -XXX,XX +XXX,XX @@ static uint64_t pauth_computepac(CPUARMState *env, uint64_t data, static uint64_t pauth_addpac(CPUARMState *env, uint64_t ptr, uint64_t modifier, ARMPACKey *key, bool data) { + ARMCPU *cpu = env_archcpu(env); ARMMMUIdx mmu_idx = arm_stage1_mmu_idx(env); ARMVAParameters param = aa64_va_parameters(env, ptr, mmu_idx, data, false); + ARMPauthFeature pauth_feature = cpu_isar_feature(pauth_feature, cpu); uint64_t pac, ext_ptr, ext, test; int bot_bit, top_bit; @@ -XXX,XX +XXX,XX @@ static uint64_t pauth_addpac(CPUARMState *env, uint64_t ptr, uint64_t modifier, */ test = sextract64(ptr, bot_bit, top_bit - bot_bit); if (test != 0 && test != -1) { - /* - * Note that our top_bit is one greater than the pseudocode's - * version, hence "- 2" here. - */ - pac ^= MAKE_64BIT_MASK(top_bit - 2, 1); + if (pauth_feature == PauthFeat_EPAC) { + pac = 0; + } else { + /* + * Note that our top_bit is one greater than the pseudocode's + * version, hence "- 2" here. + */ + pac ^= MAKE_64BIT_MASK(top_bit - 2, 1); + } } /* -- 2.34.1 From: Aaron Lindsay <aaron@os.amperecomputing.com> Signed-off-by: Aaron Lindsay <aaron@os.amperecomputing.com> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20230829232335.965414-8-richard.henderson@linaro.org Message-Id: <20230609172324.982888-6-aaron@os.amperecomputing.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- docs/system/arm/emulation.rst | 1 + target/arm/tcg/cpu64.c | 2 +- target/arm/tcg/pauth_helper.c | 21 +++++++++++++++++---- 3 files changed, 19 insertions(+), 5 deletions(-) diff --git a/docs/system/arm/emulation.rst b/docs/system/arm/emulation.rst index XXXXXXX..XXXXXXX 100644 --- a/docs/system/arm/emulation.rst +++ b/docs/system/arm/emulation.rst @@ -XXX,XX +XXX,XX @@ the following architecture extensions: - FEAT_PAN2 (AT S1E1R and AT S1E1W instruction variants affected by PSTATE.PAN) - FEAT_PAN3 (Support for SCTLR_ELx.EPAN) - FEAT_PAuth (Pointer authentication) +- FEAT_PAuth2 (Enhacements to pointer authentication) - FEAT_PMULL (PMULL, PMULL2 instructions) - FEAT_PMUv3p1 (PMU Extensions v3.1) - FEAT_PMUv3p4 (PMU Extensions v3.4) diff --git a/target/arm/tcg/cpu64.c b/target/arm/tcg/cpu64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/cpu64.c +++ b/target/arm/tcg/cpu64.c @@ -XXX,XX +XXX,XX @@ void aarch64_max_tcg_initfn(Object *obj) t = cpu->isar.id_aa64isar1; t = FIELD_DP64(t, ID_AA64ISAR1, DPB, 2); /* FEAT_DPB2 */ - t = FIELD_DP64(t, ID_AA64ISAR1, APA, PauthFeat_EPAC); + t = FIELD_DP64(t, ID_AA64ISAR1, APA, PauthFeat_2); t = FIELD_DP64(t, ID_AA64ISAR1, API, 1); t = FIELD_DP64(t, ID_AA64ISAR1, JSCVT, 1); /* FEAT_JSCVT */ t = FIELD_DP64(t, ID_AA64ISAR1, FCMA, 1); /* FEAT_FCMA */ diff --git a/target/arm/tcg/pauth_helper.c b/target/arm/tcg/pauth_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/pauth_helper.c +++ b/target/arm/tcg/pauth_helper.c @@ -XXX,XX +XXX,XX @@ static uint64_t pauth_addpac(CPUARMState *env, uint64_t ptr, uint64_t modifier, */ test = sextract64(ptr, bot_bit, top_bit - bot_bit); if (test != 0 && test != -1) { - if (pauth_feature == PauthFeat_EPAC) { + if (pauth_feature >= PauthFeat_2) { + /* No action required */ + } else if (pauth_feature == PauthFeat_EPAC) { pac = 0; } else { /* @@ -XXX,XX +XXX,XX @@ static uint64_t pauth_addpac(CPUARMState *env, uint64_t ptr, uint64_t modifier, * Preserve the determination between upper and lower at bit 55, * and insert pointer authentication code. */ + if (pauth_feature >= PauthFeat_2) { + pac ^= ptr; + } if (param.tbi) { ptr &= ~MAKE_64BIT_MASK(bot_bit, 55 - bot_bit + 1); pac &= MAKE_64BIT_MASK(bot_bit, 54 - bot_bit + 1); @@ -XXX,XX +XXX,XX @@ static uint64_t pauth_original_ptr(uint64_t ptr, ARMVAParameters param) static uint64_t pauth_auth(CPUARMState *env, uint64_t ptr, uint64_t modifier, ARMPACKey *key, bool data, int keynumber) { + ARMCPU *cpu = env_archcpu(env); ARMMMUIdx mmu_idx = arm_stage1_mmu_idx(env); ARMVAParameters param = aa64_va_parameters(env, ptr, mmu_idx, data, false); + ARMPauthFeature pauth_feature = cpu_isar_feature(pauth_feature, cpu); int bot_bit, top_bit; - uint64_t pac, orig_ptr, test; + uint64_t pac, orig_ptr, cmp_mask; orig_ptr = pauth_original_ptr(ptr, param); pac = pauth_computepac(env, orig_ptr, modifier, *key); bot_bit = 64 - param.tsz; top_bit = 64 - 8 * param.tbi; - test = (pac ^ ptr) & ~MAKE_64BIT_MASK(55, 1); - if (unlikely(extract64(test, bot_bit, top_bit - bot_bit))) { + cmp_mask = MAKE_64BIT_MASK(bot_bit, top_bit - bot_bit); + cmp_mask &= ~MAKE_64BIT_MASK(55, 1); + + if (pauth_feature >= PauthFeat_2) { + return ptr ^ (pac & cmp_mask); + } + + if ((pac ^ ptr) & cmp_mask) { int error_code = (keynumber << 1) | (keynumber ^ 1); if (param.tbi) { return deposit64(orig_ptr, 53, 2, error_code); -- 2.34.1 From: Aaron Lindsay <aaron@os.amperecomputing.com> An instruction is a 'combined' Pointer Authentication instruction if it does something in addition to PAC -- for instance, branching to or loading an address from the authenticated pointer. Knowing whether a PAC operation is 'combined' is needed to implement FEAT_FPACCOMBINE. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Aaron Lindsay <aaron@os.amperecomputing.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20230829232335.965414-9-richard.henderson@linaro.org Message-Id: <20230609172324.982888-7-aaron@os.amperecomputing.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/tcg/helper-a64.h | 4 ++ target/arm/tcg/pauth_helper.c | 71 +++++++++++++++++++++++++++------- target/arm/tcg/translate-a64.c | 12 +++--- 3 files changed, 68 insertions(+), 19 deletions(-) diff --git a/target/arm/tcg/helper-a64.h b/target/arm/tcg/helper-a64.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/helper-a64.h +++ b/target/arm/tcg/helper-a64.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_3(pacda, TCG_CALL_NO_WG, i64, env, i64, i64) DEF_HELPER_FLAGS_3(pacdb, TCG_CALL_NO_WG, i64, env, i64, i64) DEF_HELPER_FLAGS_3(pacga, TCG_CALL_NO_WG, i64, env, i64, i64) DEF_HELPER_FLAGS_3(autia, TCG_CALL_NO_WG, i64, env, i64, i64) +DEF_HELPER_FLAGS_3(autia_combined, TCG_CALL_NO_WG, i64, env, i64, i64) DEF_HELPER_FLAGS_3(autib, TCG_CALL_NO_WG, i64, env, i64, i64) +DEF_HELPER_FLAGS_3(autib_combined, TCG_CALL_NO_WG, i64, env, i64, i64) DEF_HELPER_FLAGS_3(autda, TCG_CALL_NO_WG, i64, env, i64, i64) +DEF_HELPER_FLAGS_3(autda_combined, TCG_CALL_NO_WG, i64, env, i64, i64) DEF_HELPER_FLAGS_3(autdb, TCG_CALL_NO_WG, i64, env, i64, i64) +DEF_HELPER_FLAGS_3(autdb_combined, TCG_CALL_NO_WG, i64, env, i64, i64) DEF_HELPER_FLAGS_2(xpaci, TCG_CALL_NO_RWG_SE, i64, env, i64) DEF_HELPER_FLAGS_2(xpacd, TCG_CALL_NO_RWG_SE, i64, env, i64) diff --git a/target/arm/tcg/pauth_helper.c b/target/arm/tcg/pauth_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/pauth_helper.c +++ b/target/arm/tcg/pauth_helper.c @@ -XXX,XX +XXX,XX @@ static uint64_t pauth_original_ptr(uint64_t ptr, ARMVAParameters param) } static uint64_t pauth_auth(CPUARMState *env, uint64_t ptr, uint64_t modifier, - ARMPACKey *key, bool data, int keynumber) + ARMPACKey *key, bool data, int keynumber, + uintptr_t ra, bool is_combined) { ARMCPU *cpu = env_archcpu(env); ARMMMUIdx mmu_idx = arm_stage1_mmu_idx(env); @@ -XXX,XX +XXX,XX @@ uint64_t HELPER(pacga)(CPUARMState *env, uint64_t x, uint64_t y) return pac & 0xffffffff00000000ull; } -uint64_t HELPER(autia)(CPUARMState *env, uint64_t x, uint64_t y) +static uint64_t pauth_autia(CPUARMState *env, uint64_t x, uint64_t y, + uintptr_t ra, bool is_combined) { int el = arm_current_el(env); if (!pauth_key_enabled(env, el, SCTLR_EnIA)) { return x; } - pauth_check_trap(env, el, GETPC()); - return pauth_auth(env, x, y, &env->keys.apia, false, 0); + pauth_check_trap(env, el, ra); + return pauth_auth(env, x, y, &env->keys.apia, false, 0, ra, is_combined); } -uint64_t HELPER(autib)(CPUARMState *env, uint64_t x, uint64_t y) +uint64_t HELPER(autia)(CPUARMState *env, uint64_t x, uint64_t y) +{ + return pauth_autia(env, x, y, GETPC(), false); +} + +uint64_t HELPER(autia_combined)(CPUARMState *env, uint64_t x, uint64_t y) +{ + return pauth_autia(env, x, y, GETPC(), true); +} + +static uint64_t pauth_autib(CPUARMState *env, uint64_t x, uint64_t y, + uintptr_t ra, bool is_combined) { int el = arm_current_el(env); if (!pauth_key_enabled(env, el, SCTLR_EnIB)) { return x; } - pauth_check_trap(env, el, GETPC()); - return pauth_auth(env, x, y, &env->keys.apib, false, 1); + pauth_check_trap(env, el, ra); + return pauth_auth(env, x, y, &env->keys.apib, false, 1, ra, is_combined); } -uint64_t HELPER(autda)(CPUARMState *env, uint64_t x, uint64_t y) +uint64_t HELPER(autib)(CPUARMState *env, uint64_t x, uint64_t y) +{ + return pauth_autib(env, x, y, GETPC(), false); +} + +uint64_t HELPER(autib_combined)(CPUARMState *env, uint64_t x, uint64_t y) +{ + return pauth_autib(env, x, y, GETPC(), true); +} + +static uint64_t pauth_autda(CPUARMState *env, uint64_t x, uint64_t y, + uintptr_t ra, bool is_combined) { int el = arm_current_el(env); if (!pauth_key_enabled(env, el, SCTLR_EnDA)) { return x; } - pauth_check_trap(env, el, GETPC()); - return pauth_auth(env, x, y, &env->keys.apda, true, 0); + pauth_check_trap(env, el, ra); + return pauth_auth(env, x, y, &env->keys.apda, true, 0, ra, is_combined); } -uint64_t HELPER(autdb)(CPUARMState *env, uint64_t x, uint64_t y) +uint64_t HELPER(autda)(CPUARMState *env, uint64_t x, uint64_t y) +{ + return pauth_autda(env, x, y, GETPC(), false); +} + +uint64_t HELPER(autda_combined)(CPUARMState *env, uint64_t x, uint64_t y) +{ + return pauth_autda(env, x, y, GETPC(), true); +} + +static uint64_t pauth_autdb(CPUARMState *env, uint64_t x, uint64_t y, + uintptr_t ra, bool is_combined) { int el = arm_current_el(env); if (!pauth_key_enabled(env, el, SCTLR_EnDB)) { return x; } - pauth_check_trap(env, el, GETPC()); - return pauth_auth(env, x, y, &env->keys.apdb, true, 1); + pauth_check_trap(env, el, ra); + return pauth_auth(env, x, y, &env->keys.apdb, true, 1, ra, is_combined); +} + +uint64_t HELPER(autdb)(CPUARMState *env, uint64_t x, uint64_t y) +{ + return pauth_autdb(env, x, y, GETPC(), false); +} + +uint64_t HELPER(autdb_combined)(CPUARMState *env, uint64_t x, uint64_t y) +{ + return pauth_autdb(env, x, y, GETPC(), true); } uint64_t HELPER(xpaci)(CPUARMState *env, uint64_t a) diff --git a/target/arm/tcg/translate-a64.c b/target/arm/tcg/translate-a64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/translate-a64.c +++ b/target/arm/tcg/translate-a64.c @@ -XXX,XX +XXX,XX @@ static TCGv_i64 auth_branch_target(DisasContext *s, TCGv_i64 dst, truedst = tcg_temp_new_i64(); if (use_key_a) { - gen_helper_autia(truedst, cpu_env, dst, modifier); + gen_helper_autia_combined(truedst, cpu_env, dst, modifier); } else { - gen_helper_autib(truedst, cpu_env, dst, modifier); + gen_helper_autib_combined(truedst, cpu_env, dst, modifier); } return truedst; } @@ -XXX,XX +XXX,XX @@ static bool trans_LDRA(DisasContext *s, arg_LDRA *a) if (s->pauth_active) { if (!a->m) { - gen_helper_autda(dirty_addr, cpu_env, dirty_addr, - tcg_constant_i64(0)); + gen_helper_autda_combined(dirty_addr, cpu_env, dirty_addr, + tcg_constant_i64(0)); } else { - gen_helper_autdb(dirty_addr, cpu_env, dirty_addr, - tcg_constant_i64(0)); + gen_helper_autdb_combined(dirty_addr, cpu_env, dirty_addr, + tcg_constant_i64(0)); } } -- 2.34.1 From: Aaron Lindsay <aaron@os.amperecomputing.com> Signed-off-by: Aaron Lindsay <aaron@os.amperecomputing.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20230829232335.965414-10-richard.henderson@linaro.org Message-Id: <20230609172324.982888-8-aaron@os.amperecomputing.com> [rth: Simplify fpac comparison, reusing cmp_mask] Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- docs/system/arm/emulation.rst | 2 ++ target/arm/syndrome.h | 7 +++++++ target/arm/tcg/cpu64.c | 2 +- target/arm/tcg/pauth_helper.c | 18 +++++++++++++++++- 4 files changed, 27 insertions(+), 2 deletions(-) diff --git a/docs/system/arm/emulation.rst b/docs/system/arm/emulation.rst index XXXXXXX..XXXXXXX 100644 --- a/docs/system/arm/emulation.rst +++ b/docs/system/arm/emulation.rst @@ -XXX,XX +XXX,XX @@ the following architecture extensions: - FEAT_FGT (Fine-Grained Traps) - FEAT_FHM (Floating-point half-precision multiplication instructions) - FEAT_FP16 (Half-precision floating-point data processing) +- FEAT_FPAC (Faulting on AUT* instructions) +- FEAT_FPACCOMBINE (Faulting on combined pointer authentication instructions) - FEAT_FRINTTS (Floating-point to integer instructions) - FEAT_FlagM (Flag manipulation instructions v2) - FEAT_FlagM2 (Enhancements to flag manipulation instructions) diff --git a/target/arm/syndrome.h b/target/arm/syndrome.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/syndrome.h +++ b/target/arm/syndrome.h @@ -XXX,XX +XXX,XX @@ enum arm_exception_class { EC_SYSTEMREGISTERTRAP = 0x18, EC_SVEACCESSTRAP = 0x19, EC_ERETTRAP = 0x1a, + EC_PACFAIL = 0x1c, EC_SMETRAP = 0x1d, EC_GPC = 0x1e, EC_INSNABORT = 0x20, @@ -XXX,XX +XXX,XX @@ static inline uint32_t syn_smetrap(SMEExceptionType etype, bool is_16bit) | (is_16bit ? 0 : ARM_EL_IL) | etype; } +static inline uint32_t syn_pacfail(bool data, int keynumber) +{ + int error_code = (data << 1) | keynumber; + return (EC_PACFAIL << ARM_EL_EC_SHIFT) | ARM_EL_IL | error_code; +} + static inline uint32_t syn_pactrap(void) { return EC_PACTRAP << ARM_EL_EC_SHIFT; diff --git a/target/arm/tcg/cpu64.c b/target/arm/tcg/cpu64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/cpu64.c +++ b/target/arm/tcg/cpu64.c @@ -XXX,XX +XXX,XX @@ void aarch64_max_tcg_initfn(Object *obj) t = cpu->isar.id_aa64isar1; t = FIELD_DP64(t, ID_AA64ISAR1, DPB, 2); /* FEAT_DPB2 */ - t = FIELD_DP64(t, ID_AA64ISAR1, APA, PauthFeat_2); + t = FIELD_DP64(t, ID_AA64ISAR1, APA, PauthFeat_FPACCOMBINED); t = FIELD_DP64(t, ID_AA64ISAR1, API, 1); t = FIELD_DP64(t, ID_AA64ISAR1, JSCVT, 1); /* FEAT_JSCVT */ t = FIELD_DP64(t, ID_AA64ISAR1, FCMA, 1); /* FEAT_FCMA */ diff --git a/target/arm/tcg/pauth_helper.c b/target/arm/tcg/pauth_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/pauth_helper.c +++ b/target/arm/tcg/pauth_helper.c @@ -XXX,XX +XXX,XX @@ static uint64_t pauth_original_ptr(uint64_t ptr, ARMVAParameters param) } } +static G_NORETURN +void pauth_fail_exception(CPUARMState *env, bool data, + int keynumber, uintptr_t ra) +{ + raise_exception_ra(env, EXCP_UDEF, syn_pacfail(data, keynumber), + exception_target_el(env), ra); +} + static uint64_t pauth_auth(CPUARMState *env, uint64_t ptr, uint64_t modifier, ARMPACKey *key, bool data, int keynumber, uintptr_t ra, bool is_combined) @@ -XXX,XX +XXX,XX @@ static uint64_t pauth_auth(CPUARMState *env, uint64_t ptr, uint64_t modifier, cmp_mask &= ~MAKE_64BIT_MASK(55, 1); if (pauth_feature >= PauthFeat_2) { - return ptr ^ (pac & cmp_mask); + ARMPauthFeature fault_feature = + is_combined ? PauthFeat_FPACCOMBINED : PauthFeat_FPAC; + uint64_t result = ptr ^ (pac & cmp_mask); + + if (pauth_feature >= fault_feature + && ((result ^ sextract64(result, 55, 1)) & cmp_mask)) { + pauth_fail_exception(env, data, keynumber, ra); + } + return result; } if ((pac ^ ptr) & cmp_mask) { -- 2.34.1 From: Philippe Mathieu-Daudé <philmd@linaro.org> Fix when using GCC v11.4 (Ubuntu 11.4.0-1ubuntu1~22.04) with CFLAGS=-Og: [4/6] Compiling C object libcommon.fa.p/hw_intc_arm_gicv3_its.c.o FAILED: libcommon.fa.p/hw_intc_arm_gicv3_its.c.o inlined from ‘lookup_vte’ at hw/intc/arm_gicv3_its.c:453:9, inlined from ‘vmovp_callback’ at hw/intc/arm_gicv3_its.c:1039:14: hw/intc/arm_gicv3_its.c:347:9: error: ‘vte.rdbase’ may be used uninitialized [-Werror=maybe-uninitialized] 347 | trace_gicv3_its_vte_read(vpeid, vte->valid, vte->vptsize, | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 348 | vte->vptaddr, vte->rdbase); | ~~~~~~~~~~~~~~~~~~~~~~~~~~ hw/intc/arm_gicv3_its.c: In function ‘vmovp_callback’: hw/intc/arm_gicv3_its.c:1036:13: note: ‘vte’ declared here 1036 | VTEntry vte; | ^~~ Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Message-id: 20230831131348.69032-1-philmd@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- hw/intc/arm_gicv3_its.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/hw/intc/arm_gicv3_its.c b/hw/intc/arm_gicv3_its.c index XXXXXXX..XXXXXXX 100644 --- a/hw/intc/arm_gicv3_its.c +++ b/hw/intc/arm_gicv3_its.c @@ -XXX,XX +XXX,XX @@ static MemTxResult get_vte(GICv3ITSState *s, uint32_t vpeid, VTEntry *vte) if (entry_addr == -1) { /* No L2 table entry, i.e. no valid VTE, or a memory error */ vte->valid = false; - goto out; + trace_gicv3_its_vte_read_fault(vpeid); + return MEMTX_OK; } vteval = address_space_ldq_le(as, entry_addr, MEMTXATTRS_UNSPECIFIED, &res); if (res != MEMTX_OK) { - goto out; + trace_gicv3_its_vte_read_fault(vpeid); + return res; } vte->valid = FIELD_EX64(vteval, VTE, VALID); vte->vptsize = FIELD_EX64(vteval, VTE, VPTSIZE); vte->vptaddr = FIELD_EX64(vteval, VTE, VPTADDR); vte->rdbase = FIELD_EX64(vteval, VTE, RDBASE); -out: - if (res != MEMTX_OK) { - trace_gicv3_its_vte_read_fault(vpeid); - } else { - trace_gicv3_its_vte_read(vpeid, vte->valid, vte->vptsize, - vte->vptaddr, vte->rdbase); - } + trace_gicv3_its_vte_read(vpeid, vte->valid, vte->vptsize, + vte->vptaddr, vte->rdbase); return res; } -- 2.34.1 From: Francisco Iglesias <francisco.iglesias@amd.com> Introduce the Xilinx Configuration Frame Interface (CFI) for transmitting CFI data packets between the Xilinx Configuration Frame Unit models (CFU_APB, CFU_FDRO and CFU_SFR), the Xilinx CFRAME controller (CFRAME_REG) and the Xilinx CFRAME broadcast controller (CFRAME_BCAST_REG) models (when emulating bitstream programming and readback). Signed-off-by: Francisco Iglesias <francisco.iglesias@amd.com> Reviewed-by: Sai Pavan Boddu <sai.pavan.boddu@amd.com> Acked-by: Edgar E. Iglesias <edgar@zeroasic.com> Message-id: 20230831165701.2016397-2-francisco.iglesias@amd.com Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- MAINTAINERS | 6 ++++ include/hw/misc/xlnx-cfi-if.h | 59 +++++++++++++++++++++++++++++++++++ hw/misc/xlnx-cfi-if.c | 34 ++++++++++++++++++++ hw/misc/meson.build | 1 + 4 files changed, 100 insertions(+) create mode 100644 include/hw/misc/xlnx-cfi-if.h create mode 100644 hw/misc/xlnx-cfi-if.c diff --git a/MAINTAINERS b/MAINTAINERS index XXXXXXX..XXXXXXX 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -XXX,XX +XXX,XX @@ S: Maintained F: hw/ssi/xlnx-versal-ospi.c F: include/hw/ssi/xlnx-versal-ospi.h +Xilinx Versal CFI +M: Francisco Iglesias <francisco.iglesias@amd.com> +S: Maintained +F: hw/misc/xlnx-cfi-if.c +F: include/hw/misc/xlnx-cfi-if.h + STM32F100 M: Alexandre Iooss <erdnaxe@crans.org> L: qemu-arm@nongnu.org diff --git a/include/hw/misc/xlnx-cfi-if.h b/include/hw/misc/xlnx-cfi-if.h new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/include/hw/misc/xlnx-cfi-if.h @@ -XXX,XX +XXX,XX @@ +/* + * Xilinx CFI interface + * + * Copyright (C) 2023, Advanced Micro Devices, Inc. + * + * Written by Francisco Iglesias <francisco.iglesias@amd.com> + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ +#ifndef XLNX_CFI_IF_H +#define XLNX_CFI_IF_H 1 + +#include "qemu/help-texts.h" +#include "hw/hw.h" +#include "qom/object.h" + +#define TYPE_XLNX_CFI_IF "xlnx-cfi-if" +typedef struct XlnxCfiIfClass XlnxCfiIfClass; +DECLARE_CLASS_CHECKERS(XlnxCfiIfClass, XLNX_CFI_IF, TYPE_XLNX_CFI_IF) + +#define XLNX_CFI_IF(obj) \ + INTERFACE_CHECK(XlnxCfiIf, (obj), TYPE_XLNX_CFI_IF) + +typedef enum { + PACKET_TYPE_CFU = 0x52, + PACKET_TYPE_CFRAME = 0xA1, +} xlnx_cfi_packet_type; + +typedef enum { + CFRAME_FAR = 1, + CFRAME_SFR = 2, + CFRAME_FDRI = 4, + CFRAME_CMD = 6, +} xlnx_cfi_reg_addr; + +typedef struct XlnxCfiPacket { + uint8_t reg_addr; + uint32_t data[4]; +} XlnxCfiPacket; + +typedef struct XlnxCfiIf { + Object Parent; +} XlnxCfiIf; + +typedef struct XlnxCfiIfClass { + InterfaceClass parent; + + void (*cfi_transfer_packet)(XlnxCfiIf *cfi_if, XlnxCfiPacket *pkt); +} XlnxCfiIfClass; + +/** + * Transfer a XlnxCfiPacket. + * + * @cfi_if: the object implementing this interface + * @XlnxCfiPacket: a pointer to the XlnxCfiPacket to transfer + */ +void xlnx_cfi_transfer_packet(XlnxCfiIf *cfi_if, XlnxCfiPacket *pkt); + +#endif /* XLNX_CFI_IF_H */ diff --git a/hw/misc/xlnx-cfi-if.c b/hw/misc/xlnx-cfi-if.c new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/hw/misc/xlnx-cfi-if.c @@ -XXX,XX +XXX,XX @@ +/* + * Xilinx CFI interface + * + * Copyright (C) 2023, Advanced Micro Devices, Inc. + * + * Written by Francisco Iglesias <francisco.iglesias@amd.com> + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ +#include "qemu/osdep.h" +#include "hw/misc/xlnx-cfi-if.h" + +void xlnx_cfi_transfer_packet(XlnxCfiIf *cfi_if, XlnxCfiPacket *pkt) +{ + XlnxCfiIfClass *xcic = XLNX_CFI_IF_GET_CLASS(cfi_if); + + if (xcic->cfi_transfer_packet) { + xcic->cfi_transfer_packet(cfi_if, pkt); + } +} + +static const TypeInfo xlnx_cfi_if_info = { + .name = TYPE_XLNX_CFI_IF, + .parent = TYPE_INTERFACE, + .class_size = sizeof(XlnxCfiIfClass), +}; + +static void xlnx_cfi_if_register_types(void) +{ + type_register_static(&xlnx_cfi_if_info); +} + +type_init(xlnx_cfi_if_register_types) + diff --git a/hw/misc/meson.build b/hw/misc/meson.build index XXXXXXX..XXXXXXX 100644 --- a/hw/misc/meson.build +++ b/hw/misc/meson.build @@ -XXX,XX +XXX,XX @@ specific_ss.add(when: 'CONFIG_XLNX_VERSAL', if_true: files('xlnx-versal-crl.c')) system_ss.add(when: 'CONFIG_XLNX_VERSAL', if_true: files( 'xlnx-versal-xramc.c', 'xlnx-versal-pmc-iou-slcr.c', + 'xlnx-cfi-if.c', )) system_ss.add(when: 'CONFIG_STM32F2XX_SYSCFG', if_true: files('stm32f2xx_syscfg.c')) system_ss.add(when: 'CONFIG_STM32F4XX_SYSCFG', if_true: files('stm32f4xx_syscfg.c')) -- 2.34.1 From: Francisco Iglesias <francisco.iglesias@amd.com> Introduce a model of the software programming interface (CFU_APB) of Xilinx Versal's Configuration Frame Unit. Signed-off-by: Francisco Iglesias <francisco.iglesias@amd.com> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Message-id: 20230831165701.2016397-3-francisco.iglesias@amd.com Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- MAINTAINERS | 2 + include/hw/misc/xlnx-versal-cfu.h | 231 ++++++++++++++++++ hw/misc/xlnx-versal-cfu.c | 380 ++++++++++++++++++++++++++++++ hw/misc/meson.build | 1 + 4 files changed, 614 insertions(+) create mode 100644 include/hw/misc/xlnx-versal-cfu.h create mode 100644 hw/misc/xlnx-versal-cfu.c diff --git a/MAINTAINERS b/MAINTAINERS index XXXXXXX..XXXXXXX 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -XXX,XX +XXX,XX @@ M: Francisco Iglesias <francisco.iglesias@amd.com> S: Maintained F: hw/misc/xlnx-cfi-if.c F: include/hw/misc/xlnx-cfi-if.h +F: hw/misc/xlnx-versal-cfu.c +F: include/hw/misc/xlnx-versal-cfu.h STM32F100 M: Alexandre Iooss <erdnaxe@crans.org> diff --git a/include/hw/misc/xlnx-versal-cfu.h b/include/hw/misc/xlnx-versal-cfu.h new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/include/hw/misc/xlnx-versal-cfu.h @@ -XXX,XX +XXX,XX @@ +/* + * QEMU model of the CFU Configuration Unit. + * + * Copyright (C) 2023, Advanced Micro Devices, Inc. + * + * Written by Francisco Iglesias <francisco.iglesias@amd.com> + * + * SPDX-License-Identifier: GPL-2.0-or-later + * + * References: + * [1] Versal ACAP Technical Reference Manual, + * https://www.xilinx.com/support/documentation/architecture-manuals/am011-versal-acap-trm.pdf + * + * [2] Versal ACAP Register Reference, + * https://www.xilinx.com/htmldocs/registers/am012/am012-versal-register-reference.html + */ +#ifndef HW_MISC_XLNX_VERSAL_CFU_APB_H +#define HW_MISC_XLNX_VERSAL_CFU_APB_H + +#include "hw/sysbus.h" +#include "hw/register.h" +#include "hw/misc/xlnx-cfi-if.h" + +#define TYPE_XLNX_VERSAL_CFU_APB "xlnx,versal-cfu-apb" +OBJECT_DECLARE_SIMPLE_TYPE(XlnxVersalCFUAPB, XLNX_VERSAL_CFU_APB) + +REG32(CFU_ISR, 0x0) + FIELD(CFU_ISR, USR_GTS_EVENT, 9, 1) + FIELD(CFU_ISR, USR_GSR_EVENT, 8, 1) + FIELD(CFU_ISR, SLVERR, 7, 1) + FIELD(CFU_ISR, DECOMP_ERROR, 6, 1) + FIELD(CFU_ISR, BAD_CFI_PACKET, 5, 1) + FIELD(CFU_ISR, AXI_ALIGN_ERROR, 4, 1) + FIELD(CFU_ISR, CFI_ROW_ERROR, 3, 1) + FIELD(CFU_ISR, CRC32_ERROR, 2, 1) + FIELD(CFU_ISR, CRC8_ERROR, 1, 1) + FIELD(CFU_ISR, SEU_ENDOFCALIB, 0, 1) +REG32(CFU_IMR, 0x4) + FIELD(CFU_IMR, USR_GTS_EVENT, 9, 1) + FIELD(CFU_IMR, USR_GSR_EVENT, 8, 1) + FIELD(CFU_IMR, SLVERR, 7, 1) + FIELD(CFU_IMR, DECOMP_ERROR, 6, 1) + FIELD(CFU_IMR, BAD_CFI_PACKET, 5, 1) + FIELD(CFU_IMR, AXI_ALIGN_ERROR, 4, 1) + FIELD(CFU_IMR, CFI_ROW_ERROR, 3, 1) + FIELD(CFU_IMR, CRC32_ERROR, 2, 1) + FIELD(CFU_IMR, CRC8_ERROR, 1, 1) + FIELD(CFU_IMR, SEU_ENDOFCALIB, 0, 1) +REG32(CFU_IER, 0x8) + FIELD(CFU_IER, USR_GTS_EVENT, 9, 1) + FIELD(CFU_IER, USR_GSR_EVENT, 8, 1) + FIELD(CFU_IER, SLVERR, 7, 1) + FIELD(CFU_IER, DECOMP_ERROR, 6, 1) + FIELD(CFU_IER, BAD_CFI_PACKET, 5, 1) + FIELD(CFU_IER, AXI_ALIGN_ERROR, 4, 1) + FIELD(CFU_IER, CFI_ROW_ERROR, 3, 1) + FIELD(CFU_IER, CRC32_ERROR, 2, 1) + FIELD(CFU_IER, CRC8_ERROR, 1, 1) + FIELD(CFU_IER, SEU_ENDOFCALIB, 0, 1) +REG32(CFU_IDR, 0xc) + FIELD(CFU_IDR, USR_GTS_EVENT, 9, 1) + FIELD(CFU_IDR, USR_GSR_EVENT, 8, 1) + FIELD(CFU_IDR, SLVERR, 7, 1) + FIELD(CFU_IDR, DECOMP_ERROR, 6, 1) + FIELD(CFU_IDR, BAD_CFI_PACKET, 5, 1) + FIELD(CFU_IDR, AXI_ALIGN_ERROR, 4, 1) + FIELD(CFU_IDR, CFI_ROW_ERROR, 3, 1) + FIELD(CFU_IDR, CRC32_ERROR, 2, 1) + FIELD(CFU_IDR, CRC8_ERROR, 1, 1) + FIELD(CFU_IDR, SEU_ENDOFCALIB, 0, 1) +REG32(CFU_ITR, 0x10) + FIELD(CFU_ITR, USR_GTS_EVENT, 9, 1) + FIELD(CFU_ITR, USR_GSR_EVENT, 8, 1) + FIELD(CFU_ITR, SLVERR, 7, 1) + FIELD(CFU_ITR, DECOMP_ERROR, 6, 1) + FIELD(CFU_ITR, BAD_CFI_PACKET, 5, 1) + FIELD(CFU_ITR, AXI_ALIGN_ERROR, 4, 1) + FIELD(CFU_ITR, CFI_ROW_ERROR, 3, 1) + FIELD(CFU_ITR, CRC32_ERROR, 2, 1) + FIELD(CFU_ITR, CRC8_ERROR, 1, 1) + FIELD(CFU_ITR, SEU_ENDOFCALIB, 0, 1) +REG32(CFU_PROTECT, 0x14) + FIELD(CFU_PROTECT, ACTIVE, 0, 1) +REG32(CFU_FGCR, 0x18) + FIELD(CFU_FGCR, GCLK_CAL, 14, 1) + FIELD(CFU_FGCR, SC_HBC_TRIGGER, 13, 1) + FIELD(CFU_FGCR, GLOW, 12, 1) + FIELD(CFU_FGCR, GPWRDWN, 11, 1) + FIELD(CFU_FGCR, GCAP, 10, 1) + FIELD(CFU_FGCR, GSCWE, 9, 1) + FIELD(CFU_FGCR, GHIGH_B, 8, 1) + FIELD(CFU_FGCR, GMC_B, 7, 1) + FIELD(CFU_FGCR, GWE, 6, 1) + FIELD(CFU_FGCR, GRESTORE, 5, 1) + FIELD(CFU_FGCR, GTS_CFG_B, 4, 1) + FIELD(CFU_FGCR, GLUTMASK, 3, 1) + FIELD(CFU_FGCR, EN_GLOBS_B, 2, 1) + FIELD(CFU_FGCR, EOS, 1, 1) + FIELD(CFU_FGCR, INIT_COMPLETE, 0, 1) +REG32(CFU_CTL, 0x1c) + FIELD(CFU_CTL, GSR_GSC, 15, 1) + FIELD(CFU_CTL, SLVERR_EN, 14, 1) + FIELD(CFU_CTL, CRC32_RESET, 13, 1) + FIELD(CFU_CTL, AXI_ERROR_EN, 12, 1) + FIELD(CFU_CTL, FLUSH_AXI, 11, 1) + FIELD(CFU_CTL, SSI_PER_SLR_PR, 10, 1) + FIELD(CFU_CTL, GCAP_CLK_EN, 9, 1) + FIELD(CFU_CTL, STATUS_SYNC_DISABLE, 8, 1) + FIELD(CFU_CTL, IGNORE_CFI_ERROR, 7, 1) + FIELD(CFU_CTL, CFRAME_DISABLE, 6, 1) + FIELD(CFU_CTL, QWORD_CNT_RESET, 5, 1) + FIELD(CFU_CTL, CRC8_DISABLE, 4, 1) + FIELD(CFU_CTL, CRC32_CHECK, 3, 1) + FIELD(CFU_CTL, DECOMPRESS, 2, 1) + FIELD(CFU_CTL, SEU_GO, 1, 1) + FIELD(CFU_CTL, CFI_LOCAL_RESET, 0, 1) +REG32(CFU_CRAM_RW, 0x20) + FIELD(CFU_CRAM_RW, RFIFO_AFULL_DEPTH, 18, 9) + FIELD(CFU_CRAM_RW, RD_WAVE_CNT_LEFT, 12, 6) + FIELD(CFU_CRAM_RW, RD_WAVE_CNT, 6, 6) + FIELD(CFU_CRAM_RW, WR_WAVE_CNT, 0, 6) +REG32(CFU_MASK, 0x28) +REG32(CFU_CRC_EXPECT, 0x2c) +REG32(CFU_CFRAME_LEFT_T0, 0x60) + FIELD(CFU_CFRAME_LEFT_T0, NUM, 0, 20) +REG32(CFU_CFRAME_LEFT_T1, 0x64) + FIELD(CFU_CFRAME_LEFT_T1, NUM, 0, 20) +REG32(CFU_CFRAME_LEFT_T2, 0x68) + FIELD(CFU_CFRAME_LEFT_T2, NUM, 0, 20) +REG32(CFU_ROW_RANGE, 0x6c) + FIELD(CFU_ROW_RANGE, HALF_FSR, 5, 1) + FIELD(CFU_ROW_RANGE, NUM, 0, 5) +REG32(CFU_STATUS, 0x100) + FIELD(CFU_STATUS, SEU_WRITE_ERROR, 30, 1) + FIELD(CFU_STATUS, FRCNT_ERROR, 29, 1) + FIELD(CFU_STATUS, RSVD_ERROR, 28, 1) + FIELD(CFU_STATUS, FDRO_ERROR, 27, 1) + FIELD(CFU_STATUS, FDRI_ERROR, 26, 1) + FIELD(CFU_STATUS, FDRI_READ_ERROR, 25, 1) + FIELD(CFU_STATUS, READ_FDRI_ERROR, 24, 1) + FIELD(CFU_STATUS, READ_SFR_ERROR, 23, 1) + FIELD(CFU_STATUS, READ_STREAM_ERROR, 22, 1) + FIELD(CFU_STATUS, UNKNOWN_STREAM_PKT, 21, 1) + FIELD(CFU_STATUS, USR_GTS, 20, 1) + FIELD(CFU_STATUS, USR_GSR, 19, 1) + FIELD(CFU_STATUS, AXI_BAD_WSTRB, 18, 1) + FIELD(CFU_STATUS, AXI_BAD_AR_SIZE, 17, 1) + FIELD(CFU_STATUS, AXI_BAD_AW_SIZE, 16, 1) + FIELD(CFU_STATUS, AXI_BAD_ARADDR, 15, 1) + FIELD(CFU_STATUS, AXI_BAD_AWADDR, 14, 1) + FIELD(CFU_STATUS, SCAN_CLEAR_PASS, 13, 1) + FIELD(CFU_STATUS, HC_SEC_ERROR, 12, 1) + FIELD(CFU_STATUS, GHIGH_B_ISHIGH, 11, 1) + FIELD(CFU_STATUS, GHIGH_B_ISLOW, 10, 1) + FIELD(CFU_STATUS, GMC_B_ISHIGH, 9, 1) + FIELD(CFU_STATUS, GMC_B_ISLOW, 8, 1) + FIELD(CFU_STATUS, GPWRDWN_B_ISHIGH, 7, 1) + FIELD(CFU_STATUS, CFI_SEU_CRC_ERROR, 6, 1) + FIELD(CFU_STATUS, CFI_SEU_ECC_ERROR, 5, 1) + FIELD(CFU_STATUS, CFI_SEU_HEARTBEAT, 4, 1) + FIELD(CFU_STATUS, SCAN_CLEAR_DONE, 3, 1) + FIELD(CFU_STATUS, HC_COMPLETE, 2, 1) + FIELD(CFU_STATUS, CFI_CFRAME_BUSY, 1, 1) + FIELD(CFU_STATUS, CFU_STREAM_BUSY, 0, 1) +REG32(CFU_INTERNAL_STATUS, 0x104) + FIELD(CFU_INTERNAL_STATUS, SSI_EOS, 22, 1) + FIELD(CFU_INTERNAL_STATUS, SSI_GWE, 21, 1) + FIELD(CFU_INTERNAL_STATUS, RFIFO_EMPTY, 20, 1) + FIELD(CFU_INTERNAL_STATUS, RFIFO_FULL, 19, 1) + FIELD(CFU_INTERNAL_STATUS, SEL_SFR, 18, 1) + FIELD(CFU_INTERNAL_STATUS, STREAM_CFRAME, 17, 1) + FIELD(CFU_INTERNAL_STATUS, FDRI_PHASE, 16, 1) + FIELD(CFU_INTERNAL_STATUS, CFI_PIPE_EN, 15, 1) + FIELD(CFU_INTERNAL_STATUS, AWFIFO_DCNT, 10, 5) + FIELD(CFU_INTERNAL_STATUS, WFIFO_DCNT, 5, 5) + FIELD(CFU_INTERNAL_STATUS, REPAIR_BUSY, 4, 1) + FIELD(CFU_INTERNAL_STATUS, TRIMU_BUSY, 3, 1) + FIELD(CFU_INTERNAL_STATUS, TRIMB_BUSY, 2, 1) + FIELD(CFU_INTERNAL_STATUS, HCLEANR_BUSY, 1, 1) + FIELD(CFU_INTERNAL_STATUS, HCLEAN_BUSY, 0, 1) +REG32(CFU_QWORD_CNT, 0x108) +REG32(CFU_CRC_LIVE, 0x10c) +REG32(CFU_PENDING_READ_CNT, 0x110) + FIELD(CFU_PENDING_READ_CNT, NUM, 0, 25) +REG32(CFU_FDRI_CNT, 0x114) +REG32(CFU_ECO1, 0x118) +REG32(CFU_ECO2, 0x11c) + +#define R_MAX (R_CFU_ECO2 + 1) + +#define NUM_STREAM 2 +#define WFIFO_SZ 4 + +struct XlnxVersalCFUAPB { + SysBusDevice parent_obj; + MemoryRegion iomem; + MemoryRegion iomem_stream[NUM_STREAM]; + qemu_irq irq_cfu_imr; + + /* 128-bit wfifo. */ + uint32_t wfifo[WFIFO_SZ]; + + uint32_t regs[R_MAX]; + RegisterInfo regs_info[R_MAX]; + + uint8_t fdri_row_addr; + + struct { + XlnxCfiIf *cframe[15]; + } cfg; +}; + +/** + * This is a helper function for updating a CFI data write fifo, an array of 4 + * uint32_t and 128 bits of data that are allowed to be written through 4 + * sequential 32 bit accesses. After the last index has been written into the + * write fifo (wfifo), the data is copied to and returned in a secondary fifo + * provided to the function (wfifo_ret), and the write fifo is cleared + * (zeroized). + * + * @addr: the address used when calculating the wfifo array index to update + * @value: the value to write into the wfifo array + * @wfifo: the wfifo to update + * @wfifo_out: will return the wfifo data when all 128 bits have been written + * + * @return: true if all 128 bits have been updated. + */ +bool update_wfifo(hwaddr addr, uint64_t value, + uint32_t *wfifo, uint32_t *wfifo_ret); + +#endif diff --git a/hw/misc/xlnx-versal-cfu.c b/hw/misc/xlnx-versal-cfu.c new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/hw/misc/xlnx-versal-cfu.c @@ -XXX,XX +XXX,XX @@ +/* + * QEMU model of the CFU Configuration Unit. + * + * Copyright (C) 2023, Advanced Micro Devices, Inc. + * + * Written by Edgar E. Iglesias <edgar.iglesias@gmail.com>, + * Sai Pavan Boddu <sai.pavan.boddu@amd.com>, + * Francisco Iglesias <francisco.iglesias@amd.com> + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#include "qemu/osdep.h" +#include "hw/sysbus.h" +#include "hw/register.h" +#include "hw/irq.h" +#include "qemu/bitops.h" +#include "qemu/log.h" +#include "qemu/units.h" +#include "migration/vmstate.h" +#include "hw/qdev-properties.h" +#include "hw/qdev-properties-system.h" +#include "hw/misc/xlnx-versal-cfu.h" + +#ifndef XLNX_VERSAL_CFU_APB_ERR_DEBUG +#define XLNX_VERSAL_CFU_APB_ERR_DEBUG 0 +#endif + +#define KEYHOLE_STREAM_4K (4 * KiB) +#define KEYHOLE_STREAM_256K (256 * KiB) +#define CFRAME_BROADCAST_ROW 0x1F + +bool update_wfifo(hwaddr addr, uint64_t value, + uint32_t *wfifo, uint32_t *wfifo_ret) +{ + unsigned int idx = extract32(addr, 2, 2); + + wfifo[idx] = value; + + if (idx == 3) { + memcpy(wfifo_ret, wfifo, WFIFO_SZ * sizeof(uint32_t)); + memset(wfifo, 0, WFIFO_SZ * sizeof(uint32_t)); + return true; + } + + return false; +} + +static void cfu_imr_update_irq(XlnxVersalCFUAPB *s) +{ + bool pending = s->regs[R_CFU_ISR] & ~s->regs[R_CFU_IMR]; + qemu_set_irq(s->irq_cfu_imr, pending); +} + +static void cfu_isr_postw(RegisterInfo *reg, uint64_t val64) +{ + XlnxVersalCFUAPB *s = XLNX_VERSAL_CFU_APB(reg->opaque); + cfu_imr_update_irq(s); +} + +static uint64_t cfu_ier_prew(RegisterInfo *reg, uint64_t val64) +{ + XlnxVersalCFUAPB *s = XLNX_VERSAL_CFU_APB(reg->opaque); + uint32_t val = val64; + + s->regs[R_CFU_IMR] &= ~val; + cfu_imr_update_irq(s); + return 0; +} + +static uint64_t cfu_idr_prew(RegisterInfo *reg, uint64_t val64) +{ + XlnxVersalCFUAPB *s = XLNX_VERSAL_CFU_APB(reg->opaque); + uint32_t val = val64; + + s->regs[R_CFU_IMR] |= val; + cfu_imr_update_irq(s); + return 0; +} + +static uint64_t cfu_itr_prew(RegisterInfo *reg, uint64_t val64) +{ + XlnxVersalCFUAPB *s = XLNX_VERSAL_CFU_APB(reg->opaque); + uint32_t val = val64; + + s->regs[R_CFU_ISR] |= val; + cfu_imr_update_irq(s); + return 0; +} + +static void cfu_fgcr_postw(RegisterInfo *reg, uint64_t val64) +{ + XlnxVersalCFUAPB *s = XLNX_VERSAL_CFU_APB(reg->opaque); + uint32_t val = (uint32_t)val64; + + /* Do a scan. It always looks good. */ + if (FIELD_EX32(val, CFU_FGCR, SC_HBC_TRIGGER)) { + ARRAY_FIELD_DP32(s->regs, CFU_STATUS, SCAN_CLEAR_PASS, 1); + ARRAY_FIELD_DP32(s->regs, CFU_STATUS, SCAN_CLEAR_DONE, 1); + } +} + +static const RegisterAccessInfo cfu_apb_regs_info[] = { + { .name = "CFU_ISR", .addr = A_CFU_ISR, + .rsvd = 0xfffffc00, + .w1c = 0x3ff, + .post_write = cfu_isr_postw, + },{ .name = "CFU_IMR", .addr = A_CFU_IMR, + .reset = 0x3ff, + .rsvd = 0xfffffc00, + .ro = 0x3ff, + },{ .name = "CFU_IER", .addr = A_CFU_IER, + .rsvd = 0xfffffc00, + .pre_write = cfu_ier_prew, + },{ .name = "CFU_IDR", .addr = A_CFU_IDR, + .rsvd = 0xfffffc00, + .pre_write = cfu_idr_prew, + },{ .name = "CFU_ITR", .addr = A_CFU_ITR, + .rsvd = 0xfffffc00, + .pre_write = cfu_itr_prew, + },{ .name = "CFU_PROTECT", .addr = A_CFU_PROTECT, + .reset = 0x1, + },{ .name = "CFU_FGCR", .addr = A_CFU_FGCR, + .rsvd = 0xffff8000, + .post_write = cfu_fgcr_postw, + },{ .name = "CFU_CTL", .addr = A_CFU_CTL, + .rsvd = 0xffff0000, + },{ .name = "CFU_CRAM_RW", .addr = A_CFU_CRAM_RW, + .reset = 0x401f7d9, + .rsvd = 0xf8000000, + },{ .name = "CFU_MASK", .addr = A_CFU_MASK, + },{ .name = "CFU_CRC_EXPECT", .addr = A_CFU_CRC_EXPECT, + },{ .name = "CFU_CFRAME_LEFT_T0", .addr = A_CFU_CFRAME_LEFT_T0, + .rsvd = 0xfff00000, + },{ .name = "CFU_CFRAME_LEFT_T1", .addr = A_CFU_CFRAME_LEFT_T1, + .rsvd = 0xfff00000, + },{ .name = "CFU_CFRAME_LEFT_T2", .addr = A_CFU_CFRAME_LEFT_T2, + .rsvd = 0xfff00000, + },{ .name = "CFU_ROW_RANGE", .addr = A_CFU_ROW_RANGE, + .rsvd = 0xffffffc0, + .ro = 0x3f, + },{ .name = "CFU_STATUS", .addr = A_CFU_STATUS, + .rsvd = 0x80000000, + .ro = 0x7fffffff, + },{ .name = "CFU_INTERNAL_STATUS", .addr = A_CFU_INTERNAL_STATUS, + .rsvd = 0xff800000, + .ro = 0x7fffff, + },{ .name = "CFU_QWORD_CNT", .addr = A_CFU_QWORD_CNT, + .ro = 0xffffffff, + },{ .name = "CFU_CRC_LIVE", .addr = A_CFU_CRC_LIVE, + .ro = 0xffffffff, + },{ .name = "CFU_PENDING_READ_CNT", .addr = A_CFU_PENDING_READ_CNT, + .rsvd = 0xfe000000, + .ro = 0x1ffffff, + },{ .name = "CFU_FDRI_CNT", .addr = A_CFU_FDRI_CNT, + .ro = 0xffffffff, + },{ .name = "CFU_ECO1", .addr = A_CFU_ECO1, + },{ .name = "CFU_ECO2", .addr = A_CFU_ECO2, + } +}; + +static void cfu_apb_reset(DeviceState *dev) +{ + XlnxVersalCFUAPB *s = XLNX_VERSAL_CFU_APB(dev); + unsigned int i; + + for (i = 0; i < ARRAY_SIZE(s->regs_info); ++i) { + register_reset(&s->regs_info[i]); + } + memset(s->wfifo, 0, WFIFO_SZ * sizeof(uint32_t)); + + s->regs[R_CFU_STATUS] |= R_CFU_STATUS_HC_COMPLETE_MASK; + cfu_imr_update_irq(s); +} + +static const MemoryRegionOps cfu_apb_ops = { + .read = register_read_memory, + .write = register_write_memory, + .endianness = DEVICE_LITTLE_ENDIAN, + .valid = { + .min_access_size = 4, + .max_access_size = 4, + }, +}; + +static void cfu_transfer_cfi_packet(XlnxVersalCFUAPB *s, uint8_t row_addr, + XlnxCfiPacket *pkt) +{ + if (row_addr == CFRAME_BROADCAST_ROW) { + for (int i = 0; i < ARRAY_SIZE(s->cfg.cframe); i++) { + if (s->cfg.cframe[i]) { + xlnx_cfi_transfer_packet(s->cfg.cframe[i], pkt); + } + } + } else { + assert(row_addr < ARRAY_SIZE(s->cfg.cframe)); + + if (s->cfg.cframe[row_addr]) { + xlnx_cfi_transfer_packet(s->cfg.cframe[row_addr], pkt); + } + } +} + +static uint64_t cfu_stream_read(void *opaque, hwaddr addr, unsigned size) +{ + qemu_log_mask(LOG_GUEST_ERROR, "%s: Unsupported read from addr=%" + HWADDR_PRIx "\n", __func__, addr); + return 0; +} + +static void cfu_stream_write(void *opaque, hwaddr addr, uint64_t value, + unsigned size) +{ + XlnxVersalCFUAPB *s = XLNX_VERSAL_CFU_APB(opaque); + uint32_t wfifo[WFIFO_SZ]; + + if (update_wfifo(addr, value, s->wfifo, wfifo)) { + uint8_t packet_type, row_addr, reg_addr; + + packet_type = extract32(wfifo[0], 24, 8); + row_addr = extract32(wfifo[0], 16, 5); + reg_addr = extract32(wfifo[0], 8, 6); + + /* Compressed bitstreams are not supported yet. */ + if (ARRAY_FIELD_EX32(s->regs, CFU_CTL, DECOMPRESS) == 0) { + if (s->regs[R_CFU_FDRI_CNT]) { + XlnxCfiPacket pkt = { + .reg_addr = CFRAME_FDRI, + .data[0] = wfifo[0], + .data[1] = wfifo[1], + .data[2] = wfifo[2], + .data[3] = wfifo[3] + }; + + cfu_transfer_cfi_packet(s, s->fdri_row_addr, &pkt); + + s->regs[R_CFU_FDRI_CNT]--; + + } else if (packet_type == PACKET_TYPE_CFU && + reg_addr == CFRAME_FDRI) { + + /* Load R_CFU_FDRI_CNT, must be multiple of 25 */ + s->regs[R_CFU_FDRI_CNT] = wfifo[1]; + + /* Store target row_addr */ + s->fdri_row_addr = row_addr; + + if (wfifo[1] % 25 != 0) { + qemu_log_mask(LOG_GUEST_ERROR, + "CFU FDRI_CNT is not loaded with " + "a multiple of 25 value\n"); + } + + } else if (packet_type == PACKET_TYPE_CFRAME) { + XlnxCfiPacket pkt = { + .reg_addr = reg_addr, + .data[0] = wfifo[1], + .data[1] = wfifo[2], + .data[2] = wfifo[3], + }; + cfu_transfer_cfi_packet(s, row_addr, &pkt); + } + } + } +} + +static const MemoryRegionOps cfu_stream_ops = { + .read = cfu_stream_read, + .write = cfu_stream_write, + .endianness = DEVICE_LITTLE_ENDIAN, + .valid = { + .min_access_size = 4, + .max_access_size = 8, + }, +}; + +static void cfu_apb_init(Object *obj) +{ + XlnxVersalCFUAPB *s = XLNX_VERSAL_CFU_APB(obj); + SysBusDevice *sbd = SYS_BUS_DEVICE(obj); + RegisterInfoArray *reg_array; + unsigned int i; + char *name; + + memory_region_init(&s->iomem, obj, TYPE_XLNX_VERSAL_CFU_APB, R_MAX * 4); + reg_array = + register_init_block32(DEVICE(obj), cfu_apb_regs_info, + ARRAY_SIZE(cfu_apb_regs_info), + s->regs_info, s->regs, + &cfu_apb_ops, + XLNX_VERSAL_CFU_APB_ERR_DEBUG, + R_MAX * 4); + memory_region_add_subregion(&s->iomem, + 0x0, + &reg_array->mem); + sysbus_init_mmio(sbd, &s->iomem); + for (i = 0; i < NUM_STREAM; i++) { + name = g_strdup_printf(TYPE_XLNX_VERSAL_CFU_APB "-stream%d", i); + memory_region_init_io(&s->iomem_stream[i], obj, &cfu_stream_ops, s, + name, i == 0 ? KEYHOLE_STREAM_4K : + KEYHOLE_STREAM_256K); + sysbus_init_mmio(sbd, &s->iomem_stream[i]); + g_free(name); + } + sysbus_init_irq(sbd, &s->irq_cfu_imr); +} + +static Property cfu_props[] = { + DEFINE_PROP_LINK("cframe0", XlnxVersalCFUAPB, cfg.cframe[0], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe1", XlnxVersalCFUAPB, cfg.cframe[1], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe2", XlnxVersalCFUAPB, cfg.cframe[2], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe3", XlnxVersalCFUAPB, cfg.cframe[3], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe4", XlnxVersalCFUAPB, cfg.cframe[4], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe5", XlnxVersalCFUAPB, cfg.cframe[5], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe6", XlnxVersalCFUAPB, cfg.cframe[6], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe7", XlnxVersalCFUAPB, cfg.cframe[7], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe8", XlnxVersalCFUAPB, cfg.cframe[8], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe9", XlnxVersalCFUAPB, cfg.cframe[9], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe10", XlnxVersalCFUAPB, cfg.cframe[10], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe11", XlnxVersalCFUAPB, cfg.cframe[11], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe12", XlnxVersalCFUAPB, cfg.cframe[12], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe13", XlnxVersalCFUAPB, cfg.cframe[13], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe14", XlnxVersalCFUAPB, cfg.cframe[14], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_END_OF_LIST(), +}; + +static const VMStateDescription vmstate_cfu_apb = { + .name = TYPE_XLNX_VERSAL_CFU_APB, + .version_id = 1, + .minimum_version_id = 1, + .fields = (VMStateField[]) { + VMSTATE_UINT32_ARRAY(wfifo, XlnxVersalCFUAPB, 4), + VMSTATE_UINT32_ARRAY(regs, XlnxVersalCFUAPB, R_MAX), + VMSTATE_UINT8(fdri_row_addr, XlnxVersalCFUAPB), + VMSTATE_END_OF_LIST(), + } +}; + +static void cfu_apb_class_init(ObjectClass *klass, void *data) +{ + DeviceClass *dc = DEVICE_CLASS(klass); + + dc->reset = cfu_apb_reset; + dc->vmsd = &vmstate_cfu_apb; + device_class_set_props(dc, cfu_props); +} + +static const TypeInfo cfu_apb_info = { + .name = TYPE_XLNX_VERSAL_CFU_APB, + .parent = TYPE_SYS_BUS_DEVICE, + .instance_size = sizeof(XlnxVersalCFUAPB), + .class_init = cfu_apb_class_init, + .instance_init = cfu_apb_init, + .interfaces = (InterfaceInfo[]) { + { TYPE_XLNX_CFI_IF }, + { } + } +}; + +static void cfu_apb_register_types(void) +{ + type_register_static(&cfu_apb_info); +} + +type_init(cfu_apb_register_types) diff --git a/hw/misc/meson.build b/hw/misc/meson.build index XXXXXXX..XXXXXXX 100644 --- a/hw/misc/meson.build +++ b/hw/misc/meson.build @@ -XXX,XX +XXX,XX @@ specific_ss.add(when: 'CONFIG_XLNX_VERSAL', if_true: files('xlnx-versal-crl.c')) system_ss.add(when: 'CONFIG_XLNX_VERSAL', if_true: files( 'xlnx-versal-xramc.c', 'xlnx-versal-pmc-iou-slcr.c', + 'xlnx-versal-cfu.c', 'xlnx-cfi-if.c', )) system_ss.add(when: 'CONFIG_STM32F2XX_SYSCFG', if_true: files('stm32f2xx_syscfg.c')) -- 2.34.1 From: Francisco Iglesias <francisco.iglesias@amd.com> Introduce a model of Xilinx Versal's Configuration Frame Unit's data out port (CFU_FDRO). Signed-off-by: Francisco Iglesias <francisco.iglesias@amd.com> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Message-id: 20230831165701.2016397-4-francisco.iglesias@amd.com Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- include/hw/misc/xlnx-versal-cfu.h | 12 ++++ hw/misc/xlnx-versal-cfu.c | 96 +++++++++++++++++++++++++++++++ 2 files changed, 108 insertions(+) diff --git a/include/hw/misc/xlnx-versal-cfu.h b/include/hw/misc/xlnx-versal-cfu.h index XXXXXXX..XXXXXXX 100644 --- a/include/hw/misc/xlnx-versal-cfu.h +++ b/include/hw/misc/xlnx-versal-cfu.h @@ -XXX,XX +XXX,XX @@ #include "hw/sysbus.h" #include "hw/register.h" #include "hw/misc/xlnx-cfi-if.h" +#include "qemu/fifo32.h" #define TYPE_XLNX_VERSAL_CFU_APB "xlnx,versal-cfu-apb" OBJECT_DECLARE_SIMPLE_TYPE(XlnxVersalCFUAPB, XLNX_VERSAL_CFU_APB) +#define TYPE_XLNX_VERSAL_CFU_FDRO "xlnx,versal-cfu-fdro" +OBJECT_DECLARE_SIMPLE_TYPE(XlnxVersalCFUFDRO, XLNX_VERSAL_CFU_FDRO) + REG32(CFU_ISR, 0x0) FIELD(CFU_ISR, USR_GTS_EVENT, 9, 1) FIELD(CFU_ISR, USR_GSR_EVENT, 8, 1) @@ -XXX,XX +XXX,XX @@ struct XlnxVersalCFUAPB { } cfg; }; + +struct XlnxVersalCFUFDRO { + SysBusDevice parent_obj; + MemoryRegion iomem_fdro; + + Fifo32 fdro_data; +}; + /** * This is a helper function for updating a CFI data write fifo, an array of 4 * uint32_t and 128 bits of data that are allowed to be written through 4 diff --git a/hw/misc/xlnx-versal-cfu.c b/hw/misc/xlnx-versal-cfu.c index XXXXXXX..XXXXXXX 100644 --- a/hw/misc/xlnx-versal-cfu.c +++ b/hw/misc/xlnx-versal-cfu.c @@ -XXX,XX +XXX,XX @@ static void cfu_stream_write(void *opaque, hwaddr addr, uint64_t value, } } +static uint64_t cfu_fdro_read(void *opaque, hwaddr addr, unsigned size) +{ + XlnxVersalCFUFDRO *s = XLNX_VERSAL_CFU_FDRO(opaque); + uint64_t ret = 0; + + if (!fifo32_is_empty(&s->fdro_data)) { + ret = fifo32_pop(&s->fdro_data); + } + + return ret; +} + +static void cfu_fdro_write(void *opaque, hwaddr addr, uint64_t value, + unsigned size) +{ + qemu_log_mask(LOG_GUEST_ERROR, "%s: Unsupported write from addr=%" + HWADDR_PRIx "\n", __func__, addr); +} + static const MemoryRegionOps cfu_stream_ops = { .read = cfu_stream_read, .write = cfu_stream_write, @@ -XXX,XX +XXX,XX @@ static const MemoryRegionOps cfu_stream_ops = { }, }; +static const MemoryRegionOps cfu_fdro_ops = { + .read = cfu_fdro_read, + .write = cfu_fdro_write, + .endianness = DEVICE_LITTLE_ENDIAN, + .valid = { + .min_access_size = 4, + .max_access_size = 4, + }, +}; + static void cfu_apb_init(Object *obj) { XlnxVersalCFUAPB *s = XLNX_VERSAL_CFU_APB(obj); @@ -XXX,XX +XXX,XX @@ static void cfu_apb_init(Object *obj) sysbus_init_irq(sbd, &s->irq_cfu_imr); } +static void cfu_fdro_init(Object *obj) +{ + XlnxVersalCFUFDRO *s = XLNX_VERSAL_CFU_FDRO(obj); + SysBusDevice *sbd = SYS_BUS_DEVICE(obj); + + memory_region_init_io(&s->iomem_fdro, obj, &cfu_fdro_ops, s, + TYPE_XLNX_VERSAL_CFU_FDRO, KEYHOLE_STREAM_4K); + sysbus_init_mmio(sbd, &s->iomem_fdro); + fifo32_create(&s->fdro_data, 8 * KiB / sizeof(uint32_t)); +} + +static void cfu_fdro_reset_enter(Object *obj, ResetType type) +{ + XlnxVersalCFUFDRO *s = XLNX_VERSAL_CFU_FDRO(obj); + + fifo32_reset(&s->fdro_data); +} + +static void cfu_fdro_cfi_transfer_packet(XlnxCfiIf *cfi_if, XlnxCfiPacket *pkt) +{ + XlnxVersalCFUFDRO *s = XLNX_VERSAL_CFU_FDRO(cfi_if); + + if (fifo32_num_free(&s->fdro_data) >= ARRAY_SIZE(pkt->data)) { + for (int i = 0; i < ARRAY_SIZE(pkt->data); i++) { + fifo32_push(&s->fdro_data, pkt->data[i]); + } + } else { + /* It is a programming error to fill the fifo. */ + qemu_log_mask(LOG_GUEST_ERROR, + "CFU_FDRO: CFI data dropped due to full read fifo\n"); + } +} + static Property cfu_props[] = { DEFINE_PROP_LINK("cframe0", XlnxVersalCFUAPB, cfg.cframe[0], TYPE_XLNX_CFI_IF, XlnxCfiIf *), @@ -XXX,XX +XXX,XX @@ static const VMStateDescription vmstate_cfu_apb = { } }; +static const VMStateDescription vmstate_cfu_fdro = { + .name = TYPE_XLNX_VERSAL_CFU_FDRO, + .version_id = 1, + .minimum_version_id = 1, + .fields = (VMStateField[]) { + VMSTATE_FIFO32(fdro_data, XlnxVersalCFUFDRO), + VMSTATE_END_OF_LIST(), + } +}; + static void cfu_apb_class_init(ObjectClass *klass, void *data) { DeviceClass *dc = DEVICE_CLASS(klass); @@ -XXX,XX +XXX,XX @@ static void cfu_apb_class_init(ObjectClass *klass, void *data) device_class_set_props(dc, cfu_props); } +static void cfu_fdro_class_init(ObjectClass *klass, void *data) +{ + DeviceClass *dc = DEVICE_CLASS(klass); + ResettableClass *rc = RESETTABLE_CLASS(klass); + XlnxCfiIfClass *xcic = XLNX_CFI_IF_CLASS(klass); + + dc->vmsd = &vmstate_cfu_fdro; + xcic->cfi_transfer_packet = cfu_fdro_cfi_transfer_packet; + rc->phases.enter = cfu_fdro_reset_enter; +} + static const TypeInfo cfu_apb_info = { .name = TYPE_XLNX_VERSAL_CFU_APB, .parent = TYPE_SYS_BUS_DEVICE, @@ -XXX,XX +XXX,XX @@ static const TypeInfo cfu_apb_info = { } }; +static const TypeInfo cfu_fdro_info = { + .name = TYPE_XLNX_VERSAL_CFU_FDRO, + .parent = TYPE_SYS_BUS_DEVICE, + .instance_size = sizeof(XlnxVersalCFUFDRO), + .class_init = cfu_fdro_class_init, + .instance_init = cfu_fdro_init, + .interfaces = (InterfaceInfo[]) { + { TYPE_XLNX_CFI_IF }, + { } + } +}; + static void cfu_apb_register_types(void) { type_register_static(&cfu_apb_info); + type_register_static(&cfu_fdro_info); } type_init(cfu_apb_register_types) -- 2.34.1 From: Francisco Iglesias <francisco.iglesias@amd.com> Introduce a model of Xilinx Versal's Configuration Frame Unit's Single Frame Read port (CFU_SFR). Signed-off-by: Francisco Iglesias <francisco.iglesias@amd.com> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Message-id: 20230831165701.2016397-5-francisco.iglesias@amd.com Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- include/hw/misc/xlnx-versal-cfu.h | 15 ++++++ hw/misc/xlnx-versal-cfu.c | 87 +++++++++++++++++++++++++++++++ 2 files changed, 102 insertions(+) diff --git a/include/hw/misc/xlnx-versal-cfu.h b/include/hw/misc/xlnx-versal-cfu.h index XXXXXXX..XXXXXXX 100644 --- a/include/hw/misc/xlnx-versal-cfu.h +++ b/include/hw/misc/xlnx-versal-cfu.h @@ -XXX,XX +XXX,XX @@ OBJECT_DECLARE_SIMPLE_TYPE(XlnxVersalCFUAPB, XLNX_VERSAL_CFU_APB) #define TYPE_XLNX_VERSAL_CFU_FDRO "xlnx,versal-cfu-fdro" OBJECT_DECLARE_SIMPLE_TYPE(XlnxVersalCFUFDRO, XLNX_VERSAL_CFU_FDRO) +#define TYPE_XLNX_VERSAL_CFU_SFR "xlnx,versal-cfu-sfr" +OBJECT_DECLARE_SIMPLE_TYPE(XlnxVersalCFUSFR, XLNX_VERSAL_CFU_SFR) + REG32(CFU_ISR, 0x0) FIELD(CFU_ISR, USR_GTS_EVENT, 9, 1) FIELD(CFU_ISR, USR_GSR_EVENT, 8, 1) @@ -XXX,XX +XXX,XX @@ struct XlnxVersalCFUFDRO { Fifo32 fdro_data; }; +struct XlnxVersalCFUSFR { + SysBusDevice parent_obj; + MemoryRegion iomem_sfr; + + /* 128-bit wfifo. */ + uint32_t wfifo[WFIFO_SZ]; + + struct { + XlnxVersalCFUAPB *cfu; + } cfg; +}; + /** * This is a helper function for updating a CFI data write fifo, an array of 4 * uint32_t and 128 bits of data that are allowed to be written through 4 diff --git a/hw/misc/xlnx-versal-cfu.c b/hw/misc/xlnx-versal-cfu.c index XXXXXXX..XXXXXXX 100644 --- a/hw/misc/xlnx-versal-cfu.c +++ b/hw/misc/xlnx-versal-cfu.c @@ -XXX,XX +XXX,XX @@ static void cfu_stream_write(void *opaque, hwaddr addr, uint64_t value, } } +static uint64_t cfu_sfr_read(void *opaque, hwaddr addr, unsigned size) +{ + qemu_log_mask(LOG_GUEST_ERROR, "%s: Unsupported read from addr=%" + HWADDR_PRIx "\n", __func__, addr); + return 0; +} + +static void cfu_sfr_write(void *opaque, hwaddr addr, uint64_t value, + unsigned size) +{ + XlnxVersalCFUSFR *s = XLNX_VERSAL_CFU_SFR(opaque); + uint32_t wfifo[WFIFO_SZ]; + + if (update_wfifo(addr, value, s->wfifo, wfifo)) { + uint8_t row_addr = extract32(wfifo[0], 23, 5); + uint32_t frame_addr = extract32(wfifo[0], 0, 23); + XlnxCfiPacket pkt = { .reg_addr = CFRAME_SFR, + .data[0] = frame_addr }; + + if (s->cfg.cfu) { + cfu_transfer_cfi_packet(s->cfg.cfu, row_addr, &pkt); + } + } +} + static uint64_t cfu_fdro_read(void *opaque, hwaddr addr, unsigned size) { XlnxVersalCFUFDRO *s = XLNX_VERSAL_CFU_FDRO(opaque); @@ -XXX,XX +XXX,XX @@ static const MemoryRegionOps cfu_stream_ops = { }, }; +static const MemoryRegionOps cfu_sfr_ops = { + .read = cfu_sfr_read, + .write = cfu_sfr_write, + .endianness = DEVICE_LITTLE_ENDIAN, + .valid = { + .min_access_size = 4, + .max_access_size = 4, + }, +}; + static const MemoryRegionOps cfu_fdro_ops = { .read = cfu_fdro_read, .write = cfu_fdro_write, @@ -XXX,XX +XXX,XX @@ static void cfu_apb_init(Object *obj) sysbus_init_irq(sbd, &s->irq_cfu_imr); } +static void cfu_sfr_init(Object *obj) +{ + XlnxVersalCFUSFR *s = XLNX_VERSAL_CFU_SFR(obj); + SysBusDevice *sbd = SYS_BUS_DEVICE(obj); + + memory_region_init_io(&s->iomem_sfr, obj, &cfu_sfr_ops, s, + TYPE_XLNX_VERSAL_CFU_SFR, KEYHOLE_STREAM_4K); + sysbus_init_mmio(sbd, &s->iomem_sfr); +} + +static void cfu_sfr_reset_enter(Object *obj, ResetType type) +{ + XlnxVersalCFUSFR *s = XLNX_VERSAL_CFU_SFR(obj); + + memset(s->wfifo, 0, WFIFO_SZ * sizeof(uint32_t)); +} + static void cfu_fdro_init(Object *obj) { XlnxVersalCFUFDRO *s = XLNX_VERSAL_CFU_FDRO(obj); @@ -XXX,XX +XXX,XX @@ static Property cfu_props[] = { DEFINE_PROP_END_OF_LIST(), }; +static Property cfu_sfr_props[] = { + DEFINE_PROP_LINK("cfu", XlnxVersalCFUSFR, cfg.cfu, + TYPE_XLNX_VERSAL_CFU_APB, XlnxVersalCFUAPB *), + DEFINE_PROP_END_OF_LIST(), +}; + static const VMStateDescription vmstate_cfu_apb = { .name = TYPE_XLNX_VERSAL_CFU_APB, .version_id = 1, @@ -XXX,XX +XXX,XX @@ static const VMStateDescription vmstate_cfu_fdro = { } }; +static const VMStateDescription vmstate_cfu_sfr = { + .name = TYPE_XLNX_VERSAL_CFU_SFR, + .version_id = 1, + .minimum_version_id = 1, + .fields = (VMStateField[]) { + VMSTATE_UINT32_ARRAY(wfifo, XlnxVersalCFUSFR, 4), + VMSTATE_END_OF_LIST(), + } +}; + static void cfu_apb_class_init(ObjectClass *klass, void *data) { DeviceClass *dc = DEVICE_CLASS(klass); @@ -XXX,XX +XXX,XX @@ static void cfu_fdro_class_init(ObjectClass *klass, void *data) rc->phases.enter = cfu_fdro_reset_enter; } +static void cfu_sfr_class_init(ObjectClass *klass, void *data) +{ + DeviceClass *dc = DEVICE_CLASS(klass); + ResettableClass *rc = RESETTABLE_CLASS(klass); + + device_class_set_props(dc, cfu_sfr_props); + dc->vmsd = &vmstate_cfu_sfr; + rc->phases.enter = cfu_sfr_reset_enter; +} + static const TypeInfo cfu_apb_info = { .name = TYPE_XLNX_VERSAL_CFU_APB, .parent = TYPE_SYS_BUS_DEVICE, @@ -XXX,XX +XXX,XX @@ static const TypeInfo cfu_fdro_info = { } }; +static const TypeInfo cfu_sfr_info = { + .name = TYPE_XLNX_VERSAL_CFU_SFR, + .parent = TYPE_SYS_BUS_DEVICE, + .instance_size = sizeof(XlnxVersalCFUSFR), + .class_init = cfu_sfr_class_init, + .instance_init = cfu_sfr_init, +}; + static void cfu_apb_register_types(void) { type_register_static(&cfu_apb_info); type_register_static(&cfu_fdro_info); + type_register_static(&cfu_sfr_info); } type_init(cfu_apb_register_types) -- 2.34.1 From: Francisco Iglesias <francisco.iglesias@amd.com> Introduce a model of Xilinx Versal's Configuration Frame controller (CFRAME_REG). Signed-off-by: Francisco Iglesias <francisco.iglesias@amd.com> Message-id: 20230831165701.2016397-6-francisco.iglesias@amd.com Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- MAINTAINERS | 2 + include/hw/misc/xlnx-versal-cframe-reg.h | 286 ++++++++++ hw/misc/xlnx-versal-cframe-reg.c | 697 +++++++++++++++++++++++ hw/misc/meson.build | 1 + 4 files changed, 986 insertions(+) create mode 100644 include/hw/misc/xlnx-versal-cframe-reg.h create mode 100644 hw/misc/xlnx-versal-cframe-reg.c diff --git a/MAINTAINERS b/MAINTAINERS index XXXXXXX..XXXXXXX 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -XXX,XX +XXX,XX @@ F: hw/misc/xlnx-cfi-if.c F: include/hw/misc/xlnx-cfi-if.h F: hw/misc/xlnx-versal-cfu.c F: include/hw/misc/xlnx-versal-cfu.h +F: hw/misc/xlnx-versal-cframe-reg.c +F: include/hw/misc/xlnx-versal-cframe-reg.h STM32F100 M: Alexandre Iooss <erdnaxe@crans.org> diff --git a/include/hw/misc/xlnx-versal-cframe-reg.h b/include/hw/misc/xlnx-versal-cframe-reg.h new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/include/hw/misc/xlnx-versal-cframe-reg.h @@ -XXX,XX +XXX,XX @@ +/* + * QEMU model of the Configuration Frame Control module + * + * Copyright (C) 2023, Advanced Micro Devices, Inc. + * + * Written by Francisco Iglesias <francisco.iglesias@amd.com> + * + * SPDX-License-Identifier: GPL-2.0-or-later + * + * References: + * [1] Versal ACAP Technical Reference Manual, + * https://www.xilinx.com/support/documentation/architecture-manuals/am011-versal-acap-trm.pdf + * + * [2] Versal ACAP Register Reference, + * https://www.xilinx.com/htmldocs/registers/am012/am012-versal-register-reference.html + */ +#ifndef HW_MISC_XLNX_VERSAL_CFRAME_REG_H +#define HW_MISC_XLNX_VERSAL_CFRAME_REG_H + +#include "hw/sysbus.h" +#include "hw/register.h" +#include "hw/misc/xlnx-cfi-if.h" +#include "hw/misc/xlnx-versal-cfu.h" +#include "qemu/fifo32.h" + +#define TYPE_XLNX_VERSAL_CFRAME_REG "xlnx,cframe-reg" +OBJECT_DECLARE_SIMPLE_TYPE(XlnxVersalCFrameReg, XLNX_VERSAL_CFRAME_REG) + +/* + * The registers in this module are 128 bits wide but it is ok to write + * and read them through 4 sequential 32 bit accesses (address[3:2] = 0, + * 1, 2, 3). + */ +REG32(CRC0, 0x0) + FIELD(CRC, CRC, 0, 32) +REG32(CRC1, 0x4) +REG32(CRC2, 0x8) +REG32(CRC3, 0xc) +REG32(FAR0, 0x10) + FIELD(FAR0, SEGMENT, 23, 2) + FIELD(FAR0, BLOCKTYPE, 20, 3) + FIELD(FAR0, FRAME_ADDR, 0, 20) +REG32(FAR1, 0x14) +REG32(FAR2, 0x18) +REG32(FAR3, 0x1c) +REG32(FAR_SFR0, 0x20) + FIELD(FAR_SFR0, BLOCKTYPE, 20, 3) + FIELD(FAR_SFR0, FRAME_ADDR, 0, 20) +REG32(FAR_SFR1, 0x24) +REG32(FAR_SFR2, 0x28) +REG32(FAR_SFR3, 0x2c) +REG32(FDRI0, 0x40) +REG32(FDRI1, 0x44) +REG32(FDRI2, 0x48) +REG32(FDRI3, 0x4c) +REG32(FRCNT0, 0x50) + FIELD(FRCNT0, FRCNT, 0, 32) +REG32(FRCNT1, 0x54) +REG32(FRCNT2, 0x58) +REG32(FRCNT3, 0x5c) +REG32(CMD0, 0x60) + FIELD(CMD0, CMD, 0, 5) +REG32(CMD1, 0x64) +REG32(CMD2, 0x68) +REG32(CMD3, 0x6c) +REG32(CR_MASK0, 0x70) +REG32(CR_MASK1, 0x74) +REG32(CR_MASK2, 0x78) +REG32(CR_MASK3, 0x7c) +REG32(CTL0, 0x80) + FIELD(CTL, PER_FRAME_CRC, 0, 1) +REG32(CTL1, 0x84) +REG32(CTL2, 0x88) +REG32(CTL3, 0x8c) +REG32(CFRM_ISR0, 0x150) + FIELD(CFRM_ISR0, READ_BROADCAST_ERROR, 21, 1) + FIELD(CFRM_ISR0, CMD_MISSING_ERROR, 20, 1) + FIELD(CFRM_ISR0, RW_ROWOFF_ERROR, 19, 1) + FIELD(CFRM_ISR0, READ_REG_ADDR_ERROR, 18, 1) + FIELD(CFRM_ISR0, READ_BLK_TYPE_ERROR, 17, 1) + FIELD(CFRM_ISR0, READ_FRAME_ADDR_ERROR, 16, 1) + FIELD(CFRM_ISR0, WRITE_REG_ADDR_ERROR, 15, 1) + FIELD(CFRM_ISR0, WRITE_BLK_TYPE_ERROR, 13, 1) + FIELD(CFRM_ISR0, WRITE_FRAME_ADDR_ERROR, 12, 1) + FIELD(CFRM_ISR0, MFW_OVERRUN_ERROR, 11, 1) + FIELD(CFRM_ISR0, FAR_FIFO_UNDERFLOW, 10, 1) + FIELD(CFRM_ISR0, FAR_FIFO_OVERFLOW, 9, 1) + FIELD(CFRM_ISR0, PER_FRAME_SEQ_ERROR, 8, 1) + FIELD(CFRM_ISR0, CRC_ERROR, 7, 1) + FIELD(CFRM_ISR0, WRITE_OVERRUN_ERROR, 6, 1) + FIELD(CFRM_ISR0, READ_OVERRUN_ERROR, 5, 1) + FIELD(CFRM_ISR0, CMD_INTERRUPT_ERROR, 4, 1) + FIELD(CFRM_ISR0, WRITE_INTERRUPT_ERROR, 3, 1) + FIELD(CFRM_ISR0, READ_INTERRUPT_ERROR, 2, 1) + FIELD(CFRM_ISR0, SEU_CRC_ERROR, 1, 1) + FIELD(CFRM_ISR0, SEU_ECC_ERROR, 0, 1) +REG32(CFRM_ISR1, 0x154) +REG32(CFRM_ISR2, 0x158) +REG32(CFRM_ISR3, 0x15c) +REG32(CFRM_IMR0, 0x160) + FIELD(CFRM_IMR0, READ_BROADCAST_ERROR, 21, 1) + FIELD(CFRM_IMR0, CMD_MISSING_ERROR, 20, 1) + FIELD(CFRM_IMR0, RW_ROWOFF_ERROR, 19, 1) + FIELD(CFRM_IMR0, READ_REG_ADDR_ERROR, 18, 1) + FIELD(CFRM_IMR0, READ_BLK_TYPE_ERROR, 17, 1) + FIELD(CFRM_IMR0, READ_FRAME_ADDR_ERROR, 16, 1) + FIELD(CFRM_IMR0, WRITE_REG_ADDR_ERROR, 15, 1) + FIELD(CFRM_IMR0, WRITE_BLK_TYPE_ERROR, 13, 1) + FIELD(CFRM_IMR0, WRITE_FRAME_ADDR_ERROR, 12, 1) + FIELD(CFRM_IMR0, MFW_OVERRUN_ERROR, 11, 1) + FIELD(CFRM_IMR0, FAR_FIFO_UNDERFLOW, 10, 1) + FIELD(CFRM_IMR0, FAR_FIFO_OVERFLOW, 9, 1) + FIELD(CFRM_IMR0, PER_FRAME_SEQ_ERROR, 8, 1) + FIELD(CFRM_IMR0, CRC_ERROR, 7, 1) + FIELD(CFRM_IMR0, WRITE_OVERRUN_ERROR, 6, 1) + FIELD(CFRM_IMR0, READ_OVERRUN_ERROR, 5, 1) + FIELD(CFRM_IMR0, CMD_INTERRUPT_ERROR, 4, 1) + FIELD(CFRM_IMR0, WRITE_INTERRUPT_ERROR, 3, 1) + FIELD(CFRM_IMR0, READ_INTERRUPT_ERROR, 2, 1) + FIELD(CFRM_IMR0, SEU_CRC_ERROR, 1, 1) + FIELD(CFRM_IMR0, SEU_ECC_ERROR, 0, 1) +REG32(CFRM_IMR1, 0x164) +REG32(CFRM_IMR2, 0x168) +REG32(CFRM_IMR3, 0x16c) +REG32(CFRM_IER0, 0x170) + FIELD(CFRM_IER0, READ_BROADCAST_ERROR, 21, 1) + FIELD(CFRM_IER0, CMD_MISSING_ERROR, 20, 1) + FIELD(CFRM_IER0, RW_ROWOFF_ERROR, 19, 1) + FIELD(CFRM_IER0, READ_REG_ADDR_ERROR, 18, 1) + FIELD(CFRM_IER0, READ_BLK_TYPE_ERROR, 17, 1) + FIELD(CFRM_IER0, READ_FRAME_ADDR_ERROR, 16, 1) + FIELD(CFRM_IER0, WRITE_REG_ADDR_ERROR, 15, 1) + FIELD(CFRM_IER0, WRITE_BLK_TYPE_ERROR, 13, 1) + FIELD(CFRM_IER0, WRITE_FRAME_ADDR_ERROR, 12, 1) + FIELD(CFRM_IER0, MFW_OVERRUN_ERROR, 11, 1) + FIELD(CFRM_IER0, FAR_FIFO_UNDERFLOW, 10, 1) + FIELD(CFRM_IER0, FAR_FIFO_OVERFLOW, 9, 1) + FIELD(CFRM_IER0, PER_FRAME_SEQ_ERROR, 8, 1) + FIELD(CFRM_IER0, CRC_ERROR, 7, 1) + FIELD(CFRM_IER0, WRITE_OVERRUN_ERROR, 6, 1) + FIELD(CFRM_IER0, READ_OVERRUN_ERROR, 5, 1) + FIELD(CFRM_IER0, CMD_INTERRUPT_ERROR, 4, 1) + FIELD(CFRM_IER0, WRITE_INTERRUPT_ERROR, 3, 1) + FIELD(CFRM_IER0, READ_INTERRUPT_ERROR, 2, 1) + FIELD(CFRM_IER0, SEU_CRC_ERROR, 1, 1) + FIELD(CFRM_IER0, SEU_ECC_ERROR, 0, 1) +REG32(CFRM_IER1, 0x174) +REG32(CFRM_IER2, 0x178) +REG32(CFRM_IER3, 0x17c) +REG32(CFRM_IDR0, 0x180) + FIELD(CFRM_IDR0, READ_BROADCAST_ERROR, 21, 1) + FIELD(CFRM_IDR0, CMD_MISSING_ERROR, 20, 1) + FIELD(CFRM_IDR0, RW_ROWOFF_ERROR, 19, 1) + FIELD(CFRM_IDR0, READ_REG_ADDR_ERROR, 18, 1) + FIELD(CFRM_IDR0, READ_BLK_TYPE_ERROR, 17, 1) + FIELD(CFRM_IDR0, READ_FRAME_ADDR_ERROR, 16, 1) + FIELD(CFRM_IDR0, WRITE_REG_ADDR_ERROR, 15, 1) + FIELD(CFRM_IDR0, WRITE_BLK_TYPE_ERROR, 13, 1) + FIELD(CFRM_IDR0, WRITE_FRAME_ADDR_ERROR, 12, 1) + FIELD(CFRM_IDR0, MFW_OVERRUN_ERROR, 11, 1) + FIELD(CFRM_IDR0, FAR_FIFO_UNDERFLOW, 10, 1) + FIELD(CFRM_IDR0, FAR_FIFO_OVERFLOW, 9, 1) + FIELD(CFRM_IDR0, PER_FRAME_SEQ_ERROR, 8, 1) + FIELD(CFRM_IDR0, CRC_ERROR, 7, 1) + FIELD(CFRM_IDR0, WRITE_OVERRUN_ERROR, 6, 1) + FIELD(CFRM_IDR0, READ_OVERRUN_ERROR, 5, 1) + FIELD(CFRM_IDR0, CMD_INTERRUPT_ERROR, 4, 1) + FIELD(CFRM_IDR0, WRITE_INTERRUPT_ERROR, 3, 1) + FIELD(CFRM_IDR0, READ_INTERRUPT_ERROR, 2, 1) + FIELD(CFRM_IDR0, SEU_CRC_ERROR, 1, 1) + FIELD(CFRM_IDR0, SEU_ECC_ERROR, 0, 1) +REG32(CFRM_IDR1, 0x184) +REG32(CFRM_IDR2, 0x188) +REG32(CFRM_IDR3, 0x18c) +REG32(CFRM_ITR0, 0x190) + FIELD(CFRM_ITR0, READ_BROADCAST_ERROR, 21, 1) + FIELD(CFRM_ITR0, CMD_MISSING_ERROR, 20, 1) + FIELD(CFRM_ITR0, RW_ROWOFF_ERROR, 19, 1) + FIELD(CFRM_ITR0, READ_REG_ADDR_ERROR, 18, 1) + FIELD(CFRM_ITR0, READ_BLK_TYPE_ERROR, 17, 1) + FIELD(CFRM_ITR0, READ_FRAME_ADDR_ERROR, 16, 1) + FIELD(CFRM_ITR0, WRITE_REG_ADDR_ERROR, 15, 1) + FIELD(CFRM_ITR0, WRITE_BLK_TYPE_ERROR, 13, 1) + FIELD(CFRM_ITR0, WRITE_FRAME_ADDR_ERROR, 12, 1) + FIELD(CFRM_ITR0, MFW_OVERRUN_ERROR, 11, 1) + FIELD(CFRM_ITR0, FAR_FIFO_UNDERFLOW, 10, 1) + FIELD(CFRM_ITR0, FAR_FIFO_OVERFLOW, 9, 1) + FIELD(CFRM_ITR0, PER_FRAME_SEQ_ERROR, 8, 1) + FIELD(CFRM_ITR0, CRC_ERROR, 7, 1) + FIELD(CFRM_ITR0, WRITE_OVERRUN_ERROR, 6, 1) + FIELD(CFRM_ITR0, READ_OVERRUN_ERROR, 5, 1) + FIELD(CFRM_ITR0, CMD_INTERRUPT_ERROR, 4, 1) + FIELD(CFRM_ITR0, WRITE_INTERRUPT_ERROR, 3, 1) + FIELD(CFRM_ITR0, READ_INTERRUPT_ERROR, 2, 1) + FIELD(CFRM_ITR0, SEU_CRC_ERROR, 1, 1) + FIELD(CFRM_ITR0, SEU_ECC_ERROR, 0, 1) +REG32(CFRM_ITR1, 0x194) +REG32(CFRM_ITR2, 0x198) +REG32(CFRM_ITR3, 0x19c) +REG32(SEU_SYNDRM00, 0x1a0) +REG32(SEU_SYNDRM01, 0x1a4) +REG32(SEU_SYNDRM02, 0x1a8) +REG32(SEU_SYNDRM03, 0x1ac) +REG32(SEU_SYNDRM10, 0x1b0) +REG32(SEU_SYNDRM11, 0x1b4) +REG32(SEU_SYNDRM12, 0x1b8) +REG32(SEU_SYNDRM13, 0x1bc) +REG32(SEU_SYNDRM20, 0x1c0) +REG32(SEU_SYNDRM21, 0x1c4) +REG32(SEU_SYNDRM22, 0x1c8) +REG32(SEU_SYNDRM23, 0x1cc) +REG32(SEU_SYNDRM30, 0x1d0) +REG32(SEU_SYNDRM31, 0x1d4) +REG32(SEU_SYNDRM32, 0x1d8) +REG32(SEU_SYNDRM33, 0x1dc) +REG32(SEU_VIRTUAL_SYNDRM0, 0x1e0) +REG32(SEU_VIRTUAL_SYNDRM1, 0x1e4) +REG32(SEU_VIRTUAL_SYNDRM2, 0x1e8) +REG32(SEU_VIRTUAL_SYNDRM3, 0x1ec) +REG32(SEU_CRC0, 0x1f0) +REG32(SEU_CRC1, 0x1f4) +REG32(SEU_CRC2, 0x1f8) +REG32(SEU_CRC3, 0x1fc) +REG32(CFRAME_FAR_BOT0, 0x200) +REG32(CFRAME_FAR_BOT1, 0x204) +REG32(CFRAME_FAR_BOT2, 0x208) +REG32(CFRAME_FAR_BOT3, 0x20c) +REG32(CFRAME_FAR_TOP0, 0x210) +REG32(CFRAME_FAR_TOP1, 0x214) +REG32(CFRAME_FAR_TOP2, 0x218) +REG32(CFRAME_FAR_TOP3, 0x21c) +REG32(LAST_FRAME_BOT0, 0x220) + FIELD(LAST_FRAME_BOT0, BLOCKTYPE1_LAST_FRAME_LSB, 20, 12) + FIELD(LAST_FRAME_BOT0, BLOCKTYPE0_LAST_FRAME, 0, 20) +REG32(LAST_FRAME_BOT1, 0x224) + FIELD(LAST_FRAME_BOT1, BLOCKTYPE3_LAST_FRAME_LSB, 28, 4) + FIELD(LAST_FRAME_BOT1, BLOCKTYPE2_LAST_FRAME, 8, 20) + FIELD(LAST_FRAME_BOT1, BLOCKTYPE1_LAST_FRAME_MSB, 0, 8) +REG32(LAST_FRAME_BOT2, 0x228) + FIELD(LAST_FRAME_BOT2, BLOCKTYPE3_LAST_FRAME_MSB, 0, 16) +REG32(LAST_FRAME_BOT3, 0x22c) +REG32(LAST_FRAME_TOP0, 0x230) + FIELD(LAST_FRAME_TOP0, BLOCKTYPE5_LAST_FRAME_LSB, 20, 12) + FIELD(LAST_FRAME_TOP0, BLOCKTYPE4_LAST_FRAME, 0, 20) +REG32(LAST_FRAME_TOP1, 0x234) + FIELD(LAST_FRAME_TOP1, BLOCKTYPE6_LAST_FRAME, 8, 20) + FIELD(LAST_FRAME_TOP1, BLOCKTYPE5_LAST_FRAME_MSB, 0, 8) +REG32(LAST_FRAME_TOP2, 0x238) +REG32(LAST_FRAME_TOP3, 0x23c) + +#define CFRAME_REG_R_MAX (R_LAST_FRAME_TOP3 + 1) + +#define FRAME_NUM_QWORDS 25 +#define FRAME_NUM_WORDS (FRAME_NUM_QWORDS * 4) /* 25 * 128 bits */ + +typedef struct XlnxCFrame { + uint32_t data[FRAME_NUM_WORDS]; +} XlnxCFrame; + +struct XlnxVersalCFrameReg { + SysBusDevice parent_obj; + MemoryRegion iomem; + MemoryRegion iomem_fdri; + qemu_irq irq_cfrm_imr; + + /* 128-bit wfifo. */ + uint32_t wfifo[WFIFO_SZ]; + + uint32_t regs[CFRAME_REG_R_MAX]; + RegisterInfo regs_info[CFRAME_REG_R_MAX]; + + bool rowon; + bool wcfg; + bool rcfg; + + GTree *cframes; + Fifo32 new_f_data; + + struct { + XlnxCfiIf *cfu_fdro; + uint32_t blktype_num_frames[7]; + } cfg; + bool row_configured; +}; + +#endif diff --git a/hw/misc/xlnx-versal-cframe-reg.c b/hw/misc/xlnx-versal-cframe-reg.c new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/hw/misc/xlnx-versal-cframe-reg.c @@ -XXX,XX +XXX,XX @@ +/* + * QEMU model of the Configuration Frame Control module + * + * Copyright (C) 2023, Advanced Micro Devices, Inc. + * + * Written by Francisco Iglesias <francisco.iglesias@amd.com> + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#include "qemu/osdep.h" +#include "hw/sysbus.h" +#include "hw/register.h" +#include "hw/registerfields.h" +#include "qemu/bitops.h" +#include "qemu/log.h" +#include "qemu/units.h" +#include "qapi/error.h" +#include "hw/qdev-properties.h" +#include "migration/vmstate.h" +#include "hw/irq.h" +#include "hw/misc/xlnx-versal-cframe-reg.h" + +#ifndef XLNX_VERSAL_CFRAME_REG_ERR_DEBUG +#define XLNX_VERSAL_CFRAME_REG_ERR_DEBUG 0 +#endif + +#define KEYHOLE_STREAM_4K (4 * KiB) +#define N_WORDS_128BIT 4 + +#define MAX_BLOCKTYPE 6 +#define MAX_BLOCKTYPE_FRAMES 0xFFFFF + +enum { + CFRAME_CMD_WCFG = 1, + CFRAME_CMD_ROWON = 2, + CFRAME_CMD_ROWOFF = 3, + CFRAME_CMD_RCFG = 4, + CFRAME_CMD_DLPARK = 5, +}; + +static gint int_cmp(gconstpointer a, gconstpointer b, gpointer user_data) +{ + guint ua = GPOINTER_TO_UINT(a); + guint ub = GPOINTER_TO_UINT(b); + return (ua > ub) - (ua < ub); +} + +static void cfrm_imr_update_irq(XlnxVersalCFrameReg *s) +{ + bool pending = s->regs[R_CFRM_ISR0] & ~s->regs[R_CFRM_IMR0]; + qemu_set_irq(s->irq_cfrm_imr, pending); +} + +static void cfrm_isr_postw(RegisterInfo *reg, uint64_t val64) +{ + XlnxVersalCFrameReg *s = XLNX_VERSAL_CFRAME_REG(reg->opaque); + cfrm_imr_update_irq(s); +} + +static uint64_t cfrm_ier_prew(RegisterInfo *reg, uint64_t val64) +{ + XlnxVersalCFrameReg *s = XLNX_VERSAL_CFRAME_REG(reg->opaque); + + s->regs[R_CFRM_IMR0] &= ~s->regs[R_CFRM_IER0]; + s->regs[R_CFRM_IER0] = 0; + cfrm_imr_update_irq(s); + return 0; +} + +static uint64_t cfrm_idr_prew(RegisterInfo *reg, uint64_t val64) +{ + XlnxVersalCFrameReg *s = XLNX_VERSAL_CFRAME_REG(reg->opaque); + + s->regs[R_CFRM_IMR0] |= s->regs[R_CFRM_IDR0]; + s->regs[R_CFRM_IDR0] = 0; + cfrm_imr_update_irq(s); + return 0; +} + +static uint64_t cfrm_itr_prew(RegisterInfo *reg, uint64_t val64) +{ + XlnxVersalCFrameReg *s = XLNX_VERSAL_CFRAME_REG(reg->opaque); + + s->regs[R_CFRM_ISR0] |= s->regs[R_CFRM_ITR0]; + s->regs[R_CFRM_ITR0] = 0; + cfrm_imr_update_irq(s); + return 0; +} + +static void cframe_incr_far(XlnxVersalCFrameReg *s) +{ + uint32_t faddr = ARRAY_FIELD_EX32(s->regs, FAR0, FRAME_ADDR); + uint32_t blktype = ARRAY_FIELD_EX32(s->regs, FAR0, BLOCKTYPE); + + assert(blktype <= MAX_BLOCKTYPE); + + faddr++; + if (faddr > s->cfg.blktype_num_frames[blktype]) { + /* Restart from 0 and increment block type */ + faddr = 0; + blktype++; + + assert(blktype <= MAX_BLOCKTYPE); + + ARRAY_FIELD_DP32(s->regs, FAR0, BLOCKTYPE, blktype); + } + + ARRAY_FIELD_DP32(s->regs, FAR0, FRAME_ADDR, faddr); +} + +static void cfrm_fdri_post_write(RegisterInfo *reg, uint64_t val) +{ + XlnxVersalCFrameReg *s = XLNX_VERSAL_CFRAME_REG(reg->opaque); + + if (s->row_configured && s->rowon && s->wcfg) { + + if (fifo32_num_free(&s->new_f_data) >= N_WORDS_128BIT) { + fifo32_push(&s->new_f_data, s->regs[R_FDRI0]); + fifo32_push(&s->new_f_data, s->regs[R_FDRI1]); + fifo32_push(&s->new_f_data, s->regs[R_FDRI2]); + fifo32_push(&s->new_f_data, s->regs[R_FDRI3]); + } + + if (fifo32_is_full(&s->new_f_data)) { + uint32_t addr = extract32(s->regs[R_FAR0], 0, 23); + XlnxCFrame *f = g_new(XlnxCFrame, 1); + + for (int i = 0; i < FRAME_NUM_WORDS; i++) { + f->data[i] = fifo32_pop(&s->new_f_data); + } + + g_tree_replace(s->cframes, GUINT_TO_POINTER(addr), f); + + cframe_incr_far(s); + + fifo32_reset(&s->new_f_data); + } + } +} + +static void cfrm_readout_frames(XlnxVersalCFrameReg *s, uint32_t start_addr, + uint32_t end_addr) +{ + /* + * NB: when our minimum glib version is at least 2.68 we can improve the + * performance of the cframe traversal by using g_tree_lookup_node and + * g_tree_node_next (instead of calling g_tree_lookup for finding each + * cframe). + */ + for (uint32_t addr = start_addr; addr < end_addr; addr++) { + XlnxCFrame *f = g_tree_lookup(s->cframes, GUINT_TO_POINTER(addr)); + + /* Transmit the data if a frame was found */ + if (f) { + for (int i = 0; i < FRAME_NUM_WORDS; i += 4) { + XlnxCfiPacket pkt = {}; + + pkt.data[0] = f->data[i]; + pkt.data[1] = f->data[i + 1]; + pkt.data[2] = f->data[i + 2]; + pkt.data[3] = f->data[i + 3]; + + if (s->cfg.cfu_fdro) { + xlnx_cfi_transfer_packet(s->cfg.cfu_fdro, &pkt); + } + } + } + } +} + +static void cfrm_frcnt_post_write(RegisterInfo *reg, uint64_t val) +{ + XlnxVersalCFrameReg *s = XLNX_VERSAL_CFRAME_REG(reg->opaque); + + if (s->row_configured && s->rowon && s->rcfg) { + uint32_t start_addr = extract32(s->regs[R_FAR0], 0, 23); + uint32_t end_addr = start_addr + s->regs[R_FRCNT0] / FRAME_NUM_QWORDS; + + cfrm_readout_frames(s, start_addr, end_addr); + } +} + +static void cfrm_cmd_post_write(RegisterInfo *reg, uint64_t val) +{ + XlnxVersalCFrameReg *s = XLNX_VERSAL_CFRAME_REG(reg->opaque); + + if (s->row_configured) { + uint8_t cmd = ARRAY_FIELD_EX32(s->regs, CMD0, CMD); + + switch (cmd) { + case CFRAME_CMD_WCFG: + s->wcfg = true; + break; + case CFRAME_CMD_ROWON: + s->rowon = true; + break; + case CFRAME_CMD_ROWOFF: + s->rowon = false; + break; + case CFRAME_CMD_RCFG: + s->rcfg = true; + break; + case CFRAME_CMD_DLPARK: + s->wcfg = false; + s->rcfg = false; + break; + default: + break; + }; + } +} + +static uint64_t cfrm_last_frame_bot_post_read(RegisterInfo *reg, + uint64_t val64) +{ + XlnxVersalCFrameReg *s = XLNX_VERSAL_CFRAME_REG(reg->opaque); + uint64_t val = 0; + + switch (reg->access->addr) { + case A_LAST_FRAME_BOT0: + val = FIELD_DP32(val, LAST_FRAME_BOT0, BLOCKTYPE1_LAST_FRAME_LSB, + s->cfg.blktype_num_frames[1]); + val = FIELD_DP32(val, LAST_FRAME_BOT0, BLOCKTYPE0_LAST_FRAME, + s->cfg.blktype_num_frames[0]); + break; + case A_LAST_FRAME_BOT1: + val = FIELD_DP32(val, LAST_FRAME_BOT1, BLOCKTYPE3_LAST_FRAME_LSB, + s->cfg.blktype_num_frames[3]); + val = FIELD_DP32(val, LAST_FRAME_BOT1, BLOCKTYPE2_LAST_FRAME, + s->cfg.blktype_num_frames[2]); + val = FIELD_DP32(val, LAST_FRAME_BOT1, BLOCKTYPE1_LAST_FRAME_MSB, + (s->cfg.blktype_num_frames[1] >> 12)); + break; + case A_LAST_FRAME_BOT2: + val = FIELD_DP32(val, LAST_FRAME_BOT2, BLOCKTYPE3_LAST_FRAME_MSB, + (s->cfg.blktype_num_frames[3] >> 4)); + break; + case A_LAST_FRAME_BOT3: + default: + break; + } + + return val; +} + +static uint64_t cfrm_last_frame_top_post_read(RegisterInfo *reg, + uint64_t val64) +{ + XlnxVersalCFrameReg *s = XLNX_VERSAL_CFRAME_REG(reg->opaque); + uint64_t val = 0; + + switch (reg->access->addr) { + case A_LAST_FRAME_TOP0: + val = FIELD_DP32(val, LAST_FRAME_TOP0, BLOCKTYPE5_LAST_FRAME_LSB, + s->cfg.blktype_num_frames[5]); + val = FIELD_DP32(val, LAST_FRAME_TOP0, BLOCKTYPE4_LAST_FRAME, + s->cfg.blktype_num_frames[4]); + break; + case A_LAST_FRAME_TOP1: + val = FIELD_DP32(val, LAST_FRAME_TOP1, BLOCKTYPE6_LAST_FRAME, + s->cfg.blktype_num_frames[6]); + val = FIELD_DP32(val, LAST_FRAME_TOP1, BLOCKTYPE5_LAST_FRAME_MSB, + (s->cfg.blktype_num_frames[5] >> 12)); + break; + case A_LAST_FRAME_TOP2: + case A_LAST_FRAME_BOT3: + default: + break; + } + + return val; +} + +static void cfrm_far_sfr_post_write(RegisterInfo *reg, uint64_t val) +{ + XlnxVersalCFrameReg *s = XLNX_VERSAL_CFRAME_REG(reg->opaque); + + if (s->row_configured && s->rowon && s->rcfg) { + uint32_t start_addr = extract32(s->regs[R_FAR_SFR0], 0, 23); + + /* Readback 1 frame */ + cfrm_readout_frames(s, start_addr, start_addr + 1); + } +} + +static const RegisterAccessInfo cframe_reg_regs_info[] = { + { .name = "CRC0", .addr = A_CRC0, + .rsvd = 0x00000000, + },{ .name = "CRC1", .addr = A_CRC0, + .rsvd = 0xffffffff, + },{ .name = "CRC2", .addr = A_CRC0, + .rsvd = 0xffffffff, + },{ .name = "CRC3", .addr = A_CRC0, + .rsvd = 0xffffffff, + },{ .name = "FAR0", .addr = A_FAR0, + .rsvd = 0xfe000000, + },{ .name = "FAR1", .addr = A_FAR1, + .rsvd = 0xffffffff, + },{ .name = "FAR2", .addr = A_FAR2, + .rsvd = 0xffffffff, + },{ .name = "FAR3", .addr = A_FAR3, + .rsvd = 0xffffffff, + },{ .name = "FAR_SFR0", .addr = A_FAR_SFR0, + .rsvd = 0xff800000, + },{ .name = "FAR_SFR1", .addr = A_FAR_SFR1, + .rsvd = 0xffffffff, + },{ .name = "FAR_SFR2", .addr = A_FAR_SFR2, + .rsvd = 0xffffffff, + },{ .name = "FAR_SFR3", .addr = A_FAR_SFR3, + .rsvd = 0xffffffff, + .post_write = cfrm_far_sfr_post_write, + },{ .name = "FDRI0", .addr = A_FDRI0, + },{ .name = "FDRI1", .addr = A_FDRI1, + },{ .name = "FDRI2", .addr = A_FDRI2, + },{ .name = "FDRI3", .addr = A_FDRI3, + .post_write = cfrm_fdri_post_write, + },{ .name = "FRCNT0", .addr = A_FRCNT0, + .rsvd = 0x00000000, + },{ .name = "FRCNT1", .addr = A_FRCNT1, + .rsvd = 0xffffffff, + },{ .name = "FRCNT2", .addr = A_FRCNT2, + .rsvd = 0xffffffff, + },{ .name = "FRCNT3", .addr = A_FRCNT3, + .rsvd = 0xffffffff, + .post_write = cfrm_frcnt_post_write + },{ .name = "CMD0", .addr = A_CMD0, + .rsvd = 0xffffffe0, + },{ .name = "CMD1", .addr = A_CMD1, + .rsvd = 0xffffffff, + },{ .name = "CMD2", .addr = A_CMD2, + .rsvd = 0xffffffff, + },{ .name = "CMD3", .addr = A_CMD3, + .rsvd = 0xffffffff, + .post_write = cfrm_cmd_post_write + },{ .name = "CR_MASK0", .addr = A_CR_MASK0, + .rsvd = 0x00000000, + },{ .name = "CR_MASK1", .addr = A_CR_MASK1, + .rsvd = 0x00000000, + },{ .name = "CR_MASK2", .addr = A_CR_MASK2, + .rsvd = 0x00000000, + },{ .name = "CR_MASK3", .addr = A_CR_MASK3, + .rsvd = 0xffffffff, + },{ .name = "CTL0", .addr = A_CTL0, + .rsvd = 0xfffffff8, + },{ .name = "CTL1", .addr = A_CTL1, + .rsvd = 0xffffffff, + },{ .name = "CTL2", .addr = A_CTL2, + .rsvd = 0xffffffff, + },{ .name = "CTL3", .addr = A_CTL3, + .rsvd = 0xffffffff, + },{ .name = "CFRM_ISR0", .addr = A_CFRM_ISR0, + .rsvd = 0xffc04000, + .w1c = 0x3bfff, + },{ .name = "CFRM_ISR1", .addr = A_CFRM_ISR1, + .rsvd = 0xffffffff, + },{ .name = "CFRM_ISR2", .addr = A_CFRM_ISR2, + .rsvd = 0xffffffff, + },{ .name = "CFRM_ISR3", .addr = A_CFRM_ISR3, + .rsvd = 0xffffffff, + .post_write = cfrm_isr_postw, + },{ .name = "CFRM_IMR0", .addr = A_CFRM_IMR0, + .rsvd = 0xffc04000, + .ro = 0xfffff, + .reset = 0x3bfff, + },{ .name = "CFRM_IMR1", .addr = A_CFRM_IMR1, + .rsvd = 0xffffffff, + },{ .name = "CFRM_IMR2", .addr = A_CFRM_IMR2, + .rsvd = 0xffffffff, + },{ .name = "CFRM_IMR3", .addr = A_CFRM_IMR3, + .rsvd = 0xffffffff, + },{ .name = "CFRM_IER0", .addr = A_CFRM_IER0, + .rsvd = 0xffc04000, + },{ .name = "CFRM_IER1", .addr = A_CFRM_IER1, + .rsvd = 0xffffffff, + },{ .name = "CFRM_IER2", .addr = A_CFRM_IER2, + .rsvd = 0xffffffff, + },{ .name = "CFRM_IER3", .addr = A_CFRM_IER3, + .rsvd = 0xffffffff, + .pre_write = cfrm_ier_prew, + },{ .name = "CFRM_IDR0", .addr = A_CFRM_IDR0, + .rsvd = 0xffc04000, + },{ .name = "CFRM_IDR1", .addr = A_CFRM_IDR1, + .rsvd = 0xffffffff, + },{ .name = "CFRM_IDR2", .addr = A_CFRM_IDR2, + .rsvd = 0xffffffff, + },{ .name = "CFRM_IDR3", .addr = A_CFRM_IDR3, + .rsvd = 0xffffffff, + .pre_write = cfrm_idr_prew, + },{ .name = "CFRM_ITR0", .addr = A_CFRM_ITR0, + .rsvd = 0xffc04000, + },{ .name = "CFRM_ITR1", .addr = A_CFRM_ITR1, + .rsvd = 0xffffffff, + },{ .name = "CFRM_ITR2", .addr = A_CFRM_ITR2, + .rsvd = 0xffffffff, + },{ .name = "CFRM_ITR3", .addr = A_CFRM_ITR3, + .rsvd = 0xffffffff, + .pre_write = cfrm_itr_prew, + },{ .name = "SEU_SYNDRM00", .addr = A_SEU_SYNDRM00, + },{ .name = "SEU_SYNDRM01", .addr = A_SEU_SYNDRM01, + },{ .name = "SEU_SYNDRM02", .addr = A_SEU_SYNDRM02, + },{ .name = "SEU_SYNDRM03", .addr = A_SEU_SYNDRM03, + },{ .name = "SEU_SYNDRM10", .addr = A_SEU_SYNDRM10, + },{ .name = "SEU_SYNDRM11", .addr = A_SEU_SYNDRM11, + },{ .name = "SEU_SYNDRM12", .addr = A_SEU_SYNDRM12, + },{ .name = "SEU_SYNDRM13", .addr = A_SEU_SYNDRM13, + },{ .name = "SEU_SYNDRM20", .addr = A_SEU_SYNDRM20, + },{ .name = "SEU_SYNDRM21", .addr = A_SEU_SYNDRM21, + },{ .name = "SEU_SYNDRM22", .addr = A_SEU_SYNDRM22, + },{ .name = "SEU_SYNDRM23", .addr = A_SEU_SYNDRM23, + },{ .name = "SEU_SYNDRM30", .addr = A_SEU_SYNDRM30, + },{ .name = "SEU_SYNDRM31", .addr = A_SEU_SYNDRM31, + },{ .name = "SEU_SYNDRM32", .addr = A_SEU_SYNDRM32, + },{ .name = "SEU_SYNDRM33", .addr = A_SEU_SYNDRM33, + },{ .name = "SEU_VIRTUAL_SYNDRM0", .addr = A_SEU_VIRTUAL_SYNDRM0, + },{ .name = "SEU_VIRTUAL_SYNDRM1", .addr = A_SEU_VIRTUAL_SYNDRM1, + },{ .name = "SEU_VIRTUAL_SYNDRM2", .addr = A_SEU_VIRTUAL_SYNDRM2, + },{ .name = "SEU_VIRTUAL_SYNDRM3", .addr = A_SEU_VIRTUAL_SYNDRM3, + },{ .name = "SEU_CRC0", .addr = A_SEU_CRC0, + },{ .name = "SEU_CRC1", .addr = A_SEU_CRC1, + },{ .name = "SEU_CRC2", .addr = A_SEU_CRC2, + },{ .name = "SEU_CRC3", .addr = A_SEU_CRC3, + },{ .name = "CFRAME_FAR_BOT0", .addr = A_CFRAME_FAR_BOT0, + },{ .name = "CFRAME_FAR_BOT1", .addr = A_CFRAME_FAR_BOT1, + },{ .name = "CFRAME_FAR_BOT2", .addr = A_CFRAME_FAR_BOT2, + },{ .name = "CFRAME_FAR_BOT3", .addr = A_CFRAME_FAR_BOT3, + },{ .name = "CFRAME_FAR_TOP0", .addr = A_CFRAME_FAR_TOP0, + },{ .name = "CFRAME_FAR_TOP1", .addr = A_CFRAME_FAR_TOP1, + },{ .name = "CFRAME_FAR_TOP2", .addr = A_CFRAME_FAR_TOP2, + },{ .name = "CFRAME_FAR_TOP3", .addr = A_CFRAME_FAR_TOP3, + },{ .name = "LAST_FRAME_BOT0", .addr = A_LAST_FRAME_BOT0, + .ro = 0xffffffff, + .post_read = cfrm_last_frame_bot_post_read, + },{ .name = "LAST_FRAME_BOT1", .addr = A_LAST_FRAME_BOT1, + .ro = 0xffffffff, + .post_read = cfrm_last_frame_bot_post_read, + },{ .name = "LAST_FRAME_BOT2", .addr = A_LAST_FRAME_BOT2, + .ro = 0xffffffff, + .post_read = cfrm_last_frame_bot_post_read, + },{ .name = "LAST_FRAME_BOT3", .addr = A_LAST_FRAME_BOT3, + .ro = 0xffffffff, + .post_read = cfrm_last_frame_bot_post_read, + },{ .name = "LAST_FRAME_TOP0", .addr = A_LAST_FRAME_TOP0, + .ro = 0xffffffff, + .post_read = cfrm_last_frame_top_post_read, + },{ .name = "LAST_FRAME_TOP1", .addr = A_LAST_FRAME_TOP1, + .ro = 0xffffffff, + .post_read = cfrm_last_frame_top_post_read, + },{ .name = "LAST_FRAME_TOP2", .addr = A_LAST_FRAME_TOP2, + .ro = 0xffffffff, + .post_read = cfrm_last_frame_top_post_read, + },{ .name = "LAST_FRAME_TOP3", .addr = A_LAST_FRAME_TOP3, + .ro = 0xffffffff, + .post_read = cfrm_last_frame_top_post_read, + } +}; + +static void cframe_reg_cfi_transfer_packet(XlnxCfiIf *cfi_if, + XlnxCfiPacket *pkt) +{ + XlnxVersalCFrameReg *s = XLNX_VERSAL_CFRAME_REG(cfi_if); + uint64_t we = MAKE_64BIT_MASK(0, 4 * 8); + + if (!s->row_configured) { + return; + } + + switch (pkt->reg_addr) { + case CFRAME_FAR: + s->regs[R_FAR0] = pkt->data[0]; + break; + case CFRAME_SFR: + s->regs[R_FAR_SFR0] = pkt->data[0]; + register_write(&s->regs_info[R_FAR_SFR3], 0, + we, object_get_typename(OBJECT(s)), + XLNX_VERSAL_CFRAME_REG_ERR_DEBUG); + break; + case CFRAME_FDRI: + s->regs[R_FDRI0] = pkt->data[0]; + s->regs[R_FDRI1] = pkt->data[1]; + s->regs[R_FDRI2] = pkt->data[2]; + register_write(&s->regs_info[R_FDRI3], pkt->data[3], + we, object_get_typename(OBJECT(s)), + XLNX_VERSAL_CFRAME_REG_ERR_DEBUG); + break; + case CFRAME_CMD: + ARRAY_FIELD_DP32(s->regs, CMD0, CMD, pkt->data[0]); + + register_write(&s->regs_info[R_CMD3], 0, + we, object_get_typename(OBJECT(s)), + XLNX_VERSAL_CFRAME_REG_ERR_DEBUG); + break; + default: + break; + } +} + +static uint64_t cframe_reg_fdri_read(void *opaque, hwaddr addr, unsigned size) +{ + qemu_log_mask(LOG_GUEST_ERROR, "%s: Unsupported read from addr=%" + HWADDR_PRIx "\n", __func__, addr); + return 0; +} + +static void cframe_reg_fdri_write(void *opaque, hwaddr addr, uint64_t value, + unsigned size) +{ + XlnxVersalCFrameReg *s = XLNX_VERSAL_CFRAME_REG(opaque); + uint32_t wfifo[WFIFO_SZ]; + + if (update_wfifo(addr, value, s->wfifo, wfifo)) { + uint64_t we = MAKE_64BIT_MASK(0, 4 * 8); + + s->regs[R_FDRI0] = wfifo[0]; + s->regs[R_FDRI1] = wfifo[1]; + s->regs[R_FDRI2] = wfifo[2]; + register_write(&s->regs_info[R_FDRI3], wfifo[3], + we, object_get_typename(OBJECT(s)), + XLNX_VERSAL_CFRAME_REG_ERR_DEBUG); + } +} + +static void cframe_reg_reset_enter(Object *obj, ResetType type) +{ + XlnxVersalCFrameReg *s = XLNX_VERSAL_CFRAME_REG(obj); + unsigned int i; + + for (i = 0; i < ARRAY_SIZE(s->regs_info); ++i) { + register_reset(&s->regs_info[i]); + } + memset(s->wfifo, 0, WFIFO_SZ * sizeof(uint32_t)); + fifo32_reset(&s->new_f_data); + + if (g_tree_nnodes(s->cframes)) { + /* + * Take a reference so when g_tree_destroy() unrefs it we keep the + * GTree and only destroy its contents. NB: when our minimum + * glib version is at least 2.70 we could use g_tree_remove_all(). + */ + g_tree_ref(s->cframes); + g_tree_destroy(s->cframes); + } +} + +static void cframe_reg_reset_hold(Object *obj) +{ + XlnxVersalCFrameReg *s = XLNX_VERSAL_CFRAME_REG(obj); + + cfrm_imr_update_irq(s); +} + +static const MemoryRegionOps cframe_reg_ops = { + .read = register_read_memory, + .write = register_write_memory, + .endianness = DEVICE_LITTLE_ENDIAN, + .valid = { + .min_access_size = 4, + .max_access_size = 4, + }, +}; + +static const MemoryRegionOps cframe_reg_fdri_ops = { + .read = cframe_reg_fdri_read, + .write = cframe_reg_fdri_write, + .endianness = DEVICE_LITTLE_ENDIAN, + .valid = { + .min_access_size = 4, + .max_access_size = 4, + }, +}; + +static void cframe_reg_realize(DeviceState *dev, Error **errp) +{ + XlnxVersalCFrameReg *s = XLNX_VERSAL_CFRAME_REG(dev); + + for (int i = 0; i < ARRAY_SIZE(s->cfg.blktype_num_frames); i++) { + if (s->cfg.blktype_num_frames[i] > MAX_BLOCKTYPE_FRAMES) { + error_setg(errp, + "blktype-frames%d > 0xFFFFF (max frame per block)", + i); + return; + } + if (s->cfg.blktype_num_frames[i]) { + s->row_configured = true; + } + } +} + +static void cframe_reg_init(Object *obj) +{ + XlnxVersalCFrameReg *s = XLNX_VERSAL_CFRAME_REG(obj); + SysBusDevice *sbd = SYS_BUS_DEVICE(obj); + RegisterInfoArray *reg_array; + + memory_region_init(&s->iomem, obj, TYPE_XLNX_VERSAL_CFRAME_REG, + CFRAME_REG_R_MAX * 4); + reg_array = + register_init_block32(DEVICE(obj), cframe_reg_regs_info, + ARRAY_SIZE(cframe_reg_regs_info), + s->regs_info, s->regs, + &cframe_reg_ops, + XLNX_VERSAL_CFRAME_REG_ERR_DEBUG, + CFRAME_REG_R_MAX * 4); + memory_region_add_subregion(&s->iomem, + 0x0, + &reg_array->mem); + sysbus_init_mmio(sbd, &s->iomem); + memory_region_init_io(&s->iomem_fdri, obj, &cframe_reg_fdri_ops, s, + TYPE_XLNX_VERSAL_CFRAME_REG "-fdri", + KEYHOLE_STREAM_4K); + sysbus_init_mmio(sbd, &s->iomem_fdri); + sysbus_init_irq(sbd, &s->irq_cfrm_imr); + + s->cframes = g_tree_new_full((GCompareDataFunc)int_cmp, NULL, + NULL, (GDestroyNotify)g_free); + fifo32_create(&s->new_f_data, FRAME_NUM_WORDS); +} + +static const VMStateDescription vmstate_cframe = { + .name = "cframe", + .version_id = 1, + .minimum_version_id = 1, + .fields = (VMStateField[]) { + VMSTATE_UINT32_ARRAY(data, XlnxCFrame, FRAME_NUM_WORDS), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription vmstate_cframe_reg = { + .name = TYPE_XLNX_VERSAL_CFRAME_REG, + .version_id = 1, + .minimum_version_id = 1, + .fields = (VMStateField[]) { + VMSTATE_UINT32_ARRAY(wfifo, XlnxVersalCFrameReg, 4), + VMSTATE_UINT32_ARRAY(regs, XlnxVersalCFrameReg, CFRAME_REG_R_MAX), + VMSTATE_BOOL(rowon, XlnxVersalCFrameReg), + VMSTATE_BOOL(wcfg, XlnxVersalCFrameReg), + VMSTATE_BOOL(rcfg, XlnxVersalCFrameReg), + VMSTATE_GTREE_DIRECT_KEY_V(cframes, XlnxVersalCFrameReg, 1, + &vmstate_cframe, XlnxCFrame), + VMSTATE_FIFO32(new_f_data, XlnxVersalCFrameReg), + VMSTATE_END_OF_LIST(), + } +}; + +static Property cframe_regs_props[] = { + DEFINE_PROP_LINK("cfu-fdro", XlnxVersalCFrameReg, cfg.cfu_fdro, + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_UINT32("blktype0-frames", XlnxVersalCFrameReg, + cfg.blktype_num_frames[0], 0), + DEFINE_PROP_UINT32("blktype1-frames", XlnxVersalCFrameReg, + cfg.blktype_num_frames[1], 0), + DEFINE_PROP_UINT32("blktype2-frames", XlnxVersalCFrameReg, + cfg.blktype_num_frames[2], 0), + DEFINE_PROP_UINT32("blktype3-frames", XlnxVersalCFrameReg, + cfg.blktype_num_frames[3], 0), + DEFINE_PROP_UINT32("blktype4-frames", XlnxVersalCFrameReg, + cfg.blktype_num_frames[4], 0), + DEFINE_PROP_UINT32("blktype5-frames", XlnxVersalCFrameReg, + cfg.blktype_num_frames[5], 0), + DEFINE_PROP_UINT32("blktype6-frames", XlnxVersalCFrameReg, + cfg.blktype_num_frames[6], 0), + DEFINE_PROP_END_OF_LIST(), +}; + +static void cframe_reg_class_init(ObjectClass *klass, void *data) +{ + ResettableClass *rc = RESETTABLE_CLASS(klass); + DeviceClass *dc = DEVICE_CLASS(klass); + XlnxCfiIfClass *xcic = XLNX_CFI_IF_CLASS(klass); + + dc->vmsd = &vmstate_cframe_reg; + dc->realize = cframe_reg_realize; + rc->phases.enter = cframe_reg_reset_enter; + rc->phases.hold = cframe_reg_reset_hold; + device_class_set_props(dc, cframe_regs_props); + xcic->cfi_transfer_packet = cframe_reg_cfi_transfer_packet; +} + +static const TypeInfo cframe_reg_info = { + .name = TYPE_XLNX_VERSAL_CFRAME_REG, + .parent = TYPE_SYS_BUS_DEVICE, + .instance_size = sizeof(XlnxVersalCFrameReg), + .class_init = cframe_reg_class_init, + .instance_init = cframe_reg_init, + .interfaces = (InterfaceInfo[]) { + { TYPE_XLNX_CFI_IF }, + { } + } +}; + +static void cframe_reg_register_types(void) +{ + type_register_static(&cframe_reg_info); +} + +type_init(cframe_reg_register_types) diff --git a/hw/misc/meson.build b/hw/misc/meson.build index XXXXXXX..XXXXXXX 100644 --- a/hw/misc/meson.build +++ b/hw/misc/meson.build @@ -XXX,XX +XXX,XX @@ system_ss.add(when: 'CONFIG_XLNX_VERSAL', if_true: files( 'xlnx-versal-pmc-iou-slcr.c', 'xlnx-versal-cfu.c', 'xlnx-cfi-if.c', + 'xlnx-versal-cframe-reg.c', )) system_ss.add(when: 'CONFIG_STM32F2XX_SYSCFG', if_true: files('stm32f2xx_syscfg.c')) system_ss.add(when: 'CONFIG_STM32F4XX_SYSCFG', if_true: files('stm32f4xx_syscfg.c')) -- 2.34.1 From: Francisco Iglesias <francisco.iglesias@amd.com> Introduce a model of Xilinx Versal's Configuration Frame broadcast controller (CFRAME_BCAST_REG). Signed-off-by: Francisco Iglesias <francisco.iglesias@amd.com> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Message-id: 20230831165701.2016397-7-francisco.iglesias@amd.com Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- include/hw/misc/xlnx-versal-cframe-reg.h | 17 +++ hw/misc/xlnx-versal-cframe-reg.c | 161 +++++++++++++++++++++++ 2 files changed, 178 insertions(+) diff --git a/include/hw/misc/xlnx-versal-cframe-reg.h b/include/hw/misc/xlnx-versal-cframe-reg.h index XXXXXXX..XXXXXXX 100644 --- a/include/hw/misc/xlnx-versal-cframe-reg.h +++ b/include/hw/misc/xlnx-versal-cframe-reg.h @@ -XXX,XX +XXX,XX @@ #define TYPE_XLNX_VERSAL_CFRAME_REG "xlnx,cframe-reg" OBJECT_DECLARE_SIMPLE_TYPE(XlnxVersalCFrameReg, XLNX_VERSAL_CFRAME_REG) +#define TYPE_XLNX_VERSAL_CFRAME_BCAST_REG "xlnx.cframe-bcast-reg" +OBJECT_DECLARE_SIMPLE_TYPE(XlnxVersalCFrameBcastReg, + XLNX_VERSAL_CFRAME_BCAST_REG) + /* * The registers in this module are 128 bits wide but it is ok to write * and read them through 4 sequential 32 bit accesses (address[3:2] = 0, @@ -XXX,XX +XXX,XX @@ struct XlnxVersalCFrameReg { bool row_configured; }; +struct XlnxVersalCFrameBcastReg { + SysBusDevice parent_obj; + MemoryRegion iomem_reg; + MemoryRegion iomem_fdri; + + /* 128-bit wfifo. */ + uint32_t wfifo[WFIFO_SZ]; + + struct { + XlnxCfiIf *cframe[15]; + } cfg; +}; + #endif diff --git a/hw/misc/xlnx-versal-cframe-reg.c b/hw/misc/xlnx-versal-cframe-reg.c index XXXXXXX..XXXXXXX 100644 --- a/hw/misc/xlnx-versal-cframe-reg.c +++ b/hw/misc/xlnx-versal-cframe-reg.c @@ -XXX,XX +XXX,XX @@ static const MemoryRegionOps cframe_reg_fdri_ops = { }, }; +static uint64_t cframes_bcast_reg_read(void *opaque, hwaddr addr, unsigned size) +{ + qemu_log_mask(LOG_GUEST_ERROR, "%s: Unsupported read from addr=%" + HWADDR_PRIx "\n", __func__, addr); + return 0; +} + +static void cframes_bcast_write(XlnxVersalCFrameBcastReg *s, uint8_t reg_addr, + uint32_t *wfifo) +{ + XlnxCfiPacket pkt = { + .reg_addr = reg_addr, + .data[0] = wfifo[0], + .data[1] = wfifo[1], + .data[2] = wfifo[2], + .data[3] = wfifo[3] + }; + + for (int i = 0; i < ARRAY_SIZE(s->cfg.cframe); i++) { + if (s->cfg.cframe[i]) { + xlnx_cfi_transfer_packet(s->cfg.cframe[i], &pkt); + } + } +} + +static void cframes_bcast_reg_write(void *opaque, hwaddr addr, uint64_t value, + unsigned size) +{ + XlnxVersalCFrameBcastReg *s = XLNX_VERSAL_CFRAME_BCAST_REG(opaque); + uint32_t wfifo[WFIFO_SZ]; + + if (update_wfifo(addr, value, s->wfifo, wfifo)) { + uint8_t reg_addr = extract32(addr, 4, 6); + + cframes_bcast_write(s, reg_addr, wfifo); + } +} + +static uint64_t cframes_bcast_fdri_read(void *opaque, hwaddr addr, + unsigned size) +{ + qemu_log_mask(LOG_GUEST_ERROR, "%s: Unsupported read from addr=%" + HWADDR_PRIx "\n", __func__, addr); + return 0; +} + +static void cframes_bcast_fdri_write(void *opaque, hwaddr addr, uint64_t value, + unsigned size) +{ + XlnxVersalCFrameBcastReg *s = XLNX_VERSAL_CFRAME_BCAST_REG(opaque); + uint32_t wfifo[WFIFO_SZ]; + + if (update_wfifo(addr, value, s->wfifo, wfifo)) { + cframes_bcast_write(s, CFRAME_FDRI, wfifo); + } +} + +static const MemoryRegionOps cframes_bcast_reg_reg_ops = { + .read = cframes_bcast_reg_read, + .write = cframes_bcast_reg_write, + .endianness = DEVICE_LITTLE_ENDIAN, + .valid = { + .min_access_size = 4, + .max_access_size = 4, + }, +}; + +static const MemoryRegionOps cframes_bcast_reg_fdri_ops = { + .read = cframes_bcast_fdri_read, + .write = cframes_bcast_fdri_write, + .endianness = DEVICE_LITTLE_ENDIAN, + .valid = { + .min_access_size = 4, + .max_access_size = 4, + }, +}; + static void cframe_reg_realize(DeviceState *dev, Error **errp) { XlnxVersalCFrameReg *s = XLNX_VERSAL_CFRAME_REG(dev); @@ -XXX,XX +XXX,XX @@ static Property cframe_regs_props[] = { DEFINE_PROP_END_OF_LIST(), }; +static void cframe_bcast_reg_init(Object *obj) +{ + XlnxVersalCFrameBcastReg *s = XLNX_VERSAL_CFRAME_BCAST_REG(obj); + SysBusDevice *sbd = SYS_BUS_DEVICE(obj); + + memory_region_init_io(&s->iomem_reg, obj, &cframes_bcast_reg_reg_ops, s, + TYPE_XLNX_VERSAL_CFRAME_BCAST_REG, KEYHOLE_STREAM_4K); + memory_region_init_io(&s->iomem_fdri, obj, &cframes_bcast_reg_fdri_ops, s, + TYPE_XLNX_VERSAL_CFRAME_BCAST_REG "-fdri", + KEYHOLE_STREAM_4K); + sysbus_init_mmio(sbd, &s->iomem_reg); + sysbus_init_mmio(sbd, &s->iomem_fdri); +} + +static void cframe_bcast_reg_reset_enter(Object *obj, ResetType type) +{ + XlnxVersalCFrameBcastReg *s = XLNX_VERSAL_CFRAME_BCAST_REG(obj); + + memset(s->wfifo, 0, WFIFO_SZ * sizeof(uint32_t)); +} + +static const VMStateDescription vmstate_cframe_bcast_reg = { + .name = TYPE_XLNX_VERSAL_CFRAME_BCAST_REG, + .version_id = 1, + .minimum_version_id = 1, + .fields = (VMStateField[]) { + VMSTATE_UINT32_ARRAY(wfifo, XlnxVersalCFrameBcastReg, 4), + VMSTATE_END_OF_LIST(), + } +}; + +static Property cframe_bcast_regs_props[] = { + DEFINE_PROP_LINK("cframe0", XlnxVersalCFrameBcastReg, cfg.cframe[0], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe1", XlnxVersalCFrameBcastReg, cfg.cframe[1], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe2", XlnxVersalCFrameBcastReg, cfg.cframe[2], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe3", XlnxVersalCFrameBcastReg, cfg.cframe[3], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe4", XlnxVersalCFrameBcastReg, cfg.cframe[4], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe5", XlnxVersalCFrameBcastReg, cfg.cframe[5], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe6", XlnxVersalCFrameBcastReg, cfg.cframe[6], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe7", XlnxVersalCFrameBcastReg, cfg.cframe[7], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe8", XlnxVersalCFrameBcastReg, cfg.cframe[8], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe9", XlnxVersalCFrameBcastReg, cfg.cframe[9], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe10", XlnxVersalCFrameBcastReg, cfg.cframe[10], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe11", XlnxVersalCFrameBcastReg, cfg.cframe[11], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe12", XlnxVersalCFrameBcastReg, cfg.cframe[12], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe13", XlnxVersalCFrameBcastReg, cfg.cframe[13], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_LINK("cframe14", XlnxVersalCFrameBcastReg, cfg.cframe[14], + TYPE_XLNX_CFI_IF, XlnxCfiIf *), + DEFINE_PROP_END_OF_LIST(), +}; + static void cframe_reg_class_init(ObjectClass *klass, void *data) { ResettableClass *rc = RESETTABLE_CLASS(klass); @@ -XXX,XX +XXX,XX @@ static void cframe_reg_class_init(ObjectClass *klass, void *data) xcic->cfi_transfer_packet = cframe_reg_cfi_transfer_packet; } +static void cframe_bcast_reg_class_init(ObjectClass *klass, void *data) +{ + DeviceClass *dc = DEVICE_CLASS(klass); + ResettableClass *rc = RESETTABLE_CLASS(klass); + + dc->vmsd = &vmstate_cframe_bcast_reg; + device_class_set_props(dc, cframe_bcast_regs_props); + rc->phases.enter = cframe_bcast_reg_reset_enter; +} + static const TypeInfo cframe_reg_info = { .name = TYPE_XLNX_VERSAL_CFRAME_REG, .parent = TYPE_SYS_BUS_DEVICE, @@ -XXX,XX +XXX,XX @@ static const TypeInfo cframe_reg_info = { } }; +static const TypeInfo cframe_bcast_reg_info = { + .name = TYPE_XLNX_VERSAL_CFRAME_BCAST_REG, + .parent = TYPE_SYS_BUS_DEVICE, + .instance_size = sizeof(XlnxVersalCFrameBcastReg), + .class_init = cframe_bcast_reg_class_init, + .instance_init = cframe_bcast_reg_init, +}; + static void cframe_reg_register_types(void) { type_register_static(&cframe_reg_info); + type_register_static(&cframe_bcast_reg_info); } type_init(cframe_reg_register_types) -- 2.34.1 From: Francisco Iglesias <francisco.iglesias@amd.com> Connect the Configuration Frame Unit (CFU_APB, CFU_FDRO and CFU_SFR) to the Versal machine. Signed-off-by: Francisco Iglesias <francisco.iglesias@amd.com> Acked-by: Edgar E. Iglesias <edgar@zeroasic.com> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Message-id: 20230831165701.2016397-8-francisco.iglesias@amd.com Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- include/hw/arm/xlnx-versal.h | 16 ++++++++++++++ hw/arm/xlnx-versal.c | 42 ++++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+) diff --git a/include/hw/arm/xlnx-versal.h b/include/hw/arm/xlnx-versal.h index XXXXXXX..XXXXXXX 100644 --- a/include/hw/arm/xlnx-versal.h +++ b/include/hw/arm/xlnx-versal.h @@ -XXX,XX +XXX,XX @@ #include "hw/misc/xlnx-versal-crl.h" #include "hw/misc/xlnx-versal-pmc-iou-slcr.h" #include "hw/net/xlnx-versal-canfd.h" +#include "hw/misc/xlnx-versal-cfu.h" #define TYPE_XLNX_VERSAL "xlnx-versal" OBJECT_DECLARE_SIMPLE_TYPE(Versal, XLNX_VERSAL) @@ -XXX,XX +XXX,XX @@ struct Versal { XlnxEFuse efuse; XlnxVersalEFuseCtrl efuse_ctrl; XlnxVersalEFuseCache efuse_cache; + XlnxVersalCFUAPB cfu_apb; + XlnxVersalCFUFDRO cfu_fdro; + XlnxVersalCFUSFR cfu_sfr; OrIRQState apb_irq_orgate; } pmc; @@ -XXX,XX +XXX,XX @@ struct Versal { #define VERSAL_GEM1_WAKE_IRQ_0 59 #define VERSAL_ADMA_IRQ_0 60 #define VERSAL_XRAM_IRQ_0 79 +#define VERSAL_CFU_IRQ_0 120 #define VERSAL_PMC_APB_IRQ 121 #define VERSAL_OSPI_IRQ 124 #define VERSAL_SD0_IRQ_0 126 @@ -XXX,XX +XXX,XX @@ struct Versal { #define MM_PMC_EFUSE_CACHE 0xf1250000 #define MM_PMC_EFUSE_CACHE_SIZE 0x00C00 +#define MM_PMC_CFU_APB 0xf12b0000 +#define MM_PMC_CFU_APB_SIZE 0x10000 +#define MM_PMC_CFU_STREAM 0xf12c0000 +#define MM_PMC_CFU_STREAM_SIZE 0x1000 +#define MM_PMC_CFU_SFR 0xf12c1000 +#define MM_PMC_CFU_SFR_SIZE 0x1000 +#define MM_PMC_CFU_FDRO 0xf12c2000 +#define MM_PMC_CFU_FDRO_SIZE 0x1000 +#define MM_PMC_CFU_STREAM_2 0xf1f80000 +#define MM_PMC_CFU_STREAM_2_SIZE 0x40000 + #define MM_PMC_CRP 0xf1260000U #define MM_PMC_CRP_SIZE 0x10000 #define MM_PMC_RTC 0xf12a0000 diff --git a/hw/arm/xlnx-versal.c b/hw/arm/xlnx-versal.c index XXXXXXX..XXXXXXX 100644 --- a/hw/arm/xlnx-versal.c +++ b/hw/arm/xlnx-versal.c @@ -XXX,XX +XXX,XX @@ static void versal_create_ospi(Versal *s, qemu_irq *pic) qdev_connect_gpio_out(orgate, 0, pic[VERSAL_OSPI_IRQ]); } +static void versal_create_cfu(Versal *s, qemu_irq *pic) +{ + SysBusDevice *sbd; + + /* CFU FDRO */ + object_initialize_child(OBJECT(s), "cfu-fdro", &s->pmc.cfu_fdro, + TYPE_XLNX_VERSAL_CFU_FDRO); + sbd = SYS_BUS_DEVICE(&s->pmc.cfu_fdro); + + sysbus_realize(sbd, &error_fatal); + memory_region_add_subregion(&s->mr_ps, MM_PMC_CFU_FDRO, + sysbus_mmio_get_region(sbd, 0)); + + /* CFU APB */ + object_initialize_child(OBJECT(s), "cfu-apb", &s->pmc.cfu_apb, + TYPE_XLNX_VERSAL_CFU_APB); + sbd = SYS_BUS_DEVICE(&s->pmc.cfu_apb); + + sysbus_realize(sbd, &error_fatal); + memory_region_add_subregion(&s->mr_ps, MM_PMC_CFU_APB, + sysbus_mmio_get_region(sbd, 0)); + memory_region_add_subregion(&s->mr_ps, MM_PMC_CFU_STREAM, + sysbus_mmio_get_region(sbd, 1)); + memory_region_add_subregion(&s->mr_ps, MM_PMC_CFU_STREAM_2, + sysbus_mmio_get_region(sbd, 2)); + sysbus_connect_irq(sbd, 0, pic[VERSAL_CFU_IRQ_0]); + + /* CFU SFR */ + object_initialize_child(OBJECT(s), "cfu-sfr", &s->pmc.cfu_sfr, + TYPE_XLNX_VERSAL_CFU_SFR); + + sbd = SYS_BUS_DEVICE(&s->pmc.cfu_sfr); + + object_property_set_link(OBJECT(&s->pmc.cfu_sfr), + "cfu", OBJECT(&s->pmc.cfu_apb), &error_abort); + + sysbus_realize(sbd, &error_fatal); + memory_region_add_subregion(&s->mr_ps, MM_PMC_CFU_SFR, + sysbus_mmio_get_region(sbd, 0)); +} + static void versal_create_crl(Versal *s, qemu_irq *pic) { SysBusDevice *sbd; @@ -XXX,XX +XXX,XX @@ static void versal_realize(DeviceState *dev, Error **errp) versal_create_pmc_iou_slcr(s, pic); versal_create_ospi(s, pic); versal_create_crl(s, pic); + versal_create_cfu(s, pic); versal_map_ddr(s); versal_unimp(s); -- 2.34.1 From: Francisco Iglesias <francisco.iglesias@amd.com> Connect the Configuration Frame controller (CFRAME_REG) and the Configuration Frame broadcast controller (CFRAME_BCAST_REG) to the Versal machine. Signed-off-by: Francisco Iglesias <francisco.iglesias@amd.com> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Message-id: 20230831165701.2016397-9-francisco.iglesias@amd.com Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- include/hw/arm/xlnx-versal.h | 69 +++++++++++++++++++++ hw/arm/xlnx-versal.c | 113 ++++++++++++++++++++++++++++++++++- 2 files changed, 181 insertions(+), 1 deletion(-) diff --git a/include/hw/arm/xlnx-versal.h b/include/hw/arm/xlnx-versal.h index XXXXXXX..XXXXXXX 100644 --- a/include/hw/arm/xlnx-versal.h +++ b/include/hw/arm/xlnx-versal.h @@ -XXX,XX +XXX,XX @@ #include "hw/misc/xlnx-versal-pmc-iou-slcr.h" #include "hw/net/xlnx-versal-canfd.h" #include "hw/misc/xlnx-versal-cfu.h" +#include "hw/misc/xlnx-versal-cframe-reg.h" #define TYPE_XLNX_VERSAL "xlnx-versal" OBJECT_DECLARE_SIMPLE_TYPE(Versal, XLNX_VERSAL) @@ -XXX,XX +XXX,XX @@ OBJECT_DECLARE_SIMPLE_TYPE(Versal, XLNX_VERSAL) #define XLNX_VERSAL_NR_IRQS 192 #define XLNX_VERSAL_NR_CANFD 2 #define XLNX_VERSAL_CANFD_REF_CLK (24 * 1000 * 1000) +#define XLNX_VERSAL_NR_CFRAME 15 struct Versal { /*< private >*/ @@ -XXX,XX +XXX,XX @@ struct Versal { XlnxVersalCFUAPB cfu_apb; XlnxVersalCFUFDRO cfu_fdro; XlnxVersalCFUSFR cfu_sfr; + XlnxVersalCFrameReg cframe[XLNX_VERSAL_NR_CFRAME]; + XlnxVersalCFrameBcastReg cframe_bcast; OrIRQState apb_irq_orgate; } pmc; @@ -XXX,XX +XXX,XX @@ struct Versal { #define MM_PMC_CFU_STREAM_2 0xf1f80000 #define MM_PMC_CFU_STREAM_2_SIZE 0x40000 +#define MM_PMC_CFRAME0_REG 0xf12d0000 +#define MM_PMC_CFRAME0_REG_SIZE 0x1000 +#define MM_PMC_CFRAME0_FDRI 0xf12d1000 +#define MM_PMC_CFRAME0_FDRI_SIZE 0x1000 +#define MM_PMC_CFRAME1_REG 0xf12d2000 +#define MM_PMC_CFRAME1_REG_SIZE 0x1000 +#define MM_PMC_CFRAME1_FDRI 0xf12d3000 +#define MM_PMC_CFRAME1_FDRI_SIZE 0x1000 +#define MM_PMC_CFRAME2_REG 0xf12d4000 +#define MM_PMC_CFRAME2_REG_SIZE 0x1000 +#define MM_PMC_CFRAME2_FDRI 0xf12d5000 +#define MM_PMC_CFRAME2_FDRI_SIZE 0x1000 +#define MM_PMC_CFRAME3_REG 0xf12d6000 +#define MM_PMC_CFRAME3_REG_SIZE 0x1000 +#define MM_PMC_CFRAME3_FDRI 0xf12d7000 +#define MM_PMC_CFRAME3_FDRI_SIZE 0x1000 +#define MM_PMC_CFRAME4_REG 0xf12d8000 +#define MM_PMC_CFRAME4_REG_SIZE 0x1000 +#define MM_PMC_CFRAME4_FDRI 0xf12d9000 +#define MM_PMC_CFRAME4_FDRI_SIZE 0x1000 +#define MM_PMC_CFRAME5_REG 0xf12da000 +#define MM_PMC_CFRAME5_REG_SIZE 0x1000 +#define MM_PMC_CFRAME5_FDRI 0xf12db000 +#define MM_PMC_CFRAME5_FDRI_SIZE 0x1000 +#define MM_PMC_CFRAME6_REG 0xf12dc000 +#define MM_PMC_CFRAME6_REG_SIZE 0x1000 +#define MM_PMC_CFRAME6_FDRI 0xf12dd000 +#define MM_PMC_CFRAME6_FDRI_SIZE 0x1000 +#define MM_PMC_CFRAME7_REG 0xf12de000 +#define MM_PMC_CFRAME7_REG_SIZE 0x1000 +#define MM_PMC_CFRAME7_FDRI 0xf12df000 +#define MM_PMC_CFRAME7_FDRI_SIZE 0x1000 +#define MM_PMC_CFRAME8_REG 0xf12e0000 +#define MM_PMC_CFRAME8_REG_SIZE 0x1000 +#define MM_PMC_CFRAME8_FDRI 0xf12e1000 +#define MM_PMC_CFRAME8_FDRI_SIZE 0x1000 +#define MM_PMC_CFRAME9_REG 0xf12e2000 +#define MM_PMC_CFRAME9_REG_SIZE 0x1000 +#define MM_PMC_CFRAME9_FDRI 0xf12e3000 +#define MM_PMC_CFRAME9_FDRI_SIZE 0x1000 +#define MM_PMC_CFRAME10_REG 0xf12e4000 +#define MM_PMC_CFRAME10_REG_SIZE 0x1000 +#define MM_PMC_CFRAME10_FDRI 0xf12e5000 +#define MM_PMC_CFRAME10_FDRI_SIZE 0x1000 +#define MM_PMC_CFRAME11_REG 0xf12e6000 +#define MM_PMC_CFRAME11_REG_SIZE 0x1000 +#define MM_PMC_CFRAME11_FDRI 0xf12e7000 +#define MM_PMC_CFRAME11_FDRI_SIZE 0x1000 +#define MM_PMC_CFRAME12_REG 0xf12e8000 +#define MM_PMC_CFRAME12_REG_SIZE 0x1000 +#define MM_PMC_CFRAME12_FDRI 0xf12e9000 +#define MM_PMC_CFRAME12_FDRI_SIZE 0x1000 +#define MM_PMC_CFRAME13_REG 0xf12ea000 +#define MM_PMC_CFRAME13_REG_SIZE 0x1000 +#define MM_PMC_CFRAME13_FDRI 0xf12eb000 +#define MM_PMC_CFRAME13_FDRI_SIZE 0x1000 +#define MM_PMC_CFRAME14_REG 0xf12ec000 +#define MM_PMC_CFRAME14_REG_SIZE 0x1000 +#define MM_PMC_CFRAME14_FDRI 0xf12ed000 +#define MM_PMC_CFRAME14_FDRI_SIZE 0x1000 +#define MM_PMC_CFRAME_BCAST_REG 0xf12ee000 +#define MM_PMC_CFRAME_BCAST_REG_SIZE 0x1000 +#define MM_PMC_CFRAME_BCAST_FDRI 0xf12ef000 +#define MM_PMC_CFRAME_BCAST_FDRI_SIZE 0x1000 + #define MM_PMC_CRP 0xf1260000U #define MM_PMC_CRP_SIZE 0x10000 #define MM_PMC_RTC 0xf12a0000 diff --git a/hw/arm/xlnx-versal.c b/hw/arm/xlnx-versal.c index XXXXXXX..XXXXXXX 100644 --- a/hw/arm/xlnx-versal.c +++ b/hw/arm/xlnx-versal.c @@ -XXX,XX +XXX,XX @@ #define XLNX_VERSAL_RCPU_TYPE ARM_CPU_TYPE_NAME("cortex-r5f") #define GEM_REVISION 0x40070106 -#define VERSAL_NUM_PMC_APB_IRQS 3 +#define VERSAL_NUM_PMC_APB_IRQS 18 #define NUM_OSPI_IRQ_LINES 3 static void versal_create_apu_cpus(Versal *s) @@ -XXX,XX +XXX,XX @@ static void versal_create_pmc_apb_irq_orgate(Versal *s, qemu_irq *pic) * - RTC * - BBRAM * - PMC SLCR + * - CFRAME regs (input 3 - 17 to the orgate) */ object_initialize_child(OBJECT(s), "pmc-apb-irq-orgate", &s->pmc.apb_irq_orgate, TYPE_OR_IRQ); @@ -XXX,XX +XXX,XX @@ static void versal_create_ospi(Versal *s, qemu_irq *pic) static void versal_create_cfu(Versal *s, qemu_irq *pic) { SysBusDevice *sbd; + DeviceState *dev; + int i; + const struct { + uint64_t reg_base; + uint64_t fdri_base; + } cframe_addr[] = { + { MM_PMC_CFRAME0_REG, MM_PMC_CFRAME0_FDRI }, + { MM_PMC_CFRAME1_REG, MM_PMC_CFRAME1_FDRI }, + { MM_PMC_CFRAME2_REG, MM_PMC_CFRAME2_FDRI }, + { MM_PMC_CFRAME3_REG, MM_PMC_CFRAME3_FDRI }, + { MM_PMC_CFRAME4_REG, MM_PMC_CFRAME4_FDRI }, + { MM_PMC_CFRAME5_REG, MM_PMC_CFRAME5_FDRI }, + { MM_PMC_CFRAME6_REG, MM_PMC_CFRAME6_FDRI }, + { MM_PMC_CFRAME7_REG, MM_PMC_CFRAME7_FDRI }, + { MM_PMC_CFRAME8_REG, MM_PMC_CFRAME8_FDRI }, + { MM_PMC_CFRAME9_REG, MM_PMC_CFRAME9_FDRI }, + { MM_PMC_CFRAME10_REG, MM_PMC_CFRAME10_FDRI }, + { MM_PMC_CFRAME11_REG, MM_PMC_CFRAME11_FDRI }, + { MM_PMC_CFRAME12_REG, MM_PMC_CFRAME12_FDRI }, + { MM_PMC_CFRAME13_REG, MM_PMC_CFRAME13_FDRI }, + { MM_PMC_CFRAME14_REG, MM_PMC_CFRAME14_FDRI }, + }; + const struct { + uint32_t blktype0_frames; + uint32_t blktype1_frames; + uint32_t blktype2_frames; + uint32_t blktype3_frames; + uint32_t blktype4_frames; + uint32_t blktype5_frames; + uint32_t blktype6_frames; + } cframe_cfg[] = { + [0] = { 34111, 3528, 12800, 11, 5, 1, 1 }, + [1] = { 38498, 3841, 15361, 13, 7, 3, 1 }, + [2] = { 38498, 3841, 15361, 13, 7, 3, 1 }, + [3] = { 38498, 3841, 15361, 13, 7, 3, 1 }, + }; /* CFU FDRO */ object_initialize_child(OBJECT(s), "cfu-fdro", &s->pmc.cfu_fdro, @@ -XXX,XX +XXX,XX @@ static void versal_create_cfu(Versal *s, qemu_irq *pic) memory_region_add_subregion(&s->mr_ps, MM_PMC_CFU_FDRO, sysbus_mmio_get_region(sbd, 0)); + /* CFRAME REG */ + for (i = 0; i < ARRAY_SIZE(s->pmc.cframe); i++) { + g_autofree char *name = g_strdup_printf("cframe%d", i); + + object_initialize_child(OBJECT(s), name, &s->pmc.cframe[i], + TYPE_XLNX_VERSAL_CFRAME_REG); + + sbd = SYS_BUS_DEVICE(&s->pmc.cframe[i]); + dev = DEVICE(&s->pmc.cframe[i]); + + if (i < ARRAY_SIZE(cframe_cfg)) { + object_property_set_int(OBJECT(dev), "blktype0-frames", + cframe_cfg[i].blktype0_frames, + &error_abort); + object_property_set_int(OBJECT(dev), "blktype1-frames", + cframe_cfg[i].blktype1_frames, + &error_abort); + object_property_set_int(OBJECT(dev), "blktype2-frames", + cframe_cfg[i].blktype2_frames, + &error_abort); + object_property_set_int(OBJECT(dev), "blktype3-frames", + cframe_cfg[i].blktype3_frames, + &error_abort); + object_property_set_int(OBJECT(dev), "blktype4-frames", + cframe_cfg[i].blktype4_frames, + &error_abort); + object_property_set_int(OBJECT(dev), "blktype5-frames", + cframe_cfg[i].blktype5_frames, + &error_abort); + object_property_set_int(OBJECT(dev), "blktype6-frames", + cframe_cfg[i].blktype6_frames, + &error_abort); + } + object_property_set_link(OBJECT(dev), "cfu-fdro", + OBJECT(&s->pmc.cfu_fdro), &error_fatal); + + sysbus_realize(SYS_BUS_DEVICE(dev), &error_fatal); + + memory_region_add_subregion(&s->mr_ps, cframe_addr[i].reg_base, + sysbus_mmio_get_region(sbd, 0)); + memory_region_add_subregion(&s->mr_ps, cframe_addr[i].fdri_base, + sysbus_mmio_get_region(sbd, 1)); + sysbus_connect_irq(sbd, 0, + qdev_get_gpio_in(DEVICE(&s->pmc.apb_irq_orgate), + 3 + i)); + } + + /* CFRAME BCAST */ + object_initialize_child(OBJECT(s), "cframe_bcast", &s->pmc.cframe_bcast, + TYPE_XLNX_VERSAL_CFRAME_BCAST_REG); + + sbd = SYS_BUS_DEVICE(&s->pmc.cframe_bcast); + dev = DEVICE(&s->pmc.cframe_bcast); + + for (i = 0; i < ARRAY_SIZE(s->pmc.cframe); i++) { + g_autofree char *propname = g_strdup_printf("cframe%d", i); + object_property_set_link(OBJECT(dev), propname, + OBJECT(&s->pmc.cframe[i]), &error_fatal); + } + + sysbus_realize(sbd, &error_fatal); + + memory_region_add_subregion(&s->mr_ps, MM_PMC_CFRAME_BCAST_REG, + sysbus_mmio_get_region(sbd, 0)); + memory_region_add_subregion(&s->mr_ps, MM_PMC_CFRAME_BCAST_FDRI, + sysbus_mmio_get_region(sbd, 1)); + /* CFU APB */ object_initialize_child(OBJECT(s), "cfu-apb", &s->pmc.cfu_apb, TYPE_XLNX_VERSAL_CFU_APB); sbd = SYS_BUS_DEVICE(&s->pmc.cfu_apb); + dev = DEVICE(&s->pmc.cfu_apb); + + for (i = 0; i < ARRAY_SIZE(s->pmc.cframe); i++) { + g_autofree char *propname = g_strdup_printf("cframe%d", i); + object_property_set_link(OBJECT(dev), propname, + OBJECT(&s->pmc.cframe[i]), &error_fatal); + } sysbus_realize(sbd, &error_fatal); memory_region_add_subregion(&s->mr_ps, MM_PMC_CFU_APB, -- 2.34.1 From: Richard Henderson <richard.henderson@linaro.org> STGP writes to tag memory, it does not check it. This happened to work because we wrote tag memory first so that the check always succeeded. Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20230901203103.136408-1-richard.henderson@linaro.org Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/tcg/translate-a64.c | 41 +++++++++++++--------------------- 1 file changed, 15 insertions(+), 26 deletions(-) diff --git a/target/arm/tcg/translate-a64.c b/target/arm/tcg/translate-a64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/translate-a64.c +++ b/target/arm/tcg/translate-a64.c @@ -XXX,XX +XXX,XX @@ static bool trans_STGP(DisasContext *s, arg_ldstpair *a) tcg_gen_addi_i64(dirty_addr, dirty_addr, offset); } - if (!s->ata) { - /* - * TODO: We could rely on the stores below, at least for - * system mode, if we arrange to add MO_ALIGN_16. - */ - gen_helper_stg_stub(cpu_env, dirty_addr); - } else if (tb_cflags(s->base.tb) & CF_PARALLEL) { - gen_helper_stg_parallel(cpu_env, dirty_addr, dirty_addr); - } else { - gen_helper_stg(cpu_env, dirty_addr, dirty_addr); - } - - mop = finalize_memop(s, MO_64); - clean_addr = gen_mte_checkN(s, dirty_addr, true, false, 2 << MO_64, mop); - + clean_addr = clean_data_tbi(s, dirty_addr); tcg_rt = cpu_reg(s, a->rt); tcg_rt2 = cpu_reg(s, a->rt2); /* - * STGP is defined as two 8-byte memory operations and one tag operation. - * We implement it as one single 16-byte memory operation for convenience. - * Rebuild mop as for STP. - * TODO: The atomicity with LSE2 is stronger than required. - * Need a form of MO_ATOM_WITHIN16_PAIR that never requires - * 16-byte atomicity. + * STGP is defined as two 8-byte memory operations, aligned to TAG_GRANULE, + * and one tag operation. We implement it as one single aligned 16-byte + * memory operation for convenience. Note that the alignment ensures + * MO_ATOM_IFALIGN_PAIR produces 8-byte atomicity for the memory store. */ - mop = MO_128; - if (s->align_mem) { - mop |= MO_ALIGN_8; - } - mop = finalize_memop_pair(s, mop); + mop = finalize_memop_atom(s, MO_128 | MO_ALIGN, MO_ATOM_IFALIGN_PAIR); tmp = tcg_temp_new_i128(); if (s->be_data == MO_LE) { @@ -XXX,XX +XXX,XX @@ static bool trans_STGP(DisasContext *s, arg_ldstpair *a) } tcg_gen_qemu_st_i128(tmp, clean_addr, get_mem_index(s), mop); + /* Perform the tag store, if tag access enabled. */ + if (s->ata) { + if (tb_cflags(s->base.tb) & CF_PARALLEL) { + gen_helper_stg_parallel(cpu_env, dirty_addr, dirty_addr); + } else { + gen_helper_stg(cpu_env, dirty_addr, dirty_addr); + } + } + op_addr_ldstpair_post(s, a, dirty_addr, offset); return true; } -- 2.34.1 From: Colton Lewis <coltonlewis@google.com> Due to recent KVM changes, QEMU is setting a ptimer offset resulting in unintended trap and emulate access and a consequent performance hit. Filter out the PTIMER_CNT register to restore trapless ptimer access. Quoting Andrew Jones: Simply reading the CNT register and writing back the same value is enough to set an offset, since the timer will have certainly moved past whatever value was read by the time it's written. QEMU frequently saves and restores all registers in the get-reg-list array, unless they've been explicitly filtered out (with Linux commit 680232a94c12, KVM_REG_ARM_PTIMER_CNT is now in the array). So, to restore trapless ptimer accesses, we need a QEMU patch to filter out the register. See https://lore.kernel.org/kvmarm/gsntttsonus5.fsf@coltonlewis-kvm.c.googlers.com/T/#m0770023762a821db2a3f0dd0a7dc6aa54e0d0da9 for additional context. Cc: qemu-stable@nongnu.org Signed-off-by: Andrew Jones <andrew.jones@linux.dev> Signed-off-by: Colton Lewis <coltonlewis@google.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Tested-by: Colton Lewis <coltonlewis@google.com> Message-id: 20230831190052.129045-1-coltonlewis@google.com Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/kvm64.c | 1 + 1 file changed, 1 insertion(+) diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/kvm64.c +++ b/target/arm/kvm64.c @@ -XXX,XX +XXX,XX @@ typedef struct CPRegStateLevel { */ static const CPRegStateLevel non_runtime_cpregs[] = { { KVM_REG_ARM_TIMER_CNT, KVM_PUT_FULL_STATE }, + { KVM_REG_ARM_PTIMER_CNT, KVM_PUT_FULL_STATE }, }; int kvm_arm_cpreg_level(uint64_t regidx) -- 2.34.1 From: Richard Henderson <richard.henderson@linaro.org> Provide a stub implementation, as a write is a "request". Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20230831232441.66020-2-richard.henderson@linaro.org Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper.c | 64 +++++++++++++++++++++++++++++---------------- 1 file changed, 41 insertions(+), 23 deletions(-) diff --git a/target/arm/helper.c b/target/arm/helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper.c +++ b/target/arm/helper.c @@ -XXX,XX +XXX,XX @@ void register_cp_regs_for_features(ARMCPU *cpu) }; modify_arm_cp_regs(v8_idregs, v8_user_idregs); #endif - /* RVBAR_EL1 is only implemented if EL1 is the highest EL */ + /* + * RVBAR_EL1 and RMR_EL1 only implemented if EL1 is the highest EL. + * TODO: For RMR, a write with bit 1 set should do something with + * cpu_reset(). In the meantime, "the bit is strictly a request", + * so we are in spec just ignoring writes. + */ if (!arm_feature(env, ARM_FEATURE_EL3) && !arm_feature(env, ARM_FEATURE_EL2)) { - ARMCPRegInfo rvbar = { - .name = "RVBAR_EL1", .state = ARM_CP_STATE_BOTH, - .opc0 = 3, .opc1 = 0, .crn = 12, .crm = 0, .opc2 = 1, - .access = PL1_R, - .fieldoffset = offsetof(CPUARMState, cp15.rvbar), + ARMCPRegInfo el1_reset_regs[] = { + { .name = "RVBAR_EL1", .state = ARM_CP_STATE_BOTH, + .opc0 = 3, .opc1 = 0, .crn = 12, .crm = 0, .opc2 = 1, + .access = PL1_R, + .fieldoffset = offsetof(CPUARMState, cp15.rvbar) }, + { .name = "RMR_EL1", .state = ARM_CP_STATE_BOTH, + .opc0 = 3, .opc1 = 0, .crn = 12, .crm = 0, .opc2 = 2, + .access = PL1_RW, .type = ARM_CP_CONST, + .resetvalue = arm_feature(env, ARM_FEATURE_AARCH64) } }; - define_one_arm_cp_reg(cpu, &rvbar); + define_arm_cp_regs(cpu, el1_reset_regs); } define_arm_cp_regs(cpu, v8_idregs); define_arm_cp_regs(cpu, v8_cp_reginfo); @@ -XXX,XX +XXX,XX @@ void register_cp_regs_for_features(ARMCPU *cpu) if (cpu_isar_feature(aa64_sel2, cpu)) { define_arm_cp_regs(cpu, el2_sec_cp_reginfo); } - /* RVBAR_EL2 is only implemented if EL2 is the highest EL */ + /* + * RVBAR_EL2 and RMR_EL2 only implemented if EL2 is the highest EL. + * See commentary near RMR_EL1. + */ if (!arm_feature(env, ARM_FEATURE_EL3)) { - ARMCPRegInfo rvbar[] = { - { - .name = "RVBAR_EL2", .state = ARM_CP_STATE_AA64, - .opc0 = 3, .opc1 = 4, .crn = 12, .crm = 0, .opc2 = 1, - .access = PL2_R, - .fieldoffset = offsetof(CPUARMState, cp15.rvbar), - }, - { .name = "RVBAR", .type = ARM_CP_ALIAS, - .cp = 15, .opc1 = 0, .crn = 12, .crm = 0, .opc2 = 1, - .access = PL2_R, - .fieldoffset = offsetof(CPUARMState, cp15.rvbar), - }, + static const ARMCPRegInfo el2_reset_regs[] = { + { .name = "RVBAR_EL2", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 4, .crn = 12, .crm = 0, .opc2 = 1, + .access = PL2_R, + .fieldoffset = offsetof(CPUARMState, cp15.rvbar) }, + { .name = "RVBAR", .type = ARM_CP_ALIAS, + .cp = 15, .opc1 = 0, .crn = 12, .crm = 0, .opc2 = 1, + .access = PL2_R, + .fieldoffset = offsetof(CPUARMState, cp15.rvbar) }, + { .name = "RMR_EL2", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 4, .crn = 12, .crm = 0, .opc2 = 2, + .access = PL2_RW, .type = ARM_CP_CONST, .resetvalue = 1 }, }; - define_arm_cp_regs(cpu, rvbar); + define_arm_cp_regs(cpu, el2_reset_regs); } } @@ -XXX,XX +XXX,XX @@ void register_cp_regs_for_features(ARMCPU *cpu) { .name = "RVBAR_EL3", .state = ARM_CP_STATE_AA64, .opc0 = 3, .opc1 = 6, .crn = 12, .crm = 0, .opc2 = 1, .access = PL3_R, - .fieldoffset = offsetof(CPUARMState, cp15.rvbar), - }, + .fieldoffset = offsetof(CPUARMState, cp15.rvbar), }, + { .name = "RMR_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 12, .crm = 0, .opc2 = 2, + .access = PL3_RW, .type = ARM_CP_CONST, .resetvalue = 1 }, + { .name = "RMR", .state = ARM_CP_STATE_AA32, + .cp = 15, .opc1 = 0, .crn = 12, .crm = 0, .opc2 = 2, + .access = PL3_RW, .type = ARM_CP_CONST, + .resetvalue = arm_feature(env, ARM_FEATURE_AARCH64) }, { .name = "SCTLR_EL3", .state = ARM_CP_STATE_AA64, .opc0 = 3, .opc1 = 6, .crn = 1, .crm = 0, .opc2 = 0, .access = PL3_RW, -- 2.34.1 From: Richard Henderson <richard.henderson@linaro.org> The cortex-a710 is a first generation ARMv9.0-A processor. Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20230831232441.66020-3-richard.henderson@linaro.org Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- docs/system/arm/virt.rst | 1 + hw/arm/virt.c | 1 + target/arm/tcg/cpu64.c | 212 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 214 insertions(+) diff --git a/docs/system/arm/virt.rst b/docs/system/arm/virt.rst index XXXXXXX..XXXXXXX 100644 --- a/docs/system/arm/virt.rst +++ b/docs/system/arm/virt.rst @@ -XXX,XX +XXX,XX @@ Supported guest CPU types: - ``cortex-a57`` (64-bit) - ``cortex-a72`` (64-bit) - ``cortex-a76`` (64-bit) +- ``cortex-a710`` (64-bit) - ``a64fx`` (64-bit) - ``host`` (with KVM only) - ``neoverse-n1`` (64-bit) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index XXXXXXX..XXXXXXX 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -XXX,XX +XXX,XX @@ static const char *valid_cpus[] = { ARM_CPU_TYPE_NAME("cortex-a55"), ARM_CPU_TYPE_NAME("cortex-a72"), ARM_CPU_TYPE_NAME("cortex-a76"), + ARM_CPU_TYPE_NAME("cortex-a710"), ARM_CPU_TYPE_NAME("a64fx"), ARM_CPU_TYPE_NAME("neoverse-n1"), ARM_CPU_TYPE_NAME("neoverse-v1"), diff --git a/target/arm/tcg/cpu64.c b/target/arm/tcg/cpu64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/cpu64.c +++ b/target/arm/tcg/cpu64.c @@ -XXX,XX +XXX,XX @@ static void aarch64_neoverse_v1_initfn(Object *obj) aarch64_add_sve_properties(obj); } +static const ARMCPRegInfo cortex_a710_cp_reginfo[] = { + { .name = "CPUACTLR_EL1", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 0, .crn = 15, .crm = 1, .opc2 = 0, + .access = PL1_RW, .type = ARM_CP_CONST, .resetvalue = 0, + .accessfn = access_actlr_w }, + { .name = "CPUACTLR2_EL1", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 0, .crn = 15, .crm = 1, .opc2 = 1, + .access = PL1_RW, .type = ARM_CP_CONST, .resetvalue = 0, + .accessfn = access_actlr_w }, + { .name = "CPUACTLR3_EL1", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 0, .crn = 15, .crm = 1, .opc2 = 2, + .access = PL1_RW, .type = ARM_CP_CONST, .resetvalue = 0, + .accessfn = access_actlr_w }, + { .name = "CPUACTLR4_EL1", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 0, .crn = 15, .crm = 1, .opc2 = 3, + .access = PL1_RW, .type = ARM_CP_CONST, .resetvalue = 0, + .accessfn = access_actlr_w }, + { .name = "CPUECTLR_EL1", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 0, .crn = 15, .crm = 1, .opc2 = 4, + .access = PL1_RW, .type = ARM_CP_CONST, .resetvalue = 0, + .accessfn = access_actlr_w }, + { .name = "CPUECTLR2_EL1", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 0, .crn = 15, .crm = 1, .opc2 = 5, + .access = PL1_RW, .type = ARM_CP_CONST, .resetvalue = 0, + .accessfn = access_actlr_w }, + { .name = "CPUPPMCR_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 0, .crn = 15, .crm = 2, .opc2 = 4, + .access = PL3_RW, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "CPUPWRCTLR_EL1", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 0, .crn = 15, .crm = 2, .opc2 = 7, + .access = PL1_RW, .type = ARM_CP_CONST, .resetvalue = 0, + .accessfn = access_actlr_w }, + { .name = "ATCR_EL1", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 0, .crn = 15, .crm = 7, .opc2 = 0, + .access = PL1_RW, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "CPUACTLR5_EL1", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 0, .crn = 15, .crm = 8, .opc2 = 0, + .access = PL1_RW, .type = ARM_CP_CONST, .resetvalue = 0, + .accessfn = access_actlr_w }, + { .name = "CPUACTLR6_EL1", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 0, .crn = 15, .crm = 8, .opc2 = 1, + .access = PL1_RW, .type = ARM_CP_CONST, .resetvalue = 0, + .accessfn = access_actlr_w }, + { .name = "CPUACTLR7_EL1", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 0, .crn = 15, .crm = 8, .opc2 = 2, + .access = PL1_RW, .type = ARM_CP_CONST, .resetvalue = 0, + .accessfn = access_actlr_w }, + { .name = "ATCR_EL2", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 4, .crn = 15, .crm = 7, .opc2 = 0, + .access = PL2_RW, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "AVTCR_EL2", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 4, .crn = 15, .crm = 7, .opc2 = 1, + .access = PL2_RW, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "CPUPPMCR_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 15, .crm = 2, .opc2 = 0, + .access = PL3_RW, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "CPUPPMCR2_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 15, .crm = 2, .opc2 = 1, + .access = PL3_RW, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "CPUPPMCR4_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 15, .crm = 2, .opc2 = 4, + .access = PL3_RW, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "CPUPPMCR5_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 15, .crm = 2, .opc2 = 5, + .access = PL3_RW, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "CPUPPMCR6_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 15, .crm = 2, .opc2 = 6, + .access = PL3_RW, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "CPUACTLR_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 15, .crm = 4, .opc2 = 0, + .access = PL3_RW, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "ATCR_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 15, .crm = 7, .opc2 = 0, + .access = PL3_RW, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "CPUPSELR_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 15, .crm = 8, .opc2 = 0, + .access = PL3_RW, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "CPUPCR_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 15, .crm = 8, .opc2 = 1, + .access = PL3_RW, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "CPUPOR_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 15, .crm = 8, .opc2 = 2, + .access = PL3_RW, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "CPUPMR_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 15, .crm = 8, .opc2 = 3, + .access = PL3_RW, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "CPUPOR2_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 15, .crm = 8, .opc2 = 4, + .access = PL3_RW, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "CPUPMR2_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 15, .crm = 8, .opc2 = 5, + .access = PL3_RW, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "CPUPFR_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 15, .crm = 8, .opc2 = 6, + .access = PL3_RW, .type = ARM_CP_CONST, .resetvalue = 0 }, + + /* + * Stub RAMINDEX, as we don't actually implement caches, BTB, + * or anything else with cpu internal memory. + * "Read" zeros into the IDATA* and DDATA* output registers. + */ + { .name = "RAMINDEX_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 1, .opc1 = 6, .crn = 15, .crm = 0, .opc2 = 0, + .access = PL3_W, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "IDATA0_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 15, .crm = 0, .opc2 = 0, + .access = PL3_R, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "IDATA1_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 15, .crm = 0, .opc2 = 1, + .access = PL3_R, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "IDATA2_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 15, .crm = 0, .opc2 = 2, + .access = PL3_R, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "DDATA0_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 15, .crm = 1, .opc2 = 0, + .access = PL3_R, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "DDATA1_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 15, .crm = 1, .opc2 = 1, + .access = PL3_R, .type = ARM_CP_CONST, .resetvalue = 0 }, + { .name = "DDATA2_EL3", .state = ARM_CP_STATE_AA64, + .opc0 = 3, .opc1 = 6, .crn = 15, .crm = 1, .opc2 = 2, + .access = PL3_R, .type = ARM_CP_CONST, .resetvalue = 0 }, +}; + +static void aarch64_a710_initfn(Object *obj) +{ + ARMCPU *cpu = ARM_CPU(obj); + + cpu->dtb_compatible = "arm,cortex-a710"; + set_feature(&cpu->env, ARM_FEATURE_V8); + set_feature(&cpu->env, ARM_FEATURE_NEON); + set_feature(&cpu->env, ARM_FEATURE_GENERIC_TIMER); + set_feature(&cpu->env, ARM_FEATURE_AARCH64); + set_feature(&cpu->env, ARM_FEATURE_CBAR_RO); + set_feature(&cpu->env, ARM_FEATURE_EL2); + set_feature(&cpu->env, ARM_FEATURE_EL3); + set_feature(&cpu->env, ARM_FEATURE_PMU); + + /* Ordered by Section B.4: AArch64 registers */ + cpu->midr = 0x412FD471; /* r2p1 */ + cpu->revidr = 0; + cpu->isar.id_pfr0 = 0x21110131; + cpu->isar.id_pfr1 = 0x00010000; /* GIC filled in later */ + cpu->isar.id_dfr0 = 0x16011099; + cpu->id_afr0 = 0; + cpu->isar.id_mmfr0 = 0x10201105; + cpu->isar.id_mmfr1 = 0x40000000; + cpu->isar.id_mmfr2 = 0x01260000; + cpu->isar.id_mmfr3 = 0x02122211; + cpu->isar.id_isar0 = 0x02101110; + cpu->isar.id_isar1 = 0x13112111; + cpu->isar.id_isar2 = 0x21232042; + cpu->isar.id_isar3 = 0x01112131; + cpu->isar.id_isar4 = 0x00010142; + cpu->isar.id_isar5 = 0x11011121; /* with Crypto */ + cpu->isar.id_mmfr4 = 0x21021110; + cpu->isar.id_isar6 = 0x01111111; + cpu->isar.mvfr0 = 0x10110222; + cpu->isar.mvfr1 = 0x13211111; + cpu->isar.mvfr2 = 0x00000043; + cpu->isar.id_pfr2 = 0x00000011; + cpu->isar.id_aa64pfr0 = 0x1201111120111112ull; /* GIC filled in later */ + cpu->isar.id_aa64pfr1 = 0x0000000000000221ull; + cpu->isar.id_aa64zfr0 = 0x0000110100110021ull; /* with Crypto */ + cpu->isar.id_aa64dfr0 = 0x000011f010305611ull; + cpu->isar.id_aa64dfr1 = 0; + cpu->id_aa64afr0 = 0; + cpu->id_aa64afr1 = 0; + cpu->isar.id_aa64isar0 = 0x0221111110212120ull; /* with Crypto */ + cpu->isar.id_aa64isar1 = 0x0010111101211032ull; + cpu->isar.id_aa64mmfr0 = 0x0000022200101122ull; + cpu->isar.id_aa64mmfr1 = 0x0000000010212122ull; + cpu->isar.id_aa64mmfr2 = 0x1221011110101011ull; + cpu->clidr = 0x0000001482000023ull; + cpu->gm_blocksize = 4; + cpu->ctr = 0x000000049444c004ull; + cpu->dcz_blocksize = 4; + /* TODO FEAT_MPAM: mpamidr_el1 = 0x0000_0001_0006_003f */ + + /* Section B.5.2: PMCR_EL0 */ + cpu->isar.reset_pmcr_el0 = 0xa000; /* with 20 counters */ + + /* Section B.6.7: ICH_VTR_EL2 */ + cpu->gic_num_lrs = 4; + cpu->gic_vpribits = 5; + cpu->gic_vprebits = 5; + cpu->gic_pribits = 5; + + /* Section 14: Scalable Vector Extensions support */ + cpu->sve_vq.supported = 1 << 0; /* 128bit */ + + /* + * The cortex-a710 TRM does not list CCSIDR values. The layout of + * the caches are in text in Table 7-1, Table 8-1, and Table 9-1. + * + * L1: 4-way set associative 64-byte line size, total either 32K or 64K. + * L2: 8-way set associative 64 byte line size, total either 256K or 512K. + */ + cpu->ccsidr[0] = make_ccsidr64(4, 64, 64 * KiB); /* L1 dcache */ + cpu->ccsidr[1] = cpu->ccsidr[0]; /* L1 icache */ + cpu->ccsidr[2] = make_ccsidr64(8, 64, 512 * KiB); /* L2 cache */ + + /* FIXME: Not documented -- copied from neoverse-v1 */ + cpu->reset_sctlr = 0x30c50838; + + define_arm_cp_regs(cpu, cortex_a710_cp_reginfo); + + aarch64_add_pauth_properties(obj); + aarch64_add_sve_properties(obj); +} + /* * -cpu max: a CPU with as many features enabled as our emulation supports. * The version of '-cpu max' for qemu-system-arm is defined in cpu32.c; @@ -XXX,XX +XXX,XX @@ static const ARMCPUInfo aarch64_cpus[] = { { .name = "cortex-a55", .initfn = aarch64_a55_initfn }, { .name = "cortex-a72", .initfn = aarch64_a72_initfn }, { .name = "cortex-a76", .initfn = aarch64_a76_initfn }, + { .name = "cortex-a710", .initfn = aarch64_a710_initfn }, { .name = "a64fx", .initfn = aarch64_a64fx_initfn }, { .name = "neoverse-n1", .initfn = aarch64_neoverse_n1_initfn }, { .name = "neoverse-v1", .initfn = aarch64_neoverse_v1_initfn }, -- 2.34.1 From: Richard Henderson <richard.henderson@linaro.org> Perform the check for EL2 enabled in the security space and the TIDCP bit in an out-of-line helper. Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20230831232441.66020-4-richard.henderson@linaro.org Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/helper.h | 1 + target/arm/tcg/op_helper.c | 13 +++++++++++++ target/arm/tcg/translate-a64.c | 16 ++++++++++++++-- target/arm/tcg/translate.c | 27 +++++++++++++++++++++++++++ 4 files changed, 55 insertions(+), 2 deletions(-) diff --git a/target/arm/helper.h b/target/arm/helper.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper.h +++ b/target/arm/helper.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_2(check_bxj_trap, TCG_CALL_NO_WG, void, env, i32) DEF_HELPER_4(access_check_cp_reg, cptr, env, i32, i32, i32) DEF_HELPER_FLAGS_2(lookup_cp_reg, TCG_CALL_NO_RWG_SE, cptr, env, i32) +DEF_HELPER_FLAGS_2(tidcp_el1, TCG_CALL_NO_WG, void, env, i32) DEF_HELPER_3(set_cp_reg, void, env, cptr, i32) DEF_HELPER_2(get_cp_reg, i32, env, cptr) DEF_HELPER_3(set_cp_reg64, void, env, cptr, i64) diff --git a/target/arm/tcg/op_helper.c b/target/arm/tcg/op_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/op_helper.c +++ b/target/arm/tcg/op_helper.c @@ -XXX,XX +XXX,XX @@ const void *HELPER(lookup_cp_reg)(CPUARMState *env, uint32_t key) return ri; } +/* + * Test for HCR_EL2.TIDCP at EL1. + * Since implementation defined registers are rare, and within QEMU + * most of them are no-op, do not waste HFLAGS space for this and + * always use a helper. + */ +void HELPER(tidcp_el1)(CPUARMState *env, uint32_t syndrome) +{ + if (arm_hcr_el2_eff(env) & HCR_TIDCP) { + raise_exception_ra(env, EXCP_UDEF, syndrome, 2, GETPC()); + } +} + void HELPER(set_cp_reg)(CPUARMState *env, const void *rip, uint32_t value) { const ARMCPRegInfo *ri = rip; diff --git a/target/arm/tcg/translate-a64.c b/target/arm/tcg/translate-a64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/translate-a64.c +++ b/target/arm/tcg/translate-a64.c @@ -XXX,XX +XXX,XX @@ static void handle_sys(DisasContext *s, bool isread, bool need_exit_tb = false; TCGv_ptr tcg_ri = NULL; TCGv_i64 tcg_rt; + uint32_t syndrome; + + if (crn == 11 || crn == 15) { + /* + * Check for TIDCP trap, which must take precedence over + * the UNDEF for "no such register" etc. + */ + syndrome = syn_aa64_sysregtrap(op0, op1, op2, crn, crm, rt, isread); + switch (s->current_el) { + case 1: + gen_helper_tidcp_el1(cpu_env, tcg_constant_i32(syndrome)); + break; + } + } if (!ri) { /* Unknown register; this might be a guest error or a QEMU @@ -XXX,XX +XXX,XX @@ static void handle_sys(DisasContext *s, bool isread, /* Emit code to perform further access permissions checks at * runtime; this may result in an exception. */ - uint32_t syndrome; - syndrome = syn_aa64_sysregtrap(op0, op1, op2, crn, crm, rt, isread); gen_a64_update_pc(s, 0); tcg_ri = tcg_temp_new_ptr(); diff --git a/target/arm/tcg/translate.c b/target/arm/tcg/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/translate.c +++ b/target/arm/tcg/translate.c @@ -XXX,XX +XXX,XX @@ void gen_gvec_uaba(unsigned vece, uint32_t rd_ofs, uint32_t rn_ofs, tcg_gen_gvec_3(rd_ofs, rn_ofs, rm_ofs, opr_sz, max_sz, &ops[vece]); } +static bool aa32_cpreg_encoding_in_impdef_space(uint8_t crn, uint8_t crm) +{ + static const uint16_t mask[3] = { + 0b0000000111100111, /* crn == 9, crm == {c0-c2, c5-c8} */ + 0b0000000100010011, /* crn == 10, crm == {c0, c1, c4, c8} */ + 0b1000000111111111, /* crn == 11, crm == {c0-c8, c15} */ + }; + + if (crn >= 9 && crn <= 11) { + return (mask[crn - 9] >> crm) & 1; + } + return false; +} + static void do_coproc_insn(DisasContext *s, int cpnum, int is64, int opc1, int crn, int crm, int opc2, bool isread, int rt, int rt2) @@ -XXX,XX +XXX,XX @@ static void do_coproc_insn(DisasContext *s, int cpnum, int is64, } } + if (cpnum == 15 && aa32_cpreg_encoding_in_impdef_space(crn, crm)) { + /* + * Check for TIDCP trap, which must take precedence over the UNDEF + * for "no such register" etc. It shares precedence with HSTR, + * but raises the same exception, so order doesn't matter. + */ + switch (s->current_el) { + case 1: + gen_helper_tidcp_el1(cpu_env, tcg_constant_i32(syndrome)); + break; + } + } + if (!ri) { /* * Unknown register; this might be a guest error or a QEMU -- 2.34.1 From: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20230831232441.66020-5-richard.henderson@linaro.org Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- docs/system/arm/emulation.rst | 1 + target/arm/cpu.h | 5 +++++ target/arm/helper.h | 1 + target/arm/tcg/cpu64.c | 1 + target/arm/tcg/op_helper.c | 20 ++++++++++++++++++++ target/arm/tcg/translate-a64.c | 5 +++++ target/arm/tcg/translate.c | 6 ++++++ 7 files changed, 39 insertions(+) diff --git a/docs/system/arm/emulation.rst b/docs/system/arm/emulation.rst index XXXXXXX..XXXXXXX 100644 --- a/docs/system/arm/emulation.rst +++ b/docs/system/arm/emulation.rst @@ -XXX,XX +XXX,XX @@ the following architecture extensions: - FEAT_SME_I16I64 (16-bit to 64-bit integer widening outer product instructions) - FEAT_SPECRES (Speculation restriction instructions) - FEAT_SSBS (Speculative Store Bypass Safe) +- FEAT_TIDCP1 (EL0 use of IMPLEMENTATION DEFINED functionality) - FEAT_TLBIOS (TLB invalidate instructions in Outer Shareable domain) - FEAT_TLBIRANGE (TLB invalidate range instructions) - FEAT_TTCNP (Translation table Common not private translations) diff --git a/target/arm/cpu.h b/target/arm/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -XXX,XX +XXX,XX @@ static inline bool isar_feature_aa64_hcx(const ARMISARegisters *id) return FIELD_EX64(id->id_aa64mmfr1, ID_AA64MMFR1, HCX) != 0; } +static inline bool isar_feature_aa64_tidcp1(const ARMISARegisters *id) +{ + return FIELD_EX64(id->id_aa64mmfr2, ID_AA64MMFR1, TIDCP1) != 0; +} + static inline bool isar_feature_aa64_uao(const ARMISARegisters *id) { return FIELD_EX64(id->id_aa64mmfr2, ID_AA64MMFR2, UAO) != 0; diff --git a/target/arm/helper.h b/target/arm/helper.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/helper.h +++ b/target/arm/helper.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_2(check_bxj_trap, TCG_CALL_NO_WG, void, env, i32) DEF_HELPER_4(access_check_cp_reg, cptr, env, i32, i32, i32) DEF_HELPER_FLAGS_2(lookup_cp_reg, TCG_CALL_NO_RWG_SE, cptr, env, i32) +DEF_HELPER_FLAGS_2(tidcp_el0, TCG_CALL_NO_WG, void, env, i32) DEF_HELPER_FLAGS_2(tidcp_el1, TCG_CALL_NO_WG, void, env, i32) DEF_HELPER_3(set_cp_reg, void, env, cptr, i32) DEF_HELPER_2(get_cp_reg, i32, env, cptr) diff --git a/target/arm/tcg/cpu64.c b/target/arm/tcg/cpu64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/cpu64.c +++ b/target/arm/tcg/cpu64.c @@ -XXX,XX +XXX,XX @@ void aarch64_max_tcg_initfn(Object *obj) t = FIELD_DP64(t, ID_AA64MMFR1, XNX, 1); /* FEAT_XNX */ t = FIELD_DP64(t, ID_AA64MMFR1, ETS, 1); /* FEAT_ETS */ t = FIELD_DP64(t, ID_AA64MMFR1, HCX, 1); /* FEAT_HCX */ + t = FIELD_DP64(t, ID_AA64MMFR1, TIDCP1, 1); /* FEAT_TIDCP1 */ cpu->isar.id_aa64mmfr1 = t; t = cpu->isar.id_aa64mmfr2; diff --git a/target/arm/tcg/op_helper.c b/target/arm/tcg/op_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/op_helper.c +++ b/target/arm/tcg/op_helper.c @@ -XXX,XX +XXX,XX @@ void HELPER(tidcp_el1)(CPUARMState *env, uint32_t syndrome) } } +/* + * Similarly, for FEAT_TIDCP1 at EL0. + * We have already checked for the presence of the feature. + */ +void HELPER(tidcp_el0)(CPUARMState *env, uint32_t syndrome) +{ + /* See arm_sctlr(), but we also need the sctlr el. */ + ARMMMUIdx mmu_idx = arm_mmu_idx_el(env, 0); + int target_el = mmu_idx == ARMMMUIdx_E20_0 ? 2 : 1; + + /* + * The bit is not valid unless the target el is aa64, but since the + * bit test is simpler perform that first and check validity after. + */ + if ((env->cp15.sctlr_el[target_el] & SCTLR_TIDCP) + && arm_el_is_aa64(env, target_el)) { + raise_exception_ra(env, EXCP_UDEF, syndrome, target_el, GETPC()); + } +} + void HELPER(set_cp_reg)(CPUARMState *env, const void *rip, uint32_t value) { const ARMCPRegInfo *ri = rip; diff --git a/target/arm/tcg/translate-a64.c b/target/arm/tcg/translate-a64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/translate-a64.c +++ b/target/arm/tcg/translate-a64.c @@ -XXX,XX +XXX,XX @@ static void handle_sys(DisasContext *s, bool isread, */ syndrome = syn_aa64_sysregtrap(op0, op1, op2, crn, crm, rt, isread); switch (s->current_el) { + case 0: + if (dc_isar_feature(aa64_tidcp1, s)) { + gen_helper_tidcp_el0(cpu_env, tcg_constant_i32(syndrome)); + } + break; case 1: gen_helper_tidcp_el1(cpu_env, tcg_constant_i32(syndrome)); break; diff --git a/target/arm/tcg/translate.c b/target/arm/tcg/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/translate.c +++ b/target/arm/tcg/translate.c @@ -XXX,XX +XXX,XX @@ static void do_coproc_insn(DisasContext *s, int cpnum, int is64, * but raises the same exception, so order doesn't matter. */ switch (s->current_el) { + case 0: + if (arm_dc_feature(s, ARM_FEATURE_AARCH64) + && dc_isar_feature(aa64_tidcp1, s)) { + gen_helper_tidcp_el0(cpu_env, tcg_constant_i32(syndrome)); + } + break; case 1: gen_helper_tidcp_el1(cpu_env, tcg_constant_i32(syndrome)); break; -- 2.34.1 From: Richard Henderson <richard.henderson@linaro.org> The linux kernel detects and enables this bit. Once trapped, EC_SYSTEMREGISTERTRAP is treated like EC_UNCATEGORIZED, so no changes required within linux-user/aarch64/cpu_loop.c. Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20230831232441.66020-6-richard.henderson@linaro.org Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- target/arm/cpu.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/target/arm/cpu.c b/target/arm/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/cpu.c +++ b/target/arm/cpu.c @@ -XXX,XX +XXX,XX @@ static void arm_cpu_reset_hold(Object *obj) SCTLR_EnDA | SCTLR_EnDB); /* Trap on btype=3 for PACIxSP. */ env->cp15.sctlr_el[1] |= SCTLR_BT0; + /* Trap on implementation defined registers. */ + if (cpu_isar_feature(aa64_tidcp1, cpu)) { + env->cp15.sctlr_el[1] |= SCTLR_TIDCP; + } /* and to the FP/Neon instructions */ env->cp15.cpacr_el1 = FIELD_DP64(env->cp15.cpacr_el1, CPACR_EL1, FPEN, 3); -- 2.34.1 From: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com> Now that we have Eager Page Split support added for ARM in the kernel, enable it in Qemu. This adds, -eager-split-size to -accel sub-options to set the eager page split chunk size. -enable KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE. The chunk size specifies how many pages to break at a time, using a single allocation. Bigger the chunk size, more pages need to be allocated ahead of time. Reviewed-by: Gavin Shan <gshan@redhat.com> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com> Message-id: 20230905091246.1931-1-shameerali.kolothum.thodi@huawei.com Signed-off-by: Peter Maydell <peter.maydell@linaro.org> --- include/sysemu/kvm_int.h | 1 + accel/kvm/kvm-all.c | 1 + target/arm/kvm.c | 61 ++++++++++++++++++++++++++++++++++++++++ qemu-options.hx | 15 ++++++++++ 4 files changed, 78 insertions(+) diff --git a/include/sysemu/kvm_int.h b/include/sysemu/kvm_int.h index XXXXXXX..XXXXXXX 100644 --- a/include/sysemu/kvm_int.h +++ b/include/sysemu/kvm_int.h @@ -XXX,XX +XXX,XX @@ struct KVMState uint64_t kvm_dirty_ring_bytes; /* Size of the per-vcpu dirty ring */ uint32_t kvm_dirty_ring_size; /* Number of dirty GFNs per ring */ bool kvm_dirty_ring_with_bitmap; + uint64_t kvm_eager_split_size; /* Eager Page Splitting chunk size */ struct KVMDirtyRingReaper reaper; NotifyVmexitOption notify_vmexit; uint32_t notify_window; diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index XXXXXXX..XXXXXXX 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -XXX,XX +XXX,XX @@ static void kvm_accel_instance_init(Object *obj) /* KVM dirty ring is by default off */ s->kvm_dirty_ring_size = 0; s->kvm_dirty_ring_with_bitmap = false; + s->kvm_eager_split_size = 0; s->notify_vmexit = NOTIFY_VMEXIT_OPTION_RUN; s->notify_window = 0; s->xen_version = 0; diff --git a/target/arm/kvm.c b/target/arm/kvm.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/kvm.c +++ b/target/arm/kvm.c @@ -XXX,XX +XXX,XX @@ #include "exec/address-spaces.h" #include "hw/boards.h" #include "hw/irq.h" +#include "qapi/visitor.h" #include "qemu/log.h" const KVMCapabilityInfo kvm_arch_required_capabilities[] = { @@ -XXX,XX +XXX,XX @@ int kvm_arch_init(MachineState *ms, KVMState *s) } } + if (s->kvm_eager_split_size) { + uint32_t sizes; + + sizes = kvm_vm_check_extension(s, KVM_CAP_ARM_SUPPORTED_BLOCK_SIZES); + if (!sizes) { + s->kvm_eager_split_size = 0; + warn_report("Eager Page Split support not available"); + } else if (!(s->kvm_eager_split_size & sizes)) { + error_report("Eager Page Split requested chunk size not valid"); + ret = -EINVAL; + } else { + ret = kvm_vm_enable_cap(s, KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE, 0, + s->kvm_eager_split_size); + if (ret < 0) { + error_report("Enabling of Eager Page Split failed: %s", + strerror(-ret)); + } + } + } + kvm_arm_init_debug(s); return ret; @@ -XXX,XX +XXX,XX @@ bool kvm_arch_cpu_check_are_resettable(void) return true; } +static void kvm_arch_get_eager_split_size(Object *obj, Visitor *v, + const char *name, void *opaque, + Error **errp) +{ + KVMState *s = KVM_STATE(obj); + uint64_t value = s->kvm_eager_split_size; + + visit_type_size(v, name, &value, errp); +} + +static void kvm_arch_set_eager_split_size(Object *obj, Visitor *v, + const char *name, void *opaque, + Error **errp) +{ + KVMState *s = KVM_STATE(obj); + uint64_t value; + + if (s->fd != -1) { + error_setg(errp, "Unable to set early-split-size after KVM has been initialized"); + return; + } + + if (!visit_type_size(v, name, &value, errp)) { + return; + } + + if (value && !is_power_of_2(value)) { + error_setg(errp, "early-split-size must be a power of two"); + return; + } + + s->kvm_eager_split_size = value; +} + void kvm_arch_accel_class_init(ObjectClass *oc) { + object_class_property_add(oc, "eager-split-size", "size", + kvm_arch_get_eager_split_size, + kvm_arch_set_eager_split_size, NULL, NULL); + + object_class_property_set_description(oc, "eager-split-size", + "Eager Page Split chunk size for hugepages. (default: 0, disabled)"); } diff --git a/qemu-options.hx b/qemu-options.hx index XXXXXXX..XXXXXXX 100644 --- a/qemu-options.hx +++ b/qemu-options.hx @@ -XXX,XX +XXX,XX @@ DEF("accel", HAS_ARG, QEMU_OPTION_accel, " split-wx=on|off (enable TCG split w^x mapping)\n" " tb-size=n (TCG translation block cache size)\n" " dirty-ring-size=n (KVM dirty ring GFN count, default 0)\n" + " eager-split-size=n (KVM Eager Page Split chunk size, default 0, disabled. ARM only)\n" " notify-vmexit=run|internal-error|disable,notify-window=n (enable notify VM exit and set notify window, x86 only)\n" " thread=single|multi (enable multi-threaded TCG)\n", QEMU_ARCH_ALL) SRST @@ -XXX,XX +XXX,XX @@ SRST is disabled (dirty-ring-size=0). When enabled, KVM will instead record dirty pages in a bitmap. + ``eager-split-size=n`` + KVM implements dirty page logging at the PAGE_SIZE granularity and + enabling dirty-logging on a huge-page requires breaking it into + PAGE_SIZE pages in the first place. KVM on ARM does this splitting + lazily by default. There are performance benefits in doing huge-page + split eagerly, especially in situations where TLBI costs associated + with break-before-make sequences are considerable and also if guest + workloads are read intensive. The size here specifies how many pages + to break at a time and needs to be a valid block size which is + 1GB/2MB/4KB, 32MB/16KB and 512MB/64KB for 4KB/16KB/64KB PAGE_SIZE + respectively. Be wary of specifying a higher size as it will have an + impact on the memory. By default, this feature is disabled + (eager-split-size=0). + ``notify-vmexit=run|internal-error|disable,notify-window=n`` Enables or disables notify VM exit support on x86 host and specify the corresponding notify window to trigger the VM exit if enabled. -- 2.34.1