From nobody Wed May 8 19:54:16 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1604666694; cv=none; d=zohomail.com; s=zohoarc; b=Yov8YNTWkG2cNVUVTRI5NYS1Gn/keEjLwpNoYPP5UdPv/S3akbYmS+46geUmcML+HIX8WwyRniBClOljdb42fvAs/bEe0YLjKC/fMDkbTeLggP8V1iTt0QEhCXd0Zqw8OOWtvau7F3l35FIyxaplSmvSOZPlM/8lO3Wzu6jRF4o= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1604666694; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:Reply-To:References:Sender:Subject:To; bh=j3SKRXBdexUuDbCbzZbNTLHCTFpWBd8K3jdx0eTp/K8=; b=gIJzLbMNpMW3emtImWf4IrarwCBLyfKmQvcy5nXeflSVgpZJ1sRrejZupjbVUXIxHazNPPgXmoGkWqgj59Dzpmhh7+2WPfIc82YQKxYp0VCiAHnGZATu6P8CYIjBFEdqV/ggbwo1NfaUvtw/PXcPEBfzKuHwGAr2h2PHvW1oPqw= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1604666694162676.9946056241463; Fri, 6 Nov 2020 04:44:54 -0800 (PST) Received: from localhost ([::1]:41656 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kb16r-0005H3-3u for importer@patchew.org; Fri, 06 Nov 2020 07:44:53 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:59908) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kb154-0002ki-6S; Fri, 06 Nov 2020 07:43:02 -0500 Received: from relay.sw.ru ([185.231.240.75]:48164 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kb14w-0001LB-VT; Fri, 06 Nov 2020 07:43:01 -0500 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.94) (envelope-from ) id 1kb14T-007cZw-UE; Fri, 06 Nov 2020 15:42:25 +0300 To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, jcody@redhat.com, kwolf@redhat.com, mreitz@redhat.com, armbru@redhat.com, dgilbert@redhat.com, pbonzini@redhat.com, eblake@redhat.com, marcandre.lureau@redhat.com, den@openvz.org, vsementsov@virtuozzo.com, andrey.shinkevich@virtuozzo.com Subject: [PATCH 1/2] iotests: add another bash sleep command to 247 Date: Fri, 6 Nov 2020 15:42:01 +0300 Message-Id: <1604666522-545580-2-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1604666522-545580-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1604666522-545580-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/11/06 07:42:29 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Reply-to: Andrey Shinkevich From: Andrey Shinkevich via Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This patch paves the way for the one that follows. The following patch makes the QMP monitor to read up to 4K from stdin at once. That results in running the bash 'sleep' command before the _qemu_proc_exec() starts in subshell. Another 'sleep' command with an unobtrusive 'query-status' plays as a workaround. Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/247 | 2 ++ tests/qemu-iotests/247.out | 1 + 2 files changed, 3 insertions(+) diff --git a/tests/qemu-iotests/247 b/tests/qemu-iotests/247 index 87e37b3..7d316ec 100755 --- a/tests/qemu-iotests/247 +++ b/tests/qemu-iotests/247 @@ -59,6 +59,8 @@ TEST_IMG=3D"$TEST_IMG.4" _make_test_img $size {"execute":"block-commit", "arguments":{"device":"format-4", "top-node": "format-2", "base-node":"fo= rmat-0", "job-id":"job0"}} EOF +sleep 1 +echo '{"execute":"query-status"}' if [ "${VALGRIND_QEMU}" =3D=3D "y" ]; then sleep 10 else diff --git a/tests/qemu-iotests/247.out b/tests/qemu-iotests/247.out index e909e83..13d9547 100644 --- a/tests/qemu-iotests/247.out +++ b/tests/qemu-iotests/247.out @@ -17,6 +17,7 @@ QMP_VERSION {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event"= : "BLOCK_JOB_COMPLETED", "data": {"device": "job0", "len": 134217728, "offs= et": 134217728, "speed": 0, "type": "commit"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event"= : "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": "job0"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event"= : "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "job0"}} +{"return": {"status": "running", "singlestep": false, "running": true}} {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event"= : "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}} *** done --=20 1.8.3.1 From nobody Wed May 8 19:54:16 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1604666911; cv=none; d=zohomail.com; s=zohoarc; b=ag+B3kf9mOjQLC15WnO6IgTuhEg3TWIv529YWHraFq53O254F8yV4oTmCq8pcpiCVQkHbzi/NuHOeRmu/9JL08HkL0asrf85wwmoRQMifyEpeZluhmZDeNa/Y9HxF6h5o4JJ+wWXI4crJfKB0FMgvgdTrfiL1mx6JXaFkgIaDPI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1604666911; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:Reply-To:References:Sender:Subject:To; bh=hvqtltfltdSSHqZ7Bl8fmU0TyJ97E7l1WqdmGlZ6ghY=; b=GfmXUV/CmjeLz959CQdwlJu60JSn7Kny3MokBI1GqBofq7cLinbQZeWYvTalNC9kkeqsm5yLcT6TgQn8mYcWnOK60qNw9WEVNAevchBZKgfQlQ4bozi6VKz3cJNFjqdLBzGXMI6uoaraJAJeG8ONmxq2uV+udOCVjiS0VNOnjus= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 16046669110461006.5850062663036; Fri, 6 Nov 2020 04:48:31 -0800 (PST) Received: from localhost ([::1]:56590 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kb1AL-000317-LW for importer@patchew.org; Fri, 06 Nov 2020 07:48:29 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:60062) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kb15O-00034n-Be; Fri, 06 Nov 2020 07:43:22 -0500 Received: from relay.sw.ru ([185.231.240.75]:48318 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kb15K-0001Tf-EK; Fri, 06 Nov 2020 07:43:21 -0500 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.94) (envelope-from ) id 1kb14r-007cZw-Fg; Fri, 06 Nov 2020 15:42:49 +0300 To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, jcody@redhat.com, kwolf@redhat.com, mreitz@redhat.com, armbru@redhat.com, dgilbert@redhat.com, pbonzini@redhat.com, eblake@redhat.com, marcandre.lureau@redhat.com, den@openvz.org, vsementsov@virtuozzo.com, andrey.shinkevich@virtuozzo.com Subject: [PATCH 2/2] monitor: increase amount of data for monitor to read Date: Fri, 6 Nov 2020 15:42:02 +0300 Message-Id: <1604666522-545580-3-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1604666522-545580-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1604666522-545580-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/11/06 07:42:29 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Reply-to: Andrey Shinkevich From: Andrey Shinkevich via Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" QMP and HMP monitors read one byte at a time from the socket or stdin, which is very inefficient. With 100+ VMs on the host, this results in multiple extra system calls and CPU overuse. This patch increases the amount of read data up to 4096 bytes that fits the buffer size on the channel level. Suggested-by: Denis V. Lunev Signed-off-by: Andrey Shinkevich --- chardev/char-fd.c | 64 ++++++++++++++++++++++++++++++++++++++++++= +++- chardev/char-socket.c | 54 +++++++++++++++++++++++++++----------- chardev/char.c | 40 +++++++++++++++++++++++++++++ include/chardev/char.h | 15 +++++++++++ monitor/monitor.c | 2 +- tests/qemu-iotests/247.out | 2 +- 6 files changed, 159 insertions(+), 18 deletions(-) diff --git a/chardev/char-fd.c b/chardev/char-fd.c index 1cd62f2..6194fe6 100644 --- a/chardev/char-fd.c +++ b/chardev/char-fd.c @@ -33,6 +33,8 @@ #include "chardev/char-fd.h" #include "chardev/char-io.h" =20 +#include "monitor/monitor-internal.h" + /* Called with chr_write_lock held. */ static int fd_chr_write(Chardev *chr, const uint8_t *buf, int len) { @@ -41,7 +43,7 @@ static int fd_chr_write(Chardev *chr, const uint8_t *buf,= int len) return io_channel_send(s->ioc_out, buf, len); } =20 -static gboolean fd_chr_read(QIOChannel *chan, GIOCondition cond, void *opa= que) +static gboolean fd_chr_read_hmp(QIOChannel *chan, void *opaque) { Chardev *chr =3D CHARDEV(opaque); FDChardev *s =3D FD_CHARDEV(opaque); @@ -71,6 +73,66 @@ static gboolean fd_chr_read(QIOChannel *chan, GIOConditi= on cond, void *opaque) return TRUE; } =20 +static gboolean fd_chr_read_qmp(QIOChannel *chan, void *opaque) +{ + static JSONthrottle thl =3D {0}; + uint8_t *start; + Chardev *chr =3D CHARDEV(opaque); + FDChardev *s =3D FD_CHARDEV(opaque); + int len, size, pos; + ssize_t ret; + + if (!thl.load) { + len =3D sizeof(thl.buf); + if (len > s->max_size) { + len =3D s->max_size; + } + if (len =3D=3D 0) { + return TRUE; + } + + ret =3D qio_channel_read( + chan, (gchar *)thl.buf, len, NULL); + if (ret =3D=3D 0) { + remove_fd_in_watch(chr); + qemu_chr_be_event(chr, CHR_EVENT_CLOSED); + thl =3D (const JSONthrottle){0}; + return FALSE; + } + if (ret < 0) { + return TRUE; + } + thl.load =3D ret; + thl.cursor =3D 0; + } + + size =3D thl.load; + start =3D thl.buf + thl.cursor; + pos =3D qemu_chr_end_position((const char *) start, size, &thl); + if (pos >=3D 0) { + size =3D pos + 1; + } + + qemu_chr_be_write(chr, start, size); + thl.cursor +=3D size; + thl.load -=3D size; + + return TRUE; +} + +static gboolean fd_chr_read(QIOChannel *chan, GIOCondition cond, void *opa= que) +{ + Chardev *chr =3D CHARDEV(opaque); + CharBackend *be =3D chr->be; + Monitor *mon =3D (Monitor *)be->opaque; + + if (monitor_is_qmp(mon)) { + return fd_chr_read_qmp(chan, opaque); + } + + return fd_chr_read_hmp(chan, opaque); +} + static int fd_chr_read_poll(void *opaque) { Chardev *chr =3D CHARDEV(opaque); diff --git a/chardev/char-socket.c b/chardev/char-socket.c index 213a4c8..8335e8c 100644 --- a/chardev/char-socket.c +++ b/chardev/char-socket.c @@ -520,30 +520,54 @@ static void tcp_chr_disconnect(Chardev *chr) =20 static gboolean tcp_chr_read(QIOChannel *chan, GIOCondition cond, void *op= aque) { + static JSONthrottle thl =3D {0}; + uint8_t *start; Chardev *chr =3D CHARDEV(opaque); SocketChardev *s =3D SOCKET_CHARDEV(opaque); - uint8_t buf[CHR_READ_BUF_LEN]; - int len, size; + int len, size, pos; =20 if ((s->state !=3D TCP_CHARDEV_STATE_CONNECTED) || s->max_size <=3D 0) { return TRUE; } - len =3D sizeof(buf); - if (len > s->max_size) { - len =3D s->max_size; - } - size =3D tcp_chr_recv(chr, (void *)buf, len); - if (size =3D=3D 0 || (size =3D=3D -1 && errno !=3D EAGAIN)) { - /* connection closed */ - tcp_chr_disconnect(chr); - } else if (size > 0) { - if (s->do_telnetopt) { - tcp_chr_process_IAC_bytes(chr, s, buf, &size); + + if (!thl.load) { + len =3D sizeof(thl.buf); + if (len > s->max_size) { + len =3D s->max_size; + } + size =3D tcp_chr_recv(chr, (void *)thl.buf, len); + if (size =3D=3D 0 || (size =3D=3D -1 && errno !=3D EAGAIN)) { + /* connection closed */ + tcp_chr_disconnect(chr); + thl =3D (const JSONthrottle){0}; + return TRUE; } - if (size > 0) { - qemu_chr_be_write(chr, buf, size); + if (size < 0) { + return TRUE; } + thl.load =3D size; + thl.cursor =3D 0; + } + + size =3D thl.load; + start =3D thl.buf + thl.cursor; + pos =3D qemu_chr_end_position((const char *) start, size, &thl); + if (pos >=3D 0) { + size =3D pos + 1; + } + len =3D size; + + if (s->do_telnetopt) { + tcp_chr_process_IAC_bytes(chr, s, start, &size); + } + if (size > 0) { + qemu_chr_be_write(chr, start, size); + thl.cursor +=3D size; + thl.load -=3D size; + } else { + thl.cursor +=3D len; + thl.load -=3D len; } =20 return TRUE; diff --git a/chardev/char.c b/chardev/char.c index aa42821..251c97c 100644 --- a/chardev/char.c +++ b/chardev/char.c @@ -1178,6 +1178,46 @@ GSource *qemu_chr_timeout_add_ms(Chardev *chr, guint= ms, return source; } =20 +/* + * Split the incoming buffered stream so that the QMP monitor queue is not + * overwhelmed with requests. The function looks for the last paired + * brace/bracket in a JSON command. + */ +int qemu_chr_end_position(const char *buf, int size, JSONthrottle *thl) +{ + int i; + + for (i =3D 0; i < size; i++) { + switch (buf[i]) { + case ' ': + case '\n': + case '\r': + continue; + case '{': + thl->brace_count++; + break; + case '}': + thl->brace_count--; + break; + case '[': + thl->bracket_count++; + break; + case ']': + thl->bracket_count--; + break; + default: + break; + } + /* The same condition as it is in the json_message_process_token()= */ + if ((thl->brace_count > 0 || thl->bracket_count > 0) + && thl->brace_count >=3D 0 && thl->bracket_count >=3D 0) { + continue; + } + return i; + } + return -1; +} + void qemu_chr_cleanup(void) { object_unparent(get_chardevs_root()); diff --git a/include/chardev/char.h b/include/chardev/char.h index db42f0a..5f02731 100644 --- a/include/chardev/char.h +++ b/include/chardev/char.h @@ -70,6 +70,14 @@ struct Chardev { DECLARE_BITMAP(features, QEMU_CHAR_FEATURE_LAST); }; =20 +typedef struct { + uint8_t buf[CHR_READ_BUF_LEN]; + int load; + int cursor; + int brace_count; + int bracket_count; +} JSONthrottle; + /** * qemu_chr_new_from_opts: * @opts: see qemu-config.c for a list of valid options @@ -141,6 +149,13 @@ Chardev *qemu_chr_new_mux_mon(const char *label, const= char *filename, void qemu_chr_change(QemuOpts *opts, Error **errp); =20 /** + * Split the incoming buffered stream so that the QMP monitor queue is not + * overwhelmed with requests. The function looks for the last paired + * brace/bracket in a JSON command. + */ +int qemu_chr_end_position(const char *buf, int size, JSONthrottle *thl); + +/** * qemu_chr_cleanup: * * Delete all chardevs (when leaving qemu) diff --git a/monitor/monitor.c b/monitor/monitor.c index 84222cd..43d2d3b 100644 --- a/monitor/monitor.c +++ b/monitor/monitor.c @@ -566,7 +566,7 @@ int monitor_can_read(void *opaque) { Monitor *mon =3D opaque; =20 - return !qatomic_mb_read(&mon->suspend_cnt); + return !qatomic_mb_read(&mon->suspend_cnt) ? CHR_READ_BUF_LEN : 0; } =20 void monitor_list_append(Monitor *mon) diff --git a/tests/qemu-iotests/247.out b/tests/qemu-iotests/247.out index 13d9547..2758662 100644 --- a/tests/qemu-iotests/247.out +++ b/tests/qemu-iotests/247.out @@ -12,12 +12,12 @@ QMP_VERSION {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event"= : "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "job0"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event"= : "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "job0"}} {"return": {}} +{"return": {"status": "running", "singlestep": false, "running": true}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event"= : "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "job0"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event"= : "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "job0"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event"= : "BLOCK_JOB_COMPLETED", "data": {"device": "job0", "len": 134217728, "offs= et": 134217728, "speed": 0, "type": "commit"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event"= : "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": "job0"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event"= : "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "job0"}} -{"return": {"status": "running", "singlestep": false, "running": true}} {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event"= : "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}} *** done --=20 1.8.3.1